Baby is sick and not getting better?

We took our son to the doctors because of what we thought was just a cold and turns out he also had an ear infection. The doctor gave him Amoxicillin and he started taking that 3 days ago. Our son has not gotten an ounce better yet and sounds like his lungs are even more congested. Am I just being paranoid? At what point should I take him back to the doctors? Thanks