Wrong, the breaking of your eardrums is what fuckes you over. you falling to the ground is what can fuck your brain over.
You balance is mainly controlled by your eyes and ears. Those 2 signals have to match, if they don't (like on a ship, where your eyes say you are not moving, but your ears say you are) then you get sick. Loose balance and throw up and shit.
If the ears are damages to much, like lets say an explosions, then you have the same, just a lot worse. You fall to the ground since you just lost all your balance, feel like shit and probably even throw up.
Hence people on the ground crawling and deaf when you throw a large pressure wave in there direction.
Not 100% sure of this, but breaking an eardrum is sometimes also labeled as having a concussion, since it is very similar in many ways. The brain getting smashed into you own skull isn't really and instant death, and is mostly only dangerous with side impacts. It takes a few hours for the brain to swell and cause serious problems. (unless you just plowed a hammer in your head, but then it is more the pieces of skull that cut there way into your brain that are the problem)
Also, we are talking about air pressure and the rapid change of it, a boxer hitting you in the face has a lot more force. people who died of overpressure rarely show visible marks on the outside. Just blood in the mouth and ears.
If you want to compare it with a hit to the head, then i propose hitting someone with a flat hand where your wrist hits the temple and your flat hand the ear, one semi hard blow can give a person a concussion and make him go down and stay down. Saw someone do it about 5 years ago, i was rather amazed.
If you want to counter the presure, close eyes, nose, ears and mouth.