Why? Well that part is complicated.
It's obvious why it's rude to burp: you're subjecting people to something they don't want to hear.
It's obvious why it's rude to interrupt someone: you're treating them as though they are unimportant, and being disrespectful.
But whispering to the person next to you isn't doing either of those things. You're not subjecting the group to anything, and you're not interfering with their speaking.
Some would say that whispering is rude because it gives the impression that you're talking about another person in the room. But that theory doesn't hold up, because we have a similar negative reaction to any "1 to 1" communication in a public venue, even if the communication isn't "whispered," or otherwise obscured.
For instance, it's rude to discuss topics that the group isn't in on. Even though everyone can hear the discussion, and are free to talk amongst themselves, they take it personally that a private conversation is going on in their presence. The classic example of this is "shop-talk," where people that work together talk about their jobs in social settings.
The real problem, then, isn't one of paranoia. It's an issue of "us" and "them."
When you "whisper," you draw a line in the social sand. By creating private communication in the midst of public communication, you inadvertently create a class structure. You've made a distinction between people, which is something people often resent.
Which brings me, quite naturally, to the Walkman.
Technology provides lots of new opportunities for 1:1 communication, and each time a new one pops up we have to reconcile it socially. When cell phones became common there was something of a backlash against them, if you used your cell in public people thought you were self-important.
Portable music devices (which, as time goes on, are also portable video devices) are an interesting case of 1:1 communication. Traditionally, entertainment has been communal in nature because the only efficient way to send it out was to a group. But as technology advances, entertainment has become more personal.*
And I wonder how we'll reconcile it.
*Except for reading, which has always been a 1:1 thing. Imagine how bad it must have been when the skill of reading became common! Not only was a person engaging in a 1:1 communication, it was a form of expression that many people couldn't interpret.