Why do people who watch TV typically watch so much of it, and why do people who stop watching TV do so? And why do people who don't watch TV often instinctively have such a poor opinion of it? After all, they usually used to watch it too, didn't they?
For the purpose of explaining my theory, I'm going to start with the conclusion I reached and work back from there there:
People dislike TV because it's an inherently sensationalist, emotive and passive medium. It positively discourages critical thought and subtly but definitely encourages a receptive, passive, uncritical mindset.
Constrained input frequency
Television drip-feeds you information without any activity on your part - indeed, you're better off not moving or doing anything else, as any activity on your part will only distract from the medium. Unlike reading (where you can read as fast as you feel comfortable), the speed of information-flow is limited by the television. This encourages a passive, receptive mindset, as there's literally nothing you can do to affect the incoming information flow without degrading it (e.g., by fast-forwarding).
Water, water everywhere...
Compare the amount of time and raw information in the written sentence
"The man walked twenty metres down the road."
compared with a video imparting exactly the same thing. Also compare the amount of useful, important data in the written sentence versus the sheer volume of unimportant information you have to assimilate from a video to get the same amount of data.
Nevertheless, with TV you still have to sift through all this unimportant information to select out the important parts, and this extra cognitive workload impairs and discourages any other thoughts you may be having. Additionally, having to handle such a comparatively large amount of input in real-time at least reduces the amount of attention you have to consider, analyse and critically evaluate what you're receiving.
Compared to other media
Reading, in contrast, is like accessing pre-filtered meaning - very little written text doesn't relate data essential to the communication, and text which does break this rule quickly becomes dull, windy and boring. Text is low-bandwidth (lower even than radio) and isn't inherently interesting to look at, so since the form won't hold your interest the content is required to be more interesting. Written text typically has a higher ratio of data-to-information - analogous to the idea of the signal:noise ratio in electronics.
Likewise, while radio has many of the same faults as TV (no random access, constrained input frequency) its lower bandwidth (audio compared to video) also communicates meaning much more efficiently than television - again, it has a higher data:information ratio.
Poor at communicating meaning
TV therefore communicates surprisingly inefficiently, in terms of the amount of raw information you have to sift through to extract meaning. This means that while it typically requires a large amount of attention to parse out the incoming information, it imparts relatively little actual data or knowledge... and what data it does impart is drip-fed to you at a rate much slower than you could typically assimilate it if it were presented in a more condensed or refined form.
Finally, its imposed (and slow) rhythm, high-bandwidth but data-poor input and a complete lack of interactivity (indeed, a disincentive to any activity at all) means no matter what you're watching, the medium itself acts to cultivate a more passive, receptive and uncritical mindset than you would otherwise experience.
Don't get me wrong - obviously there are plenty of situations where information could only sensibly be transmitted in a video medium (sports events, any communication where movement and visual change over time are important aspects, etc), and in any one particular instance these effects are typically very small. However, when you compare the very nature of the medium of TV to other media (books, the web... arguably even computer games) it's hard not to come to the conclusion that it's the least interactive, least efficient and most unchallenging of all the mainstream media we typically use.
 Finally, if any extra detail is required or desired by the receiver (What man? How old was he? Was he wearing a hat? What colour?), compare the amount of mental exercise required be the receiver to imagine all these things compared to merely observing them.
 But the rather more worrying question is: are they cumulative? If one gets used to regularly existing in a passive, receptive, uncritical mindset does that make it easier (and more common) to to experience it in future? Obviously I'm not trying to claim TV turns people into mooing idiots or anything so excessive... nevertheless, at the very least it raises interesting questions...