"In Sea of Data, Not All Numbers Are Equal"

Read the full article by Jon Cohen in The Washington Post, November 14, 2006. Cohen is The Post's director of polling.

Excerpts:

One vogue approach to the glut of polls this year was to surrender judgment, assume all polls were equal and average their findings. Political junkies bookmarked Web sites that aggregated polls and posted five- and 10-poll averages.

But, perhaps unsurprisingly, averages work only "on average." For example, the posted averages on the Maryland governor's and Senate races showed them as closely competitive; they were not. Polls from The Post and Gallup showed those races as solidly Democratic in June, September and October, just as they were on Election Day.......

averaging polls encourages the already excessive attention paid to horse-race numbers. Preelection polls are not meant to be crystal balls. Putting a number on the status of the race is a necessary part of preelection polls, but much is lost if it's the only one.

We need standards, not averages. There's certainly a place for averages. My investment portfolio, for example, would be in better shape today if I had invested in broad indexes of securities instead of fancying myself a stock-picker. At the same time, I'd be in a much tighter financial position if I took investment advice from spam e-mails as seriously as that from accredited financial experts.......

Pollsters sometimes disagree about how to conduct surveys, but the high-quality polling we should pay attention to is based on an established method undergirded by statistical theory.

The gold standard in news polling remains interviewers making telephone calls to people randomly selected from a sample of a definable, reachable population. To be sure, the luster on the method is not as shiny as it once was, but I'd always choose tarnished precious metals over fool's gold.

Before anyone feels condemned to night classes and bell curves to sort through the glut of polls, let me say that the primary filtering burden should rest with the news media. It's ironic in a field that prides itself on sorting reliable sources from bogus ones that so many treat all numbers -- including poll estimates -- equally, and as valid on their face.

News organizations should be aware that they give immediate credibility to the "facts" they air or print. But this is not an argument for some sort of media monopoly on polling information.

Comment: I agree! JAD