Reading Polls Carefully

There is a new Harris Poll that is presented as demonstrating that there is very wide spread belief in the United States in a number of propositions about Barack Obama that seem to be patently ridiculous. An article by the Director of Polling at ABC News points out that the Harris Poll methods were seriously flawed, including poor sampling methodology:

The poll starts by telling respondents “here are some things people have said about President Obama,” then asking if they think each is true or false. Fifteen statements follow, with all (excluding “he is a Muslim”) unrelentingly negative. “True” answers run from a high of 40 percent, for “he is a socialist,” to a low of 13 percent, for “he wants the terrorists to win.”

The problems are fundamental. “Some people have said” is a biasing introductory phrase; it imbues the subsequent statements with an air of credibility – particularly when you don’t note that others say something else. (That approach can have problems of its own; the “some people” vs. “other people” format implies equivalence.)

The subsequent statements, for their part, are classically unbalanced – there’s no alternative proposition to consider. A wealth of academic literature, neatly summarized here, demonstrates that questions constructed in this fashion – true/false, agree/disagree – carry a heavy dose of what’s known as acquiescence bias. They overstate agreement with whatever’s been posited, often by a very substantial margin. (This reflects avoidance of cognitive burden, which tends to happen disproportionately with less-educated respondents, as is reflected in Harris’ results.)

Using all negative statements, rather than a mix of negative and positive ones, reflects another non-standard approach, one that can further bias responses. (The ordering of items, unclear in the Harris release, can be troublesome as well.)

The criticism reminds us that we have to be careful about believing polling results since not only can polls be misleading, but it is possible for polling organizations to deliberately create polls that leave a mistaken impression in the unwary reader.

On the other hand, the poll does seem to be credible in suggesting that the more education held by the respondent, the less likely the respondent was to give foolish answers to the items in the poll.