Iran or North Korea More of a Threat? It Depends…

If anyone doubts that "public opinion" is impossible to measure precisely, consider the following. In a survey conducted Nov. 19-21, Gallup asked whether Iran and, separately, North Korea, represented a threat to the United States. The exact wording of the question and the results are shown below:

Which comes closest to your view -- (Iran/North Korea) poses an immediate threat to the United States, (Iran/North Korea) poses a long-term threat to the U.S., but not an immediate threat, or (Iran/North Korea) does not pose a threat to the United States at all?

In half the sample (Form A), Gallup asked first about Iran and then North Korea; and in the other half of the sample (Form B), Gallup asked about North Korea and then Iran. The overall results found 23% saying Iran was an immediate threat, and 20% saying that about North Korea. About 6 in 10 Americans saw each country as a long-term threat, while 14% to 15% said each country posed no threat. Our conclusion: Americans saw little difference in the threat each country poses.

Question Order Influences "Public Opinion"

The order of asking the questions greatly influenced people's judgment as to whether each country was an immediate or a long-term threat, although question order had essentially no effect on the percentage who said each country posed no threat.

In Form A, when Iran was mentioned first, 16% said Iran was an immediate threat. In Form B, when Iran was mentioned after North Korea, 30% said Iran was an immediate threat. Mentioning North Korea first boosted the percentage of people saying Iran is an immediate threat.

A similar effect occurred in the public's threat assessment of North Korea. In Form B, when North Korea was mentioned first, just 16% said that country was an immediate threat. But when North Korea was mentioned after Iran, 23% said North Korea was an immediate threat.

Thus, for both countries, the percentage of people saying the country was an immediate threat increased after the other country had been mentioned. For Iran, the increase was 14 percentage points (from 16% when mentioned first to 30% when mentioned second), while for North Korea, the increase was a more modest 7 percentage points (from 16% to 23%).

These results illustrate how influential a questionnaire design can be on the opinion that polls measure. There are at least three other ways the questions could have been asked besides what Gallup did (Design #4 below), and the results would have all given a different view of public opinion.

• Design #1: Often, when there are only two similar items that need to be evaluated, polling organizations will conduct a split sample, with half the sample asked only about one item and the other half asked about the other. If Gallup had done that for this survey, we would have reported that Americans see no difference in threat of each country to the United States. Sixteen percent each see Iran and, separately, North Korea, as an immediate threat.
  
  
• Design #2: Many polling organizations arbitrarily choose to list one item first and the other item second. Had Iran been listed first, followed by North Korea, Gallup would have reported that more people felt North Korea was an immediate threat (23%) than felt that about Iran (16%).
  
  
• Design #3: Had North Korea been listed first, followed by Iran, the opposite conclusion would have been reported: More people said Iran was a threat (30%) than said that about North Korea (16%).
  
  
• Design #4: Finally, in the questionnaire version actually used by Gallup, the results show that a slightly larger percentage of Americans said Iran was a threat (23%) than said that about North Korea (20%).

Note the major differences between Design #2 and Design #3 -- either more people see Iran as a threat (by 14 points) or more people see North Korea as a threat (by 7 points). Then there is Design #1, which says people see no differences between the two countries, and Design #4, which says there are only slight differences.

How do we interpret these conflicting results? Which should we report in our news articles? And do these four different results invalidate public opinion polls in general?

In next week's article, I will argue that rather than invalidate polls, the different percentages provide useful insights into the dynamics of public opinion. But the information reported here demonstrates why different polling organizations can report diametrically opposed results, even though interviewing could have occurred at the same time and with identical sample characteristics. Public opinion, it turns out, is much more enigmatic than most people believe.