What We Learned in Nashville

PRINCETON, NJ -- Author Joe Klein recently made the following assertion in his TIME Magazine cover story about Democrats: "Polling is not much of a science when only 5% of people contacted by phone -- that’s the current average -- actually agree to answer questions." That sounds pretty damaging, except that no one in the polling business knows where that figure comes from. As far as we know, it’s totally inaccurate. The cooperation rate (the percentage of people who finish phone interviews) is much closer to 50% in most current polling -- a far cry from Klein’s mysterious 5% figure.

Klein’s inaccurate reference was among many issues discussed at the recently completed American Association of Public Opinion Research (AAPOR) conference in Nashville, the annual convention at which pollsters and survey practitioners gather to share knowledge and report research findings. I wish critics of polling could come to these meetings and see just how seriously pollsters take their responsibility, and how hard they’re working to improve the quality and accuracy of survey research.

A big focus at this year’s conference was -- as you might guess from my comments above -- response rates. I’ve talked about response rates before in these columns. Just to review, by response rates we mean the percentage of the phone numbers selected in a survey sample (itself selected to be representative of the overall U.S. population) that end up resulting in usable, completed interviews. There are a lot of reasons why response rates are not 100%. People refuse to be interviewed once contacted, some phone numbers persistently ring with no one answering, some phone numbers are persistently busy, more and more phone numbers are answered by answering machines or screened by caller ID machines (which means that they are not answered). And so forth. There is little question that all of these factors, taken as a whole, have resulted in a lower percentage of initial samples of telephone numbers yielding completed interviews. That fact was confirmed by several papers at the AAPOR conference.

But the key questions are: What does that mean, how devastating is it for polling, and should steps be taken to try to increase response rates?

One of the major points underscored at the conference in Nashville was that all of the recent media attention on response rates (fueled in part by columnist, pundit, and gadfly Arianna Huffington’s "discovery" of response rates a few years ago) is quite belated. Survey scientists have been analyzing issues relating to sampling, including response rates, for as long as they have been conducting surveys.

Second, the papers presented at the conference confirmed a fundamentally important conclusion. Response rates are not immutable. In fact, they are quite variable, depending in large part on how much time and money the survey researcher decides to spend on a particular survey. Surveys, as noted in one review conducted by Ohio State Professor Jon A. Krosnick, and his colleagues Allyson L. Holbrook and Alison Pfent, can and do have response rates as low as 10% or as high as 70%. The response rate for a particular survey is, to a significant degree, under the control of the researcher. It all comes down to time and money.

There are a number of techniques researchers can use to increase the percentage of the initial phone numbers in a random sample that ultimately yield completed interviews. Those techniques (all of which follow common sense, by the way) include: make the survey as short as possible (because fewer respondents will drop out), keep the survey "in the field" as long as possible, keep calling numbers that initially yield no answers and "busies" because this allows more opportunities to reach respondents who might be home infrequently or on vacation, offer a financial incentive to participate (duh!), send a letter ahead of time telling the person that a phone call is coming, and develop a group of "closers" -- interviewers who do nothing but call people back who have refused to be interviewed. And so on.

Some surveys done under contract to the government employ all of the above techniques. They start out with an initial phone sample, and then spend months (one survey discussed at the conference was in the field for 245 days) calling and calling each number in an attempt to reach someone in that household. When a person answers but refuses to complete the interview, the firms conducting these surveys often employ refusal conversion experts to call back and attempt to persuade those who don’t initially comply to go along with the program.

Most of our Gallup polls (and other surveys that focus on reactions to fast-breaking news events) can’t do a lot of this. We have to be in and out of the field within a matter of days, or the data we gather are stale.

For example, the week of May 19 found us measuring the public’s reaction to the recent terrorist bombings in Saudi Arabia and Casablanca. To keep this survey in the field for two or three months wouldn’t do us much good at all. We would be combining interviews conducted in May with those conducted in June or July or August, when there, no doubt, would have been a huge number of changes in the real world situation. The aggregated results wouldn’t be interpretable. We needed a very short three-day field period to allow us to measure the short-term reaction of the public to this specific news event.

For another example, we measured the feelings of the public about Annika Sorenstram’s appearance in the Bank of America Colonial PGA golf tournament. Media coverage flared up in the week or so before the tournament began, but soon after it ended (and when she did not make the cut), the intensive coverage predictably died down. We needed a poll to pick up reaction in a short time period. That’s what we did, but the response rates involved had to be lower than they would have been had the poll been in the field for a longer period.

Research firms are thus left to make decisions on the necessity and feasibility of investing a great amount of time and money in conducting surveys, based on the returns on this type of investment.

These types of decisions are not specific to the survey research industry. Many areas of industry and science involve calculations of the value of investments to improve a process versus the incremental returns those process improvements make. The billions of dollars spent on NASA’s Apollo program involved incredibly high levels of investment in multiple levels of redundancy because it was judged that the cost of failure was so high. In similar fashion, the U.S. Navy’s modern nuclear-powered submarines cost billions because of the extraordinary efforts to reduce the chances to near zero of the smallest untoward event.

But in modern medicine, the cost-benefit analysis of expensive, time-consuming tests is being more closely evaluated than ever before. It’s less and less clear than the lives saved from extraordinary efforts to detect early cancer (for example, as a result of whole body CAT scans or PSA tests for men) are worth the costs in terms of money and the unneeded pain and suffering involved in follow-up diagnostic procedures.

Similarly, for pollsters, as I have noted, it’s clear that a major investment in time and money will yield higher response rates. But what is the payoff? Do higher response rates really yield higher quality data? Where is the balance between the cost of increasing response rates and the risk of decreasing a sample’s "representativeness" if these investments are not made?

That’s what a lot of pollsters’ discussions in Nashville were all about. The results of the research presented at the conference suggested -- as has a lot of previous research -- that there just doesn’t seem to be a major increase in accuracy associated with the time, money, and effort it takes to increase response rates. Low response rates mean that we get a sample of a sample -- we complete interviews with a percentage of a larger sample initially selected. But that sample of a sample seems to work pretty well. Going to the effort to get a higher and higher percentage of the initial sample in the final sample doesn’t appear to have a lot of payoff. The results of surveys in which the response rates are lower are in many ways just as accurate as the results of surveys in which response rates are higher. The added time and investment -- at this point -- don’t seem to have a dramatically positive payoff.

The conference left us with several messages. There is no question that response rates are indeed lower than they have been at points in the past. There is no question that survey researchers can control response rates and increase them if they are willing to spend the time and money. The value of investing in producing higher response values, however, has still not been definitively demonstrated.

Most importantly, survey researchers take response rates very seriously. We study them, we focus on them at scholarly conferences like the one in Nashville, and they are the topic of scholarly papers and articles. They have been and continue to be a serious and important topic of our research focus.