Gallup has statistically adjusted previously reported Life Evaluation Scores spanning Jan. 2, 2008 through April 6, 2009. The adjustments involve respondents' scores on two questions that ask Americans to rate their current and future life on a Cantril Self-Anchoring Striving ladder scale with steps numbered from 0 to 10. These ladder scores are in turn used to categorize Americans as "thriving," "struggling," and "suffering," and Americans' representations in these groupings have also been adjusted. The statistical adjustments were made on the basis of an in-depth analysis of the effect of a series of political questions included at the beginning of the Gallup Daily tracking survey in 2008. Asking some respondents these politically oriented questions depressed positive responses on the Cantril Scale ladder questions that immediately followed them. The overall effect of the retroactive statistical adjustments slightly increased the average scores on the Cantril Scale ladder scores throughout 2008, although the general patterns as previously reported remain the same.
On Jan. 2, 2008, Gallup and Healthways initiated a breakthrough daily tracking of Americans' health and wellbeing based on 1,000 completed interviews per night with a random sample of Americans. The primary focus of this massive undertaking is to provide a continuing update on one of the most important components of America life today: the wellbeing and health status of its residents.
The tracking program coincidentally began in a high-profile election year, and Gallup made the decision to augment the tracking with a varying series of questions asking Americans about political matters, including presidential nomination preferences, general election preferences, and associated partisanship and likelihood to vote measures. Following a decades-long Gallup practice, these questions were included at the beginning of each survey questionnaire. The long series of carefully developed health and wellbeing questions were and have been kept in exactly the same order throughout the tracking.
Beginning on Jan. 6, 2009, Gallup reduced the set of political questions and only asked them of a randomly selected half of each night's sample. The other half of each night's sample was not asked political questions and began their interview with the standard battery of wellbeing and health questions.
Gallup analysts noted in early 2009 that respondents who continued to receive the political questions first in the survey seemed to give more negative responses on the Gallup-Healthways Well-Being Index ladder items. Gallup analysts determined that by inserting a simple "re-orienting" question ("Are you satisfied or dissatisfied with your personal life today?") after the political questions, the effect of the political questions asked of half the sample was largely mitigated. The use of this re-orienting question was initiated on April 6, 2009, and has been continued to date. Half of all respondents receive a political question first (about presidential job approval), followed by the re-orienting question about personal life satisfaction, and then the standard series of wellbeing and health evaluation questions. The other half of respondents from each night's sample go directly into the wellbeing and health questions.
The effect of question order or context effect is well-documented in survey research; what is asked first can affect that which comes later. In this situation, the surprising finding for the 2008 data was the degree to which asking respondents one or more politically oriented questions in a survey affected their responses to subsequent, totally unrelated questions asking them to evaluate their current and future life. The data show that in today's political environment, simply asking respondents to rate the president or to indicate for which presidential candidate they plan to vote may cause them to evaluate their current and future life slightly more negatively in subsequent questions.
Gallup analysts, working with Gallup Senior Scientist Dr. Angus Deaton of the Department of Economics at Princeton University, have examined this question order phenomenon in a number of ways over the past two years. Several features of the over-time nature of the Gallup-Healthways project provide an excellent basis for these analyses. First, changes made to the structure of the questionnaire over time provide the opportunity to track the effect of different questionnaire construction and order sequences. Second, the split-sample procedures put in place between Jan. 6 and April 5, 2009, provide a real-world experimental design that allows for direct comparisons of two random conditions. Third, because the survey has several questions asked in exactly the same way over the entire life of the project, it provides stable measures that can be used to calibrate expected responses to specific other questions of interest.
Taking advantage of this last fact, Gallup analysts and Angus Deaton estimated the statistical relationship between a set of two questions asking about standard of living (located late in the survey and highly correlated with the Cantril Scale ladder questions) and scores on the Cantril Scale ladder using data only from respondents in 2009 who did not receive political questions first. A number of demographic, political party, and regional variables were then added into the model, which helped refine estimates of what each respondent could be expected to score on the Cantril Scale ladder questions. Finally, Gallup analysts evaluated the split-half data from Jan. 6 through April 5, 2009 to measure differences that resulted from the effect of the political questions, and used the results of this procedure as an additional input into the overall statistical model.
All of these procedures take advantage of the availability of the 2009 data in which there were no political questions included in the survey. Using just these data, the statistical relationship between respondents' answers to standard of living and background questions asked late in the survey and those respondents' answers to the Cantril ladder questions asked at the beginning of the survey can be calculated. The results provide an excellent way of estimating how respondents would have answered the Cantril Scale ladder questions if they did not answer political questions first.
Following these assumptions, Gallup used statistical estimates to approximate Cantril Scale measures for all respondents from Jan. 2, 2008 through Jan. 5, 2009. The resulting Cantril ladder estimates were then compared to the actual Cantril Scale ladder scores given by these respondents - all of whom answered political questions first during the survey. Because the effect of the political questions in general is to depress (make more negative) some respondents' self-placement on the Cantril Life Evaluation ladder, the re-estimated scores for 2008 are in general at least slightly more positive than the actual scores the respondents gave.
Subsequent follow-up analysis found that these statistical procedures in general do a good job of estimating scores for respondents in the non-political survey environment. This was determined by estimating Cantril ladder scores for respondents in 2009 and 2010 who in fact did not answer political questions first. Gallup then compared the estimated scores with the actual Cantril ladder scores each respondent gave. The estimates proved to be close to the actual scores, particularly at the monthly level. In other words, without any knowledge of how respondents actually rated themselves on the two Cantril life evaluation ladder scales, the statistical estimating procedures did a good job of correctly estimating what respondents actually scored on the Cantril ladder scales.
As a result, Gallup made the decision to use these statistical procedures to retrospectively re-estimate the scores on the Cantril Scale for all respondents for Jan. 2 through Dec. 30, 2008. This procedure assigns each respondent interviewed in 2008 two Cantril Scale scores that Gallup estimates he or she would have given had he or she not answered political questions first during the survey. This has the effect, as noted, of increasing the average Cantril Scale scores for this time period. The average self-placement on the Cantril Scale measuring current life conditions was 6.5 throughout 2008 with the political questions asked first, while the average re-estimated score for 2008 was 6.8. There was no difference in scores on the future life conditions between the estimated scores and the actual scores.
The small changes that resulted from using the re-estimating procedures had an effect on Gallup's Life Evaluation Index that is derived from the Cantril Scale scores. Specifically, the percentage of Americans in 2008 categorized as thriving increased from 45.1% in the original data to 48.9% using the re-estimates. The percentage categorized as struggling decreased from 50.6% to 46.8%, and the percentage classified as suffering was essentially unchanged from 4.3% to 4.4%. The overall Well-Being Index scores, of which these Life Evaluation scores are a part, became slightly more positive using the re-estimated data, increasing from 65.9 to 66.5. The broad picture of the overall over-time trends from the time period 2008 through the summer of 2011 did not change. The year 2008 remains the most negative year Gallup has measured so far; just not quite as negative as it was before the re-estimates were put into place.
From this point forward, Gallup will retroactively adjust trend analysis based on Cantril Scale scores, including 2008 data, to incorporate the re-estimated numbers. Gallup.com analyses affected by the new estimates will be carefully noted. Additionally, Gallup will use the data for the first three months of 2009 based only on those respondents who did not receive the political questions first in the survey context.These data provide a fascinating insight into the way in which Americans' minds work. Once asked to think about political questions, survey respondents apparently move into a more negative frame of mind in general, and when immediately thereafter are asked to place themselves on a life evaluation ladder, are not quite so positive as they would have been had they not been reminded about politics. The political questions asked first in 2008 were fairly innocuous; for example, "What party do you identify with?" "Who would you vote for in the presidential election?" "How likely are you to vote?" and, for some months, presidential approval. But apparently even asking respondents these questions is enough to "spoil the day" for some respondents, and to cause them to rethink how they evaluate their life.