Survey Context Effects on Middle East Sympathies

Survey research professionals are well aware that the way respondents answer a question can be influenced by the questions that precede it. “Context effects” can occur when results to a specific question vary when it is asked in different surveys, especially when those surveys cover different topics.
Some context effects are easy to anticipate and avoid. For example, suppose a researcher wanted to include an open-ended question that asked people to name the most important issues to their vote. If that question followed questions asking about crime and healthcare, we would expect more people to mention crime and healthcare as important issues to their vote than would have if they had not been reminded about those issues. Or, people may answer the presidential job approval question differently if it were asked after questions about the economy during an economic boom than during a recession. Survey questions can activate thoughts or frame a subject in a particular way — in turn causing people to respond to subsequent questions in ways they otherwise would not.
But context effects can be unexpected and may not be discovered until a question has been repeated many times and it becomes apparent that the same question in different contexts yields different results. This is what occurred for Gallup’s long-standing trend question on sympathies toward the Israelis and Palestinians in the Middle East conflict.
Gallup has asked Americans this question since 1988, and it has been included in Gallup’s annual World Affairs survey each February since 2001. An examination of the trend results on this question from the World Affairs survey versus other Gallup surveys showed a consistent difference. Specifically, the percentage of adults who did not express an opinion about the conflict or who expressed a neutral opinion (favoring both sides or neither side) was consistently lower on World Affairs surveys than on others, while the percentage expressing sympathy toward Israel was considerably higher. The average differences are well beyond what we would attribute to normal sampling error.

Given that the World Affairs surveys date back to 2001 and most of the non-World Affairs surveys on which the question was asked were conducted before then, it is possible that the shifts represent real changes in public opinion on the Middle East over time. It may be that people have become more likely to hold opinions about the Middle East conflict and have increasingly sympathized with Israel in recent years. Such a shift is certainly plausible given the United States’ strong ties to Israel.
However, an analysis of World Affairs surveys and other surveys conducted at about the same time indicate this is not the case. Roughly contemporaneous surveys conducted by Gallup and other organizations show consistent differences in the percentage sympathizing with the Israelis when compared with the World Affairs survey results.

That leaves two possible explanations for the differences between surveys:
Gallup tested these hypotheses in a Feb. 12-28 survey of 1,932 U.S. adults.
The experiment was positioned toward the end of the questionnaire. The questions preceding the experiment asked about President Donald Trump’s handling of his job, ratings of U.S. politicians, personal finance ratings, personal life satisfaction ratings and society’s treatment of women. Thus, the survey format’s lack of questions on international matters provided a test for whether the heavy focus on international matters in the World Affairs surveys encourages respondents to express opinions on foreign policy matters they might not otherwise. The theory is that after respondents answer some questions on international affairs, they may become more focused on the topic and more comfortable expressing opinions (including weakly held ones) in response to subsequent questions on that same topic.
Survey respondents were randomly assigned to different versions of the survey (roughly half to “Form A” and roughly half to “Form B”). Form A respondents were asked the Middle East sympathies question after being asked to rate several countries on a favorability scale, including Israel and the Palestinian Authority — the standard sequence in World Affairs surveys. Form B respondents were asked the sympathies question without that country favorability question preceding it, as on other non-World Affairs surveys on which the sympathies question was asked.
The experiment results were persuasive. The differences seen in past surveys appear to be context effects produced by asking the country favorability question before the Middle East sympathies question.
When respondents were asked the sympathies question after rating countries on a favorability scale, 58% said they were sympathetic to the Israelis, compared with 47% who did not rate countries before answering the sympathies question. Twenty-two percent of respondents who rated Israel and other countries had no opinion or did not favor either the Israelis or Palestinians, compared with 32% of respondents who were not asked to rate countries. These differences are similar in size to those found between World Affairs surveys and other surveys in the past.
Additionally, the 58% of respondents in the Feb. 12-28 experiment who rated countries and were sympathetic to the Israelis essentially duplicates the 59% who expressed the same opinion in the Feb. 1-10 World Affairs survey. The similarity in these results also refutes the hypothesis that World Affairs surveys’ focus on international issues is a factor in the differences in sympathy toward the Israelis and “no opinion/no preference” responses observed in past World Affairs and non-World Affairs surveys.

A closer look at the results reveals that the country favorability question has little impact on whether people sympathize with the Palestinians. Rather, the priming effect of asking country favorability appears to push people toward sympathizing with Israel rather than expressing no opinion (or not taking a side). This suggests that those who do not hold especially strong opinions on the Middle East are most susceptible to survey context effects. And because Americans in general are more likely to have positive opinions of Israel than the Palestinian Authority, those with weakly held opinions who are influenced by the survey context are more likely to say they sympathize with the Israelis than the Palestinians.
The context effects were apparent in key subgroups, including by education and by party identification, with similarly sized effects among those with more versus less education and among Republicans, Democrats and independents.

The experiment results indicate that Americans respond differently to the question about their sympathies toward the two sides in the Middle East dispute depending on whether they rated each of those countries (among others) immediately before they were asked the sympathies question.
One question that cannot be answered from the experiment is whether the percentage of Americans who are sympathetic toward Israel is closer to 50% (as it is when not preceded by country ratings) or 60% (as when preceded by country ratings). However, regardless of whether the percentage is closer to 50% or 60%, the findings are still clear — that many more Americans sympathize with the Israelis than the Palestinians.
While, as in this example, some context effects are difficult to anticipate, there are some steps that can be taken to minimize their occurrence. Gallup usually puts questions that are especially important to a survey — such as presidential job approval — or that are particularly vulnerable to survey context (e.g., open-ended questions) near the front of the survey to make the results less susceptible to context effects.
Of course, in a long survey not all questions can be asked near the front. For these, if the goal is to compare results on certain questions in the current survey to those from previous surveys, it is important to try and maintain as similar a context as possible. In World Affairs surveys and others in the Gallup Poll Social Series, Gallup largely maintains the same questions in the same order, year after year. Even if there is a known context effect, as on the Middle East sympathies, maintaining a similar context helps researchers be more confident that any changes they see over time are likely due to real attitude change as opposed to context effects or other types of survey bias.
In addition to reminding survey researchers about the care they need to take in designing surveys, awareness of context effects also helps those who are looking at the data be better consumers of it. Those who study public opinion and want to understand changes in attitudes over time would be well-served to dig deeper into the surveys to see if the change might be due to context rather than a shift in attitudes. Often, the answer is not immediately clear — but as more results are collected using the same questions over time, the conclusion as to whether a past result represented a definite change in public opinion or was a potential context effect becomes increasingly apparent.