By Yuliya Dudaronak, Research Director
In public opinion surveys, data analysis and interpretation are only as useful as the questions asked. Questions should act as appropriate measures of people’s opinions, behaviors, and experiences to help reveal an accurate picture of the situation at hand. Therefore, the wording of a question is important, as it shapes the parameters in which respondents can provide relevant and accurate data.
For example, the same question can be asked in many different ways:
Therefore, the goal of writing/selecting a survey question is to choose the question that every potential respondent will:
- Interpret in the same way;
- Be able to respond to accurately;
- Be willing to answer.
This is even more important when exploring topics that are complex or highly sensitive. For example, at ORB, we often look at recruitment methods and tactics of violent extremist organizations (VEOs). In December 2021, ORB ran a face-to-face survey with over 2000 respondents in selected governorates of three northern states of Nigeria (Yobe, Adamawa, and Borno). As part of this survey, respondents were randomly selected to receive one of two versions of a question pertaining to recruitment methods and tactics.
Half of all respondents were asked to identify recruitment tactics from a predetermined list in a “yes or no format”: In your community, are the following tactics being used by VEOs to recruit/attract new members or try and get your community to support them?
The other half of respondents were asked to name recruitment tactics in an open-ended format: In your community, what are some of the main tactics used by VEOs to recruit/attract new members or try and get your community to support them?
This experiment tried to determine if respondents across different question formats would identify similar tactics at similar rates, potentially shedding light on how to best quantify VEO recruitment methods to monitor change over time.
These findings above (Figure 2) indicate that kidnapping/abduction, anti-government rhetoric, and financial incentives are consistently named among the top VEO tactics, regardless of question format. However, the number one tactic cited in the open-ended question is ‘incentives to youth’, while the number one tactic cited in the closed-ended question is ‘abduction/kidnapping.’ This example demonstrates that the structure of the question drew respondents to two different conclusions. These major discrepancies raise concerns about which data is more accurate, which has programmatic implications.
The Role of Recall Bias
What could explain these discrepancies? A potential answer may be recall bias. Recall bias is a systematic undercount or overcount of the frequency with which a certain behavior occurred. (1)
this case, the experiment shows that prompted responses on a variety of known tactics may be undercounted or overcounted in comparison to open ended responses—perhaps because certain tactics are less visible, or less discussed openly. This effect may explain why tactics such as infiltration of social circles seldom appear as open-ended responses. Inversely, it may also explain why widely discussed methods, such as financial incentives, come up nearly twice as often when asked in open-ended question compared to a pre-determined list.
How then, do we navigate instances of confirmation bias/selecting the best question? As in many questions of research design, the true answer is ‘it depends’.
More specifically, it depends on the goals of the research. In this case, the goal of the research was to assess the population’s familiarity with a specific set of tactics. Thus, using a predetermined list works best for this situation. However, if the goal is to ‘take community’s pulse’ on how they believe VEOs engage in their communities, asking completely open-ended questions is likely the more appropriate method. At ORB, we make sure to engage in detailed discussions with clients about the goals of the research and how they are planning to use the data to make sure we offer the methodologies particularly suited to the goals of that specific project.
(1) “Recall Bias,” American Psychological Association (APA), APA Dictionary of Psychology. Accessed October 13, 2021, at: https://dictionary.apa.org/recall-bias