The Toolkit
The Toolkit
Survey Development
Once we have identified the priority risks to measure, we can assess whether social desirability bias is likely to influence responses and, where relevant, proceed with designing List Experiment questions.
01Introductory statement
Each List Experiment question should begin with a clear introductory statement explaining what respondents need to do: indicate how many statements apply to them, rather than which specific statements apply.
Example framework:
“Please select how many of the following statements apply to you/your household/your farm/your community. Please do not tell me which statements apply; only share the total number that apply.”
This framing is critical to maintaining confidentiality: the List Experiment works by collecting a combined response across several statements rather than a direct response to the sensitive item.
02Designing the neutral statements
The neutral statements form the foundation of a List Experiment. They should be easy to answer, relevant to respondents, and varied enough to protect the sensitive statement.
- Keep them harmless and non-sensitive: They should not refer to the sensitive behavior or create discomfort, judgment, or fear.
- Make them relevant: Items should reflect experiences familiar to respondents’ everyday lives and be relevant to their geographic and cultural context. For farmers, this means drawing on locally relevant farming activities, labor practices, or household decision-making. Statements that are unfamiliar, geographically inappropriate, or disconnected from respondents’ lived experiences can reduce comprehension and distort response patterns.
- Keep them close to the topic: The statements should relate to the broader theme without revealing or hinting at the sensitive behavior. This helps the sensitive statement blend naturally into the list and reduces the risk of fleeing behavior, where respondents deliberately under-report the number of applicable statements to avoid revealing sensitive information.
- Mix likely and less likely statements: Include statements that respondents can reasonably agree or disagree with. If every statement is commonly true or untrue, respondents may select similar counts, making the sensitive statement easier to infer.
- Avoid negatively correlated statements: Including statements that are negatively correlated, where respondents who agree with one are less likely to agree with another, can help reduce variation in total counts. These considerations should inform question development and piloting, alongside the need to keep statements simple and relevant to respondents.
- Balancing list length and respondent privacy: Shorter lists may be easier for respondents to understand and can reduce variability in responses, but they may also provide less protection for the sensitive statement. The number of non-sensitive statements should therefore balance respondent comprehension, statistical precision, and privacy. A list of three non-sensitive statements and one sensitive statement provides a practical starting point, although the most appropriate design may vary by context.
03Designing the sensitive statement
The treatment group receives the same neutral statements as the control group, plus one sensitive statement. The sensitive statement should:
- Clearly capture the behavior or experience being measured.
- Be framed neutrally, without judgment or language that makes respondents feel accused.
- Follow the same structure and style as the other statements.
Select a statement type to highlight its example statements in both lists.
Control List
3 non-sensitive statements
- Neutral statement: Women in your community typically take care of household chores.
- Neutral statement: Men in your community are usually seen as the head of the household.
- Disagreeable statement: Women and men in your community spend the same amount of time working on the farms.
Treatment List
3 control statements +
1 sensitive statement
- Neutral statement: Women in your community typically take care of household chores.
- Neutral statement: Men in your community are usually seen as the head of the household.
- Sensitive statement: Women in your community face behavior of a violent or unwanted sexual nature when working on coffee farms.
- Disagreeable statement: Women and men in your community spend the same amount of time working on the farms.
04Pilot and refine statements
Before full implementation, test the questions with the target respondent group. Feedback from enumerators and pilot respondents helps identify whether statements are understandable, relevant, and appropriately framed.
Pilot data collection often reveals issues with the design or framing of List Experiment statements that may not have been apparent during the initial questionnaire development. Common challenges include:
- Limited contextual relevance: Some statements may not reflect the everyday experiences or practices of the target respondent group.
- Lack of clarity: Certain statements may be framed in ways that respondents find difficult to understand or interpret consistently.
- Unexpected agreeableness: Neutral statements initially thought to be disagreeable, in fact, be widely accepted within the local context, reducing their suitability for measuring sensitive behaviors.
Following the pilot, researchers should review these issues and refine the questions accordingly. This may involve simplifying the wording, adjusting the framing, selecting more contextually relevant themes, or replacing statements that do not function as intended. Where changes are substantial, the revised questions should be piloted again to assess whether they are better understood.
Pair List Experiments with Direct Verification Questions
Pairing a List Experiment with a direct question helps test whether social desirability bias is affecting responses. Comparing the prevalence from direct questioning with the List Experiment estimate shows whether respondents may be under-reporting when asked outright.
This comparison is particularly useful when applying List Experiments in a new population or context, where we may not know whether a topic that seems sensitive to researchers is actually sensitive to respondents. If the two estimates are close, there may be little evidence of social desirability bias, and the added complexity of a List Experiment may not be warranted. If the List Experiment estimate is clearly higher, direct questioning may be under-reporting the behavior, supporting continued use of the indirect method.
Over successive studies, this evidence can refine the approach. Topics that show little difference between direct and indirect measurement can later be monitored with simpler direct questions, while topics that consistently show under-reporting through direct questioning may continue to need indirect methods.
The wording of the direct question should match the sensitive statement as closely as possible, so both measure the same behavior. The direct question should also be administered to all respondents. Further, one must decide its placement in advance. Asking it before the list can prime respondents; placing it after the list is generally preferable.
For highly sensitive behaviors, asking directly may be inappropriate or may affect responses to the List Experiment. In such cases, researchers can omit the direct verification question and rely solely on the List Experiment to estimate the prevalence of the behavior.
The Toolkit
Sampling and Randomization
List Experiments generally require larger samples than direct questions because prevalence is estimated from the difference between two group averages. This introduces additional variability, making it important to have sufficient respondents in both the treatment and control groups to generate meaningful estimates.
The sample should adequately reflect the population being studied, including relevant locations, worker types, farm sizes, or supply-chain tiers where risks may be concentrated. List Experiments do not correct for sampling bias, so careful consideration of who is included in the survey remains essential.
Balancing indirect and direct questioning:
Where List Experiments are paired with direct verification questions, the survey design should consider how best to collect both measures without unnecessarily reducing the sample available for the List Experiment. One option is to administer direct questions to a smaller subset of respondents, although this may limit the precision of the direct estimate. Where feasible, asking both questions to separate groups can provide useful insights into whether respondents are under-reporting sensitive behaviors when questioned directly.
Considering alternative designs:
More advanced approaches, such as double List Experiments, can improve statistical precision without necessarily increasing the sample size. In this design, respondents answer two different control lists, and the sensitive item is added to one list for half of the sample and to the other list for the remaining half. Each group therefore acts as the treatment group for one list and the control group for the other, producing two prevalence estimates from the same sample. Averaging these estimates increases precision without increasing sample size. While such designs may be useful in certain contexts, they also introduce additional complexity and may not be practical for every study.
The Toolkit
Data Collection
Enumerator preparation
Before fieldwork, Research Assistants should complete a dry run or mock interview covering the List Experiment module. Training should ensure enumerators understand both the treatment/control structure and the mechanics of administering the questions without compromising anonymity. Enumerators should review the list questions in advance and flag any terminology or instructions they are uncomfortable administering.
Enumerators should also follow several administration principles:
- Read the statements exactly as written: Don’t alter their meaning, framing, or sentiment. Where clarification is needed, the enumerator can explain the intended meaning conversationally, but should not rewrite or simplify the statement itself.
- Don’t prompt based on earlier responses: If a list response seems inconsistent with something the respondent said earlier, the enumerator should not point out the discrepancy or ask them to revise their answer. Record the response as given.
- Don’t reveal treatment status: Respondents should not be told whether they are in the treatment or control group; the enumerator simply administers the list matching the respondent’s random assignment.
- Allow enough time to process the list: Pause between statements where useful and avoid rushing, particularly at signs of hesitation or confusion.
- Maintain neutral delivery: Statements should not be read with an interrogative or leading tone, and treatment and control versions should be delivered with the same neutral phrasing and emphasis. Training should also cover avoiding any implication that respondents should answer each statement individually.
Introducing the List Methodology
Before administering the main List Experiment questions, we explain the methodology to respondents using a simple, non-sensitive example.
We ask respondents to report how many statements apply to them, rather than identifying which specific statements are true, with responses ranging from 0 to 3 or 0 to 4 depending on the number of statements. For example, we present three statements:
- The sun rises in the morning;
- The sky is blue during the day;
- The sun shines at midnight.
The enumerator explains that the correct response is 2, as only the first two statements are true. This demonstrates how respondents should count applicable statements without revealing which ones they agree with.
We then test respondent comprehension using two practice questions on non-sensitive themes:
- Farming and everyday life: Respondents hear four statements:
- You are a farmer;
- you grow coffee;
- you live in Kenya; and
- you eat food and drink water.
They are asked to report how many apply to them, from 0 to 4. For the intended respondent group of Kenyan coffee farmers, the expected answer is 4.
- Sleep patterns: Respondents hear three statements:
- I did not sleep at all yesterday;
- I slept for less than or exactly 6 hours yesterday; and
- I slept for more than 6 hours yesterday.
They are asked to report how many apply to them, from 0 to 3. Since these statements are mutually exclusive and collectively exhaustive, the expected answer is 1.
These practice questions allow enumerators to assess whether respondents understand the counting exercise and provide clarification where needed. Once respondents demonstrate comprehension, we proceed to the main List Experiment questions.
Administering the List Experiment
Before starting the List Experiment questions, the enumerator should clearly explain how the exercise works. Respondents should first be walked through an example question, then two comprehension questions, before moving to the main list questions. These steps shouldn’t be skipped or rushed.
Respondents should report only the total number of statements that apply, not which ones—and enumerators should reinforce this throughout the section. A practical technique is to ask respondents to count the applicable statements on their fingers while listening to the full list.
The enumerator should read all statements in the list before asking for a response. Statements should not be administered one at a time as a series of “yes/no” questions, as this would undermine the anonymity the method provides.
Each enumerator should administer both treatment and control interviews, rather than being assigned exclusively to one group. This reduces the risk that differences in enumerator style or behavior become confounded with treatment assignment.
Each enumerator doesn’t need to complete exactly the same number of treatment and control interviews, but the totals should stay roughly balanced overall. The field team should track completion numbers regularly and adjust allocations if imbalances emerge.
Do No Harm and Respondent Safeguarding
When collecting data on sensitive human rights risks, researchers must prioritize the safety, privacy, and wellbeing of respondents and enumerators. Researchers should follow these safeguards:
- Informed consent: Clearly explain the purpose of the survey, how responses will be used, and that participation is voluntary. Respondents should be able to skip questions or withdraw at any time without consequences.
- Respondent privacy: Conduct interviews in private, away from employers, supervisors, or others who may influence responses. Ensure that respondents are asked a question about this at the start of the phone survey.
- Safeguarding and referrals: Train enumerators to recognize signs of distress and respond appropriately to outright disclosures of abuse or exploitation. Establish clear safeguarding protocols and identify relevant local support services before data collection begins.
- Reporting and dissemination: Present findings in aggregate, avoid reporting results for small sub-groups that could identify respondents, and assess whether sharing findings could expose individuals or communities to retaliation.
What the data looks like
| Respondent ID | Group | Control List Count (0-3) |
Treatment List Count (0-4) |
Direct Question (Yes/No) |
|---|---|---|---|---|
| R-001 | Control | 2 | Not applicable | No |
| R-002 | Treatment | Not applicable | 3 | Yes |
| R-003 | Treatment | Not applicable | 1 | No |
| R-004 | Control | 1 | Not applicable | Yes |
| R-005 | Control | 3 | Not applicable | No |
| R-006 | Treatment | Not applicable | 2 | Yes |
| R-007 | Treatment | Not applicable | 4 | No |
| R-008 | Control | 1 | Not applicable | Yes |
| R-009 | Control | 2 | Not applicable | No |
| R-010 | Treatment | Not applicable | 2 | Yes |
The Toolkit
Analysis
Indirect (list) estimate
We take the average number of statements selected by the treatment group and subtract the average for the control group. Since the only difference between the two lists is the inclusion of the sensitive statement, this difference estimates the proportion of respondents to whom the sensitive behavior applies. For example, if the treatment group selects more statements on average than the control group, that gap, the treatment average minus the control average, is the estimated prevalence, expressed as a percentage.
Direct estimate
We also calculate the proportion of respondents who answer “yes” when asked the corresponding sensitive question directly. This provides a benchmark against which the indirect estimate can be compared.
Confidence Interval
Because the indirect estimate is based on the difference between two group averages, it is subject to sampling uncertainty. We therefore use a confidence interval around the estimate, which provides a lower and upper bound for the estimated prevalence.
If the direct estimate falls outside this interval, we interpret this as evidence that the List Experiment is capturing risks that direct questioning may be missing. We classify this as “evidence of sensitivity,” suggesting that indirect questioning is useful for measuring the risk.
If the direct estimate falls within the interval, the two approaches are not statistically distinguishable at the confidence level. We classify this as “no clear evidence of sensitivity.” This does not mean the risk is absent or has low prevalence. Instead, this suggests that respondents may be comfortable reporting the risk directly, and that list-based questioning may not be necessary in subsequent survey rounds.
At 60 Decibels, we use a 90% confidence level as standard practice.
How the analysis works
Standard error: a measure of how much the indirect estimate may vary due to sampling. In this example, the standard error is assumed to be 0.06 for illustrative purposes.
If direct reporting falls within the lower and upper bound, there is no clear evidence of sensitivity. If it falls below the lower bound, there is evidence of sensitivity.