EDI | Enabling Decisions Insightfully | Views from the field | No. 1
How to minimize attrition and non-response rates in mobile phone surveys? Experiences from the field
Johanna CHOUMERT NKOLO, Céline GUIMAS, Amy KAHN, Caitlin MCKEE, Caitlin SPENCE
Mobile phone surveys are becoming increasingly popular as a cheaper means of gathering household data in developing countries. Mobile phone penetration rates are growing at a fast pace. Measured as mobile cellular subscriptions per 100 people penetration rates are now 27 for Ethiopia, 56 for Tanzania and 72 for Kenya (2013 World Bank data). Conventional face-to-face household surveys, are the gold standard due to the scope and depth of interview data that can be collected but comparative to Mobile Phone surveys they require a longer time in the field and consequently incur greater costs driven by interviewer allowances and travel expenses. Furthermore, the frequency at which mobile phone surveys can be conducted means the questions asked can, potentially, be more topical and tailored to ongoing events.
On the flip side to the apparent cost savings several limitations are often noted regarding the use of mobile phone surveys. In addition to less attentive respondents, there may be a non-random heterogeneity of phone network coverage and mobile phone ownership. This leads to potential sample bias with a possible under-representation of isolated areas or poorest populations. In addition, reviews on panel studies have documented high attrition rates, i.e. households drop out of the survey.
Therefore, when conducting a mobile phone survey, one may wonder if there is an ‘impossible trinity’ between (i) data quality and representative sample (ii) time effectiveness (iii) and cost effectiveness.
EDI has a history of successfully conducting mobile phone surveys with low attrition and non-response rates. Building on our experience in conducting large scale household surveys in Tanzania, EDI proposes survey designs and field strategies aimed at reducing the attrition rate in mobile phone surveys.
Our strategy consists in providing mobile phones to respondents after a comprehensive field-based baseline interview. The underlying idea is to provide an incentive for ongoing participation and ensure that respondents are contactable. Then, we ask respondents the times that suit them best for their phone interviews. A text messaging service is used to remind them of an upcoming interview. In some cases, this text service is further used to maintain their interest in the project by, for instance, sending them information about interesting findings from the research.
On some projects solar chargers have been distributed with the phone. This ensures that access to electricity is not a cause of sample bias. Enumerators are instructed to train respondents on how to use both the mobile phone and the charger at the end of the interview, which further aids the contact availability of the respondent. The field teams also activate the mobile phones prior to the survey, to ensure that they are immediately usable and to minimize effort required from the respondent. A final approach is to send ‘airtime’ vouchers (phone credit) after follow-up interviews to the number of the mobile phone provided. This incentivizes respondents to answer calls and participate in additional phone survey rounds as the respondent can use this credit for personal calls.
During the face-to-face, field-based baseline interview, additional phone numbers are collected in case the mobile phone provided cannot be reached during the follow-up phone survey. Typically, up to three additional phone numbers are recorded, which may include the respondent’s own numbers, other household members’ numbers, or those of neighbours. The aim is to ensure that there are as many phone number options as possible at the time of the subsequent phone survey, when trying to track the respondent. This is particularly important in surveys where there is a significant time gap between the baseline and the follow-up phone round, which increases the likelihood of mobile phones being misplaced, stolen, sold or given away for various reasons. In Tanzania it is also common for people to replace or update their phone numbers (using a new SIM card) fairly often.
The typical EDI protocol in mobile phone surveys is to attempt to call each respondent (using the various phone numbers collected) at least four times before marking them as unreachable. Each phone call attempt is recorded in the data, and interviewer comments are documented noting down anything important regarding each attempt (for example, a phone number may no longer be in use, and therefore should not be used in future call attempts).
Local authorities are also utilised when respondents cannot be reached. In particular, they may be asked to track down respondents that have their mobile phones off (for example due to lack of electricity to charge them), encouraging them to turn on their phones and/or keep them charged at particular times so that the interviewers can phone them accordingly.
Overall, our strategy has been successful in several phone surveys conducted in Tanzania. In a three rounds phone survey conducted on 5,000 households, we reached an overall attrition rate of 2.6%. In another one, some 200 households were interviewed for a period of 25 weeks and we achieved an attrition rate of only 4 households (2%). Taking stock of our experience, we propose several insights which will be useful to policy makers and researchers in designing surveys. Our experience allows for cost-efficient surveys while minimizing bias which may result from attrition.
References:
- Croke, K., Dabalen, A., Demombybes, G., Giugale, M., Hoogeveen, J., 2012. Collecting High Frequency Panel Data in Africa Using Mobile Phone Interviews, Policy Research Working Papers. The World Bank.
- Dillon, B., 2012. Using mobile phones to collect panel data in developing countries. Journal of International Development, 24, 518–527.
- Lee, U., 2003. Panel Attrition in Survey Data: a Literature Review. CSSR Working Paper No. 41. Centre for Social Science Research University of Cape Town.
- Leo, B., Morello, R., Mellon, J., Peixoto, T., Davenport, S., 2015. Do Mobile Phone Surveys Work in Poor Countries? CGD Working Paper 398. Washington, DC: Center for Global Development.
