Dan Bunter
The importance of piloting a survey can often be overlooked, but it is one of the most crucial project phases when seeking to deliver high quality data from a large-scale data collection.
After a recent project trip to Tanzania with EDI Global, the importance of piloting survey tools before the enumerator training and data collection phases was reinforced to me. The project was an endline data collection for an impact evaluation conducted by NORC at the University of Chicago on the Waache Wasome (Let them Learn) programme.
This brief blog outlines what a pilot survey is, includes reflections on my recent piloting experience and concludes with recommendations on running an effective pilot.
What is a Pilot Survey?
According to UNESCO, a pilot survey is defined as a “Survey that is conducted with few individuals of the target population or the survey sample, in order to test and refine the survey instruments before the main data collection across the target population or full sample”.
To add to this definition, piloting can also be used to test out ALL aspects of large-scale survey data collection such as the sampling strategy, field protocols and data processing. Testing out these different elements during the pilot allows the project team to solve a variety of issues before the main data collection, therefore increasing the chances of delivering high quality data.
Reflections on a Recent Pilot Survey Experience
The pilot for the endline data collection with NORC took place in one of the main sample regions with four schools who had received the programme interventions but were not part of the evaluation sample. These schools presented ideal piloting conditions as the respondents were able to answer questions which focused on the Waache Wasome programme interventions. The team could also pilot all three survey instruments including school girls, teachers and parents.
The real value of the pilot came during the debrief sessions each evening after data collection. The mixture of client expertise on the issues raised by the survey, local contextual understanding of language and culture from EDI’s Tanzanian co-ordination team and our own survey expertise created dynamic and rich discussions which led to important changes in each of the survey instruments. Examples of changes included:
-
-
Changing the order of responses in a multi-response list to ensure responses are thematically grouped
-
Rewriting or deleting questions that did not provide satisfactory answers
-
Establishing correct field protocols for experimental sections of the survey
-
Fine-tuning translations
-
Debrief discussions varied in content from miniscule survey design issues to broader questions about ethics and dealing with sensitive topics raised by the questionnaires. Although at times discussions led to long debates about the meaning of one word in Kiswahili, it was essential to take time to solve such issues before the training of supervisors and enumerators began.
The importance of piloting surveys before the main data collection was highlighted in this project due to the number of small but significant changes made to the survey tools and field protocols. These changes all ultimately contributed to improving the quality of data collected.
Recommendations
If you are looking to have a successful pilot of a survey, here are some recommendations to follow based on my recent experience:
1) Seek out ideal piloting conditions. Sample respondents should have demographic similarities to the respondents in the main data collection and if possible (though often not feasible), they would have experienced the intervention(s) being evaluated. Ideally, interview these respondents in similar environments to the real survey, that may be in a school or in a household or a workplace.
2) Have more than one enumerator test the questionnaire. This improves the chances of spotting issues during an interview and can better highlight broader survey issues if the problem is raised by multiple enumerators.
3) Pilot enumerators should already be aware of the project or have past survey experience to help improve the quality of feedback. Ideally, the pilot enumerators will go on to be supervisors, data processors or the project co-ordinator in the main data collection and will therefore have lots of expertise and experience to offer throughout the pilot.
4) Debrief discussions must include local knowledge and expertise. Meanings of words in local languages or dialects and understanding respondent attitudes towards specific survey topics, can only come from individuals from the region or country of where the data collection takes place. This knowledge should never be excluded from debrief discussions.
5) Allow time from the conclusion of the pilot to the beginning of enumerator training (ideally one week or more depending on the number of survey instruments). This ensures all the learning from piloting can be effectively adapted into the survey tools and field protocols to best inform the enumerator training sessions. A rushed pilot and training phase are likely to overlook important learning points to improve data quality.
