Dr. Johanna Choumert Nkolo
The 7th Conference of the European Survey Research Association (ESRA) was held in Lisbon, Portugal, from 17th to 21st July 2017. The conference intended to address the most recent research and innovations relevant to survey methodology and data collection. The programme attests to the dynamism of the field, with about 850 participants sharing their work and ideas in sessions covering a wide range of exciting topics such as mixed methods surveys, using paradata to improve survey data quality, assessing data quality, and doing surveys in developing countries, amongst others.
In the first part of this blog post I provide a summary of the ESRA 2017 conference. Secondly, I share some thoughts on assessing data quality and how data quality is defined.
Two keynotes, Mixed-mode surveys and Cross-cultural surveys
|
The conference started with a keynote speech by Edith de Leeuw (Utrecht University) on “Mixed Mode: Past, Present, and Future” in which she summarised the state of the art in traditional mixed-mode surveys and discussed implications for mixed device surveys. Mixed-mode surveys are facilitated today by better coverage, increasing response rates and reducing costs. Nonetheless, they can still result in measurement errors. Some recommendations to implement such surveys can be found in her latest publication on the topic: de Leeuw, E.D., Klausch, L.T. & Hox, J.J.C.M. (2017). Mixed Mode Research: Issues in Design and Analysis. In Paul Biemer (Eds.), Total Survey Error in Practice, Willey. Additional References on the topic can be found here. |
 |
|
The second keynote speech, by Beth-Ellen Pennell (University of Michigan), covered a topic very relevant to EDI research, that is “Trends and Developments in Multinational, Multiregional, and Multicultural (3MC) Surveys”. In this speech, she discussed recent developments in 3MC research, with a focus on low-resource countries which are often characterised by having several languages and dialects, out of date census data, displaced populations, and low literacy rates. She raised several interesting questions such as the staggering of data collections in 3MC surveys, the cost/quality trade-offs, and the use of paradata to monitor field teams and data quality. She recommended the use of real-time interviewer-level indicators to guide quality control interventions, as well as Geographic Information Systems (GIS) combined with timestamps to track the movement of interviewers in field. |
References: CCSG (2016). Cross-Cultural Survey Guidelines. Johnson, T., Pennell, B.-E., Stoop, I., & Dorer B. (forthcoming). Advances in Survey Methods in Multinational, Multiregional and Multicultural Contexts. Hoboken, New Jersey: John Wiley & Sons. Workshop on “Comparative Survey Design and Implementation (CSDI)”, an initiative that gathers researchers interested in cross-cultural surveys and data harmonisation. Next workshop in 2018. |
The paradata marathon (4 sessions, 17 presentations)
Paradata are data about the data collection process. Most researchers focus on the survey data and auxiliary data. However, there are a number of useful lessons that can be learnt from the analysis of paradata. Below are summaries of selected presentations from these sessions on “Using paradata to assess and improve survey data quality”:
-
Paradata, and timestamps in particular, are a powerful tool to monitor fieldwork, detect survey errors, and improve data quality. An excellent illustration of this was presented by Caroline Vandenplas (KU Leuven) using data from the European Social Survey (ESS), in her presentation “Fieldwork monitoring and managing with time-related paradata”.
-
How to use interview duration data to inform about survey quality during the data collection? Statistical and multi-level analysis bring some interesting insights on how paradata can be used to monitor field work. In her presentation “Monitoring interview duration data: from a Statistical Process Control perspective”, using the ESS, Jiayun Jin (KU Leuven) analyzed interview duration to detect interviews which appear to be too long or too short and identified the characteristics of respondents and interviewers which influence interview duration.
-
Still using data from the ESS, in her presentation “The use of auxiliary and event data in tracking an inhomogeneity of substantive results from surveys in cross-national studies. Example of ESS”, Teresa Zmijewska-Jedrzejczyk (Institute of Philosophy and Sociology, Polish Academy of Sciences) underlined the importance of contextual effects (context effects of survey climate) and recommended using paradata together with events data to evaluate survey data quality.
-
In the presentation “Paradata as an aide to questionnaire design: Improving quality and reducing burden” (Office for National Statistics, UK), it was stressed that the use of paradata allows researchers to quickly spot problems with questionnaire design and improve it, with the example of ONS’s business surveys. Such paradata include timestamps, call records, …
-
In “Interview speed, interviewer experience and the fieldwork process”, Celine Wuyts (KU Leuven) extended previous research on the effects of interviewers’ survey experience and experience accumulated over the course of fieldwork on interview speed using the ESS data. One interesting finding is the negative association between time since the last interview and interview speed suggesting that discontinuities in fieldwork may slow down experience accumulation.
-
In “What took you so long? The determinants of interview length” Tobias Schmidt (Deutsche Bundesbank) provided an analysis of interview length for the Computer Assisted Personal Interviewing (CAPI) German Wealth Survey, and variables such as socio-demographic characteristics of respondents, interviewers’ characteristics, interview characteristics, paradata, …
-
Surveys on sensitive topics are the object of much research on how to obtain true responses from respondents. In “Hesitations in socially desirable responses in a smartphone survey”, Michael Schober (New School for Social Research) examined how respondents produce socially desirable answers using data from an experiment conducted on iPhone users, with some being interviewed by professional interviewers and others by an automated spoken dialog interviewing system. Response latencies (speech paradata) provide some impressive results by displaying some differences between the answering process of respondents talking to a human interviewer and an automated speech programme.
-
In “Using paradata-based key performance indicators to monitor implementation of a split-ballot experiment”, Aneta Guenova (U.S. Department of State) showed how Key Performance Indicators (KPIs) derived from paradata can be used to monitor survey data collection. Using a CAPI survey of 1500 respondents, asking parallel scale questions to two sub-groups (with respondents being randomly allocated to one of the two versions), she showed how even a small amount of paradata is an easy and cost-effective way to check the data. This could include the number of interviews per sampling point, the number of interviews for each questionnaire version, date and time of each interview. Due to budget constraints, survey paradata was reviewed at two points of fieldwork (though ideally, day-to-day monitoring should be implemented).
-
I also presented in this session a paper entitled “Using timestamps to monitor fieldwork and evaluate data quality: Experiences from a household survey in Tanzania” (co-authored with Henry Cust and Callum Taylor). We argue that monitoring data quality is an essential part of any serious, large-scale data collection process. It helps to identify errors and inconsistencies in the data and results in a cleaner, more accurate dataset. The increasing use of CAPI has also increased the possibilities of researchers. CAPI allows users to capture a wide range of paradata which can be used to complement the questionnaire data, metadata and auxiliary data. For instance, timestamps record the exact time when a selected question is answered. They also provide useful information throughout the interview that can be used in various ways. The purpose of this contribution was to show how timestamps should be implemented in a CAPI survey and used before, during and after data collection. We used timestamps collected for a 800-household survey conducted in Tanzania in November and December 2016. For each 1-hour questionnaire, we collected over 30 timestamps. The information collected was used during the 3 phases of fieldwork preparation, real-time fieldwork monitoring, and evaluation of data quality. Overall, our results corroborate the importance of collecting and analysing paradata to monitor fieldwork and ensure data quality for micro data collection in developing countries.
Assessing data quality
Firstly, what is data quality?
What does Google say about it? Actually, Wikipedia has an answer “Data quality refers to the condition of a set of values of qualitative or quantitative variables. There are many definitions of data quality but data is generally considered high quality if it is “fit for [its] intended uses in operations, decision making and planning.”. Alternatively, data is deemed of high quality if it correctly represents the real-world construct to which it refers. Furthermore, apart from these definitions, as data volume increases, the question of internal data consistency becomes significant, regardless of fitness for use for any particular external purpose. People’s views on data quality can often be in disagreement, even when discussing the same set of data used for the same purpose. Data cleansing may be required in order to ensure data quality”.
I had raised the question in a previous post following presentations at the Oxford CSAE 2016 Conference. Defining data quality is not an easy task. As a development economist, I consider a data set to be of quality if it provides accurate information about the reality or the outcomes of interest, and I must confess some obsession for sampling bias and measurement errors! In the various conferences I attended, including the ESRA conference, listening to presentations or discussing with researchers, I reached the conclusion that there is no single definition or approach to data quality in researchers’ or data collectors’ minds. Some proxies could be the presence of outliers, missing consumption data (for household surveys) … Some researchers put forward discrepancies in spousal answers, revealing different knowledge on the topic, conflicts in the relationship or social desirability. This is an interesting point; indeed, in most household surveys, interviewers typically interview one respondent (e.g. household head, most knowledgeable household member, randomly selected respondent or respondent selected according to certain criteria such as having had a baby in the last 6 months). Would we get different outcomes by interviewing other household members? Using the German Family Panel, and the items on division of labor, conflicts in the relationship and disagreement in the relationship, Miriam Truebner (University of Bonn) proposed an analysis of the predictors of spousal discrepancies, such as knowledge, social desirability, relationship quality, etc. Other researchers raised the possibility that poor quality results can be due to poor quality data which calls for excluding poor quality respondents. Although I would agree that one should exclude an interviewer who is not performing well, with the view that interviewers are the suppliers of questions to respondents, I think that excluding “low-quality” respondents would lead to serious sample selection bias. Other researchers proxy data quality by interview duration, or by the number of “Don’t know” answers or missing values…In the case of surveys carried out in multiple countries, the issue of harmonisation between countries has been discussed by many organisations such as for the Demographic and Health Surveys (DHS) and the Living Standards Measurement Study (LSMS). For other surveys, one could however question how the harmonisation has been carried out especially when the data collection and the data cleaning was carried out by different organisations.
The debate is not closed and will be the topic of a future dedicated blog post.
Other topics
In a previous blog, I discussed some ways to deal with sensitive topics. Here are some additional resources on the topic with a focus on surveying respondents on domestic violence [1] [2]. The conference also had four sessions on “Researching sensitive topics”.
Collecting data in conflict areas raises numerous challenges well described by presenters in the session “Survey research in Conflict Areas: Learnings from Case Studies”. The “HiCN Households in Conflict Network” provides information about relevant surveys and household datasets in conflict zones.
Conferences like the ESRA Conference or the International Conference on Social Science Methodology provide great opportunities for EDI to contribute to the broader research dialogue linking robust academic research and field experience. Survey methodology is a rapidly evolving field with the availability of more diverse and cheaper technology (tablets, mobile phones, internet…). All projects that EDI implement are designed to be in line with international best practices, and we are always pushing to improve internal processes and contribute to the survey research field with the aim of driving up data quality.
