Relevance & Research Question

Long surveys present high burden for respondents. For a long time the rule of thumb for length of self-completion surveys was not to exceed 15-20 minutes. More surveys are moving towards self-completion designs due to increasing survey costs and also due to high rates of device ownership and internet access in the UK and other countries. For some social surveys 20 minutes is not enough to continue collecting high quality data required. Some studies experimented with longer questionnaires and obtained reassuring results, for example, European Social Survey (ESS). However, more evidence is needed in this under-researched area. We collected the first wave of the UK Generations and Gender Survey (GGS) where only online mode of data collection was available to respondents. The median time spent on the questionnaire is around 40 minutes which is much higher than the advice given to survey practitioners in the past. It is important to assess different aspects of data quality of the long questionnaire. The main research question is: is long questionnaire associated with poorer data quality?

Methods & Data

We analyse the GGS collected in the UK. The GGS is a part of a global data collection infrastructure focused on population and family dynamics. The GGS collects demographic, economic, and social attitude data on young and mid-life adults (18-59) as they enter into adulthood, form partnerships, and raise children. We assess different data quality indicators: break-off rate, item nonresponse, different response style behaviours, consent to participation in the second wave of the survey among other indicators. We first conduct descriptive analysis and then use different logistic regression models to investigate data quality in the UK GGS.

Results

The results are reassuring and suggest that even though the GGS questionnaire is long and complex and interviewers are not there to guide the respondents through the process, the data quality is not poor.
Added Value
This study contributes to the under-researched area of long online questionnaires. This assessment suggests that, when carefully designed, long questionnaires do not represent risk to data quality and can be successfully implemented in self-completion surveys.

Relevance & Research Question: In an era where smartphones have become ubiquitous, the use of mobile devices for survey completion has become prevalent, underscoring the need to understand their impact on data quality. Long gone are instructions for respondents to use alternative devices. While earlier research suggested that mobile device responses resulted in lower data quality, recent studies based on mobile-first survey designs challenge this view, indicating that smartphone usage does not inherently degrade data quality. Methods & Data: This study contributes to this evolving body of research by examining smartphone survey completion in CRONOS-2, the ESS probability-based panel fielded across 12 European countries from November 2021 to March 2023. Results: We investigate response patterns over time and analyse demographic differences between smartphone and PC respondents, discussing how these insights might be used for targeted respondent recruitment. In addition, we explore survey completion times and data quality indicators, including item non-response and satisficing behaviors, drawing comparisons between PC and smartphone respondents. In this study, we also examine potential differences in break-off rates between smartphone and PC respondents. This approach is informed by previous research, which suggests that smartphones might lead to higher break-offs. Should this be the case, our goal is to identify any problematic items that could lead to such outcomes. Added Value: The findings from our research are intended to assist survey researchers and practitioners in the design and execution of high-quality online surveys in a mobile-centric world.

Relevance & Research Question

More and more large-scale surveys offer web mode as part of a mixed-mode design or as the only mode. Respondents can access the web survey from various devices including desktop computer, tablet, and smartphones. The proportion of respondents using smartphones to complete web surveys is growing. Some studies indicated that the breakoff rate is higher for respondents completing on a smartphone than on other devices. Especially in the context of large-scale social surveys with long completion times, this could lead to more breakoffs at an early stage of the survey. There is also evidence that other data quality indicators vary across devices. Those variations may increase for questions asked later in a survey, as respondents experience a greater burden toward the end of a survey. However, different devices may also address different individuals, which would result in a more balanced sample. We investigate the impact of the growing use of smartphones in large-scale surveys.

Methods & Data

We use data from the German General Social Survey (ALLBUS) 2021 and 2023. ALLBUS is a register-based cross-sectional survey of persons aged 18 and older in Germany with a completion time of around 50 minutes. Both surveys were conducted in a self-administered mixed-mode design (mail and web). We compare the sample composition and data quality between different devices. The data quality includes breakoffs, the final question before breakoff, item nonresponse, completion time, and providing an answer to an open question.

Results

The proportion of smartphones used to participate in the survey increases from 5.7% in ALLBUS 2021 to 17.3% in ALLBUS 2023. Preliminary results show that smartphone users restart the survey more often than desktop users. Moreover, breakoff occurs earlier and at a greater rate for smartphone users compared to desktop users. There are significant socio-demographic differences between smartphone and desktop users.

Added Value

Examining differences in devices improves our understanding of how respondents behave and the potential impact on data quality, especially in the later stages of large-scale surveys.