Relevance & Research Question
Nowadays, designing online surveys means designing for mixed device surveys. One of the challenges in designing mixed device surveys is the presentation of grid questions. In this experiment we compare various design options for grid questions. Our main research questions are: 1) To what extent do these different grid designs differ with respect to response quality and respondent satisfaction? 2) Does this differ for respondents on PCs and respondents on smartphones?
Methods & Data In 2023 an experiment was conducted with a sample of 12060 persons of the general Dutch population aged 16 and older. Sample units were randomly assigned to an online survey in either the standard stylesheet as currently used by Statistics Netherlands (n=2824, 40% of the sample) or an experimental stylesheet (n=7236, 60% of the sample).

Within the current stylesheet, half of the sample units were randomly assigned to the standard grid design as currently used (a table format for large screens and a stem-fixed vertical scrollable format for small screens) and the other half to a general stem-fixed grid design (stem-fixed design for both the large and the small screen). Within the experimental stylesheet, one third of the sample was randomly assigned to either the general stem-fix grid design, a carrousel grid design (in which only one item is displayed at the time and after answering one item, the next item automatically ‘flies in‘) or an accordion grid design (all items are presented vertically on one page, and answer options are automatically closed and unfolded after an item is answered).

Various indicators are used to assess response quality, e.g. break-off, item non response, straightlining, mid-point reporting. Respondent satisfaction is assessed with a set of evaluation questions at the end of the questionnaire.

Results Data are currently being analyzed.

Added Value This experiment with a general population sample adds to the knowledge of previous studies on grids. which have mainly been conducted with (access) panels.

Towards a mobile web questionnaire for the Vacation Survey: UX design challenges

Vivian Meertens & Maaike Kompier

Key words: Mobile Web Questionnaire Design, Smartphone First Design, Vacation Survey, Statistics Netherlands, UX testing, Qualitative Approach, Mixed Device Surveys

Relevance & Research Question: —your text here—

Despite the fact that online surveys are not always fit for small screens and mobile device navigation, the number of respondents that start online surveys on mobile devices instead of PC or laptop device, is still growing. Statistics Netherlands (CBS) has responded to this trend by developing and designing mixed device surveys. This study focuses on the redesign of the Vacation Survey, applying a smartphone first approach.

The Vacation Survey is a web only panel survey, that could only be completed on a PC or laptop. The layered design with a master detail approach was formatted in such a way that a large screen was needed to be able to complete the questionnaire. Despite a warning in the invitation letter that a PC or laptop should be used to complete the questionnaire, 14.5% of first-time logins in 2023 were via smartphones, resulting in a redesign with a smartphone first approach. The study examines the applicability and understandability of the Vacation Survey’s layered design, specifically its master-detail approach, from a user experience (UX) design perspective.

Results: —your text here—
This study shares key findings of the qualitative UX test conducted at the CBS Userlab. It will explore how visual design aspects influence respondent behaviour on mobile devices, stressing the importance of observing human interaction when filling in a questionnaire on a mobile phone. The results emphasize the need for thoughtful UX design in mobile web questionnaires to enhanced user engagement and response accuracy.

Added Value: —your text here
The study provides valuable insights into challenges and implications of transitioning social surveys to mobile devices. By discussing the necessary adaptations for a functional, user-friendly mobile questionnaire, this research contributes to the broader field of survey methodology, offering guidance for future survey designs that accommodate the growing trend of mobile device usage.

Relevance & Research Question: Recall-based travel diaries require respondents to report their travel behaviour over a period ranging from one to seven days. During this period, they are asked to indicate the start and end times and locations, modes of transport, distances, and the number of people on each trip. Depending on the mode, additional questions are asked to gather information on ticket types and costs or fuel types. Due to the specificity of the requested information and its non-centrality for most respondents, travel diaries pose a substantial burden, increasing the risk of satisficing behaviours and trip underreporting. Methods & Data: In this presentation, we describe key decisions made during the design of the Wales National Travel Survey. This push-to-web project includes a questionnaire and a 2-day travel diary programmed into the survey. Results: Critical aspects of these decisions include the focus of the recall (trip, activity, or location based) and the sequence of follow-up questions (interleaved vs. roster approach). Recent literature suggests that location-based diaries align better with respondents’ cognitive processes than trip-based diaries and help reduce underreporting. Therefore, a location-based travel diary was proposed with an auto-complete field to match inputs with known addresses or postcodes. Interactive maps were also proposed for user testing. While they can be particularly useful when respondents have difficulty describing locations or when places lack formal addresses, previous research warns that advanced diary features can increase drop-off rates. Regarding the follow-up sequence, due to mixed findings in the literature and limited information on the performance of these approaches in web-based travel diaries, experimentation is planned to understand how each approach performs in terms of the accuracy of the filter questions and the follow-up questions. Additionally, this presentation discusses the challenges and options for gathering distance data in recall-based travel diaries, along with learnings from the early phases of diary testing based on the application of a Questionnaire Appraisal System and cognitive/usability interviews. Added Value: These findings offer valuable insights into the design of complex web-based surveys with multiple loops and non-standard features, extending beyond travel diaries.