Relevance & Research Question
Data collection with smartphone sensors and apps is increasingly used in substantive and market research. However, app studies suffer from low response rates and adherence. To increase participant engagement, some app-based studies provide personalized feedback or tailored insights that are generated based on the data participants provide and shown back to them. To date, studies on the effectiveness of personalized feedback on participation and adherence have been inconclusive. Given that implementing such feedback is costly, better insights are needed. We use app process data (paradata) to answer the research question: ‘What is the effect of personalized feedback on participation rates and participant engagement in an app-based general population study?’.
Methods & Data
We use data collected using the Household Budged Survey app of Statistics Netherlands. A probability-based sample of 1441 households in the Netherlands was invited to download the app and report all their expenditures for two weeks. The app lets participants fill in their expenses manually or scan receipts. Random half of the households received personalized feedback: graphs and figures summarizing all their expenses. We use paradata of the participants’ actions in the app to study whether participants that received personalized feedback differ in their app usage and study adherence compared to participants that did not receive feedback.
Results

From 1441 households, 290 participated in the study. We find no effect on participation rates and study engagement when promising feedback in the invitation stage and in providing participants with personalized feedback. We will present detailed findings of our analysis of paradata to demonstrate how participants interacted with the personalized feedback (e.g., number and duration of engagements, changes to reporting of expenses in the feedback and no-feedback groups), as well as and whether the feedback had an effect on specific subgroups.

Added Value
The study advances our understanding of personalized feedback since paradata can show us how participants interact with personalized feedback. In light of our findings, researchers and practitioners can use personalized feedback as a tool to increase study engagement and/or make informed decisions on whether they should spend resources on research app feature of providing feedback.

Relevance & Research Question

As part of the EU project Smart Survey Implementation (SSI), the Federal Statistical Office of Germany (Destatis) is participating in the development of Smart Surveys. Smart Surveys combine traditional question-based surveys and smart features, that collect data by accessing device sensor data. One smart feature is a receipt scanner, making use of the smartphone camera, allowing participants to upload pictures of shopping receipts in a survey app. The aim is to reduce respondent burden in diary-based household budget surveys. However, smart features can only help to simplify surveys if they can be easily used. Therefore, the usability of the receipt scanning function was tested with potential respondents.

Methods & Data

Destatis conducted qualitative usability tests with a diary-based household budget app, featuring a prototype of a receipt scanner. 19 participants used the app, while their interaction with the app was observed, followed by an interview on their experiences.

Results

Given the choice between manual input and using the scanner, respondents prefer the scan function to record purchases. Participants appreciate the fast and easy way to record receipts, compared to the manual input of purchases. Making use of the scan function does not raise privacy concerns – at least not as long as an advantage is clearly given and as the publisher of the app is known and trusted.

All participants were able to use the scan function, although user friendliness of the current state of development proved to be insufficient. As a prototype was tested, the scan results were prone to errors, which had to be corrected. Respondents do not accept having to correct data, as the effort involved is perceived as too high and results are expected to be accurate.

Added Value

The study shows that a receipt scanning function per se is highly appreciated. However, in order to be used by the respondents, it is imperative that the function works pefectly, that its operation is easy to understand and involves little effort. Concerning the further development of this smart feature, the results confirm us in our approach but also show, where improvements are needed.

Relevance & Research Question

Diary studies are used to capture detailed respondent behaviour, but they are highly burdensome for respondents, resulting in nonresponse and measurement-errors. Smart surveys (i.e using a smartphone or activity tracker) can make use of sensors that can replace questions, reducing the response burden. Researchers can then collect diary data for a prolonged time period.

We aim to answer the question “How many days should we collect data for a smart diary study in behavioural statistics?”, considering the response burden and the inter- and intra-personal variability of statistics.

Methods & Data

We use data from four one-week smart diary studies: a physical activity study (2021, N = 414), a travel study (2018 and 2022, N = 185), and a budget study (2022, N = 10,421). All studies used probability-based samples, were conducted by Statistics Netherlands, and have different levels of response burden.

For each data source we calculate two statistics based on the frequency and duration of the studied behaviour. For each statistic we separate the inter-individual and intra-individual variance. We calculate the intraclass-correlation-coefficient (ICC), the proportion of intra-individual variance out of the total variance. We then calculate the reliability of a study period of 7 or fewer days, based on these variance components.

Results

The intra-individual variance for 7 days is high for each of the studies, meaning that not every day looks the same for studied individuals. The ICC is approximately 50% for travel statistics, 55% for physical activity statistics, and 90% for expenditure statistics. The reliability for the first two studies is around 0.80 at 7 days, meaning that 7 days provide enough information. While for expenditure, reliability is around 0.50 at 7 days, implying more days are useful.

Added Value

Our results can guide the decision on the number of diary days by taking into consideration burden and data quality. The results can help practitioners efficiently allocate resources (e.g., lower sample size, shorter data collection), reducing the costs. Our conclusions and recommendations have relevance across a wide range of research areas concerning studies on (time-use) behaviour, and studies using (smartphone) sensors or apps.