Relevance & Research Question

Linking web tracking and survey data opens new research areas. We record participants browsing behavior via a browser plugin. Tracking allows for measuring behavior that individuals tend to recall inaccurately and reduces the survey burden. However, previous studies found that participants were reluctant to participate in tracking studies. To increase participation rates, monetary incentives are widely used. These can be granted unconditionally, conditional on participation or as a combination of both. It is, however, unclear (1) how large conditional incentives should be and whether unconditional incentives can additionally increase participation rates. Additionally, we are interested in, (2) whether these effects are the same for a convenience sample and a probability-based sample.

Methods & Data

To answer our research questions, we conduct a 2x3 factorial experiment with approximately 2600 panelists of a new panel. Panelists are recruited via Meta-ads and via a German general population survey (ALLBUS). The first factor is whether panelists receive a prepaid incentive of 5 Euro or not. The second factor is the amount of the postpaid incentive (10,25 or 40 Euro), conditional on 30 out of 60 active days in the tracking period.

We investigate (1a) consent for participation in a web tracking study, (1b) actual installation of the browser plugin. We will present logistic regression models. Additionally, (2) we will investigate the differences between Meta and ALLBUS recruited participants.

Results

Using a smaller dataset of our first field period including only participants recruited via Meta-ads, we find that the unconditional incentive has a positive relation with consent but not with installation. For the amount of the conditional incentive, we cannot see an effect yet. We will analyze a larger dataset for the results that will be presented at the conference. Results for our second research question will be available by the time of the conference.

Added Value

Little is known on how incentives and other factors impact participation in tracking studies as most research only investigates hypothetical consent. Our study adds to the knowledge on the incentive amount needed to recruit participants via Meta and via a general population survey into a web tracking study.

Relevance & Research Question:

Vignette and conjoint experiments are extensively utilised in the field of social sciences. These methodologies assess preferences in hypothetical scenarios, exploring decision-making in complex choices with varied attributes, aiming to align survey responses more closely with real-world decisions. However, survey experiments are only valid externally to the extent that stated intentions align with real-world behaviour. This study uses a unique dataset that allows us to compare the outcomes of a vignette experiment (which assessed willingness to participate in a social media data donation study) to actual participation in a real data donation study involving the same survey respondents.

Methods & Data:

A vignette experiment embedded in an online survey of a non-probability-based panel was conducted in Hungary in May 2022 (n=1000). Respondents expressed their willingness to participate in hypothetical data donation studies. In a mixed factorial design, five treatment dimensions were varied in the study descriptions (platform, range of data, upload/download time, monetary-, and non-monetary incentive). In February 2023, the same participants were invited to a real data donation study which had almost the same characteristics as the ones described in the vignettes.

Results:

The correlation between the self-reported willingness and actual participation was only 0.29. Moreover, the drivers of willingness and actual participation were different. For instance, education was one of the strongest predictors of willingness, yet was not significantly associated with actual participation. We also found differences regarding the effect of privacy beliefs or the Big-Five personality traits.

Added Value:

This study contributed to the literature by validating the results of a vignette experiment using within-person comparisons from behavioural data. The results suggest that vignette experiments may strongly suffer from hypothetical or other biases, at least in scenarios when the personal risk and burden are high, and underscore the importance of improving external validity of such experiments.

Relevance & Research Question

Using apps as a survey mode offers promising features. The use of passive measurements on smartphones can be beneficial for response burden, by replacing traditional survey questions, and for data quality as it can reduce recall bias. However, not everyone is able or willing to participate in an app-based study, causing coverage issues and nonresponse. We investigate whether a mixed-mode design can be effective for our goals by analyzing who chooses to participate in an app study and who prefers a web questionnaire.

Methods & Data

We report on a study by Statistics Netherlands (winter 2022-2023) for which we invited 2544 individuals from a cross-sectional sample of the Dutch population. We asked individuals to use a smartphone app to collect their travel data, or participate in a web questionnaire. We combine a concurrent mixed-mode design with a “push-to-app” design by offering the web questionnaire at different moments: directly in the invitation letter or in one of the reminders. Invitees are asked to participate in one mode. We assess whether participation is related to individual characteristics with registry data.

Results

More people register in the app (11.5%) than in the questionnaire (7.0%). Total registration rates are higher when the web questionnaire is offered directly (19.8%) than in the first (18.5%) or second reminder (15.8%). The app registration rate does not increase much when the web questionnaire is offered later, suggesting that certain people have a mode-preference for the app. Most striking is the age effect. The app attracts younger participants while older participants are overrepresented in the web questionnaire. Combining the two yields a more balanced sample.

Added Value

We show that with a mixed-mode design, we can attract more respondents than with an app-only design in a probability based sample. With the use of population registries we are able to improve our understanding of who participates in app- and web-studies. Additionally, our analysis can contribute to the design of future diary studies combining a smartphone app and a web questionnaire.