Relevance & research question:

Researchers are increasingly using social media platforms for survey recruitment. Typically, advertisements are distributed through these platforms to motivate users to participate in an online survey. To date, there is little empirical evidence on how the content and design characteristics of advertisements can affect response quality in surveys based on social media recruitment. This project is the first comprehensive study of the effects of ad design on response quality in surveys recruited via social media.

Methods and data:

We use data from the SoMeRec survey, which was conducted via Facebook ads in Germany and the United States in June 2023 and focused primarily on climate change and migration. The survey ad campaign featured 15 images with different thematic associations to climate change and migration, including strong and loose associations and neutral images. A commercial access panel company was contracted to include identical survey questions serving as benchmark comparison. The Facebook sample consisted of 7,139 respondents in Germany and 13,022 in the US, while the access panel consisted of 1,555 surveys in Germany and 1,576 surveys in the US. In our analyses, we compare common data quality indicators, including completion time, straightlining, item non-response, and follow-up availability, across different ad features.

Results:

First analysis show that survey completion time is higher for thematic ad designs compared to neutral ads and the reference sample. There are differences in the overall item non-response rate, with higher item non-response for the immigration-themed ad designs. There are no significant differences in straightlining between samples and ad designs. Finally, respondents recruited through neutral ads were more likely to be available for follow up surveys than those recruited through themed ads.

Added value:

Our study advances the literature by studying the general population in Germany and the US, by testing various indicators of survey data quality, and by including a benchmark survey of respondents not recruited through social media. The results clearly indicate an effect of ad design on survey data quality and highlight the importance of sample and recruitment design for estimates based on social media recruitment and online surveys.

Relevance & Research Question: Social media advertisement is becoming an increasingly popular method of recruiting participants for studies in the social sciences. Recently, more and more participants of surveys are recruited via social media. This method of recruitment has been particularly prominent for recruiting special populations for surveys, such as migrants or LGBT persons, but recently meta has significantly reduced these selection criteria. However, meta still allows the selection of common socio-demographic characteristics, such as age and gender, when placing an ad. Meta estimates these socio-demographic characteristics based on the user's data. With this information, we took an non-probabilistic quota-sampling-like approach by specifying to meta the desired peoples' proportions for socio-demographic characteristics which should click on the ad and be directed to the recruitment survey of our nonprobabilistic panel.

However, the volatile and hard to control nature of social media recruitment opens it up to scrutiny and demands evaluation. In this study we assess coverage issues and cost effectiveness of utilizing Meta advertisement in recruiting respondents for a non-probabilistic online panel, we consider three aspects in detail. First, we evaluate the extent to which the targeting criteria, namely age and gender achieve a balanced sample at different stages of the registration process into the panel and give recommendations for adjustments. Furthermore, we validate whether these social media targeting criteria are reliable and agree with the survey answers. Third, we assess the cost structure in the light of the response propensities at the different stages of the recruitment process and investigate whether expensive social media ad groups pay off in the long-term.

Methods & Data: We are using data from the recruiment of the new GESIS Panel Plus. The recuitmenr process includes several steps and we sill consider each step individually using multivariate analysis methods.

Results: First results suggest that expensive recruitment groups do not pay off in the long term.

Added Value: These research will open up the black box of cost structure in relation to socio - demographic attributes when using Meta as recruitment frame for cross-sectional and longitudinal surveys.