Relevance & Research Question
Cost- and time-efficient web surveys have replaced other survey modes. These efficiencies can potentially cover the increasing demand for survey data. However, since web surveys suffer from low response rates, researchers start considering social media platforms as sources for respondent recruitment. Although these platforms provide advertisement and targeting systems, the data quality and integrity of web surveys recruited through social media might be threatened by bots. Bots have the potential to shift survey outcomes and thus political and social decisions. This is alarming since there is almost no research on bots and their completion characteristics in web surveys. Importantly, existing research does not distinguish between conventional (rule-based) and sophisticated (AI-based) bots. We therefore address the following research question: Do bots varying in sophistication show different web survey completion characteristics?
Methods & Data
We programmed four bots with different levels of sophistication: two conventional (rule-based) and two sophisticated (AI-based) bots. To this end, we conducted a literature search to compile a list of capabilities that are key for bots to successfully complete web surveys. The two AI-based bots were additionally linked to the Large Language Model “Gemini Pro” (Google). We ran each bot N = 100 times through a web survey on equal gender partnerships and tested several bot prevention and detection measures, such as CAPTCHAs and invisible honey pot questions.
Results
The results indicate that both rule-based and AI-based bots come with impressive completion rates (up to 100%). In addition, we can prove conventional wisdom about bots in web surveys wrong: CAPTCHAs and honey pot questions pose no challenges. However, there are clear differences between rule-based and AI-based bots when it comes to web survey completion. For example, conventional bots are faster and select more options in check-all-that-apply questions than their AI-based counterparts.
Added Value
By distinguishing rule-based and AI-based bots, this study stands out of previous studies and shows that both types of bots have different completion characteristics. In addition, it provides a useful framework for future research on the prevention and detection of bots in web surveys.

Relevance and Research Question

Self-report measures for sensitive topics, such as stereotypes and prejudices, are often compromised by social desirability bias. Indirect psychological bias tests offer a promising alternative by measuring implicit biases through reaction times, decision errors under time pressure, priming effects, and memory performance. Traditionally, these tests are conducted in controlled lab environments, which limits the sample size and diversity due to logistical constraints. However, when conducted online, researchers cannot control participants' choice of device - but device type is associated with systematic differences (e.g. screen size, input method, and test environment) that may influence results. We developed a tool for integrating indirect psychological bias tests into online surveys: MIND.set. This study used MIND.set to investigate two key questions: 1) How reliably can implicit biases be detected in online survey contexts? and 2) How does the type of device used (e.g., mobile vs. desktop) impact test outcomes?

Methods and Data

In 2023, we conducted an online survey with 2,707 participants from the general population, using quotas for gender, age, and education. Participants were randomly assigned to one of five indirect bias tests implemented via the MIND.set platform: Implicit Association Test (IAT), Affect Misattribution Procedure (AMP), Shooter Task (ST), Avoidance Task (AT), and Source Monitoring Paradigm (SMP). All tests focused on stereotypes of Arab-looking men versus White men, specifically regarding perceived threat. Participants self-selected their devices (mobile or desktop), and our preregistered hypotheses (OSF) examined the influence of device type on bias detection and bias scores.

Results

The analyses confirmed implicit biases on at least one bias indicator across all five tests. Crucially, bias scores were largely unaffected by device type. While minor variations were observed, these did not significantly undermine the reliability of results across different devices.

Added Value

The MIND.set platform enhances the accessibility of indirect bias testing by offering a robust infrastructure for online research. This study is the first to systematically investigate device effects across multiple indirect bias tests, providing critical insights for researchers seeking to incorporate such tests into online surveys.

Relevance & Research Question

Accurate coding of open-ended responses in quantitative surveys is critical for generating insights. However, traditional manual coding methods are time-consuming and costly. LLMs present an opportunity to revolutionize this process. The research question explored in this study: How can LLM-based coding be optimized to outperform both human coders and baseline LLM implementations in terms of accuracy?

The goal of this research is to better understand how to improve automated coding via foundation models, and to assess the impact on coding quality of various strategies aimed at improving on vanilla LLM use. In particular, we considered the effect of:

1/ few shot learning: how helpful is it to provide general or case-based examples?

2/ prompt optimization: what is the best way to ask the LLM to perform the seemingly easy task of applying labels to verbatims?

3/ input optimization: is there a way to format input labels so as to make it easier for the LLM to correctly apply them?

4/ model choice: do newer generation LLMs fare better than older ones? are quicker/lighter models good enough?

Methods & Data

This research employed comparative tests across 4 coding methods tested on multiple datasets from real-world surveys: (1) Human manual coding by client (=benchmark), (2) Human coding by external suppliers, (3) Initial implementation of an LLM-based coding tool (BARI V1), and (4) An optimized version of the LLM tool (BARI V2) enhanced through iterative improvements in prompt engineering, training data alignment and feedback loops. Key performance metrics being always coding accuracy (measured against benchmark ‘Client’).
Results

Our results suggest that:

1/ general few shot learning is not particularly helpful

2/ some prompting strategies do fare better, specially on trickier inputs

3/ input format actually crucial the most crucial factor

4/ smaller models aren't bad compared to bigger ones

Added Value

This study showcases how optimizations of LLMs can bridge the gap between AI and human in coding open-ended survey responses. The findings provide insights into leveraging AI for more efficient and accurate data analysis, highlighting a transformative approach for researchers, practitioners, and industry stakeholders.