Relevance & Research Question:

Survey measures of political knowledge, widely used in political science research, face challenges in online administration due to potential cheating. Previous research reveals a significant proportion of participants resort to online searches when answering political knowledge questions, casting doubt on measurement quality. Existing studies testing potential interventions to curb cheating have relied on indirect measures of cheating, such as catch questions. This study introduces a novel approach, employing direct observations of participants' Internet browsing via web trackers, combined with an experimental design testing two strategies to prevent cheating (instructions and time limit). The paper explores three research questions: what proportion of participants looks up information when posed political knowledge questions (RQ.1)? What is the impact of the interventions on the likelihood of individuals looking up information (RQ.2)? How do estimates from direct observations differ from indirect proxies (e.g., self-reports, paradata) (RQ.3)?
Methods & Data:

A web survey experiment (N = 1,200) in Spain was deployed within an opt-in access online panel. Cross quotas for age and gender, and quotas for educational level, and region were used to ensure a sample matching on these variables to the Internet adult population. Participants answered six knowledge questions on political facts and current events. Cheating was identified by analysing URLs from web tracking data, and alternative indirect measures were applied, including catch questions, self-reports, and paradata.
Results:

Two noteworthy patterns emerge. Firstly, cheating prevalence from web tracking data is below 5%, markedly smaller than levels estimated by indirect measures (2 to 7 times larger). Secondly, based on web tracking data the anti-cheating interventions have no effect. Nonetheless, using indirect measures of cheating we find that both interventions significantly reduce the likelihood of cheating.
Added Value:

This study pioneers the integration of web tracking data and experimental design to examine cheating in online political knowledge assessments. Despite requiring further validation, the substantial differences between web tracking data and indirect approaches suggest two competing conclusion: either cheating in online surveys is substantially lower than first thought, or web tracking data may not be suitable for identifying cheating in online surveys.

Relevance & Research Question
Designing high-quality survey questions is crucial for reliable and valid research outcomes. However, this process often relies on subjective expertise. In response to this challenge, Saris and colleagues developed the Survey Quality Predictor (SQP), a web-based tool to predict the quality of survey items for continuous latent variables. The research questions driving this presentation are: How can the quality of survey items be predicted? How can SQP 3.0 be effectively integrated into the questionnaire development process?
Methods & Data
The quality prediction algorithm (i.e., random forest) of the latest SQP version (3.0) is grounded in a comprehensive analysis involving more than 6,000 survey questions that were part of multitrait-multimethod (MTMM) experiments in 28 languages and 33 countries. The quality prediction of new survey items is based on their linguistic and formal characteristics (e.g., layout and polarity of the answer scale). It is important to note that SQP is not designed to replace traditional methods like cognitive pretesting but serves as a complementary tool in the development phase of questionnaires.
Results
This presentation showcases practical applications of SQP 3.0 in the questionnaire development process. The audience will gain insights into how SQP predicts the quality of survey items. Also, researchers will get to know how they can leverage SQP to identify survey items, enhance item quality before data collection, and detect discrepancies between source and translated versions of survey items.
Added Value
By incorporating SQP into the questionnaire development toolkit, researchers can enhance the efficiency and objectivity of their survey design processes, ultimately contributing to the advancement of survey research methodologies. In addition, I will highlight the collaborative nature of SQP as an ongoing and evolving research project on survey data quality, emphasizing avenues for potential collaboration among researchers.

Relevance & Research Question: Probability-based online and mixed-method panels are widely used in scientific research, but not as much for market research or political opinion polling. This presentation will explore the case of "Panelstory", the first Hungarian probability-based mixed-method panel, which was established in 2022 with the purpose of utilizing scientific methods to address market research and political opinion polling issues.
Methods & Data: We will provide a thorough assessment of panel data based on the total survey error framework to evaluate the quality of indicators such as financial situation, alcohol consumption, interest in politics, health, marital status and media use. Additionally, we will examine the panel composition, response rates, dropout, and recruitment statistics. Non-probability online data collections, face-to-face surveys, and administrative data will be used as reference points. We also relate this to the characteristics of Internet penetration.
Results: The research conducted thus far has revealed that Hungary's Internet penetration rate (82 percent) necessitates a mixed-method design. This is due to the fact that a clear pattern of Internet penetration has been identified in correlation with the indicators being studied. Based on the characteristics of internet penetration in Hungary, in 67 percent of the estimates were biased. For relevant research dimensions such as interest in politics, religiosity, health and marital status, the online data collection significantly under- or overestimates the likely real population proportions.The results of single-mode and mixed-method are notably different in terms of all of the indicators tested.
Added Value: It is especially important to assess how surveys from probability-based online and mixed-method panels compare to traditional methods such as face-to-face and single-mode designs. This presentation will provide a discussion of a new panel, highlighting both the advantages and potential issues of using scientific results in terms of data quality.