Relevance & Research Question: Simple descriptive results can (now) be generated with ChatGPT in a relatively resource-efficient way - e.g. by generating a syntax code at the push of a button and then supporting the user in interpreting the output. The skills required to formulate the prompts and validate the descriptive results generated are still rather "manageable" compared to more complex analysis procedures (e.g. classification analysis). Errors that can occur start with the use of inappropriate analysis methods and range from the use of incorrect syntax to incorrect interpretation of results. This leads to the question: what are the benefits - and what are the risks of generating (and possibly using) incorrect results - of the new possibilities offered by AI-based data analysis?
Methods & Data: Based on this, the article will focus on the following aspects (using replication datasets with available, tested syntax codes) in the course of replication of already published studies
- Errors in the generated syntax code (e.g. omitting important steps, suggesting inappropriate statistical tests)
- Number of trials required (until an acceptable result is obtained or it is determined that no 'acceptable' result can be obtained)
- Usefulness of the results (e.g., clarity of interpretation, compactness).
Results: The findings can be summarised as follows. The use of tools such as ChatGPT
(1) is convincing for generating a decision basis for the choice of analysis method.
(2) can support simple descriptive data analysis, result description and interpretation in a resource-efficient way.
(3) is only advisable for generating a syntax for carrying out a complex procedure (e.g. MDS) with the appropriate expertise (possible for checking and adapting various specifics) - as it is very error-prone.
Added Value: The presentation focuses on the application of AI-supported data analysis in 'everyday research', which can be amateurish in nature, and emphasises the need to have the necessary skills to ensure the required quality of results. The aim of the resulting research is to develop a specific strategy for efficient 'scientific use'.