Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Saved in:
| Title: | Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization |
|---|---|
| Language: | English |
| Authors: | Simone Zhang (ORCID |
| Source: | Sociological Methods & Research. 2025 54(3):1197-1242. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 46 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Artificial Intelligence, Online Surveys, Responses, Social Science Research, Research Problems, Computer Use |
| DOI: | 10.1177/00491241251327130 |
| ISSN: | 0049-1241 1552-8294 |
| Abstract: | The growing popularity of generative artificial intelligence (AI) tools presents new challenges for data quality in online surveys and experiments. This study examines participants' use of large language models to answer open-ended survey questions and describes empirical tendencies in human versus large language model (LLM)-generated text responses. In an original survey of research participants recruited from a popular online platform for sourcing social science research subjects, 34 percent reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants' LLM use in online research. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1475709 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | The growing popularity of generative artificial intelligence (AI) tools presents new challenges for data quality in online surveys and experiments. This study examines participants' use of large language models to answer open-ended survey questions and describes empirical tendencies in human versus large language model (LLM)-generated text responses. In an original survey of research participants recruited from a popular online platform for sourcing social science research subjects, 34 percent reported using LLMs to help them answer open-ended survey questions. Simulations comparing human-written responses from three pre-ChatGPT studies with LLM-generated text reveal that LLM responses are more homogeneous and positive, particularly when they describe social groups in sensitive questions. These homogenization patterns may mask important underlying social variation in attitudes and beliefs among human subjects, raising concerns about data validity. Our findings shed light on the scope and potential consequences of participants' LLM use in online research. |
|---|---|
| ISSN: | 0049-1241 1552-8294 |
| DOI: | 10.1177/00491241251327130 |
