Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study
Saved in:
| Title: | Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study |
|---|---|
| Language: | English |
| Authors: | Masaki Eguchi, Kotaro Takizawa (ORCID |
| Source: | TESOL Quarterly: A Journal for Teachers of English to Speakers of Other Languages and of Standard English as a Second Dialect. 2025 59(1):S183-S219. |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 37 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Artificial Intelligence, Role Playing, Foreign Countries, College Students, Communicative Competence (Languages), English (Second Language), Computer Uses in Education, Second Language Learning, Second Language Instruction, Dialogs (Language) |
| Geographic Terms: | Japan |
| DOI: | 10.1002/tesq.70028 |
| ISSN: | 0039-8322 1545-7249 |
| Abstract: | This study investigates the nature of co-construction in roleplays conducted with human versus AI interlocutors for assessing interactional competence (IC) in L2 English. Seventy-five university students in Japan completed roleplay tasks with both human tutors and an AI agent. The AI agent is a multimodal dialog system integrated with a large language model (LLM), designed to allow synchronous interaction with the participant through autonomous turn-taking. Using conversation analysis, 24 interactions were analyzed to investigate how participants managed preference organization, sequence expansion, and turn-taking. The analysis revealed that the AI-delivered roleplays elicited some IC-relevant practices and that participants treated the roleplay as a co-constructed interaction, responding contingently to the AI's contributions. While the data suggested both human and AI interlocutors maintained mutual understanding, striking differences in turn-taking practices were observed, including more frequent overlaps and inter-turn gaps in the AI-delivered condition. The study concludes that LLM-integrated multimodal dialog systems, by producing recognizable verbal actions and multimodal signals, have the potential to effectively elicit co-constructed interactional performances relevant to IC assessment. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1490581 |
| Database: | ERIC |
| Abstract: | This study investigates the nature of co-construction in roleplays conducted with human versus AI interlocutors for assessing interactional competence (IC) in L2 English. Seventy-five university students in Japan completed roleplay tasks with both human tutors and an AI agent. The AI agent is a multimodal dialog system integrated with a large language model (LLM), designed to allow synchronous interaction with the participant through autonomous turn-taking. Using conversation analysis, 24 interactions were analyzed to investigate how participants managed preference organization, sequence expansion, and turn-taking. The analysis revealed that the AI-delivered roleplays elicited some IC-relevant practices and that participants treated the roleplay as a co-constructed interaction, responding contingently to the AI's contributions. While the data suggested both human and AI interlocutors maintained mutual understanding, striking differences in turn-taking practices were observed, including more frequent overlaps and inter-turn gaps in the AI-delivered condition. The study concludes that LLM-integrated multimodal dialog systems, by producing recognizable verbal actions and multimodal signals, have the potential to effectively elicit co-constructed interactional performances relevant to IC assessment. |
|---|---|
| ISSN: | 0039-8322 1545-7249 |
| DOI: | 10.1002/tesq.70028 |
