View in EDS

Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study

Saved in:
Bibliographic Details
Title: Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study
Language: English
Authors: Masaki Eguchi, Kotaro Takizawa (ORCID 0000-0003-2166-9996), Mao Saeki, Fuma Kurata, Shungo Suzuki (ORCID 0000-0002-6327-3298), Yoichi Matsuyama, Yasuyo Sawaki
Source: TESOL Quarterly: A Journal for Teachers of English to Speakers of Other Languages and of Standard English as a Second Dialect. 2025 59(1):S183-S219.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 37
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Artificial Intelligence, Role Playing, Foreign Countries, College Students, Communicative Competence (Languages), English (Second Language), Computer Uses in Education, Second Language Learning, Second Language Instruction, Dialogs (Language)
Geographic Terms: Japan
DOI: 10.1002/tesq.70028
ISSN: 0039-8322
1545-7249
Abstract: This study investigates the nature of co-construction in roleplays conducted with human versus AI interlocutors for assessing interactional competence (IC) in L2 English. Seventy-five university students in Japan completed roleplay tasks with both human tutors and an AI agent. The AI agent is a multimodal dialog system integrated with a large language model (LLM), designed to allow synchronous interaction with the participant through autonomous turn-taking. Using conversation analysis, 24 interactions were analyzed to investigate how participants managed preference organization, sequence expansion, and turn-taking. The analysis revealed that the AI-delivered roleplays elicited some IC-relevant practices and that participants treated the roleplay as a co-constructed interaction, responding contingently to the AI's contributions. While the data suggested both human and AI interlocutors maintained mutual understanding, striking differences in turn-taking practices were observed, including more frequent overlaps and inter-turn gaps in the AI-delivered condition. The study concludes that LLM-integrated multimodal dialog systems, by producing recognizable verbal actions and multimodal signals, have the potential to effectively elicit co-constructed interactional performances relevant to IC assessment.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1490581
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1490581
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Masaki+Eguchi%22">Masaki Eguchi</searchLink><br /><searchLink fieldCode="AR" term="%22Kotaro+Takizawa%22">Kotaro Takizawa</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2166-9996">0000-0003-2166-9996</externalLink>)<br /><searchLink fieldCode="AR" term="%22Mao+Saeki%22">Mao Saeki</searchLink><br /><searchLink fieldCode="AR" term="%22Fuma+Kurata%22">Fuma Kurata</searchLink><br /><searchLink fieldCode="AR" term="%22Shungo+Suzuki%22">Shungo Suzuki</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6327-3298">0000-0002-6327-3298</externalLink>)<br /><searchLink fieldCode="AR" term="%22Yoichi+Matsuyama%22">Yoichi Matsuyama</searchLink><br /><searchLink fieldCode="AR" term="%22Yasuyo+Sawaki%22">Yasuyo Sawaki</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22TESOL+Quarterly%3A+A+Journal+for+Teachers+of+English+to+Speakers+of+Other+Languages+and+of+Standard+English+as+a+Second+Dialect%22"><i>TESOL Quarterly: A Journal for Teachers of English to Speakers of Other Languages and of Standard English as a Second Dialect</i></searchLink>. 2025 59(1):S183-S219.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 37
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Role+Playing%22">Role Playing</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Communicative+Competence+%28Languages%29%22">Communicative Competence (Languages)</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Uses+in+Education%22">Computer Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Instruction%22">Second Language Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Dialogs+%28Language%29%22">Dialogs (Language)</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Japan%22">Japan</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/tesq.70028
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0039-8322<br />1545-7249
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study investigates the nature of co-construction in roleplays conducted with human versus AI interlocutors for assessing interactional competence (IC) in L2 English. Seventy-five university students in Japan completed roleplay tasks with both human tutors and an AI agent. The AI agent is a multimodal dialog system integrated with a large language model (LLM), designed to allow synchronous interaction with the participant through autonomous turn-taking. Using conversation analysis, 24 interactions were analyzed to investigate how participants managed preference organization, sequence expansion, and turn-taking. The analysis revealed that the AI-delivered roleplays elicited some IC-relevant practices and that participants treated the roleplay as a co-constructed interaction, responding contingently to the AI's contributions. While the data suggested both human and AI interlocutors maintained mutual understanding, striking differences in turn-taking practices were observed, including more frequent overlaps and inter-turn gaps in the AI-delivered condition. The study concludes that LLM-integrated multimodal dialog systems, by producing recognizable verbal actions and multimodal signals, have the potential to effectively elicit co-constructed interactional performances relevant to IC assessment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1490581
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1490581
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/tesq.70028
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 37
        StartPage: S183
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Role Playing
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Communicative Competence (Languages)
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Computer Uses in Education
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Second Language Instruction
        Type: general
      – SubjectFull: Dialogs (Language)
        Type: general
      – SubjectFull: Japan
        Type: general
    Titles:
      – TitleFull: Human- versus Artificial Intelligence-Delivered Roleplay Tasks for Assessing Interactional Competence: An Applied Conversation Analytic Study
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Masaki Eguchi
      – PersonEntity:
          Name:
            NameFull: Kotaro Takizawa
      – PersonEntity:
          Name:
            NameFull: Mao Saeki
      – PersonEntity:
          Name:
            NameFull: Fuma Kurata
      – PersonEntity:
          Name:
            NameFull: Shungo Suzuki
      – PersonEntity:
          Name:
            NameFull: Yoichi Matsuyama
      – PersonEntity:
          Name:
            NameFull: Yasuyo Sawaki
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0039-8322
            – Type: issn-electronic
              Value: 1545-7249
          Numbering:
            – Type: volume
              Value: 59
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: TESOL Quarterly: A Journal for Teachers of English to Speakers of Other Languages and of Standard English as a Second Dialect
              Type: main
ResultId 1