View in EDS

AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08

Saved in:
Bibliographic Details
Title: AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08
Language: English
Authors: Ikkyu Choi, Jiangang Hao, Chen Li, Michael Fauss, Jakub Novák
Source: ETS Research Report Series. Dec 2024.
Availability: ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/
Peer Reviewed: Y
Page Count: 18
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Descriptors: Writing Tests, Automation, Cheating, Plagiarism, Proximity, Program Effectiveness, Language Proficiency, English (Second Language), Language Tests
Assessment and Survey Identifiers: Test of English as a Foreign Language
ISSN: 2330-8516
Abstract: A frequently encountered security issue in writing tests is nonauthentic text submission: Test takers submit texts that are not their own but rather are copies of texts prepared by someone else. In this report, we propose AutoESD, a human-in-the-loop and automated system to detect nonauthentic texts for a large-scale writing tests, and report its performance on an operational data set. The AutoESD system utilizes multiple automated text similarity measures to identify suspect texts and provides an analytics-enhanced web application to help human experts review the identified texts. To evaluate the performance of AutoESD, we obtained its similarity measures on "TOEFL iBT®" test writing responses collected from multiple remote administrations and examined their distributions. The results were highly encouraging in that the distributional characteristics of AutoESD similarity measures were effective in identifying suspect texts and the measures could be computed quickly without affecting the operational score turnaround timeline.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1459542
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1459542
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1459542
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ikkyu+Choi%22">Ikkyu Choi</searchLink><br /><searchLink fieldCode="AR" term="%22Jiangang+Hao%22">Jiangang Hao</searchLink><br /><searchLink fieldCode="AR" term="%22Chen+Li%22">Chen Li</searchLink><br /><searchLink fieldCode="AR" term="%22Michael+Fauss%22">Michael Fauss</searchLink><br /><searchLink fieldCode="AR" term="%22Jakub+Novák%22">Jakub Novák</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ETS+Research+Report+Series%22"><i>ETS Research Report Series</i></searchLink>. Dec 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 18
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Writing+Tests%22">Writing Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Cheating%22">Cheating</searchLink><br /><searchLink fieldCode="DE" term="%22Plagiarism%22">Plagiarism</searchLink><br /><searchLink fieldCode="DE" term="%22Proximity%22">Proximity</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Proficiency%22">Language Proficiency</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Test+of+English+as+a+Foreign+Language%22">Test of English as a Foreign Language</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2330-8516
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: A frequently encountered security issue in writing tests is nonauthentic text submission: Test takers submit texts that are not their own but rather are copies of texts prepared by someone else. In this report, we propose AutoESD, a human-in-the-loop and automated system to detect nonauthentic texts for a large-scale writing tests, and report its performance on an operational data set. The AutoESD system utilizes multiple automated text similarity measures to identify suspect texts and provides an analytics-enhanced web application to help human experts review the identified texts. To evaluate the performance of AutoESD, we obtained its similarity measures on "TOEFL iBT®" test writing responses collected from multiple remote administrations and examined their distributions. The results were highly encouraging in that the distributional characteristics of AutoESD similarity measures were effective in identifying suspect texts and the measures could be computed quickly without affecting the operational score turnaround timeline.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1459542
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1459542
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
    Subjects:
      – SubjectFull: Writing Tests
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Cheating
        Type: general
      – SubjectFull: Plagiarism
        Type: general
      – SubjectFull: Proximity
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Language Proficiency
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: Test of English as a Foreign Language
        Type: general
    Titles:
      – TitleFull: AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ikkyu Choi
      – PersonEntity:
          Name:
            NameFull: Jiangang Hao
      – PersonEntity:
          Name:
            NameFull: Chen Li
      – PersonEntity:
          Name:
            NameFull: Michael Fauss
      – PersonEntity:
          Name:
            NameFull: Jakub Novák
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-electronic
              Value: 2330-8516
          Titles:
            – TitleFull: ETS Research Report Series
              Type: main
ResultId 1