AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08
Saved in:
| Title: | AutoESD: An Automated System for Detecting Nonauthentic Texts for High-Stakes Writing Tests. Research Report. ETS RR-24-08 |
|---|---|
| Language: | English |
| Authors: | Ikkyu Choi, Jiangang Hao, Chen Li, Michael Fauss, Jakub Novák |
| Source: | ETS Research Report Series. Dec 2024. |
| Availability: | ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/ |
| Peer Reviewed: | Y |
| Page Count: | 18 |
| Publication Date: | 2024 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Writing Tests, Automation, Cheating, Plagiarism, Proximity, Program Effectiveness, Language Proficiency, English (Second Language), Language Tests |
| Assessment and Survey Identifiers: | Test of English as a Foreign Language |
| ISSN: | 2330-8516 |
| Abstract: | A frequently encountered security issue in writing tests is nonauthentic text submission: Test takers submit texts that are not their own but rather are copies of texts prepared by someone else. In this report, we propose AutoESD, a human-in-the-loop and automated system to detect nonauthentic texts for a large-scale writing tests, and report its performance on an operational data set. The AutoESD system utilizes multiple automated text similarity measures to identify suspect texts and provides an analytics-enhanced web application to help human experts review the identified texts. To evaluate the performance of AutoESD, we obtained its similarity measures on "TOEFL iBT®" test writing responses collected from multiple remote administrations and examined their distributions. The results were highly encouraging in that the distributional characteristics of AutoESD similarity measures were effective in identifying suspect texts and the measures could be computed quickly without affecting the operational score turnaround timeline. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1459542 |
| Database: | ERIC |
| Abstract: | A frequently encountered security issue in writing tests is nonauthentic text submission: Test takers submit texts that are not their own but rather are copies of texts prepared by someone else. In this report, we propose AutoESD, a human-in-the-loop and automated system to detect nonauthentic texts for a large-scale writing tests, and report its performance on an operational data set. The AutoESD system utilizes multiple automated text similarity measures to identify suspect texts and provides an analytics-enhanced web application to help human experts review the identified texts. To evaluate the performance of AutoESD, we obtained its similarity measures on "TOEFL iBT®" test writing responses collected from multiple remote administrations and examined their distributions. The results were highly encouraging in that the distributional characteristics of AutoESD similarity measures were effective in identifying suspect texts and the measures could be computed quickly without affecting the operational score turnaround timeline. |
|---|---|
| ISSN: | 2330-8516 |
