View in EDS

Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02

Saved in:
Bibliographic Details
Title: Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02
Language: English
Authors: Paul Deane, Duanli Yan, Katherine Castellano, Yigal Attali, Michelle Lamar, Mo Zhang, Ian Blood, James V. Bruno, Chen Li, Wenju Cui, Chunyi Ruan, Colleen Appel, Kofi James, Rodolfo Long, Farah Qureshi
Source: ETS Research Report Series. Dec 2024.
Availability: ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/
Peer Reviewed: Y
Page Count: 64
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Education
Secondary Education
Descriptors: Writing (Composition), Essays, Models, Elementary School Students, Secondary School Students, Validity, Reliability, Natural Language Processing, Artificial Intelligence, Writing Evaluation, Institutional Characteristics, Student Characteristics, Scoring, Automation
ISSN: 2330-8516
Abstract: This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS' digital writing service, Criterion®. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater® scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1459602
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1459602
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1459602
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Paul+Deane%22">Paul Deane</searchLink><br /><searchLink fieldCode="AR" term="%22Duanli+Yan%22">Duanli Yan</searchLink><br /><searchLink fieldCode="AR" term="%22Katherine+Castellano%22">Katherine Castellano</searchLink><br /><searchLink fieldCode="AR" term="%22Yigal+Attali%22">Yigal Attali</searchLink><br /><searchLink fieldCode="AR" term="%22Michelle+Lamar%22">Michelle Lamar</searchLink><br /><searchLink fieldCode="AR" term="%22Mo+Zhang%22">Mo Zhang</searchLink><br /><searchLink fieldCode="AR" term="%22Ian+Blood%22">Ian Blood</searchLink><br /><searchLink fieldCode="AR" term="%22James+V%2E+Bruno%22">James V. Bruno</searchLink><br /><searchLink fieldCode="AR" term="%22Chen+Li%22">Chen Li</searchLink><br /><searchLink fieldCode="AR" term="%22Wenju+Cui%22">Wenju Cui</searchLink><br /><searchLink fieldCode="AR" term="%22Chunyi+Ruan%22">Chunyi Ruan</searchLink><br /><searchLink fieldCode="AR" term="%22Colleen+Appel%22">Colleen Appel</searchLink><br /><searchLink fieldCode="AR" term="%22Kofi+James%22">Kofi James</searchLink><br /><searchLink fieldCode="AR" term="%22Rodolfo+Long%22">Rodolfo Long</searchLink><br /><searchLink fieldCode="AR" term="%22Farah+Qureshi%22">Farah Qureshi</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ETS+Research+Report+Series%22"><i>ETS Research Report Series</i></searchLink>. Dec 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: ETS. Rosedale Road, Mailstop 19R, Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 64
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Writing+%28Composition%29%22">Writing (Composition)</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Characteristics%22">Institutional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2330-8516
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper presents a multidimensional model of variation in writing quality, register, and genre in student essays, trained and tested via confirmatory factor analysis of 1.37 million essay submissions to ETS' digital writing service, Criterion®. The model was also validated with several other corpora, which indicated that it provides a reasonable fit for essay data from 4th grade to college. It includes an analysis of the test-retest reliability of each trait, longitudinal trends by trait, both within the school year and from 4th to 12th grades, and analysis of genre differences by trait, using prompts from the Criterion topic library aligned with the major modes of writing (exposition, argumentation, narrative, description, process, comparison and contrast, and cause and effect). It demonstrates that many of the traits are about as reliable as overall e-rater® scores, that the trait model can be used to build models somewhat more closely aligned with human scores than standard e-rater models, and that there are large, significant trait differences by genre, consistent with genre differences in trait patterns described in the larger literature. Some of the traits demonstrated clear trends between successive revisions. Students using Criterion appear to have consistently improved grammar, usage, and spelling after getting Criterion feedback and to have marginally improved essay organization. Many of the traits also demonstrated clear grade level trends. These features indicate that the trait model could be used to support more detailed scoring and reporting for writing assessments and learning tools.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1459602
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1459602
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 64
    Subjects:
      – SubjectFull: Writing (Composition)
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Validity
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Institutional Characteristics
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Automation
        Type: general
    Titles:
      – TitleFull: Modeling Writing Traits in a Formative Essay Corpus. Research Report. ETS RR-24-02
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Paul Deane
      – PersonEntity:
          Name:
            NameFull: Duanli Yan
      – PersonEntity:
          Name:
            NameFull: Katherine Castellano
      – PersonEntity:
          Name:
            NameFull: Yigal Attali
      – PersonEntity:
          Name:
            NameFull: Michelle Lamar
      – PersonEntity:
          Name:
            NameFull: Mo Zhang
      – PersonEntity:
          Name:
            NameFull: Ian Blood
      – PersonEntity:
          Name:
            NameFull: James V. Bruno
      – PersonEntity:
          Name:
            NameFull: Chen Li
      – PersonEntity:
          Name:
            NameFull: Wenju Cui
      – PersonEntity:
          Name:
            NameFull: Chunyi Ruan
      – PersonEntity:
          Name:
            NameFull: Colleen Appel
      – PersonEntity:
          Name:
            NameFull: Kofi James
      – PersonEntity:
          Name:
            NameFull: Rodolfo Long
      – PersonEntity:
          Name:
            NameFull: Farah Qureshi
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-electronic
              Value: 2330-8516
          Titles:
            – TitleFull: ETS Research Report Series
              Type: main
ResultId 1