View in EDS

Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach

Saved in:
Bibliographic Details
Title: Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach
Language: English
Authors: Matt Barney, Stefanie A. Wind, Vaishak Krishna
Source: International Journal of Assessment Tools in Education. 2026 13(1):224-247.
Availability: International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate
Peer Reviewed: Y
Page Count: 24
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Descriptors: Artificial Intelligence, Natural Language Processing, Ethics, Performance Based Assessment, Persuasive Discourse, Measurement Techniques, Psychometrics, Computer Assisted Testing
ISSN: 2148-7456
Abstract: As AI becomes prevalent in all stages of assessment procedures, it is essential to develop procedures to ensure that its use supports ethical and psychometrically defensible measurement. In this study, we consider how measurement principles can be directly incorporated into an ethical reasoning performance assessment in which Large Language Models (LLMs) serve as raters. We demonstrate how a measurement approach can be used to obtain defensible measures of LLM-generated text related to ethics, prompts designed to elicit text-based ethical persuasion responses, and individual learners. We demonstrate how measurement quality indicators can serve as guardrails to help mitigate potential AI-related risks that can impact learners, such as hallucinations or errors. We describe a novel approach to designing, implementing, and evaluating performance assessments with AI, with the goal of enabling effective personalized learning experiences.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1495732
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1495732
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1495732
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Matt+Barney%22">Matt Barney</searchLink><br /><searchLink fieldCode="AR" term="%22Stefanie+A%2E+Wind%22">Stefanie A. Wind</searchLink><br /><searchLink fieldCode="AR" term="%22Vaishak+Krishna%22">Vaishak Krishna</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Assessment+Tools+in+Education%22"><i>International Journal of Assessment Tools in Education</i></searchLink>. 2026 13(1):224-247.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Ethics%22">Ethics</searchLink><br /><searchLink fieldCode="DE" term="%22Performance+Based+Assessment%22">Performance Based Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Persuasive+Discourse%22">Persuasive Discourse</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement+Techniques%22">Measurement Techniques</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2148-7456
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As AI becomes prevalent in all stages of assessment procedures, it is essential to develop procedures to ensure that its use supports ethical and psychometrically defensible measurement. In this study, we consider how measurement principles can be directly incorporated into an ethical reasoning performance assessment in which Large Language Models (LLMs) serve as raters. We demonstrate how a measurement approach can be used to obtain defensible measures of LLM-generated text related to ethics, prompts designed to elicit text-based ethical persuasion responses, and individual learners. We demonstrate how measurement quality indicators can serve as guardrails to help mitigate potential AI-related risks that can impact learners, such as hallucinations or errors. We describe a novel approach to designing, implementing, and evaluating performance assessments with AI, with the goal of enabling effective personalized learning experiences.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1495732
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1495732
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 224
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Ethics
        Type: general
      – SubjectFull: Performance Based Assessment
        Type: general
      – SubjectFull: Persuasive Discourse
        Type: general
      – SubjectFull: Measurement Techniques
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
    Titles:
      – TitleFull: Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Matt Barney
      – PersonEntity:
          Name:
            NameFull: Stefanie A. Wind
      – PersonEntity:
          Name:
            NameFull: Vaishak Krishna
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2148-7456
          Numbering:
            – Type: volume
              Value: 13
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: International Journal of Assessment Tools in Education
              Type: main
ResultId 1