Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach
Saved in:
| Title: | Using Large Language Models to Evaluate Ethical Persuasion Text: A Measurement Modeling Approach |
|---|---|
| Language: | English |
| Authors: | Matt Barney, Stefanie A. Wind, Vaishak Krishna |
| Source: | International Journal of Assessment Tools in Education. 2026 13(1):224-247. |
| Availability: | International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate |
| Peer Reviewed: | Y |
| Page Count: | 24 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Artificial Intelligence, Natural Language Processing, Ethics, Performance Based Assessment, Persuasive Discourse, Measurement Techniques, Psychometrics, Computer Assisted Testing |
| ISSN: | 2148-7456 |
| Abstract: | As AI becomes prevalent in all stages of assessment procedures, it is essential to develop procedures to ensure that its use supports ethical and psychometrically defensible measurement. In this study, we consider how measurement principles can be directly incorporated into an ethical reasoning performance assessment in which Large Language Models (LLMs) serve as raters. We demonstrate how a measurement approach can be used to obtain defensible measures of LLM-generated text related to ethics, prompts designed to elicit text-based ethical persuasion responses, and individual learners. We demonstrate how measurement quality indicators can serve as guardrails to help mitigate potential AI-related risks that can impact learners, such as hallucinations or errors. We describe a novel approach to designing, implementing, and evaluating performance assessments with AI, with the goal of enabling effective personalized learning experiences. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1495732 |
| Database: | ERIC |
| Abstract: | As AI becomes prevalent in all stages of assessment procedures, it is essential to develop procedures to ensure that its use supports ethical and psychometrically defensible measurement. In this study, we consider how measurement principles can be directly incorporated into an ethical reasoning performance assessment in which Large Language Models (LLMs) serve as raters. We demonstrate how a measurement approach can be used to obtain defensible measures of LLM-generated text related to ethics, prompts designed to elicit text-based ethical persuasion responses, and individual learners. We demonstrate how measurement quality indicators can serve as guardrails to help mitigate potential AI-related risks that can impact learners, such as hallucinations or errors. We describe a novel approach to designing, implementing, and evaluating performance assessments with AI, with the goal of enabling effective personalized learning experiences. |
|---|---|
| ISSN: | 2148-7456 |
