View in EDS

Statistical and Qualitative Analysis of ChatGPT and Human Raters in Preservice Teachers' Writing Assessment

Saved in:
Bibliographic Details
Title: Statistical and Qualitative Analysis of ChatGPT and Human Raters in Preservice Teachers' Writing Assessment
Language: English
Authors: Bahadir Gülden (ORCID 0000-0003-1917-8813), Huzeyfe Bilge (ORCID 0000-0001-7664-488X), Pinar Kanik Uysal (ORCID 0000-0003-1208-9535)
Source: International Journal of Assessment Tools in Education. 2026 13(1):248-269.
Availability: International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate
Peer Reviewed: Y
Page Count: 22
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Preservice Teachers, Writing Evaluation, Artificial Intelligence, Evaluation Methods, Feedback (Response), Technology Uses in Education, Foreign Countries, Undergraduate Students, Writing Skills, Reliability, Scores, Barriers, Expertise, Turkish, Language Teachers, Scoring, Writing Assignments
Geographic Terms: Turkey
ISSN: 2148-7456
Abstract: Teachers spend a significant amount of time providing feedback. This study compared expert and ChatGPT assessments and feedback on written texts to determine the suitability of AI for writing skill assessments that are time-consuming to assess and provide feedback. Three experts and ChatGPT graded 14 Turkish undergraduate students' assignments using rubric that included content, language use, vocabulary, organization, and mechanics, and justified their decisions. The study involved document review and triangulation, a qualitative design. In addition, an intraclass correlation coefficient was used to assess the consistency of the ChatGPT and the experts' scores. All feedback was qualitatively analyzed to identify the strengths and weaknesses of the experts and their similarities with ChatGPT. Experts and ChatGPT had moderate to weak consistency in the writing subscales, while good reliability was found in the total score. Experts excelled in 'explanatory feedback', 'interpretation' and 'experience', while ChatGPT excelled in 'automation and continuity' and 'data processing capacity'. Experts' weaknesses included 'limited time and energy' and 'comparison bias', while ChatGPT's weaknesses were 'ambiguous expressions' and 'repetition'. The study also found that experts and ChatGPT preferred to provide constructive and supportive feedback.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1495754
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1495754
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1495754
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Statistical and Qualitative Analysis of ChatGPT and Human Raters in Preservice Teachers' Writing Assessment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Bahadir+Gülden%22">Bahadir Gülden</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1917-8813">0000-0003-1917-8813</externalLink>)<br /><searchLink fieldCode="AR" term="%22Huzeyfe+Bilge%22">Huzeyfe Bilge</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7664-488X">0000-0001-7664-488X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Pinar+Kanik+Uysal%22">Pinar Kanik Uysal</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1208-9535">0000-0003-1208-9535</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Assessment+Tools+in+Education%22"><i>International Journal of Assessment Tools in Education</i></searchLink>. 2026 13(1):248-269.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Journal of Assessment Tools in Education. Pamukkale University, Faculty of Education, Kinikli Campus, Denizli 20070, Turkey. e-mail: ijate.editor@gmail.com; Web site: https://dergipark.org.tr/en/pub/ijate
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Preservice+Teachers%22">Preservice Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Skills%22">Writing Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Expertise%22">Expertise</searchLink><br /><searchLink fieldCode="DE" term="%22Turkish%22">Turkish</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Teachers%22">Language Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Assignments%22">Writing Assignments</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Turkey%22">Turkey</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2148-7456
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Teachers spend a significant amount of time providing feedback. This study compared expert and ChatGPT assessments and feedback on written texts to determine the suitability of AI for writing skill assessments that are time-consuming to assess and provide feedback. Three experts and ChatGPT graded 14 Turkish undergraduate students' assignments using rubric that included content, language use, vocabulary, organization, and mechanics, and justified their decisions. The study involved document review and triangulation, a qualitative design. In addition, an intraclass correlation coefficient was used to assess the consistency of the ChatGPT and the experts' scores. All feedback was qualitatively analyzed to identify the strengths and weaknesses of the experts and their similarities with ChatGPT. Experts and ChatGPT had moderate to weak consistency in the writing subscales, while good reliability was found in the total score. Experts excelled in 'explanatory feedback', 'interpretation' and 'experience', while ChatGPT excelled in 'automation and continuity' and 'data processing capacity'. Experts' weaknesses included 'limited time and energy' and 'comparison bias', while ChatGPT's weaknesses were 'ambiguous expressions' and 'repetition'. The study also found that experts and ChatGPT preferred to provide constructive and supportive feedback.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1495754
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1495754
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 248
    Subjects:
      – SubjectFull: Preservice Teachers
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Writing Skills
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Barriers
        Type: general
      – SubjectFull: Expertise
        Type: general
      – SubjectFull: Turkish
        Type: general
      – SubjectFull: Language Teachers
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Writing Assignments
        Type: general
      – SubjectFull: Turkey
        Type: general
    Titles:
      – TitleFull: Statistical and Qualitative Analysis of ChatGPT and Human Raters in Preservice Teachers' Writing Assessment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Bahadir Gülden
      – PersonEntity:
          Name:
            NameFull: Huzeyfe Bilge
      – PersonEntity:
          Name:
            NameFull: Pinar Kanik Uysal
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2148-7456
          Numbering:
            – Type: volume
              Value: 13
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: International Journal of Assessment Tools in Education
              Type: main
ResultId 1