Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool.
Saved in:
| Title: | Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool. |
|---|---|
| Authors: | Potter, Andrew1 ahpotter@asu.edu, Patne, Nishad A.1 npatne@asu.edu, Ahmed, Ishrat1 Ishrat.Ahmed.1@asu.edu, Islam, Rezwana1 rislam11@asu.edu, Allen, Laura A.2 lallen@umn.edu, McNamara, Danielle S.1 dsmcnamara@asu.edu, Serhan, Zeinab1 zserhan@asu.edu, Öncel, Püren2 oncel001@umn.edu, Arner, Tracy1 tarner@asu.edu, Roscoe, Rod D.1 rod.roscoe@asu.edu, Crossley, Scott A.1 scott.crossley@vanderbilt.edu |
| Source: | Journal of Educational Data Mining. 2026, Vol. 18 Issue 1, p113-155. 43p. |
| Subject Terms: | *Educational technology, *Formative evaluation, *Qualitative research, Participatory design, Language models, Natural language processing, Human-computer interaction, Data analytics |
| Abstract: | This study introduces a hybrid human-AI workflow to qualitative data analysis within the participatory design of the Writing Analytics Toolkit (WAT), an open-source platform that provides formative feedback on student writing using natural language processing. The toolkit includes a classroom-facing implementation (WAT Classroom; WAT-C), designed to support instruction, and a researcher-facing implementation (WAT Researcher; WAT-R), designed to support analytic and validation workflows. Nine experienced college writing instructors (with 97 cumulative years of teaching) participated in focus group sessions to evaluate an early prototype of the classroom version of WAT (WAT-C), offering formative input on usability, instructional alignment, and feedback clarity. To analyze the resulting qualitative data, we employed a novel AI-augmented analytic process: GPT-4o, integrated within a secure, retrieval-augmented system, to generate inductive codes and preliminary themes from transcripts. These AI-generated outputs were iteratively reviewed, critiqued, refined, and synthesized by researchers, supporting both analytical scalability and interpretive rigor. This human-AI partnership enabled efficient thematic exploration while preserving methodological transparency and researcher judgment. Findings from both qualitative and complementary survey data identified four key design priorities: (1) clearer, more concise feedback, (2) increased instructor customization, (3) reduced administrative burden, and (4) a simplified user interface. These insights directly informed subsequent revisions to WAT-C, including a redesigned feedback interface, customizable metric targets, learning management system integration, and a more intuitive layout. This work illustrates how large language models (LLMs) can support inductive qualitative analysis within participatory design workflows. Moreover, results demonstrate how this workflow can inform iterative educational technology development. Implications include the need to ensure ethical oversight, researcher-led interpretation, and alignment with instructional priorities when incorporating AI into the design of educational technologies. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Educational Data Mining is the property of International Educational Data Mining Society and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 192782013 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Potter%2C+Andrew%22">Potter, Andrew</searchLink><relatesTo>1</relatesTo><i> ahpotter@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Patne%2C+Nishad+A%2E%22">Patne, Nishad A.</searchLink><relatesTo>1</relatesTo><i> npatne@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Ahmed%2C+Ishrat%22">Ahmed, Ishrat</searchLink><relatesTo>1</relatesTo><i> Ishrat.Ahmed.1@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Islam%2C+Rezwana%22">Islam, Rezwana</searchLink><relatesTo>1</relatesTo><i> rislam11@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Allen%2C+Laura+A%2E%22">Allen, Laura A.</searchLink><relatesTo>2</relatesTo><i> lallen@umn.edu</i><br /><searchLink fieldCode="AR" term="%22McNamara%2C+Danielle+S%2E%22">McNamara, Danielle S.</searchLink><relatesTo>1</relatesTo><i> dsmcnamara@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Serhan%2C+Zeinab%22">Serhan, Zeinab</searchLink><relatesTo>1</relatesTo><i> zserhan@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Öncel%2C+Püren%22">Öncel, Püren</searchLink><relatesTo>2</relatesTo><i> oncel001@umn.edu</i><br /><searchLink fieldCode="AR" term="%22Arner%2C+Tracy%22">Arner, Tracy</searchLink><relatesTo>1</relatesTo><i> tarner@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Roscoe%2C+Rod+D%2E%22">Roscoe, Rod D.</searchLink><relatesTo>1</relatesTo><i> rod.roscoe@asu.edu</i><br /><searchLink fieldCode="AR" term="%22Crossley%2C+Scott+A%2E%22">Crossley, Scott A.</searchLink><relatesTo>1</relatesTo><i> scott.crossley@vanderbilt.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Journal+of+Educational+Data+Mining%22">Journal of Educational Data Mining</searchLink>. 2026, Vol. 18 Issue 1, p113-155. 43p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Educational+technology%22">Educational technology</searchLink><br />*<searchLink fieldCode="DE" term="%22Formative+evaluation%22">Formative evaluation</searchLink><br />*<searchLink fieldCode="DE" term="%22Qualitative+research%22">Qualitative research</searchLink><br /><searchLink fieldCode="DE" term="%22Participatory+design%22">Participatory design</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+language+processing%22">Natural language processing</searchLink><br /><searchLink fieldCode="DE" term="%22Human-computer+interaction%22">Human-computer interaction</searchLink><br /><searchLink fieldCode="DE" term="%22Data+analytics%22">Data analytics</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This study introduces a hybrid human-AI workflow to qualitative data analysis within the participatory design of the Writing Analytics Toolkit (WAT), an open-source platform that provides formative feedback on student writing using natural language processing. The toolkit includes a classroom-facing implementation (WAT Classroom; WAT-C), designed to support instruction, and a researcher-facing implementation (WAT Researcher; WAT-R), designed to support analytic and validation workflows. Nine experienced college writing instructors (with 97 cumulative years of teaching) participated in focus group sessions to evaluate an early prototype of the classroom version of WAT (WAT-C), offering formative input on usability, instructional alignment, and feedback clarity. To analyze the resulting qualitative data, we employed a novel AI-augmented analytic process: GPT-4o, integrated within a secure, retrieval-augmented system, to generate inductive codes and preliminary themes from transcripts. These AI-generated outputs were iteratively reviewed, critiqued, refined, and synthesized by researchers, supporting both analytical scalability and interpretive rigor. This human-AI partnership enabled efficient thematic exploration while preserving methodological transparency and researcher judgment. Findings from both qualitative and complementary survey data identified four key design priorities: (1) clearer, more concise feedback, (2) increased instructor customization, (3) reduced administrative burden, and (4) a simplified user interface. These insights directly informed subsequent revisions to WAT-C, including a redesigned feedback interface, customizable metric targets, learning management system integration, and a more intuitive layout. This work illustrates how large language models (LLMs) can support inductive qualitative analysis within participatory design workflows. Moreover, results demonstrate how this workflow can inform iterative educational technology development. Implications include the need to ensure ethical oversight, researcher-led interpretation, and alignment with instructional priorities when incorporating AI into the design of educational technologies. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Journal of Educational Data Mining is the property of International Educational Data Mining Society and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=192782013 |
| RecordInfo | BibRecord: BibEntity: Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 43 StartPage: 113 Subjects: – SubjectFull: Educational technology Type: general – SubjectFull: Formative evaluation Type: general – SubjectFull: Qualitative research Type: general – SubjectFull: Participatory design Type: general – SubjectFull: Language models Type: general – SubjectFull: Natural language processing Type: general – SubjectFull: Human-computer interaction Type: general – SubjectFull: Data analytics Type: general Titles: – TitleFull: Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Potter, Andrew – PersonEntity: Name: NameFull: Patne, Nishad A. – PersonEntity: Name: NameFull: Ahmed, Ishrat – PersonEntity: Name: NameFull: Islam, Rezwana – PersonEntity: Name: NameFull: Allen, Laura A. – PersonEntity: Name: NameFull: McNamara, Danielle S. – PersonEntity: Name: NameFull: Serhan, Zeinab – PersonEntity: Name: NameFull: Öncel, Püren – PersonEntity: Name: NameFull: Arner, Tracy – PersonEntity: Name: NameFull: Roscoe, Rod D. – PersonEntity: Name: NameFull: Crossley, Scott A. IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Text: 2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 21572100 Numbering: – Type: volume Value: 18 – Type: issue Value: 1 Titles: – TitleFull: Journal of Educational Data Mining Type: main |
| ResultId | 1 |
