Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool.
Saved in:
| Title: | Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool. |
|---|---|
| Authors: | Potter, Andrew1 ahpotter@asu.edu, Patne, Nishad A.1 npatne@asu.edu, Ahmed, Ishrat1 Ishrat.Ahmed.1@asu.edu, Islam, Rezwana1 rislam11@asu.edu, Allen, Laura A.2 lallen@umn.edu, McNamara, Danielle S.1 dsmcnamara@asu.edu, Serhan, Zeinab1 zserhan@asu.edu, Öncel, Püren2 oncel001@umn.edu, Arner, Tracy1 tarner@asu.edu, Roscoe, Rod D.1 rod.roscoe@asu.edu, Crossley, Scott A.1 scott.crossley@vanderbilt.edu |
| Source: | Journal of Educational Data Mining. 2026, Vol. 18 Issue 1, p113-155. 43p. |
| Subject Terms: | *Educational technology, *Formative evaluation, *Qualitative research, Participatory design, Language models, Natural language processing, Human-computer interaction, Data analytics |
| Abstract: | This study introduces a hybrid human-AI workflow to qualitative data analysis within the participatory design of the Writing Analytics Toolkit (WAT), an open-source platform that provides formative feedback on student writing using natural language processing. The toolkit includes a classroom-facing implementation (WAT Classroom; WAT-C), designed to support instruction, and a researcher-facing implementation (WAT Researcher; WAT-R), designed to support analytic and validation workflows. Nine experienced college writing instructors (with 97 cumulative years of teaching) participated in focus group sessions to evaluate an early prototype of the classroom version of WAT (WAT-C), offering formative input on usability, instructional alignment, and feedback clarity. To analyze the resulting qualitative data, we employed a novel AI-augmented analytic process: GPT-4o, integrated within a secure, retrieval-augmented system, to generate inductive codes and preliminary themes from transcripts. These AI-generated outputs were iteratively reviewed, critiqued, refined, and synthesized by researchers, supporting both analytical scalability and interpretive rigor. This human-AI partnership enabled efficient thematic exploration while preserving methodological transparency and researcher judgment. Findings from both qualitative and complementary survey data identified four key design priorities: (1) clearer, more concise feedback, (2) increased instructor customization, (3) reduced administrative burden, and (4) a simplified user interface. These insights directly informed subsequent revisions to WAT-C, including a redesigned feedback interface, customizable metric targets, learning management system integration, and a more intuitive layout. This work illustrates how large language models (LLMs) can support inductive qualitative analysis within participatory design workflows. Moreover, results demonstrate how this workflow can inform iterative educational technology development. Implications include the need to ensure ethical oversight, researcher-led interpretation, and alignment with instructional priorities when incorporating AI into the design of educational technologies. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of Educational Data Mining is the property of International Educational Data Mining Society and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
Be the first to leave a comment!
