Evaluating Multi-Knowledge Component Interpretability of Deep Knowledge Tracing Models in Programming
Saved in:
| Title: | Evaluating Multi-Knowledge Component Interpretability of Deep Knowledge Tracing Models in Programming |
|---|---|
| Language: | English |
| Authors: | Yang Shi, Tiffany Barnes, Min Chi, Thomas Price |
| Source: | International Educational Data Mining Society. 2024. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 8 |
| Publication Date: | 2024 |
| Sponsoring Agency: | National Science Foundation (NSF) |
| Contract Number: | 2013502 2112635 |
| Document Type: | Speeches/Meeting Papers Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Algorithms, Artificial Intelligence, Models, Programming, Knowledge Level, College Students, Educational Technology |
| Geographic Terms: | Virginia |
| Abstract: | Knowledge tracing (KT) models have been a commonly used tool for tracking students' knowledge status. Recent advances in deep knowledge tracing (DKT) have demonstrated increased performance for knowledge tracing tasks in many datasets. However, interpreting students' states on single knowledge components (KCs) from DKT models could be challenging when tracking multiple KCs in one student submission attempt. In this paper, we evaluate the ability of DKT models to track students' knowledge using AUC scores. We further propose two possible solutions to improve multi-KC tracking performance: incorporating a layer that explicitly represents knowledge of each KC and incorporating code features into the DKT models. In experiments, we compare DKT to the proposed models and evaluate KC tracking performance in an introductory computer science course (CS1) dataset. Our results indicate that while all four models perform similarly on problem correctness predictions, incorporating KC layers may lead to limited improvement for KC tracking performance. Through a hand-labeled dataset with KC-specific correctness, our research shows that DKT has a limited performance when tracking multiple skills, especially when tracking incorrect submissions. We present potential ways, including designing a layer or incorporating student code information in the models, while the results show that only the layer yielded improvements. [For the complete proceedings, see ED675485.] |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | ED675549 |
| Database: | ERIC |
| Abstract: | Knowledge tracing (KT) models have been a commonly used tool for tracking students' knowledge status. Recent advances in deep knowledge tracing (DKT) have demonstrated increased performance for knowledge tracing tasks in many datasets. However, interpreting students' states on single knowledge components (KCs) from DKT models could be challenging when tracking multiple KCs in one student submission attempt. In this paper, we evaluate the ability of DKT models to track students' knowledge using AUC scores. We further propose two possible solutions to improve multi-KC tracking performance: incorporating a layer that explicitly represents knowledge of each KC and incorporating code features into the DKT models. In experiments, we compare DKT to the proposed models and evaluate KC tracking performance in an introductory computer science course (CS1) dataset. Our results indicate that while all four models perform similarly on problem correctness predictions, incorporating KC layers may lead to limited improvement for KC tracking performance. Through a hand-labeled dataset with KC-specific correctness, our research shows that DKT has a limited performance when tracking multiple skills, especially when tracking incorrect submissions. We present potential ways, including designing a layer or incorporating student code information in the models, while the results show that only the layer yielded improvements. [For the complete proceedings, see ED675485.] |
|---|
