Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory
Saved in:
| Title: | Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory |
|---|---|
| Language: | English |
| Authors: | Hao Fang, Aiwei Mu, Guosheng Xing, Xingyu Chen, Seng Yue Wong |
| Source: | International Review of Research in Open and Distributed Learning. 2026 27(1):155-179. |
| Availability: | Athabasca University Press. 1200, 10011-109 Street, Edmonton, AB T5J 3S8, Canada. Tel: 780-497-3412; Fax: 780-421-3298; e-mail: irrodl@athabascau.ca; Web site: http://www.irrodl.org |
| Peer Reviewed: | Y |
| Page Count: | 25 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Health Sciences, Medical Education, Allied Health Occupations Education, Learning Analytics, Artificial Intelligence, Educational Theories, Discourse Analysis, Networks |
| ISSN: | 1492-3831 |
| Abstract: | Online learning environments tend not to provide the social and pedagogical cues of physical classrooms, so evaluating student engagement and emotional states in real time becomes challenging. Current methods depend mainly upon facial expression recognition or textual sentiment analysis, constraining the depth and accuracy of behavioral interpretation. This research suggests a multimodal learning analytics framework that combines visual and textual data to infer learner emotions and engagement for improving the interpretability, responsiveness, and pedagogical value of learning analytics systems in digital education. Two datasets were created: (a) a facial expression dataset of 10,000 grayscale images annotated over five emotion categories and (b) an engagement dataset of 4,000 images annotated according to behavioral indicators. Concurrently, 1,667 learner feedback responses from massive open online courses were prepared for sentiment analysis. Convolutional neural networks (CNNs) were used for emotion and engagement classification, and a fine-tuned BERT (bidirectional encoder representations from transformers) model for sentiment analysis. A rule-based integration engine combined outputs to create multidimensional behavioural typologies. The CNN models reached >92% validation accuracy for both emotion detection and engagement detection tasks, whereas the BERT sentiment classifier achieved F1 = 0.87 and 88.1% accuracy. The multimodal integration procedure identified four unique learner behavior typologies (e.g., students who were cognitively engaged but visually disengaged). The framework offers an accurate, interpretable, and scalable real-time learning analytics solution. Compared with previous methods, it overcomes significant limitations and offers a useful resource for facilitating adaptive, data-based instruction interventions, especially in online and health science education. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1501202 |
| Database: | ERIC |
Be the first to leave a comment!
