Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory
Saved in:
| Title: | Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory |
|---|---|
| Language: | English |
| Authors: | Hao Fang, Aiwei Mu, Guosheng Xing, Xingyu Chen, Seng Yue Wong |
| Source: | International Review of Research in Open and Distributed Learning. 2026 27(1):155-179. |
| Availability: | Athabasca University Press. 1200, 10011-109 Street, Edmonton, AB T5J 3S8, Canada. Tel: 780-497-3412; Fax: 780-421-3298; e-mail: irrodl@athabascau.ca; Web site: http://www.irrodl.org |
| Peer Reviewed: | Y |
| Page Count: | 25 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Health Sciences, Medical Education, Allied Health Occupations Education, Learning Analytics, Artificial Intelligence, Educational Theories, Discourse Analysis, Networks |
| ISSN: | 1492-3831 |
| Abstract: | Online learning environments tend not to provide the social and pedagogical cues of physical classrooms, so evaluating student engagement and emotional states in real time becomes challenging. Current methods depend mainly upon facial expression recognition or textual sentiment analysis, constraining the depth and accuracy of behavioral interpretation. This research suggests a multimodal learning analytics framework that combines visual and textual data to infer learner emotions and engagement for improving the interpretability, responsiveness, and pedagogical value of learning analytics systems in digital education. Two datasets were created: (a) a facial expression dataset of 10,000 grayscale images annotated over five emotion categories and (b) an engagement dataset of 4,000 images annotated according to behavioral indicators. Concurrently, 1,667 learner feedback responses from massive open online courses were prepared for sentiment analysis. Convolutional neural networks (CNNs) were used for emotion and engagement classification, and a fine-tuned BERT (bidirectional encoder representations from transformers) model for sentiment analysis. A rule-based integration engine combined outputs to create multidimensional behavioural typologies. The CNN models reached >92% validation accuracy for both emotion detection and engagement detection tasks, whereas the BERT sentiment classifier achieved F1 = 0.87 and 88.1% accuracy. The multimodal integration procedure identified four unique learner behavior typologies (e.g., students who were cognitively engaged but visually disengaged). The framework offers an accurate, interpretable, and scalable real-time learning analytics solution. Compared with previous methods, it overcomes significant limitations and offers a useful resource for facilitating adaptive, data-based instruction interventions, especially in online and health science education. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1501202 |
| Database: | ERIC |
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGmhkBiWzCKvU4jy08zNJnKAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDDhud-9fOHFxPieRmwIBEICBm_wqSUml09Q8qVcZYDzHxLOuRABMfMhcjt4dsYjqLs-SJ3fH10zZnxLYfCNVr1usjJcnB9qT2PXr0RrnAHcVzSavqB46xfUGGXkCP6KhVtUUIMiRe4qYM13ua6zYbG5jnEySuok6I3B_zbKTfab3hJYahai5t3ghNy2ij9pz35othN7j4dMso2XVksvUwY4c6qlA7Zj3qz6MfBxw Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1501202 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1501202 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Hao+Fang%22">Hao Fang</searchLink><br /><searchLink fieldCode="AR" term="%22Aiwei+Mu%22">Aiwei Mu</searchLink><br /><searchLink fieldCode="AR" term="%22Guosheng+Xing%22">Guosheng Xing</searchLink><br /><searchLink fieldCode="AR" term="%22Xingyu+Chen%22">Xingyu Chen</searchLink><br /><searchLink fieldCode="AR" term="%22Seng+Yue+Wong%22">Seng Yue Wong</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22International+Review+of+Research+in+Open+and+Distributed+Learning%22"><i>International Review of Research in Open and Distributed Learning</i></searchLink>. 2026 27(1):155-179. – Name: Avail Label: Availability Group: Avail Data: Athabasca University Press. 1200, 10011-109 Street, Edmonton, AB T5J 3S8, Canada. Tel: 780-497-3412; Fax: 780-421-3298; e-mail: irrodl@athabascau.ca; Web site: http://www.irrodl.org – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 25 – Name: DatePubCY Label: Publication Date Group: Date Data: 2026 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Health+Sciences%22">Health Sciences</searchLink><br /><searchLink fieldCode="DE" term="%22Medical+Education%22">Medical Education</searchLink><br /><searchLink fieldCode="DE" term="%22Allied+Health+Occupations+Education%22">Allied Health Occupations Education</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Theories%22">Educational Theories</searchLink><br /><searchLink fieldCode="DE" term="%22Discourse+Analysis%22">Discourse Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Networks%22">Networks</searchLink> – Name: ISSN Label: ISSN Group: ISSN Data: 1492-3831 – Name: Abstract Label: Abstract Group: Ab Data: Online learning environments tend not to provide the social and pedagogical cues of physical classrooms, so evaluating student engagement and emotional states in real time becomes challenging. Current methods depend mainly upon facial expression recognition or textual sentiment analysis, constraining the depth and accuracy of behavioral interpretation. This research suggests a multimodal learning analytics framework that combines visual and textual data to infer learner emotions and engagement for improving the interpretability, responsiveness, and pedagogical value of learning analytics systems in digital education. Two datasets were created: (a) a facial expression dataset of 10,000 grayscale images annotated over five emotion categories and (b) an engagement dataset of 4,000 images annotated according to behavioral indicators. Concurrently, 1,667 learner feedback responses from massive open online courses were prepared for sentiment analysis. Convolutional neural networks (CNNs) were used for emotion and engagement classification, and a fine-tuned BERT (bidirectional encoder representations from transformers) model for sentiment analysis. A rule-based integration engine combined outputs to create multidimensional behavioural typologies. The CNN models reached >92% validation accuracy for both emotion detection and engagement detection tasks, whereas the BERT sentiment classifier achieved F1 = 0.87 and 88.1% accuracy. The multimodal integration procedure identified four unique learner behavior typologies (e.g., students who were cognitively engaged but visually disengaged). The framework offers an accurate, interpretable, and scalable real-time learning analytics solution. Compared with previous methods, it overcomes significant limitations and offers a useful resource for facilitating adaptive, data-based instruction interventions, especially in online and health science education. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2026 – Name: AN Label: Accession Number Group: ID Data: EJ1501202 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1501202 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 25 StartPage: 155 Subjects: – SubjectFull: Health Sciences Type: general – SubjectFull: Medical Education Type: general – SubjectFull: Allied Health Occupations Education Type: general – SubjectFull: Learning Analytics Type: general – SubjectFull: Artificial Intelligence Type: general – SubjectFull: Educational Theories Type: general – SubjectFull: Discourse Analysis Type: general – SubjectFull: Networks Type: general Titles: – TitleFull: Multimodal Engagement and Sentiment Analytics in Health Science Education: A Learning Analytics Framework Integrating AI and Pedagogical Theory Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Hao Fang – PersonEntity: Name: NameFull: Aiwei Mu – PersonEntity: Name: NameFull: Guosheng Xing – PersonEntity: Name: NameFull: Xingyu Chen – PersonEntity: Name: NameFull: Seng Yue Wong IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2026 Identifiers: – Type: issn-electronic Value: 1492-3831 Numbering: – Type: volume Value: 27 – Type: issue Value: 1 Titles: – TitleFull: International Review of Research in Open and Distributed Learning Type: main |
| ResultId | 1 |
