Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time
Saved in:
| Title: | Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time |
|---|---|
| Language: | English |
| Authors: | Xander Beberman, Sarah Novicoff, Ana Trindade Ribeiro, Carly Robinson, Susanna Loeb, Society for Research on Educational Effectiveness (SREE) |
| Source: | Society for Research on Educational Effectiveness. 2025. |
| Availability: | Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/ |
| Peer Reviewed: | Y |
| Publication Date: | 2025 |
| Document Type: | Reports - Research |
| Education Level: | Early Childhood Education Elementary Education Kindergarten Primary Education Grade 1 Grade 2 |
| Descriptors: | Electronic Learning, Tutoring, Student Characteristics, Institutional Characteristics, Public Schools, Kindergarten, Grade 1, Grade 2, Program Effectiveness, Emergent Literacy, Literacy Education, Time Factors (Learning), Classification |
| Geographic Terms: | Texas |
| Abstract: | Background/Context: Virtual tutoring has grown significantly as a personalized learning tool, especially in response to the academic disruptions caused by the COVID-19 pandemic. While out-of-school virtual tutoring has demonstrated moderate to large positive effects on student learning (0.20-0.40 SDs; Carlana and La Ferrara, 2024; Deacon and Chojnacki, 2023; Gortazar et al., 2024; Roschelle et al., 2020), in-school virtual tutoring shows significantly smaller impacts (0.01-0.08 SDs; Kraft et al., 2022; Ready et al., 2024; Robinson et al., 2024). Although many factors may explain this discrepancy, we hypothesize that in-school virtual tutoring can be more susceptible to disruptions similar to classroom instruction (Kraft and Monti-Nussbaum, 2021), affecting the overall instruction time. Prior research has shown that instructional time is a critical determinant of learning outcomes (Yesil Dagli (2019); Gromada and Shewbridge (2016); Holland et al. (2015); Kraft and Novicoff (2024); Patall et al. (2010), but due to difficulties in data collection, there has been limited empirical work quantifying how disruptions can reduce instruction time and affect learning. Purpose/Objective/Research Question: The primary objective of this study is to investigate how much instructional time is lost due to disruptions during in-school virtual tutoring sessions and to analyze the relationship between these disruptions and student and school characteristics. Specifically, the study aims to identify different types of disruptions and quantify the time they consume during tutoring, using a text classification model trained on tutor transcripts annotated by humans. The underlying hypothesis is that a substantial portion of virtual tutoring time in schools is spent addressing disruptions, which contributes to the diminished effectiveness observed in such settings. Setting: Our study uses data collected for the RCT conducted in Robinson et al., 2024. The virtual tutoring intervention occurred across 12 campuses within the same public school system in Texas. Tutoring sessions were administered virtually during the school day, using a platform that connected students with remote tutors nationwide to deliver early literacy instruction to participating students assigned to one of the treatment groups. Population/Participants/Subjects: Data was collected from 1,357 students in kindergarten through second grade and 192 tutors who participated in an RCT to evaluate the effectiveness of in-school virtual tutoring (Robinson et al., 2024). Over 26,000 tutoring sessions were analyzed, with 19,448 individual tutoring sessions (1:1 student-tutor ratio) and 7,150 paired tutoring sessions (2:1 tutor-student ratio). Students were diverse in terms of demographics and learning needs, including those receiving free or reduced lunch, English learners, and students with disabilities. Tutoring was focused on early literacy development, and the main outcomes for the RCT were achievement scores on DIBELS and MAP Reading. Intervention/Program/Practice: The tutoring intervention involved early literacy virtual instruction delivered via a video conferencing platform. Tutors and students engaged in individualized or paired tutoring sessions using structured literacy materials. Sessions were recorded, and transcript and audio data were collected. A typical session lasted around 20-30 minutes, and the program implementation occurred from November 2022 to May 2023. Research Design: We use natural language processing (NLP) to classify and analyze types of disruptions during virtual tutoring. We developed a custom multi-class text classifier trained on tutor utterances labeled by human annotators. Sessions were selected from a two-week window in April 2023, and regression models were used to assess the relationship between session characteristics related to the students and their school and the prevalence of different types of disruptions. Data Collection and Analysis: We transcribed all 26,714 session recordings from the RCT's tutoring sessions for this study. We created a human-annotated set of 2,501 annotations of 1,488 unique utterances from 255 sessions to train our classifier. We achieved an inter-rater reliability of 0.71 using Krippendorff's alpha across 19 human annotators. Disruption categories included tech problems, student disruptions, background noise, and issues involving additional students or substitute tutors. A RoBERTa-based classifier was fine-tuned using 80% of the labeled data, with oversampling of less frequent categories to address class imbalance. Accuracy reached 74% overall (80% for disruption vs. non-disruption). We estimated disruption durations based on the classifier labeling of each utterance and utterance timestamps. Our analysis used session-level aggregation of disruption time, examining how disruption time varied by student composition (e.g., 1:1 vs. 2:1 tutoring), demographics, and school for the students in the session. We estimate the following regression, where i is the session, y_i is the percentage of session time spent on one of the categories from our disruption classifier, 2:1 is a dummy for students assigned to paired tutoring (instead of individual), X represents a vector of student demographics and other characteristics, [delta] represents school (s) fixed effects, and represents grade fixed effects. y[subscript i] = [beta subscript 0] + [beta subscript 1] (2 : 1t)[subscript i] + X[subscript i][gamma] + [delta subscript s(i)] + [lambda subscript g(i)] + [epsilon subscript i]. Findings/Results: We found that approximately 25% of tutoring time was spent managing disruptions. Of this, 11.8% of time was attributed to student behavior, and 9% to technology issues. Pair tutoring (2:1) sessions were associated with significantly less usable instructional time--nearly 9 percentage points lower than individual (1:1) sessions. Sessions with at least one female student had more usable instructional time (+3 p.p.), suggesting possible behavioral differences by gender. Tech issues were less prevalent among second-grade students, likely due to their greater experience with technology. We also found that tech issues and background noise disruptions correlated more strongly with specific schools, suggesting variations in the implementation environment affected the amount of instruction time available to students through virtual tutoring. Conclusions: Disruptions are a significant drain on instructional time in in-school virtual tutoring settings, with both student behavior and environmental factors contributing. Notably, disruptions varied systematically by session type, student demographics, and school, suggesting that targeted program design and implementation changes--such as quieter tutoring locations, better tech support, and smaller group sizes--could mitigate some of these effects. As we continue to work on this study, our next steps include extending our analysis to the full set of tutoring sessions and investigating whether lower levels of school environment disruptions, such as tech issues and background noise, also correlate with larger learning gains for students by estimating differential impact on student performance on the end-of-year DIBELS assessment. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Access URL: | https://www.sree.org/2025-conference |
| Accession Number: | ED677776 |
| Database: | ERIC |
| Abstract: | Background/Context: Virtual tutoring has grown significantly as a personalized learning tool, especially in response to the academic disruptions caused by the COVID-19 pandemic. While out-of-school virtual tutoring has demonstrated moderate to large positive effects on student learning (0.20-0.40 SDs; Carlana and La Ferrara, 2024; Deacon and Chojnacki, 2023; Gortazar et al., 2024; Roschelle et al., 2020), in-school virtual tutoring shows significantly smaller impacts (0.01-0.08 SDs; Kraft et al., 2022; Ready et al., 2024; Robinson et al., 2024). Although many factors may explain this discrepancy, we hypothesize that in-school virtual tutoring can be more susceptible to disruptions similar to classroom instruction (Kraft and Monti-Nussbaum, 2021), affecting the overall instruction time. Prior research has shown that instructional time is a critical determinant of learning outcomes (Yesil Dagli (2019); Gromada and Shewbridge (2016); Holland et al. (2015); Kraft and Novicoff (2024); Patall et al. (2010), but due to difficulties in data collection, there has been limited empirical work quantifying how disruptions can reduce instruction time and affect learning. Purpose/Objective/Research Question: The primary objective of this study is to investigate how much instructional time is lost due to disruptions during in-school virtual tutoring sessions and to analyze the relationship between these disruptions and student and school characteristics. Specifically, the study aims to identify different types of disruptions and quantify the time they consume during tutoring, using a text classification model trained on tutor transcripts annotated by humans. The underlying hypothesis is that a substantial portion of virtual tutoring time in schools is spent addressing disruptions, which contributes to the diminished effectiveness observed in such settings. Setting: Our study uses data collected for the RCT conducted in Robinson et al., 2024. The virtual tutoring intervention occurred across 12 campuses within the same public school system in Texas. Tutoring sessions were administered virtually during the school day, using a platform that connected students with remote tutors nationwide to deliver early literacy instruction to participating students assigned to one of the treatment groups. Population/Participants/Subjects: Data was collected from 1,357 students in kindergarten through second grade and 192 tutors who participated in an RCT to evaluate the effectiveness of in-school virtual tutoring (Robinson et al., 2024). Over 26,000 tutoring sessions were analyzed, with 19,448 individual tutoring sessions (1:1 student-tutor ratio) and 7,150 paired tutoring sessions (2:1 tutor-student ratio). Students were diverse in terms of demographics and learning needs, including those receiving free or reduced lunch, English learners, and students with disabilities. Tutoring was focused on early literacy development, and the main outcomes for the RCT were achievement scores on DIBELS and MAP Reading. Intervention/Program/Practice: The tutoring intervention involved early literacy virtual instruction delivered via a video conferencing platform. Tutors and students engaged in individualized or paired tutoring sessions using structured literacy materials. Sessions were recorded, and transcript and audio data were collected. A typical session lasted around 20-30 minutes, and the program implementation occurred from November 2022 to May 2023. Research Design: We use natural language processing (NLP) to classify and analyze types of disruptions during virtual tutoring. We developed a custom multi-class text classifier trained on tutor utterances labeled by human annotators. Sessions were selected from a two-week window in April 2023, and regression models were used to assess the relationship between session characteristics related to the students and their school and the prevalence of different types of disruptions. Data Collection and Analysis: We transcribed all 26,714 session recordings from the RCT's tutoring sessions for this study. We created a human-annotated set of 2,501 annotations of 1,488 unique utterances from 255 sessions to train our classifier. We achieved an inter-rater reliability of 0.71 using Krippendorff's alpha across 19 human annotators. Disruption categories included tech problems, student disruptions, background noise, and issues involving additional students or substitute tutors. A RoBERTa-based classifier was fine-tuned using 80% of the labeled data, with oversampling of less frequent categories to address class imbalance. Accuracy reached 74% overall (80% for disruption vs. non-disruption). We estimated disruption durations based on the classifier labeling of each utterance and utterance timestamps. Our analysis used session-level aggregation of disruption time, examining how disruption time varied by student composition (e.g., 1:1 vs. 2:1 tutoring), demographics, and school for the students in the session. We estimate the following regression, where i is the session, y_i is the percentage of session time spent on one of the categories from our disruption classifier, 2:1 is a dummy for students assigned to paired tutoring (instead of individual), X represents a vector of student demographics and other characteristics, [delta] represents school (s) fixed effects, and represents grade fixed effects. y[subscript i] = [beta subscript 0] + [beta subscript 1] (2 : 1t)[subscript i] + X[subscript i][gamma] + [delta subscript s(i)] + [lambda subscript g(i)] + [epsilon subscript i]. Findings/Results: We found that approximately 25% of tutoring time was spent managing disruptions. Of this, 11.8% of time was attributed to student behavior, and 9% to technology issues. Pair tutoring (2:1) sessions were associated with significantly less usable instructional time--nearly 9 percentage points lower than individual (1:1) sessions. Sessions with at least one female student had more usable instructional time (+3 p.p.), suggesting possible behavioral differences by gender. Tech issues were less prevalent among second-grade students, likely due to their greater experience with technology. We also found that tech issues and background noise disruptions correlated more strongly with specific schools, suggesting variations in the implementation environment affected the amount of instruction time available to students through virtual tutoring. Conclusions: Disruptions are a significant drain on instructional time in in-school virtual tutoring settings, with both student behavior and environmental factors contributing. Notably, disruptions varied systematically by session type, student demographics, and school, suggesting that targeted program design and implementation changes--such as quieter tutoring locations, better tech support, and smaller group sizes--could mitigate some of these effects. As we continue to work on this study, our next steps include extending our analysis to the full set of tutoring sessions and investigating whether lower levels of school environment disruptions, such as tech issues and background noise, also correlate with larger learning gains for students by estimating differential impact on student performance on the end-of-year DIBELS assessment. |
|---|
