When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295
Saved in:
| Title: | When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295 |
|---|---|
| Language: | English |
| Authors: | Elizabeth Huffaker, Carly D. Robinson, Emanuele Bardelli, Sara White, Susanna Loeb, Annenberg Institute for School Reform at Brown University |
| Source: | Annenberg Institute for School Reform at Brown University. 2025. |
| Availability: | Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/ |
| Peer Reviewed: | N |
| Page Count: | 56 |
| Publication Date: | 2025 |
| Document Type: | Reports - Research |
| Education Level: | Early Childhood Education Elementary Education Grade 3 Primary Education Grade 4 Intermediate Grades Grade 5 Middle Schools Grade 6 Grade 7 Junior High Schools Secondary Education Grade 8 |
| Descriptors: | Intervention, Tutoring, Electronic Learning, Grade 3, Grade 4, Grade 5, Grade 6, Grade 7, Grade 8, Suburban Schools, School Districts, Mathematics Achievement, Reading Achievement, Academic Achievement, Synchronous Communication, Small Group Instruction, Mathematics Tests, Reading Tests, Scores, Achievement Tests, Educational Legislation, State Legislation, Program Effectiveness, Elementary School Students, Middle School Students, Supplementary Education, Mathematics Instruction, Reading Instruction, Educational Research |
| Geographic Terms: | Texas |
| Assessment and Survey Identifiers: | State of Texas Assessments of Academic Readiness (STAAR), Measures of Academic Progress |
| Abstract: | As school districts focus on improving learning, they can learn not only from when and where interventions work--but also from why they sometimes do not. Policymakers widely embraced high-impact tutoring as an evidence-supported strategy to address learning delays from the COVID-19 pandemic. However, scaling these promising practices can be difficult, and not all implementations will be effective. Many districts have turned to third-party virtual tutoring providers to deliver student supports during the school day. Using random assignment, we evaluate the impacts of one such program for 3rd through 8th grade students in a suburban Texas school district. Compared with students assigned to the comparison interventions, we find no effect of assignment to virtual tutoring on math achievement, and, for reading, we find a moderate negative effect on the state end-of-year assessment (i.e., -0.09 SD) and no effect on a low-stakes exam. Drawing from frameworks for interpreting null or unexpected results in education experiments, we find further evidence of subject-specific heterogeneity in the implementation and efficacy and identify coverage of standards-aligned material as a moderator of estimated effectiveness relative to "business-as-usual" interventions. This paper offers strategies to identify factors contributing to null or unexpected results and highlights implications for designing policy-relevant studies to assess educational interventions. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | ED678261 |
| Database: | ERIC |
| Abstract: | As school districts focus on improving learning, they can learn not only from when and where interventions work--but also from why they sometimes do not. Policymakers widely embraced high-impact tutoring as an evidence-supported strategy to address learning delays from the COVID-19 pandemic. However, scaling these promising practices can be difficult, and not all implementations will be effective. Many districts have turned to third-party virtual tutoring providers to deliver student supports during the school day. Using random assignment, we evaluate the impacts of one such program for 3rd through 8th grade students in a suburban Texas school district. Compared with students assigned to the comparison interventions, we find no effect of assignment to virtual tutoring on math achievement, and, for reading, we find a moderate negative effect on the state end-of-year assessment (i.e., -0.09 SD) and no effect on a low-stakes exam. Drawing from frameworks for interpreting null or unexpected results in education experiments, we find further evidence of subject-specific heterogeneity in the implementation and efficacy and identify coverage of standards-aligned material as a moderator of estimated effectiveness relative to "business-as-usual" interventions. This paper offers strategies to identify factors contributing to null or unexpected results and highlights implications for designing policy-relevant studies to assess educational interventions. |
|---|
