View in EDS

When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295

Saved in:
Bibliographic Details
Title: When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295
Language: English
Authors: Elizabeth Huffaker, Carly D. Robinson, Emanuele Bardelli, Sara White, Susanna Loeb, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 56
Publication Date: 2025
Document Type: Reports - Research
Education Level: Early Childhood Education
Elementary Education
Grade 3
Primary Education
Grade 4
Intermediate Grades
Grade 5
Middle Schools
Grade 6
Grade 7
Junior High Schools
Secondary Education
Grade 8
Descriptors: Intervention, Tutoring, Electronic Learning, Grade 3, Grade 4, Grade 5, Grade 6, Grade 7, Grade 8, Suburban Schools, School Districts, Mathematics Achievement, Reading Achievement, Academic Achievement, Synchronous Communication, Small Group Instruction, Mathematics Tests, Reading Tests, Scores, Achievement Tests, Educational Legislation, State Legislation, Program Effectiveness, Elementary School Students, Middle School Students, Supplementary Education, Mathematics Instruction, Reading Instruction, Educational Research
Geographic Terms: Texas
Assessment and Survey Identifiers: State of Texas Assessments of Academic Readiness (STAAR), Measures of Academic Progress
Abstract: As school districts focus on improving learning, they can learn not only from when and where interventions work--but also from why they sometimes do not. Policymakers widely embraced high-impact tutoring as an evidence-supported strategy to address learning delays from the COVID-19 pandemic. However, scaling these promising practices can be difficult, and not all implementations will be effective. Many districts have turned to third-party virtual tutoring providers to deliver student supports during the school day. Using random assignment, we evaluate the impacts of one such program for 3rd through 8th grade students in a suburban Texas school district. Compared with students assigned to the comparison interventions, we find no effect of assignment to virtual tutoring on math achievement, and, for reading, we find a moderate negative effect on the state end-of-year assessment (i.e., -0.09 SD) and no effect on a low-stakes exam. Drawing from frameworks for interpreting null or unexpected results in education experiments, we find further evidence of subject-specific heterogeneity in the implementation and efficacy and identify coverage of standards-aligned material as a moderator of estimated effectiveness relative to "business-as-usual" interventions. This paper offers strategies to identify factors contributing to null or unexpected results and highlights implications for designing policy-relevant studies to assess educational interventions.
Abstractor: As Provided
Entry Date: 2026
Accession Number: ED678261
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678261
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678261
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Elizabeth+Huffaker%22">Elizabeth Huffaker</searchLink><br /><searchLink fieldCode="AR" term="%22Carly+D%2E+Robinson%22">Carly D. Robinson</searchLink><br /><searchLink fieldCode="AR" term="%22Emanuele+Bardelli%22">Emanuele Bardelli</searchLink><br /><searchLink fieldCode="AR" term="%22Sara+White%22">Sara White</searchLink><br /><searchLink fieldCode="AR" term="%22Susanna+Loeb%22">Susanna Loeb</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 56
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+3%22">Grade 3</searchLink><br /><searchLink fieldCode="EL" term="%22Primary+Education%22">Primary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="EL" term="%22Intermediate+Grades%22">Intermediate Grades</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+8%22">Grade 8</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Tutoring%22">Tutoring</searchLink><br /><searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+3%22">Grade 3</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+8%22">Grade 8</searchLink><br /><searchLink fieldCode="DE" term="%22Suburban+Schools%22">Suburban Schools</searchLink><br /><searchLink fieldCode="DE" term="%22School+Districts%22">School Districts</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Achievement%22">Reading Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Synchronous+Communication%22">Synchronous Communication</searchLink><br /><searchLink fieldCode="DE" term="%22Small+Group+Instruction%22">Small Group Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Tests%22">Reading Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Legislation%22">Educational Legislation</searchLink><br /><searchLink fieldCode="DE" term="%22State+Legislation%22">State Legislation</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Supplementary+Education%22">Supplementary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Instruction%22">Mathematics Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Instruction%22">Reading Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Texas%22">Texas</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22State+of+Texas+Assessments+of+Academic+Readiness+%28STAAR%29%22">State of Texas Assessments of Academic Readiness (STAAR)</searchLink><br /><searchLink fieldCode="SU" term="%22Measures+of+Academic+Progress%22">Measures of Academic Progress</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: As school districts focus on improving learning, they can learn not only from when and where interventions work--but also from why they sometimes do not. Policymakers widely embraced high-impact tutoring as an evidence-supported strategy to address learning delays from the COVID-19 pandemic. However, scaling these promising practices can be difficult, and not all implementations will be effective. Many districts have turned to third-party virtual tutoring providers to deliver student supports during the school day. Using random assignment, we evaluate the impacts of one such program for 3rd through 8th grade students in a suburban Texas school district. Compared with students assigned to the comparison interventions, we find no effect of assignment to virtual tutoring on math achievement, and, for reading, we find a moderate negative effect on the state end-of-year assessment (i.e., -0.09 SD) and no effect on a low-stakes exam. Drawing from frameworks for interpreting null or unexpected results in education experiments, we find further evidence of subject-specific heterogeneity in the implementation and efficacy and identify coverage of standards-aligned material as a moderator of estimated effectiveness relative to "business-as-usual" interventions. This paper offers strategies to identify factors contributing to null or unexpected results and highlights implications for designing policy-relevant studies to assess educational interventions.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678261
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678261
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 56
    Subjects:
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Tutoring
        Type: general
      – SubjectFull: Electronic Learning
        Type: general
      – SubjectFull: Grade 3
        Type: general
      – SubjectFull: Grade 4
        Type: general
      – SubjectFull: Grade 5
        Type: general
      – SubjectFull: Grade 6
        Type: general
      – SubjectFull: Grade 7
        Type: general
      – SubjectFull: Grade 8
        Type: general
      – SubjectFull: Suburban Schools
        Type: general
      – SubjectFull: School Districts
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Reading Achievement
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Synchronous Communication
        Type: general
      – SubjectFull: Small Group Instruction
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Reading Tests
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Educational Legislation
        Type: general
      – SubjectFull: State Legislation
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Supplementary Education
        Type: general
      – SubjectFull: Mathematics Instruction
        Type: general
      – SubjectFull: Reading Instruction
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Texas
        Type: general
      – SubjectFull: State of Texas Assessments of Academic Readiness (STAAR)
        Type: general
      – SubjectFull: Measures of Academic Progress
        Type: general
    Titles:
      – TitleFull: When Interventions Don't Move the Needle: Insights from Null Results in Education Research. EdWorkingPaper No. 25-1295
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Elizabeth Huffaker
      – PersonEntity:
          Name:
            NameFull: Carly D. Robinson
      – PersonEntity:
          Name:
            NameFull: Emanuele Bardelli
      – PersonEntity:
          Name:
            NameFull: Sara White
      – PersonEntity:
          Name:
            NameFull: Susanna Loeb
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1