View in EDS

Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083

Saved in:
Bibliographic Details
Title: Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083
Language: English
Authors: Catherine Mata, Katharine Meyer, Lindsay Page, Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2025.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 34
Publication Date: 2025
Sponsoring Agency: Ascendium Education Group, Inc.
Document Type: Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Randomized Controlled Trials, Higher Education, Intervention, Large Group Instruction, In Person Learning, Artificial Intelligence, Statistical Analysis, Effect Size, State Universities, Outreach Programs, Undergraduate Students, Technology Integration, Research Methodology, Chemistry, College Science
Geographic Terms: Georgia (Atlanta)
Abstract: Randomized controlled trials are the reference method for causal inference, but field experiments in educational settings must balance statistical power with the risk of contamination. This study examines crossover and spillover contamination in a large-enrollment, in-person college course implementing an AI-enabled chatbot intervention. We compare two randomization approaches, individual-level and laboratory-level, to assess contamination risks. Contrary to expectations, no crossover occurred under student-level randomization. However, survey data suggest potential spillover, with treatment-group students reporting that they shared chatbot messages with peers. Using estimated contamination levels, we assess changes in minimum detectable effect size (MDES) and show that individual-level randomization remains preferable. Our findings offer practical guidance for balancing contamination risk and statistical power when designing RCTs in interactive educational settings.
Abstractor: As Provided
Entry Date: 2026
Accession Number: ED678174
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED678174
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED678174
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Catherine+Mata%22">Catherine Mata</searchLink><br /><searchLink fieldCode="AR" term="%22Katharine+Meyer%22">Katharine Meyer</searchLink><br /><searchLink fieldCode="AR" term="%22Lindsay+Page%22">Lindsay Page</searchLink><br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 34
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Ascendium Education Group, Inc.
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Large+Group+Instruction%22">Large Group Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22In+Person+Learning%22">In Person Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Effect+Size%22">Effect Size</searchLink><br /><searchLink fieldCode="DE" term="%22State+Universities%22">State Universities</searchLink><br /><searchLink fieldCode="DE" term="%22Outreach+Programs%22">Outreach Programs</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Integration%22">Technology Integration</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Chemistry%22">Chemistry</searchLink><br /><searchLink fieldCode="DE" term="%22College+Science%22">College Science</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Georgia+%28Atlanta%29%22">Georgia (Atlanta)</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Randomized controlled trials are the reference method for causal inference, but field experiments in educational settings must balance statistical power with the risk of contamination. This study examines crossover and spillover contamination in a large-enrollment, in-person college course implementing an AI-enabled chatbot intervention. We compare two randomization approaches, individual-level and laboratory-level, to assess contamination risks. Contrary to expectations, no crossover occurred under student-level randomization. However, survey data suggest potential spillover, with treatment-group students reporting that they shared chatbot messages with peers. Using estimated contamination levels, we assess changes in minimum detectable effect size (MDES) and show that individual-level randomization remains preferable. Our findings offer practical guidance for balancing contamination risk and statistical power when designing RCTs in interactive educational settings.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED678174
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED678174
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 34
    Subjects:
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Higher Education
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Large Group Instruction
        Type: general
      – SubjectFull: In Person Learning
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Effect Size
        Type: general
      – SubjectFull: State Universities
        Type: general
      – SubjectFull: Outreach Programs
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Technology Integration
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Chemistry
        Type: general
      – SubjectFull: College Science
        Type: general
      – SubjectFull: Georgia (Atlanta)
        Type: general
    Titles:
      – TitleFull: Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: Catherine Mata
      – PersonEntity:
          Name:
            NameFull: Katharine Meyer
      – PersonEntity:
          Name:
            NameFull: Lindsay Page
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 29
              M: 08
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1