Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083
Saved in:
| Title: | Examining the Relationship between Randomization Strategies and Contamination in Higher Education Interventions. EdWorkingPaper No. 24-1083 |
|---|---|
| Language: | English |
| Authors: | Catherine Mata, Katharine Meyer, Lindsay Page, Annenberg Institute for School Reform at Brown University |
| Source: | Annenberg Institute for School Reform at Brown University. 2025. |
| Availability: | Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/ |
| Peer Reviewed: | N |
| Page Count: | 34 |
| Publication Date: | 2025 |
| Sponsoring Agency: | Ascendium Education Group, Inc. |
| Document Type: | Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Randomized Controlled Trials, Higher Education, Intervention, Large Group Instruction, In Person Learning, Artificial Intelligence, Statistical Analysis, Effect Size, State Universities, Outreach Programs, Undergraduate Students, Technology Integration, Research Methodology, Chemistry, College Science |
| Geographic Terms: | Georgia (Atlanta) |
| Abstract: | Randomized controlled trials are the reference method for causal inference, but field experiments in educational settings must balance statistical power with the risk of contamination. This study examines crossover and spillover contamination in a large-enrollment, in-person college course implementing an AI-enabled chatbot intervention. We compare two randomization approaches, individual-level and laboratory-level, to assess contamination risks. Contrary to expectations, no crossover occurred under student-level randomization. However, survey data suggest potential spillover, with treatment-group students reporting that they shared chatbot messages with peers. Using estimated contamination levels, we assess changes in minimum detectable effect size (MDES) and show that individual-level randomization remains preferable. Our findings offer practical guidance for balancing contamination risk and statistical power when designing RCTs in interactive educational settings. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | ED678174 |
| Database: | ERIC |
| Abstract: | Randomized controlled trials are the reference method for causal inference, but field experiments in educational settings must balance statistical power with the risk of contamination. This study examines crossover and spillover contamination in a large-enrollment, in-person college course implementing an AI-enabled chatbot intervention. We compare two randomization approaches, individual-level and laboratory-level, to assess contamination risks. Contrary to expectations, no crossover occurred under student-level randomization. However, survey data suggest potential spillover, with treatment-group students reporting that they shared chatbot messages with peers. Using estimated contamination levels, we assess changes in minimum detectable effect size (MDES) and show that individual-level randomization remains preferable. Our findings offer practical guidance for balancing contamination risk and statistical power when designing RCTs in interactive educational settings. |
|---|
