Evaluating the Effectiveness of Writing Interventions for Students in Grades K-5: A Comprehensive Meta-Analysis
Saved in:
| Title: | Evaluating the Effectiveness of Writing Interventions for Students in Grades K-5: A Comprehensive Meta-Analysis |
|---|---|
| Language: | English |
| Authors: | Alyson Collins, Stephen Ciullo, Steve Graham, Joong won Lee, Society for Research on Educational Effectiveness (SREE) |
| Source: | Society for Research on Educational Effectiveness. 2024. |
| Availability: | Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/ |
| Peer Reviewed: | Y |
| Publication Date: | 2024 |
| Intended Audience: | Researchers |
| Document Type: | Information Analyses |
| Education Level: | Elementary Education |
| Descriptors: | Program Effectiveness, Writing Instruction, Elementary School Students, Intervention, Writing Skills, Meta Analysis, Effect Size, Writing Research, Research Methodology, At Risk Students, Students with Disabilities, English Language Learners, Low Income Students, Student Characteristics, Writing Strategies, Computer Assisted Instruction |
| Assessment and Survey Identifiers: | National Assessment of Educational Progress |
| Abstract: | Background/Context: Many students in the United States need access to effective writing instruction, with some requiring intensive intervention. Results from the National Assessment of Education Progress (NAEP) in writing over the past 25 years revealed that most students in the U.S. have yet to attain a proficient level of written expression (e.g., National Center for Education Statistics, 2012; Persky et al., 2002). Moreover, subgroup analysis of NAEP outcomes suggests a critical need to improve writing outcomes historically marginalized populations, including English Learners, students from low-income school districts, and students with a disability (Mo & Troia, 2017). Extending beyond students' needs, classroom teachers have expressed challenges with accessing effective instructional methods for teaching writing, as general educators as well as special educators reported feeling under-prepared to teach writing (Graham et al., 2023). As such, a comprehensive meta-analysis on writing instruction in Grades K-5 is necessary for two reasons. First, researchers need to expand upon previous systematic reviews because most meta-analyses in writing have narrowly focused on a single writing outcome (e.g., writing quality; Graham et al., 2012) or limited findings to a specific student population (e.g., students with learning disabilities (Gillespie & Graham, 2014). Although evaluating specific outcomes of writing interventions and their impact on certain populations of learners may inform everyday educational practice, a comprehensive examination of the existing research in writing greatly expands our knowledge of the extent which instruction is effective on a range of skills (e.g., handwriting, spelling), cognitive factors (e.g., motivation), and types of writers. Second, a comprehensive meta-analysis provides comparisons and contrasts to overall effect sizes alongside deeper investigations of effects for subgroups of interventions, outcomes, and learners. Ultimately, this meta-analysis extends prior reviews writing intervention research in Grades K to 5 by evaluating current evidence to shape future directions for classroom instruction and research. Purpose/Objective/Research Question: The current meta-analysis involved a comprehensive and systematic review of the writing intervention literature for students in Grades K-5. The study aimed to analyze the extant writing research to understand which writing interventions are effective, for whom, and on which writing outcomes. By synthesizing this body of research, researchers can make informed recommendations to pre-service teacher education programs, professional development providers, educational policy makers, and writing intervention researchers. Our hypothesis was that writing interventions using experimental and/or quasi-experimental designs would result in greater performance for students in Grades K to 5 in the treatment conditions (with moderate effect sizes), including when writing intervention is delivered to specific populations of learners (e.g., students with disabilities). Two research questions were investigated: 1. What writing interventions are effective on what writing outcomes for students in Grades K to 5? 2. What writing interventions are effective for which learners (i.e., for whom)? Research Design & Data Analysis: This meta-analysis investigated the extent to which writing interventions are effective on writing outcomes (e.g., quality, spelling) for students in Grades K to 5. Researchers applied rigorous systematic search methods to locate studies including electronic database searches, reference harvesting, journal hand searches, inspection of previously conducted reviews, and an author contact effort. Inclusion criteria were: (a) participants in Grade K to 5, (b) the study investigated a writing intervention, (c) the study included at least one writing outcome measure, (d) studies used randomized control trials (RCT) or quasi-experimental designs with pretest. No restrictions were placed on date of the study, the country in which the study was implemented, or language of instruction; however, all studies were reported in English. Eligible studies were implemented in a typical school setting (e.g., hospitals and clinic settings were excluded). Approximately 20% of studies were independently screened and coded by a second researcher. Reliability for screening and coding exceeded 92% and 80%, respectively. For data analysis, researchers utilized Hedge's g (Hedges, 1981) standardized mean difference effect size (ES) to estimate the effects of writing interventions when compared to control conditions, with ESs adjusted when pretest scores were reported. R packages metafor and clubSandwich (Pustejovsky, 2022) were used to apply Robust Variance Estimation (RVE) procedures and to estimate the average ES for categories of writing interventions (e.g., strategy instruction, spelling interventions, computer-assisted instruction). Additionally, separate models were estimated for each writing outcome (e.g., holistic quality, spelling, writing output) and subgroup of learners, with a value of [rho] = 0.80 applied to estimate between-study variance (Hedges et al., 2010). Moderator analyses further explored if study characteristics including intervention duration, group size, or language of instruction explained heterogeneity in study ES. Results identified 362 eligible studies that included 3,727 ESs. Robust Variance Estimation (RVE) models were conducted. The full model (including all ES) produced moderate effects in favor of students receiving treatment (ES = 0.34). Subgroup analyses were also conducted. These additional RVE models revealed that writing interventions, on average, have large effects on outcomes of genre elements (ES = 0.89) and organizational processes (ES = 2.14). Conversely, on average, smaller effects were reported for writing mechanics outcomes (spelling ES = 0.41) and conventions (ES = 0.33). Another noteworthy findings was that simply providing students with time to write revealed a small ES that was not statistically significant. Further, large effects were reported across student populations for writing strategy instruction, including self-regulated strategy development (SRSD). In addition, computer-assisted interventions produced a small effect size (and not statistically significant), but the corpus of included technology-based studies was relatively small (n=12). Conclusions: Findings from this comprehensive meta-analysis has important implications for how educators use writing interventions to improve outcomes for different populations of learners. Discussion will consider implications for using different approaches to teaching writing, underscoring interventions with potential to have greater impact on elementary students. In addition, discussion consider areas needing further attention in pre- and in-service professional to influence on-the-ground teacher practice. Discussion will conclude with considerations for future research in writing. |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Access URL: | https://www.sree.org/2024-conference |
| Accession Number: | ED663432 |
| Database: | ERIC |
| Abstract: | Background/Context: Many students in the United States need access to effective writing instruction, with some requiring intensive intervention. Results from the National Assessment of Education Progress (NAEP) in writing over the past 25 years revealed that most students in the U.S. have yet to attain a proficient level of written expression (e.g., National Center for Education Statistics, 2012; Persky et al., 2002). Moreover, subgroup analysis of NAEP outcomes suggests a critical need to improve writing outcomes historically marginalized populations, including English Learners, students from low-income school districts, and students with a disability (Mo & Troia, 2017). Extending beyond students' needs, classroom teachers have expressed challenges with accessing effective instructional methods for teaching writing, as general educators as well as special educators reported feeling under-prepared to teach writing (Graham et al., 2023). As such, a comprehensive meta-analysis on writing instruction in Grades K-5 is necessary for two reasons. First, researchers need to expand upon previous systematic reviews because most meta-analyses in writing have narrowly focused on a single writing outcome (e.g., writing quality; Graham et al., 2012) or limited findings to a specific student population (e.g., students with learning disabilities (Gillespie & Graham, 2014). Although evaluating specific outcomes of writing interventions and their impact on certain populations of learners may inform everyday educational practice, a comprehensive examination of the existing research in writing greatly expands our knowledge of the extent which instruction is effective on a range of skills (e.g., handwriting, spelling), cognitive factors (e.g., motivation), and types of writers. Second, a comprehensive meta-analysis provides comparisons and contrasts to overall effect sizes alongside deeper investigations of effects for subgroups of interventions, outcomes, and learners. Ultimately, this meta-analysis extends prior reviews writing intervention research in Grades K to 5 by evaluating current evidence to shape future directions for classroom instruction and research. Purpose/Objective/Research Question: The current meta-analysis involved a comprehensive and systematic review of the writing intervention literature for students in Grades K-5. The study aimed to analyze the extant writing research to understand which writing interventions are effective, for whom, and on which writing outcomes. By synthesizing this body of research, researchers can make informed recommendations to pre-service teacher education programs, professional development providers, educational policy makers, and writing intervention researchers. Our hypothesis was that writing interventions using experimental and/or quasi-experimental designs would result in greater performance for students in Grades K to 5 in the treatment conditions (with moderate effect sizes), including when writing intervention is delivered to specific populations of learners (e.g., students with disabilities). Two research questions were investigated: 1. What writing interventions are effective on what writing outcomes for students in Grades K to 5? 2. What writing interventions are effective for which learners (i.e., for whom)? Research Design & Data Analysis: This meta-analysis investigated the extent to which writing interventions are effective on writing outcomes (e.g., quality, spelling) for students in Grades K to 5. Researchers applied rigorous systematic search methods to locate studies including electronic database searches, reference harvesting, journal hand searches, inspection of previously conducted reviews, and an author contact effort. Inclusion criteria were: (a) participants in Grade K to 5, (b) the study investigated a writing intervention, (c) the study included at least one writing outcome measure, (d) studies used randomized control trials (RCT) or quasi-experimental designs with pretest. No restrictions were placed on date of the study, the country in which the study was implemented, or language of instruction; however, all studies were reported in English. Eligible studies were implemented in a typical school setting (e.g., hospitals and clinic settings were excluded). Approximately 20% of studies were independently screened and coded by a second researcher. Reliability for screening and coding exceeded 92% and 80%, respectively. For data analysis, researchers utilized Hedge's g (Hedges, 1981) standardized mean difference effect size (ES) to estimate the effects of writing interventions when compared to control conditions, with ESs adjusted when pretest scores were reported. R packages metafor and clubSandwich (Pustejovsky, 2022) were used to apply Robust Variance Estimation (RVE) procedures and to estimate the average ES for categories of writing interventions (e.g., strategy instruction, spelling interventions, computer-assisted instruction). Additionally, separate models were estimated for each writing outcome (e.g., holistic quality, spelling, writing output) and subgroup of learners, with a value of [rho] = 0.80 applied to estimate between-study variance (Hedges et al., 2010). Moderator analyses further explored if study characteristics including intervention duration, group size, or language of instruction explained heterogeneity in study ES. Results identified 362 eligible studies that included 3,727 ESs. Robust Variance Estimation (RVE) models were conducted. The full model (including all ES) produced moderate effects in favor of students receiving treatment (ES = 0.34). Subgroup analyses were also conducted. These additional RVE models revealed that writing interventions, on average, have large effects on outcomes of genre elements (ES = 0.89) and organizational processes (ES = 2.14). Conversely, on average, smaller effects were reported for writing mechanics outcomes (spelling ES = 0.41) and conventions (ES = 0.33). Another noteworthy findings was that simply providing students with time to write revealed a small ES that was not statistically significant. Further, large effects were reported across student populations for writing strategy instruction, including self-regulated strategy development (SRSD). In addition, computer-assisted interventions produced a small effect size (and not statistically significant), but the corpus of included technology-based studies was relatively small (n=12). Conclusions: Findings from this comprehensive meta-analysis has important implications for how educators use writing interventions to improve outcomes for different populations of learners. Discussion will consider implications for using different approaches to teaching writing, underscoring interventions with potential to have greater impact on elementary students. In addition, discussion consider areas needing further attention in pre- and in-service professional to influence on-the-ground teacher practice. Discussion will conclude with considerations for future research in writing. |
|---|
