View in EDS HTML Full Text PDF Full Text

The Spanish- and English-Language Literacy Impacts of 'Descubriendo la Lectura' across Three Experimental Replications

Saved in:
Bibliographic Details
Title: The Spanish- and English-Language Literacy Impacts of 'Descubriendo la Lectura' across Three Experimental Replications
Language: English
Authors: Trisha H. Borman (ORCID 0000-0003-4176-4202), Geoffrey D. Borman (ORCID 0000-0002-7039-8208), So Jung Park, Bo Zhu, Scott Houghton
Source: Journal of Research on Educational Effectiveness. 2024 17(4):978-1007.
Availability: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Peer Reviewed: Y
Page Count: 30
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Education
Early Childhood Education
Grade 1
Primary Education
Descriptors: Literacy, Elementary School Students, Grade 1, Spanish, Bilingual Students, English (Second Language), English Language Learners, Randomized Controlled Trials, Tutoring, Reading Skills, Early Intervention
Geographic Terms: California, Illinois, Texas, Wisconsin, Arizona
DOI: 10.1080/19345747.2023.2269936
ISSN: 1934-5747
1934-5739
Abstract: We present findings across three randomized trials of "Descubriendo la Lectura" (DLL), an intervention designed to improve literacy skills of Spanish-speaking first graders struggling with reading. DLL is a one-to-one Spanish-language literacy program lasting 12-20 weeks offered to a school's lowest performing emerging bilingual first graders. Directly replicating procedures across three cohorts, we examined Spanish and English literacy outcomes for approximately 400 students across 30 schools, 10 districts, and five states. Analyses revealed statistically significant effects of student-level assignment to DLL on all 11 Spanish-language outcomes, with an average impact of d = 0.57. Impacts across the four English-language outcomes were positive but not statistically significant. Descriptive results suggest that the Spanish-language impacts are reliably replicated, most likely due to the intervention's strong implementation supports.
Abstractor: As Provided
Notes: https://www.openicpsr.org/openicpsr/project/193586
Entry Date: 2024
Accession Number: EJ1447262
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGZs8U3LhS-XhGzDqJW3xjeAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDIF0OUxM4xarZie09gIBEICBmlCXYwqIBrtyxJ3iCkb2qK-ip6-SeGx6qabw7WO7l861O6A-GFJMV7jvzaZQPjits0EhC6zsm3RFxDW2G55yNi0EcJWWVRTyGsdrkLDIbxOp3LKvHLDySJVyL2p6hHqb3IdrXj_1L2MJ0XWylXd4pPVlRbJ1NOf20jLnR_eWRJmiQKe5jHCJEicLL3Zh8gkfSqAMXceFj9dQfv0=
Text:
  Availability: 1
  Value: <anid>AN0180591483;[5ew9]01oct.24;2024Nov04.05:50;v2.2.500</anid> <title id="AN0180591483-1">The Spanish- and English-Language Literacy Impacts of Descubriendo la Lectura across Three Experimental Replications </title> <p>We present findings across three randomized trials of Descubriendo la Lectura (DLL), an intervention designed to improve literacy skills of Spanish-speaking first graders struggling with reading. DLL is a one-to-one Spanish-language literacy program lasting 12–20 weeks offered to a school's lowest performing emerging bilingual first graders. Directly replicating procedures across three cohorts, we examined Spanish and English literacy outcomes for approximately 400 students across 30 schools, 10 districts, and five states. Analyses revealed statistically significant effects of student-level assignment to DLL on all 11 Spanish-language outcomes, with an average impact of d = 0.57. Impacts across the four English-language outcomes were positive but not statistically significant. Descriptive results suggest that the Spanish-language impacts are reliably replicated, most likely due to the intervention's strong implementation supports.</p> <p>Keywords: Early literacy; English learners; high-intensity tutoring; randomized controlled trial; replication</p> <p>Prevailing educational policies and practices have failed to capitalize on the strengths of Latinx English learners (ELs), including the unique potential of bilingualism—a versatile asset with cognitive, social-emotional learning, and future employment benefits (Bialystock, [<reflink idref="bib7" id="ref1">7</reflink>]; Chen & Padilla, [<reflink idref="bib17" id="ref2">17</reflink>]; Nagy et al., [<reflink idref="bib50" id="ref3">50</reflink>]; Portes & Hao, [<reflink idref="bib59" id="ref4">59</reflink>]; Vom Dorp, [<reflink idref="bib76" id="ref5">76</reflink>]). A consensus of evidence indicates that the most effective strategy for cultivating the early literacy skills of emerging bilingual students is a strengths-based approach, which draws on the Spanish-language skills that students possess as they begin formal schooling. Research suggests that early elementary bilingual programs produce stronger long-term English proficiency and English language arts (ELA) achievement compared with English-only programs while simultaneously developing students' Spanish-language assets and promoting bilingualism (Francis et al., [<reflink idref="bib32" id="ref6">32</reflink>]; National Academies of Sciences Engineering & Medicine, 2017; Rolstad et al., [<reflink idref="bib66" id="ref7">66</reflink>]; Slavin & Cheung, [<reflink idref="bib69" id="ref8">69</reflink>]; Umansky & Reardon, [<reflink idref="bib72" id="ref9">72</reflink>]; Valentino & Reardon, [<reflink idref="bib73" id="ref10">73</reflink>]). To take advantage of the strengths associated with bilingual education, educators need proven programs and strategies that enrich opportunities for ELs.</p> <p>As of 2019, ELs accounted for more than five million (10%) of the total K–12 student population, with a majority being Spanish speakers (77%) and approximately 15% of ELs concentrated in first grade (de Brey et al., [<reflink idref="bib25" id="ref11">25</reflink>]; National Center for Education Statistics, [<reflink idref="bib52" id="ref12">52</reflink>]). As the fastest growing population of students, a recent projection suggested that ELs will comprise one of every four students in the United States by 2025 (National Education Association, [<reflink idref="bib53" id="ref13">53</reflink>]). As the Spanish-speaking EL population in the United States continues to grow (National Center for Education Statistics, [<reflink idref="bib52" id="ref14">52</reflink>]), schools have increasingly turned to bilingual programming, particularly dual-language approaches, to support students' language needs (Arteagoitia & Yen, [<reflink idref="bib1" id="ref15">1</reflink>]; Escamilla, [<reflink idref="bib29" id="ref16">29</reflink>]). With the significant historical and current barriers to school opportunities and success that Latinx ELs have faced, such as poorly funded schools, suboptimal learning environments, less experienced teachers, and English-only programming (Galindo & Reardon, [<reflink idref="bib34" id="ref17">34</reflink>]; Gandara & Hopkins, [<reflink idref="bib35" id="ref18">35</reflink>]; Reardon & Galindo, [<reflink idref="bib63" id="ref19">63</reflink>]), effective Spanish-language literacy supports will be more important than ever in the coming years.</p> <p>Indeed, all students, including ELs, face literacy challenges during their first years of formal schooling. Intervening early, when many ELs confront reading and language difficulties simultaneously, can mitigate longer term future literacy challenges that are more difficult to resolve (Cheung & Slavin, [<reflink idref="bib18" id="ref20">18</reflink>]; Francis et al., [<reflink idref="bib33" id="ref21">33</reflink>]; Knudsen et al., [<reflink idref="bib43" id="ref22">43</reflink>]). Providing literacy supports in first grade can be particularly beneficial, before reading complexity increases (Wanzek et al., [<reflink idref="bib78" id="ref23">78</reflink>]; Wanzek & Vaughn, [<reflink idref="bib77" id="ref24">77</reflink>]) and before bilingual programs transition to increasing the use of English instruction (Calderón & Slakk, [<reflink idref="bib16" id="ref25">16</reflink>]). What scalable evidence-based solutions are available to support the beginning-literacy skill development and school success of those Spanish-language ELs who are struggling?</p> <p>Our review of the What Works Clearinghouse (WWC) website,[<reflink idref="bib1" id="ref26">1</reflink>] multiple meta-analyses (Ludwig et al., [<reflink idref="bib46" id="ref27">46</reflink>]; Richards-Tutor et al., [<reflink idref="bib64" id="ref28">64</reflink>]), and research reviews (August et al., [<reflink idref="bib4" id="ref29">4</reflink>]; August & Shanahan, [<reflink idref="bib5" id="ref30">5</reflink>]; Baker et al., [<reflink idref="bib6" id="ref31">6</reflink>]; Cheung & Slavin, [<reflink idref="bib18" id="ref32">18</reflink>], [<reflink idref="bib19" id="ref33">19</reflink>]; Gersten et al., [<reflink idref="bib36" id="ref34">36</reflink>]; Snyder et al., [<reflink idref="bib71" id="ref35">71</reflink>]) revealed very few proven programs designed to support early elementary ELs who are struggling with reading. One review, in fact, prominently noted that there were "dramatically" fewer studies of quality reading interventions for ELs relative to those for native English speakers (Richards-Tutor et al., [<reflink idref="bib64" id="ref36">64</reflink>], p. 161). Indeed, our review identified only three programs with any evidence of impact on emerging bilingual students' Spanish-language outcomes: <emph>Enhanced Proactive Reading</emph> (currently distributed by McGraw Hill Education as <emph>Intervenciones tempranas de la lectura</emph>; see Vaughn, Cirino, et al., [<reflink idref="bib74" id="ref37">74</reflink>]; Vaughn, Linan-Thompson, et al., [<reflink idref="bib75" id="ref38">75</reflink>]); the former Spanish-language adaptation of Success for All, <emph>Exito Para Todos</emph> (see Dianda & Flaherty, [<reflink idref="bib27" id="ref39">27</reflink>]; Nunnery et al., [<reflink idref="bib56" id="ref40">56</reflink>]); and the intervention that is the subject of our study here, <emph>Descubriendo la Lectura</emph> (DLL).</p> <hd id="AN0180591483-2">The DLL Program</hd> <p>DLL is the Spanish-language reconstruction of Reading Recovery (RR), providing one-to-one literacy tutoring in Spanish to first-grade ELs struggling to learn to read. The DLL program comprises material and personnel resources, advanced training and ongoing professional development (PD), and high-quality student instruction designed to improve literacy skills and comprehension for emerging bilingual students for whom Spanish is their first language. The overall DLL theory of change is summarized in Figure 1.</p> <p>PHOTO (COLOR): Figure 1. Theory of change for the Descubriendo la Lectura program.</p> <hd id="AN0180591483-3">Resources</hd> <p>At the top left side of Figure 1, which is the heart of DLL, are several evidence-based literacy strategies, including early intervention, native language instruction, essential research-based reading components, tutoring, and self-directed learning. Also, as highlighted among the other resources summarized within the left panel, DLL calls for rigorous teacher selection and training standards, along with a personnel structure designed to include coordination across a district, a school, and the DLL teacher. Finally, a variety of proprietary material resources support the delivery of DLL as a replicable program.</p> <hd id="AN0180591483-4">Early Intervention</hd> <p>DLL reaches students in first grade before reading complexity increases and many bilingual programs transition to more English instruction (Calderón et al., [<reflink idref="bib15" id="ref41">15</reflink>]; Wanzek & Vaughn, [<reflink idref="bib77" id="ref42">77</reflink>]). Research suggests that students face substantial difficulties catching up if they fall behind in reading in first grade (Juel, [<reflink idref="bib42" id="ref43">42</reflink>]; Lyon & Fletcher, [<reflink idref="bib47" id="ref44">47</reflink>]; Shaywitz et al., [<reflink idref="bib67" id="ref45">67</reflink>]). Investing in early literacy instruction has profound, positive effects on students' neurological and skill development and long-term academic trajectories (Cheung & Slavin, [<reflink idref="bib18" id="ref46">18</reflink>]; Francis et al., [<reflink idref="bib33" id="ref47">33</reflink>]; Juel, [<reflink idref="bib42" id="ref48">42</reflink>]; Knudsen et al., [<reflink idref="bib43" id="ref49">43</reflink>]).</p> <hd id="AN0180591483-5">Native Language Learning</hd> <p>Aligned to early instruction in most bilingual programs (Calderón et al., [<reflink idref="bib15" id="ref50">15</reflink>]), DLL is provided in a student's native language. Prior research (Francis et al., [<reflink idref="bib32" id="ref51">32</reflink>]; Greene, [<reflink idref="bib38" id="ref52">38</reflink>]; McField & McField, [<reflink idref="bib49" id="ref53">49</reflink>]; Rolstad et al., [<reflink idref="bib66" id="ref54">66</reflink>]; Slavin & Cheung, [<reflink idref="bib69" id="ref55">69</reflink>]; Willig, [<reflink idref="bib81" id="ref56">81</reflink>]) indicates that teaching students in both English and their native language produces stronger literacy results than English-only programs. Research also demonstrates that fostering children's first language development has positive impacts on acquiring the second language (August & Hakuta, [<reflink idref="bib3" id="ref57">3</reflink>]; Cummins, [<reflink idref="bib24" id="ref58">24</reflink>]; Durgunoğlu et al., [<reflink idref="bib28" id="ref59">28</reflink>]; Gottardo, [<reflink idref="bib37" id="ref60">37</reflink>]). Differing amounts of English instruction can support initial learning, but literacy programs using native language instruction promote long-term academic gains (Ashdown & Simic, [<reflink idref="bib2" id="ref61">2</reflink>]).</p> <hd id="AN0180591483-6">Essential Components of Reading Instruction</hd> <p>DLL lessons are grounded in Marie Clay's scholarship on literacy development (Clay [<reflink idref="bib20" id="ref62">20</reflink>], [<reflink idref="bib22" id="ref63">22</reflink>]). Clay ([<reflink idref="bib22" id="ref64">22</reflink>]) argued that a child challenged by literacy learning requires a skilled teacher who is professionally trained and who embraces the complexity of literacy learning required for supporting the range of diversity among literacy learners. DLL incorporates the National Reading Panel's essential elements—phonemic awareness, phonics, fluency, vocabulary, and text comprehension (National Institute of Child Health & Human Development, 2000)—which are emphasized across student lessons.</p> <hd id="AN0180591483-7">Tutoring</hd> <p>Research supports DLL's use of one-to-one instruction (i.e., tutoring) for students struggling with reading, including ELs (Cheung & Slavin, [<reflink idref="bib19" id="ref65">19</reflink>]; Slavin et al., [<reflink idref="bib70" id="ref66">70</reflink>]). Several DLL components align to evidence-based tutoring practices: coordination with the classroom teacher, intensive tutor training, well-structured sessions, progress monitoring, and a daily cadence (Planning and Evaluation Service, Office of the Deputy Secretary, [<reflink idref="bib58" id="ref67">58</reflink>]). DLL instruction is tailored to the individual needs of each student. For example, teachers use running records that identify the source(s) of students' challenges (e.g., meaning, syntax), serve as the basis for coaching conversations with their teacher leader, and inform the next day's lesson. This individualized approach highlights a child's ever-changing processing and the teacher's observation, adaptation, and deep understanding of that child's strengths (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref68">61</reflink>]).</p> <hd id="AN0180591483-8">Self-Directed Learning</hd> <p>Aligned to Clay's emphasis on independent learning (Clay, [<reflink idref="bib21" id="ref69">21</reflink>]), DLL fosters active learning through opportunities to problem-solve, self-monitor, read books, and write messages with substantial independence and teachers serving as a guide (DeFord, [<reflink idref="bib26" id="ref70">26</reflink>]). DLL aims to support sustainable literacy gains by providing opportunities to employ writing, motivation, and independence (May et al., [<reflink idref="bib48" id="ref71">48</reflink>]).</p> <hd id="AN0180591483-9">Teacher Selection and Training</hd> <p>Rather than a program that can be purchase, DLL is an "investment in the development of teacher expertise" (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref72">61</reflink>], p. 5). High teaching standards begin with selection criteria for teachers: they must hold state-required certification; have at least 3 years of overall teaching experience; and have demonstrated, through teacher evaluation, successful bilingual teaching (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref73">61</reflink>], p. 8). When entering the program, teachers receive training from teacher leaders in a yearlong master's level RR class and then "bridge" to DLL in the second year, during which they must teach lessons and implement all aspects of DLL. Alternatively, if enough teachers are training in DLL at once, a teacher leader may choose to train teachers directly in DLL.</p> <hd id="AN0180591483-10">Personnel</hd> <p>DLL devotes substantial attention to ensuring that the program is implemented in coordination with school systems (May et al., [<reflink idref="bib48" id="ref74">48</reflink>]). It calls for a district-level site coordinator who supports and sustains implementation of DLL within a district and facilitates communication with teacher leaders who support DLL teachers and work with school-level administrators to ensure that DLL is integrated within schools, rather than being an "add-on" program. Within schools, the program calls for coordination among DLL teachers and principals, classroom teachers, and parents.</p> <hd id="AN0180591483-11">Materials</hd> <p>In addition to using the validated <emph>Instrumento de Observación</emph> (<emph>IdO</emph>), a Spanish-language diagnostic assessment used by teachers to measure student skill levels and progress, teachers use several other forms to assess and monitor student progress, including a running record form that allows them to calculate error ratios and accuracy rates and to determine which sources of information (e.g., meaning, syntax, visual) children are using when they make errors or self-corrections. Additional monitoring forms allow teachers to measure change across time in text level and known reading and writing vocabulary. A daily lesson form provides a platform for teachers to plan daily lessons, take notes on their systematic observations during the lesson, and make instructional decisions both in the moment and for future lessons. Teachers also have an extensive collection of books so that students can read a new book during each lesson, and they also employ engaging manipulatives such as word cutouts for story creation and assembly.</p> <hd id="AN0180591483-12">Activities</hd> <p>With these resources as a foundation, DLL's implementation relies on rigorous teacher training that supports quality student instruction through the evidence-based intervention. This instruction is reinforced by ongoing PD and communication across key stakeholders. These central DLL program activities are in the middle panel of Figure 1.</p> <hd id="AN0180591483-13">Training</hd> <p>The research on tutoring supports interventions delivered by trained teachers (Slavin et al., [<reflink idref="bib70" id="ref75">70</reflink>]) and emphasizes intensive training (Planning and Evaluation Service, Office of the Deputy Secretary, [<reflink idref="bib58" id="ref76">58</reflink>]). DLL training requires the completion of yearlong graduate-level coursework. As they train, DLL teachers also teach students and ultimately "demonstrate effective teaching" (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref77">61</reflink>], p. 11). The graduate-level courses focus on theory, an analysis of children's reading and writing behaviors, and the design of individualized lessons tailored to the specific needs of individual children. In addition, DLL teachers receive direct training from DLL teacher leaders who observe student lessons through a one-way glass.</p> <hd id="AN0180591483-14">Quality Student Instruction</hd> <p>With qualified teachers trained to implement an evidence-based intervention, DLL provides quality student instruction. Trained literacy teachers provide daily 30-minute, one-to-one instruction to DLL students for 12–20 weeks, with the goal of meeting the literacy skills of their peers (Reading Recovery Council of North America, [<reflink idref="bib62" id="ref78">62</reflink>]). Prescribed lesson activities include rereading familiar books; reading a recently assigned book while teachers take a running record; working with letters or words using magnetic letters; writing a story; assembling a child's cutup story; and reading a new, strategically selected book (Reading Recovery Council of North America, [<reflink idref="bib62" id="ref79">62</reflink>]). Lessons also include instruction on letter and sound relationships, spelling patterns, comprehension, producing oral language, and increasing vocabularies (Clay, [<reflink idref="bib21" id="ref80">21</reflink>]; Reading Recovery Council of North America, [<reflink idref="bib62" id="ref81">62</reflink>]). Teachers are required to keep records of their student's progress as a basis for future lessons and submit data to their assigned teacher leader to inform coaching discussions.</p> <hd id="AN0180591483-15">Ongoing Professional Development</hd> <p>Ongoing PD and coaching is another cornerstone of DLL often associated with student success (Slavin & Cheung, [<reflink idref="bib69" id="ref82">69</reflink>]; Joyce & Showers, [<reflink idref="bib41" id="ref83">41</reflink>]; Planning and Evaluation Service, Office of the Deputy Secretary, [<reflink idref="bib58" id="ref84">58</reflink>]). Throughout their involvement with DLL, DLL teachers must participate in at least six PD sessions each year, and teachers are expected to participate in behind-the-glass sessions, in which DLL teacher leaders observe their instruction and provide feedback. DLL teacher leaders coach DLL teachers during the school year by discussing strategy, observing instruction, and/or modeling instructional approaches.</p> <hd id="AN0180591483-16">Ongoing Professional Development and Communication across Stakeholders</hd> <p>In addition to an interconnected personnel structure between the district and school, DLL requires that DLL teachers work closely with building administrators, classroom teachers, and the parents of individual students (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref85">61</reflink>]). Rather than an isolated intervention within a school, the program aims to foster communication across stakeholders to support student learning. Implementation research supports this coordinated approach, including the importance of school-level administration support (Bryk et al., [<reflink idref="bib14" id="ref86">14</reflink>]), collaboration (Patterson et al., [<reflink idref="bib57" id="ref87">57</reflink>]), and coordination with classroom teachers (Planning and Evaluation Service, Office of the Deputy Secretary, [<reflink idref="bib58" id="ref88">58</reflink>]). Along with ongoing PD, these communications across stakeholders help support quality student instruction.</p> <hd id="AN0180591483-17">Student Outcomes</hd> <p>When these activities (training, quality instruction, ongoing PD, and communication) are implemented as intended, they support the student literacy outcomes, which appear on the right-hand side of Figure 1. That is, DLL teachers implement an evidence-based one-on-one tutoring intervention in coordination with the school community, which supports students' Spanish-language literacy skill development, including skills such as decoding, vocabulary building, and fluency, to help promote overall reading, writing, and oral proficiency. This focus on heritage language development supports emerging bilingual students and facilitates their later development of English-language skills.</p> <hd id="AN0180591483-18">Prior Evidence of DLL Impacts on Student Outcomes</hd> <p>Two prior randomized controlled trials (RCTs) demonstrated consistently large and positive impacts of DLL on the Spanish-language literacy outcomes of ELs (Borman et al., [<reflink idref="bib13" id="ref89">13</reflink>], [<reflink idref="bib12" id="ref90">12</reflink>]), with both studies receiving the WWC's highest evidence rating—<emph>meets standards without reservations</emph>—in recently published individual study reviews.[<reflink idref="bib2" id="ref91">2</reflink>] The first RCT, conducted in 2011–12 (Borman et al., [<reflink idref="bib12" id="ref92">12</reflink>]), comprised 187 (98 treatment and 89 control) students who were served by 27 previously trained DLL teachers in 19 schools in two districts from two states. The researchers used two student assessments to measure academic progress: the <emph>IdO</emph> (Escamilla et al., [<reflink idref="bib30" id="ref93">30</reflink>])[<reflink idref="bib3" id="ref94">3</reflink>] and the <emph>Logramos</emph> ([<reflink idref="bib45" id="ref95">45</reflink>]).[<reflink idref="bib4" id="ref96">4</reflink>] Treatment students served by DLL outperformed wait-list control students on all <emph>IdO</emph> and <emph>Logramos</emph> measures. Effect sizes (<emph>d</emph>) for <emph>IdO</emph> outcomes ranged from <emph>d</emph><bold></bold>=<bold></bold>0.48 to <emph>d</emph><bold></bold>=<bold></bold>1.24, and effect sizes for the broad-based, standardized <emph>Logramos</emph> outcomes ranged from <emph>d</emph><bold></bold>=<bold></bold>0.36 to <emph>d</emph><bold></bold>=<bold></bold>0.48.</p> <p>The more recent study conducted in 2016–17 (Borman et al., [<reflink idref="bib13" id="ref97">13</reflink>]) included approximately 150 students from 22 schools across three states. In addition to the <emph>IdO</emph> and <emph>Logramos</emph> assessments, this study included the broad-based English-language ITBS. The findings revealed results similar to those in the prior study, with statistically significant student-level impacts of assignment to DLL services on all <emph>IdO</emph> posttest outcomes, ranging from <emph>d</emph><bold></bold>=<bold></bold>0.36 to <emph>d</emph><bold></bold>=<bold></bold>0.97 across the six literacy subtests, and for the four <emph>Logramos</emph> subtests, ranging from <emph>d</emph><bold></bold>=<bold></bold>0.18 to <emph>d</emph><bold></bold>=<bold></bold>0.34. The impacts on Spanish-language outcomes did not translate into statistically significant English literacy impacts as measured by the ITBS, but this result is not surprising given DLL's focus on students' Spanish-language literacy development.</p> <hd id="AN0180591483-19">Method</hd> <p>In this section, we report the final results of the replicated findings across three independent yearly cohorts from 30 schools across 10 districts in five states: Arizona, California, Illinois, Texas, and Wisconsin. With direct replications of the intervention and research methods across three independent first-grade cohorts, this study offers the largest body of evidence to date about the overall Spanish-language and English-language impacts of DLL and the relative consistency of those impacts across replications. In addition to describing the overall impacts, we describe the extent to which those impacts varied across the 30 participating schools.</p> <hd id="AN0180591483-20">Sample</hd> <p>Schools were recruited to participate in this study if they had been implementing DLL for at least 1 year prior to the start of the study. Students were eligible for DLL services if their home language was Spanish and if they were among the lowest scoring students in their schools as determined by the <emph>IdO</emph>. (See the Measures section for a detailed description of all measures.) The combined sample across the three cohorts—2016–17, 2017–18, and 2018–19—included nearly 400 first-graders who were struggling readers, and whose home language was Spanish, nested within the 30 schools. The Institutional Review Board (IRB) of the American Institutes for Research® (FWA00003952, IRB00000436) reviewed the project for compliance with the Common Rule and approved it under determination 85649/04410. The IRB also determined that there were no risks greater than those ordinarily encountered in daily life or during routine testing or instructional activities. Because the conditions stipulated in 45 CFR §46.117(c) were met, a waiver of documentation of consent was granted.</p> <p>All sampled schools offered transitional bilingual programs, which provided Spanish-language regular-classroom reading and language arts instruction to their first-grade ELs. DLL teachers provided supplemental one-to-one Spanish-language literacy tutoring to approximately four students each semester, and each child progressed through the program at an individualized pace, generally lasting 12–20 weeks. All students were native Spanish speakers.</p> <hd id="AN0180591483-21">Study Design</hd> <p>For this study, we employed an RCT design in which the lowest performing first-grade native Spanish-speaking students were randomly assigned to immediate treatment or a delayed, poststudy treatment condition. Our study focused on the fall semester outcomes for students and contrasted the pre-post outcomes for the immediately treated students to the pre-post outcomes of those students whose treatment was delayed until after the study was completed. Following typical DLL practices, during the fall of each year, DLL teachers administered the <emph>IdO</emph>, a Spanish literacy assessment, to all Spanish-speaking students who were identified by their classroom teachers as performing below grade level in literacy in their school. Based on the <emph>IdO</emph> results, the lowest performing students at each campus were identified and randomized to an immediate or a poststudy delayed DLL treatment condition. Those students randomized to the immediately treated condition received DLL services at the beginning of the academic year, and those in the delayed condition received services 12–20 weeks later—as the immediately served students exited the program.</p> <p>The researchers achieved this randomized design by generating a random number for each eligible child (i.e., the lowest performing ELs) at the beginning of the school year. Teachers served those with lower randomly assigned numbers first, and those students with higher numbers remained on a waiting list and were served later, at the conclusion of the study period. As each immediately treated student completed DLL, the teachers served the next child with the lowest randomly assigned number on the waiting list. As students assigned to immediate service during the fall completed DLL, they were tested using three instruments: the <emph>IdO</emph>, <emph>Logramos,</emph> and the ITBS. At the same time, prior to entry into DLL, the next child on the randomly determined waiting list also was administered the three instruments. All students—both immediate and delayed treatment—were administered the <emph>IdO</emph>, <emph>Logramos</emph>, and ITBS assessments at baseline in August or September, prior to being assigned to DLL, and all students were again tested with all three instruments after 12–20 weeks—between November and February. The timeline for program delivery and testing for the immediate- and delayed-treatment students appears in Figure 2.</p> <p>PHOTO (COLOR): Figure 2. Timeline of DLL program delivery and student assessments.</p> <hd id="AN0180591483-22">Measures</hd> <p>The three student assessments were the <emph>IdO</emph> (Escamilla et al., [<reflink idref="bib30" id="ref98">30</reflink>]), the <emph>Logramos</emph> reading achievement test, and the ITBS. The <emph>IdO</emph> is the Spanish-language version of the <emph>Observation Survey of Early Literacy Achievement</emph> (<emph>OS</emph>; Clay, [<reflink idref="bib20" id="ref99">20</reflink>]). The <emph>OS</emph> and the <emph>IdO</emph> measure student performance on the following six subtests: Letter Identification, Ohio Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds in Words, and Text Reading Level (Clay, [<reflink idref="bib22" id="ref100">22</reflink>]). Both <emph>OS</emph>, for English-language students, and <emph>IdO</emph>, for Spanish-language students, are administered as a regular component of the Reading Recovery and DLL programs, respectively. The <emph>OS</emph> and the <emph>IdO</emph> tests are administered at the beginning of the first-grade school year as a screener to determine eligibility for RR and DLL services and after students complete services to measure academic progress. The <emph>IdO</emph> also has been used to examine student outcomes in prior research studies, including Escamilla et al. ([<reflink idref="bib30" id="ref101">30</reflink>]).</p> <p> <emph>IdO</emph> is not a literal translation of the English <emph>OS</emph> but rather a conceptual recreation which measures literacy skill acquisition for young native Spanish speakers. Both <emph>OS</emph> and <emph>IdO</emph> are tools designed for the systematic observation of young children's early literacy competencies. The National Center on Intensive Intervention reviewed <emph>OS</emph>, giving it its highest rating of "convincing evidence."[<reflink idref="bib5" id="ref102">5</reflink>] Construct validity for the <emph>IdO</emph> was evaluated, using data from approximately 500 first-grade students, yielding Cronbach's alphas from <emph>α</emph><bold></bold>= 0.51 to <emph>α</emph><bold></bold>= 0.82 (Escamilla et al., [<reflink idref="bib30" id="ref103">30</reflink>]). Concurrent validity was assessed by comparing the six subtests of <emph>IdO</emph> with a norm-referenced test, <emph>Aprenda: La prueba de logros en Español</emph> (Pearson, 3rd ed.).[<reflink idref="bib6" id="ref104">6</reflink>] Content validity was supported through a series of translations and back translations (Escamilla et al., [<reflink idref="bib30" id="ref105">30</reflink>]).</p> <p> <emph>Logramos</emph> is the Spanish-language version of the ITBS achievement test battery and is available from Riverside Insights.[<reflink idref="bib7" id="ref106">7</reflink>] The items in <emph>Logramos</emph> follow the scope and sequence of Form E of the ITBS and have been translated, when appropriate, from the original English items in Form E of the ITBS. Although the vast majority of items were direct translations of the English test into Spanish, some items required adaptation and replacement of English items in the Spanish version to target the same skills and maintain the underlying psychometrics of the test items.</p> <p>Finally, to assess students' English-language skills, we administered the ITBS. For both the ITBS and <emph>Logramos</emph>, we administered three subtests during each assessment—Vocabulary/Vocabulario, Reading/Lectura, and Language/Lenguaje—which were combined to create an overall total score. Vocabulary/Vocabulario includes 26 items. Reading/Lectura, includes 35 items and assesses literacy in the following domains: Literary text/Texto literario; Informational text/Texto informative; Explicit meaning/Significado explícito; Implicit meaning/Significado implícito; key idea/ideas principals. The Language section of the assessment includes 34 items assessing Spelling/Ortografía, Capitalization/Uso de mayúsculas, Punctuation/Puntuación, and Written Expression/Expresión escrita.</p> <p>The <emph>Logramos</emph> and <emph>ITBS</emph> tests are standardized, norm-referenced, and not overaligned with the intervention and, therefore, provide fair assessments of achievement change for both treatment- and control-group students. Both norm-referenced tests were administered as additional assessments for the purposes of the study and are not typically given to DLL students under routine, nonstudy conditions. All three assessments (<emph>IdO</emph>, <emph>Logramos</emph>, and ITBS posttests) were proctored by a DLL teacher leader or another teacher who did not provide DLL or classroom instruction to the tested students.</p> <p>Beyond these test scores, we collected demographic data for each student through the International Data Evaluation Center ([<reflink idref="bib40" id="ref107">40</reflink>]) at The Ohio State University, which gathers data on all RR and DLL sites and students each year.</p> <hd id="AN0180591483-23">Program Implementation</hd> <p>For this study, the program was implemented under routine conditions that would support any typical DLL implementation. The services provided one-on-one tutoring in Spanish for first-grade Spanish-speaking students receiving regular-classroom literacy instruction in Spanish and experiencing substantial challenges with reading and writing. The program offered daily 30-minute instruction for 12–20 weeks for students performing in the bottom 20–25% of their class, as specified by the DLL protocol (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref108">61</reflink>]). DLL-trained literacy teachers provided intensive, individualized support as a complement to students' regular-classroom instruction, with the goal of raising students' literacy skills to the level of their peers (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref109">61</reflink>]). DLL teachers worked with classroom teachers to avoid providing this supplemental literacy instruction during the regular classroom literacy block. Typically, DLL services were offered at times that did not interrupt students' core academic instruction.</p> <p>As the delayed-treatment students awaited DLL services, they experienced business as usual literacy instruction, which comprised Spanish-language regular classroom literacy instruction and, in some cases, small-group supplemental reading instruction or, in a smaller number of cases, special education services. The DLL teachers monitored the academic progress of all their assigned immediate- and delayed-treatment students throughout the school year, occasionally providing supplemental small-group instruction to their delayed-treatment students. However, only immediate-treatment students received the one-to-one DLL programming during the first 12–20 weeks of the school year.</p> <hd id="AN0180591483-24">Features of the DLL Daily Lessons</hd> <p>Teachers had flexibility to adapt lessons based on individual student needs, but the program required that several activities occur during each student's daily 30-minute lesson. These activities included rereading familiar books; reading a recently assigned book while teachers took a running record; working with letters or words using magnetic letters; writing a story; assembling a child's cutup story; and reading a new, strategically selected book (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref110">61</reflink>]). Lessons provided teaching on letter and sound relationships, spelling patterns, comprehension, producing oral language, and increasing vocabularies (Clay, [<reflink idref="bib21" id="ref111">21</reflink>]; Reading Recovery Council of North America, [<reflink idref="bib61" id="ref112">61</reflink>]). Rather than a prescribed curriculum, however, teachers had the flexibility to adjust lessons based on each child's strengths and needs, offering support and strategic guidance as needed. Further, teachers provided extensive opportunities for student problem solving and self-correction with limited interruption (Clay, [<reflink idref="bib21" id="ref113">21</reflink>]).</p> <p>Individually tailored instruction is a key defining feature of the program. With typical implementations of DLL, the teaching decisions depend on careful observation and an understanding of student progress; thus, program implementation relies on and demands extensive record-keeping. Teachers maintain records of each student—using data to inform their practice—by employing a tool that captures literacy progress before entering the program, when leaving the program, and at the end of the school year (the <emph>IdO</emph>; Reading Recovery Council of North America, [<reflink idref="bib61" id="ref114">61</reflink>]). Teachers also use a variety of tools to routinely assess student progress, including a planning report, daily monitoring tools, and daily assessment forms. The running records form is an example of a crucial assessment tool that allows for responsive instruction. Using this form, DLL teachers calculate error ratios, calculate accuracy rates and self-correction ratios, and determine which sources of information (meaning, syntax, visual) children used when they made errors or self-corrections. They also made notes about fluency, observed strategic processing, and the use of high frequency words. DLL teachers use the running record analysis to determine the focus of the next day's lesson.</p> <hd id="AN0180591483-25">Additional Teacher and Teacher Leader Activities</hd> <p>To provide high-quality, individualized instruction, the program includes rigorous teacher training and standards. The present study evaluated the program as run by already-trained DLL teachers, who had to keep complete records on each student as a basis for instruction, use the <emph>IdO</emph>, communicate with parents and teachers, submit data to the DLL teacher leader, and monitor the progress of students who exited the program (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref115">61</reflink>]).</p> <p>DLL teachers must regularly consult with DLL teacher leaders about those students who are not making satisfactory progress, and DLL teachers receive at least one school visit from a teacher leader annually (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref116">61</reflink>]). In addition to their training role, teacher leaders provide program implementation oversight in schools by regularly collecting program data on student achievement, providing PD to teachers (at least four sessions per year), serving as specialists to support teachers facing struggling readers, supporting the development of school teams, and creating annual site reports for the university training center (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref117">61</reflink>]). Teacher leaders themselves receive continuous support through PD activities, attending three sessions throughout the year: once in the fall and once in the spring at their respective university training centers and once in the summer at an annual program-sponsored institute.</p> <p>Ultimately, DLL extends the RR approach (Sirinides et al., [<reflink idref="bib68" id="ref118">68</reflink>]) to ELs struggling with English literacy by first addressing literacy in their native Spanish language. The program aims to improve early reading and writing outcomes for the lowest achieving ELs through a focused approach that offers the needed boost to catch up to their classmates, which, ultimately, aims to improve long-term academic achievement (Reading Recovery Council of North America, [<reflink idref="bib61" id="ref119">61</reflink>]).</p> <hd id="AN0180591483-26">Analytical Methods</hd> <p></p> <hd id="AN0180591483-27">Intent-to-Treat Impact Analyses</hd> <p>We fit two-level hierarchical linear models (HLMs) to estimate the student-level intent-to-treat impact, or the impact of being assigned to receive DLL services, on each outcome measures. The primary analysis of interest is at the student-level because students were randomly assigned within schools in this study. To account for the nested structure of data—students nested within schools—the model includes random effects for the school-level intercept and the treatment slope. The model adjusts outcomes by using a student-level pretest measure, thus improving model fit and statistically controlling for any potential chance pretest differences between treatment and control groups. Finally, we account for the blocked random assignment by yearly cohort by including cohort fixed effects. The two-level model used for the overall impact analysis is specified as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>Y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn><mi>j</mi></mrow></msub></mrow><mo>+</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn><mi>j</mi></mrow></msub></mrow><mrow><mo stretchy="true">(</mo><mrow><mi>D</mi><mi>L</mi><mrow><msub><mrow><mi>L</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo></mrow><mo>+</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn><mi>j</mi></mrow></msub></mrow><mrow><mo stretchy="true">(</mo><mrow><mi mathvariant="italic">PRETES</mi><mrow><msub><mrow><mi>T</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo></mrow><mo /><mo>+</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn><mi>j</mi></mrow></msub></mrow><mrow><msub><mrow><mrow><mrow><mo stretchy="true">(</mo><mrow><mi mathvariant="italic">COHORT</mi><mrow><msub><mrow><mn>2</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo></mrow></mrow></mrow><mrow /></msub></mrow><mo>+</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn><mi>j</mi></mrow></msub></mrow><mrow><mo stretchy="true">(</mo><mrow><mi mathvariant="italic">COHORT</mi><mrow><msub><mrow><mn>3</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo></mrow><mo /><mo>+</mo><mrow><msub><mrow><mi>r</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib1" id="ref120">1</reflink>)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>0</mn><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi mathvariant="normal">γ</mi></mrow><mrow><mn>00</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mrow><mi>u</mi></mrow><mrow><mn>0</mn><mi>j</mi></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib2" id="ref121">2</reflink>)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi mathvariant="normal">γ</mi></mrow><mrow><mn>10</mn></mrow></msub></mrow><mo>+</mo><mrow><msub><mrow><mi>u</mi></mrow><mrow><mn>1</mn><mi>j</mi></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib3" id="ref122">3</reflink>)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>2</mn><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi mathvariant="normal">γ</mi></mrow><mrow><mn>20</mn></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib4" id="ref123">4</reflink>)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi mathvariant="normal">γ</mi></mrow><mrow><mn>30</mn></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib5" id="ref124">5</reflink>)</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn><mi>j</mi></mrow></msub></mrow><mo>=</mo><mrow><msub><mrow><mi mathvariant="normal">γ</mi></mrow><mrow><mn>40</mn></mrow></msub></mrow></mrow></math> </ephtml> (<reflink idref="bib6" id="ref125">6</reflink>)</p> <p>Beginning with the student-level model (Equation (<reflink idref="bib1" id="ref126">1</reflink>)),</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mi>Y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo /></math> </ephtml> represents the test score of student <emph>i</emph> in school <emph>j</emph>; the model intercept, <emph>β</emph><subs>0</subs><emph><subs>j</subs></emph>, represents the grand mean for control group members with average pretest scores from Cohort 1 (reference group); <emph>β</emph><subs>1</subs><emph><subs>j</subs></emph> is a coefficient representing the impact of assignment to DLL, with <emph>DLL<subs>ij</subs></emph> representing an indicator variable equal to 1 for DLL and 0 for the control group. The coefficient <emph>β</emph><subs>2</subs> is the association between the pretest measure</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mrow><mo stretchy="false">(</mo><mi mathvariant="italic">PRETEST</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo stretchy="false">)</mo></math> </ephtml> and the outcome, which we include to reduce unexplained variance in the outcome measures and improve the precision of our treatment-effect estimates. The indicator variables</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>2</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo></math> </ephtml> and</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>3</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo><mo>,</mo></math> </ephtml> which specify the year in which a student received treatment, account for the fixed effects for each cohort, with Cohort 1 as the referent category. Finally, <emph>r<subs>ij</subs></emph> is the student-specific error term.</p> <p>Equations (<reflink idref="bib2" id="ref127">2</reflink>)–(<reflink idref="bib5" id="ref128">5</reflink>) are the Level 2 school-level models. With Equation (<reflink idref="bib2" id="ref129">2</reflink>), we estimate the extent to which the mean achievement intercept varies across schools. With Equation (<reflink idref="bib3" id="ref130">3</reflink>), we estimate how the impact of assignment to DLL, <emph>β</emph><subs>1</subs><emph><subs>j</subs></emph>, may vary across DLL sites. Because we have no compelling theoretical or practical reason to believe that the cross-level relationships between the student-level premeasure and outcomes and student-level cohort indicators and outcomes should vary across schools, the Level 2 equations estimate the fixed effects of these covariates. Equation (<reflink idref="bib4" id="ref131">4</reflink>) estimates the fixed effect of the pretest covariate, and Equations (<reflink idref="bib5" id="ref132">5</reflink>) and (<reflink idref="bib6" id="ref133">6</reflink>) estimate the fixed effects for the two cohort indicators. In these equations, the mean effects of the pretest and the cohort indicators are predicted solely by their respective school-level intercepts, γ.</p> <p>Most importantly, this two-level model estimates both the overall student-level impact of receiving DLL across schools (<emph>β</emph><subs>1</subs><emph><subs>j</subs></emph>) and the random variation in DLL's impact across schools, which is captured by the variance term <emph>u</emph><subs>1</subs><emph><subs>j</subs></emph> in Equation (<reflink idref="bib3" id="ref134">3</reflink>). The student-level impact speaks to the overall effectiveness of DLL, and the key random site-level component of our model addresses the extent to which DLL impacts may vary across schools. In this respect, the model provides evidence about the overall DLL effect and the extent to which that effect may depend on differences across implementation sites.</p> <p>In addition to this main impact model, we specified an exploratory model with cohort-by-treatment interaction terms to evaluate potential differences in treatment effects by cohort. The Level 1 model is as follows:</p> <p>Graph</p> <p> <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mtable columnalign="left"><mtr><mtd><mrow><msub><mrow><mi>Y</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo>=</mo><mo /><mi>α</mi><mo>+</mo><mo /><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>1</mn></mrow></msub></mrow><mrow><msub><mrow><mo>(</mo><mi mathvariant="italic">DLL</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo>)</mo><mo>+</mo><mrow><msub><mrow><mo /><mi>β</mi></mrow><mrow><mn>2</mn></mrow></msub></mrow><mrow><msub><mrow><mo>(</mo><mi mathvariant="italic">PRETEST</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow><mo>)</mo><mo>+</mo><mo /><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>3</mn></mrow></msub></mrow><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>2</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo><mo>+</mo><mo /><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>4</mn></mrow></msub></mrow><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>3</mn></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo><mo>+</mo><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>5</mn></mrow></msub></mrow><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>2</mn><mi mathvariant="normal">*</mi><mi mathvariant="italic">DLL</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo><mo>+</mo><mo /><mrow><msub><mrow><mi>β</mi></mrow><mrow><mn>6</mn></mrow></msub></mrow><mo stretchy="true">(</mo><mrow><mrow><msub><mrow><mi mathvariant="italic">COHORT</mi><mn>3</mn><mi mathvariant="normal">*</mi><mi mathvariant="italic">DLL</mi></mrow><mrow><mi>i</mi><mi>j</mi></mrow></msub></mrow></mrow><mo stretchy="true">)</mo><mo>+</mo><mo /><mrow><msub><mrow><mi>r</mi></mrow><mrow><mi>i</mi><mi>j</mi><mo /></mrow></msub></mrow></mtd></mtr></mtable></math> </ephtml> (<reflink idref="bib7" id="ref135">7</reflink>)</p> <p>We specified the Level 2 equations for this exploratory model in the same way as our primary analytic model discussed previously. We treated the additional student-level cohort-by-DLL interaction terms as fixed effects at Level 2 of the model. In all models, the pre-post scores were standardized (<emph>M</emph><bold></bold>=<bold></bold>1; <emph>SD</emph><bold></bold>=<bold></bold>0). Therefore, the coefficient for the treatment variable can be interpreted directly as the covariate-adjusted standardized mean difference between treatment and control.</p> <hd id="AN0180591483-28">Cross-Site Distribution of Site Mean DLL Impacts</hd> <p>After estimating the overall impacts of DLL, we further explored the cross-site variation of DLL impacts across the 30 participating schools. Our previously specified multilevel impact analyses employed empirical Bayes (EB) estimates (or shrinkage estimators), which, as noted by Raudenbush and Bryk ([<reflink idref="bib60" id="ref136">60</reflink>]), have the smallest mean squared error for predicting a specific parameter value but are biased toward the overall mean of those parameter values. As a result, the cross-site variance of EB estimates (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mi mathvariant="italic">Var</mi><mo stretchy="false">(</mo><mrow><mrow><mrow><mover accent="true"><mrow><mi>B</mi></mrow><mo>̂</mo></mover></mrow></mrow></mrow></math> </ephtml> <emph> <sups>2</sups> <subs>j</subs> </emph>)) from our previous two-level HLMs with a randomly varying DLL treatment coefficient tended to understate the cross-site variance of true mean program effects (</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mrow><mi>τ</mi></mrow><mrow><mi>B</mi></mrow><mrow><mn>2</mn></mrow></msubsup></mrow></math> </ephtml> ).</p> <p>To address this overshrinkage, we employed methods outlined by Bloom et al. ([<reflink idref="bib8" id="ref137">8</reflink>]), which, for our study, involved multiplying each EB estimate by the ratio of the random effect covariance estimate for the school-specific DLL treatment effect to the variance of the raw EB estimates. Multiplying the deviation of each EB estimate from its grand mean "stretches" these deviations to produce a sample variance that equals the estimated variance of true program effects. Because of the relatively large number of subtests, we restricted these exploratory estimates of the cross-site distribution of DLL intent-to-treat impacts to the overall total score outcomes from each literacy assessment. For <emph>IdO</emph>, we used the total score and for the <emph>Logramos</emph> and ITBS assessments, we report results for each test's ELA total score. We present descriptive statistics, including the mean and standard deviation of each outcome, and produce histograms, which display the distributions of impacts across the 30 school-based sites.</p> <hd id="AN0180591483-29">Results</hd> <p></p> <hd id="AN0180591483-30">Attrition</hd> <p>Attrition in an RCT occurs when the final analytical sample is different from the initial sample that was randomly assigned at baseline. In our case, attrition occurred only when the participating students failed to complete the assigned pretests and posttests. Both overall attrition and differential attrition, between treatment and control groups, can affect the statistical equivalence of the initial randomized sample and can potentially create bias in outcome estimates (What Works Clearinghouse, [<reflink idref="bib79" id="ref138">79</reflink>]). Table 1 summarizes the number of students with complete data included in the final analytical samples, the treatment and control attrition rates, and the overall and differential attrition rates. Relative to WWC's attrition standards, which describe guidelines for determining low, moderate, and high attrition, based on a combination of overall and differential attrition rates, each measure meets the criteria for "low" attrition according to the WWC Version 5.0 standards (What Works Clearinghouse, [<reflink idref="bib79" id="ref139">79</reflink>]). Therefore, the small amount of data attrition from our analytical sample is unlikely to bias our results.</p> <p>Table 1. Data attrition from randomly assigned treatment (<emph>n</emph> = 206) and control (<emph>n</emph> = 195) samples by assessment.</p> <p> <ephtml> <table><thead><tr><td /><td>Analytic sample sizes</td><td>Data attrition rates</td></tr><tr><td /><td>Treatment</td><td>Control</td><td>Treatment (%)</td><td>Control (%)</td><td>Overall (%)</td><td>Differential (%)</td></tr></thead><tbody valign="top"><tr><td><italic>IdO</italic></td><td char=".">197</td><td char=".">178</td><td char=".">4</td><td char=".">9</td><td char=".">6</td><td char=".">4</td></tr><tr><td><italic>Logramos</italic></td><td char=".">177</td><td char=".">160</td><td char=".">14</td><td char=".">18</td><td char=".">16</td><td char=".">4</td></tr><tr><td>ITBS</td><td char=".">167</td><td char=".">156</td><td char=".">19</td><td char=".">20</td><td char=".">19</td><td char=".">1</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note.</emph> IDO: <emph>Instrumento de Observación</emph>; ITBS: Iowa Tests of Basic Skills.</p> <hd id="AN0180591483-31">Baseline Equivalence</hd> <p>We also evaluated baseline equivalence between treatment and control students on all pretest measures from the <emph>IdO</emph>, <emph>Logramos</emph>, and ITBS assessments. This check for baseline equivalence helps ensure that no substantial pretreatment differences exist between the treatment and control students that may compromise analyses of their posttest outcomes. Table 2 shows the treatment and control means and standard deviations and the standardized treatment-control mean differences for each subtest. Following WWC procedures (What Works Clearinghouse, [<reflink idref="bib79" id="ref140">79</reflink>]s), when the difference between treatment and control groups on observable baseline characteristics is greater than 0.25 <emph>SDs</emph>, the two groups are judged to be not equivalent. When the difference in baseline characteristics is greater than 0.05 but less than 0.25 <emph>SDs</emph>, the analysis should include a statistical adjustment for the baseline measure. Differences of less than or equal to 0.05 <emph>SDs</emph> require no statistical adjustment. As shown in Table 2, none of the 15 pretests had a standardized mean difference greater than 0.25. To adjust for baseline differences less than 0.25 <emph>SDs</emph>, though, and to improve the power and precision of our impact analyses, we included the applicable pretest covariate for each analysis of the 15 achievement outcomes.</p> <p>Table 2. Baseline treatment-control equivalence for <emph>Instrumento de Observación</emph> (<emph>IdO</emph>), <emph>Logramos</emph>, and Iowa test of basic skills (ITBS) pretest measures.</p> <p> <ephtml> <table><thead><tr><td /><td>Treatment</td><td>Control</td><td /></tr><tr><td><italic>IdO</italic> pretest measures</td><td><italic>M</italic></td><td><italic>SD</italic></td><td><italic>M</italic></td><td><italic>SD</italic></td><td>Standardized mean difference</td></tr></thead><tbody valign="top"><tr><td>Letter identification</td><td char=".">44.61</td><td char=".">14.54</td><td char=".">45.20</td><td char=".">13.34</td><td char=".">−0.04</td></tr><tr><td>Sounds in words</td><td char=".">24.19</td><td char=".">11.26</td><td char=".">22.90</td><td char=".">10.94</td><td char=".">0.12</td></tr><tr><td>Writing vocabulary</td><td char=".">10.72</td><td char=".">7.45</td><td char=".">10.82</td><td char=".">7.92</td><td char=".">−0.01</td></tr><tr><td>Concepts about print</td><td char=".">11.13</td><td char=".">3.40</td><td char=".">10.69</td><td char=".">3.54</td><td char=".">0.13</td></tr><tr><td>Ohio Word Test</td><td char=".">7.86</td><td char=".">5.74</td><td char=".">7.52</td><td char=".">5.56</td><td char=".">0.06</td></tr><tr><td>Text Reading Level</td><td char=".">1.41</td><td char=".">1.52</td><td char=".">1.10</td><td char=".">1.34</td><td char=".">0.22</td></tr><tr><td>Total score</td><td char=".">463.6</td><td char=".">39.15</td><td char=".">457.39</td><td char=".">41.11</td><td char=".">0.15</td></tr><tr><td /><td>(<italic>n</italic> = 197)</td><td>(<italic>n</italic> = 178)</td><td char="." /></tr><tr><td><italic>Logramos</italic> pretest measures</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>Reading</td><td char=".">145.86</td><td char=".">35.92</td><td char=".">141.38</td><td char=".">40.14</td><td char=".">0.12</td></tr><tr><td>Language</td><td char=".">169.25</td><td char=".">14.34</td><td char=".">169.31</td><td char=".">17.98</td><td char=".">0.00</td></tr><tr><td>Vocabulary</td><td char=".">159.08</td><td char=".">15.58</td><td char=".">159.84</td><td char=".">10.25</td><td char=".">−0.06</td></tr><tr><td>ELA total</td><td char=".">154.43</td><td char=".">37.11</td><td char=".">151.21</td><td char=".">42.14</td><td char=".">0.08</td></tr><tr><td /><td>(<italic>n</italic> = 177)</td><td>(<italic>n</italic> = 160)</td><td char="." /></tr><tr><td>ITBS pretest measures</td><td /><td /><td char="." /></tr><tr><td>Reading</td><td char=".">120.41</td><td char=".">38.29</td><td char=".">116.57</td><td char=".">42.54</td><td char=".">0.09</td></tr><tr><td>Language</td><td char=".">123.09</td><td char=".">13.88</td><td char=".">123.04</td><td char=".">10.15</td><td char=".">0.00</td></tr><tr><td>Vocabulary</td><td char=".">123.75</td><td char=".">16.37</td><td char=".">122.74</td><td char=".">16.60</td><td char=".">0.06</td></tr><tr><td>ELA total</td><td char=".">115.44</td><td char=".">36.95</td><td char=".">111.58</td><td char=".">40.96</td><td char=".">0.10</td></tr><tr><td /><td>(<italic>n</italic> = 167)</td><td>(<italic>n</italic> = 156)</td><td char="." /></tr></tbody></table> </ephtml> </p> <hd id="AN0180591483-32">Impact Analyses</hd> <p>The results are presented separately for each literacy outcome. For each outcome, we first estimated a main impact model and then estimated a second model, which assesses any potential cohort-by-treatment interaction effects. As noted earlier, treatment coefficients can be interpreted as the standardized mean differences (or effect sizes) between treatment and control students on the three literacy assessments.</p> <hd id="AN0180591483-33">Instrumento de Observación</hd> <p>Table 3 shows the estimated DLL effects on the total score and all six <emph>IdO</emph> subtest outcomes: Letter Identification, Ohio Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds and Text Reading Sounds, and Text Reading Level. The results for the main impact analyses, labeled as Model 1 for each <emph>IdO</emph> outcome shown in Table 3, reveal statistically significant student-level impacts of assignment to DLL services on all posttest outcomes, ranging from <emph>d</emph><bold></bold>=<bold></bold>0.36 for the Sounds in Words outcome to <emph>d</emph><bold></bold>=<bold></bold>0.97 for the Text Reading Level subtest. The results for each outcome labeled Model 2 evaluate the cohort-by-treatment interaction effects. As suggested by the nonsignificant cohort-by-treatment interactions for each outcome, these exploratory analyses revealed no evidence to suggest that there were differences for the DLL impacts across the three independent yearly student cohorts.</p> <p>Table 3. Multilevel estimates of Intention-to-Treat impacts on <emph>Instrumento de Observación</emph> outcomes.</p> <p> <ephtml> <table><thead><tr><td /><td>Letter identification</td><td>Sounds in words</td><td>Writing vocabulary</td><td>Concepts about print</td><td>Ohio Word Test</td><td>Text Reading Level</td><td>Total score</td></tr><tr><td>Fixed effects</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td></tr></thead><tbody valign="top"><tr><td>Intercept</td><td char=".">−0.40***</td><td char=".">−0.38**</td><td char=".">−0.21</td><td char=".">−0.25</td><td char=".">−0.52***</td><td char=".">−0.59***</td><td char=".">−0.49***</td><td char=".">−0.49***</td><td char=".">−0.26*</td><td char=".">−0.30*</td><td char=".">−0.49***</td><td char=".">−0.57***</td><td char=".">−0.50***</td><td char=".">−0.54***</td></tr><tr><td /><td char=".">(0.10)</td><td char=".">(0.12)</td><td char=".">(0.12)</td><td char=".">(0.13)</td><td char=".">(0.10)</td><td char=".">(0.11)</td><td char=".">(0.09)</td><td char=".">(0.11)</td><td char=".">(0.12)</td><td char=".">(0.13)</td><td char=".">(0.08)</td><td char=".">(0.10)</td><td char=".">(0.09)</td><td char=".">(0.10)</td></tr><tr><td>Treatment</td><td char=".">0.72***</td><td char=".">0.67***</td><td char=".">0.36**</td><td char=".">0.42*</td><td char=".">0.87***</td><td char=".">0.99***</td><td char=".">0.87***</td><td char=".">0.87***</td><td char=".">0.49***</td><td char=".">0.57***</td><td char=".">0.97***</td><td char=".">1.13***</td><td char=".">0.91***</td><td char=".">0.98***</td></tr><tr><td /><td char=".">(0.12)</td><td char=".">(0.16)</td><td char=".">(0.12)</td><td char=".">(0.17)</td><td char=".">(0.11)</td><td char=".">(0.15)</td><td char=".">(0.08)</td><td char=".">(0.12)</td><td char=".">(0.12)</td><td char=".">(0.16)</td><td char=".">(0.11)</td><td char=".">(0.15)</td><td char=".">(0.10)</td><td char=".">(0.14)</td></tr><tr><td>Pretest</td><td char=".">0.52***</td><td char=".">0.52***</td><td char=".">0.50***</td><td char=".">0.50***</td><td char=".">0.40***</td><td char=".">0.40***</td><td char=".">0.38***</td><td char=".">0.38***</td><td char=".">0.44***</td><td char=".">0.44***</td><td char=".">0.39***</td><td char=".">0.39***</td><td char=".">0.54***</td><td char=".">0.54***</td></tr><tr><td /><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td><td char=".">(0.04)</td></tr><tr><td>Cohort 2</td><td char=".">0.02</td><td char=".">−0.03</td><td char=".">0.04</td><td char=".">0.11</td><td char=".">0.07</td><td char=".">0.15</td><td char=".">0.03</td><td char=".">0.03</td><td char=".">0.02</td><td char=".">0.03</td><td char=".">−0.02</td><td char=".">0.09</td><td char=".">0.03</td><td char=".">0.09</td></tr><tr><td /><td char=".">(0.09)</td><td char=".">(0.13)</td><td char=".">(0.09)</td><td char=".">(0.14)</td><td char=".">(0.09)</td><td char=".">(0.12)</td><td char=".">(0.09)</td><td char=".">(0.12)</td><td char=".">(0.09)</td><td char=".">(0.14)</td><td char=".">(0.09)</td><td char=".">(0.13)</td><td char=".">(0.07)</td><td char=".">(0.11)</td></tr><tr><td>Cohort 3</td><td char=".">−0.02</td><td char=".">−0.03</td><td char=".">−0.02</td><td char=".">0.02</td><td char=".">0.06</td><td char=".">0.22</td><td char=".">0.02</td><td char=".">−0.01</td><td char=".">−0.03</td><td char=".">0.15</td><td char=".">−0.04</td><td char=".">0.16</td><td char=".">−0.00</td><td char=".">0.07</td></tr><tr><td /><td char=".">(0.11)</td><td char=".">(0.17)</td><td char=".">(0.12)</td><td char=".">(0.19)</td><td char=".">(0.11)</td><td char=".">(0.16)</td><td char=".">(0.11)</td><td char=".">(0.16)</td><td char=".">(0.12)</td><td char=".">(0.18)</td><td char=".">(0.11)</td><td char=".">(0.16)</td><td char=".">(0.10)</td><td char=".">(0.14)</td></tr><tr><td>Treatment * Cohort 2</td><td char="." /><td char=".">0.11</td><td char="." /><td char=".">−0.13</td><td char="." /><td char=".">−0.16</td><td char="." /><td char=".">−0.01</td><td char="." /><td char=".">−0.03</td><td char="." /><td char=".">−0.20</td><td char="." /><td char=".">−0.11</td></tr><tr><td /><td char="." /><td char=".">(0.18)</td><td char="." /><td char=".">(0.19)</td><td char="." /><td char=".">(0.17)</td><td char="." /><td char=".">(0.17)</td><td char="." /><td char=".">(0.19)</td><td char="." /><td char=".">(0.18)</td><td char="." /><td char=".">(0.15)</td></tr><tr><td>Treatment * Cohort 3</td><td char="." /><td char=".">0.02</td><td char="." /><td char=".">−0.06</td><td char="." /><td char=".">−0.31</td><td char="." /><td char=".">0.04</td><td char="." /><td char=".">−0.31</td><td char="." /><td char=".">−0.39</td><td char="." /><td char=".">−0.13</td></tr><tr><td /><td char="." /><td char=".">(0.23)</td><td char="." /><td char=".">(0.24)</td><td char="." /><td char=".">(0.21)</td><td char="." /><td char=".">(0.20)</td><td char="." /><td char=".">(0.24)</td><td char="." /><td char=".">(0.22)</td><td char="." /><td char=".">(0.19)</td></tr><tr><td>Random effects</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>School intercept variance</td><td char=".">0.14***</td><td char=".">0.14***</td><td char=".">0.22***</td><td char=".">0.23***</td><td char=".">0.14***</td><td char=".">0.15***</td><td char=".">0.10***</td><td char=".">0.10***</td><td char=".">0.20***</td><td char=".">0.20***</td><td char=".">0.04</td><td char=".">0.04</td><td char=".">0.11***</td><td char=".">0.11***</td></tr><tr><td>School treatment impact variance</td><td char=".">0.24***</td><td char=".">0.23***</td><td char=".">0.23***</td><td char=".">0.26***</td><td char=".">0.19***</td><td char=".">0.18***</td><td char=".">0.01</td><td char=".">0.01</td><td char=".">0.25***</td><td char=".">0.23***</td><td char=".">0.15**</td><td char=".">0.16</td><td char=".">0.19***</td><td char=".">0.19***</td></tr><tr><td>School intercept/impact correlation</td><td char=".">−0.17</td><td char=".">−0.16</td><td char=".">−0.22*</td><td char=".">−0.24*</td><td char=".">−0.06</td><td char=".">−0.06</td><td char=".">0.02</td><td char=".">0.02</td><td char=".">−0.21</td><td char=".">−0.20</td><td char=".">−0.06</td><td char=".">−0.07</td><td char=".">−0.10</td><td char=".">−0.10</td></tr><tr><td>Student-level residual variance</td><td char=".">0.52***</td><td char=".">0.52***</td><td char=".">0.60***</td><td char=".">0.60***</td><td char=".">0.47***</td><td char=".">0.47***</td><td char=".">0.50***</td><td char=".">0.50***</td><td char=".">0.60***</td><td char=".">0.60***</td><td char=".">0.54***</td><td char=".">0.54***</td><td char=".">0.36***</td><td char=".">0.37***</td></tr></tbody></table> </ephtml> </p> <ulist> <item>2 <emph>Note.</emph> Student <emph>n</emph> = 375; school <emph>n</emph> = 30. Standard errors presented in parentheses.</item> <item>3 *<emph>p</emph> < 0.05; **<emph>p</emph> < 0.01; <emph>***p</emph> < 0.001. (<reflink idref="bib1" id="ref141">1</reflink>) Refers to Model 1. (<reflink idref="bib2" id="ref142">2</reflink>) Refers to Model 2.</item> </ulist> <p>Finally, in the bottom panel of Table 3, we summarize the random effects, including the variance of the school-level intercept,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>τ</mi></mrow><mrow><mn>2</mn></mrow></msup></mrow><mo>;</mo></math> </ephtml> the school-level variance of the DLL treatment effect, ζ<sups>2</sups>; the correlation, <emph>ρ</emph>, between the school intercept and treatment effect; and the unexplained student-level residual variance,</p> <p>Graph</p> <p> <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>σ</mi></mrow><mrow><mn>2</mn></mrow></msup></mrow><mo>.</mo></math> </ephtml> The results for the school-level variance of the DLL impacts reveal statistically significant treatment-effect variability across the 30 schools for all <emph>IdO</emph> outcomes, except for the Concepts About Print subtest. For instance, the results for the <emph>IdO</emph> total score at the far right-side of Table 3 show an average <emph>d</emph><bold></bold>=<bold></bold>0.91, with a random effect covariance estimate for the school-level impacts of 0.19. Taking the square-root of this covariance estimate yields a standard deviation of 0.44.</p> <p>To further explore this variability in DLL impacts across the 30 participating schools, we used methods outlined by Bloom et al. ([<reflink idref="bib8" id="ref143">8</reflink>]) to describe the estimated variance of true DLL program effects. We restricted our estimates of the cross-site distribution of DLL intent-to-treat impacts to the total score for the <emph>IdO</emph> assessment, which is equivalent to <emph>d</emph><bold></bold>=<bold></bold>0.91 with an adjusted standard deviation of the school-specific DLL impacts of 0.58 <emph>SDs</emph>. Figure 3 is a histogram of the adjusted EB estimates of school-specific impact estimates for the 30 schools included in the overall analysis of <emph>IdO</emph> outcomes. Of the 30 schools, 29 schools (97%) have positive impact estimates, and 12 schools (40%) have impact estimates greater than 1 <emph>SD</emph>. The figure shows only two relative outliers, one school with an estimated positive impact of approximately <emph>d</emph><bold></bold>=<bold></bold>2.60 and one with a negative impact of <emph>d</emph><bold></bold>=<bold></bold>−0.50.</p> <p>PHOTO (COLOR): Figure 3. Adjusted empirical Bayes estimates of School-Specific impacts of DLL on mean IdO outcomes.</p> <hd id="AN0180591483-34">Logramos</hd> <p>Students who received DLL services outperformed those who had not yet received services on all <emph>Logramos</emph> subtests, and the associated effect sizes were statistically significant, ranging from <emph>d</emph> = 0.26 to <emph>d</emph> = 0.57. Model 1 results shown in Table 4 indicate the average DLL effects for the four <emph>Logramos</emph> outcomes: Reading, Language, Vocabulary, and ELA total. As was the case for the <emph>IdO</emph> outcomes, the exploratory analyses of each <emph>Logramos</emph> outcome labeled as Model 2 suggest no evidence of statistically significant differences for the DLL impacts across the three independent yearly student cohorts.</p> <p>Table 4. Multilevel estimates of Intention-to-Treat impact on <emph>Logramos</emph> outcomes.</p> <p> <ephtml> <table><thead><tr><td>Fixed effects</td><td>Reading</td><td>Language</td><td>Vocabulary</td><td>English/Language Arts total</td></tr><tr><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td></tr></thead><tbody valign="top"><tr><td>Intercept</td><td char=".">−0.23</td><td char=".">−0.25</td><td char=".">−0.23</td><td char=".">−0.22</td><td char=".">−0.20</td><td char=".">−0.30</td><td char=".">−0.20</td><td char=".">−0.19</td></tr><tr><td /><td char=".">(0.17)</td><td char=".">(0.18)</td><td char=".">(0.19)</td><td char=".">(0.19)</td><td char=".">(0.19)</td><td char=".">(0.21)</td><td char=".">(0.11)</td><td char=".">(0.12)</td></tr><tr><td>Treatment</td><td char=".">0.34<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">0.36<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">0.20***</td><td char=".">0.18</td><td char=".">0.30<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">0.48<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">0.25***</td><td char=".">0.24<xref ref-type="table-fn" rid="tfn5">*</xref></td></tr><tr><td /><td char=".">(0.11)</td><td char=".">(0.16)</td><td char=".">(0.06)</td><td char=".">(0.10)</td><td char=".">(0.14)</td><td char=".">(0.19)</td><td char=".">(0.07)</td><td char=".">(0.11)</td></tr><tr><td>Pretest</td><td char=".">0.39***</td><td char=".">0.39***</td><td char=".">0.07<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">0.08<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">0.05</td><td char=".">0.05</td><td char=".">0.63***</td><td char=".">0.63***</td></tr><tr><td /><td char=".">(0.06)</td><td char=".">(0.06)</td><td char=".">(0.03)</td><td char=".">(0.03)</td><td char=".">(0.05)</td><td char=".">(0.05)</td><td char=".">(0.04)</td><td char=".">(0.04)</td></tr><tr><td>Cohort 2</td><td char=".">0.00</td><td char=".">0.01</td><td char=".">0.10</td><td char=".">0.07</td><td char=".">0.01</td><td char=".">0.19</td><td char=".">0.04</td><td char=".">−0.00</td></tr><tr><td /><td char=".">(0.11)</td><td char=".">(0.15)</td><td char=".">(0.08)</td><td char=".">(0.10)</td><td char=".">(0.11)</td><td char=".">(0.16)</td><td char=".">(0.08)</td><td char=".">(0.11)</td></tr><tr><td>Cohort 3</td><td char=".">−0.02</td><td char=".">0.09</td><td char=".">0.01</td><td char=".">0.09</td><td char=".">−0.12</td><td char=".">−0.02</td><td char=".">−0.07</td><td char=".">0.04</td></tr><tr><td /><td char=".">(0.15)</td><td char=".">(0.23)</td><td char=".">(0.11)</td><td char=".">(0.15)</td><td char=".">(0.15)</td><td char=".">(0.24)</td><td char=".">(0.11)</td><td char=".">(0.16)</td></tr><tr><td>Treatment<xref ref-type="table-fn" rid="tfn5">*</xref> Cohort 2</td><td char="." /><td char=".">−0.00</td><td char="." /><td char=".">0.07</td><td char="." /><td char=".">−0.31</td><td char="." /><td char=".">0.07</td></tr><tr><td /><td char="." /><td char=".">(0.19)</td><td char="." /><td char=".">(0.13)</td><td char="." /><td char=".">(0.20)</td><td char="." /><td char=".">(0.14)</td></tr><tr><td>Treatment<xref ref-type="table-fn" rid="tfn5">*</xref> Cohort 3</td><td char="." /><td char=".">−0.17</td><td char="." /><td char=".">−0.13</td><td char="." /><td char=".">−0.17</td><td char="." /><td char=".">−0.16</td></tr><tr><td /><td char="." /><td char=".">(0.27)</td><td char="." /><td char=".">(0.18)</td><td char="." /><td char=".">(0.28)</td><td char="." /><td char=".">(0.19)</td></tr><tr><td>Random effects</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>School intercept variance</td><td char=".">0.58***</td><td char=".">0.58***</td><td char=".">0.97***</td><td char=".">0.95***</td><td char=".">0.85***</td><td char=".">0.89***</td><td char=".">0.23***</td><td char=".">0.21***</td></tr><tr><td>School treatment impact variance</td><td char=".">0.13<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">0.13<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">0.00</td><td char=".">0.00</td><td char=".">0.36<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">0.41</td><td char=".">0.04</td><td char=".">0.03</td></tr><tr><td>School intercept/ impact correlation</td><td char=".">−0.27<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">−0.27<xref ref-type="table-fn" rid="tfn5">*</xref>*</td><td char=".">0.00</td><td char=".">0.01</td><td char=".">−0.56***</td><td char=".">−0.60</td><td char=".">−0.09<xref ref-type="table-fn" rid="tfn5">*</xref></td><td char=".">−0.08<xref ref-type="table-fn" rid="tfn5">*</xref></td></tr><tr><td>Student-level residual variance</td><td char=".">0.61***</td><td char=".">0.62***</td><td char=".">0.29***</td><td char=".">0.29***</td><td char=".">0.65***</td><td char=".">0.65***</td><td char=".">0.32***</td><td char=".">0.32***</td></tr></tbody></table> </ephtml> </p> <ulist> <item>4 <emph>Note.</emph> Student <emph>n</emph> = 337; school <emph>n</emph> = 30. Standard errors presented in parentheses.</item> <item>5 *<emph>p</emph> < 0.05; **<emph>p</emph> < 0.01; <emph>***p</emph> < 0.001. (<reflink idref="bib1" id="ref144">1</reflink>) Refers to Model 1. (<reflink idref="bib2" id="ref145">2</reflink>) Refers to Model 2.</item> </ulist> <p>The random effects, including the school-level variance of the DLL treatment effect, ζ<sups>2</sups>, are shown in the bottom panel of Table 4. These outcomes reveal statistically significant treatment-effect variability across the 30 schools for two of the four <emph>Logramos</emph> outcomes: Reading and Vocabulary. For instance, the results for the <emph>Logramos</emph> Vocabulary subtest yielded an estimated average effect size of <emph>d</emph><bold></bold>=<bold></bold>0.30, with a random effect covariance estimate for the school-level impacts of 0.36, which is equivalent to 0.60 <emph>SDs</emph>.</p> <p>We calculated adjusted EB estimates of the school-specific impacts on the composite <emph>Logramos</emph> total score for the 30 schools. Figure 4 provides a graphical representation of this school-level variability in DLL impacts on the <emph>Logramos</emph> total score, which was equivalent to <emph>d</emph><bold></bold>=<bold></bold>0.25 with an adjusted standard deviation of the school-specific DLL impacts of 0.23. As with the <emph>IdO</emph> outcome, 29 of the 30 schools (97%) had positive impacts on the <emph>Logramos</emph> Total Language Arts outcome, ranging from <emph>d</emph><bold></bold>=<bold></bold>−0.04 to <emph>d</emph><bold></bold>=<bold></bold>0.94.</p> <p>PHOTO (COLOR): Figure 4. Adjusted empirical Bayes estimates of School-Specific impacts of DLL on mean Logramos outcomes.</p> <hd id="AN0180591483-35">ITBS</hd> <p>Students assigned to DLL scored somewhat higher on all ITBS subtests than those assigned to the control group as demonstrated by positive effect sizes; however, the effect sizes were not statistically significant. Model 1 in Table 5 shows the average impacts of assignment to DLL on each ITBS outcome: Reading, Language, Vocabulary, and ELA total. Similar to the Spanish-language outcomes, the results from Model 2 reveal no statistically significant cohort-by-treatment interaction effects. The random effects shown at the bottom of Table 5 do not reveal substantial treatment-effect variability across the 30 schools. The results suggested no statistically significant treatment effect heterogeneity for the ITBS outcomes.</p> <p>Table 5. Multilevel estimates of Intention-to-Treat impact on <emph>Iowa test of Basic skills</emph> outcomes.</p> <p> <ephtml> <table><thead><tr><td>Fixed Effects</td><td>Reading</td><td>Language</td><td>Vocabulary</td><td>English/Language Arts total</td></tr><tr><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td><td>(1)</td><td>(2)</td></tr></thead><tbody valign="top"><tr><td>Intercept</td><td char=".">0.10</td><td char=".">0.11</td><td char=".">−0.07</td><td char=".">−0.06</td><td char=".">0.08</td><td char=".">0.07</td><td char=".">0.12</td><td char=".">0.12</td></tr><tr><td /><td char=".">(0.15)</td><td char=".">(0.16)</td><td char=".">(0.20)</td><td char=".">(0.21)</td><td char=".">(0.19)</td><td char=".">(0.21)</td><td char=".">(0.13)</td><td char=".">(0.14)</td></tr><tr><td>Treatment</td><td char=".">0.06</td><td char=".">0.03</td><td char=".">0.06</td><td char=".">0.06</td><td char=".">0.07</td><td char=".">0.08</td><td char=".">0.02</td><td char=".">0.02</td></tr><tr><td /><td char=".">(0.08)</td><td char=".">(0.13)</td><td char=".">(0.06)</td><td char=".">(0.10)</td><td char=".">(0.13)</td><td char=".">(0.19)</td><td char=".">(0.07)</td><td char=".">(0.12)</td></tr><tr><td>Pretest</td><td char=".">0.35***</td><td char=".">0.35***</td><td char=".">0.02</td><td char=".">0.02</td><td char=".">−0.01</td><td char=".">−0.01</td><td char=".">0.37***</td><td char=".">0.37***</td></tr><tr><td /><td char=".">(0.06)</td><td char=".">(0.06)</td><td char=".">(0.03)</td><td char=".">(0.03)</td><td char=".">(0.05)</td><td char=".">(0.05)</td><td char=".">(0.05)</td><td char=".">(0.05)</td></tr><tr><td>Cohort 2</td><td char=".">−0.27<xref ref-type="table-fn" rid="tfn7">*</xref>*</td><td char=".">−0.29<xref ref-type="table-fn" rid="tfn7">*</xref></td><td char=".">−0.08</td><td char=".">−0.09</td><td char=".">−0.19</td><td char=".">−0.12</td><td char=".">−0.30<xref ref-type="table-fn" rid="tfn7">*</xref>*</td><td char=".">−0.29<xref ref-type="table-fn" rid="tfn7">*</xref></td></tr><tr><td /><td char=".">(0.10)</td><td char=".">(0.14)</td><td char=".">(0.08)</td><td char=".">(0.10)</td><td char=".">(0.11)</td><td char=".">(0.17)</td><td char=".">(0.09)</td><td char=".">(0.13)</td></tr><tr><td>Cohort 3</td><td char=".">−0.24</td><td char=".">−0.28</td><td char=".">−0.03</td><td char=".">−0.02</td><td char=".">−0.05</td><td char=".">−0.22</td><td char=".">−0.23</td><td char=".">−0.24</td></tr><tr><td /><td char=".">(0.14)</td><td char=".">(0.19)</td><td char=".">(0.11)</td><td char=".">(0.15)</td><td char=".">(0.16)</td><td char=".">(0.24)</td><td char=".">(0.13)</td><td char=".">(0.17)</td></tr><tr><td>Treatment<xref ref-type="table-fn" rid="tfn7">*</xref> Cohort 2</td><td char="." /><td char=".">0.02</td><td char="." /><td char=".">0.01</td><td char="." /><td char=".">−0.13</td><td char="." /><td char=".">−0.00</td></tr><tr><td /><td char="." /><td char=".">(0.18)</td><td char="." /><td char=".">(0.13)</td><td char="." /><td char=".">(0.21)</td><td char="." /><td char=".">(0.16)</td></tr><tr><td>Treatment<xref ref-type="table-fn" rid="tfn7">*</xref> Cohort 3</td><td char="." /><td char=".">0.08</td><td char="." /><td char=".">−0.02</td><td char="." /><td char=".">0.25</td><td char="." /><td char=".">0.02</td></tr><tr><td /><td char="." /><td char=".">(0.23)</td><td char="." /><td char=".">(0.17)</td><td char="." /><td char=".">(0.28)</td><td char="." /><td char=".">(0.20)</td></tr><tr><td>Random effects</td><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /><td char="." /></tr><tr><td>School intercept variance</td><td char=".">0.43***</td><td char=".">0.43***</td><td char=".">1.12***</td><td char=".">1.12***</td><td char=".">0.81***</td><td char=".">0.84***</td><td char=".">0.33***</td><td char=".">0.33***</td></tr><tr><td>School treatment impact variance</td><td char=".">0.00</td><td char=".">0.00</td><td char=".">0.01</td><td char=".">0.01</td><td char=".">0.28<xref ref-type="table-fn" rid="tfn7">*</xref></td><td char=".">0.32<xref ref-type="table-fn" rid="tfn7">*</xref></td><td char=".">0.00</td><td char=".">0.00</td></tr><tr><td>School intercept/impact correlation</td><td char=".">0.01</td><td char=".">0.01</td><td char=".">−0.12***</td><td char=".">−0.12***</td><td char=".">−0.47***</td><td char=".">−0.50***</td><td char=".">0.02</td><td char=".">0.02</td></tr><tr><td>Student-level residual variance</td><td char=".">0.48***</td><td char=".">0.49***</td><td char=".">0.26***</td><td char=".">0.26***</td><td char=".">0.64***</td><td char=".">0.64***</td><td char=".">0.40***</td><td char=".">0.40***</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 <emph>Note.</emph> Student <emph>n</emph> = 323; school <emph>n</emph> = 30. Standard errors presented in parentheses.</item> <item>7 *<emph>p</emph> < 0.05; **<emph>p</emph> < 0.01; <emph>***p</emph> < 0.001. (<reflink idref="bib1" id="ref146">1</reflink>) Refers to Model 1. (<reflink idref="bib2" id="ref147">2</reflink>) Refers to Model 2.</item> </ulist> <p>Although the ITBS outcomes revealed no statistically significant treatment effects or cross-site treatment effect variability, we calculated adjusted EB estimates of the school-specific impacts for the 30 schools to examine their variability descriptively. Figure 5 shows this school-level variability in DLL impacts on the ITBS total score composite outcome, which was equivalent to <emph>d</emph><bold></bold>=<bold></bold>0.02 with an adjusted standard deviation of the school-specific DLL impacts of 0.04 <emph>SDs</emph>. Of the 30 schools, 24 schools (80%) had positive impact estimates. Overall, the estimated impacts ranged from <emph>d</emph><bold></bold>=<bold></bold>−0.08 to <emph>d</emph><bold></bold>=<bold></bold>0.08. Thus, no schools had impacts greater than 0.10 <emph>SDs</emph>, and no schools had negative impacts less than 0.10 <emph>SDs</emph>.</p> <p>PHOTO (COLOR): Figure 5. Adjusted empirical Bayes estimates of School-Specific impacts of DLL on mean ITBS outcomes.</p> <hd id="AN0180591483-36">Multiple Comparisons Corrections</hd> <p>Because we tested the statistical significance of multiple outcomes within one literacy domain, we applied the Benjamini-Hochberg method to adjust for multiple comparisons to control for a false-discovery rate.[<reflink idref="bib8" id="ref148">8</reflink>] The results suggest that after adjusting for multiple comparisons across the seven <emph>IdO</emph> and four <emph>Logramos</emph> tests, the statistically significant results held. In other words, our analyses indicated that the treated students served by DLL outperformed the control students on all Spanish assessments (i.e., all <emph>IdO</emph> and <emph>Logramos</emph> measures). Of course, because there were no statistically significant impacts across the four ITBS measures, there was no need to make the corresponding adjustments to these outcomes.</p> <hd id="AN0180591483-37">Discussion</hd> <p>Our analyses indicate that treatment students served by DLL outperformed control students on all Spanish-language <emph>IdO</emph> and <emph>Logramos</emph> measures. We found statistically significant student-level impacts of assignment to DLL services on all <emph>IdO</emph> posttest outcomes, ranging from <emph>d</emph><bold></bold>=<bold></bold>0.36 for the Sounds in Words outcome to <emph>d</emph><bold></bold>=<bold></bold>0.97 for the Text Reading subtest. The DLL effect sizes for the <emph>Logramos</emph> posttests ranged from <emph>d</emph> = 0.20 to <emph>d</emph> = 0.34. No analyses of interaction effects suggest that the impacts of DLL depend on the year of instructional delivery: 2016–17, 2017–18, or 2018–19. Further, our analyses of the school-to-school differences in DLL impacts consistently revealed that the overall Spanish-language impacts are larger than they are variable, with an <emph>IdO</emph> total score outcome equivalent to <emph>d</emph><bold></bold>=<bold></bold>0.91 and a standard deviation of 0.58 and an average <emph>Logramos</emph> outcome equivalent to <emph>d</emph><bold></bold>=<bold></bold>0.25 with a standard deviation of 0.23. Although 80% of the schools also realized positive impacts on the ITBS assessment, the highly positive Spanish-language outcomes did not appear to translate immediately into statistically significant English-language impacts.</p> <hd id="AN0180591483-38">Interpreting the Magnitudes of the DLL Impacts</hd> <p>Similar to the sister English-language version (RR), DLL demonstrates strong impacts for the sampled students across many dimensions of literacy in reliable ways that are replicated across years and from teacher to teacher. Comparing our results to those from the recent large-scale, multicohort evaluation of RR reported by Sirinides et al. ([<reflink idref="bib68" id="ref149">68</reflink>]), the magnitudes of the impacts are quite similar. The RR effect size for the overall score from the <emph>OS</emph> was nearly one full standard deviation and similarly DLL's impact on the <emph>IdO</emph> total score was equivalent to 0.91 standard deviations. RR impacts on the English-language ITBS, which were somewhat greater than one-third of a standard deviation, were comparable to those found for the Spanish-language <emph>Logramos</emph> assessment, which ranged from 0.20 to 0.34. Based on the first-year results from the national study of RR, May et al. ([<reflink idref="bib48" id="ref150">48</reflink>]) indicated that 81% of the 147 participating RR schools had positive English-language ITBS impacts on the participating treatment students. Although our sample was considerably smaller, with only 30 schools, all but one school (97%) in the current study of DLL had positive impacts on the Spanish-language <emph>Logramos</emph> assessment.</p> <p>Other benchmarks may provide additional context to understand the impacts of DLL. For instance, Kraft's ([<reflink idref="bib44" id="ref151">44</reflink>]) review of nearly 250 recent RCTs of education interventions suggested that a "large" effect size is 0.20 or greater. More specifically, by examining the distribution of impacts across those 242 RCTs, the impacts of DLL on the <emph>Logramos</emph> and <emph>IdO</emph> total scores are, respectively, above the 75th percentile and close to the 99th percentile of that distribution. When comparing the largest impact of 0.97 standard deviation units to other more intuitive benchmarks provided by Hill et al. ([<reflink idref="bib39" id="ref152">39</reflink>]), the DLL effect is equivalent to a learning advantage relative to controls that is approximately equal to the amount of reading achievement growth experienced by the typical first-grade student during a full school year. With effect sizes ranging from <emph>d</emph><bold></bold>=<bold></bold>0.20 to <emph>d</emph><bold></bold>=<bold></bold>0.34 on the <emph>Logramos</emph> assessment, the impacts of DLL were equivalent to approximately one fifth to more than one third of the achievement growth found for the typical first-grade student across a full year of school. Finally, compared with typical supplemental Title I programs, which were reviewed in a meta-analysis by Borman and D'Agostino ([<reflink idref="bib10" id="ref153">10</reflink>]), which frequently serve as the funding source for DLL, the DLL impacts on the <emph>Logramos</emph> assessment are more than two to three times larger, and the <emph>IdO</emph> impacts are greater than nine times larger than those observed in supplemental Title I programs. Indeed, as a supplemental intervention spanning only 12–20 weeks, DLL appears to produce impressive impacts of a magnitude not typically seen for educational programs of any type.</p> <hd id="AN0180591483-39">Interpreting the Variability of the DLL Impacts</hd> <p>Relatively few benchmarks are available to assess the consistency of impacts found across the 30 participating DLL sites. Recently, there has been growing interest in investigating treatment-effect heterogeneity in multisite RCTs of education interventions (Bloom et al., [<reflink idref="bib8" id="ref154">8</reflink>]; Bloom & Spybrook, [<reflink idref="bib9" id="ref155">9</reflink>]; Weiss et al., [<reflink idref="bib80" id="ref156">80</reflink>]). In reviewing results from 16 multisite RCTs, Weiss et al. ([<reflink idref="bib80" id="ref157">80</reflink>]) concluded that findings of treatment impacts and site-to-site variability of the impacts follow several patterns. Specifically, the authors found that studies of education interventions fall into one of four categories: (<reflink idref="bib1" id="ref158">1</reflink>) yielding consistent near-zero impacts across sites; (<reflink idref="bib2" id="ref159">2</reflink>) yielding near-zero average impacts with substantial cross-site variation; (<reflink idref="bib3" id="ref160">3</reflink>) yielding consistent positive impacts across sites; and (<reflink idref="bib4" id="ref161">4</reflink>) yielding large average positive impacts with substantial cross-site variation. Our results would seem to fall into the latter two categories. Concluding which of these two characterizations best fits our results requires more careful consideration of the magnitudes of our cross-site variation.</p> <p>Weiss et al. ([<reflink idref="bib80" id="ref162">80</reflink>]) reported that about one third of the 16 multisite education RCTs that they reviewed yielded little cross-site impact variation, ranging from 0 to 0.05 standard deviations (<emph>SD</emph>s). Another third of the estimates ranged from 0.05 <emph>SDs</emph> to 0.15 <emph>SDs</emph>, indicating moderate cross-site variation, and the final third ranged from 0.15 <emph>SDs</emph> to 0.35 <emph>SDs</emph>, which suggested considerable cross-site variation. On the surface, our estimates of cross-site variation for the Spanish-language outcomes (0.25 <emph>SDs</emph> to more than 0.5 <emph>SDs</emph>) seem to align with the category of findings characterized by Weiss et al. as "large average positive impact[s] with substantial cross-site variation" (p. 13). Yet our cross-site impact estimates also can be described by the authors' other description of "consistently positive impacts across sites" (p. 13). Specifically, only one of the 30 sites in our sample had nonpositive impacts for the <emph>Logramos</emph> and <emph>IdO</emph> assessments.</p> <p>When examining the reasons behind greater consistency in cross-site impacts, Weiss et al. ([<reflink idref="bib80" id="ref163">80</reflink>]) identified three explanations: (<reflink idref="bib1" id="ref164">1</reflink>) highly specific and consistent interventions that tend to deliver similar services across sites; (<reflink idref="bib2" id="ref165">2</reflink>) low-intensity programs, which are characterized by small treatment-control service contrasts and limited opportunity for cross-site differences; and (<reflink idref="bib3" id="ref166">3</reflink>) interventions delivered to treatment and control students in the same school building yielding smaller treatment-control service contrasts than when treatment and control students are served in separate and potentially more variable school contexts.</p> <p>Because DLL is a highly specific intervention (explanation #1) and we studied both treatment and control students in the same building (explanation #3), we would expect consistent impacts across sites. However, because DLL is not a low-intensity program, (explanation #2), this may induce greater opportunity for impact variation across sites. Although the sample of 16 multisite studies reviewed by Weiss et al. ([<reflink idref="bib80" id="ref167">80</reflink>]) revealed no evidence that cross-site impact variation increases as the magnitude of cross-site mean impacts increases, none of the 16 studies had effect sizes approaching the magnitudes of the DLL intervention. Perhaps it is this combination of factors that explain the consistently positive but relatively variable impacts across the 30 participating schools in our study.</p> <hd id="AN0180591483-40">Conclusion</hd> <p>DLL is predicated on the notion that responsive, high-quality Spanish-language literacy programs require coordinated supports for students who are struggling with early literacy learning. DLL provides supplemental resources needed for Spanish-speaking students to participate—and potentially thrive—in their school's Spanish-language literacy program. Also, furthering the development of learning in one's native language can transfer to learning a second language and since DLL is taught in Spanish, it may contribute to students acquiring not only Spanish but also English literacy skills as they progress through elementary school. Indeed, much research indicates that teaching reading in a student's native language may be the optimal approach to improving reading in English. Research from multiple studies on a variety of languages supports the notion that learning literacy in one's first language augments literacy in the second (Richards-Tutor et al., [<reflink idref="bib64" id="ref168">64</reflink>]; Roberts, [<reflink idref="bib65" id="ref169">65</reflink>]). In the initial months of intervention, though, our results do not suggest that such transference occurred. As past research has suggested, this process of improved Spanish-language literacy helping students acquire additional English-language skills may take several years (Umansky & Reardon, [<reflink idref="bib72" id="ref170">72</reflink>]). Given the range of Spanish literacy skills that we measured, more nuanced analyses of which particular skills may predict students' future Spanish and English development could help further elucidate this process and suggest important points of focus for early intervention.</p> <p>Our sample was relatively small, therefore more research is necessary to document the generalizability of these favorable results across other districts, schools, teachers, and students. However, these experimental outcomes across 30 schools are very promising. The literacy outcomes evaluated revealed little evidence of variability across years, suggesting the reliability and replicability of DLL's impacts. Although impacts varied across sites, the impacts were positive in all but one site. Future research should investigate the extent to which these effects are sustained across multiple years, and whether these early advantages in students' native language of Spanish translate, as past evidence suggests, into later advantages in English literacy.</p> <p>In addition, future work should systematically link data concerning the quality and variability of implementation across sites and teachers with impact data. Clearly, key strengths of the DLL program are its well-specified guidelines and practices and ongoing oversight, support, and training by teacher leaders and university-based trainers. However, future work might provide greater understandings of how variability in the contexts in which DLL is implemented, the quality and quantity of DLL instruction and how it may vary, and how such contextual and programmatic variations may be associated with differences in student outcomes. Although there was some variability in DLL teachers' experience with DLL, on average, teachers had been delivering DLL for 8 years. Furthermore, all school sites had been implementing DLL when our study began. Our cost analysis suggests DLL requires significant financial commitment (Borman et al., [<reflink idref="bib11" id="ref171">11</reflink>]), so it is possible that newly implementing sites may not yield the large and positive impacts observed in this study or the variation in those impacts may be larger or smaller. Thus, understanding impact and variation in the context of early implementers is also important.</p> <p>Finally, future longitudinal studies could help inform how the program may prevent the need for other potentially more costly interventions, including special education, continuing specialized services for ELs, and retentions in grade. Prior quasi-experimental evidence suggests that DLL students maintain average to high literacy performance through elementary school (Escamilla et al., [<reflink idref="bib31" id="ref172">31</reflink>]; Neal & Kelly, [<reflink idref="bib55" id="ref173">55</reflink>]), and studies of early literacy programs emphasizing native language instruction suggest similar long-term benefits (Ashdown & Simic, [<reflink idref="bib2" id="ref174">2</reflink>]; Richards-Tutor et al., [<reflink idref="bib64" id="ref175">64</reflink>]). Therefore, even though DLL targets intensive resources for first-grade struggling readers, it is quite possible that these early investments have long-term benefits that may, ultimately, save schools and districts valuable resources and prevent future remediation efforts.</p> <hd id="AN0180591483-41">Open Research Statements</hd> <p></p> <hd id="AN0180591483-42">Study and Analysis Plan Registration</hd> <p>There is no study and analysis plan registration associated with this manuscript.</p> <hd id="AN0180591483-43">Data, Code, and Materials Transparency</hd> <p>The data and code underlying the results reported in this manuscript are available in openICPSR: https://<ulink href="http://www.openicpsr.org/openicpsr/project/193586/">www.openicpsr.org/openicpsr/project/193586/</ulink></p> <hd id="AN0180591483-44">Design and Analysis Reporting Guidelines</hd> <p>Not applicable.</p> <hd id="AN0180591483-45">Transparency Declaration</hd> <p>The lead author (the manuscript's guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.</p> <hd id="AN0180591483-46">Replication Statement</hd> <p>This manuscript reports an original study.</p> <hd id="AN0180591483-47">Open Scholarship</hd> <p>Graph</p> <p>This article has earned the https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/ badge for Open Data. The data are openly accessible at https://<ulink href="http://www.openicpsr.org/openicpsr/project/193586/version/V1/view">www.openicpsr.org/openicpsr/project/193586/version/V1/view</ulink>.</p> <hd id="AN0180591483-48">Disclosure Statement</hd> <p>No potential conflict of interest was reported by the author(s).</p> <ref id="AN0180591483-49"> <title> References </title> <blist> <bibl id="bib1" idref="ref15" type="bt">1</bibl> <bibtext> Arteagoitia, I., & Yen, S. J. (2020). Equity in representing literacy growth in dual language bilingual education for emerging bilingual learners. TESOL Quarterly, 54 (3), 719 – 742. https://doi.org/10.1002/tesq.588</bibtext> </blist> <blist> <bibl id="bib2" idref="ref61" type="bt">2</bibl> <bibtext> Ashdown, J., & Simic, O. (2000). Is early literacy intervention effective for English language learners? Evidence from reading recovery. Literacy Teaching and Learning, 5 (1), 27 – 42. https://readingrecovery.org/wp-content/uploads/2017/03/LTL_5.1-Ashdown-Simic.pdf.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref57" type="bt">3</bibl> <bibtext> August, D. and Hakuta, K., Eds. (1997). Improving schooling for language minority children: A research agenda. National Academy Press.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref29" type="bt">4</bibl> <bibtext> August, D., McCardle, P., & Shanahan, T. (2014). Developing literacy in English language learners: Findings from a review of the experimental research. School Psychology Review, 43 (4), 490 – 498. https://doi.org/10.1080/02796015.2014.12087417</bibtext> </blist> <blist> <bibl id="bib5" idref="ref30" type="bt">5</bibl> <bibtext> August, D., & Shanahan, T. (2006). Developing literacy in second-language learners: Report of the National Literacy Panel on language minority children and youth. Lawrence Erlbaum.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref31" type="bt">6</bibl> <bibtext> Baker, S., Geva, E., Kieffer, M. J., Lesaux, N., Linan-Thompson, S., Morris, J., Proctor, C. P., Russell, R., Gersten, R., Dimino, J., Jayanthi, M., Haymond, K., & Newman-Gonchar, R. (2014). Teaching academic content and literacy to English learners in elementary and middle school (NCEE 2014-4012). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/english_learners_pg_040114.pdf</bibtext> </blist> <blist> <bibl id="bib7" idref="ref1" type="bt">7</bibl> <bibtext> Bialystock, E. (2001). Bilingualism in development. Language, literacy, and cognition. Cambridge University Press.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref137" type="bt">8</bibl> <bibtext> Bloom, H. S., Raudenbush, S. W., Weiss, M. J., & Porter, K. (2017). Using multisite experiments to study cross-site variation in treatment effects: A hybrid approach with fixed intercepts and a random treatment coefficient. Journal of Research on Educational Effectiveness, 10 (4), 817 – 842. https://doi.org/10.1080/19345747.2016.1264518</bibtext> </blist> <blist> <bibl id="bib9" idref="ref155" type="bt">9</bibl> <bibtext> Bloom, H. S., & Spybrook, J. (2017). Assessing the precision of multisite trials for estimating the parameters of a cross-site population distribution of program effects. Journal of Research on Educational Effectiveness, 10 (4), 877 – 902. https://doi.org/10.1080/19345747.2016.1271069</bibtext> </blist> <blist> <bibtext> Borman, G. D., & D'Agostino, J. V. (1996). Title I and student achievement: A meta-analysis of federal evaluation results. Educational Evaluation and Policy Analysis, 18 (4), 309 – 326. https://eric.ed.gov/?id=EJ538585 https://doi.org/10.3102/01623737018004309</bibtext> </blist> <blist> <bibtext> Borman, G. D., Brodziak de los Reyes, I., Borman, T. H., Carbuccia-Abbott, M., Houghton, S., Park, S. J., Zhu, B., & Martin, A. (2022). The costs and impacts of Descubriendo la Lectura: Evidence from a multisite experimental study. Division of Leadership and Innovation, Arizona State University.</bibtext> </blist> <blist> <bibtext> Borman, G. D., Borman, T. H., Park, S. J., & Houghton, S. (2020). A multisite randomized controlled trial of the effectiveness of Descubriendo la Lectura. American Educational Research Journal, 57 (5), 1995 – 2020. https://doi.org/10.3102/0002831219890612</bibtext> </blist> <blist> <bibtext> Borman, T. H., Borman, G. D., Houghton, S., Park, S. J., Zhu, B., Martin, A., & Wilkinson-Flicker, S. (2019). Addressing literacy needs of struggling Spanish-speaking first graders: First-year results from a national randomized controlled trial of Descubriendo la Lectura. AERA Open, 5 (3), 233285841987048. https://doi.org/10.1177/2332858419870488</bibtext> </blist> <blist> <bibtext> Bryk, A. S., Gomez, L., & Grunow, A. (2010). Getting ideas into action: Building networked improvement communities in education. Carnegie Foundation for the Advancement of Teaching. https://<ulink href="http://www.carnegiefoundation.org/wp-content/uploads/2014/09/bryk-gomez%5fbuilding-nics-education.pdf">www.carnegiefoundation.org/wp-content/uploads/2014/09/bryk-gomez%5fbuilding-nics-education.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Calderón, M., Slavin, R., & Sanchez, M. (2011). Effective instruction for English learners. The Future of Children, 21 (1), 103 – 127. https://doi.org/10.1353/foc.2011.0007</bibtext> </blist> <blist> <bibtext> Calderón, M. E., & Slakk, S. (2019). Success with multicultural newcomers & English learners: Proven practices for school leadership teams. ASCD.</bibtext> </blist> <blist> <bibtext> Chen, X., & Padilla, A. M. (2019). Role of bilingualism and biculturalism as assets in positive psychology: Conceptual dynamic GEAR model. Frontiers in Psychology, 10, 2122. https://doi.org/10.3389/fpsyg.2019.02122</bibtext> </blist> <blist> <bibtext> Cheung, A., & Slavin, R. E. (2005). Effective reading programs for English language learners and other language minority students. Bilingual Research Journal, 29 (2), 241 – 267. https://doi.org/10.1080/15235882.2005.10162835</bibtext> </blist> <blist> <bibtext> Cheung, A. C., & Slavin, R. E. (2012). Effective reading programs for Spanish-dominant English language learners (ELLs) in the elementary grades: A synthesis of research. Review of Educational Research, 82 (4), 351 – 395. https://doi.org/10.3102/0034654312465472</bibtext> </blist> <blist> <bibtext> Clay, M. M. (1993). An observation survey of early literacy achievement. Heinemann.</bibtext> </blist> <blist> <bibtext> Clay, M. M. (2001). Change over time in children's literacy development. Heinemann.</bibtext> </blist> <blist> <bibtext> Clay, M. M. (2005). Literacy lessons designed for individuals. Heinemann.</bibtext> </blist> <blist> <bibtext> Clay, M. M. (2019). Observation survey of early literacy achievement (4th ed.). Heinemann.</bibtext> </blist> <blist> <bibtext> Cummins, J. (2005, September 23). Teaching for cross-language transfer in dual language education: Possibilities and pitfalls [Paper presentation]. TESOL Symposium on Dual Language Education: Teaching and Learning Two Languages in the EFL Setting, Bogazici University.</bibtext> </blist> <blist> <bibtext> de Brey, C., Zhang, A., & Duffy, S. (2022). Digest of education statistics. U.S. Department of Education, National Center for Education Statistics. https://nces.ed.gov/programs/digest/d20/index.asp</bibtext> </blist> <blist> <bibtext> DeFord, D. (2007). Marie Clay: An honored mentor, colleague, and friend. Journal of Reading Recovery, 7 (1), 101 – 103. <ulink href="http://readingrecovery.org/images/pdfs/Journals/JRR/Vol7%5fNo1%5fFall-2007/JRR%5f7.1-DeFord.pdf">http://readingrecovery.org/images/pdfs/Journals/JRR/Vol7%5fNo1%5fFall-2007/JRR%5f7.1-DeFord.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Dianda, M., & Flaherty, J. (1995, April). Effects of Success for All on the reading achievement of first graders in California bilingual programs [Paper presentation]. Annual Meeting of the American Educational Research Association, San Francisco, CA, United States.</bibtext> </blist> <blist> <bibtext> Durgunoğlu, A. Y., Nagy, W. E., & Hancin-Bhatt, B. J. (1993). Cross-language transfer of phonological awareness. Journal of Educational Psychology, 85 (3), 453 – 465. https://doi.org/10.1037/0022-0663.85.3.453</bibtext> </blist> <blist> <bibtext> Escamilla, K. (2018). Growing up with the Bilingual Education Act: One educator's journey. Bilingual Research Journal, 41 (4), 369 – 387. https://doi.org/10.1080/15235882.2018.1529641</bibtext> </blist> <blist> <bibtext> Escamilla, K., Andrade, A., Basurto, A., & Ruiz, O. (1996). Instrumento de Observación de los logros de la lecto-escritura inicial. Heinemann.</bibtext> </blist> <blist> <bibtext> Escamilla, K., Loera, M., Ruiz, O., & Rodríguez, Y. (1998). An examination of sustaining effects in Descubriendo la Lectura programs. Literacy Teaching and Learning. An International Journal of Early Reading and Writing, 3 (2), 59 – 81.</bibtext> </blist> <blist> <bibtext> Francis, D., Lesaux, N. K., & August, D. (2006). Language of instruction. In D. L. August & T. Shanahan (Eds.), Developing literacy in a second language: Report of the National Literacy Panel (pp. 365 – 410). Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> Francis, D. J., Shaywitz, S. E., Stuebing, K. K., Shaywitz, B. A., & Fletcher, J. M. (1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88 (1), 3 – 17. https://doi.org/10.1037/0022-0663.88.1.3</bibtext> </blist> <blist> <bibtext> Galindo, C., & Reardon, S. F. (2006). Hispanic students' educational experiences and opportunities during kindergarten. National Task Force on Early Childhood Education for Hispanics. https://stacks.stanford.edu/file/druid:xz483fw4636/hispanic_experience.pdf.</bibtext> </blist> <blist> <bibtext> Gandara, P., & Hopkins, M. (Eds.). (2010). Forbidden language: English learners and restrictive language policies. Teachers College Press.</bibtext> </blist> <blist> <bibtext> Gersten, R., Baker, S. K., Shanahan, T., Linan-Thompson, S., Collins, P., & Scarcella, R. (2007). Effective literacy and English language instruction for English learners in the elementary grades (NCEE 2007-4011). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/20074011.pdf.</bibtext> </blist> <blist> <bibtext> Gottardo, A. (2002). The relationship between language and reading skills in bilingual Spanish-English speakers. Topics in Language Disorders, 22 (5), 46 – 70. https://doi.org/10.1097/00011363-200211000-00008</bibtext> </blist> <blist> <bibtext> Greene, J. (1997). A meta-analysis of the Rossell and Baker review of bilingual education research. Bilingual Research Journal, 21 (2–3), 103 – 122. https://doi.org/10.1080/15235882.1997.10668656</bibtext> </blist> <blist> <bibtext> Hill, C. J., Bloom, H. S., Black, A. R., & Lipsey, M. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2 (3), 172 – 177. https://doi.org/10.1111/j.1750-8606.2008.00061.x</bibtext> </blist> <blist> <bibtext> International Data Evaluation Center. (2012). 2012–2013 Descubriendo la Lectura statistical abstract for the U.S. The Ohio State University. https://readingrecovery.org/wp-content/uploads/2023/02/2021-2022-Descubriendo-la-Lectura-National-Summary-Report-for-the-United-States.pdf.</bibtext> </blist> <blist> <bibtext> Joyce, B. R., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Association for Supervision & Curriculum Development.</bibtext> </blist> <blist> <bibtext> Juel, C. (1988). Learning to read and write: A longitudinal study 54 four children from first through fourth grades. Journal of Educational Psychology, 80 (4), 437 – 447. https://eric.ed.gov/?id=EJ389953 https://doi.org/10.1037/0022-0663.80.4.437</bibtext> </blist> <blist> <bibtext> Knudsen, E., Heckman, J., Cameron, J., & Shonkoff, J. P. (2006). Economic, neurobiological, and behavioral perspectives on building America's future workforce. Proceedings of the National Academy of Sciences of the United States of America, 103 (27), 10155 – 10162. https://doi.org/10.1073/pnas.0600888103</bibtext> </blist> <blist> <bibtext> Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49 (4), 241 – 253. https://doi.org/10.3102/0013189X20912798</bibtext> </blist> <blist> <bibtext> Logramos. (2006). 2005 Norms and score conversions with technical information. Riverside. <ulink href="http://www.riverpub.com/scoring/pdf/">http://www.riverpub.com/scoring/pdf/</ulink>.</bibtext> </blist> <blist> <bibtext> Ludwig, C., Guo, K., & Georgiou, G. K. (2019). Are reading interventions for English language learners effective? A meta-analysis. Journal of Learning Disabilities, 52 (3), 220 – 231. https://doi.org/10.1177/0022219419825855</bibtext> </blist> <blist> <bibtext> Lyon, G. R., & Fletcher, J. M. (2001). Early warning systems: How to prevent reading disabilities. Education Matters, 1 (2), 22 – 29.</bibtext> </blist> <blist> <bibtext> May, H., Gray, A., Gillespie, J. N., Sirinides, P., Sam, C., Goldsworthy, H., Atmijo, M., & Tognatta, N. (2013). Evaluation of the i3 scale-up of Reading Recovery. Year one report (2011–12). Consortium for Policy Research in Education. <ulink href="http://www.cpre.org/sites/default/files/researchreport/1488%5freadingrecoveryreport.pdf">http://www.cpre.org/sites/default/files/researchreport/1488%5freadingrecoveryreport.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> McField, G., & McField, D. (2014). The consistent outcome of bilingual education programs: A meta-analysis of meta-analyses. In G. McField (Ed.), The miseducation of English learners (pp. 267 – 299). Information Age.</bibtext> </blist> <blist> <bibtext> Nagy, W., Berninger, V., Abbott, R., Vaughan, K., & Vermeulen, K. (2003). Relationship of morphology and other language skills to literacy skills in at-risk second graders and at-risk fourth grade writers. Journal of Educational Psychology, 95 (4), 730 – 742. https://doi.org/10.1037/0022-0663.95.4.730</bibtext> </blist> <blist> <bibtext> National Academies of Sciences, Engineering, and Medicine. (2017). Promoting the educational success of children and youth learning English: Promising futures. National Academies Press.</bibtext> </blist> <blist> <bibtext> National Center for Education Statistics. (2022). English learners in public schools. Condition of Education. U.S. Department of Education, Institute of Education Sciences. https://nces.ed.gov/programs/coe/indicator/cgf.</bibtext> </blist> <blist> <bibtext> National Education Association. (2020). English language learners. https://<ulink href="http://www.nea.org/resource-library/english-language-learners">www.nea.org/resource-library/english-language-learners</ulink>.</bibtext> </blist> <blist> <bibtext> National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups. <ulink href="http://www.nichd.nih.gov/publications/pubs/nrp/documents/report.pdf">http://www.nichd.nih.gov/publications/pubs/nrp/documents/report.pdf</ulink></bibtext> </blist> <blist> <bibtext> Neal, J., & Kelly, P. (1999). The success of Reading Recovery for English language learners and Descubriendo la Lectura for bilingual students in California. Literacy Teaching and Learning, 4 (2), 81 – 108.</bibtext> </blist> <blist> <bibtext> Nunnery, J., Slavin, R., Ross, S., Smith, L., Hunter, P., & Stubbs, J. (1997, March 24–28). Effects of full and partial implementations of Success for All on student reading achievement in English and Spanish [Paper presentation]. Annual meeting of the American Educational Research Association, Chicago, IL, United States.</bibtext> </blist> <blist> <bibtext> Patterson, N., Weaver, J., Fletcher, J., Connor, B., Thomas, A., & Ross, C. (2018). Leveraging literacies through collaborative, source‐based planning and teaching in social studies and language arts. Journal of Adolescent & Adult Literacy, 61 (5), 523 – 532. https://<ulink href="http://www.jstor.org/stable/26631714">www.jstor.org/stable/26631714</ulink> https://doi.org/10.1002/jaal.712</bibtext> </blist> <blist> <bibtext> Planning and Evaluation Service, Office of the Deputy Secretary. (2001). Evidence that tutoring works. U.S. Department of Education. https://<ulink href="http://www.gpo.gov/fdsys/pkg/ERIC-ED464343/pdf/ERIC-ED464343.pdf">www.gpo.gov/fdsys/pkg/ERIC-ED464343/pdf/ERIC-ED464343.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> Portes, A., & Hao, L. (2002). The price of uniformity: Language, family and personality adjustment in the immigrant second generation. Ethnic and Racial Studies, 25 (6), 889 – 912. https://doi.org/10.1080/0141987022000009368</bibtext> </blist> <blist> <bibtext> Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage.</bibtext> </blist> <blist> <bibtext> Reading Recovery Council of North America. (2017). Standards and guidelines of Reading Recovery in the United States (8th ed.). https://readingrecovery.org/wp-content/uploads/2018/12/SG2018-Master-Full.pdf.</bibtext> </blist> <blist> <bibtext> Reading Recovery Council of North America. (2021). Lessons. https://readingrecovery.org/reading-recovery/teaching-children/lessons/.</bibtext> </blist> <blist> <bibtext> Reardon, S. F., & Galindo, C. (2009). The Hispanic-White achievement gap in math and reading in the elementary grades. American Educational Research Journal, 46 (3), 853 – 891. https://doi.org/10.3102/0002831209333184</bibtext> </blist> <blist> <bibtext> Richards-Tutor, C., Baker, D. L., Gersten, R., Baker, S. K., & Smith, J. M. (2016). The effectiveness of reading interventions for English learners: A research synthesis. Exceptional Children, 82 (2), 144 – 169. https://doi.org/10.1177/0014402915585483</bibtext> </blist> <blist> <bibtext> Roberts, C. A. (1994). Transferring literacy skills from L1 to L2: From theory to practice. The Journal of Educational Issues of Language Minority Students, 13, 209 – 221. https://eric.ed.gov/?id=EJ489585.</bibtext> </blist> <blist> <bibtext> Rolstad, K., Mahoney, K., & Glass, G. (2005). The big picture: A meta-analysis of program effectiveness research on English language learners. Educational Policy, 19 (4), 572 – 594. https://doi.org/10.1177/0895904805278067</bibtext> </blist> <blist> <bibtext> Shaywitz, S. E., Fletcher, J. M., Holahan, J. M., Shneider, A. E., Marchione, K. E., Stuebing, K. K., Francis, D. J., Pugh, K. R., & Shaywitz, B. A. (1999). Persistence of dyslexia: The Connecticut Longitudinal Study at adolescence. Pediatrics, 104 (6), 1351 – 1359. https://doi.org/10.1542/peds.104.6.1351</bibtext> </blist> <blist> <bibtext> Sirinides, P., Gray, A., & May, H. (2018). The impacts of Reading Recovery at scale: Results from the 4-year i3 external evaluation. Educational Evaluation and Policy Analysis, 40 (3), 316 – 335. https://doi.org/10.3102/0162373718764828</bibtext> </blist> <blist> <bibtext> Slavin, R., & Cheung, A. (2004). How do English language learners learn to read? Educational Leadership, 61 (6), 52 – 57.</bibtext> </blist> <blist> <bibtext> Slavin, R. E., Lake, C., Davis, S., & Madden, N. A. (2011). Effective programs for struggling readers: A best-evidence synthesis. Educational Research Review, 6 (1), 1 – 26. https://doi.org/10.1016/j.edurev.2010.07.002</bibtext> </blist> <blist> <bibtext> Snyder, E., Witmer, S. E., & Schmitt, H. (2017). English language learners and reading instruction: A review of the literature. Preventing School Failure: Alternative Education for Children and Youth, 61 (2), 136 – 145. https://doi.org/10.1080/1045988X.2016.1219301</bibtext> </blist> <blist> <bibtext> Umansky, I., & Reardon, S. F. (2014). Reclassification patterns among Latino English learner students in bilingual, dual immersion, and English immersion classrooms. American Educational Research Journal, 51 (5), 879 – 912. https://doi.org/10.3102/000283121454511</bibtext> </blist> <blist> <bibtext> Valentino, R. A., & Reardon, S. F. (2015). Effectiveness of four instructional programs designed to serve English learners: Variation by ethnicity and initial English proficiency. Educational Evaluation and Policy Analysis, 37 (4), 612 – 637. https://doi.org/10.3102/0162373715573310</bibtext> </blist> <blist> <bibtext> Vaughn, S., Cirino, P. T., Linan-Thompson, S., Mathes, P. G., Carlson, C. D., Hagan, E. C., Pollard-Durodola, S. D., Fletcher, J. M., & Francis, D. J. (2006). Effectiveness of a Spanish intervention and an English intervention for English-language learners at risk for reading problems. American Educational Research Journal, 43 (3), 449 – 487. https://doi.org/10.3102/00028312043003449</bibtext> </blist> <blist> <bibtext> Vaughn, S., Linan-Thompson, S., Mathes, P. G., Cirino, P. T., Carlson, C. D., Pollard-Durodola, S. D., Cardenas-Hagan, E., & Francis, D. J. (2006). Effectiveness of Spanish intervention for first-grade English language learners at risk for reading difficulties. Journal of Learning Disabilities, 39 (1), 56 – 73. https://doi.org/10.1177/00222194060390010601</bibtext> </blist> <blist> <bibtext> Vom Dorp, I. E. (2000). Biliteracy, monoliteracy and self-concept in native Spanish-dominant and native English-dominant fifth graders [Unpublished doctoral dissertation]. University of New Mexico.</bibtext> </blist> <blist> <bibtext> Wanzek, J., & Vaughn, S. (2007). Research-based implications from extensive early reading interventions. School Psychology Review, 36 (4), 541 – 561. https://eric.ed.gov/?id=EJ788353 https://doi.org/10.1080/02796015.2007.12087917</bibtext> </blist> <blist> <bibtext> Wanzek, J., Stevens, E. A., Williams, K. J., Scammacca, N., Vaughn, S., & Sargent, K. (2018). Current evidence on the effects of intensive early reading interventions. Journal of Learning Disabilities, 51 (6), 612 – 624. https://doi.org/10.1177/0022219418775110</bibtext> </blist> <blist> <bibtext> What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook (Version 5.0). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://ies.ed.gov/ncee/wwc/Handbooks.</bibtext> </blist> <blist> <bibtext> Weiss, M. J., Bloom, H. S., Verbitsky-Savitz, N., Gupta, H., Vigil, A. E., & Cullinan, D. N. (2017). How much do the effects of education and training programs vary across sites? Evidence from past multisite randomized trials. Journal of Research on Educational Effectiveness, 10 (4), 843 – 876. https://doi.org/10.1080/19345747.2017.1300719</bibtext> </blist> <blist> <bibtext> Willig, A. (1985). A meta-analysis of selected studies on the effectiveness of bilingual education. Review of Educational Research, 55 (3), 269 – 317. https://doi.org/10.3102/00346543055003269</bibtext> </blist> </ref> <ref id="AN0180591483-50"> <title> Footnotes </title> <blist> <bibtext> See https://ies.ed.gov/ncee/wwc.</bibtext> </blist> <blist> <bibtext> The WWC reviews of these studies are available at https://ies.ed.gov/ncee/wwc/study/89881 and https://ies.ed.gov/ncee/wwc/Study/90683.</bibtext> </blist> <blist> <bibtext> <emph>IdO</emph> is the Spanish version of RR's Observation Survey (Clay, [23]), which tests six literacy tasks (letter identification, word test [i.e., reading vocabulary], concepts about print, writing vocabulary, phonemic awareness, and text reading) that, taken together, can measure emerging literacy abilities for early readers.</bibtext> </blist> <blist> <bibtext> The <emph>Logramos</emph> is an evidence-based Spanish-language literacy assessment that parallels the English-language Iowa Test of Basic Skills (ITBS) and is used to monitor growth and achievement in Spanish literacy skills.</bibtext> </blist> <blist> <bibtext> See https://charts.intensiveintervention.org/screening/tool/?id=77c5c64492268897.</bibtext> </blist> <blist> <bibtext> See https://<ulink href="http://www.pearsonassessments.com/store/usassessments/en/Store/Professional-Assessments/Academic-Learning/Brief/Aprenda%3A-La-Prueba-de-Logros-en-Espa%C3%B1ol-%7C-Tercera-edici%C3%B3n/p/100000585.html">www.pearsonassessments.com/store/usassessments/en/Store/Professional-Assessments/Academic-Learning/Brief/Aprenda%3A-La-Prueba-de-Logros-en-Espa%C3%B1ol-%7C-Tercera-edici%C3%B3n/p/100000585.html</ulink>.</bibtext> </blist> <blist> <bibtext> See https://riversideinsights.com/log_tercera.</bibtext> </blist> <blist> <bibtext> Specifically, we rank ordered the statistically significant findings within the domain in ascending order of the <emph>p</emph>-values, such that <emph>p</emph><subs>1</subs> ≤ <emph>p</emph><subs>2</subs> ≤ <emph>p</emph><subs>3</subs> ≤ <emph>p</emph><subs>4</subs> ≤ <emph>p</emph><subs>5</subs> ≤ <emph>p</emph><subs>6</subs> ≤ <emph>p<subs>M</subs></emph>. For each <emph>p</emph>-value, <emph>p</emph><subs>1-M</subs>, we computed: <emph>p<subs>i</subs></emph> = α<emph><subs>i</subs></emph>/<emph>M</emph>, where <emph>i</emph> is the rank for <emph>p<subs>i</subs></emph>, with <emph>i</emph> = 1, 2,...<emph>M</emph>; <emph>M</emph> is the total number of findings within the domain; and α is the target level of statistical significance.</bibtext> </blist> </ref> <aug> <p>By Trisha H. Borman; Geoffrey D. Borman; So Jung Park; Bo Zhu and Scott Houghton</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib17" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib50" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib59" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib76" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib32" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib66" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib69" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib72" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib73" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib25" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib52" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib53" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib29" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib34" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib35" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib63" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib18" firstref="ref20"></nolink> <nolink nlid="nl18" bibid="bib33" firstref="ref21"></nolink> <nolink nlid="nl19" bibid="bib43" firstref="ref22"></nolink> <nolink nlid="nl20" bibid="bib78" firstref="ref23"></nolink> <nolink nlid="nl21" bibid="bib77" firstref="ref24"></nolink> <nolink nlid="nl22" bibid="bib16" firstref="ref25"></nolink> <nolink nlid="nl23" bibid="bib46" firstref="ref27"></nolink> <nolink nlid="nl24" bibid="bib64" firstref="ref28"></nolink> <nolink nlid="nl25" bibid="bib19" firstref="ref33"></nolink> <nolink nlid="nl26" bibid="bib36" firstref="ref34"></nolink> <nolink nlid="nl27" bibid="bib71" firstref="ref35"></nolink> <nolink nlid="nl28" bibid="bib74" firstref="ref37"></nolink> <nolink nlid="nl29" bibid="bib75" firstref="ref38"></nolink> <nolink nlid="nl30" bibid="bib27" firstref="ref39"></nolink> <nolink nlid="nl31" bibid="bib56" firstref="ref40"></nolink> <nolink nlid="nl32" bibid="bib15" firstref="ref41"></nolink> <nolink nlid="nl33" bibid="bib42" firstref="ref43"></nolink> <nolink nlid="nl34" bibid="bib47" firstref="ref44"></nolink> <nolink nlid="nl35" bibid="bib67" firstref="ref45"></nolink> <nolink nlid="nl36" bibid="bib38" firstref="ref52"></nolink> <nolink nlid="nl37" bibid="bib49" firstref="ref53"></nolink> <nolink nlid="nl38" bibid="bib81" firstref="ref56"></nolink> <nolink nlid="nl39" bibid="bib24" firstref="ref58"></nolink> <nolink nlid="nl40" bibid="bib28" firstref="ref59"></nolink> <nolink nlid="nl41" bibid="bib37" firstref="ref60"></nolink> <nolink nlid="nl42" bibid="bib20" firstref="ref62"></nolink> <nolink nlid="nl43" bibid="bib22" firstref="ref63"></nolink> <nolink nlid="nl44" bibid="bib70" firstref="ref66"></nolink> <nolink nlid="nl45" bibid="bib58" firstref="ref67"></nolink> <nolink nlid="nl46" bibid="bib61" firstref="ref68"></nolink> <nolink nlid="nl47" bibid="bib21" firstref="ref69"></nolink> <nolink nlid="nl48" bibid="bib26" firstref="ref70"></nolink> <nolink nlid="nl49" bibid="bib48" firstref="ref71"></nolink> <nolink nlid="nl50" bibid="bib62" firstref="ref78"></nolink> <nolink nlid="nl51" bibid="bib41" firstref="ref83"></nolink> <nolink nlid="nl52" bibid="bib14" firstref="ref86"></nolink> <nolink nlid="nl53" bibid="bib57" firstref="ref87"></nolink> <nolink nlid="nl54" bibid="bib13" firstref="ref89"></nolink> <nolink nlid="nl55" bibid="bib12" firstref="ref90"></nolink> <nolink nlid="nl56" bibid="bib30" firstref="ref93"></nolink> <nolink nlid="nl57" bibid="bib45" firstref="ref95"></nolink> <nolink nlid="nl58" bibid="bib40" firstref="ref107"></nolink> <nolink nlid="nl59" bibid="bib68" firstref="ref118"></nolink> <nolink nlid="nl60" bibid="bib60" firstref="ref136"></nolink> <nolink nlid="nl61" bibid="bib79" firstref="ref138"></nolink> <nolink nlid="nl62" bibid="bib44" firstref="ref151"></nolink> <nolink nlid="nl63" bibid="bib39" firstref="ref152"></nolink> <nolink nlid="nl64" bibid="bib10" firstref="ref153"></nolink> <nolink nlid="nl65" bibid="bib80" firstref="ref156"></nolink> <nolink nlid="nl66" bibid="bib65" firstref="ref169"></nolink> <nolink nlid="nl67" bibid="bib11" firstref="ref171"></nolink> <nolink nlid="nl68" bibid="bib31" firstref="ref172"></nolink> <nolink nlid="nl69" bibid="bib55" firstref="ref173"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1447262
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Spanish- and English-Language Literacy Impacts of 'Descubriendo la Lectura' across Three Experimental Replications
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Trisha+H%2E+Borman%22">Trisha H. Borman</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4176-4202">0000-0003-4176-4202</externalLink>)<br /><searchLink fieldCode="AR" term="%22Geoffrey+D%2E+Borman%22">Geoffrey D. Borman</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7039-8208">0000-0002-7039-8208</externalLink>)<br /><searchLink fieldCode="AR" term="%22So+Jung+Park%22">So Jung Park</searchLink><br /><searchLink fieldCode="AR" term="%22Bo+Zhu%22">Bo Zhu</searchLink><br /><searchLink fieldCode="AR" term="%22Scott+Houghton%22">Scott Houghton</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+on+Educational+Effectiveness%22"><i>Journal of Research on Educational Effectiveness</i></searchLink>. 2024 17(4):978-1007.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 30
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="EL" term="%22Primary+Education%22">Primary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Literacy%22">Literacy</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="DE" term="%22Spanish%22">Spanish</searchLink><br /><searchLink fieldCode="DE" term="%22Bilingual+Students%22">Bilingual Students</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22English+Language+Learners%22">English Language Learners</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Tutoring%22">Tutoring</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Skills%22">Reading Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Early+Intervention%22">Early Intervention</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22California%22">California</searchLink><br /><searchLink fieldCode="DE" term="%22Illinois%22">Illinois</searchLink><br /><searchLink fieldCode="DE" term="%22Texas%22">Texas</searchLink><br /><searchLink fieldCode="DE" term="%22Wisconsin%22">Wisconsin</searchLink><br /><searchLink fieldCode="DE" term="%22Arizona%22">Arizona</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2023.2269936
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1934-5747<br />1934-5739
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We present findings across three randomized trials of "Descubriendo la Lectura" (DLL), an intervention designed to improve literacy skills of Spanish-speaking first graders struggling with reading. DLL is a one-to-one Spanish-language literacy program lasting 12-20 weeks offered to a school's lowest performing emerging bilingual first graders. Directly replicating procedures across three cohorts, we examined Spanish and English literacy outcomes for approximately 400 students across 30 schools, 10 districts, and five states. Analyses revealed statistically significant effects of student-level assignment to DLL on all 11 Spanish-language outcomes, with an average impact of d = 0.57. Impacts across the four English-language outcomes were positive but not statistically significant. Descriptive results suggest that the Spanish-language impacts are reliably replicated, most likely due to the intervention's strong implementation supports.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://www.openicpsr.org/openicpsr/project/193586
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1447262
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1447262
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2023.2269936
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
        StartPage: 978
    Subjects:
      – SubjectFull: Literacy
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Grade 1
        Type: general
      – SubjectFull: Spanish
        Type: general
      – SubjectFull: Bilingual Students
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: English Language Learners
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Tutoring
        Type: general
      – SubjectFull: Reading Skills
        Type: general
      – SubjectFull: Early Intervention
        Type: general
      – SubjectFull: California
        Type: general
      – SubjectFull: Illinois
        Type: general
      – SubjectFull: Texas
        Type: general
      – SubjectFull: Wisconsin
        Type: general
      – SubjectFull: Arizona
        Type: general
    Titles:
      – TitleFull: The Spanish- and English-Language Literacy Impacts of 'Descubriendo la Lectura' across Three Experimental Replications
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Trisha H. Borman
      – PersonEntity:
          Name:
            NameFull: Geoffrey D. Borman
      – PersonEntity:
          Name:
            NameFull: So Jung Park
      – PersonEntity:
          Name:
            NameFull: Bo Zhu
      – PersonEntity:
          Name:
            NameFull: Scott Houghton
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1934-5747
            – Type: issn-electronic
              Value: 1934-5739
          Numbering:
            – Type: volume
              Value: 17
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Research on Educational Effectiveness
              Type: main
ResultId 1