View in EDS HTML Full Text PDF Full Text

Aiming Further: Addressing the Need for High Quality Longitudinal Research in Education

Saved in:
Bibliographic Details
Title: Aiming Further: Addressing the Need for High Quality Longitudinal Research in Education
Language: English
Authors: Watts, Tyler W., Bailey, Drew H., Li, Chen
Source: Grantee Submission. 2019.
Peer Reviewed: Y
Page Count: 21
Publication Date: 2019
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305A160176
Document Type: Opinion Papers
Reports - Evaluative
Descriptors: Educational Research, Longitudinal Studies, Educational Quality, Randomized Controlled Trials, Followup Studies, Educational Experiments, Educational Finance
DOI: 10.1080/19345747.2019.1644692
Abstract: Theories regarding the long-term effects of educational interventions are often assumed, but rarely tested using experimental methods. In the following commentary, we argue that the shortage of randomized control trials with long-term follow-up presents serious problems for the field, as it hampers our ability to develop educational programs that produce long-lasting effects, and creates incentives for the research community to focus too much attention on short-run impacts. We present steps that both researchers and funders could take to substantially improve the educational research literature by investing in educational experiments with long-term follow-up. [This paper was published in "Journal of Research on Educational Evaluation."]
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2020
Accession Number: ED608804
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFYo7s0g2fvmAy6vn7quiOOAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDAFIvEVpKxQECj2WxwIBEICBmpUnjXxG_bhLN7Mvu5mXk21hz67xn3qy0-td7Ldf6OMenK3RJqKFwSm6kHR-DqPTrZAr99gWT1vHs8wJ8D13HxWk0j88OsCRhxXIPU-AQ9EubXJi98UK8oON9y6TN-W-VhuYU58eCtLFLFWc7V_AIjN_JhgmoHJOLyjWvVo1XYQjsg_7M-xbQSglW213b_GKG5-vcprmKfeeeec=
Text:
  Availability: 1
  Value: <anid>AN0140233866;[5ew9]01oct.19;2019Dec10.04:02;v2.2.500</anid> <title id="AN0140233866-1">Aiming Further: Addressing the Need for High-Quality Longitudinal Research in Education </title> <p>Theories regarding the long-term effects of educational interventions are often assumed, but rarely tested using experimental methods. In the following commentary, we argue that the shortage of randomized control trials with long-term follow-up presents serious problems for the field, as it hampers our ability to develop educational programs that produce long-lasting effects, and creates incentives for the research community to focus too much attention on short-run impacts. We presents steps that both researchers and funders could take to substantially improve the educational research literature by investing in educational experiments with long-term follow-up.</p> <p>Keywords: educational evaluation longitudinal; research; commentary</p> <p>In educational research, the importance of longer-run follow-ups has been continually identified as a key priority for the field, with policy reports (Martin, McBridge, Brims, Doubell, Pote, & Clarke, [<reflink idref="bib22" id="ref1">22</reflink>]; McCormick, Hsueh, Weiland, & Bangser, [<reflink idref="bib23" id="ref2">23</reflink>]; Phillips et al., [<reflink idref="bib27" id="ref3">27</reflink>]), conference keynote addresses (see SREE invited lectures by Duncan [[<reflink idref="bib12" id="ref4">12</reflink>]] and Singer [[<reflink idref="bib31" id="ref5">31</reflink>]]), and "future directions" sections of research manuscripts noting the need to conduct evaluations with longitudinal follow-up. In recent years, the field has experienced substantial growth in the use of randomized control trials (RCTs) for the evaluation of educational programs, and at the same time, the wide availability of secondary administrative data sources has made longitudinal follow-up for these RCTs more possible than ever before (Penner & Dodge, [<reflink idref="bib26" id="ref6">26</reflink>]). However, despite these important innovations, educational interventions reporting long-run follow-up are still scarce, leaving a critical gap in the evaluation literature. In this commentary, we argue that this gap hampers the field's progress, stifling our ability to empirically test fundamental theories regarding long-run development and incentivizing research practices that are counterproductive to our widely held goals. Below, we offer several options that researchers and funders could pursue to substantially strengthen our understanding of how educational programs influence long-term student outcomes.</p> <hd id="AN0140233866-2">The Need for RCTs With Longitudinal Follow-Up</hd> <p>Educational research has benefitted greatly from longitudinal studies using correlational and quasi-experimental designs. Correlational studies have identified potential targets for educational interventions, and quasi-experimental studies have generated additional sources of data for estimating internally valid program impacts. However, quasi-experimental studies often carry limitations that complicate, or prevent altogether, longitudinal follow-up because the comparison group receives the treatment in a later period (e.g., age-of-entry regression discontinuity designs for public pre-k, Weiland & Yoshikawa, [<reflink idref="bib36" id="ref7">36</reflink>]; difference-in-differences designs for school accountability, Dee & Jacob, [<reflink idref="bib9" id="ref8">9</reflink>]). Further, the instruments providing "exogenous" variation in most quasi-experimental studies are often subject to assumptions that are difficult to fully test, and correlational research is even further compromised by omitted variable bias. These limitations leave RCTs as the "gold standard" of evidence for educational program evaluation.[<reflink idref="bib1" id="ref9">1</reflink>]</p> <p>Fortunately, educational RCTs have become much more common in recent years, partly due to the growing influence of The Institute for Education Sciences (IES). Since its inception in 2002, IES has become the dominant funder of educational intervention evaluations. Yet, despite explicitly calling for "follow-up" studies as part of its annual request for applications (RFA), a survey of funded IES projects highlights the severe lack of longitudinal follow-up in educational research. Using IES's public database of funded research grants and contracts (https://ies.ed.gov/funding/grantsearch/), we searched for all studies funded under the "Efficacy and Replication" category, which focuses on "the evaluation of fully-developed education interventions ... [in] authentic education settings" and "follow-up studies of students" (i.e., "Goal 3" grants; see https://ies.ed.gov/funding/). This search returned 394 abstracts from funded grants, which we further narrowed to 370 abstracts that used some form of the term "random" and that we determined were RCTs. We then coded any study abstract that used the term "longitudinal," "follow-up," or "long-term" to record the furthest follow-up assessment planned post intervention (172 studies used one of these terms along with the term "random").</p> <p>We found that only 27 of the 370 (7.3%) funded RCTs had discernable follow-up plans past 2 years after the end of the intervention. From this group of 27, 12 studies planned to follow students between 2 and 4 years post-intervention, and 15 planned to follow students over 4 years. The lack of longitudinal follow-up is partly due to the mechanics of IES funding, as grants do not typically last more than 5 years. However, we recorded only 20 studies that were dedicated follow-up studies with the purpose of tracking a sample that had already been examined in a previous evaluation. Thus, although the field has moved substantially toward RCT evaluations, these studies have largely lacked the longitudinal follow-up needed to assess whether the interventions in question make sustained impacts on child outcomes.</p> <p>This research gap has led to several issues that continue to hamper the field's progress. First, researchers continue to rely on correlational evidence linking academic achievement measures to adult outcomes in order to project program impacts when follow-up measures are unavailable (Kraft, [<reflink idref="bib19" id="ref10">19</reflink>]). This projection is often made implicitly in introduction and discussion sections when researchers cite correlational studies to motivate the intervention at hand, or the projection is made explicitly when researchers use reported correlations between test scores and earnings to make labor-market impact projections for cost-benefit analyses (Deming, [<reflink idref="bib10" id="ref11">10</reflink>]; Krueger, [<reflink idref="bib20" id="ref12">20</reflink>]). Despite growing indications that this approach might provide inaccurate long-term estimates, both by underestimating (Bartik, [<reflink idref="bib2" id="ref13">2</reflink>]; Fredriksson, Öckert, & Oosterbeek, [<reflink idref="bib14" id="ref14">14</reflink>]) and overestimating (Chetty et al., [<reflink idref="bib6" id="ref15">6</reflink>]) the long-run effect of interventions, it continues to be widely used (Bartik, Gormley, & Adelstein, [<reflink idref="bib3" id="ref16">3</reflink>]; Kline & Walters, [<reflink idref="bib17" id="ref17">17</reflink>]). This practice may lead researchers and practitioners to make inefficient investment decisions based solely on short-run impacts when long-term impacts are left unmeasured.</p> <p>A single-minded focus on end-of-treatment outcomes also creates problematic incentives for researchers. As with the overalignment problem between interventions and outcome measures (i.e., "teaching to the test"; see Slavin, [<reflink idref="bib32" id="ref18">32</reflink>]; Koretz, [<reflink idref="bib18" id="ref19">18</reflink>]), the alignment between the intervention and the <emph>timing</emph> of outcome measurement may incentivize curricula and pedagogy tailored to a narrower set of academic skills than would be ideal for maximizing students' long-run success. This short-run focus may lead to interventions that are unlikely to complement students' subsequent educational experiences, while simultaneously creating little incentive for collaborative projects that would align children's educational experiences over a multiyear period (Stipek, Franke, Clements, Farran, & Coburn, [<reflink idref="bib34" id="ref20">34</reflink>]). The popular hypothesis that proposed interventions might be necessary, but not sufficient, for spurring long-lasting change without complementary improvements to later educational quality could be tested directly. Perhaps most importantly, focus on short-run impacts gives researchers few incentives to think about their own unique solutions to generating impacts on students' long-run outcomes, a problem that would benefit from diverse teams of researchers working toward the same goal (Brooks-Gunn, [<reflink idref="bib4" id="ref21">4</reflink>]).</p> <p>Many factors likely contribute to the lack of longitudinal follow-up following educational interventions. Sample attrition following the end of treatment erodes study power over time, and researchers may consider several factors—the possibility of disappointing short-run fadeout on test scores; subsequent home, administrative, and curricular practices outside of the researcher's control; time that could be spent designing new interventions—as limiting the appeal of a resource-intensive follow-up. Indeed, substantial resources are required to collect follow-up data for large-scale RCTs, and as our coding exercise illustrated, researchers may simply lack the support needed to pursue follow-up studies. However, because we were not able to observe the complete pool of IES applications (only the studies that actually received funding), our coding exercise could not test whether the lack of follow-up funding was due to the applicant pool (i.e., few studies seek follow-up funding) or the grant selection process (i.e., follow-up studies are submitted but not selected). Given that the number of funded follow-up studies remains remarkably low, it seems plausible that both the applicant pool and the grant selection process could benefit from a greater focus on longer-run outcomes. Consequently, we provide recommendations to funders and researchers in the sections below.</p> <hd id="AN0140233866-3">Recommendations for Funders</hd> <p>New RFAs should encourage researchers to prespecify hypotheses for whether (and if so, how) their proposed intervention would affect long-term outcomes. For IES, this policy could ask researchers to incorporate their long-term hypotheses into their logic model, which would incentivize researchers to think carefully about the possible long-term implications of the intervention proposed. Such a policy would work best if coupled with an official preregistration database, like the new Registry of Efficacy and Effectiveness Studies (Spybrook, Anderson, & Maynard, [<reflink idref="bib33" id="ref22">33</reflink>]), a preregistration website designed specifically for educational interventions (engagement with this registry has now become an encouraged component of new IES RFAs).</p> <p>It should be noted that interventions need not affect long-run outcomes in order to be worthwhile or informative. For example, funders and researchers may find merit in a study examining the effects of a preschool reading curriculum, regardless of the curriculum's effects on long-term reading achievement. However, if such a study had only short-run goals in mind, then this should be made explicit in both the framing of the study and the stated theory of change. In this case, future long-term follow-up could be pursued only for exploratory purposes.</p> <p>More often, researchers hint at predictions about the long-run importance of a particular intervention or intervention target by citing the relatively small experimental literature that has included long-run follow-up (e.g., Heckman, [<reflink idref="bib15" id="ref23">15</reflink>], cited over 3,000 times) or correlational work highlighting the predictive validity of a particular construct. Keeping with the above example of an early reading program, the proposed intervention might frame the importance of the study by citing correlational studies showing strong relations between early reading achievement and later school success (e.g., Duncan et al., [<reflink idref="bib13" id="ref24">13</reflink>], cited over 4,000 times), or they might cite influential theoretical work predicting that early boosts in reading achievement should lead to future skill acquisition (e.g., Cunha & Heckman, [<reflink idref="bib8" id="ref25">8</reflink>], cited over 2,500 times). In these cases, long-term hypotheses are made implicitly, even if the study is only funded to test impacts on short-run measures of reading achievement. By asking researchers to shift these implicit theories to explicit predictions, researchers will be given incentives to think carefully about the mechanisms that connect their intervention models to the larger goals of educational programs that researchers often discuss only superficially at the beginning of papers and grant proposals.</p> <p>Next, coupled with the prespecification of long-run hypotheses, funders could also ask researchers to provide some indications for how their long-term hypotheses could be tested. By building "future research plans" into new grant proposals, funders would ask researchers to design new intervention studies that open the possibility of future high-quality follow-up research. Such plans could include proposed partnerships with organizations that house administrative data, or researchers could even detail plans to transfer the study to other organizations that may be better suited for future waves of data collection. As we detail in the "Recommendations for Researchers" section below, some early planning for future follow-up could substantially boost a study's chances of collecting further data from their sample should researchers and funders choose to pursue long-run follow-up.</p> <hd id="AN0140233866-4">Selecting Studies for Follow-Up Funding</hd> <p>If these two changes were made to the application process for new intervention studies, funders could rely on several selection mechanisms to choose from the pool of studies that (<reflink idref="bib1" id="ref26">1</reflink>) articulated hypotheses regarding long-run effects and (<reflink idref="bib2" id="ref27">2</reflink>) provided credible research plans for testing long-term hypotheses. First, organizations could build on the current practice of calling for follow-up of existing evaluations in RFAs. Placing new emphasis on funding follow-up studies (see recent blog post from current IES director [Schneider [<reflink idref="bib29" id="ref28">29</reflink>]]), even designating some RFAs entirely for follow-up funding, could encourage researchers to apply. Further, allowing researchers to apply to extend their preexisting evaluation projects may also encourage more follow-up applications. If long-run hypotheses and research plans were already articulated in initial applications, then extension applications could be briefer and focused solely on updating the follow-up data collection plans given the current state of the research project.</p> <p>Another approach could add efficiency to the process by cutting out researcher-written follow-up applications altogether if funding agencies determined themselves which projects merit follow-up. With this plan, funders would use initial grant applications to determine which studies made plausible long-term predictions and provided details for long-run data collection plans. They could then use annual progress reports to track important design issues (e.g., study attrition, implementation fidelity) to generate a pool of high-quality studies eligible for further follow-up funding. Funders would then appoint a review panel to review already funded evaluations that were nearing project completion, and they could choose which projects were most promising for follow-up based on theories of change, reported effect sizes, and design quality. Of course, with this policy, funders would merely offer funding to keep projects going, and researchers would have to consider whether accepting the funding was a worthwhile investment of their own time and energy.</p> <p>Although these new funding options would offer improvements over the status quo, these mechanisms also carry drawbacks. If follow-up funding is contingent on showing "promising" short-run effects, then researchers would be even further incentivized to design evaluations that produce the largest short-run impacts regardless of how these impacts extend into future periods. If the primary reason to follow up on an evaluation is the size of the initial impact estimate, then this positive selection (whether selection is correlated with systematic error, such as selective reporting of the largest impacts, overaligned outcome measures, or even random error in impact estimates) will inflate end-of-treatment effect sizes. Indeed, because these incentives already exist, the current preponderance of fadeout effects in educational intervention studies (Bailey, Duncan, Odgers, & Yu, [<reflink idref="bib1" id="ref29">1</reflink>]) could be partly due to the fact that follow-up attempts almost exclusively ensue after "promising" short-run effects have been reported. Moreover, this preference for studies showing large short-run effects gives researchers few incentives to pursue interventions that move more difficult-to-alter aspects of student cognition and behavior. Such programs may have the best chance of producing long-lasting effects despite producing smaller short-run impacts when compared with narrowly targeted interventions.</p> <p>Consequently, our preferred selection mechanism would involve funders randomly selecting projects from the aforementioned pool of high-quality studies eligible for follow-up funding. A random selection process would mitigate the incentives for researchers to design evaluations that might inflate short-term impact estimates (although the pressure to publish may still encourage many of these same behaviors) and would also incentivize more careful thinking about long-term mechanisms. The random selection process would also allow for the possibility of detecting long-term impact patterns that we have little chance of detecting in educational RCTs under the status quo (e.g., null short-term impacts followed by positive long-term impacts). Thus, randomly selecting studies that prespecified long-term hypotheses and met a threshold for design quality could yield substantial benefits by realigning researcher incentives and increasing the range of studies reporting long-term effects.</p> <hd id="AN0140233866-5">Prioritizing Research Quality</hd> <p>If funders encourage researchers to prespecify plausible long-term hypotheses and future follow-up data collection plans, then the initial competitive grant review process should yield a pool of high-quality studies eligible for follow-up funding (while still funding important short-run interventions with no hypothesized long-term effects). Regardless of the specific selection mechanism pursued by funders, the field would substantially benefit if follow-up support was extended based on the quality of research, rather than the size of the short-run effect.</p> <p>This could mean that funders invest in follow-up of studies that prespecified long-run hypotheses but found disappointing short-run effects. Funding these studies may seem risky, as analyses of long-run follow-up data would qualify as "exploratory" (i.e., any long-term effects detected would not occur due to the mechanisms prespecified in the original theory of change). Nevertheless, many educational programs currently under consideration (e.g., public preschool, charter schools, after-school programs) have been hypothesized to affect a broad range of child developmental processes, and it remains unclear whether we have fully identified, or capably measured, the mediational mechanisms that might produce long-term impacts for many of these programs. For example, in early childhood research, the famous Perry Preschool Program produced strong long-term impacts on adult indicators of economic success and well-being, yet the mediational processes that led to these impacts are still not totally understood (Bailey et al., [<reflink idref="bib1" id="ref30">1</reflink>]; Heckman, Pinto, & Savelyev, 2013). Perry Preschool produced fading impacts on measures of childhood IQ, but longitudinal data collection persisted—and the study continues to yield substantial theoretical benefits as a result.</p> <p>Thus, funders must determine how much emphasis should be placed on pursuing longitudinal follow-up, some of which may be exploratory. Certainly, if we find that short-run null effects are always followed by long-run null effects, then the field could learn from this and shift priorities accordingly. Even in this case, these null-effect studies would serve as an important comparison group to studies that did find positive long-term impacts. Moreover, by investing in high-quality long-run research now, we will develop an empirical body of literature that will improve our ability to rely on short-run evidence to project long-run effects in the future.</p> <hd id="AN0140233866-6">Recommendations for Researchers</hd> <p>If more funding is extended for follow-up studies, researchers could take advantage of these resources to enhance their intervention research in several interesting ways. First, we recommend that researchers begin planning early for potential long-term follow-up. Careful consideration of plausible long-term mechanisms from the outset of intervention development could provide substantial benefits. For example, in the above-described hypothetical reading intervention, will the curriculum teach material that students in the control group are scheduled to learn months after the end of treatment? If so, in those months, is there a plausible mechanism through which the knowledge gained during the intervention would transfer to other domains? If not, is there some way to alter the curriculum or its timing to make this more likely? Designing interventions that can purposefully connect to the set of environmental experiences expected for students after leaving the intervention would raise the possibility of developing educational interventions that will produce long-lasting effects. Researchers often attribute intervention effect fadeout to the subsequent environmental experiences of intervention participants, a possibility we find plausible. However, this possibility also points to the potential usefulness of interventions designed to complement the subsequent environmental conditions of intervention participants.</p> <p>Second, we encourage researchers to take advantage of the vast amounts of secondary data now available to continue following their evaluation samples. Penner and Dodge ([<reflink idref="bib26" id="ref31">26</reflink>]) recently included this among the many benefits that can be gained by engaging with administrative data sources. Indeed, IES has funded multiple longitudinal data systems in states and large cities across the country (see full list at https://nces.ed.gov/programs/slds/), yet these large data systems have been largely underutilized. Merging secondary data sources with earlier intervention evaluation samples has already yielded highly influential findings (Chetty et al., [<reflink idref="bib6" id="ref32">6</reflink>]; Chetty, Hendren, & Katz, [<reflink idref="bib7" id="ref33">7</reflink>]; Dodge, Bai, Ladd, & Muschkin, [<reflink idref="bib11" id="ref34">11</reflink>]; Lipsey, Farran, & Durkin, [<reflink idref="bib21" id="ref35">21</reflink>]) and will likely continue to do so. Using administrative data sources also has the benefit of carrying a lower price than traditional modes of data collection, raising the possibility of pursuing long-term follow-up even when further funding is not guaranteed.</p> <p>Given the continued growth in this sector, we encourage researchers to begin communicating with organizations that maintain administrative databases early in their intervention evaluations. This would allow researchers to better understand, and collect, the information that will be needed to eventually link participant data to secondary sources. Further, researchers should reach out to these organizations to acquire information regarding the informed consent procedures that will be required to link participant data. In some cases, it may be possible to build consent for future data release into the early waves of data collection, when participant retention and recruitment presents a less severe problem. By obtaining permission for the release of records from the outset, secondary data sources could substantially help curb long-term attrition across studies.</p> <p>Of course, the benefit of these administrative sources of data should not be overstated. Such sources of data often provide measures for a narrow set of outcomes that may or may not be useful to a given study (i.e., test scores, grade point average, etc.). Further, as children move out of schools or districts over time, participants may disappear from certain databases, further eroding study power. However, partnering with organizations that house higher-level databases (e.g., the state-level databases set up by IES), rather than single schools or districts, may prove valuable as participants disperse over time.</p> <p>Finally, we recognize the need and desire to continue to develop new intervention projects for funders and researchers alike and suggest that these goals can be complementary. Ongoing innovation through the development of new interventions will generate important variation that might be used to isolate effective program features. Focusing solely on older evaluation studies could have the drawback of diverting attention from the development of newer programs. One promising approach for combining these goals could be the use of older samples to test the efficacy of new programs. If both the "new" and "old" intervention were randomly assigned, testing the effects of one intervention should have no bearing on our ability to detect effects for the other. If studies were properly powered, this would also heighten our ability to find instances of "dynamic complementarity," which is the influential idea that educational investments may positively interact across time to make long-lasting impacts on children's trajectories (Cunha & Heckman, [<reflink idref="bib8" id="ref36">8</reflink>]). Indeed, this design has been recently pursued by at least one IES-funded project.[<reflink idref="bib2" id="ref37">2</reflink>]</p> <p>Of course, researchers would have to consider whether older samples are representative of populations of interest for newer interventions. Further, because educational researchers often specialize in programs targeted to specific age groups (e.g., early childhood, adolescence, transition to adulthood), providing new interventions to older samples would incentivize further collaborations among researchers across specializations. This might lead to more programs that that align instruction and programmatic elements over the course of development.</p> <hd id="AN0140233866-7">Conclusion</hd> <p>The field could substantially benefit from more rigorous educational evaluations reporting long-term follow-up. At present, connections between short-run outcomes and long-term impacts are often assumed, but rarely tested using experimental methods. Indeed, correlational and quasi-experimental evidence should continue to play a role in longitudinal research. However, by pursuing more longitudinal follow-up of high-quality educational RCTs, funders and researchers can better test the long-run theories that are often implied by correlational work.</p> <p>Certainly, longitudinal evaluations are not without their own limitations. As longitudinal follow-up stretches into future years, the context within which the intervention was originally tested differentiates further from the status quo. This is an unfortunate, but unavoidable, limitation of longitudinal work. However, as the enduring influence of the handful of educational RCTs with long-run follow-up demonstrates (Campbell, Ramey, Pungello, Sparling, & Miller-Johnson, [<reflink idref="bib5" id="ref38">5</reflink>]; Heckman, [<reflink idref="bib15" id="ref39">15</reflink>]; McCormick et al., [<reflink idref="bib24" id="ref40">24</reflink>]; Myers, Olsen, Seftor, Young, & Tuttle, [<reflink idref="bib25" id="ref41">25</reflink>]; Schochet, Burghardt, & McConnell, [<reflink idref="bib30" id="ref42">30</reflink>]), the underlying processes tested by interventions of interest often remain surprisingly relevant over time.</p> <p>Producing long-lasting impacts on key developmental outcomes should not be considered an easy task, and the "success" or "failure" of interventions should not be judged solely on the basis of long-run effects (e.g., an intervention may be necessary, but not sufficient, for spurring long-run change on an outcome of interest). In other words, many educational programs should probably not be expected to produce "inoculation effects." However, the common practice of citing long-run experimental or correlational evidence as motivation to pursue short-run interventions that produce unknown long-run effects indicates a need for clarity on these issues.</p> <p>Thus, our longitudinal theories should be formalized and tested empirically. Perhaps researchers do expect long-run impacts of their interventions; perhaps they expect long-run impacts contingent on some measurable medium-run contextual effects; perhaps they have no specific theory in mind but merely cite long-run evidence because it is common practice to do so. Perhaps researchers refrain from discussing long-run impacts because their educational intervention serves some worthwhile short-term goal. In any of these cases, requiring applicants to make these goals explicit would make funding decisions better informed by the purpose of the proposed research (and by reviewers' judgments of whether these goals are likely to be reached)—outcomes to which we hope funding agencies and researchers aspire.</p> <p>Given the recent advancements in rigorous methodology for the evaluations of education programs, along with the new availability of administrative data sources, the opportunity for researchers and funders to support long-term follow-up has never been greater. The benefits stemming from the changes we propose would take years to accumulate, but investing in long-term follow-up projects now could yield substantial long-term benefits to the field for years to come.</p> <hd id="AN0140233866-8">Acknowledgment</hd> <p>The authors would like to thank Ana A. Whitaker, Greg Duncan, Dale Farran, Javanna Obregon, Cybele Raver, and Christina Weiland for their helpful comments on previous drafts.</p> <hd id="AN0140233866-9">Disclosures</hd> <p>The content of this article is solely the responsibility of the authors and does not represent the views of those acknowledged or the views of the Institute of Education Sciences or the Jacobs Foundation.</p> <ref id="AN0140233866-10"> <title> References </title> <blist> <bibl id="bib1" idref="ref9" type="bt">1</bibl> <bibtext> Bailey, D., Duncan, G. J., Odgers, C. L., & Yu, W. (2017). Persistence and fadeout in the impacts of child and adolescent interventions. Journal of Research on Educational Effectiveness, 10 (1), 7 – 39. doi: 10.1080/19345747.2016.1232459</bibtext> </blist> <blist> <bibl id="bib2" idref="ref13" type="bt">2</bibl> <bibtext> Bartik, T. J. (2014). From preschool to prosperity: The economic payoff to early childhood education. Kalamazoo, MI : W.E. Upjohn Institute for Employment Research. Retrieved from https://doi.org/10.17848/9780880994835</bibtext> </blist> <blist> <bibl id="bib3" idref="ref16" type="bt">3</bibl> <bibtext> Bartik, T. J., Gormley, W., & Adelstein, S. (2012). Earnings benefits of Tulsa's pre-K program for different income groups. Economics of Education Review, 31 (6), 1143 – 1161. doi: 10.1016/j.econedurev.2012.07.016</bibtext> </blist> <blist> <bibl id="bib4" idref="ref21" type="bt">4</bibl> <bibtext> Brooks-Gunn, J. (2003). Do you believe in magic?: What we can expect from early childhood intervention programs. Social Policy Report, 17 (1), 1 – 16. doi: 10.1002/j.2379-3988.2003.tb00020.x</bibtext> </blist> <blist> <bibl id="bib5" idref="ref38" type="bt">5</bibl> <bibtext> Campbell, F. A., Ramey, C. T., Pungello, E., Sparling, J., & Miller-Johnson, S. (2002). Early childhood education: Young adult outcomes from the Abecedarian Project. Applied Developmental Science, 6 (1), 42 – 57. doi: 10.1207/S1532480XADS0601_05</bibtext> </blist> <blist> <bibl id="bib6" idref="ref15" type="bt">6</bibl> <bibtext> Chetty, R., Friedman, J. N., Hilger, N., Saez, E., Schanzenbach, D. W., & Yagan, D. (2011). How does your kindergarten classroom affect your earnings? Evidence from Project Star. Quarterly Journal of Economics, 126 (4), 1593 – 1660.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref33" type="bt">7</bibl> <bibtext> Chetty, R., Hendren, N., & Katz, L. F. (2016). The effects of exposure to better neighborhoods on children: New evidence from the Moving to Opportunity experiment. American Economic Review, 106 (4), 855 – 902. doi: 10.1257/aer.20150572</bibtext> </blist> <blist> <bibl id="bib8" idref="ref25" type="bt">8</bibl> <bibtext> Cunha, F., & Heckman, J. (2007). The technology of skill formation. American Economic Review, 97 (2), 31 – 47. doi: 10.1257/aer.97.2.31</bibtext> </blist> <blist> <bibl id="bib9" idref="ref8" type="bt">9</bibl> <bibtext> Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and Management, 30 (3), 418 – 446. doi: 10.1002/pam.20586</bibtext> </blist> <blist> <bibtext> Deming, D. (2009). Early childhood intervention and life-cycle skill development: Evidence from Head Start. American Economic Journal: Applied Economics, 1 (3), 111 – 134. doi: 10.1257/app.1.3.111</bibtext> </blist> <blist> <bibtext> Dodge, K. A., Bai, Y., Ladd, H. F., & Muschkin, C. G. (2017). Impact of North Carolina's early childhood programs and policies on educational outcomes in elementary school. Child Development, 88 (3), 996 – 1014. doi: 10.1111/cdev.12645</bibtext> </blist> <blist> <bibtext> Duncan, G. J. (2015, March). Fade-out in human capital intervention: Death, miracles and resurrection. Lecture conducted for spring meeting of the Society for Research on Educational Effectiveness, Washington, DC.</bibtext> </blist> <blist> <bibtext> Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., ... Japel, C. (2007). School readiness and later achievement. Developmental Psychology, 43 (6), 1428 – 1446. doi: 10.1037/0012-1649.43.6.1428</bibtext> </blist> <blist> <bibtext> Fredriksson, P., Öckert, B., & Oosterbeek, H. (2013). Long-term effects of class size. The Quarterly Journal of Economics, 128 (1), 249 – 285. doi: 10.1093/qje/qjs048</bibtext> </blist> <blist> <bibtext> Heckman, J. J. (2006). Skill formation and the economics of investing in disadvantaged children. Science (New York, N.Y.), 312 (5782), 1900 – 1902. doi: 10.1126/science.1128898</bibtext> </blist> <blist> <bibtext> Heckman, J., Pinto, R., & Savelyev, P. (2013). Understanding the mechanisms through which an influential early childhood program boosted adult outcomes. American Economic Review, 103 (6), 2052 – 2086.</bibtext> </blist> <blist> <bibtext> Kline, P., & Walters, C. R. (2016). Evaluating public programs with close substitutes: The case of Head Start. The Quarterly Journal of Economics, 131 (4), 1795 – 1848. doi: 10.1093/qje/qjw027</bibtext> </blist> <blist> <bibtext> Koretz, D. (2005). Alignment, high stakes, and the inflation of test scores. Yearbook of the National Society for the Study of Education, 104 (2), 99 – 118. doi: 10.1111/j.1744-7984.2005.00027.x</bibtext> </blist> <blist> <bibtext> Kraft, M. A. (2018). Interpreting effect sizes of education interventions (Brown University Working Papers). Providence. Retrieved from https://scholar.harvard.edu/files/mkraft/files/kraft_2018_interpreting_effect_sizes.pdf</bibtext> </blist> <blist> <bibtext> Krueger, A. B. (2003). Economic considerations and class size. The Economic Journal, 113 (485), F34 – F63. doi: 10.1111/1468-0297.00098</bibtext> </blist> <blist> <bibtext> Lipsey, M. W., Farran, D. C., & Durkin, K. (2018). Effects of the Tennessee Prekindergarten Program on children's achievement and behavior through third grade. Early Childhood Research Quarterly, 45, 155 – 176. doi: 10.1016/j.ecresq.2018.03.005</bibtext> </blist> <blist> <bibtext> Martin, J., McBridge, T., Brims, L., Doubell, L., Pote, I., & Clarke, A. (2018). Evaluating early intervention programmes: Six common pitfalls, and how to avoid them. Retrieved from Early Intervention Foundation website: <ulink href="http://www.eif.org.uk/publication/evaluating-early-intervention-programmes-six-common-pitfalls-and-how-to-avoid-them">http://www.eif.org.uk/publication/evaluating-early-intervention-programmes-six-common-pitfalls-and-how-to-avoid-them</ulink></bibtext> </blist> <blist> <bibtext> McCormick, M., Hsueh, J., Weiland, C., & Bangser, M. (2017). The challenge of sustaining preschool impacts. Retrieved from MDRC website: https://<ulink href="http://www.mdrc.org/publication/challenge-sustaining-preschool-impacts">www.mdrc.org/publication/challenge-sustaining-preschool-impacts</ulink></bibtext> </blist> <blist> <bibtext> McCormick, M. C., Brooks-Gunn, J., Buka, S. L., Goldman, J., Yu, J., Salganik, M., ... Bauer, C. R. (2006). Early intervention in low birth weight premature infants: Results at 18 years of age for the Infant Health and Development Program. Pediatrics, 117 (3), 771 – 780. doi: 10.1542/peds.2005-1316</bibtext> </blist> <blist> <bibtext> Myers, D., Olsen, R., Seftor, N., Young, J., & Tuttle, C. (2004). The impacts of regular Upward Bound: Results from the third follow-up data collection. Washington, DC : Mathematica Policy Research.</bibtext> </blist> <blist> <bibtext> Penner, A. M., & Dodge, K. A. (2019). Using administrative data for social science and policy. RSF: The Russell Sage Foundation Journal of the Social Sciences, 5 (2), 1 – 18. doi: 10.7758/RSF.2019.5.2.01</bibtext> </blist> <blist> <bibtext> Phillips, D. A., Lipsey, M. W., Dodge, K. A., Haskins, R., Bassok, D., Burchinal, M. R., ... Weiland, C. (2017). The current state of scientific knowledge on pre-kindergarten effects. Retrieved from Brookings website: https://<ulink href="http://www.brookings.edu/wp-content/uploads/2017/04/duke%5fprekstudy%5ffinal%5f4-4-17%5fhires.pdf">www.brookings.edu/wp-content/uploads/2017/04/duke%5fprekstudy%5ffinal%5f4-4-17%5fhires.pdf</ulink></bibtext> </blist> <blist> <bibtext> Raver, C. C., Jones, S. M., Li-Grining, C., Zhai, F., Metzger, M. W., & Solomon, B. (2009). Targeting children's behavior problems in preschool classrooms: A cluster-randomized controlled trial. Journal of Consulting and Clinical Psychology, 77 (2), 302. doi: 10.1037/a0015302</bibtext> </blist> <blist> <bibtext> Schneider, M. (2019, June 19). Some thoughts on the New IES RFAs [Blog post]. Retrieved from https://ies.ed.gov/director/remarks/6-19-2019.asp</bibtext> </blist> <blist> <bibtext> Schochet, P. Z., Burghardt, J., & McConnell, S. (2008). Does Job Corps work? Impact findings from the national Job Corps study. American Economic Review, 98 (5), 1864 – 1886. doi: 10.1257/aer.98.5.1864</bibtext> </blist> <blist> <bibtext> Singer, J. (2019, March). Shaping the arc of educational research. Hedges Lecture conducted for spring meeting of the Society for Research on Educational Effectiveness, Washington, DC.</bibtext> </blist> <blist> <bibtext> Slavin, R. E. (2008). What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37 (1), 5 – 14. doi: 10.3102/0013189X08314117</bibtext> </blist> <blist> <bibtext> Spybrook, J., Anderson, D., & Maynard, R. (2019). The Registry of Efficacy and Effectiveness Studies (REES): A step toward increased transparency in education. Journal of Research on Educational Effectiveness, 12 (1), 5 – 9. doi: 10.1080/19345747.2018.1529212</bibtext> </blist> <blist> <bibtext> Stipek, D., Franke, M., Clements, D., Farran, D., & Coburn, C. (2017). PK-3: What does it mean for instruction? Social policy report. Society for Research in Child Development, 30 (2), 1 – 22. Retrieved from <ulink href="http://www.srcd.org/publications/social-policy-report">www.srcd.org/publications/social-policy-report</ulink></bibtext> </blist> <blist> <bibtext> Watts, T. W., Gandhi, J., Ibrahim, D. A., Masucci, M. D., & Raver, C. C. (2018). The Chicago School Readiness Project: Examining the long-term impacts of an early childhood intervention. Plos One, 13 (7), e0200144. doi: 10.1371/journal.pone.0200144</bibtext> </blist> <blist> <bibtext> Weiland, C., & Yoshikawa, H. (2013). Impacts of a prekindergarten program on children's mathematics, language, literacy, executive function, and emotional skills. Child Development, 84 (6), 2112 – 2130. doi: 10.1111/cdev.12099</bibtext> </blist> </ref> <ref id="AN0140233866-11"> <title> Footnotes </title> <blist> <bibtext> Here, we consider traditional RCTs where the treatment group is compared with a "business-as-usual" control group. RCTs with "waitlist" control designs also disallow for long-run follow-up.</bibtext> </blist> <blist> <bibtext> See recent work on the Chicago School Readiness Project (Raver et al., [28]; Watts, Gandhi, Ibrahim, Masucci, & Raver, [35]), which followed an early childhood intervention sample into adolescence and re-randomized the sample to a mindset intervention.</bibtext> </blist> </ref> <aug> <p>By Tyler W. Watts; Drew H. Bailey and Chen Li</p> <p>Reported by Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib22" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib23" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib27" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib12" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib31" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib26" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib36" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib19" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib10" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib20" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib14" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib17" firstref="ref17"></nolink> <nolink nlid="nl13" bibid="bib32" firstref="ref18"></nolink> <nolink nlid="nl14" bibid="bib18" firstref="ref19"></nolink> <nolink nlid="nl15" bibid="bib34" firstref="ref20"></nolink> <nolink nlid="nl16" bibid="bib33" firstref="ref22"></nolink> <nolink nlid="nl17" bibid="bib15" firstref="ref23"></nolink> <nolink nlid="nl18" bibid="bib13" firstref="ref24"></nolink> <nolink nlid="nl19" bibid="bib29" firstref="ref28"></nolink> <nolink nlid="nl20" bibid="bib11" firstref="ref34"></nolink> <nolink nlid="nl21" bibid="bib21" firstref="ref35"></nolink> <nolink nlid="nl22" bibid="bib24" firstref="ref40"></nolink> <nolink nlid="nl23" bibid="bib25" firstref="ref41"></nolink> <nolink nlid="nl24" bibid="bib30" firstref="ref42"></nolink>
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED608804
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED608804
AccessLevel: 3
PubType: Editorial & Opinion
PubTypeId: editorialOpinion
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Aiming Further: Addressing the Need for High Quality Longitudinal Research in Education
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Watts%2C+Tyler+W%2E%22">Watts, Tyler W.</searchLink><br /><searchLink fieldCode="AR" term="%22Bailey%2C+Drew+H%2E%22">Bailey, Drew H.</searchLink><br /><searchLink fieldCode="AR" term="%22Li%2C+Chen%22">Li, Chen</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2019.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 21
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305A160176
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Opinion Papers<br />Reports - Evaluative
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Longitudinal+Studies%22">Longitudinal Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Quality%22">Educational Quality</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Followup+Studies%22">Followup Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Experiments%22">Educational Experiments</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Finance%22">Educational Finance</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1080/19345747.2019.1644692
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Theories regarding the long-term effects of educational interventions are often assumed, but rarely tested using experimental methods. In the following commentary, we argue that the shortage of randomized control trials with long-term follow-up presents serious problems for the field, as it hampers our ability to develop educational programs that produce long-lasting effects, and creates incentives for the research community to focus too much attention on short-run impacts. We present steps that both researchers and funders could take to substantially improve the educational research literature by investing in educational experiments with long-term follow-up. [This paper was published in "Journal of Research on Educational Evaluation."]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED608804
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED608804
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1080/19345747.2019.1644692
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 21
    Subjects:
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Longitudinal Studies
        Type: general
      – SubjectFull: Educational Quality
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Followup Studies
        Type: general
      – SubjectFull: Educational Experiments
        Type: general
      – SubjectFull: Educational Finance
        Type: general
    Titles:
      – TitleFull: Aiming Further: Addressing the Need for High Quality Longitudinal Research in Education
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Watts, Tyler W.
      – PersonEntity:
          Name:
            NameFull: Bailey, Drew H.
      – PersonEntity:
          Name:
            NameFull: Li, Chen
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 06
              M: 12
              Type: published
              Y: 2019
          Titles:
            – TitleFull: Grantee Submission
              Type: main
ResultId 1