View in EDS HTML Full Text PDF Full Text

Identifying and Measuring the Common Elements of Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder: Development of the 'NDBI-Fi'

Saved in:
Bibliographic Details
Title: Identifying and Measuring the Common Elements of Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder: Development of the 'NDBI-Fi'
Language: English
Authors: Frost, Kyle M. (ORCID 0000-0001-7941-9485), Brian, Jessica, Gengoux, Grace W. (ORCID 0000-0002-6942-9513), Hardan, Antonio, Rieth, Sarah R., Stahmer, Aubyn (ORCID 0000-0002-1596-9848), Ingersoll, Brooke
Source: Autism: The International Journal of Research and Practice. Nov 2020 24(8):2285-2297.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Peer Reviewed: Y
Page Count: 13
Publication Date: 2020
Sponsoring Agency: National Institute on Deafness and Other Communication Disorders (NIDCD)
National Institute of Mental Health (DHHS/NIH)
Institute of Education Sciences (ED)
Congressionally Directed Medical Research Programs (CDMRP) (DOD)
Health Resources and Services Administration (DHHS), Maternal and Child Health Bureau
Contract Number: R21DC01368902
R01MH081757
R324A140005
W81XWH1010586
R40MC27704
Document Type: Journal Articles
Reports - Research
Descriptors: Autism, Pervasive Developmental Disorders, Intervention, Young Children, Behavior Rating Scales, Evidence Based Practice, Psychometrics, Test Reliability, Test Validity, Fidelity, Taxonomy, Motor Development, Children, Cognitive Ability
Assessment and Survey Identifiers: Mullen Scales of Early Learning
DOI: 10.1177/1362361320944011
ISSN: 1362-3613
Abstract: Naturalistic developmental behavioral interventions for young children with autism spectrum disorder share key elements. However, the extent of similarity and overlap in techniques among naturalistic developmental behavioral intervention models has not been quantified, and there is no standardized measure for assessing the implementation of their common elements. This article presents a multi-stage process which began with the development of a taxonomy of elements of naturalistic developmental behavioral interventions. Next, intervention experts identified the common elements of naturalistic developmental behavioral interventions using quantitative methods. An observational rating scheme of those common elements, the eight-item "NDBI-Fi," was developed. Finally, preliminary analyses of the reliability and the validity of the "NDBI-Fi" were conducted using archival data from randomized controlled trials of caregiver-implemented naturalistic developmental behavioral interventions, including 87 post-intervention caregiver-child interaction videos from five sites, as well as 29 pre-post video pairs from two sites. Evaluation of the eight-item "NDBI-Fi" measure revealed promising psychometric properties, including evidence supporting adequate reliability, sensitivity to change, as well as concurrent, convergent, and discriminant validity. Results lend support to the utility of the "NDBI-Fi" as a measure of caregiver implementation of common elements across naturalistic developmental behavioral intervention models. With additional validation, this unique measure has the potential to advance intervention science in autism spectrum disorder by providing a tool which cuts across a class of evidence-based interventions.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2020
Accession Number: EJ1270012
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGFR5ezup45wwuzGpEHgLWCAAAA4TCB3gYJKoZIhvcNAQcGoIHQMIHNAgEAMIHHBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDN4ZIwqvAYFmAC7B1wIBEICBmQtQd8pbaoY_Nno_Quf3CBJgL9CZVIKHs-Be7k8z9l5lZIEScPaUGYpXEbslZKER-QlXR64LgIJsAKLBMQYyfUVgMIjE6BcRbPQ0A8von9Hc9sSzwyQln2ImR7I9NYVpWSJXm29HfLLiRF6vIPiogDonXo6mKGrV7-iDZZG27TB4-STZfpBn4jXQz3ir48YLb6Orw75BU1bw_w==
Text:
  Availability: 1
  Value: <anid>AN0146318042;f9d01nov.20;2020Oct09.03:58;v2.2.500</anid> <title id="AN0146318042-1">Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the NDBI-Fi </title> <p>Naturalistic developmental behavioral interventions for young children with autism spectrum disorder share key elements. However, the extent of similarity and overlap in techniques among naturalistic developmental behavioral intervention models has not been quantified, and there is no standardized measure for assessing the implementation of their common elements. This article presents a multi-stage process which began with the development of a taxonomy of elements of naturalistic developmental behavioral interventions. Next, intervention experts identified the common elements of naturalistic developmental behavioral interventions using quantitative methods. An observational rating scheme of those common elements, the eight-item NDBI-Fi, was developed. Finally, preliminary analyses of the reliability and the validity of the NDBI-Fi were conducted using archival data from randomized controlled trials of caregiver-implemented naturalistic developmental behavioral interventions, including 87 post-intervention caregiver–child interaction videos from five sites, as well as 29 pre–post video pairs from two sites. Evaluation of the eight-item NDBI-Fi measure revealed promising psychometric properties, including evidence supporting adequate reliability, sensitivity to change, as well as concurrent, convergent, and discriminant validity. Results lend support to the utility of the NDBI-Fi as a measure of caregiver implementation of common elements across naturalistic developmental behavioral intervention models. With additional validation, this unique measure has the potential to advance intervention science in autism spectrum disorder by providing a tool which cuts across a class of evidence-based interventions. Naturalistic developmental behavioral interventions for young children with autism spectrum disorder share key elements. However, the extent of similarity between programs within this class of evidence-based interventions is unknown. There is also currently no tool that can be used to measure the implementation of their common elements. This article presents a multi-stage process which began with defining all intervention elements of naturalistic developmental behavioral interventions. Next, intervention experts identified the common elements of naturalistic developmental behavioral interventions using a survey. An observational rating scheme of those common elements, the eight-item NDBI-Fi, was developed. We evaluated the quality of the NDBI-Fi using videos from completed trials of caregiver-implemented naturalistic developmental behavioral interventions. Results showed that the NDBI-Fi measure has promise; it was sensitive to change, related to other similar measures, and demonstrated adequate agreement between raters. This unique measure has the potential to advance intervention science in autism spectrum disorder by providing a tool to measure the implementation of common elements across naturalistic developmental behavioral intervention models. Given that naturalistic developmental behavioral interventions have numerous shared strategies, this may ease clinicians' uncertainty about choosing the "right" intervention package. It also suggests that there may not be a need for extensive training in more than one naturalistic developmental behavioral intervention. Future research should determine whether these common elements are part of other treatment approaches to better understand the quality of services children and families receive as part of usual care.</p> <p>Keywords: autism spectrum disorders; common elements; interventions—psychosocial/behavioral; measure development</p> <hd id="AN0146318042-2">Introduction</hd> <p>Current best practices for the treatment of young children on the autism spectrum include interventions that integrate developmental and behavioral approaches and include caregivers in children's treatment ([<reflink idref="bib35" id="ref1">35</reflink>]; [<reflink idref="bib57" id="ref2">57</reflink>]). There is a growing evidence base for several such manualized interventions, broadly classified as naturalistic developmental behavioral interventions (NDBI), which supports their positive influence on children's development trajectories ([<reflink idref="bib44" id="ref3">44</reflink>]). These interventions embed teaching in naturalistic contexts rather than highly structured environments, and emphasize spontaneous initiation rather than repeated responding to adult-led prompts ([<reflink idref="bib50" id="ref4">50</reflink>]). Despite support for the efficacy of both therapist- and caregiver-implemented NDBIs ([<reflink idref="bib50" id="ref5">50</reflink>]), our knowledge of core intervention elements and treatment mechanisms in these interventions remains limited. Though NDBI developers acknowledge their individual interventions share several common elements despite differing theoretical perspectives ([<reflink idref="bib44" id="ref6">44</reflink>]), the extent to which the models are similar in practice has not been addressed systematically. Furthermore, researchers studying the various models do not articulate or measure these elements in the same way and often identify different components as fundamental to their approach. Thus, researchers and practitioners alike may benefit from additional clarity regarding which specific elements are most effective or necessary for improving outcomes.</p> <p>Development of an intervention taxonomy, or shared set of intervention elements, can support our understanding of evidence-based interventions by providing the field with standardized language and a way to describe and compare intervention ingredients across studies ([<reflink idref="bib7" id="ref7">7</reflink>]; [<reflink idref="bib30" id="ref8">30</reflink>]; [<reflink idref="bib31" id="ref9">31</reflink>]). Identifying common elements across similar evidence-based treatments allows for a more nuanced understanding of how these treatments work. Shifting the unit of analysis from a whole treatment package to individual elements ([<reflink idref="bib6" id="ref10">6</reflink>]) supports the identification of potentially active ingredients of existing NDBIs ([<reflink idref="bib10" id="ref11">10</reflink>]; [<reflink idref="bib49" id="ref12">49</reflink>]). Although common elements are not necessarily responsible for therapeutic change, their inclusion across multiple treatment packages suggests that they may be good candidates to consider in empirical research ([<reflink idref="bib11" id="ref13">11</reflink>]). Accordingly, common elements of evidence-based interventions have been examined in the context of many types of behavioral treatments for children with mental health concerns, including those targeting disruptive behavior disorders ([<reflink idref="bib11" id="ref14">11</reflink>]; [<reflink idref="bib22" id="ref15">22</reflink>]) and parenting skills ([<reflink idref="bib1" id="ref16">1</reflink>]). In addition, identifying common elements can facilitate the development of a standardized measure to better characterize similarities among treatment groups ([<reflink idref="bib12" id="ref17">12</reflink>]), including active treatment and treatment-as-usual control groups.</p> <p>A focus on individual elements of intervention packages may also improve the measurement of treatment fidelity. Measuring treatment fidelity, or adherence to the intervention protocol, is essential for understanding how treatments work, and for interpreting the results of intervention trials ([<reflink idref="bib52" id="ref18">52</reflink>]). However, most reports of treatment fidelity in the literature provide summary ratings, such as overall percent adherence to the entire treatment protocol. NDBI studies rarely link fidelity of specific intervention elements directly to intervention outcomes (see [<reflink idref="bib14" id="ref19">14</reflink>] for a notable exception); therefore, it is unclear how elements contribute to improvements in child social communication. Furthermore, among NDBIs, measures of treatment fidelity used for research often remain unpublished; therefore, limited data exist regarding which strategies contribute to the overall rating. To our knowledge, NDBI intervention fidelity measures have not been examined psychometrically in a published study, which limits the understanding of their validity, reliability across short time intervals, or sensitivity to change. Without common terminology to describe intervention elements and a common measurement tool for reporting fidelity, researchers cannot easily compare intervention elements across studies. This limitation hinders our ability to understand the key elements of NDBIs associated with positive outcomes. Finally, implementation science has recently highlighted the importance of treatment fidelity for establishing and maintaining high-quality services among community providers over time ([<reflink idref="bib17" id="ref20">17</reflink>]). Thus, the development of an NDBI fidelity tool that can guide training for community providers would be extremely helpful.</p> <p>Owing to the fact that best practice in early intervention includes caregiver involvement ([<reflink idref="bib55" id="ref21">55</reflink>]), some NDBIs have been designed specifically for caregiver delivery ([<reflink idref="bib4" id="ref22">4</reflink>]; [<reflink idref="bib20" id="ref23">20</reflink>]), while others have been adapted to caregiver-implemented formats (e.g. [<reflink idref="bib23" id="ref24">23</reflink>], [<reflink idref="bib27" id="ref25">27</reflink>]; [<reflink idref="bib41" id="ref26">41</reflink>]). However, efficacy research of caregiver-implemented NDBIs has been mixed, with some studies finding significant gains (e.g. [<reflink idref="bib2" id="ref27">2</reflink>]; [<reflink idref="bib3" id="ref28">3</reflink>]; [<reflink idref="bib14" id="ref29">14</reflink>]), and others finding null results (e.g. [<reflink idref="bib41" id="ref30">41</reflink>]). Reasons for the null effects remain unclear and could be due to multiple factors, such as a lack of efficacy, individual differences in treatment response, low treatment fidelity for key intervention ingredients, and/or high quality of community care received by control groups. Another factor unique to caregiver-implemented interventions is that caregivers vary in their implementation of intervention strategies both before and after training ([<reflink idref="bib46" id="ref31">46</reflink>]). This suggests that improving the measurement of caregiver intervention fidelity is an important avenue for understanding the efficacy of caregiver-implemented NDBIs.</p> <p>Despite the similarity of key intervention techniques across NDBIs, researchers have not developed a defined set of common intervention elements or a standardized measure for assessing intervention fidelity. This project begins to address these gaps through the following goals: (<reflink idref="bib1" id="ref32">1</reflink>) develop a taxonomy of elements of NDBIs, (<reflink idref="bib2" id="ref33">2</reflink>) identify the common elements across NDBI models, (<reflink idref="bib3" id="ref34">3</reflink>) develop an observational rating scheme to measure the common elements, and (<reflink idref="bib4" id="ref35">4</reflink>) establish preliminary reliability and validity of the new measure with a sample of children with autism spectrum disorder (ASD) and their caregivers who participated in a several randomized controlled trials (RCTs) of different caregiver-implemented NDBI models. Caregiver-implemented models were strategically selected for our preliminary validation sample because, unlike trained therapists in RCTs, caregivers have great variability in their implementation of intervention techniques in control and treatment groups, thus allowing the measurement of the full range of intervention implementation.</p> <hd id="AN0146318042-3">The current study</hd> <p>This research comprised a multi-step process which prioritized content validity in the development and validation of an intervention-independent fidelity measure ([<reflink idref="bib32" id="ref36">32</reflink>]). The steps are depicted in Figure 1. Phase 1 describes the process and results of developing a broad taxonomy of NDBI techniques and the identification of NDBI common elements. Phase 2 describes the subsequent development and evaluation of the <emph>NDBI-Fi</emph>, an observational rating scheme for measuring adherence to the common elements of NDBIs. An observational rating scheme was selected because this approach is considered the gold standard in fidelity measurement in treatment efficacy trials given its potential for providing objective and highly specific information regarding intervention providers in session behavior ([<reflink idref="bib18" id="ref37">18</reflink>]; [<reflink idref="bib33" id="ref38">33</reflink>]). In addition, observational ratings are more likely to be able to detect gradations in quality than indirect (e.g. therapist- or client-report) methods ([<reflink idref="bib43" id="ref39">43</reflink>]), making them potentially useful as a quality improvement tool.</p> <p>Graph: Figure 1. Method flowchart.</p> <hd id="AN0146318042-4">Phase 1</hd> <p></p> <hd id="AN0146318042-5">Method</hd> <p></p> <hd id="AN0146318042-6">Intervention taxonomy</hd> <p>Because our aim was to develop an observational fidelity tool that could measure common elements of NDBI, we began by reviewing individual fidelity measures. We focused on therapeutic content (i.e. NDBI strategies) rather than other potentially important aspects of caregiver-implemented interventions, such as treatment techniques performed by the coach/therapist to help the parent learn and apply the therapeutic content, aspects of the therapeutic alliance, or other treatment parameters. Though the therapist/coach's skills to effectively teach caregivers are crucial in caregiver-implemented interventions, the current study focuses on the specific strategies of individual NDBI models that are directed toward the child. This is not to suggest that these other facets of the intervention, such as quality of coaching, goal setting, and duration of treatment, are not important, but rather that they do not fit within the goal of this study.</p> <p>The first and last author requested published and unpublished NDBI fidelity measures from an expert panel of doctoral-level intervention developers, authors, and experts to develop a broad taxonomy of NDBI elements. Several authors of the [<reflink idref="bib44" id="ref40">44</reflink>] paper, as well as known colleagues who have conducted RCTs of the interventions identified by Schreibman and colleagues in their seminal paper were invited by email to collaborate. Each of these interventions has been examined in a research context and has demonstrated some evidence of efficacy as a therapist-delivered and/or caregiver-implemented intervention ([<reflink idref="bib42" id="ref41">42</reflink>]; [<reflink idref="bib50" id="ref42">50</reflink>]). A total of 11 research teams (14 individuals; 8 interventions) were contacted. One research team did not respond. Interventions examined included <emph>Early Achievements</emph> ([<reflink idref="bib28" id="ref43">28</reflink>]), <emph>Early Start Denver Model</emph> (ESDM; [<reflink idref="bib40" id="ref44">40</reflink>]), <emph>Enhanced Milieu Teaching</emph> (EMT; [<reflink idref="bib23" id="ref45">23</reflink>]; [<reflink idref="bib24" id="ref46">24</reflink>]), <emph>Joint Attention, Symbolic Play, Engagement, and Regulation</emph> (JASPER; [<reflink idref="bib25" id="ref47">25</reflink>], [<reflink idref="bib26" id="ref48">26</reflink>]) <emph>Pivotal Response Training</emph> (PRT; [<reflink idref="bib16" id="ref49">16</reflink>]; [<reflink idref="bib45" id="ref50">45</reflink>]), <emph>Project ImPACT</emph> ([<reflink idref="bib19" id="ref51">19</reflink>]), and <emph>Social Antecedent-Behavior-Consequences</emph> (ABCs) ([<reflink idref="bib4" id="ref52">4</reflink>], [<reflink idref="bib3" id="ref53">3</reflink>]). While the intervention approaches used in this study do not represent a comprehensive list of all interventions that could be characterized as NDBI, those with expertise in the above interventions agreed to collaborate on this endeavor and they represent models commonly used in the literature. Furthermore, we did not examine classroom-based interventions due to the unique features of group instruction and this study's focus on parent–child interactions.</p> <p>The first and last authors established a preliminary taxonomy of intervention elements by examining the content of available NDBI fidelity rating forms (<emph>n</emph> = 9).[<reflink idref="bib7" id="ref54">7</reflink>] The taxonomy was inclusive of intervention-specific elements (i.e. not common across all interventions), as well as those shared among multiple interventions. The process included formally defining each of the elements based on the content of the examined fidelity forms, internally refining the taxonomy over several iterations and generating examples and non-examples for each element to further clarify the definitions. The preliminary taxonomy was then refined using an adapted Delphi method. As per the Delphi method, the expert panel representing the NDBI (identified above) received the preliminary taxonomy and provided open-ended critique and commentary; they were also encouraged to add intervention elements not included in the original taxonomy. Four individuals shared the information with an additional person on their research team to respond in addition to or in place of themselves. A total of 12 individuals responded across all of the seven identified interventions (Table 1). The internal team subsequently revised the definitions and examples, yielding a refined taxonomy of 20 unique intervention elements.</p> <p>Graph</p> <p>Table 1. Number of fidelity tools, expert panel members, and survey respondents per NDBI.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left">Interventions</th><th align="left">Fidelity tools (<italic>n</italic>)</th><th align="left">Expert panel (<italic>n</italic>)</th><th align="left">Survey respondents (<italic>n</italic>)</th></tr></thead><tbody><tr><td>Project ImPACT</td><td>2</td><td>2<xref ref-type="table-fn" rid="tfn2">a</xref></td><td>4</td></tr><tr><td>ESDM</td><td>1</td><td>2</td><td>3</td></tr><tr><td>JASPER</td><td>1</td><td>2</td><td>2</td></tr><tr><td>Social ABCs</td><td>1</td><td>0</td><td>4</td></tr><tr><td>PRT</td><td>2</td><td>4</td><td>3</td></tr><tr><td>EMT</td><td>1</td><td>1</td><td>3</td></tr><tr><td>Early Achievements</td><td>1</td><td>1</td><td>2</td></tr><tr><td>Multiple NDBI</td><td>–</td><td>2</td><td>–</td></tr></tbody></table> </ephtml> </p> <p>1 NDBI: naturalistic developmental behavioral interventions; ESDM: Early Start Denver Model; JASPER: Joint Attention, Symbolic Play, Engagement, and Regulation; PRT: Pivotal Response Training; EMT: Enhanced Milieu Teaching.</p> <p>2 Two experts in Project ImPACT were lead authors.</p> <hd id="AN0146318042-7">Item reduction</hd> <p>Next, a survey was used to obtain quantitative feedback on the refined taxonomy to reduce items to the common elements and increase the content validity of the item set. The members of our expert panel nominated survey respondents who they would consider "experts in their intervention (e.g. past grad students, qualified intervention trainers, etc.)." A total of 25 individuals were nominated, 21 of whom responded to our online survey (85%). The survey presented the text for the 20 elements from the taxonomy described above. Survey respondents rated the extent to which each element was a part of the intervention protocol in which they had expertise, using the following scale (adapted from [<reflink idref="bib29" id="ref55">29</reflink>]):</p> <p></p> <ulist> <item> <emph>Essential</emph>: This item is a component of [intervention], and it is described explicitly in the intervention manual. Interventionists use it consistently during sessions.</item> <p></p> <item> <emph>Useful but non-essential</emph>: This item is a good clinical practice, and interventionists use it when providing [intervention], but it is not described in the intervention manual.</item> <p></p> <item> <emph>Neutral</emph>: I would not discourage the use of this strategy when providing [intervention], but interventionists do not typically use it, and it is not described in the intervention manual.</item> <p></p> <item> <emph>Conflicting</emph>: This item conflicts with the [intervention] intervention protocol. Intervention trainees and caregivers are discouraged from using this strategy.</item> </ulist> <p>This scale was selected because of its distinction between "essential" and "useful but non-essential" elements, which provided information on both manualized and non-manualized intervention elements.</p> <p>Next, content validity ratios (CVRs) were calculated for each item, using the following formula <ephtml> <math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>C</mi><mi>V</mi><mi>R</mi><mo>=</mo><mfrac><mrow><msub><mi>n</mi><mi>e</mi></msub><mo>−</mo><mi>N</mi><mo>/</mo><mn>2</mn></mrow><mrow><mi>N</mi><mo>/</mo><mn>2</mn></mrow></mfrac></mrow></math> </ephtml> , where <emph>n<subs>e</subs></emph> is the number of survey respondents indicating a particular response, and <emph>N</emph> is the total number of respondents ([<reflink idref="bib51" id="ref56">51</reflink>]). Survey responses from up to three individuals per intervention were used to calculate the CVR, with two intervention teams contributing only two responses. Additional responses were dropped from analysis to avoid unequal weighting of any one intervention model over another; individuals with the least amount of self-reported intervention experience were dropped. A total of 19 responses were analyzed, representing all seven NDBIs. Results were identical when all available survey data were used; we opted to present CVR results from 19 cases so that the results would not be biased toward any one intervention package.</p> <p>The CVR, which quantifies consensus, was used to quantitatively evaluate the extent to which each item was characteristic of NDBIs. The published recommended cut-off for achieving statistically significant agreement with our sample size (0.42) was used to determine which items would be retained in the final measure ([<reflink idref="bib29" id="ref57">29</reflink>]; [<reflink idref="bib51" id="ref58">51</reflink>]). CVRs were calculated for each item in two ways: (<reflink idref="bib1" id="ref59">1</reflink>) considering the number of respondents indicating a score of "essential" only and (<reflink idref="bib2" id="ref60">2</reflink>) considering the respondents who indicated a score of "essential" or "useful but non-essential." Examination of items rated as "essential" accounts for techniques specified explicitly in NDBI manuals. The addition of items rated "useful but non-essential" accounts for the fact that clinicians often draw on additional clinical skills when providing intervention beyond what is specified in a treatment manual.</p> <hd id="AN0146318042-8">Results</hd> <p></p> <hd id="AN0146318042-9">Intervention taxonomy</hd> <p>The broad taxonomy consisted of a total of 20 elements with definitions agreed on by our expert panel (Table 2; full definitions in Supplemental Material). Given the differences in terminology often used across NDBI models, these refined definitions may be useful in translating information across research teams and in the community and better defining similarities and differences between interventions.</p> <p>Graph</p> <p>Table 2. CVRs for intervention taxonomy items.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" colspan="2">Item</th><th align="left">Essential or useful</th><th align="left">Essential</th></tr></thead><tbody><tr><td>1</td><td>Face-to-face and on the child's level<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.89</bold></td><td><bold>0.68</bold></td></tr><tr><td>2</td><td>Preparing the activity space</td><td><bold>0.89</bold></td><td>0.37</td></tr><tr><td>3</td><td>Following the child's lead<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.89</bold></td><td><bold>0.89</bold></td></tr><tr><td>4</td><td>Imitating the child</td><td>0.37</td><td>0.05</td></tr><tr><td>5</td><td>Supporting turn-taking</td><td><bold>0.79</bold></td><td>0.26</td></tr><tr><td>6</td><td>Displaying positive affect and animation<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>1.00</bold></td><td><bold>0.68</bold></td></tr><tr><td>7</td><td>Engaging the child in play routines</td><td><bold>0.79</bold></td><td>0.16</td></tr><tr><td>8</td><td>Engaging the child in social routines</td><td><bold>0.79</bold></td><td>−0.37</td></tr><tr><td>9</td><td>Managing problem behavior and dysregulation</td><td><bold>1.00</bold></td><td>0.37</td></tr><tr><td>10</td><td>Modeling appropriate language<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>1.00</bold></td><td><bold>0.58</bold></td></tr><tr><td>11</td><td>Modeling gestures and joint attention</td><td><bold>0.47</bold></td><td>0.05</td></tr><tr><td>12</td><td>Modeling new play acts</td><td><bold>0.79</bold></td><td>0.37</td></tr><tr><td>13</td><td>Responding to attempts to communicate<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.89</bold></td><td><bold>0.89</bold></td></tr><tr><td>14</td><td>Using communicative temptations<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>1.00</bold></td><td><bold>0.79</bold></td></tr><tr><td>15</td><td>Pace and frequency of direct teaching opportunities<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.89</bold></td><td><bold>0.58</bold></td></tr><tr><td>16</td><td>Varying difficulty of direct teaching target</td><td><bold>0.68</bold></td><td>0.05</td></tr><tr><td>17</td><td>Using clear and appropriate teaching opportunities<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.79</bold></td><td><bold>0.79</bold></td></tr><tr><td>18</td><td>Providing motivating and relevant teaching opportunities<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>1.00</bold></td><td><bold>1.00</bold></td></tr><tr><td>19</td><td>Supporting a correct response using prompts<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.68</bold></td><td>0.37</td></tr><tr><td>20</td><td>Providing contingent natural and social reinforcement<xref ref-type="table-fn" rid="tfn3">*</xref></td><td><bold>0.89</bold></td><td><bold>0.79</bold></td></tr></tbody></table> </ephtml> </p> <p>3 Denotes items included in the <emph>NDBI-Fi</emph> measure; bold text denotes items exceeding the statistically significant cut-off of 0.42.</p> <hd id="AN0146318042-10">Item reduction</hd> <p>CVRs for "essential" items only and for "essential" or "useful but non-essential" elements are provided in Table 2. When considering both items rated "essential" and "useful but non-essential," all but one element of the 20 exceeded the cut-off indicating consensus across interventions. When considering only elements rated "essential," 10 of the 20 items exceeded the cut-off indicating consensus. One additional element, which referred to the use of prompting to support the child's response, was examined further and refined based on feedback from the survey respondents. Specifically, some interventions used a specific prompting hierarchy that was precluded based on the original wording of the item; therefore, the prompting item was modified to contain more generic language and was included in the final set of 11 common essential elements. Following the revision of this item, no items were rated as "conflicting" by more than one survey respondent.</p> <hd id="AN0146318042-11">Phase 2</hd> <p></p> <hd id="AN0146318042-12">Method</hd> <p></p> <hd id="AN0146318042-13">Participants</hd> <p>This study involved analyzing existing data from completed or ongoing treatment trials of caregiver-implemented NDBIs with children with ASD aged 7 years or younger. This age range was selected to be consistent with intervention trials of NDBI. Five sites contributed videos of caregiver–child play interactions with representation from four interventions, including Project ImPACT ([<reflink idref="bib19" id="ref61">19</reflink>]; [<reflink idref="bib21" id="ref62">21</reflink>])/ Project ImPACT for Toddlers ([<reflink idref="bib46" id="ref63">46</reflink>], [<reflink idref="bib47" id="ref64">47</reflink>]), JASPER ([<reflink idref="bib25" id="ref65">25</reflink>], [<reflink idref="bib26" id="ref66">26</reflink>]), PRT ([<reflink idref="bib16" id="ref67">16</reflink>]; [<reflink idref="bib45" id="ref68">45</reflink>]), and Social ABCs ([<reflink idref="bib4" id="ref69">4</reflink>], [<reflink idref="bib3" id="ref70">3</reflink>]). This study was approved by the Institutional Review Board (IRB) at Michigan State University, and sharing of videos was approved by IRBs at external study sites. All families consented for their videos to be used for research purposes. The study sample included 87 caregiver–child dyads randomized to either active treatment or control groups. Demographic information is reported in Table 3.</p> <p>Graph</p> <p>Table 3. Participant demographics.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" colspan="3">Children</th><th align="left" colspan="3">Caregivers</th></tr></thead><tbody><tr><td>Child's biological sex</td><td><italic>n</italic></td><td>%</td><td>Caregiver's biological sex</td><td><italic>n</italic></td><td>%</td></tr><tr><td> Male</td><td>71</td><td>81.6</td><td> Male</td><td>8</td><td>9.2</td></tr><tr><td> Female</td><td>16</td><td>18.4</td><td> Female</td><td>79</td><td>90.8</td></tr><tr><td>Child's race</td><td><italic>n</italic></td><td>%</td><td>Mother's highest complete education</td><td><italic>n</italic></td><td>%</td></tr><tr><td> White/Caucasian</td><td>48</td><td>55.2</td><td> Graduate/Professional degree</td><td>23</td><td>26.4</td></tr><tr><td> Black/African American</td><td>7</td><td>8</td><td> Bachelor's degree</td><td>29</td><td>33.3</td></tr><tr><td> Asian/Pacific Islander</td><td>19</td><td>21.8</td><td> Associate's degree</td><td>5</td><td>5.7</td></tr><tr><td> Biracial/mixed race</td><td>2</td><td>2.3</td><td> High school degree/GED</td><td>21</td><td>24.1</td></tr><tr><td> Other</td><td>8</td><td>9.2</td><td> Did not complete high school</td><td>1</td><td>1.1</td></tr><tr><td>Child's ethnicity</td><td><italic>n</italic></td><td>%</td><td> Missing</td><td>8</td><td>9.2</td></tr><tr><td> Non-Hispanic/Latino</td><td>71</td><td>81.6</td><td>Father's highest complete education</td><td><italic>n</italic></td><td>%</td></tr><tr><td> Hispanic/Latino</td><td>15</td><td>17.2</td><td> Graduate/Professional degree</td><td>22</td><td>25.3</td></tr><tr><td>MSEL</td><td><italic>M</italic></td><td><italic>SD</italic></td><td> Bachelor's degree</td><td>14</td><td>16.1</td></tr><tr><td> Average AE (months)</td><td>22.1</td><td>7.6</td><td> Associate's degree</td><td>5</td><td>5.7</td></tr><tr><td>Chronological age (months)</td><td>37.2</td><td>13.8</td><td> High school degree/GED</td><td>12</td><td>13.8</td></tr><tr><td /><td /><td /><td> Did not complete high school</td><td>0</td><td>0.0</td></tr><tr><td /><td /><td /><td> Missing</td><td>34</td><td>39.1</td></tr></tbody></table> </ephtml> </p> <p>4 MSEL: Mullen Scales of Early Learning; SD: standard deviation; AE: age equivalent.</p> <hd id="AN0146318042-14">Measures</hd> <p></p> <hd id="AN0146318042-15">NDBI-Fi</hd> <p>The 11 quantitatively derived "essential" common elements from Phase 1 were used to develop an observational rating scheme and scoring manual for the <emph>NDBI-Fi</emph> measure. The measure used a macro-level rating scheme (i.e. a 1–5 rating scale) to align with many existing fidelity measures (67% of those included in this study) and to increase the likelihood that the measure would not be burdensome or costly to use. The <emph>NDBI-Fi</emph> manual includes practical considerations for rating, item definitions, examples and non-examples, a glossary, and descriptive anchors for assigning ratings. Of the 11 common items from Phase 1, one item specified the <emph>frequency</emph> of direct teaching episodes; these teaching episodes comprise a multi-step procedure based on the principles of operant conditioning with an antecedent-behavior-consequence (ABC) structure. Four additional items focused on the <emph>quality</emph> of direct teaching episodes (<emph>Clear and appropriate, Motivating and relevant, Supporting a correct response</emph>, and <emph>Providing contingent and natural reinforcement</emph>). These were collapsed into a single item, <emph>Quality of direct teaching</emph>, to facilitate ease of coding and ensure that full teaching trials were being scored. Additional items include <emph>Face-to-face and on the child's level, Following the child's lead, Displaying positive affect and animation, Modeling appropriate language, Responding to attempts to communicate</emph>, and <emph>Using communicative temptations</emph>. Thus, the <emph>NDBI-Fi</emph> consists of an eight-item rating scheme (Table 4). The measure is available in the Supplemental Material and from the corresponding author.</p> <p>Graph</p> <p>Table 4. NDBI-Fi item descriptions.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /></colgroup><thead><tr><th align="left">Item</th><th align="left">Brief description</th></tr></thead><tbody><tr><td>1. Face-to-face and on the child's level</td><td>● Child and adult facing each other● Child and adult on similar level</td></tr><tr><td>2. Following the child's lead</td><td>● Child and adult are both active participants in child-chosen activity</td></tr><tr><td>3. Positive affect and animation</td><td>● Adult uses positive affect● Adult matches affect to child's sensory needs</td></tr><tr><td>4. Modeling appropriate language</td><td>● Adult adjusts language to the child's developmental level</td></tr><tr><td>5. Responding to attempts to communicate</td><td>● Adult verbally responds to child's communication behaviors by repeating, clarifying, or expanding</td></tr><tr><td>6. Using communicative temptations</td><td>● Adult creates situations to elicit communication from the child followed by a brief period of expectant waiting</td></tr><tr><td>7. Frequency of direct teaching episodes</td><td>● Adult directs the child to demonstrate new or emerging skills</td></tr><tr><td>8. Quality of direct teaching episodes</td><td>● Adult uses high-quality teaching strategies (e.g. clear instructions, when child is motivated, contingent reinforcement).</td></tr></tbody></table> </ephtml> </p> <p>A total of two raters, including the first author and a research assistant, piloted the rating scheme on a small set of videos to refine the descriptive rating anchors and to achieve inter-rater reliability. One rater was a graduate student with direct intervention experience in three different NDBI models, while the other rater was an undergraduate research assistant without direct intervention experience. Raters discussed scoring differences and refined items and rating anchors to improve clarity and ease of scoring. These two raters independently coded videos and held consensus meetings to discuss discrepancies in ratings until inter-rater reliability was met. Raters were considered reliable when they could rate three consecutive not previously reviewed videos according to the following criteria: (<reflink idref="bib1" id="ref71">1</reflink>) at least seven out of eight items were within 1 point, (<reflink idref="bib2" id="ref72">2</reflink>) no items were greater than 2 points apart, and (<reflink idref="bib3" id="ref73">3</reflink>) the average score was within 0.5 points (i.e. +/−0.25 points). The primary rater was kept blind to treatment condition for all videos; the secondary rater was kept blind to treatment condition when possible (39% of double-coded videos). One rater was involved in data collection for a subset of videos, and as such, blinding of both raters was not possible for these select cases.</p> <hd id="AN0146318042-16">Caregiver–child interaction</hd> <p>Videos included caregiver–child interactions from existing treatment trials. All videos involved an approximately 10-min free-play interaction between the child and the caregiver. Sites selected videos that included English-speaking participants within the treatment and control groups at random, using an online random number generator (https://<ulink href="http://www.random.org/integer-sets">www.random.org/integer-sets</ulink>). A total of 87 post-timepoint videos were collected from five intervention trials (JASPER, Project ImPACT, Project ImPACT for Toddlers, PRT, and Social ABCs), including 54 videos of dyads who received treatment and 33 videos of control participants (i.e. waitlist or treatment-as-usual; Table 5). In addition, 29 pre–post video pairs from two of the sites (Project ImPACT and Social ABCs) were used to examine sensitivity to change.</p> <p>Graph</p> <p>Table 5. Number of videos examined per intervention across group and time point.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" rowspan="2">Intervention</th><th align="left">Pre-intervention</th><th align="left" colspan="2">Post-intervention</th></tr><tr><th align="left">Treatment (<italic>n</italic>)</th><th align="left">Treatment (<italic>n</italic>)</th><th align="left">Control (<italic>n</italic>)</th></tr></thead><tbody><tr><td>Project ImPACT</td><td>24</td><td>24</td><td>9</td></tr><tr><td>Project ImPACT for Toddlers</td><td>0</td><td>8</td><td>9</td></tr><tr><td>PRT</td><td>0</td><td>10</td><td>10</td></tr><tr><td>Social ABCs</td><td>5</td><td>6</td><td>0</td></tr><tr><td>JASPER</td><td>0</td><td>6</td><td>5</td></tr></tbody></table> </ephtml> </p> <p>5 PRT: Pivotal Response Training; JASPER: Joint Attention, Symbolic Play, Engagement and Regulation.</p> <hd id="AN0146318042-17">Established NDBI fidelity</hd> <p>Caregiver treatment adherence using the established fidelity measure for each intervention was available for 76 post-treatment videos (representing Project ImPACT, Project ImPACT for Toddlers, PRT, and Social ABCs). Because intervention fidelity forms utilized different scales (Table 6), scores were transformed as necessary so that all fidelity ratings were on the same scale (with a minimum score of 1 and a maximum score of 5).</p> <p>Graph</p> <p>Table 6. Characteristics of established NDBI fidelity measures.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left">Intervention</th><th align="left"><italic>n</italic> items (subscales)</th><th align="left">Rating scale</th><th align="left">Type of coding</th><th align="left">Example items</th></tr></thead><tbody><tr><td>PRT<xref ref-type="table-fn" rid="tfn7">a</xref></td><td>8 (3)</td><td>0–1</td><td>Interval (1 min)</td><td>• Child choice• Clear opportunity</td></tr><tr><td>PRT<xref ref-type="table-fn" rid="tfn8">b</xref></td><td>6 (0)</td><td>0–1</td><td>Interval (2 min)</td><td>• Contingent reinforcement• Natural reinforcement</td></tr><tr><td>Project ImPACT<xref ref-type="table-fn" rid="tfn9">c</xref></td><td>29 (5)</td><td>1–5</td><td>Global</td><td>• Expands on child's language• Uses clear and appropriate prompts</td></tr><tr><td>Project ImPACT for Toddlers<xref ref-type="table-fn" rid="tfn7">a</xref></td><td>19 (7)</td><td>1–5</td><td>Global</td><td>• Makes comments and avoids questions• Adjusts levels of prompts based on child responding</td></tr><tr><td>Social ABCs</td><td>10 (0)</td><td>0–1</td><td>Interval (1 min)</td><td>• Positive emotion• Shared control</td></tr></tbody></table> </ephtml> </p> <ulist> <item>6 PRT: Pivotal Response Training.</item> <item>7 University of California–San Diego site.</item> <item>8 Stanford University site.</item> <item>9 Michigan State University site.</item> </ulist> <hd id="AN0146318042-18">Mullen Scales of Early Learning</hd> <p>The Mullen Scales of Early Learning (MSEL; [<reflink idref="bib34" id="ref74">34</reflink>]) is a standardized cognitive assessment with four domains that evaluate skills in the domains of visual reception, fine motor, expressive language, and receptive language. The MSEL was administered for all five intervention trials at the study sites. Age equivalent scores across all the four MSEL domains were averaged to obtain an overall estimate of child's developmental level.</p> <hd id="AN0146318042-19">Analysis plan</hd> <p>An exploratory factor analysis was used to evaluate the dimensionality of the <emph>NDBI-Fi</emph>, and Cronbach's alpha was subsequently used to evaluate internal consistency. In addition, two raters coded a total of 52 videos (60%) from three sites. Intraclass correlations (ICCs) were used to evaluate the agreement between coders on individual items as well as overall score. The ICC is the preferred metric for this type of scale; furthermore, it incorporates the magnitude of disagreement into the metric, yielding a more precise estimate of reliability than the metrics of all-or-nothing agreement ([<reflink idref="bib15" id="ref75">15</reflink>]). A single-measure, two-way mixed design based on absolute agreement was used.</p> <p>To address concurrent validity, an independent samples <emph>t</emph>-test was used to determine if caregivers who received training differed from those who did not at the post-intervention timepoint. We hypothesized that caregivers in the active study treatment groups across trials would receive a significantly higher <emph>NDBI-Fi</emph> rating at the end of the treatment phase than caregivers in control groups.</p> <p>Convergent and discriminant validity were examined using Pearson correlation to test the relationship between the <emph>NDBI-Fi</emph> and relevant constructs. We expected that overall ratings for the <emph>Established NDBI Fidelity</emph> would be significantly correlated with the <emph>NDBI-Fi Average Rating</emph> with a medium to large effect size. Next, we expected that the <emph>NDBI-Fi</emph> would not be related (i.e. a small effect size, <emph>r</emph> < 0.2) to child factors, such as child's chronological age or child's developmental age equivalent, which might impact parent–child interactions.</p> <p>For a measure, such as the <emph>NDBI-Fi</emph> to be useful in the context of intervention research, it must capture change in parent behaviors as they learn intervention techniques. To evaluate the sensitivity of the <emph>NDBI-Fi</emph> in capturing change in this context, the available subset of videos of the same dyads' pre- and post-training was rated. This analysis only included dyads in treatment conditions, though the structure and intensity of training offered to caregivers were likely different across sites. A paired sample <emph>t</emph>-test was used to assess for significant change in caregiver use of techniques from pre- to post-training. We expected that, on average, caregivers would score significantly higher on the <emph>NDBI-Fi</emph> after participating in the intervention.</p> <hd id="AN0146318042-20">Results</hd> <p>The <emph>NDBI-Fi</emph> average score (<emph>M</emph> = 3.28, <emph>SD</emph> = 0.75) was adequately normally distributed (Figure 2), with skewness of −0.13 (<emph>SE</emph> = 0.26) and kurtosis of −0.84 (<emph>SE</emph> = 0.51). Some individual items deviated from normality according to skewness and kurtosis values (Table 7), including a low-frequency behavior with positive skew (6. Communicative Temptations) and some high-frequency behaviors with negative skew (e.g. 7. Frequency of Direct Teaching). An exploratory factor analysis of all post-timepoint <emph>NDBI-Fi</emph> ratings was conducted using principal axis factoring. Two factors were extracted with eigenvalues greater than 1 (3.43, 1.12); however, the scree plot demonstrated a clear "elbow" at factor two, suggesting a one-factor solution fits the data best.</p> <p>Graph: Figure 2. Frequency distribution for the NDBI-Fi average rating for treatment cases and control cases.</p> <p>Graph</p> <p>Table 7. Mean, standard deviation, normality, and reliability of NDBI-Fi items at post-intervention.</p> <p> <ephtml> <table><colgroup><col align="left" /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /><col align="char" char="." /></colgroup><thead><tr><th align="left" rowspan="2"><italic>NDBI-Fi</italic> item</th><th align="left" rowspan="2"><italic>M</italic></th><th align="left" rowspan="2"><italic>SD</italic></th><th align="left" colspan="2">Skewness</th><th align="left" colspan="2">Kurtosis</th><th align="left" rowspan="2">ICC</th></tr><tr><th align="left">Statistic</th><th align="left"><italic>SE</italic></th><th align="left">Statistic</th><th align="left"><italic>SE</italic></th></tr></thead><tbody><tr><td>1. Face-to-Face</td><td>2.81</td><td>1.34</td><td>0.28</td><td>0.26</td><td>−1.10</td><td>0.51</td><td>0.82</td></tr><tr><td>2. Follow child's lead</td><td>3.58</td><td>1.36</td><td>−0.60</td><td>0.26</td><td>−0.93</td><td>0.51</td><td>0.63</td></tr><tr><td>3. Positive affect</td><td>3.64</td><td>1.28</td><td>−0.54</td><td>0.26</td><td>−0.93</td><td>0.51</td><td>0.78</td></tr><tr><td>4. Modeling language</td><td>3.17</td><td>1.06</td><td>−0.17</td><td>0.26</td><td>−0.87</td><td>0.51</td><td>0.61</td></tr><tr><td>5. Responding to communication</td><td>3.28</td><td>1.08</td><td>−0.12</td><td>0.26</td><td>−0.63</td><td>0.51</td><td>0.52</td></tr><tr><td>6. Communicative temptations</td><td>1.82</td><td>1.15</td><td>1.22</td><td>0.26</td><td>0.29</td><td>0.51</td><td>0.70</td></tr><tr><td>7. Frequency of direct teaching</td><td>4.05</td><td>1.00</td><td>−1.24</td><td>0.26</td><td>1.50</td><td>0.51</td><td>0.74</td></tr><tr><td>8. Quality of direct teaching</td><td>3.93</td><td>0.89</td><td>−0.60</td><td>0.26</td><td>−0.24</td><td>0.52</td><td>0.33</td></tr><tr><td>Average score</td><td>3.28</td><td>0.75</td><td>−0.13</td><td>0.26</td><td>−0.84</td><td>0.51</td><td>0.80</td></tr></tbody></table> </ephtml> </p> <p>10 SD: standard deviation; ICC: intraclass correlation; SE: standard error.</p> <hd id="AN0146318042-21">Reliability</hd> <p>The eight <emph>NDBI-Fi</emph> items as a scale yielded a Cronbach's alpha of 0.80, thereby demonstrating good internal consistency. Inter-item correlations ranged from 0.11 to 0.65. The single-measures ICC for the <emph>NDBI-Fi</emph> average rating demonstrated excellent reliability ([<reflink idref="bib8" id="ref76">8</reflink>]). Individual item ICCs ranged from 0.33 to 0.82 (Table 6); two items had poor to fair reliability, four items had good reliability, and two items had excellent reliability ([<reflink idref="bib8" id="ref77">8</reflink>]).</p> <hd id="AN0146318042-22">Concurrent validity</hd> <p>An independent samples <emph>t</emph>-test was used to compare post-timepoint ratings for caregivers in the active study treatment groups (<emph>n</emph> = 54) and control groups (<emph>n</emph> = 33). Caregivers who received training (<emph>M</emph> = 3.56, <emph>SD</emph> = 0.69) received higher <emph>NDBI-Fi</emph> average ratings than caregivers in the study control groups on average, with a large effect size (<emph>M</emph> = 2.81, <emph>SD</emph> = 0.62), <emph>t</emph>(<reflink idref="bib85" id="ref78">85</reflink>) = 5.09, <emph>p</emph> < 0.001, <emph>d</emph> = 1.12. However, there was overlap in the frequency distributions of trained and untrained caregivers, with some untrained caregivers demonstrating high fidelity and some trained caregivers demonstrating low fidelity (Figure 2) at the end of the treatment phase.</p> <hd id="AN0146318042-23">Convergent and discriminant validity</hd> <p>A Pearson correlation showed that the <emph>NDBI-Fi</emph> average rating correlated significantly with individual established intervention fidelity with a large effect size (<emph>r</emph> = 0.60, <emph>p</emph> < 0.001). As expected, caregivers who performed the interventions at higher fidelity also received higher ratings on the <emph>NDBI-Fi</emph>. Pearson correlations revealed that the NDBI-Fi average rating did not significantly correlate with either developmental level (<emph>r</emph> = 0.21, p = 0.06) or child's chronological age at the start of the study (<emph>r</emph> = 0.01, p = 0.92).</p> <hd id="AN0146318042-24">Sensitivity to change</hd> <p>Caregivers who received intervention training scored significantly higher at post-intervention on the <emph>NDBI-Fi</emph> average rating (<emph>M</emph> = 3.56, <emph>SD</emph> = 0.61) than at pre-intervention (<emph>M</emph> = 2.89, <emph>SD</emph> = 0.72), <emph>t</emph>(<reflink idref="bib28" id="ref79">28</reflink>) = 4.22, <emph>p</emph> < 0.001, <emph>d</emph> = 1.00.</p> <hd id="AN0146318042-25">Discussion</hd> <p>Various NDBIs for young children with ASD have been independently developed and validated. While researchers acknowledge common elements across these treatments ([<reflink idref="bib44" id="ref80">44</reflink>]), this study represents the first attempt to evaluate the extent to which experts systematically agree that individual elements are shared across manualized treatment packages. In addition, we present preliminary validation data of a unique measure designed to capture caregiver implementation of common intervention techniques across five NDBI trials.</p> <p>Development of the <emph>NDBI-Fi</emph> began with the creation of a taxonomy of NDBI intervention techniques (see Supplemental Material). This collaborative effort yielded a list of 20 defined elements, refined by expert clinical scientists representing seven different NDBIs, with accompanying examples and non-examples to illustrate them. Subsequent findings identified 11 "essential" common elements shared across NDBIs. These included elements, such as being face-to-face and on the child's level, following the child's lead, modeling language, positive affect and animation, responding to the child's attempts to communicate, using communicative temptations, and the frequency and quality of direct teaching episodes. Furthermore, these elements were examined across five trials of four different NDBIs to validate an intervention-independent fidelity measure. The <emph>NDBI-Fi</emph> demonstrated adequate psychometric properties, as well as preliminary evidence of convergent validity and sensitivity to change. Results suggested that the reliability of some items was limited, and attempts should be made to improve these items or adjust coding practices to support higher reliability. In particular, the inter-rater reliability for the <emph>Quality of direct teaching</emph> item suggests the need for further refinement. Although evidence is preliminary, it is our hope that the ongoing development of this measure will help spark innovative research that cuts across interventions by providing a mechanism for measuring the implementation of common elements of NDBIs during intervention trials.</p> <p>The <emph>NDBI-Fi</emph> item development process revealed that many clinical best practices are shared among NDBIs but not necessarily included across all NDBI treatment manuals and fidelity forms. This was indicated by a discrepancy in the number of items for which there was consensus while examining "essential" ratings only (i.e. items explicitly described in the intervention manual; <emph>n</emph> = 11) as compared to a combination of "essential" and "useful but non-essential" ratings (i.e. items implemented but not manualized; <emph>n</emph> = 19). This result suggests that eight of the broad items are commonly implemented while delivering NDBIs regardless of whether these practices are defined in their treatment manuals or fidelity measures. The presence of these common practices may compromise direct comparison of different NDBIs and obscure our understanding of which techniques promote improvement in child outcomes.</p> <p>While we found mean-level group differences in <emph>NDBI-Fi</emph> scores between caregivers with and without training, our data also demonstrated variability within these groups, with some untrained caregivers demonstrating the use of several NDBI strategies and some trained caregivers demonstrating the limited use of strategies. This highlights the fact that many of these intervention strategies are also natural parenting techniques that families may use to some extent without training. However, the extensiveness of implementation likely varies across families. In future research, it will be important to consider how <emph>change</emph> in caregiver fidelity of implementation relates to child improvement, in addition to standard between-group comparisons. In practice, this finding has implications for the use of stepped-care models in caregiver-implemented interventions for ASD ([<reflink idref="bib38" id="ref81">38</reflink>]; [<reflink idref="bib53" id="ref82">53</reflink>]). Caregivers who do not intuitively use many of these strategies may have the most to gain from training and may require a higher level of support to be successful. However, caregivers who do intuitively use some NDBI strategies may benefit from less intensive training, or training targeting other areas of need.</p> <p>Finally, research in implementation science has documented barriers to providing evidence-based interventions in the community for social services more broadly ([<reflink idref="bib36" id="ref83">36</reflink>]; [<reflink idref="bib37" id="ref84">37</reflink>]) and for ASD interventions specifically ([<reflink idref="bib5" id="ref85">5</reflink>]; [<reflink idref="bib39" id="ref86">39</reflink>]; [<reflink idref="bib48" id="ref87">48</reflink>]; [<reflink idref="bib56" id="ref88">56</reflink>]). Research suggests that practitioners have concerns about the use of packaged treatment manuals, perhaps due to the perceived inflexibility of treatment manuals, or difficulty knowing which treatment manual(s) to use at what time. This study demonstrates that NDBIs have numerous shared strategies, which may alleviate clinicians' uncertainty about choosing the "right" intervention package. It also suggests that there may not be a need for extensive training in more than one NDBI, given the demonstrated overlap across treatment models.</p> <hd id="AN0146318042-26">Limitations and future directions</hd> <p>This report constitutes a preliminary validation of the <emph>NDBI-Fi</emph>. Future research should attempt to evaluate this measure across additional NDBIs. Analyses using a greater number of caregiver–child interaction videos and intervention models would allow for a more rigorous assessment of the validity of the measure. This study did not account for the dose and duration of intervention due to limited space and a focus on evaluating the <emph>NDBI-Fi</emph> measure; however, this may be possible in future research. While we found preliminary evidence that the measure was sensitive to change from pre- to post-intervention among caregivers who received NDBI training, a group by time interaction would be a more rigorous test. Particularly given that some caregivers obtained high scores without training, a more in-depth assessment of change is warranted, including shorter term changes and changes that may occur without intervention. Comparing the <emph>NDBI-Fi</emph> and other fidelity measures in terms of their sensitivity to change would be useful to better understand this issue. Furthermore, data on inter-rater reliability suggest that while training someone without direct intervention experience in rating the <emph>NDBI-Fi</emph> can be achieved, it yields reliability estimates that are acceptable but could be improved.</p> <p>In the item reduction stage of Phase 1 of this study, we selected a pool of respondents hand-picked by NDBI developers. We did so because these individuals are intimately familiar with the interventions as they are meant to be delivered. However, in the community, these interventions may be delivered alongside other treatments or merged with other types of treatment elements not considered part of NDBIs. This group of individuals could not speak to how community providers may use these elements, or whether these elements are parts of other types of interventions as well. Future research should attempt to clarify if and how often the techniques we identified are utilized in different intervention approaches, such as more structured approaches based on applied behavior analysis, or those used in special education and speech-language pathology. Understanding the extent to which NDBI intervention elements are part of other treatment approaches is important for understanding what exactly comprises "usual care" early intervention services. Such work is essential for refining our understanding of what constitutes NDBIs as a class of interventions and how they are distinct from other practices in early intervention. However, the <emph>NDBI-Fi</emph> was not designed to evaluate the full breadth of intervention techniques found in other types of interventions and cannot be used to evaluate the quality of such services. Nonetheless, it is our hope that the iterative development process of this measure may prove useful for characterizing other types of treatments as well.</p> <p>This study was limited to examining common elements used across a selection of NDBIs for young children with ASD, and we consider it the first step in an ongoing process of better characterizing and measuring this class of early interventions. It is important to reiterate that these common elements are not necessarily the most important or "active" ingredients responsible for child change; they are simply items that were common across several manualized treatment packages. While NDBIs are acknowledged to have key similarities, it is not known whether they also share active ingredients or exert change in unique ways. Identification of common elements is necessary to determine the unique features of individual interventions as well. To develop the science of NDBIs and better understand the active ingredients and mechanisms through which they exert change, researchers will need to build upon this and other work to understand the full range of treatment elements that comprise these complex interventions. Furthermore, understanding how these treatments work will require the design of creative experimental studies that can examine the causal relationship between implementing specific treatment techniques and child outcomes. For example, single-case experimental designs or group designs (e.g. dismantling trials, factorial experiments), which systematically examine the effects of these common elements, could reveal which, if any, of them contributes to observed changes in child social communication ([<reflink idref="bib9" id="ref89">9</reflink>]; [<reflink idref="bib13" id="ref90">13</reflink>]; [<reflink idref="bib14" id="ref91">14</reflink>]; [<reflink idref="bib54" id="ref92">54</reflink>]). Measurement tools which cut across intervention models are necessary for advancing this goal.</p> <hd id="AN0146318042-27">Supplemental Material</hd> <p>S1-NDBI-Fi_Broad_Item_Definitions_Only.pdf</p> <p>S1-NDBI-Fi_Broad_Item_Definitions_Only – Supplemental material for Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the NDBI-Fi</p> <p></p> <p>Supplemental material, S1-NDBI-Fi_Broad_Item_Definitions_Only for Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the <emph>NDBI-Fi</emph> by Kyle M Frost, Jessica Brian, Grace W Gengoux, Antonio Hardan, Sarah R Rieth, Aubyn Stahmer and Brooke Ingersoll in Autism</p> <p></p> <hd id="AN0146318042-28">Supplemental Material</hd> <p>S2-NDBI-Fi_Common_Items_Manual_2-2019.pdf</p> <p>S2-NDBI-Fi_Common_Items_Manual_2-2019 – Supplemental material for Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the NDBI-Fi</p> <p></p> <p>Supplemental material, S2-NDBI-Fi_Common_Items_Manual_2-2019 for Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the <emph>NDBI-Fi</emph> by Kyle M Frost, Jessica Brian, Grace W Gengoux, Antonio Hardan, Sarah R Rieth, Aubyn Stahmer and Brooke Ingersoll in Autism</p> <p></p> <p>The authors thank the families whose participation in research made this work possible. In addition, they thank Kaylin Russell, who dedicated so much of her time to coding for this project. This project was completed with the expert input and fidelity rating forms of numerous experts in ASD intervention (listed in alphabetical order) and their trainees and colleagues: Susan Bryson, Geraldine Dawson, Helen Flanagan, Ann Kaiser, Connie Kasari, So Hyun Kim, Catherine Lord, Rebecca Landa, Mendy Minjarez, Jennifer Nietfeld, Sally Rogers, Stephanie Shire, and Isabel Smith.</p> <ref id="AN0146318042-29"> <title> References </title> <blist> <bibl id="bib1" idref="ref16" type="bt">1</bibl> <bibtext> Barth R. P., Liggett-Creel K. (2014). Common components of parenting programs for children birth to eight years of age involved with child welfare services. Children and Youth Services Review, 40, 6–12.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref27" type="bt">2</bibl> <bibtext> Bradshaw J., Koegel L. K., Koegel R. L. (2017). Improving functional language and social motivation with a parent-mediated intervention for toddlers with autism spectrum disorder. Journal of Autism and Developmental Disorders, 47(8), 2443–2458. https://doi.org/10.1007/s10803-017-3155-8</bibtext> </blist> <blist> <bibl id="bib3" idref="ref28" type="bt">3</bibl> <bibtext> Brian J. A., Smith I. M., Zwaigenbaum L., Bryson S. E. (2017). Cross-site randomized control trial of the Social ABCs caregiver-mediated intervention for toddlers with autism spectrum disorder: RCT of caregiver-mediated early ASD intervention. Autism Research, 10(10), 1700–1711. https://doi.org/10.1002/aur.1818</bibtext> </blist> <blist> <bibl id="bib4" idref="ref22" type="bt">4</bibl> <bibtext> Brian J. A., Smith I. M., Zwaigenbaum L., Roberts W., Bryson S. E. (2016). The Social ABCs caregiver-mediated intervention for toddlers with autism spectrum disorder: Feasibility, acceptability, and evidence of promise from a multisite study. Autism Research, 9(8), 899–912.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref85" type="bt">5</bibl> <bibtext> Brookman-Frazee L., Stahmer A., Chlebwoski C., Suhrheinrich J., Aarons G. (2016, December). Characterizing implementation processes and influences in two community effectiveness trials for autism [Conference session]. 9th Annual Conference on the Science of Dissemination and Implementation, Washington, DC, United States.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref10" type="bt">6</bibl> <bibtext> Chorpita B. F., Daleiden E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566–579. https://doi.org/10.1037/a0014565</bibtext> </blist> <blist> <bibl id="bib7" idref="ref7" type="bt">7</bibl> <bibtext> Chorpita B. F., Daleiden E. L., Weisz J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7(1), 5–20.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref76" type="bt">8</bibl> <bibtext> Cicchetti D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref89" type="bt">9</bibl> <bibtext> Collins L. M., Trail J. B., Kugler K. C., Baker T. B., Piper M. E., Mermelstein R. J. (2014). Evaluating individual intervention components: Making decisions based on the results of a factorial screening experiment. Translational Behavioral Medicine, 4(3), 238–251. https://doi.org/10.1007/s13142-013-0239-7</bibtext> </blist> <blist> <bibtext> Embry D. D., Biglan A. (2008). Evidence-based Kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychology Review, 11(3), 75–113. https://doi.org/10.1007/s10567-008-0036-x</bibtext> </blist> <blist> <bibtext> Garland A. F., Hawley K. M., Brookman-Frazee L., Hurlburt M. S. (2008). Identifying common elements of evidence-based psychosocial treatments for children's disruptive behavior problems. Journal of the American Academy of Child and Adolescent Psychiatry, 47(5), 505–514. https://doi.org/10.1097/CHI.0b013e31816765c2</bibtext> </blist> <blist> <bibtext> Godfrey E., Chalder T., Ridsdale L., Seed P., Ogden J. (2007). Investigating the active ingredients of cognitive behaviour therapy and counselling for patients with chronic fatigue in primary care: Developing a new process measure to assess treatment fidelity and predict outcome. British Journal of Clinical Psychology, 46(Pt 3), 253–272. https://doi.org/10.1348/014466506x147420</bibtext> </blist> <blist> <bibtext> Guidi J., Brakemeier E.-L., Bockting C. L. H., Cosci F., Cuijpers P., Jarrett R. B., Linden M., Marks I., Peretti C. S., Rafanelli C., Rief W., Schneider S., Schnyder U., Sensky T., Tomba E., Vazquez C., Vieta E., Zipfel S., Wright J. H., Fava G. A. (2018). Methodological recommendations for trials of psychological interventions. Psychotherapy and Psychosomatics, 87(5), 276–284. https://doi.org/10.1159/000490574</bibtext> </blist> <blist> <bibtext> Gulsrud A. C., Hellemann G., Shire S., Kasari C. (2016). Isolating active ingredients in a parent-mediated social communication intervention for toddlers with autism spectrum disorder. Journal of Child Psychology and Psychiatry, 57(5), 606–613. https://doi.org/10.1111/jcpp.12481</bibtext> </blist> <blist> <bibtext> Hallgren K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023</bibtext> </blist> <blist> <bibtext> Hardan A. Y., Gengoux G. W., Berquist K. L., Libove R. A., Ardel C. M., Phillips J., Frazier T. W., Minjarez M. B. (2015). A randomized controlled trial of pivotal response treatment group for parents of children with autism. Journal of Child Psychology and Psychiatry, 56(8), 884–892. https://doi.org/10.1111/jcpp.12354</bibtext> </blist> <blist> <bibtext> Hogue A., Dauber S., Lichvar E., Bobek M., Henderson C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2</bibtext> </blist> <blist> <bibtext> Hogue A., Liddle H. A., Rowe C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy: Theory, Research, Practice, Training, 33(2), 332–345. https://doi.org/10.1037/0033-3204.33.2.332</bibtext> </blist> <blist> <bibtext> Ingersoll B., Dvortcsak A. (2010). Teaching social communication to children with autism: A practitioner's guide to parent training and a manual for parents. Guilford Press.</bibtext> </blist> <blist> <bibtext> Ingersoll B., Dvortcsak A. (2019). Teaching social communication to children with autism and other developmental delays: The project ImPACT guide to coaching parents and the project ImPACT manual for parents (2nd ed.). Guilford Press.</bibtext> </blist> <blist> <bibtext> Ingersoll B., Wainer A. L., Berger N. I., Pickard K. E., Bonter N. (2016). Comparison of a self-directed and therapist-assisted telehealth parent-mediated intervention for children with ASD: A pilot RCT. Journal of Autism and Developmental Disorders, 46(7), 2275–2284. https://doi.org/10.1007/s10803-016-2755-z</bibtext> </blist> <blist> <bibtext> Kaehler L. A., Jacobs M., Jones D. J. (2016). Distilling common history and practice elements to inform dissemination: Hanf-model BPT programs as an example. Clinical Child and Family Psychology Review, 19(3), 236–258. https://doi.org/10.1007/s10567-016-0210-5</bibtext> </blist> <blist> <bibtext> Kaiser A. P., Hancock T. B., Nietfeld J. P. (2000). The effects of parent-implemented enhanced milieu teaching on the social communication of children who have autism. Early Education & Development, 11(4), 423–446. https://doi.org/10.1207/s15566935eed1104_4</bibtext> </blist> <blist> <bibtext> Kaiser A. P., Hester P. P. (1994). Generalized effects of enhanced milieu teaching. Journal of Speech, Language, and Hearing Research, 37(6), 1320–1340.</bibtext> </blist> <blist> <bibtext> Kasari C., Freeman S., Paparella T. (2006). Joint attention and symbolic play in young children with autism: A randomized controlled intervention study: Joint attention and symbolic play in young children with autism. Journal of Child Psychology and Psychiatry, 47(6), 611–620. https://doi.org/10.1111/j.1469-7610.2005.01567.x</bibtext> </blist> <blist> <bibtext> Kasari C., Gulsrud A. C., Wong C., Kwon S., Locke J. (2010). Randomized controlled caregiver mediated joint engagement intervention for toddlers with autism. Journal of Autism and Developmental Disorders, 40(9), 1045–1056. https://doi.org/10.1007/s10803-010-0955-5</bibtext> </blist> <blist> <bibtext> Kasari C., Lawton K., Shih W., Barker T. V., Landa R., Lord C., Orlich F., King B., Wetherby A., Senturk D. (2014). Caregiver-mediated intervention for low-resourced preschoolers with autism: An RCT. Pediatrics, 134, e72–e79. https://doi.org/10.1542/peds.2013-3229</bibtext> </blist> <blist> <bibtext> Landa R. J., Holman K. C., O'Neill A. H., Stuart E. A. (2011). Intervention targeting development of socially synchronous engagement in toddlers with autism spectrum disorder: A randomized controlled trial: RCT of social intervention for toddlers with ASD. Journal of Child Psychology and Psychiatry, 52(1), 13–21. https://doi.org/10.1111/j.1469-7610.2010.02288.x</bibtext> </blist> <blist> <bibtext> Lawshe C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575.</bibtext> </blist> <blist> <bibtext> Lokker C., McKibbon K. A., Colquhoun H., Hempel S. (2015). A scoping review of classification schemes of interventions to promote and integrate evidence into practice in healthcare. Implementation Science, 10(1), Article 27. https://doi.org/10.1186/s13012-015-0220-6</bibtext> </blist> <blist> <bibtext> McHugh R. K., Murray H. W., Barlow D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47(11), 946–953. https://doi.org/10.1016/j.brat.2009.07.005</bibtext> </blist> <blist> <bibtext> McKenzie J. F., Wood M. L., Kotecki J. E., Clark J. K., Brey R. A. (1999). Establishing content validity: Using qualitative and quantitative steps. American Journal of Health Behavior, 23(4), 311–318.</bibtext> </blist> <blist> <bibtext> Mowbray C. T., Holter M. C., Teague G. B., Bybee D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 26, 315–340.</bibtext> </blist> <blist> <bibtext> Mullen E. M. (1995). Mullen scales of early learning. AGS Circle Pines.</bibtext> </blist> <blist> <bibtext> National Research Council. (2001). Educating children with autism. National Academies Press.</bibtext> </blist> <blist> <bibtext> Osterling K. L., Austin M. J. (2008). The dissemination and utilization of research for promoting evidence-based practice. Journal of Evidence-Based Social Work, 5(1–2), 295–319.</bibtext> </blist> <blist> <bibtext> Pagoto S. L., Spring B., Coups E. J., Mulvaney S., Coutu M., Ozakinci G. (2007). Barriers and facilitators of evidence-based practice perceived by behavioral science health professionals. Journal of Clinical Psychology, 63(7), 695–705.</bibtext> </blist> <blist> <bibtext> Phaneuf L., McIntyre L. L. (2011). The application of a three-tier model of intervention to parent training. Journal of Positive Behavior Interventions, 13(4), 198–207. https://doi.org/10.1177/1098300711405337</bibtext> </blist> <blist> <bibtext> Pickard K. E., Kilgore A. N., Ingersoll B. (2016). Using community partnerships to better understand the barriers to using an evidence-based, parent-mediated intervention for autism spectrum disorder in a medicaid system. American Journal of Community Psychology, 57(3–4), 391–403.</bibtext> </blist> <blist> <bibtext> Rogers S. J., Dawson G. (2010). Early start Denver model for young children with autism: Promoting language, learning, and engagement. Guilford Press.</bibtext> </blist> <blist> <bibtext> Rogers S. J., Estes A., Lord C., Vismara L., Winter J., Fitzpatrick A., Guo M., Dawson G. (2012). Effects of a brief Early Start Denver Model (ESDM)–based parent intervention on toddlers at risk for autism spectrum disorders: A randomized controlled trial. Journal of the American Academy of Child & Adolescent Psychiatry, 51(10), 1052–1065. https://doi.org/10.1016/j.jaac.2012.08.003</bibtext> </blist> <blist> <bibtext> Sandbank M., Bottema-Beutel K., Crowley S., Cassidy M., Dunham K., Feldman J. I., Crank J., Albarran S. A., Raj S., Mahbub P., Woynaroski T. G. (2020). Project AIM: Autism intervention meta-analysis for studies of young children. Psychological Bulletin, 146(1), 1–29. https://doi.org/10.1037/bul0000215</bibtext> </blist> <blist> <bibtext> Schoenwald S. K., Garland A. F., Chapman J. E., Frazier S. L., Sheidow A. J., Southam-Gerow M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0</bibtext> </blist> <blist> <bibtext> Schreibman L., Dawson G., Stahmer A. C., Landa R., Rogers S. J., McGee G. G., Kasari C., Ingersoll B., Kaiser A. P., Bruinsma Y., McNerney E., Wetherby A., Halladay A. (2015). Naturalistic developmental behavioral interventions: Empirically validated treatments for autism spectrum disorder. Journal of Autism and Developmental Disorders, 45(8), 2411–2428. https://doi.org/10.1007/s10803-015-2407-8</bibtext> </blist> <blist> <bibtext> Schreibman L., Koegel R. L. (2005). Training for parents of children with autism: Pivotal responses, generalization, and individualization of interventions. In Hibbs E. D., Jensen P. S. (Eds.), Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practice (pp. 605–631). American Psychological Association.</bibtext> </blist> <blist> <bibtext> Stahmer A. C., Brookman-Frazee L., Rieth S. R., Stoner J. T., Feder J. D., Searcy K., Wang T. (2017). Parent perceptions of an adapted evidence-based practice for toddlers with autism in a community setting. Autism, 21(2), 217–230. https://doi.org/10.1177/1362361316637580</bibtext> </blist> <blist> <bibtext> Stahmer A. C., Rieth S. R., Dickson K. S., Feder J., Burgeson M., Searcy K., Brookman-Frazee L. (2020). Project ImPACT for Toddlers: Pilot outcomes of a community adaptation of an intervention for autism risk. Autism, 24, 617–632. https://doi.org/10.1177/1362361319878080</bibtext> </blist> <blist> <bibtext> Suhrheinrich J., Rieth S. R., Dickson K. S., Stahmer A. C. (2020). Exploring associations between inner-context factors and implementation outcomes. Exceptional Children, 86, 155–173.</bibtext> </blist> <blist> <bibtext> Tate D. F., Lytle L. A., Sherwood N. E., Haire-Joshu D., Matheson D., Moore S. M., Loria C. M., Pratt C., Ward D. S., Belle S. H., Michie S. (2016). Deconstructing interventions: Approaches to studying behavior change techniques across obesity interventions. Translational Behavioral Medicine, 6(2), 236–243. https://doi.org/10.1007/s13142-015-0369-1</bibtext> </blist> <blist> <bibtext> Tiede G., Walton K. M. (2019). Meta-analysis of naturalistic developmental behavioral interventions for young children with autism spectrum disorder. Autism, 23, 2080–2095. https://doi.org/10.1177/1362361319836371</bibtext> </blist> <blist> <bibtext> Veneziano L., Hooper J. (1997). A method for quantifying content validity of health-related questionnaires. American Journal of Health Behavior, 21(1), 67–70.</bibtext> </blist> <blist> <bibtext> Wainer A. L., Ingersoll B. (2013). Intervention fidelity: An essential component for understanding ASD parent training research and practice. Clinical Psychology: Science and Practice, 20(3), 335–357.</bibtext> </blist> <blist> <bibtext> Wainer A. L., Ingersoll B. (2015). Increasing access to an ASD imitation intervention via a telehealth parent training program. Journal of Autism and Developmental Disorders, 45(12), 3877–3890.</bibtext> </blist> <blist> <bibtext> Ward-Horner J., Sturmey P. (2010). Component analyses using single-subject experimental designs: A review. Journal of Applied Behavior Analysis, 43(4), 685–704. https://doi.org/10.1901/jaba.2010.43-685</bibtext> </blist> <blist> <bibtext> Wong C., Odom S. L., Hume K. A., Cox A. W., Fettig A., Kucharczyk S., Brock M. E., Plavnick J. B., Fleury V. P., Schultz T. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45(7), 1951–1966. https://doi.org/10.1007/s10803-014-2351-z</bibtext> </blist> <blist> <bibtext> Wood J. J., McLeod B. D., Klebanoff S., Brookman-Frazee L. (2015). Toward the implementation of evidence-based interventions for youth with autism spectrum disorders in schools and community agencies. Behavior Therapy, 46(1), 83–95.</bibtext> </blist> <blist> <bibtext> Zwaigenbaum L., Bauman M. L., Choueiri R., Kasari C., Carter A., Granpeesheh D., Mailloux Z., Smith Roley S., Wagner S., Fein D., Pierce K., Buie T., Davis P. A., Newschaffer C., Robins D., Wetherby A., Stone W. L., Yirmiya N., Estes A.,...Natowicz M. R. (2015). Early intervention for children with autism spectrum disorder under 3 years of age: Recommendations for practice and research. Pediatrics, 136, S60–S81. https://doi.org/10.1542/peds.2014-3667E</bibtext> </blist> </ref> <ref id="AN0146318042-30"> <title> Footnotes </title> <blist> <bibtext> K.M.F. contributed to the study conceptualization, measure development, data coding, analysis, interpretation, and manuscript preparation. J.B., G.W.G., A.H., S.R.R., and A.S. contributed to data analysis, interpretation, and manuscript preparation. B.I. contributed to study conceptualization, measure development, analysis, interpretation, and manuscript preparation.</bibtext> </blist> <blist> <bibtext> The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: B.I. receives royalties from the sale of one of the manuals used in the research. Royalties are donated to the research.</bibtext> </blist> <blist> <bibtext> The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the following grants: Autism Speaks #5773, PI: Hardan; R21DC01368902, PI: Hardan; R01MH081757, PI: Lord; US Department of Education Grant: R324A140005, PI: Stahmer; Autism Speaks Canada ASCFS-2013-13, PI: Brian; Craig Foundation and the Stollery Children's Hospital Foundation, PI: Brian; Congressionally Directed Medical Research Programs W81XWH-10-1-0586, PI: Ingersoll; Health Resources and Services Administration–Maternal and Child Health Bureau R40MC27704, PI: Ingersoll. The content and conclusions are those of the authors and should not be construed as the official position or policy of nor should any endorsements be inferred by HRSA or the US Government.</bibtext> </blist> <blist> <bibtext> This study was approved by the Institutional Review Board (IRB) at Michigan State University and sharing of data was approved by IRBs at external study sites. All participants consented for their data to be used for research purposes.</bibtext> </blist> <blist> <bibtext> Kyle M Frost</bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext>https://orcid.org/0000-0001-7941-9485 Grace W Gengoux</bibtext> </blist> <blist> <bibtext>Graph</bibtext> </blist> <blist> <bibtext>https://orcid.org/0000-0002-6942-9513 Aubyn Stahmer</bibtext> </blist> <blist> <bibtext>Graph https://orcid.org/0000-0002-1596-9848</bibtext> </blist> <blist> <bibtext> Supplemental material for this article is available online.</bibtext> </blist> <blist> <bibtext> For two of the interventions (Project ImPACT and PRT), we obtained fidelity forms from two different research teams.</bibtext> </blist> </ref> <aug> <p>By Kyle M Frost; Jessica Brian; Grace W Gengoux; Antonio Hardan; Sarah R Rieth; Aubyn Stahmer and Brooke Ingersoll</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib35" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib57" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib44" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib50" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib30" firstref="ref8"></nolink> <nolink nlid="nl6" bibid="bib31" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib10" firstref="ref11"></nolink> <nolink nlid="nl8" bibid="bib49" firstref="ref12"></nolink> <nolink nlid="nl9" bibid="bib11" firstref="ref13"></nolink> <nolink nlid="nl10" bibid="bib22" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref17"></nolink> <nolink nlid="nl12" bibid="bib52" firstref="ref18"></nolink> <nolink nlid="nl13" bibid="bib14" firstref="ref19"></nolink> <nolink nlid="nl14" bibid="bib17" firstref="ref20"></nolink> <nolink nlid="nl15" bibid="bib55" firstref="ref21"></nolink> <nolink nlid="nl16" bibid="bib20" firstref="ref23"></nolink> <nolink nlid="nl17" bibid="bib23" firstref="ref24"></nolink> <nolink nlid="nl18" bibid="bib27" firstref="ref25"></nolink> <nolink nlid="nl19" bibid="bib41" firstref="ref26"></nolink> <nolink nlid="nl20" bibid="bib46" firstref="ref31"></nolink> <nolink nlid="nl21" bibid="bib32" firstref="ref36"></nolink> <nolink nlid="nl22" bibid="bib18" firstref="ref37"></nolink> <nolink nlid="nl23" bibid="bib33" firstref="ref38"></nolink> <nolink nlid="nl24" bibid="bib43" firstref="ref39"></nolink> <nolink nlid="nl25" bibid="bib42" firstref="ref41"></nolink> <nolink nlid="nl26" bibid="bib28" firstref="ref43"></nolink> <nolink nlid="nl27" bibid="bib40" firstref="ref44"></nolink> <nolink nlid="nl28" bibid="bib24" firstref="ref46"></nolink> <nolink nlid="nl29" bibid="bib25" firstref="ref47"></nolink> <nolink nlid="nl30" bibid="bib26" firstref="ref48"></nolink> <nolink nlid="nl31" bibid="bib16" firstref="ref49"></nolink> <nolink nlid="nl32" bibid="bib45" firstref="ref50"></nolink> <nolink nlid="nl33" bibid="bib19" firstref="ref51"></nolink> <nolink nlid="nl34" bibid="bib29" firstref="ref55"></nolink> <nolink nlid="nl35" bibid="bib51" firstref="ref56"></nolink> <nolink nlid="nl36" bibid="bib21" firstref="ref62"></nolink> <nolink nlid="nl37" bibid="bib47" firstref="ref64"></nolink> <nolink nlid="nl38" bibid="bib34" firstref="ref74"></nolink> <nolink nlid="nl39" bibid="bib15" firstref="ref75"></nolink> <nolink nlid="nl40" bibid="bib85" firstref="ref78"></nolink> <nolink nlid="nl41" bibid="bib38" firstref="ref81"></nolink> <nolink nlid="nl42" bibid="bib53" firstref="ref82"></nolink> <nolink nlid="nl43" bibid="bib36" firstref="ref83"></nolink> <nolink nlid="nl44" bibid="bib37" firstref="ref84"></nolink> <nolink nlid="nl45" bibid="bib39" firstref="ref86"></nolink> <nolink nlid="nl46" bibid="bib48" firstref="ref87"></nolink> <nolink nlid="nl47" bibid="bib56" firstref="ref88"></nolink> <nolink nlid="nl48" bibid="bib13" firstref="ref90"></nolink> <nolink nlid="nl49" bibid="bib54" firstref="ref92"></nolink>
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1270012
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1270012
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Identifying and Measuring the Common Elements of Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder: Development of the 'NDBI-Fi'
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Frost%2C+Kyle+M%2E%22">Frost, Kyle M.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7941-9485">0000-0001-7941-9485</externalLink>)<br /><searchLink fieldCode="AR" term="%22Brian%2C+Jessica%22">Brian, Jessica</searchLink><br /><searchLink fieldCode="AR" term="%22Gengoux%2C+Grace+W%2E%22">Gengoux, Grace W.</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6942-9513">0000-0002-6942-9513</externalLink>)<br /><searchLink fieldCode="AR" term="%22Hardan%2C+Antonio%22">Hardan, Antonio</searchLink><br /><searchLink fieldCode="AR" term="%22Rieth%2C+Sarah+R%2E%22">Rieth, Sarah R.</searchLink><br /><searchLink fieldCode="AR" term="%22Stahmer%2C+Aubyn%22">Stahmer, Aubyn</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-1596-9848">0000-0002-1596-9848</externalLink>)<br /><searchLink fieldCode="AR" term="%22Ingersoll%2C+Brooke%22">Ingersoll, Brooke</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Autism%3A+The+International+Journal+of+Research+and+Practice%22"><i>Autism: The International Journal of Research and Practice</i></searchLink>. Nov 2020 24(8):2285-2297.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 13
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Institute on Deafness and Other Communication Disorders (NIDCD)<br />National Institute of Mental Health (DHHS/NIH)<br />Institute of Education Sciences (ED)<br />Congressionally Directed Medical Research Programs (CDMRP) (DOD)<br />Health Resources and Services Administration (DHHS), Maternal and Child Health Bureau
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R21DC01368902<br />R01MH081757<br />R324A140005<br />W81XWH1010586<br />R40MC27704
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Autism%22">Autism</searchLink><br /><searchLink fieldCode="DE" term="%22Pervasive+Developmental+Disorders%22">Pervasive Developmental Disorders</searchLink><br /><searchLink fieldCode="DE" term="%22Intervention%22">Intervention</searchLink><br /><searchLink fieldCode="DE" term="%22Young+Children%22">Young Children</searchLink><br /><searchLink fieldCode="DE" term="%22Behavior+Rating+Scales%22">Behavior Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Evidence+Based+Practice%22">Evidence Based Practice</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Taxonomy%22">Taxonomy</searchLink><br /><searchLink fieldCode="DE" term="%22Motor+Development%22">Motor Development</searchLink><br /><searchLink fieldCode="DE" term="%22Children%22">Children</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Ability%22">Cognitive Ability</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Mullen+Scales+of+Early+Learning%22">Mullen Scales of Early Learning</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/1362361320944011
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1362-3613
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Naturalistic developmental behavioral interventions for young children with autism spectrum disorder share key elements. However, the extent of similarity and overlap in techniques among naturalistic developmental behavioral intervention models has not been quantified, and there is no standardized measure for assessing the implementation of their common elements. This article presents a multi-stage process which began with the development of a taxonomy of elements of naturalistic developmental behavioral interventions. Next, intervention experts identified the common elements of naturalistic developmental behavioral interventions using quantitative methods. An observational rating scheme of those common elements, the eight-item "NDBI-Fi," was developed. Finally, preliminary analyses of the reliability and the validity of the "NDBI-Fi" were conducted using archival data from randomized controlled trials of caregiver-implemented naturalistic developmental behavioral interventions, including 87 post-intervention caregiver-child interaction videos from five sites, as well as 29 pre-post video pairs from two sites. Evaluation of the eight-item "NDBI-Fi" measure revealed promising psychometric properties, including evidence supporting adequate reliability, sensitivity to change, as well as concurrent, convergent, and discriminant validity. Results lend support to the utility of the "NDBI-Fi" as a measure of caregiver implementation of common elements across naturalistic developmental behavioral intervention models. With additional validation, this unique measure has the potential to advance intervention science in autism spectrum disorder by providing a tool which cuts across a class of evidence-based interventions.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1270012
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1270012
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/1362361320944011
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 2285
    Subjects:
      – SubjectFull: Autism
        Type: general
      – SubjectFull: Pervasive Developmental Disorders
        Type: general
      – SubjectFull: Intervention
        Type: general
      – SubjectFull: Young Children
        Type: general
      – SubjectFull: Behavior Rating Scales
        Type: general
      – SubjectFull: Evidence Based Practice
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Taxonomy
        Type: general
      – SubjectFull: Motor Development
        Type: general
      – SubjectFull: Children
        Type: general
      – SubjectFull: Cognitive Ability
        Type: general
      – SubjectFull: Mullen Scales of Early Learning
        Type: general
    Titles:
      – TitleFull: Identifying and Measuring the Common Elements of Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder: Development of the 'NDBI-Fi'
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Frost, Kyle M.
      – PersonEntity:
          Name:
            NameFull: Brian, Jessica
      – PersonEntity:
          Name:
            NameFull: Gengoux, Grace W.
      – PersonEntity:
          Name:
            NameFull: Hardan, Antonio
      – PersonEntity:
          Name:
            NameFull: Rieth, Sarah R.
      – PersonEntity:
          Name:
            NameFull: Stahmer, Aubyn
      – PersonEntity:
          Name:
            NameFull: Ingersoll, Brooke
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 11
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 1362-3613
          Numbering:
            – Type: volume
              Value: 24
            – Type: issue
              Value: 8
          Titles:
            – TitleFull: Autism: The International Journal of Research and Practice
              Type: main
ResultId 1