View in EDS HTML Full Text PDF Full Text

Validation of the Observer-Reported Communication Ability (ORCA) Measure for Individuals with Angelman Syndrome

Saved in:
Bibliographic Details
Title: Validation of the Observer-Reported Communication Ability (ORCA) Measure for Individuals with Angelman Syndrome
Language: English
Authors: Zigler, Christina K., Lin, Li, McFatrich, Molly, Lucas, Nicole, Gordon, Kelly L., Jones, Harrison N., Berent, Allyson, Panagoulias, Jennifer, Evans, Paula, Reeve, Bryce B.
Source: American Journal on Intellectual and Developmental Disabilities. May 2023 128(3):204-218.
Availability: American Association on Intellectual and Developmental Disabilities. P.O. Box 1897, Lawrence, KS 66044-1897. Tel: 785-843-1235; Fax: 785-843-1274; e-mail: AJMR@allenpress.com; Web site: https://meridian.allenpress.com/aaidd
Peer Reviewed: Y
Page Count: 15
Publication Date: 2023
Document Type: Journal Articles
Reports - Research
Descriptors: Genetic Disorders, Test Validity, Observation, Communication Skills, Measures (Individuals), Test Reliability
ISSN: 1944-7515
1944-7558
Abstract: There is a critical need for high-quality clinical outcome assessments to capture the important aspects of communication ability of individuals with Angelman syndrome (AS). To center the perspective of caregivers, our team developed the novel Observer-Reported Communication Ability (ORCA) measure using best practice guidelines, with the goal of developing a measure that could be administered to caregivers directly without the need for a certified administrator for use in clinical trials. To refine the draft measure, we conducted two rounds of cognitive interviews with 24 caregivers and a quantitative study including 249 caregivers. The results from both studies support the overall content validity, construct validity, and the reliability of the ORCA measure for individuals with AS[greater than or equal to]2 years old for use in research contexts. Future work should explore the responsiveness of ORCA measures to changes over time in a diverse sample.
Abstractor: As Provided
Entry Date: 2023
Access URL: https://meridian.allenpress.com/ajidd/article-abstract/128/3/204/492586/Validation-of-the-Observer-Reported-Communication?redirectedFrom=fulltext
Accession Number: EJ1387041
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFN4fKnLrm5Sm7GNf8-WF8lAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCsHYEy1AwmP1SfbBgIBEICBmst6P3rSVPd4PRaJXqPr_swawrO685_BybcVT8DtNue-35eqPBPIq4OrUxhaOhErEV1sYbtqJs-2jFRfuvKWMQRQEiEniraLga5xznDEqTswzrklGuM7vqxjM710eBDI0NlXZn_6ysuQtldNx3T6DO49n0nF2ekjKSM9XJ07ixhpoORkzfH8xAl-GTr2SOBObeBeKjX8w6wUFzY=
Text:
  Availability: 1
  Value: <anid>AN0163393725;[8z1j]01may.23;2024Aug16.14:17;v2.2.500</anid> <title id="AN0163393725-1">Validation of the Observer-Reported Communication Ability (ORCA) Measure for Individuals With Angelman Syndrome </title> <p>There is a critical need for high-quality clinical outcome assessments to capture the important aspects of communication ability of individuals with Angelman syndrome (AS). To center the perspective of caregivers, our team developed the novel Observer-Reported Communication Ability (ORCA) measure using best practice guidelines, with the goal of developing a measure that could be administered to caregivers directly without the need for a certified administrator for use in clinical trials. To refine the draft measure, we conducted two rounds of cognitive interviews with 24 caregivers and a quantitative study including 249 caregivers. The results from both studies support the overall content validity, construct validity, and the reliability of the ORCA measure for individuals with AS > 2 years old for use in research contexts. Future work should explore the responsiveness of ORCA measures to changes over time in a diverse sample.</p> <p>Keywords: Angelman syndrome; children; communication; validation; caregivers</p> <p>Angelman syndrome (AS) is a rare, neurodevelopmental disorder that significantly impacts the quality of life of the individual and their family through deficits in a number of areas, including communication ([<reflink idref="bib47" id="ref1">47</reflink>]). For individuals with AS, communication is characterized by minimal to no speech production, limited expressive vocabulary, and greater receptive versus expressive language ability ([<reflink idref="bib20" id="ref2">20</reflink>]; [<reflink idref="bib22" id="ref3">22</reflink>]; [<reflink idref="bib31" id="ref4">31</reflink>]; [<reflink idref="bib32" id="ref5">32</reflink>]). Individuals with AS generally use a wide variety of communication modalities including nonsymbolic (gestures, vocalizations, physical manipulation of others), symbolic (signs, speech), and low- and high-tech augmentative and alternative communication (AAC) devices ([<reflink idref="bib31" id="ref6">31</reflink>]). As a primary domain of daily functioning, communication is extremely important, as it directly impacts quality of life, socialization, safety, caregiving, and clinical care. Thus, it is not surprising that caregivers of individuals with AS have identified communication as a key, family-centered indicator of treatment efficacy for clinical trials ([<reflink idref="bib48" id="ref7">48</reflink>]). It is also ethically imperative to support the development of children's ability to communicate wants, needs, and ideas as effectively and efficiently as possible ([<reflink idref="bib39" id="ref8">39</reflink>]). This includes requesting, refusing, understanding others, and connecting with people in their environment.</p> <p>Existing communication measures have rigorous validity evidence in certain contexts, however, none were developed using input from caregivers and families of individuals with AS and most require a trained clinician to administer and interpret scores. Although useful for their standardization and normative data, assessments designed primarily for identifying developmental delays and guiding clinicians in developmental care plans leave many individuals with AS scoring at the floor of these measures, masking meaningful changes in communication ability (see [<reflink idref="bib22" id="ref9">22</reflink>] for an illustrative example). Additionally, individuals with AS function poorly in unknown environments with strangers due to dyspraxia/apraxia ([<reflink idref="bib15" id="ref10">15</reflink>]) and a focus on verbal speech does not cover the different modalities utilized by these individuals; thus, even a well-administered clinician-reported outcome (ClinRO) or performance-based outcome (PerfO) measure is unlikely to completely capture communication ability in individuals with AS ([<reflink idref="bib7" id="ref11">7</reflink>]; [<reflink idref="bib15" id="ref12">15</reflink>]; [<reflink idref="bib32" id="ref13">32</reflink>]). Thus, the field could benefit from the addition of a high-quality, caregiver-reported measure of communication ability for use alongside other ClinRO and PerfO measures. Ideally, the content of this measure would focus on meaningful and relevant aspects of an individual's communication ability from the perspective of the family, be sensitive enough to detect change over time, and allow for independent administration via caregiver self-report.</p> <p>To meet this need, our team developed the Observer-Reported Communication Ability (ORCA) measure for the AS population using best practice guidelines provided by the U.S. Food and Drug Administration (FDA), The Professional Society for Health Economics and Outcomes Research (ISPOR), the International Society of Quality of Life Outcomes (ISOQOL), the Patient-Reported Outcomes Measurement Information Systems (PROMIS), and other organizations ([<reflink idref="bib10" id="ref14">10</reflink>]; [<reflink idref="bib35" id="ref15">35</reflink>]; [<reflink idref="bib36" id="ref16">36</reflink>]; [<reflink idref="bib42" id="ref17">42</reflink>], [<reflink idref="bib43" id="ref18">43</reflink>]). The first step in this process was to conduct concept elicitation interviews with relevant stakeholders to identify the important components of communication and identify the language caregivers' use when describing their child's communication. We integrated this qualitative data from caregivers of individuals with AS and clinicians, including speech-language pathologists (SLPs), who have experience working with these families, to draft the ORCA measure. The goal of this article was to evaluate the content validity and psychometric properties of the draft ORCA measure in a sample of caregivers of individuals with AS through two phases: cognitive interviewing and a cross-sectional validation study. We used cognitive interviewing (commonly called <emph>debriefing</emph>) to ensure appropriateness of the items and coverage of the important aspects of the construct ([<reflink idref="bib50" id="ref19">50</reflink>], in this case <emph>communication ability</emph>) and to ensure understanding of all components on the measure (e.g., instructions, items, response options, and recall period; [<reflink idref="bib30" id="ref20">30</reflink>]). Our cross-sectional study was designed to quantitatively evaluate the psychometric properties of the measure, including reliability and construct validity (via comparing the ORCA scores to scores with other caregiver-reported measures). Our core study team included measurement methodologists and SLPs, and we were guided by parents and relatives of children with AS via active engagement in regular research team meetings. These individuals are also listed authors on this publication (A.B., J.P., & P.E). The overall goal of our work is to design a caregiver-reported measure that can be used in longitudinal research studies to examine an individual's change over time in communication ability.</p> <hd id="AN0163393725-2">Methods</hd> <p></p> <hd id="AN0163393725-3">Experimental Design</hd> <p>Following established guidelines for developing clinical outcome assessments ([<reflink idref="bib35" id="ref21">35</reflink>]; [<reflink idref="bib44" id="ref22">44</reflink>]), we conducted a two-phase exploration of the psychometric properties of the novel ORCA measure. Phase 1 involved cognitive interviewing with caregivers of children with AS to support content validity ([<reflink idref="bib26" id="ref23">26</reflink>]), and Phase 2 involved a quantitative psychometric evaluation to provide evidence of construct validity and reliability. Both phases were approved by the relevant Institutional Review Board, and the study conformed to standards in the U.S. Federal Policy for the Protection of Human Subjects.</p> <hd id="AN0163393725-4">The Observer-Reported Communication Ability (ORCA) Measure</hd> <p>The ORCA measure is a novel measure designed to assess a child's typical communication ability over the past 30 days from the perspective of their primary caregiver. The ORCA measure includes concepts across expressive, receptive, and pragmatic forms of communication (Table 1). The measure was designed to be completed by a caregiver independently, without the need for a trained administrator present (e.g., SLP or other clinician). The initial content was based off of qualitative interviews with caregivers and clinicians (please see companion article in this issue, "Exploring Communication Ability in Individuals With Angelman Syndrome: Findings From Qualitative Interviews With Caregivers") and published work in early communication development ([<reflink idref="bib3" id="ref24">3</reflink>]; [<reflink idref="bib21" id="ref25">21</reflink>]; [<reflink idref="bib29" id="ref26">29</reflink>]), supporting meaningfulness and relevance to families.</p> <p>PHOTO (COLOR)</p> <p>Prior to cognitive interviewing, the initial version of the ORCA measure consisted of 80 questions/items and the recall period was "currently." Items were organized into communication concepts (e.g., refusal, requesting, responding to their name) and asked about observable communication behaviors that were identified by caregivers in the concept elicitation study. This "concept by behavior" structure is similar to other existing measures of communication ability (e.g., The Communication Matrix, designed for clinical care contexts, [<reflink idref="bib33" id="ref27">33</reflink>]; [<reflink idref="bib37" id="ref28">37</reflink>]). Draft response options were developed from the language caregivers used in concept elicitation interviews when describing their child's communication skills. Please see Table 1 for an example item set for the concept of "refusal."</p> <p>A set of communication-specific descriptive items were also collected in tandem to capture important and detailed information about the individual's unique ways of communicating, including their vocabulary within each modality (e.g., gestures/signs, sounds, words/word approximations, etc.) and complexity of messaging (e.g., combining two symbols together to communicate one message). Although supported through the initial qualitative work, all of the aspects on the initial draft version of the ORCA measure (e.g., instructions, item wording, response options, and recall period) were intended to be evaluated within this study and refined as needed.</p> <hd id="AN0163393725-5">Cognitive Interviewing</hd> <p></p> <hd id="AN0163393725-6">Participants and Recruitment</hd> <p>In-depth cognitive interviews were conducted with adult caregivers of individuals with AS who were able to read, speak, and understand English. Their child had to be at least 2 years of age and caregivers had to report molecular confirmation of their child's diagnosis (e.g., DNA methylation test, FISH, CGH, or sequencing). Caregivers also had to live with the individual with AS and reside in the United States. Recruitment of caregivers was stratified by child age, using categories of 2–7 years, 8–12 years, 13–17 years, and ≥ 18 years, to ensure representation of communication ability across the lifespan. We also sought heterogeneity among the caregivers in terms of their child's AS genotype: deletion positive, mutation/<emph>UBE3A</emph>, imprinting center defect (ICD), or uniparental disomy (UPD). Purposeful sampling was used across genotype to closely represent the national prevalence (as per [<reflink idref="bib4" id="ref29">4</reflink>]). A recruitment flyer was posted on the Facebook page of the Foundation for Angelman Syndrome Therapeutics (FAST), a patient advocacy organization, and was subsequently circulated within other similar forums (parent support groups, etc.).</p> <p>Our team conducted two rounds of cognitive interviewing with 24 caregivers, with 12 in each round. Interviews were conducted between June and October, 2019. Participants received a $50 gift card for participating. A majority of the caregivers were mothers and White (Table 2), with an almost even split between male and female individuals with AS. The majority of individuals with AS had deletion positive genotype (58%), matching national prevalence ([<reflink idref="bib4" id="ref30">4</reflink>]).</p> <p>PHOTO (COLOR)</p> <hd id="AN0163393725-7">Procedures</hd> <p>Two rounds of cognitive interviewing were performed using retrospective probing techniques (as described in [<reflink idref="bib49" id="ref31">49</reflink>], [<reflink idref="bib50" id="ref32">50</reflink>]). The target sample size of 12 caregivers per round (three per age group) meets the recommended sample size for cognitive interviewing ([<reflink idref="bib49" id="ref33">49</reflink>]). Individual interviews were designed to last 60–90 min, and were conducted over the phone. Prior to the interview, study personnel mailed a paper copy of the ORCA measure to the participant with instructions to wait to complete it until the time of the interview. On the call, after obtaining verbal consent, trained interviewers began by providing a brief overview of interview procedures and then asked the participant to complete the ORCA measure. Participants were instructed to circle any questions or words that were hard to understand and complete the measure independently. After completing the ORCA measure, the interviewer then used structured probes to ask about understandability, readability, content, over- and underrepresentation of the items, and performance of the overall measure. Interviewers also evaluated the appropriateness of the recall period, response options, and instructions. All interviews were audio recorded and transcribed. Interviewers completed a debriefing form after each interview to document and summarize main findings.</p> <p>Between the first and second round of interviews, the updated version of the ORCA measure was sent to the FACITtrans Company to review the translatability of the measure into different languages (e.g., Spanish, Chinese). The company provided the study team with a line-by-line review of the ORCA measure, which included a difficulty rating (0 = <emph>no issues</emph>, 1 = <emph>minor difficulty to translate</emph>, 2 = <emph>moderate difficulty</emph>, 3 = <emph>a lot of difficulty</emph>, and 4 = <emph>impossible to translate</emph>). For ratings of 1–3, alternative wordings were provided by the company to improve the likelihood of success in future language translations. This step is recommended as a way to evaluate if a measure can be meaningfully translated in the future ([<reflink idref="bib1" id="ref34">1</reflink>]), but it is not a full translation process, nor is it a formal evaluation of cross-cultural validity.</p> <hd id="AN0163393725-8">Analysis</hd> <p>After each interview, interviewers made note of any problems or issues with the ORCA measure's instructions, items, and response options, and documented their impressions in structured debriefing forms. After the first round of interviews was completed (<emph>n</emph> = 12), team members reviewed debriefing forms and subsequently made decisions on what parts of the ORCA measure required revision. Revisions were discussed among team members, who included measurement experts, SLPs, and caregivers of children with AS. Items were defined as being "significantly revised" if their revision involved (<reflink idref="bib1" id="ref35">1</reflink>) adding or removing a word(s) that changed the meaning of a phrase, (<reflink idref="bib2" id="ref36">2</reflink>) word substitutions that in the judgment of the investigators were more than a semantic simplification, or (<reflink idref="bib3" id="ref37">3</reflink>) significant changes to the response options (e.g., changing from a severity to a frequency scale). Significant changes made after Round 1 (including those indicated by the translatability review) were evaluated in Round 2, and an item-tracking matrix was used to capture all revisions and justifications for changes.</p> <hd id="AN0163393725-9">Readability</hd> <p>We estimated the reading level of the modified text using online software (Readable) to obtain the Flesch-Kincaid Grade Level ([<reflink idref="bib23" id="ref38">23</reflink>]), which provides a numeric score representing "readability" of text that can be interpreted as the United States grade level of education. Consensus is generally that medical materials for adult patients, including outcome measures, should be targeted to a sixth-grade reading level or below (Douglas & Kelly-Campbell, 2018; [<reflink idref="bib12" id="ref39">12</reflink>]; [<reflink idref="bib51" id="ref40">51</reflink>]).</p> <hd id="AN0163393725-10">Psychometric Testing</hd> <p></p> <hd id="AN0163393725-11">Participants</hd> <p>Eligibility criteria for caregivers enrolled in the psychometric study was the same as for cognitive interviewing with the additional criteria that the individual with AS in their care had to be between the ages of 2−40 years old. The age range was broad to encompass communication abilities from a large group of individuals with AS. Although not specifically recruited, participants who participated in the previous qualitative work were eligible to participate in this study.</p> <p>The electronic survey was initiated by 295 caregivers of individuals with AS, with 249 having adequate completeness on their responses on the ORCA measure and thus were included in analyses. The majority of caregivers were women (88.2%) who were on average 41.6 years old (Table 2). Individuals with AS ranged in age from 2–39 years, were an average age of 10.5 years, and there was representation from all genotypes (Table 2). Caregiver education varied, with one reporting less than high school diploma (0.4%), 24 reporting high school degree or equivalent (9.7%), 58 reporting some college/university (23.5%), 105 reporting college/university degree (42.5%), and 59 reporting a postgraduate degree (23.9%). Family income also varied; 50 reported annual household income between 0 and $60,000 (20%); 27 between $60,000 and $80,000 (10.8%); 39 between $80,001 and $100,00 (15.7%); 82 between $100,001 and $250,000 (32.9%); and 26 over $250,000 (10.0%, 23 caregivers chose not to answer this question; 9.2%).</p> <hd id="AN0163393725-12">Procedures</hd> <p>Recruitment was remote and included a web link posted on the patient advocacy organization's Facebook page. Interested participants could click on the link and complete the survey via the Research Electronic Data Capture (REDCap) system. The REDCap survey was active from December 6, 2019 through January 10, 2020. Participants were not compensated for participating in this portion of the study.</p> <p>After providing informed consent, the survey included demographic information from the caregiver about the individual with AS including seizure activity, recent illness, and hospitalizations. Participants were asked to enter their email address in order to receive an email reminder 5 to 12 days later to complete the ORCA measure a second time (to be used to estimate test-retest reliability).</p> <hd id="AN0163393725-13">Measures</hd> <p>Additional measures were also collected to evaluate construct validity. The Communication and Symbolic Behaviors Scale DP Infant-Toddler Checklist (CSBS; [<reflink idref="bib46" id="ref41">46</reflink>]; [<reflink idref="bib45" id="ref42">45</reflink>]) was chosen as a corresponding measure of communication ability. The CSBS was developed as a tool to screen children for delays in early social communication ([<reflink idref="bib46" id="ref43">46</reflink>]) and has published support for validity when used with young children (6–24 months) for this purpose. It was chosen for this study due to its short length (24 items) and because it was specifically developed for caregivers to complete independently without a trained administrator present. It was not designed specifically for older children, however, it does include nonverbal communication behaviors like gestures, facial expressions, and positive affect, which are commonly used by individuals with AS. The CSBS has been previously utilized in AS samples, although published validity evidence has been quantitative and not focused on content validity ([<reflink idref="bib16" id="ref44">16</reflink>]; [<reflink idref="bib17" id="ref45">17</reflink>]). Higher CSBS scores represent better communication ability.</p> <p>To further evaluate construct validity, the study team also included measures of mobility (the PROMIS Parent Proxy Physical Function–Mobility 8-item short form; PROMIS-PF; [<reflink idref="bib18" id="ref46">18</reflink>]) and sleep disturbance (the PROMIS Parent Proxy Sleep Disturbance 8-item short form; PROMIS-Sleep; [<reflink idref="bib13" id="ref47">13</reflink>]). We expected a moderate but positive association between scores on the ORCA and mobility measures. We did not have strong a priori hypotheses about the association of sleep and communication, although we expected correlations to be weaker in magnitude when compared to mobility, and thus, it was considered an "exploratory" indicator. The recall period for both these measures was "the past 7 days."</p> <hd id="AN0163393725-14">Statistical Analyses</hd> <p>We explored structural validity, internal consistency and test-retest reliability, presence of floor/ceiling effects, known-groups validity, construct validity, and an exploration of potential minimal clinically important differences. Details for each analysis are described below.</p> <hd id="AN0163393725-15">Structural Validity</hd> <p>Based on our conceptual framework, we anticipated an overall communication ability construct that included concepts within expressive, receptive, and pragmatic forms of communication. Thus, we used confirmatory factor analysis (CFA) for categorical response data using weighted least squares means and variance adjusted (WLSMV) estimation ([<reflink idref="bib2" id="ref48">2</reflink>]) implemented in MPLUS software [version 8.2]. Fit statistics were evaluated and included the comparative fit index (CFI; >.95 for very good fit), Tucker-Lewis Index (TLI; >.95 for very good fit), root mean square error of approximation (RMSEA, <.06 for very good fit; [<reflink idref="bib25" id="ref49">25</reflink>], pp. 142–145).</p> <hd id="AN0163393725-16">Score Distribution and Reliability</hd> <p>Following confirmation of the factor structure, the ORCA measure was scored based on the caregiver's assessment of an individual's ability to "master" communication behaviors leveled within each of the communication concepts reflective of expressive (e.g., seek attention), receptive (e.g., making choices), and pragmatic (e.g., greeting) forms of communication. Mastery was defined as an individual performing the behavior frequently and consistently over the past 30 days, aligned with the wording of the response options and specified recall period. Graded response item response theory (IRT) modeling was used to calibrate the scale, and the model adjusted for the discrimination and difficulty levels of the concepts ([<reflink idref="bib38" id="ref50">38</reflink>]). Thus, IRT modeling was applied to the 23 communication concepts within the ORCA measure; individual concepts potentially included multiple questions on the ORCA measure. For example, the concept "Respond to Familiar Directions" includes the caregiver's answers to four ORCA questions related to their child's mastery of "stopping" something (question 17a), following "one-step directions" (question 17b), "two-step directions" (question 17c), or "three-step directions" (question 17d, all which indicated differing levels of skill within the concept. IRT scoring used the expected a posteriori (EAP) estimate based on the response patterns observed in the data, with scores transformed to a <emph>T</emph>-score metric with mean 50 and standard deviation of 10.</p> <p>We reported reliability in terms of: (<reflink idref="bib1" id="ref51">1</reflink>) the extent to which the ORCA measures' IRT-based information function showed high information across the range of participant scores; (<reflink idref="bib2" id="ref52">2</reflink>) Cronbach's coefficient alpha for internal consistency (threshold >.70 for group level assessment; [<reflink idref="bib35" id="ref53">35</reflink>]; [<reflink idref="bib41" id="ref54">41</reflink>]); and (<reflink idref="bib3" id="ref55">3</reflink>) test-retest reliability using a two-way mixed effects model on absolute agreement to calculate interclass correlation coefficients (thresholds of.70; [<reflink idref="bib24" id="ref56">24</reflink>]). We also evaluated the percentage of individuals who scored at the lowest (floor) and highest (ceiling) possible ORCA score.</p> <hd id="AN0163393725-17">Known-Groups Validity</hd> <p>We examined difference in ORCA score among AS genotypes (deletion positive, mutation, UPD, and ICD) using a one-way between-group ANOVA and conducted pairwise comparisons on group least-square means. We expected individuals with deletion positive genotype to have lower communication ability scores on the ORCA measure compared to individuals with other genotypes ([<reflink idref="bib11" id="ref57">11</reflink>]; [<reflink idref="bib22" id="ref58">22</reflink>]; [<reflink idref="bib19" id="ref59">19</reflink>]). We also compared ORCA scores between individuals with and without seizures using an independent samples <emph>t</emph> test. We expected children that experienced a seizure in the past year to have lower communication ability scores on the ORCA measure than those who had not.</p> <hd id="AN0163393725-18">Age and ORCA Scores</hd> <p>To explore the relationship between age and ORCA scores, we estimated the Pearson correlation coefficient. Our hypothesis was that there would be small to moderate positive correlations between communication ability and age, as significant variability was expected in ability across children at similar chronological ages. To further explore score ranges within age cohorts, we also report mean, standard deviations, and minimum/maximum for children 2–4 years old, 5–12 years old, and 13–35 years old.</p> <hd id="AN0163393725-19">Construct Validity</hd> <p>We estimated Pearson correlations between the ORCA scores and scores on the CSBS, PROMIS-PF and PROMIS-Sleep. Correlations were classified as: small (0.10–0.29), moderate (0.30–0.49), strong (0.50–0.69), and very strong (> 0.70; [<reflink idref="bib5" id="ref60">5</reflink>], [<reflink idref="bib6" id="ref61">6</reflink>]). We expected the ORCA to have very strong correlations with the CSBS, moderate to strong correlations with PROMIS-PF, and small correlations with PROMIS-sleep.</p> <hd id="AN0163393725-20">Exploration of Minimal Important Differences</hd> <p>We also performed an exploratory analysis of possible minimal important differences (MID) estimates. Using a distribution-based approach, we compared score differences between known groups to standard effect sizes (Cohen's effect size between.2 [small] and.5 [moderate]). Although this data is cross-sectional, previous work suggests that roughly a 2–5 point difference (i.e.,.2–.5 <emph>SDs</emph>) may represent a MID ([<reflink idref="bib27" id="ref62">27</reflink>]). Although this provides an initial estimate of potential MIDs that could be detected, future work using longitudinal data and anchor-based methods would be more appropriate to define what is minimally "important" to stakeholders ([<reflink idref="bib9" id="ref63">9</reflink>]).</p> <hd id="AN0163393725-21">Results</hd> <p></p> <hd id="AN0163393725-22">Cognitive Interviewing</hd> <p></p> <hd id="AN0163393725-23">Round 1</hd> <p>After the first round of cognitive interviewing, data suggested the addition of a more specified recall period and changes to the response options. A detailed description of all modifications made across both rounds of interviews can be found in the item tracking matrix (accessed via https://populationhealth.duke.edu/supplemental-files).</p> <hd id="AN0163393725-24">Recall Period</hd> <p>The first version of the ORCA measure purposely did not have any specified recall period other than "currently" so that interviewers could probe on the period caregivers automatically used when considering their child's communication ability. The vast majority of caregivers reported thinking over a long period of time (a month or more) to reflect "typical" communication. When asked about an appropriate recall period, caregivers reported that one week of communication would not be the best representation of ability as their child's communication could be influenced by changes in routine, illness, or seizure activity. After carefully considering characteristics of communication in these individuals, accurate recall, and potential data collection within a future clinical trial ([<reflink idref="bib28" id="ref64">28</reflink>]), the study team decided to set the recall period at 30 days and evaluate this addition in Round 2. The study team hypothesized that 30 days would be long enough that someone's communication would not be completely overwhelmed by short term issues (e.g., illness), but short enough that the caregiver could accurately recall their child's behavior.</p> <hd id="AN0163393725-25">Response Options</hd> <p>When evaluated in the first round of cognitive interviewing, caregivers reported interpreting "consistently" in varied ways, with most of them translating them to frequencies. For example, one participant stated, "I would say ['yes, but not consistently'] if he grabbed but only some of the time or occasionally ... if it's something their child does every day or their go-to method, then I would say 'yes, consistently.'" Based on feedback from caregivers and through extensive discussions with the study team, the response options were changed to "No or only once," "Sometimes," and "Yes, almost all the time," and evaluated again in Round 2. These response options were designed to represent skill <bold>mastery</bold> ("Yes, almost all the time"), meaning individuals could and did perform the behavior frequently and consistently, and <bold>emerging</bold> skills ("sometimes"), meaning individuals may be starting to use these behaviors more consistently, but not to the same extent as mastered skills.</p> <hd id="AN0163393725-26">Translatability Review</hd> <p>The final report on the ORCA measure showed no major expected issues with future language translations; no wording or phrasing were deemed impossible or extremely difficult to translate. Words deemed "moderately difficult" to translate were changed based on the FACITtrans' recommendations prior to Round 2 of cognitive interviewing so they could be evaluated. Notable changes included changing all pronouns to the third person singular (e.g., from "they" to "he/she") in items referring to the child.</p> <hd id="AN0163393725-27">Round 2</hd> <p>During the second round of cognitive interviewing, interviewers confirmed that questions on the key concepts (e.g., seeking attention, refusing an object) captured the types of communication behaviors caregivers had observed in their children, and supported the recall period of 30 days. Interviewers also asked caregivers to share examples of these behaviors, and feedback from caregivers was consistent with the intended definitions of concepts.</p> <p>Round 2 feedback indicated that the new frequency response options reflected the caregivers' personal conceptualizations of "mastery" in communication skills. In agreement with our formative work (please see companion article in this issue, "Exploring Communication Ability in Individuals With Angelman Syndrome: Findings From Qualitative Interviews With Caregivers"), caregivers reported confidence that their child had mastered a skill when they used it frequently, consistently, and in/across different settings (e.g., home and school). Based on their feedback, the response options of "No or only once," "Sometimes," and "Yes, almost all the time" reflected these observations and generally represented the lack of a skill, the emergence of a skill, and mastery of a skill (respectively). Although there was still some heterogeneity in how participants reported interpreting the response options (particularly around the interpretation of "sometimes"), the study team felt that overall, the new response options performed better than previous options.</p> <hd id="AN0163393725-28">Reading Level</hd> <p>The estimated Flesch-Kincaid grade level was 5, corresponding to a fifth-grade reading level. The most advanced words that were identified were "approximations" (referring to "word approximations") and "conversation."</p> <hd id="AN0163393725-29">Psychometric Testing</hd> <p></p> <hd id="AN0163393725-30">Structural Validity</hd> <p>Based on item fit statistics, two concepts and their related items were identified for possible removal ("telling about the past," "telling about feeling sick"). The concept "telling about feeling sick" had low associations with other communication ability concepts and was removed by the research team without impacting content validity of the ORCA measure (see Discussion section). The concept "telling about the past" represented a very high ability item that was infrequently reported by caregivers, thus model estimates for this item were not stable. A one-factor CFA model including the remaining concepts found evidence for very good model fit; CFI = 0.96; TLI = 0.95; RMSEA = 0.06, 90% CI [0.05–0.07], supporting the reporting of one total score to represent overall "communication ability" across the included concepts.</p> <hd id="AN0163393725-31">Score Distribution and Reliability</hd> <p>The distribution of ORCA total scores was roughly bell-shaped (Figure 1). Only three participants (1.2%) had the lowest possible ORCA score (<emph>T</emph> score = 25.8) and no one was at the ceiling (highest possible = 83.8; highest in sample = 76.4). The IRT-based information function showed high information across the range of participant scores with most of the information aligning with the mean of communication ability (Figure 2). Internal consistency (Cronbach's coefficient alpha) was 0.90. Among the 170 caregivers who completed the ORCA measure at both time points, the test-retest reliability was excellent (ICC = 0.91; 95% CI [0.88–0.93]).</p> <p>PHOTO (COLOR): Figure 1 Distribution of Observer-Reported Communication Ability (ORCA) T-Scores in the Calibration Sample of 249 Individuals With Angelman Syndrome</p> <p>PHOTO (COLOR): Figure 2 Item-Response Theory (IRT) Test Information Function Across the Range of Observer-Reported Communication Ability (ORCA) T-Scores</p> <hd id="AN0163393725-32">Known-Groups Validity</hd> <p>Individuals with Deletion Positive genotype had communication ability scores that were significantly lower than the other genotypes (Table 3). Average scores for the other three genotypes were not statistically different from each other. Additionally, children that experienced a seizure in the past year had significantly lower ORCA scores (<emph>M</emph> = 48.5, <emph>SD</emph> = 9.7) than children who had not experienced a seizure in the past year (<emph>M</emph> = 51.9, <emph>SD</emph> = 9.2; <emph>t</emph> = 2.86, <emph>df</emph> = 246, <emph>p</emph> <.01).</p> <p>PHOTO (COLOR)</p> <hd id="AN0163393725-33">Age and ORCA Scores</hd> <p>The correlation between age and the ORCA score was small, but statistically significant (<emph>r</emph> = 0.15; <emph>p</emph> = 0.021). Average scores within each age group varied, with significant overlap in score range across age categories (Table 3).</p> <hd id="AN0163393725-34">Construct Validity</hd> <p>Total scores on the ORCA measure were strongly correlated with the CSBS total score (<emph>r</emph> =.83, <emph>p</emph> <.001). A moderate-strong correlation was found between ORCA score and PROMIS-PF measure (<emph>r</emph> =.53, <emph>p</emph> <.001). No association was found between the ORCA measure and the PROMIS-Sleep Disturbance measure (<emph>r</emph> = −.09, <emph>p</emph> =.15). There also was no association between PROMIS-Sleep Disturbance scores with the CSBS communication measure (<emph>r</emph> = −.07, <emph>p</emph> =.31).</p> <hd id="AN0163393725-35">Minimal Important Differences</hd> <p>Mean differences in scores between individuals with deletion positive and other genotypes in our sample ranged from 10.4 to 14.0, which exceeded the threshold of 5 (.5 <emph>SD</emph>).</p> <hd id="AN0163393725-36">Discussion</hd> <p>The ORCA measure is a communication ability measure designed for use in clinical trials and centering the perspectives of the caregivers of individuals with AS. The development work was a partnership between advocacy, caregivers, and academia, with the goal of creating a tool that would capture this clinically meaningful domain with this unique patient community in mind. The results from cognitive interviewing and the psychometric evaluation support the overall content validity, construct validity, and the reliability of the measure for this use. Although some expected refinement of concepts occurred within the two-phased evaluations, overall, the content remained stable.</p> <p>As the ORCA measure was specifically designed for use in clinical trials for individuals with AS, we paid particular attention to two aspects of the measure during cognitive interviewing: the recall period and the response options. The recall period was intentionally left ambiguous in the first round of interviews so that we could explore the timeframes that caregivers naturally considered when answering the questions, without undue influence by the study team. Based on caregiver feedback, the final 30-day recall period was long enough to capture a range of typical communication behaviors without reflecting fluctuations in communication based on illness or schedule changes. Additionally, the final recall period was long enough to capture stable and meaningful changes in mastered communication skills while allowing caregivers confidence in their recall of each observable communication behavior.</p> <p>The response options were modified after the first round of cognitive interviewing to better reflect how caregivers discussed emerging versus mastered communication skills. There was heterogeneity in how caregivers were interpreting the initial response options, and some reported confusion on how to map them to concepts. Because most of the caregivers were interpreting the original response options ("consistently" vs. "not consistently") in terms of frequency, this language was used directly.</p> <p>The psychometric results also provided quantitative support for the ORCA measure and the reporting of one total score representing overall communication ability. The score distribution was not skewed and there were minimal floor effects (i.e., only three individuals were at the lowest score) and no ceiling effects in this sample. This is an improvement over existing communication measures that are used with individuals with AS, and supports the idea that this measure is uniquely tailored to the communication ability levels seen within this population. High quality outcome measures are critical for clinical trials, as we are less likely to identify positive shifts in the distribution when it is skewed or there is minimal variability within a sample. There were also minimal concerns based on internal stability, reliability, or construct validity.</p> <p>In terms of content, the psychometric results identified two communication concepts that needed further review by the study team. First, the concept "telling about feeling sick" had poor item performance. This item was originally included in the ORCA measure based on results from our formative work; caregivers and SLPs identified this as a potentially meaningful skill for children in this context (please see companion article in this issue, "Exploring Communication Ability in Individuals With Angelman Syndrome: Findings From Qualitative Interviews With Caregivers"). However, the study team also recognized that these concepts were not always relevant to every family, depending on their priorities for their child and skills that they were targeting/reinforcing. Thus, when it showed poor item performance, the study team felt comfortable that removing these items would not impact the overall content validity of the ORCA measure. Children who can inform caregivers of pain and illness have higher level expressive and receptive skills, and these should be captured through other relevant and better performing items (e.g., directing attention to a body part, answering <emph>yes/no</emph> questions like "does your head hurt?"). The second concept that the item level statistics identified was "telling about the past." This represented a very high ability item that caregivers infrequently reported seeing in their children, thus model estimates for this item were not stable. However, we believed that it was important for content validity to continue to capture this advanced level skill, especially in the context of a clinical trial when improvement in communication is the goal. Thus, at this stage of measure development, we recommend that this item remain as part of the ORCA measure, but will not contribute to the total score without further evaluation.</p> <p>Although our samples were well representative of age cohorts and AS genotypes, the majority of caregivers in our sample were White and had at least some college or graduate education. This is consistent with the current underrepresentation of minority populations in genetic studies and reflects systemic structural barriers that can limit or delay diagnosis and treatment for rare diseases, especially for communities of color ([<reflink idref="bib8" id="ref65">8</reflink>]; [<reflink idref="bib14" id="ref66">14</reflink>]). Future work should continually ensure the ORCA measure's content and psychometric properties are equally supported when used with all families, including those who are most strongly impacted by institutionalized and structural racism. We also limited our recruitment to English-speaking families in the continental United States and required parents to confirm that their child's diagnosis was based on genetic testing. The latter is not available to all families, so it could represent a financial barrier to participation. We also limited the sample to children who were 2 years old and older, and thus, do not recommend the use of the ORCA measure in children below this threshold.</p> <p>The current work, although supportive of the modified ORCA measure, does not fully address all aspects of reliability and validity. Future studies are planned to explore the relationship of the ORCA measure with other performance-based or clinician-reported communication measures, like the Vineland Adaptive Behavior Scales ([<reflink idref="bib40" id="ref67">40</reflink>]), as well as longitudinal validity in existing developmental cohorts and after known interventions. We also recommend a deeper exploration of the relationship between scores and age, as normative values would be useful for interpreting clinical trial results. We also note that the ORCA measure was developed specifically for individuals with AS, but the measure may have applications to other neurodevelopmental disorders that have similar communication profiles.</p> <hd id="AN0163393725-37">Conclusion</hd> <p>The inclusion of high quality and meaningful clinical outcome measures is crucial to ensure upcoming clinical trials capture the outcomes that matter to individuals with AS and their families. The novel ORCA measure was designed to capture expressive, receptive, and pragmatic communication ability for individuals with AS ≥ 2 years of age from the perspective of their caregiver when completed independently, without the need for a trained administrator. The results of our study provide initial evidence supporting the validity and reliability of this measure in this context of use, although evaluations of the responsiveness over time due to changes in communication with diverse samples are still needed. We believe that the ORCA measure can provide meaningful and useful information about the communication ability of an individual with AS when included in future clinical trials. In particular, the addition of a disease-specific caregiver-centered measure, like the ORCA measure, alongside other performance-based and clinician-reported measures could provide a comprehensive assessment of the child's skills and changes over time in response to treatment through the eyes of experts and their family.</p> <p> <emph>This study was presented at the International Society for Quality of Life Research (ISOQOL) 27th Annual Conference (abstract & poster presentation). We would like to express our great appreciation for the caregivers of individuals with Angelman syndrome who participate in this study. This study was funded by the Foundation for Angelman Syndrome Therapeutics (FT2018-005). The authors have developed the technology that is being used in the study. If the technology is commercially successful in the future, the developers and Duke University may benefit financially</emph>.</p> <ref id="AN0163393725-38"> <title> References </title> <blist> <bibl id="bib1" idref="ref34" type="bt">1</bibl> <bibtext> Acquadro, C., Patrick, D. L., Eremenco, S., Martin, M. L., Kuliś, D., Correia, H., Conway, K., & International Society for Quality of Life Research (ISOQOL) Translation and Cultural Adaptation Special Interest Group (TCA-SIG). (2017). Emerging good practices for Translatability Assessment (TA) of Patient-Reported Outcome (PRO) measures. Journal of Patient-Reported Outcomes, 2(1), 8. https://doi.org/10.1186/s41687-018-0035-8</bibtext> </blist> <blist> <bibl id="bib2" idref="ref36" type="bt">2</bibl> <bibtext> Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13(2), 186– 203. https://doi.org/10.1207/s15328007sem1302_2</bibtext> </blist> <blist> <bibl id="bib3" idref="ref24" type="bt">3</bibl> <bibtext> Beuker, K. T., Rommelse, N. N., Donders, R., & Buitelaar, J. K. (2013). Development of early communication skills in the first two years of life. Infant Behavior and Development, 36(1), 71– 83.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref29" type="bt">4</bibl> <bibtext> Clayton-Smith, J., & Laan, L. (2003). Angelman syndrome: A review of the clinical and genetic aspects. Journal of Medical Genetics, 40(2), 87– 95. https://doi.org/10.1136/jmg.40.2.87</bibtext> </blist> <blist> <bibl id="bib5" idref="ref60" type="bt">5</bibl> <bibtext> Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155– 159. https://doi.org/10.1037//0033-2909.112.1.155</bibtext> </blist> <blist> <bibl id="bib6" idref="ref61" type="bt">6</bibl> <bibtext> Cohen, J. (2013). Statistical power analysis for the behavioral sciences. Academic Press.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref11" type="bt">7</bibl> <bibtext> Cousins, M., & Smyth, M. M. (2005). Progression and development in developmental coordination disorder. In Sugden D. & Chambers M. (Eds.), Children with developmental coordination disorde r (pp. 119–134). Whurr Publishers.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref65" type="bt">8</bibl> <bibtext> D'Angelo, C. S., Hermes, A., McMaster, C. R., Prichep, E., Richer, É., van der Westhuizen, F. H., Repetto, G. M., Mengchun, G., Malherbe, H., Reichardt, J. K. V., Arbour, L., Hudson M., du Plessis, K., Haendel, M., Wilcox, P., Lynch, S. A., Rind, S., Easteal, S., Estivill X., Thomas, Y., & Baynam, G. (2020). Barriers and considerations for diagnosing rare diseases in indigenous populations. Frontiers in Pediatrics, 8, 1– 11. https://doi.org/10.3389/fped.2020.579924</bibtext> </blist> <blist> <bibl id="bib9" idref="ref63" type="bt">9</bibl> <bibtext> de Vet, H. C., Terwee, C. B., Ostelo, R. W., Beckerman, H., Knol, D. L., & Bouter, L. M. (2006). Minimal changes in health status questionnaires: Distinction between minimally detectable change and minimally important change. Health and Quality of Life Outcomes, 4(1), 1– 5.</bibtext> </blist> <blist> <bibtext> DeWalt, D. A., Rothrock, N., Yount, S., & Stone, A. A. (2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical Care, 45(5 Suppl 1), S12– S21. https://doi.org/10.1097/01.mlr.0000254567.79743.e2</bibtext> </blist> <blist> <bibtext> Didden, R., Korzilius, H., Duker, P., & Curfs, L. (2004). Communicative functioning in individuals with Angelman syndrome: A comparative study. Disability and Rehabilitation, 26(21–22), 1263– 1267. https://doi.org/10.1080/09638280412331280271</bibtext> </blist> <blist> <bibtext> Eltorai, A. E., Ghanian, S., Adams, C. A., Jr., Born, C. T., & Daniels, A. H. (2014). Readability of patient education materials on the American Association for Surgery of Trauma website. Archives of Trauma Research, 3(2), e18161. https://doi.org/10.5812/atr.18161</bibtext> </blist> <blist> <bibtext> Forrest, C. B., Meltzer, L. J., Marcus, C. L., de la Motte, A., Kratchman, A., Buysse, D. J., Pilkonis, P. A., Becker, B. D., & Bevans, K. B. (2018). Development and validation of the PROMIS Pediatric Sleep Disturbance and Sleep-Related Impairment item banks. Sleep, 41(6), 1– 13. https://doi.org/10.1093/sleep/zsy054</bibtext> </blist> <blist> <bibtext> Fraiman, Y. S., & Wojcik, M. H. (2020). The influence of social determinants of health on the genetic diagnostic odyssey: Who remains undiagnosed, why, and to what effect? Pediatric Research, 89, 295– 300. https://doi.org/10.1038/s41390-020-01151-5</bibtext> </blist> <blist> <bibtext> Grieco, J. C., Bahr, R. H., Schoenberg, M. R., Conover, L., Mackie, L. N., & Weeber, E. J. (2018). Quantitative measurement of communication ability in children with Angelman syndrome. Journal of Applied Rsearch in Intellectual Disabilities, 31, e49– e58. https://doi.org/10.1111/jar.12305</bibtext> </blist> <blist> <bibtext> Hamrick, L. R., & Tonnsen, B. L., (2019). Validating and applying the CSBS-ITC in neurogenetic syndromes. American Journal on Intellectual and Developmental Disabilities., 124(3), 263– 285. https://doi.org/10.1352/1944-7558-124.3.263</bibtext> </blist> <blist> <bibtext> Hamrick, L. R., Haney, A. M., Kelleher, B. L., & Lane, S. P. (2020). Using generalizability theory to evaluate the comparative reliability of developmental measures in neurogenetic syndrome and low-risk populations. Journal of Neurodevelopmental Disorders, 12, 16. https://doi.org/10.1186/s11689-020-09318-1</bibtext> </blist> <blist> <bibtext> Irwin, D. E., Gross, H. E., Stucky, B. D., Thissen, D., DeWitt, E. M., Lai, J. S., Amtmann, D., Khastou, L., Varni, J. W., & DeWalt, D. A. (2012). Development of six PROMIS pediatrics proxy-report item banks. Health and Quality of Life Outcomes, 10, 22. https://doi.org/10.1186/1477-7525-10-22</bibtext> </blist> <blist> <bibtext> Jolleff, N., Emmerson, F., Ryan, M., & McConachie, H. (2006). Communication skills in Angelman syndrome: Matching phenotype to genotype. Advances in Speech Language Pathology, 8(1), 28– 33. https://doi.org/10.1080/14417040500459684</bibtext> </blist> <blist> <bibtext> Jolleff, N., & Ryan, M. M. (1993). Communication development in Angelman's syndrome. Archives of Disease in Childhood, 69(1), 148– 150. https://doi.org/10.1136/adc.69.1.148</bibtext> </blist> <blist> <bibtext> Kaiser A. P., & Roberts M. Y. (2011). Advances in early communication and language intervention. Journal of Early Intervention, 33(4), 298– 309. https://doi.org/10.1177/1053815111429968</bibtext> </blist> <blist> <bibtext> Keute, M., Miller, M. T., Krishnan, M. L., Sadhwani, A., Chamberlain, S., Thibert, R. L., Tan, W., Bird, L. M., & Hipp, J. F. (2020). Angelman syndrome genotypes manifest varying degrees of clinical severity and developmental impairment. Molecular Psychiatry, 26, 3625– 3633. https://doi.org/10.1038/s41380-020-0858-6</bibtext> </blist> <blist> <bibtext> Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel. Research Branch Report 8–75. Chief of Naval Technical Training: Naval Air Station Memphis.</bibtext> </blist> <blist> <bibtext> Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155– 163. https://doi.org/10.1016/j.jcm.2016.02.012</bibtext> </blist> <blist> <bibtext> McDonald, R. (1999). Test theory: A unified treatment. Lawrence Earlbaum.</bibtext> </blist> <blist> <bibtext> Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology, 63(7), 737– 745. https://doi.org/10.1016/j.jclinepi.2010.02.006.</bibtext> </blist> <blist> <bibtext> Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality of life: The remarkable universality of half a standard deviation. Medical Care, 41(5), 582– 592. https://doi.org/10.1097/01.MLR.0000062554.74615.4C</bibtext> </blist> <blist> <bibtext> Norquist, J. M., Girman, C., Fehnel, S., DeMuro-Mercon, C., & Santanello, N. (2012). Choice of recall period for patient-reported outcome (PRO) measures: Criteria for consideration. Quality of Life Research, 21(6), 1013– 1020. https://doi.org/10.1007/s11136-011-0003-8</bibtext> </blist> <blist> <bibtext> Oller, D. K., Eilers, R. E., Neal, A. R., & Schwartz, H. K. (1999). Precursors to speech in infancy: The prediction of speech and language disorders. Journal of Communication Disorders, 32(4), 223– 245.</bibtext> </blist> <blist> <bibtext> Patrick, D. L., Burke, L. B., Gwaltney, C. J., Leidy, N. K., Martin, M. L., Molsen, E., & Ring, L. (2011). Content validity—Establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: Part 2—Assessing respondent understanding. Value in Health: The Journal of the International Society for Pharmacoeconomics and Outcomes Research, 14(8), 978– 988. https://doi.org/10.1016/j.jval.2011.06.013</bibtext> </blist> <blist> <bibtext> Pearson, E., Wilde, L., Heald, M., Royston, R., & Oliver, C. (2019). Communication in Angelman syndrome: A scoping review. Developmental Medicine and Child Neurology, 61, 1266– 1274. https://doi.org/10.1111/dmcn.14257</bibtext> </blist> <blist> <bibtext> Penner, K. A., Johnston, J., Faircloth, B. H., Irish, P., & Williams, C. A. (1993). Communication, cognition, and social interaction in the Angelman syndrome. American Journal of Medical Genetics, 46(1), 34– 39. https://doi.org/10.1002/ajmg.1320460108</bibtext> </blist> <blist> <bibtext> Quinn, E. D., & Rowland, C. (2017). Exploring expressive communication skills in a cross-sectional sample of children and young adults with Angelman syndrome. American Journal of Speech-Language Pathology, 26(2), 369– 382. https://doi.org/10.1044/2016_ajslp-15-0075</bibtext> </blist> <blist> <bibtext> Readable [Computer software].(2022). https://readable.com/</bibtext> </blist> <blist> <bibtext> Reeve, B. B., Wyrwich, K. W., Wu, A. W., Velikova, G., Terwee, C. B., Snyder, C. F., Schwartz, C., Revicki, D. A., Moinpour, C. M., McLeod, L. D., Lyons, J.C., Lenderking, W. R., Hinds, P. S., Hays, R. D., Greenhalgh, J., Gershon, R., Feeny, D., Fayers, P. M., Cella, D., ... Butt, Z. (2013). ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research, 22(8), 1889– 1905. https://doi.org/10.1007/s11136-012-0344-y</bibtext> </blist> <blist> <bibtext> Rothman, M., Burke, L., Erickson, P., Leidy, N. K., Patrick, D. L., & Petrie, C. D. (2009). Use of existing patient-reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluating and documenting content validity for the use of existing instruments and their modification PRO Task Force report. Value Health, 12(8), 1075– 1083. https://doi.org/10.1111/j.1524-4733.2009.00603.x</bibtext> </blist> <blist> <bibtext> Rowland, C., & Fried-Oken, M. (2010). Communication matrix: A clinical and research assessment tool targeting children with severe communication disorders. Journal of Pediatric Rehabilitation Medicine, 3, 319– 329. https://doi.org/10.3233/PRM-2010-0144</bibtext> </blist> <blist> <bibtext> Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. (Psychometrika Monograph, No. 17). Psychometric Society, Richmond, VA.</bibtext> </blist> <blist> <bibtext> Sigafoos, J., Woodyatt, G., Keen, D., Tait, K., Tucker, M., Roberts-Pennell, D., & Pittendreigh, N. (2000). Identifying potential communicative acts in children with developmental and physical disabilities. Communication Disorders Quarterly, 21(2), 77– 86. https://doi.org/10.1177/152574010002100202</bibtext> </blist> <blist> <bibtext> Sparrow, S. S., Cicchetti, D., & Balla, D. A. (2005). Vineland Adaptive Behavior Scales, Second Edition (Vineland-II). APA PsycTests. https://psycnet.apa.org/doi/10.1037/t15164-000</bibtext> </blist> <blist> <bibtext> Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34– 42. https://doi.org/10.1016/j.jclinepi.2006.03.012</bibtext> </blist> <blist> <bibtext> U.S. Department of Health and Human Services Food and Drug Administration/Center for Drug Evaluation and Research.(2009). Guidance for industry: Patient-reported outcome measures: use in medical product development to support labeling claims. https://<ulink href="http://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf">www.fda.gov/downloads/drugs/guidances/ucm193282.pdf</ulink>.</bibtext> </blist> <blist> <bibtext> U.S. Food & Drug Administration.(2019). Patient-focused drug development guidance public workshop on guidance 4: Incorporating clinical outcome assessments into endpoints for regulatory decision-making. https://<ulink href="http://www.fda.gov/media/132505/download">www.fda.gov/media/132505/download</ulink></bibtext> </blist> <blist> <bibtext> U.S. Food & Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, and Center for Devices and Radiological Health.(2006). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims: draft guidance. Health and Quality of Life Outcomes, 4, 79. https://doi.org/10.1186/1477-7525-4-79</bibtext> </blist> <blist> <bibtext> Wetherby, A. M., Allen, L., Cleary, J., Kublin, K., & Goldstein, H. (2002). Validity and reliability of the communication and symbolic behavior scales developmental profile with very young children. Journal of Speech, Language, and Hearing Research, 45(6), 1202– 1218. https://doi.org/10.1044/1092-4388(2002/097)</bibtext> </blist> <blist> <bibtext> Wetherby, A. M., & Prizant, G. (2003). CSBS DP Manual. First normed edition. Brookes.</bibtext> </blist> <blist> <bibtext> Wheeler, A. C., Sacco, P., & Cabo, R. (2017). Unmet clinical needs and burden in Angelman syndrome: A review of the literature. Orphanet Journal of Rare Diseases, 12(1), 164. https://doi.org/10.1186/s13023-017-0716-z</bibtext> </blist> <blist> <bibtext> Willgoss, T., Cassater, D., Connor, S., Krishnan, M. L., Miller, M. T., Dias-Barbosa, C., Phillips, D., McCormack, J., Bird, L. M., Burdine, R. D., Claridge, S., & Bichell, T. J. (2020). Measuring what matters to individuals with Angelman syndrome and their families: Development of a patient-centered disease concept model. Child Psychiatry & Human Development, 52, 654– 668. https://doi.org/10.1007/s10578-020-01051-z</bibtext> </blist> <blist> <bibtext> Willis, G. B. (2004). Cognitive interviewing revisited: A useful technique, in theory?In Presser S., Rothgeb J. M., Couper M. P., Lessler J. T., Martin E., Martin J., & Singer E. (Eds.), Methods for testing and evaluating survey questionnaires (pp.23– 44). John Wiley & Sons.</bibtext> </blist> <blist> <bibtext> Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.</bibtext> </blist> <blist> <bibtext> Wilson, M. (2009). Readability and patient education materials used for low-income populations. Clinical Nurse Specialist, 23, 33– 40.</bibtext> </blist> </ref> <aug> <p>By Christina K. Zigler; Li Lin; Molly McFatrich; Nicole Lucas; Kelly L. Gordon; Harrison N. Jones; Allyson Berent; Jennifer Panagoulias; Paula Evans and Bryce B. Reeve</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib47" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib20" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib22" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib31" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib32" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib48" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib39" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib15" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib10" firstref="ref14"></nolink> <nolink nlid="nl10" bibid="bib35" firstref="ref15"></nolink> <nolink nlid="nl11" bibid="bib36" firstref="ref16"></nolink> <nolink nlid="nl12" bibid="bib42" firstref="ref17"></nolink> <nolink nlid="nl13" bibid="bib43" firstref="ref18"></nolink> <nolink nlid="nl14" bibid="bib50" firstref="ref19"></nolink> <nolink nlid="nl15" bibid="bib30" firstref="ref20"></nolink> <nolink nlid="nl16" bibid="bib44" firstref="ref22"></nolink> <nolink nlid="nl17" bibid="bib26" firstref="ref23"></nolink> <nolink nlid="nl18" bibid="bib21" firstref="ref25"></nolink> <nolink nlid="nl19" bibid="bib29" firstref="ref26"></nolink> <nolink nlid="nl20" bibid="bib33" firstref="ref27"></nolink> <nolink nlid="nl21" bibid="bib37" firstref="ref28"></nolink> <nolink nlid="nl22" bibid="bib49" firstref="ref31"></nolink> <nolink nlid="nl23" bibid="bib23" firstref="ref38"></nolink> <nolink nlid="nl24" bibid="bib12" firstref="ref39"></nolink> <nolink nlid="nl25" bibid="bib51" firstref="ref40"></nolink> <nolink nlid="nl26" bibid="bib46" firstref="ref41"></nolink> <nolink nlid="nl27" bibid="bib45" firstref="ref42"></nolink> <nolink nlid="nl28" bibid="bib16" firstref="ref44"></nolink> <nolink nlid="nl29" bibid="bib17" firstref="ref45"></nolink> <nolink nlid="nl30" bibid="bib18" firstref="ref46"></nolink> <nolink nlid="nl31" bibid="bib13" firstref="ref47"></nolink> <nolink nlid="nl32" bibid="bib25" firstref="ref49"></nolink> <nolink nlid="nl33" bibid="bib38" firstref="ref50"></nolink> <nolink nlid="nl34" bibid="bib41" firstref="ref54"></nolink> <nolink nlid="nl35" bibid="bib24" firstref="ref56"></nolink> <nolink nlid="nl36" bibid="bib11" firstref="ref57"></nolink> <nolink nlid="nl37" bibid="bib19" firstref="ref59"></nolink> <nolink nlid="nl38" bibid="bib27" firstref="ref62"></nolink> <nolink nlid="nl39" bibid="bib28" firstref="ref64"></nolink> <nolink nlid="nl40" bibid="bib14" firstref="ref66"></nolink> <nolink nlid="nl41" bibid="bib40" firstref="ref67"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1387041
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Validation of the Observer-Reported Communication Ability (ORCA) Measure for Individuals with Angelman Syndrome
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Zigler%2C+Christina+K%2E%22">Zigler, Christina K.</searchLink><br /><searchLink fieldCode="AR" term="%22Lin%2C+Li%22">Lin, Li</searchLink><br /><searchLink fieldCode="AR" term="%22McFatrich%2C+Molly%22">McFatrich, Molly</searchLink><br /><searchLink fieldCode="AR" term="%22Lucas%2C+Nicole%22">Lucas, Nicole</searchLink><br /><searchLink fieldCode="AR" term="%22Gordon%2C+Kelly+L%2E%22">Gordon, Kelly L.</searchLink><br /><searchLink fieldCode="AR" term="%22Jones%2C+Harrison+N%2E%22">Jones, Harrison N.</searchLink><br /><searchLink fieldCode="AR" term="%22Berent%2C+Allyson%22">Berent, Allyson</searchLink><br /><searchLink fieldCode="AR" term="%22Panagoulias%2C+Jennifer%22">Panagoulias, Jennifer</searchLink><br /><searchLink fieldCode="AR" term="%22Evans%2C+Paula%22">Evans, Paula</searchLink><br /><searchLink fieldCode="AR" term="%22Reeve%2C+Bryce+B%2E%22">Reeve, Bryce B.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22American+Journal+on+Intellectual+and+Developmental+Disabilities%22"><i>American Journal on Intellectual and Developmental Disabilities</i></searchLink>. May 2023 128(3):204-218.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: American Association on Intellectual and Developmental Disabilities. P.O. Box 1897, Lawrence, KS 66044-1897. Tel: 785-843-1235; Fax: 785-843-1274; e-mail: AJMR@allenpress.com; Web site: https://meridian.allenpress.com/aaidd
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Genetic+Disorders%22">Genetic Disorders</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Observation%22">Observation</searchLink><br /><searchLink fieldCode="DE" term="%22Communication+Skills%22">Communication Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Measures+%28Individuals%29%22">Measures (Individuals)</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1944-7515<br />1944-7558
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: There is a critical need for high-quality clinical outcome assessments to capture the important aspects of communication ability of individuals with Angelman syndrome (AS). To center the perspective of caregivers, our team developed the novel Observer-Reported Communication Ability (ORCA) measure using best practice guidelines, with the goal of developing a measure that could be administered to caregivers directly without the need for a certified administrator for use in clinical trials. To refine the draft measure, we conducted two rounds of cognitive interviews with 24 caregivers and a quantitative study including 249 caregivers. The results from both studies support the overall content validity, construct validity, and the reliability of the ORCA measure for individuals with AS[greater than or equal to]2 years old for use in research contexts. Future work should explore the responsiveness of ORCA measures to changes over time in a diverse sample.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://meridian.allenpress.com/ajidd/article-abstract/128/3/204/492586/Validation-of-the-Observer-Reported-Communication?redirectedFrom=fulltext" linkWindow="_blank">https://meridian.allenpress.com/ajidd/article-abstract/128/3/204/492586/Validation-of-the-Observer-Reported-Communication?redirectedFrom=fulltext</link>
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1387041
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1387041
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
        StartPage: 204
    Subjects:
      – SubjectFull: Genetic Disorders
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Observation
        Type: general
      – SubjectFull: Communication Skills
        Type: general
      – SubjectFull: Measures (Individuals)
        Type: general
      – SubjectFull: Test Reliability
        Type: general
    Titles:
      – TitleFull: Validation of the Observer-Reported Communication Ability (ORCA) Measure for Individuals with Angelman Syndrome
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Zigler, Christina K.
      – PersonEntity:
          Name:
            NameFull: Lin, Li
      – PersonEntity:
          Name:
            NameFull: McFatrich, Molly
      – PersonEntity:
          Name:
            NameFull: Lucas, Nicole
      – PersonEntity:
          Name:
            NameFull: Gordon, Kelly L.
      – PersonEntity:
          Name:
            NameFull: Jones, Harrison N.
      – PersonEntity:
          Name:
            NameFull: Berent, Allyson
      – PersonEntity:
          Name:
            NameFull: Panagoulias, Jennifer
      – PersonEntity:
          Name:
            NameFull: Evans, Paula
      – PersonEntity:
          Name:
            NameFull: Reeve, Bryce B.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Type: published
              Y: 2023
          Identifiers:
            – Type: issn-print
              Value: 1944-7515
            – Type: issn-electronic
              Value: 1944-7558
          Numbering:
            – Type: volume
              Value: 128
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: American Journal on Intellectual and Developmental Disabilities
              Type: main
ResultId 1