View in EDS HTML Full Text PDF Full Text

Improving the Measurement of School Climate Using Item Response Theory

Saved in:
Bibliographic Details
Title: Improving the Measurement of School Climate Using Item Response Theory
Language: English
Authors: Lindstrom Johnson, Sarah, Reichenberg, Ray E., Shukla, Kathan, Waasdorp, Tracy E., Bradshaw, Catherine P.
Source: Grantee Submission. 2019.
Peer Reviewed: Y
Page Count: 31
Publication Date: 2019
Sponsoring Agency: Department of Education (ED)
Institute of Education Sciences (ED)
National Institute of Justice (NIJ) (DOJ)
Contract Number: R305H150027
2014CKBX0005
Document Type: Reports - Research
Education Level: Elementary Secondary Education
Secondary Education
Descriptors: Item Response Theory, Educational Environment, Accountability, Educational Legislation, Federal Legislation, Elementary Secondary Education, Measurement Techniques, Institutional Characteristics, Secondary School Students, Test Items, Item Analysis, Academic Achievement, Institutional Evaluation, Secondary Schools
Laws, Policies and Program Identifiers: Every Student Succeeds Act 2015
DOI: 10.1111/emip.12296
Abstract: The United States government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the "Every Student Succeeds Act". Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item-response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n= 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the 'mirt' package in R using unidimensional item response theory. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure. [This paper was published in "Educational Measurement: Issues and Practice" v38 n4 2019 (EJ1236481).]
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2020
Accession Number: ED604048
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFCA-lziiqqr_Z2LVYikKV7AAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDCHWccNHWCBviKgMuQIBEICBmqiAnJNRua8zxYtHy_rUYX2IUM8f5HVWF8GVj9EwZO4OFHPkmX_79mcEfddsZpbr_ifGO7O5CFM6MZVWOU2-ETvf4ghGEUtd-DQxBWmMkRZ8LhAF9wOti2GTJWZ9nh_eKbj8C2aJLBpNBPfrFfDsOKfVMU8QhsVtP9cCo5MYILiJ2IRUlwXLZxk6oLmspjEuEF_IDEvdecVBQw8=
Text:
  Availability: 1
  Value: <anid>AN0140917066;ems01dec.19;2019Dec30.03:26;v2.2.500</anid> <title id="AN0140917066-1">Improving the Measurement of School Climate Using Item Response Theory </title> <p>The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.</p> <p>Keywords: item response theory; measurement invariance; school accountability; school climate</p> <p>School climate has been defined as the quality and character of school life and relates to norms, values, and expectations that foster supportive environments and feelings of engagement and safety. A favorable school climate has been associated with both improved behavioral and academic outcomes for students including increased academic achievement and reduced suspension, absenteeism, truancy, dropout, drug use, and violent and aggressive behavior (for a review, see Thapa, Cohen, Guffey, & Higgins‐D'Alessandro, [<reflink idref="bib37" id="ref1">37</reflink>]). School climate is conceptualized as a property of the school, but traditionally assessed through individuals' perceptions (Bradshaw, Waasdorp, Debnam, & Lindstrom Johnson, [<reflink idref="bib9" id="ref2">9</reflink>]). These perceptions, both at the individual level and at the collective level, are shaped by internal and external factors as well as by shared experiences of school life (Cohen, McCabe, Michelli, & Pickeral, [<reflink idref="bib11" id="ref3">11</reflink>]). Yet important debates still remain regarding the definition of school climate and its measurement, which have implications for determining the causal link among climate, academics, and behavioral outcomes (Payne, [<reflink idref="bib33" id="ref4">33</reflink>]).</p> <p>The U.S. federal government has become increasingly focused on school climate. As of the 2017–2018 school year, the <emph>Every Student Succeeds Act</emph> (ESSA) requires that states, along with traditional indicators of academics (e.g., graduation and proficiency in reading and math), include one other indicator of school quality or success, such as student engagement, safety, or school climate in their accountability indicator. The majority of states have chosen to use archival data (e.g., chronic absenteeism, access to advanced coursework or career exploration offerings) to meet the reporting requirement (Achieve, [<reflink idref="bib1" id="ref5">1</reflink>]), and relatively few states have opted to administer a survey of students' perceptions. This decision likely reflects, at least in part, the current state of the field of school climate research including uncertainty surrounding its conceptualization as well as its measurement. The intent of this article is to inform the measurement of school climate and its possible use as an accountability indicator.</p> <hd id="AN0140917066-2">Defining and Measuring School Climate</hd> <p>Conceptualizations of school climate can vary in breadth and specificity ([<reflink idref="bib25" id="ref6">25</reflink>]), which has a direct implication for what is measured and reported. As such, there remains a considerable conceptual debate regarding the parameters of school climate. Specifically, some recent work suggests that school climate might best be captured by a narrower measure of student engagement (Payne, [<reflink idref="bib33" id="ref7">33</reflink>]). Other related work has defined school climate as "the quality and consistency of interpersonal interactions within the school community that influence children's cognitive, social, and psychological development" (Haynes, Emmons, & Ben‐Avie, [<reflink idref="bib18" id="ref8">18</reflink>], p, 322). This conceptualization posits that the constructs of school safety and aspects of the school environment are determinants of school climate, or outcomes of school climate, but they are not part of the specific measure of school climate. This distinction reflects a broader debate regarding whether the focus should be on school safety as compared to school climate (Benbenishty, Astor, Roziner, & Wrabel, [<reflink idref="bib4" id="ref9">4</reflink>]).</p> <p>Nevertheless, the U.S. Department of Education (USDOE) put forth an inclusive model of school climate that reflects both student safety and the school environment, as well as student engagement. Specifically, <emph>engagement</emph> focuses on relationships among students, staff, and families that are built on trust and respect and foster connection with the school. These features of school climate are thought to be fostered by a school <emph>environment</emph> with clear rules and expectations and supports for learning. Engagement and environment create and are supported by physical and emotional <emph>safety</emph> (National Center on Safe Supportive Learning Environments, [<reflink idref="bib32" id="ref10">32</reflink>]). Part of the reason for this broad conceptualization may reflect a desire to encourage consideration of all of these constructs, which individually have been linked with student learning and behavioral outcomes ([<reflink idref="bib25" id="ref11">25</reflink>]). However, the use of a multidimensional model of school climate presents difficulties in understanding how to aggregate these data into a single accountability measure. In fact, the desire for a school‐level indicator of school climate for school‐level accountability was a motivating factor behind the USDOE's Safe and Supportive Schools grant; this multi‐million federal dollar initiative funded 11 states to develop a comprehensive measure of school climate and pilot the measures in high schools to inform the implementation of evidence‐based programs to improve school climate (Bradshaw et al., [<reflink idref="bib7" id="ref12">7</reflink>]; Shaw, [<reflink idref="bib36" id="ref13">36</reflink>]).</p> <hd id="AN0140917066-3">Individual Variability in Perceptions of School Climate</hd> <p>Although there is substantial variability at the classroom and school level, the majority of variability in perceptions of school climate is attributable to individual differences (Fan, Williams, & Corkin, [<reflink idref="bib17" id="ref14">17</reflink>]; Koth, Bradshaw, & Leaf, [<reflink idref="bib23" id="ref15">23</reflink>]). Recent efforts to validate surveys of school climate have provided evidence of measurement invariance across a range of demographic factors (e.g., gender, age, and race; Bear, Gaskins, Blank, & Chen, [<reflink idref="bib3" id="ref16">3</reflink>]; Bradshaw et al., [<reflink idref="bib9" id="ref17">9</reflink>]). These findings give confidence that school climate can be conceptualized the same across groups, and that discrepancies that do occur are meaningful and not solely attributable to differences in measurement quality. For example, research has found that girls are more likely to report a positive school climate (Kuperminc, Leadbeater, Emmons, & Blatt, [<reflink idref="bib24" id="ref18">24</reflink>]) including a higher achievement motivation (Koth et al., [<reflink idref="bib23" id="ref19">23</reflink>]) and better relationships with teachers (Crosnoe, Johnson, & Elder, [<reflink idref="bib14" id="ref20">14</reflink>]), whereas boys are more likely to report lower levels of order and discipline (Koth et al., [<reflink idref="bib23" id="ref21">23</reflink>]) and disciplinary problems (Crosnoe et al., [<reflink idref="bib14" id="ref22">14</reflink>]). Youth of color tend to report less supportive relationships with their teachers, have lower perceptions of equity, and perceive the environment as less safe (Bottiani, Bradshaw, & Mendelson, [<reflink idref="bib5" id="ref23">5</reflink>]; Fan et al., [<reflink idref="bib17" id="ref24">17</reflink>]). School climate perceptions have been shown to decrease through the transition from elementary to middle school (Espinoza & Juvonen, [<reflink idref="bib16" id="ref25">16</reflink>]) and across middle school (Way, Reddy, & Rhodes, [<reflink idref="bib42" id="ref26">42</reflink>]) and improve throughout the course of high school (Bradshaw et al., [<reflink idref="bib9" id="ref27">9</reflink>]; Crosnoe et al., [<reflink idref="bib14" id="ref28">14</reflink>]). The influence of school climate on outcomes has also been shown to vary (i.e., be moderated) by gender (Crosnoe et al., [<reflink idref="bib14" id="ref29">14</reflink>]; Henry, Farrell, Schoeny, Tolan, & Dymnicki, [<reflink idref="bib20" id="ref30">20</reflink>]; Kuperminc et al., [<reflink idref="bib24" id="ref31">24</reflink>]), race (Crosnoe et al., [<reflink idref="bib14" id="ref32">14</reflink>]; Espinoza & Juvonen, [<reflink idref="bib16" id="ref33">16</reflink>]; Kuperminc et al., [<reflink idref="bib24" id="ref34">24</reflink>]), and age (Henry et al., [<reflink idref="bib20" id="ref35">20</reflink>]).</p> <p>Less empirical research has explicitly focused on differences in perceptions by levels of academic success or parental education. A study examining latent profiles of student perceptions of school climate found that students who perceived a more positive climate had higher mean academic outcomes and came from families with higher levels of parental education than those who perceived a negative climate (Cornell, Shukla, & Konold, [<reflink idref="bib12" id="ref36">12</reflink>]). Students who perform better academically may have different experiences with teachers, or be more likely to feel engaged to school (Battistich, Solomon, Kim, Watson, & Schaps, [<reflink idref="bib2" id="ref37">2</reflink>]; Kuperminc et al., [<reflink idref="bib24" id="ref38">24</reflink>]). Further, Fan and colleagues ([<reflink idref="bib17" id="ref39">17</reflink>]) found evidence of an association between parental educational level and perceptions of order, safety, and discipline but not teacher–student relationships or fairness and clarity of school rules.</p> <hd id="AN0140917066-4">Application of Item Response Theory to the Assessment of School Climate</hd> <p>An innovative aspect of the current article was the use of item response theory (IRT) analyses to examine a measure of school climate that is aligned with the USDOE's conceptualization, with the overarching goal of creating a more parsimonious yet psychometrically sound measure of climate. Although IRT has been previously applied to school climate scales (see Mo, Yang, & Hu, [<reflink idref="bib30" id="ref40">30</reflink>]), we capitalized on the existence of measures across three possible domains of school climate (e.g., safety, engagement, and environment) to both understand the ability of diverse items to create a scale as well as explore how the varying domain scales operated. Specifically, we assessed differential item functioning (DIF) for each scale in relation to a broad range of individual demographic characteristics, including maternal education (a measure of socioeconomic status) and grades (a measure of academic success). We also evaluated the ability of each scale to assess the continuum of perceptions of school climate. Taken together, the result of the IRT analyses is intended to further illustrate the validity, reliability, and potential usability of this particular measure of school climate for schools and state education agencies, as well as educational researchers. As such, the current study had a dual focus on informing both the measurement and conceptualization of school climate, which ultimately may inform schools' and states' use of surveys to meet expectations outlined in ESSA.</p> <hd id="AN0140917066-5">Method</hd> <p></p> <hd id="AN0140917066-6">Initial Data Collection</hd> <p></p> <hd id="AN0140917066-7">Procedures</hd> <p>Data for the study came from the Maryland Safe and Supportive Schools (MDS3) Initiative, which is a collaborative effort of the Maryland Department of Education (MSDE), Johns Hopkins University, and Sheppard Pratt Health System aimed at improving school climate and student outcomes. The MDS3 School Climate Student Survey (Bradshaw et al., [<reflink idref="bib9" id="ref41">9</reflink>]) was developed as a self‐report survey and is delivered online to students, staff, and parents in public middle and high schools across the state of Maryland. In the current study, we drew upon data from 111 schools across 13 Maryland school districts. Districts were approached for participation by the MSDE. Upon expressing interest in the MDS3 Initiative, district‐specific principal meetings were conducted to obtain school‐level and principal commitment to the project. The anonymous survey was administered using a passive consent and youth assent process, and all participation was voluntary. Letters were sent home to parents providing information about the survey and the larger initiative. The survey was administered online in language arts classrooms at participating high schools. School staff provided instructions for students to complete the survey following a written protocol developed by the research team. The nonidentifiable data were obtained from MSDE for analysis for the current article. The nonidentifiable data analysis was approved by researchers' Institutional Review Boards.</p> <hd id="AN0140917066-8">Participants</hd> <p>Data come from 69,513 secondary school students; 46% of students were in Grades 6, 7, and 8 (i.e., middle school) with the remainder in high school. Approximately half of the students identified as male. The sample was fairly diverse with 48.8% of students identifying as White, 25.7% as Black, and 9.6% as Hispanic. Almost 30% of students reported their mothers had a high school education or less with 55% reporting their mothers had obtained a high school degree. A majority of students (79.2%) self‐reported earning mostly A's or B's on their last report card. Further description of participant demographics and school demographics can be seen in Table .</p> <p>Sample Demographics</p> <p> <ephtml> <table><thead><tr><th /><th align="center">Total Sample (<italic>N</italic> = 69,513)</th></tr></thead><tbody><tr><td>Gender</td></tr><tr><td>Female</td><td>32,127 (49.7%)</td></tr><tr><td>Male</td><td>32,513 (50.3%)</td></tr><tr><td>Race</td></tr><tr><td>Native American/American Indian</td><td>1,536 (2.4%)</td></tr><tr><td>White</td><td>31,556 (48.8%)</td></tr><tr><td>Hispanic/Latino</td><td>6,230 (9.6%)</td></tr><tr><td>Asian/Pacific Islander</td><td>3,462 (5.4%)</td></tr><tr><td>Black/African American</td><td>16,628 (25.7%)</td></tr><tr><td>Native Hawaiian/Other Pacific Islander</td><td>385 (.6%)</td></tr><tr><td>Other</td><td>4,849 (7.5%)</td></tr><tr><td>Grade</td></tr><tr><td>6<sup>th</sup></td><td>11,069 (17.1%)</td></tr><tr><td>7<sup>th</sup></td><td>10,006 (15.5%)</td></tr><tr><td>8<sup>th</sup></td><td>8,645 (13.4%)</td></tr><tr><td>9<sup>th</sup></td><td>9,957 (15.4%)</td></tr><tr><td>10<sup>th</sup></td><td>9,528 (14.7%)</td></tr><tr><td>11<sup>th</sup></td><td>8,658 (13.4%)</td></tr><tr><td>12<sup>th</sup></td><td>6,807 (10.5%)</td></tr><tr><td>Maternal Education</td></tr><tr><td>Did not graduate high school</td><td>2,468 (8.8%)</td></tr><tr><td>Graduated high school</td><td>5,716 (20.4%)</td></tr><tr><td>Attended some college</td><td>4,274 (15.3%)</td></tr><tr><td>Graduated college</td><td>15,562 (55.5%)</td></tr><tr><td>Grades Last Report Card</td></tr><tr><td>Mostly A's</td><td>27,604 (42.8%)</td></tr><tr><td>Mostly B's</td><td>23,489 (36.4%)</td></tr><tr><td>Mostly C's</td><td>10,377 (16.1%)</td></tr><tr><td>Mostly D's</td><td>2,201 (3.4%)</td></tr><tr><td>Mostly F's</td><td>886 (1.4%)</td></tr><tr><td>School Characteristics (N = 111 schools)</td><td>M (SD)</td></tr><tr><td>% Suspension</td><td>11.6 (10.5)</td></tr><tr><td>School Enrollment</td><td>1,059.1 (429.6)</td></tr><tr><td>% FARMS</td><td>39.2 (18.0)</td></tr></tbody></table> </ephtml> </p> <hd id="AN0140917066-9">Instrument Design</hd> <p>The MDS3 School Climate Student Survey (Bradshaw et al., [<reflink idref="bib9" id="ref42">9</reflink>]) was developed by the Johns Hopkins Center for Youth Violence Prevention in collaboration with project partners. Researchers from the Center undertook a comprehensive review of the literature focusing on the three domains of school climate included in the USDOE ([<reflink idref="bib39" id="ref43">39</reflink>]) model (i.e., safety, engagement, and environment). Additionally, focus groups were held with students, district personnel, and school administrators to understand the operationalization of school contextual factors for each of the different stakeholders. The full survey contains over 150 items and takes approximately 20 minutes for students to complete online.</p> <p>The initial measurement model was created using an exploratory and confirmatory factor analytic approach in a high school sample (Bradshaw et al., [<reflink idref="bib9" id="ref44">9</reflink>]). Fifty‐six core items based on previously validated domains of safety, engagement, and the school environment were identified. Each domain contains multiple scales: <emph>Safety</emph> includes the scales perceived safety (four items; <emph>α</emph> =.64), bullying and aggression (four items; <emph>α</emph> =.63), and general drug use (three items; <emph>α</emph> =.87); <emph>Engagement</emph> includes the factors connection to teachers (six items; <emph>α</emph> =.86), student connectedness (five items; <emph>α</emph> =.87), academic engagement (four items; <emph>α</emph> =.79), whole school connectedness (four items; <emph>α</emph> =.82), culture of equity (four items; <emph>α</emph> =.83), parent engagement (four items; <emph>α</emph> =.74); and <emph>Environment</emph> includes the factors rules and consequences (five items; <emph>α</emph> =.73), physical comfort (four items; <emph>α</emph> =.79), support (four items; <emph>α</emph> =.76), and disorder (five items; <emph>α</emph> =.58). All answer choices were on a 4‐point Likert scale from <emph>strongly agree</emph> to <emph>strongly disagree</emph>, whereby all items were coded with high score representing a more favorable school climate. A specific research question for this article was the extent to which a subset of items from each domain could create a domain‐scale (i.e., <emph>Safety</emph>, <emph>Engagement</emph>, and <emph>Environment</emph>).</p> <p>IRT analyses explored measurement invariance across groups by gender (male/female), race (minority/nonminority), and grade in school (upper/lower classman). These analyses found evidence of scalar invariance (Bradshaw et al., [<reflink idref="bib9" id="ref45">9</reflink>]). Additional analyses also found measurement invariance across middle and high school students' reports (Waasdorp, Lindstrom Johnson, Shukla, & Bradshaw, [<reflink idref="bib41" id="ref46">41</reflink>]). Other studies have compared the functioning of scales to observations of the school social and physical environment (Bradshaw, Milam, Furr‐Holden, & Lindstrom Johnson, [<reflink idref="bib6" id="ref47">6</reflink>]). However, to date, there have been no efforts to address parsimony, assess differential functioning by academic success and socioeconomic status, and explore item and scale functioning across the continuum of perceptions of climate. Comparisons of these findings across the three scales were also examined.</p> <hd id="AN0140917066-10">Overview of the Item Analysis Procedures</hd> <p>Using the mirt package (Chalmers, [<reflink idref="bib10" id="ref48">10</reflink>]) in R (R Development Core Team, [<reflink idref="bib34" id="ref49">34</reflink>]), item characteristics were estimated for each of the three domains (e.g., safety, engagement, and environment) through unidimensional IRT using a graded response model (Samejima, [<reflink idref="bib35" id="ref50">35</reflink>]). These estimates included item location, item discrimination, and item information. The decision to estimate unidimensional models was based on (a) computational ease and (b) the lack of any hypothesized cross‐loadings (i.e., multidimensional models would have been specified to exhibit simple structure). The item and test information estimates resulting from the fitting of the IRT models as well as a selection of local and global fit statistics (discussed below) were then used to further refine the scales with the goal of creating the shortest scales possible while retaining a breadth of constructs and an adequate amount of test information across the spectrum of participant perspectives (i.e., 3 <emph>SD</emph>s below to 3 <emph>SD</emph>s above a neutral perspective on school climate). Adequacy was determined by converting test information to reliability using the equation put forth by Thissen ([<reflink idref="bib38" id="ref51">38</reflink>]) and targeting a reliability estimate greater than.70 across the spectrum. Global fit statistics of final models were assessed including root mean squared error of approximation (RMSEA), comparative fit index (CFI), the squared root mean standardized residuals (SRMSR), and the <emph>M</emph><sups>2</sups> statistic (see Hu & Bentler, [<reflink idref="bib22" id="ref52">22</reflink>] for the first three and Maydeu‐Olivares & Joe, [<reflink idref="bib26" id="ref53">26</reflink>] for the last). Reliability statistics were also assessed including McDonald's Omega total (ω<emph><subs>t</subs></emph>; McDonald, [<reflink idref="bib27" id="ref54">27</reflink>]) and Cronbach's <emph>α</emph> (Cortina, [<reflink idref="bib13" id="ref55">13</reflink>]).</p> <p>DIF (Holland & Wainer, [<reflink idref="bib21" id="ref56">21</reflink>]) analyses were conducted in order to identify any items exhibiting bias across any of five areas: gender, minority status, academic success, status as a middle school or high school student, or level of maternal education. Each of these variables was dichotomized with the resulting designations being [male; female], [White, non‐White], [A/B, C, or worse], [middle school, high school], [less than a high school education, high school education or greater], respectively. Typical methods of detecting DIF often involve conducting likelihood ratio tests of nested models wherein one model constrains the item parameters to be equal across groups and a comparison model allows those parameters to vary. These methods were not feasible in this case given the large sample size as any such test would be drastically overpowered and, therefore, overly sensitive to potentially spurious group differences. To address this potential concern, effect sizes (see Meade, [<reflink idref="bib28" id="ref57">28</reflink>]) were used in lieu of significance tests following the method of Meade and Wright ([<reflink idref="bib29" id="ref58">29</reflink>]). The use of such effect sizes necessitated defining criteria for what constitutes an acceptable difference because no criteria or recommendations currently exist in the DIF literature. For the purposes of the current study, it was decided that typical effect size interpretations would be used for standardized metrics and that an expected focal group difference, relative to the reference group, of one scale point at the test level would be utilized for unstandardized metrics. Finally, factor scores (theta values) were estimated using the expected a posteriori (EAP) method (Embretson & Reise, [<reflink idref="bib15" id="ref59">15</reflink>]). Mean differences in distributions of theta by the above group differences were also explored using Hedges's <emph>g</emph> as a measure of effect sizes (Hedges, [<reflink idref="bib19" id="ref60">19</reflink>]).</p> <hd id="AN0140917066-11">Results</hd> <p></p> <hd id="AN0140917066-12">Measure Creation</hd> <p>A primary goal of the scale construction process was to create a scale that provided a reliable estimate of a respondent's perspectives of school climate within each of the three domains (e.g., safety, engagement, and environment). To this end, item and test information curves were examined and used in the scale refinement process. A total of 30 of the 56 core items were retained across the three scales (10 items per scale). Table  presents descriptive statistics as well as the IRT parameter estimates for the items included in the final scales.</p> <p>Descriptive Statistics and IRT Parameter Estimates for Final Scale Items</p> <p> <ephtml> <table><thead><tr><th /><th align="center">Mean</th><th align="center"><italic>SD</italic></th><th align="center"><italic>b<sub>1</sub></italic></th><th align="center"><italic>b<sub>2</sub></italic></th><th align="center"><italic>b<sub>3</sub></italic></th><th align="center"><italic>a</italic></th></tr></thead><tbody><tr><td align="center">SAFETY</td></tr><tr><td>I feel safe at this school</td><td>2.07</td><td>.73</td><td>−2.88</td><td>−1.63</td><td>1.01</td><td>1.37</td></tr><tr><td>I feel safe going to and from this school</td><td>2.21</td><td>.72</td><td>−3.54</td><td>−2.19</td><td>.70</td><td>1.12</td></tr><tr><td>Students at this school try to stop bullying</td><td>1.42</td><td>.91</td><td>−1.76</td><td>.07</td><td>2.40</td><td>1.02</td></tr><tr><td>Seen someone else being bullied0002<sup>,</sup></td><td>.49</td><td>.50</td><td>.04</td><td align="center">NA</td><td align="center">NA</td><td>.96</td></tr><tr><td>Harassment or bullying of students problem0002</td><td>1.44</td><td>1.03</td><td>−1.01</td><td>.05</td><td>1.23</td><td>1.86</td></tr><tr><td>Physical fighting between students problem0002</td><td>1.59</td><td>.97</td><td>−1.54</td><td>−.15</td><td>1.29</td><td>1.53</td></tr><tr><td>Students carrying guns or knives0002</td><td>2.47</td><td>.88</td><td>−2.01</td><td>−1.47</td><td>−.56</td><td>2.01</td></tr><tr><td>Programs to deal with violence and conflict0002</td><td>1.64</td><td>.90</td><td>−3.17</td><td>−.67</td><td>2.66</td><td>.66</td></tr><tr><td>Students drug use problem0002</td><td>1.73</td><td>1.19</td><td>−1.00</td><td>−.26</td><td>.43</td><td>1.82</td></tr><tr><td>Students alcohol use problem0002</td><td>1.96</td><td>1.16</td><td>−1.33</td><td>−.59</td><td>.10</td><td>1.60</td></tr><tr><td align="center">ENGAGEMENT</td></tr><tr><td>My teachers tell me when I do a good job</td><td>1.97</td><td>.84</td><td>−1.86</td><td>−.93</td><td>.75</td><td>2.16</td></tr><tr><td>I enjoy learning at this school</td><td>1.74</td><td>.93</td><td>−1.34</td><td>−.56</td><td>1.01</td><td>2.26</td></tr><tr><td>My teachers believe that I can do well in school</td><td>2.20</td><td>.78</td><td>−2.11</td><td>−1.38</td><td>.38</td><td>2.30</td></tr><tr><td>My teachers listen when I have something to say</td><td>1.91</td><td>.83</td><td>−1.67</td><td>−.81</td><td>.85</td><td>2.63</td></tr><tr><td>I feel like I belong</td><td>1.87</td><td>.90</td><td>−1.59</td><td>−.76</td><td>.86</td><td>2.07</td></tr><tr><td>Materials reflect my culture, ethnicity, and identity</td><td>1.65</td><td>.90</td><td>−1.98</td><td>−.46</td><td>1.62</td><td>1.24</td></tr><tr><td>My teachers care about me</td><td>1.95</td><td>.83</td><td>−1.55</td><td>−.83</td><td>.72</td><td>3.40</td></tr><tr><td>Students trust one another</td><td>1.54</td><td>.90</td><td>−1.56</td><td>−.22</td><td>1.67</td><td>1.50</td></tr><tr><td>Do good at school, my parents hear about it</td><td>1.51</td><td>1.03</td><td>−1.30</td><td>−.04</td><td>1.30</td><td>1.39</td></tr><tr><td>Students and staff feel pride in this school</td><td>1.87</td><td>.89</td><td>−1.68</td><td>−.73</td><td>.86</td><td>2.07</td></tr><tr><td align="center">ENVIRONMENT</td></tr><tr><td>Students listen to the teachers</td><td>1.45</td><td>.81</td><td>−1.41</td><td>−.06</td><td>1.99</td><td>2.01</td></tr><tr><td>Teachers can handle students who disrupt the class</td><td>1.56</td><td>.85</td><td>−1.39</td><td>−.23</td><td>1.49</td><td>2.34</td></tr><tr><td>Students are rewarded for positive behavior</td><td>1.47</td><td>.92</td><td>−1.44</td><td>−.03</td><td>1.76</td><td>1.42</td></tr><tr><td>Everyone knows what the school rules are</td><td>1.84</td><td>.85</td><td>−2.11</td><td>−.78</td><td>1.22</td><td>1.47</td></tr><tr><td>Teachers at this school help students with their problems</td><td>1.85</td><td>.85</td><td>−1.87</td><td>−.75</td><td>1.07</td><td>1.85</td></tr><tr><td>The school is usually clean and well‐maintained</td><td>1.53</td><td>.90</td><td>−1.58</td><td>−.29</td><td>1.92</td><td>1.33</td></tr><tr><td>It is easy for teachers at my school to control the students</td><td>1.37</td><td>.85</td><td>−1.25</td><td>.16</td><td>1.87</td><td>1.97</td></tr><tr><td>Broken windows, doors, or desks in this school<sup>*</sup></td><td>1.94</td><td>.90</td><td>−3.37</td><td>−1.27</td><td>1.20</td><td>.80</td></tr><tr><td>There are clear rules about student behavior</td><td>1.98</td><td>.81</td><td>−2.18</td><td>−1.11</td><td>.94</td><td>1.65</td></tr><tr><td>Misbehaving students get away with it<sup>*</sup></td><td>1.44</td><td>.87</td><td>−2.31</td><td>.15</td><td>2.85</td><td>.84</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note. b =</emph> threshold values or location parameter<emph>, a</emph> = discrimination parameter.</p> <ulist> <item>2 <sups>*</sups>Items were reverse coded.</item> <item>3 Item was scored dichotomously (i.e., 0/1). All other items were scored 0–3.</item> </ulist> <p>Table  presents descriptive statistics as well as IRT parameter estimates for the items included in the final version of the scales. The discrimination parameter (<emph>a</emph>) refers to an item's ability to differentiate between respondents of different trait levels, whereas the threshold parameters (<emph>b</emph><subs>1</subs>, <emph>b</emph><subs>2</subs>, and <emph>b</emph><subs>3</subs>) indicate the difficulty of the item. In the case of the graded response model, these thresholds indicate the points at which a respondent would have the same probability of endorsing any of the categories below the threshold compared as they would for the categories above the threshold. In the case of the first item on the <emph>Safety</emph> scale ("I feel safe at this school."), for example, the probability of a respondent 1.01 <emph>SD</emph>s above the mean on the latent <emph>Safety</emph> trait choosing <emph>strongly agree</emph> (i.e., <emph>p</emph>[<emph>x</emph> = 3] =.50) would be the same as their probability of choosing any of the other responses (i.e., <emph>p</emph>[<emph>x =</emph> 0 or <emph>x</emph> = 1 or <emph>x</emph> = 2] =.50). As can be seen in Table , the discrimination parameter estimates (<emph>a</emph>) for the majority of items in the <emph>Safety</emph> scale and some in the <emph>Environment</emph> scale were fairly low. As the discrimination parameter assesses the ability of an item to differentiate between two individuals, low values indicate items that provide limited information. From a psychometric perspective, an ideal scale would include a mix of items that provide moderate information across the scale as well as items that provide high information at varying ability levels.</p> <p>Figure  presents the final test information curves for each of the three scales of <emph>Safety</emph>, <emph>Engagement</emph>, and <emph>Environment</emph>. For convenience, horizontal lines have been drawn on the test information curves at the points along the <emph>Y</emph>‐axis corresponding to reliability estimates of.70,.80, and.90.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/EMS/01dec19/emip12296-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="emip12296-fig-0001.jpg" title="Test information curves by scale." /> </p> <p></p> <p>Figure  presents the test information curves for each of the three scales. These curves indicate the information that the scale is able to provide as well as the reliability of the test for a given level of the latent trait. The shape of the curves suggests that the items in the scales tended to provide more information (and, thus are more reliable) for respondents with less favorable perceptions of school climate. This was particularly true for the <emph>Safety</emph> scale for which estimates of scale reliability drop below acceptable levels at perceptions 1 <emph>SD</emph> above mean (e.g., see Figure  where the reliability drops below.8 at 1 <emph>SD</emph> and below.7 at 2 <emph>SD</emph>s above). Upon examination of the threshold values <emph>b<subs>i</subs></emph> in Table  (which refer to the points at which the probability of choosing response <emph>i</emph> + 1 or higher is equal to.5), it can be seen that most items were located on the left side of the school climate spectrum, suggesting that these items might be better at capturing negative perceptions of school climate than positive. For model fit, the values of the RMSEA, CFI, and SRMR indices generally met the criteria for satisfactory global fit set forth in the literature with a few exceptions (see Table ). The CFI value for the <emph>Environment</emph> model (.867) fell below the commonly applied criteria of.90 and the SRMR value for the <emph>Safety</emph> model was larger than the suggested cutoff of.08. Neither of these departures was deemed to be reason enough for discarding the respective models. All reliability statistics were acceptable with the exception of McDonald's ([<reflink idref="bib27" id="ref61">27</reflink>]) hierarchical Omega (ω<emph><subs>h</subs></emph>) for Safety. Specifically, the <emph>Safety</emph> scale seemed to have less general commonality and more group commonality. This suggests the existence of a subset of items in the <emph>Safety</emph> scale that were strongly related beyond the general factor. The other two scales of <emph>Environment</emph> and <emph>Engagement</emph> had stronger general factors and less variance associated with subsets of items. Additional details about global model fit and reliability statistics for each of the three unidimensional models can be found in Table . Correlations between the three factors were moderate to high (<emph>Safety</emph> with <emph>Engagement</emph>, <emph>r</emph> =.619, <emph>p</emph> < .05; <emph>Safety</emph> with <emph>Environment</emph>, <emph>r</emph> =.656, <emph>p</emph> < .05; <emph>Engagement</emph> with <emph>Environment</emph>, <emph>r</emph> =.853, <emph>p</emph> < .05).</p> <p>Model Fit and Reliability Statistics for Each of the Three Scales</p> <p> <ephtml> <table><thead><tr><th>Model</th><th align="center">CFI</th><th align="center">SRMSR</th><th align="center">RMSEA [CI]</th><th align="center"><italic>M</italic><sup>2</sup> (<italic>df</italic>)</th><th align="center">α</th><th align="center">ω<italic><sub>t</sub></italic></th></tr></thead><tbody><tr><td>Safety</td><td>.954</td><td>.108</td><td>.058 [.056,.060]</td><td>3,631.94 (17)</td><td>.80</td><td>.86</td></tr><tr><td>Engagement</td><td>.929</td><td>.048</td><td>.075 [.073,.076]</td><td>5,063.65 (15)</td><td>.89</td><td>.92</td></tr><tr><td>Environment</td><td>.867</td><td>.061</td><td>.079 [.077,.081]</td><td>5,504.27 (15)</td><td>.82</td><td>.86</td></tr></tbody></table> </ephtml> </p> <hd id="AN0140917066-14">Individual Differences in Perceptions of School Climate</hd> <p>Results from the DIF analyses are presented in Table . Included in this table are two effect size estimates: the signed test difference in the sample (STDS) and the expected test score standardized difference (ETSSD). Note that the STDS uses signed estimates of group difference and, as such represents the difference in expected scale scores averaged across all focal group respondents (Meade, [<reflink idref="bib28" id="ref62">28</reflink>]). The STDS estimate is interpreted in the metric of the scale of the instrument (e.g., 0–30 for a 10‐item instrument with each item being scored [0,1,2,3]). The ETSSD estimate is a standardized effect size; the same rules that apply to interpreting other effect sizes can be used to interpret these values. The results indicate that there are no substantial measurement differences by gender, minority status, academic success, status as a middle school or high school student, or level of maternal education. The largest differences are among students of differing academic status, with students reporting getting A's and B's on their report card on average scoring.595 points higher on the <emph>Safety</emph> scale. However, this is still considered a small effect (i.e., ETSSD of –.112). Overall, the <emph>Environment</emph> scale exhibited the lowest level of differences among subgroups, with the possible exception of minority status.</p> <p>DIF by Subgroup</p> <p> <ephtml> <table><thead><tr><th>DIF Estimates</th></tr><tr><th /><th /><th align="center">Safety</th><th align="center">Engagement</th><th align="center">Environment</th></tr><tr><th>Ref. Group</th><th>Focal Group</th><th align="center">STDS</th><th align="center">ETSSD</th><th align="center">STDS</th><th align="center">ETSSD</th><th align="center">STDS</th><th align="center">ETSSD</th></tr></thead><tbody><tr><td>Male</td><td>Female</td><td>−.156</td><td>−.034</td><td>.239</td><td>.040</td><td>−.004</td><td>−.001</td></tr><tr><td>Non‐White</td><td>White</td><td>−.113</td><td>−.024</td><td>.097</td><td>.017</td><td>.206</td><td>.043</td></tr><tr><td>A/B Grade</td><td>C/D/F Grade</td><td>−.585</td><td>−.119</td><td>−.229</td><td>−.037</td><td>.054</td><td>.011</td></tr><tr><td>Middle</td><td>High</td><td>−.059</td><td>−.013</td><td>.083</td><td>.014</td><td>.014</td><td>.003</td></tr><tr><td>Mat. Ed. < HS</td><td>Mat. Ed. ≥ HS</td><td>−.101</td><td>−.021</td><td>.063</td><td>.010</td><td>.068</td><td>.014</td></tr></tbody></table> </ephtml> </p> <p>DIF by Subgroup</p> <p> <ephtml> <table><thead><tr><th>Effect Sizes of Differences</th></tr><tr><th /><th /><th align="center">Mean Diff.</th><th align="center">Effect Size</th><th align="center">Mean Diff.</th><th align="center">Effect Size</th><th align="center">Mean Diff.</th><th align="center">Effect Size</th></tr></thead><tbody><tr><td>Male</td><td>Female</td><td>.125</td><td>−.138</td><td>.082</td><td>−.086</td><td>.058</td><td>−.062</td></tr><tr><td>Non‐White</td><td>White</td><td>−.085</td><td>.094</td><td>−.127</td><td>.134</td><td>−.138</td><td>−.002</td></tr><tr><td>A/B Grade</td><td>C/D/F Grade</td><td>.227</td><td>−.251</td><td>.420</td><td>−.449</td><td>.319</td><td>−.347</td></tr><tr><td>Middle</td><td>High</td><td>−.093</td><td>.103</td><td>−.023</td><td>.024</td><td>−.016</td><td>.017</td></tr><tr><td>Mat. Ed. < HS</td><td>Mat. Ed. = HS+</td><td>−.154</td><td>.171</td><td>−.163</td><td>.173</td><td>−.132</td><td>.143</td></tr></tbody></table> </ephtml> </p> <p>4 <emph>Note</emph>. STDS, Signed Test Difference in the Sample; ETSSD, Expected Test Score Standardized Difference; Mat. Ed, Maternal education.</p> <p>Factor scores (i.e., theta values) were plotted for each of the subgroups; Table  presents effect sizes of difference. Most are considered small, with some differences by students' self‐reported grade in school being slightly more substantial. Examining plots of factor scores by subgroups shows a positively shifted distribution (see Figure ), whereby students who received higher grades had more favorable perceptions of safety, engagement, and environment. To a lesser degree, students whose parents had more than a high school degree also held more favorable perceptions of safety, engagement, and environment.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/EMS/01dec19/emip12296-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="emip12296-fig-0002.jpg" title="Selected comparisons of theta distributions by subgroup: (a) Safety, (b) engagement, and (c) environment" /> </p> <p></p> <hd id="AN0140917066-16">Discussion</hd> <p>This article is novel in its application of IRT to improving the measurement of students' perceptions of school climate using the USDOE's model. Using this method, we created parsimonious scales measuring the three key USDOE ([<reflink idref="bib39" id="ref63">39</reflink>]) domains of school climate (i.e., safety, engagement, and environment). This was done with specific attention to ensure that items in the scales covered the breadth of topics from component subscales. Additionally, we demonstrated that the scales equivalently assessed safety, engagement, and environment across gender, minority status, academic success, status as a junior or high school student, and maternal education, but that differences in perceptions, particularly by academic success, existed. Attention to these measurement differences is important as individual evaluations of school climate are often aggregated to the school level as a metric of school performance (Battistich et al., [<reflink idref="bib2" id="ref64">2</reflink>]; Way et al., [<reflink idref="bib42" id="ref65">42</reflink>]).</p> <p>Through the process of evaluating item functioning and then scale composition, we identified that our measure was better able to assess differences in individual's perceptions of poor school climate than of excellent school climate. This was most apparent in the test information curves (see Figure ) that show more information below the average theta than above. This is primarily the result of "easy" items, in which the probability of getting an answer "correct" is fairly high. Take for example the item "Students carrying guns or knives at this school is a problem." At around 2 <emph>SD</emph>s below the mean, students have an equal probability of saying they <emph>strongly agree</emph> with this statement versus any other response. At.5 <emph>SD</emph> still below the mean, they have an equal probability of saying they <emph>strongly disagree</emph> with this statement versus any other response. Although all three scales provided more information at the less favorable end of perceptions, this was particularly problematic for assessments of school safety. This may be the artifact of students' feeling safe at school (Musu‐Gillette et al., [<reflink idref="bib31" id="ref66">31</reflink>]). However, it does have implications for measurement accuracy, as it suggests that we can more reliably measure the perceptions of those who perceive their school to have poor school climate than excellent school climate, and may support the need for a more nuanced understand of safety that goes beyond acts of violence and perceptions of safety (Embretson & Reise, [<reflink idref="bib15" id="ref67">15</reflink>]). It also has implications for school climate interventions as it suggests that schools with more positive school climates may be less invested in school climate assessments and interventions (Bradshaw et al., [<reflink idref="bib9" id="ref68">9</reflink>]).</p> <p>The current study is one of the first to examine measurement invariance for students reporting different levels of academic engagement as well as differing levels of socioeconomic status (Bear et al., [<reflink idref="bib3" id="ref69">3</reflink>]; Bradshaw et al., [<reflink idref="bib9" id="ref70">9</reflink>]). Our results suggest that differences resulting from the models across these domains as well as gender, minority status, and level of school could be attributed as meaningful and not as the result of error. Additionally, effect size DIF estimates suggest limited mean differences in perceptions of school climate across groups at the same level of perceptions of school climate (i.e., theta). This is a potentially novel contribution to the understanding of individual differences in school climate perceptions as it takes into account group difference in perceptions of climate (i.e., items function the same for girls who view the school climate as less favorable as boys who view the climate as less favorable). Although most group differences in theta were small, there were moderate differences for students who had higher grades. Our results suggest that schools with a higher percentage of students receiving A's and B's will receive higher aggregated school climate scores simply as a reflection of measurement difference; therefore, schools with a higher concentration of better performing students on average tend to rate their school climate more favorably. This suggests the difficulty of disentangling school climate and achievement and may explain differential findings around the relationship between school climate and achievement, particularly those that involve the aggregation of student perceptions (Benbenishty et al., [<reflink idref="bib4" id="ref71">4</reflink>]).</p> <p>An additional contribution of this article is its potential to inform discussions regarding the definition of school climate. This is important as it has direct implications for what is measured and used as an accountability measure. Due to the multidimensional nature of school climate, surveys of student perceptions can be lengthy, which may dissuade states from using them. Furthermore, it is difficult to take the data from a multidimensional construct to create a single accountability indicator. Together, our results suggest the possibility of measuring school climate across the broad categories of safety, engagement, and environment. More work is needed to understand how these three indicators might be aggregated (Bradshaw et al., [<reflink idref="bib9" id="ref72">9</reflink>]) or how they might be causally related (Payne, [<reflink idref="bib33" id="ref73">33</reflink>]). Nevertheless, our findings suggest that engagement and environment are more highly correlated than safety, which may reflect the conceptual challenges in defining safety.</p> <hd id="AN0140917066-17">Strengths, Limitations, and Next Steps</hd> <p>A strength of this study is that we started with a previously validated measure of school climate, which is consistent with the USDOE's conceptualization of school climate. We aimed to create a more parsimonious measure that would be more time‐efficient for schools to use but also psychometrically strong from an IRT perspective. Future work should also begin to address the likely correlations among the various components of school climate (i.e., safety, engagement, and environment), allowing for a possible aggregate measure of school climate. Additional work should also focus on aspects of external validity by determining the extent to which the scale scores relate to student behavioral indicators of interest to educators and policy makers, such as suspensions, academic performance, and high school completion. Moreover, future studies could also contrast the concurrent and predictive validity of the long versus short version in reference to these and other student behavioral indicators to ensure that the shortened version is in fact sufficiently predictive of particular outcomes of interest. An important limitation to note is that the data were drawn from one state, and middle and high schools and students. Staff and even parents may provide additional valuable insights into school climate (Waasdorp, Pas, O'Brennan, & Bradshaw, [<reflink idref="bib40" id="ref74">40</reflink>]).</p> <hd id="AN0140917066-18">Conclusion</hd> <p>This article applied sophisticated analytic techniques used for assessment design to measures of school climate, with the overarching goal of making a measure, which is consistent with the USDOE's model of school climate, more efficient while not compromising its validity and reliability. Leveraging advanced psychometric tools, which have largely focused on academic and other measurement topics, we helped to advance the field of school climate assessment. In doing so we also contribute to debates about the scope and conceptualization of school climate. This line of work is particularly timely in light of ESSA's emphasis on school climate and related constructs. Furthermore, our findings highlight the importance of disentangling student background from school variables. Although a common language is emerging regarding the various theorized dimensions of school climate, the findings of this study advance the conversation by providing insight on how to both efficiently and precisely measure these three core dimensions of school climate. This type of empirical work is critical to support the inclusion of a broader array of school factors into discussions of accountability for school leaders.</p> <hd id="AN0140917066-19">Acknowledgments</hd> <p>This work was funded in part by grants from the U.S. Department of Education to the Maryland State Department of Education and the Institute for Educational Sciences (R305H150027) and the National Institute of Justice (2014‐CK‐BX‐0005) to Catherine Bradshaw.</p> <ref id="AN0140917066-20"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref5" type="bt">1</bibl> <bibtext> Sarah Lindstrom Johnson, Arizona State University, Social Sciences Room 116, PO Box 873701, Tempe AZ 85287‐3701; Sarahlj@asu.edu. Ray E. Reichenberg, University of Nebraska, 269 Louise Pound Hall, 512 N. 12th Street, Lincoln, NE 68588‐0365; rreichenberg@unl.edu. Kathan Shukla, Indian Institute of Management, Ahmedabad Vastrapur, Ahmedabud 380015 Gujarat, India; kathans@iiama.ac.in. Tracy E. Waasdorp, Johns Hopkins School of Public Health 415 N. Washington Street, Baltimore, MD 21231; twaasdo1@jhu.edu. Catherine P. Bradshaw, University of Virginia 112‐D Bavaro Hall, 417 Emmet Street South, PO Box 400260 Charlottesville, VA 22904‐4260; Catherine.bradshaw@virginia.edu.</bibtext> </blist> </ref> <ref id="AN0140917066-21"> <title> References </title> <blist> <bibtext> Achieve. (2019). Accountability in state ESSA plans. Retrieved from https://states.achieve.org/essa-tracker</bibtext> </blist> <blist> <bibl id="bib2" idref="ref37" type="bt">2</bibl> <bibtext> Battistich, V., Solomon, D., Kim, D. I., Watson, M., & Schaps, E. (1995). Schools as communities, poverty levels of student populations, and students' attitudes, motives, and performance: A multilevel analysis. American Educational Research Journal, 32, 627 – 658.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref16" type="bt">3</bibl> <bibtext> Bear, G. G., Gaskins, C., Blank, J., & Chen, F. F. (2011). Delaware School Climate Survey—Student: Its factor structure, concurrent validity, and reliability. Journal of School Psychology, 49 (2), 157 – 174.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref9" type="bt">4</bibl> <bibtext> Benbenishty, R., Astor, R. A., Roziner, I., & Wrabel, S. L. (2016). Testing the causal links between school climate, school violence, and school academic performance: A cross‐lagged panel autoregressive model. Educational Researcher, 45 (3), 197 – 206.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref23" type="bt">5</bibl> <bibtext> Bottiani, J. H., Bradshaw, C. P., & Mendelson, T. (2016). Inequality in black and white high school students' perceptions of school support: An examination of race in context. Journal of Youth and Adolescence, 45, 1176 – 1191.</bibtext> </blist> <blist> <bibl id="bib6" idref="ref47" type="bt">6</bibl> <bibtext> Bradshaw, C. P., Milam, A. J., Furr‐Holden, C. D. M., & Lindstrom Johnson, S. (2015). The School Assessment for Environmental Typology (SAfETy): An observational measure of the school environment. American Journal of Community Psychology, 56 (3–4), 280 – 292.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref12" type="bt">7</bibl> <bibtext> Bradshaw, C. P., Pas, E., Bloom, J., Barrett, S., Hershfeldt, P., Alexander, A., ... Leaf, P. (2012). A state‐wide collaboration to promote safe and supportive schools: The PBIS Maryland Initiative. Administration and Policy in Mental Health and Mental Health Services Research, 39 (4), 225 – 237.</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Bradshaw, C. P., Pas, E. T., Debnam, K. J., & Lindstrom Johnson, S. (2015). A focus on implementation of Positive Behavioral Interventions and Supports (PBIS) in high schools: Associations with bullying and other indicators of school disorder. School Psychology Review, 44, 480 – 498.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref2" type="bt">9</bibl> <bibtext> Bradshaw, C. P., Waasdorp, T. E., Debnam, K. J., & Lindstrom Johnson, S. L. (2014). Measuring school climate in high schools: A focus on safety, engagement, and the environment. Journal of School Health, 84, 593 – 604.</bibtext> </blist> <blist> <bibtext> Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48 (6), 1 – 29.</bibtext> </blist> <blist> <bibtext> Cohen, J., McCabe, L., Michelli, N. M., & Pickeral, T. (2009). School climate: Research, policy, practice, and teacher education. Teachers College Record, 111 (1), 180 – 213.</bibtext> </blist> <blist> <bibtext> Cornell, D., Shukla, K., & Konold, T. R. (2016). Authoritative school climate and student academic engagement, grades, and aspirations in middle and high schools. AERA Open, 2, 1 – 18.</bibtext> </blist> <blist> <bibtext> Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78 (1), 98 – 104.</bibtext> </blist> <blist> <bibtext> Crosnoe, R., Johnson, M. K., & Elder, G. H., Jr. (2004). Intergenerational bonding in school: The behavioral and contextual correlates of student‐teacher relationships. Sociology of Education, 77 (1), 60 – 81.</bibtext> </blist> <blist> <bibtext> Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ : Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> Espinoza, G., & Juvonen, J. (2011). Perceptions of the school social context across the transition to middle school: Heightened sensitivity among Latino students? Journal of Educational Psychology, 103, 749 – 758.</bibtext> </blist> <blist> <bibtext> Fan, W., Williams, C. M., & Corkin, D. M. (2011). A multilevel analysis of student perceptions of school climate: The effect of social and academic risk factors. Psychology in the Schools, 48, 632 – 647.</bibtext> </blist> <blist> <bibtext> Haynes, N. M., Emmons, C., & Ben‐Avie, M. (1997). School climate as a factor in student adjustment and achievement. Journal of Educational and Psychological Consultation, 8 (3), 321 – 329.</bibtext> </blist> <blist> <bibtext> Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6 (2), 107 – 128.</bibtext> </blist> <blist> <bibtext> Henry, D. B., Farrell, A. D., Schoeny, M. E., Tolan, P. H., & Dymnicki, A. B. (2011). Influence of school‐level variables on aggression and associated attitudes of middle school students. Journal of School Psychology, 49, 481 – 503.</bibtext> </blist> <blist> <bibtext> Holland, P. W., & Wainer, H. (2012). Differential item functioning. New York, NY : Routledge.</bibtext> </blist> <blist> <bibtext> Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6 (1), 1 – 55.</bibtext> </blist> <blist> <bibtext> Koth, C. W., Bradshaw, C. P., & Leaf, P. J. (2008). A multilevel study of predictors of student perceptions of school climate: The effect of classroom‐level factors. Journal of Educational Psychology, 100 (1), 96 – 104.</bibtext> </blist> <blist> <bibtext> Kuperminc, G. P., Leadbeater, B. J., Emmons, C., & Blatt, S. J. (1997). Perceived school climate and difficulties in the social adjustment of middle school students. Applied Developmental Science, 1 (2), 76 – 88.</bibtext> </blist> <blist> <bibtext> Lindstrom Johnson, S., Waasdorp, T., & Bradshaw, C. (in press). School climate. In Encyclopedia of education. Abindgon, UK : Routledge.</bibtext> </blist> <blist> <bibtext> Maydeu‐Olivares, A. & Joe, H. (2006). Limited information goodness‐of‐fit testing in multidimensional contingency tables. Psychometrika, 71 (4), 713 – 732.</bibtext> </blist> <blist> <bibtext> McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ : Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> Meade, A. W. (2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728 – 743.</bibtext> </blist> <blist> <bibtext> Meade, A. W., & Wright, N. A. (2012). Solving the measurement invariance anchor item problem in item response theory. Journal of Applied Psychology, 97, 1016 – 1031.</bibtext> </blist> <blist> <bibtext> Mo, L., Yang, F., & Hu, X. (2011). An empirical examination of IRT information for school climate surveys. Educational Research and Evaluation, 17 (1), 33 – 45.</bibtext> </blist> <blist> <bibtext> Musu‐Gillette, L., Zhang, A., Wang, K., Zhang, J., Kemp, J., Kiliberti, M., & Ouderker, D. A. (2018). Indicators of school crime and safety: 2017. Washington, DC : National Center for Education Statistics.</bibtext> </blist> <blist> <bibtext> National Center on Safe and Supportive Learning Environments. (2018). School climate. Retrieved from https://safesupportivelearning.ed.gov/safe-and-healthy-students/school-climate</bibtext> </blist> <blist> <bibtext> Payne, A. (2018). Creating and sustaining a positive and communal school climate: Contemporary research, present obstacles, and future directions. Washington, DC : US Department of Justice.</bibtext> </blist> <blist> <bibtext> R Development Core Team. (2008). R: A language and environment for statistical computing. Vienna, Austria : R Foundation for Statistical Computing.</bibtext> </blist> <blist> <bibtext> Samejima, F. (1997). Graded Response Model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of Modern Item Response Theory. New York, NY : Springer.</bibtext> </blist> <blist> <bibtext> Shaw, N. (2013). States use school score cards to target climate problems. Retrieved from https://<ulink href="http://www.edweek.org/ew/articles/2013/03/27/26climate.h32.html">www.edweek.org/ew/articles/2013/03/27/26climate.h32.html</ulink></bibtext> </blist> <blist> <bibtext> Thapa, A., Cohen, J., Guffey, S., & Higgins‐D'Alessandro, A. (2013). A review of school climate research. Review of Educational Research, 83 (3), 357 – 385.</bibtext> </blist> <blist> <bibtext> Thissen, D. (2000). Reliability and measurement precision. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 159 – 184). Mahwah, NJ : Lawrence Erlbaum.</bibtext> </blist> <blist> <bibtext> U.S. Department of Education. (2009). Safe and supportive schools model. Retrieved from <ulink href="http://safesupportiveschools.ed.gov/index.php?id=33">http://safesupportiveschools.ed.gov/index.php?id=33</ulink>. Accessed March 11 2013.</bibtext> </blist> <blist> <bibtext> Waasdorp, T. E., Pas, E. T., O'Brennan, L. M., & Bradshaw, C. P. (2011). A multilevel perspective on the climate of bullying: Discrepancies among students, school staff, and parents. Journal of School Violence, 10 (2), 115 – 132.</bibtext> </blist> <blist> <bibtext> Waasdorp, T., Lindstrom Johnson, S., Shukla, K., & Bradshaw, C. (in press). School climate measurement: Invariance across middle and high school students. Children in Schools.</bibtext> </blist> <blist> <bibtext> Way, N., Reddy, R., & Rhodes, J. (2007). Students' perceptions of school climate during the middle school years: Associations with trajectories of psychological and behavioral adjustment. American Journal of Community Psychology, 40 (3–4), 194 – 213.</bibtext> </blist> </ref> <aug> <p>By Sarah Lindstrom Johnson; Ray E. Reichenberg; Kathan Shukla; Tracy E. Waasdorp and Catherine P. Bradshaw</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib37" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib11" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib33" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib25" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib18" firstref="ref8"></nolink> <nolink nlid="nl6" bibid="bib32" firstref="ref10"></nolink> <nolink nlid="nl7" bibid="bib36" firstref="ref13"></nolink> <nolink nlid="nl8" bibid="bib17" firstref="ref14"></nolink> <nolink nlid="nl9" bibid="bib23" firstref="ref15"></nolink> <nolink nlid="nl10" bibid="bib24" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib14" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib16" firstref="ref25"></nolink> <nolink nlid="nl13" bibid="bib42" firstref="ref26"></nolink> <nolink nlid="nl14" bibid="bib20" firstref="ref30"></nolink> <nolink nlid="nl15" bibid="bib12" firstref="ref36"></nolink> <nolink nlid="nl16" bibid="bib30" firstref="ref40"></nolink> <nolink nlid="nl17" bibid="bib39" firstref="ref43"></nolink> <nolink nlid="nl18" bibid="bib41" firstref="ref46"></nolink> <nolink nlid="nl19" bibid="bib10" firstref="ref48"></nolink> <nolink nlid="nl20" bibid="bib34" firstref="ref49"></nolink> <nolink nlid="nl21" bibid="bib35" firstref="ref50"></nolink> <nolink nlid="nl22" bibid="bib38" firstref="ref51"></nolink> <nolink nlid="nl23" bibid="bib22" firstref="ref52"></nolink> <nolink nlid="nl24" bibid="bib26" firstref="ref53"></nolink> <nolink nlid="nl25" bibid="bib27" firstref="ref54"></nolink> <nolink nlid="nl26" bibid="bib13" firstref="ref55"></nolink> <nolink nlid="nl27" bibid="bib21" firstref="ref56"></nolink> <nolink nlid="nl28" bibid="bib28" firstref="ref57"></nolink> <nolink nlid="nl29" bibid="bib29" firstref="ref58"></nolink> <nolink nlid="nl30" bibid="bib15" firstref="ref59"></nolink> <nolink nlid="nl31" bibid="bib19" firstref="ref60"></nolink> <nolink nlid="nl32" bibid="bib31" firstref="ref66"></nolink> <nolink nlid="nl33" bibid="bib40" firstref="ref74"></nolink>
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED604048
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED604048
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Improving the Measurement of School Climate Using Item Response Theory
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Lindstrom+Johnson%2C+Sarah%22">Lindstrom Johnson, Sarah</searchLink><br /><searchLink fieldCode="AR" term="%22Reichenberg%2C+Ray+E%2E%22">Reichenberg, Ray E.</searchLink><br /><searchLink fieldCode="AR" term="%22Shukla%2C+Kathan%22">Shukla, Kathan</searchLink><br /><searchLink fieldCode="AR" term="%22Waasdorp%2C+Tracy+E%2E%22">Waasdorp, Tracy E.</searchLink><br /><searchLink fieldCode="AR" term="%22Bradshaw%2C+Catherine+P%2E%22">Bradshaw, Catherine P.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2019.
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 31
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Department of Education (ED)<br />Institute of Education Sciences (ED)<br />National Institute of Justice (NIJ) (DOJ)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305H150027<br />2014CKBX0005
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Environment%22">Educational Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Accountability%22">Accountability</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Legislation%22">Educational Legislation</searchLink><br /><searchLink fieldCode="DE" term="%22Federal+Legislation%22">Federal Legislation</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement+Techniques%22">Measurement Techniques</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Characteristics%22">Institutional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Analysis%22">Item Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Evaluation%22">Institutional Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+Schools%22">Secondary Schools</searchLink>
– Name: SubjectThesaurus
  Label: Laws, Policies and Program Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Every+Student+Succeeds+Act+2015%22">Every Student Succeeds Act 2015</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/emip.12296
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The United States government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the "Every Student Succeeds Act". Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item-response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n= 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the 'mirt' package in R using unidimensional item response theory. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure. [This paper was published in "Educational Measurement: Issues and Practice" v38 n4 2019 (EJ1236481).]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED604048
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED604048
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.12296
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 31
    Subjects:
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Educational Environment
        Type: general
      – SubjectFull: Accountability
        Type: general
      – SubjectFull: Educational Legislation
        Type: general
      – SubjectFull: Federal Legislation
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Measurement Techniques
        Type: general
      – SubjectFull: Institutional Characteristics
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Item Analysis
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Institutional Evaluation
        Type: general
      – SubjectFull: Secondary Schools
        Type: general
      – SubjectFull: Every Student Succeeds Act 2015
        Type: general
    Titles:
      – TitleFull: Improving the Measurement of School Climate Using Item Response Theory
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lindstrom Johnson, Sarah
      – PersonEntity:
          Name:
            NameFull: Reichenberg, Ray E.
      – PersonEntity:
          Name:
            NameFull: Shukla, Kathan
      – PersonEntity:
          Name:
            NameFull: Waasdorp, Tracy E.
      – PersonEntity:
          Name:
            NameFull: Bradshaw, Catherine P.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2019
          Titles:
            – TitleFull: Grantee Submission
              Type: main
ResultId 1