View in EDS HTML Full Text PDF Full Text

Measuring Scientific Classroom Discourse: The 'DiISC Version 2.0's' Validity and Use in Observing Secondary Science Lessons

Saved in:
Bibliographic Details
Title: Measuring Scientific Classroom Discourse: The 'DiISC Version 2.0's' Validity and Use in Observing Secondary Science Lessons
Language: English
Authors: Elizabeth Lewis (ORCID 0000-0002-3429-3003), Lyrica Lucas, Brandon Helding, Amy Tankersley, Elizabeth Hasseler, Ana Rivero, Dale Baker
Source: School Science and Mathematics. 2026 126(2):143-158.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 16
Publication Date: 2026
Sponsoring Agency: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
Contract Number: 1540797
Document Type: Journal Articles
Reports - Evaluative
Education Level: Secondary Education
Descriptors: Science Education, Science Instruction, Teaching Methods, Secondary School Science, Discussion (Teaching Technique), Science Teachers, Inquiry, Discourse Analysis, Measures (Individuals), Discourse Communities, Student Evaluation, Oral Language, Written Language, Factor Structure, Test Validity, Academic Language
DOI: 10.1111/ssm.18325
ISSN: 0036-6803
1949-8594
Abstract: To continue to support long-term, ongoing science curriculum and instruction reform efforts in the United States, there is a significant need to be able to reliably measure teachers' discourse-rich, inquiry-based instruction. In this external validation study, we present the "Discourse in Inquiry Science Classrooms Version 2.0" (DiISC 2.0) as a valuable observational instrument for researchers to investigate multiple aspects of science teachers' lessons. The DiISC 2.0 is grounded in a research-based conceptual framework of a scientific classroom discourse community, focused on fundamental socioconstructivist characteristics of lessons, including scientific inquiry, oral and written discourse, and academic language development. We collected and analyzed new data from 807 science lessons to develop and expand the instrument's original validity argument beyond the associated professional development program for broader use. We determined the DiISC's factor structure and examined its correlation with the "Electronic Quality of Inquiry Protocol." Finally, some items that did not represent DiISC constructs were removed. Thus, the "DiISC 2.0" instrument can be used to measure students' opportunities to learn science and provide feedback to teachers on their progress toward building an inclusive scientific classroom discourse community for all students, in particular, historically marginalized groups and multilingual learners.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1502629
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwHMA-QXZLebxLVsL9kVOEutAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDE1IFBFrUSyxbiKnKgIBEICBm8Qc5ODvrpcj-3XwIeKtZOf0scAa2I5sbm_3lXOJws7SZJE77-LQmbqRRoGgyohbPTGQ1rpHbpUyTv4pm4waQwceCWFXbZvCkbmj9TQ_WlrYYTXwzI80CqBs4mbp3ILZrntFWolcx4KUhCnY2jYuYdvP1WtRHV4xcg9Qz2n_uoT2utByWWAfSI_IOjcpEkjVBTHCoqy4V4EwkowO
Text:
  Availability: 1
  Value: <anid>AN0192956349;ssm01apr.26;2026Apr15.02:40;v2.2.500</anid> <title id="AN0192956349-1">Measuring scientific classroom discourse: The DiISC Version 2.0's validity and use in observing secondary science lessons </title> <p>To continue to support long‐term, ongoing science curriculum and instruction reform efforts in the United States, there is a significant need to be able to reliably measure teachers' discourse‐rich, inquiry‐based instruction. In this external validation study, we present the Discourse in Inquiry Science Classrooms Version 2.0 (DiISC 2.0) as a valuable observational instrument for researchers to investigate multiple aspects of science teachers' lessons. The DiISC 2.0 is grounded in a researchbased conceptual framework of a scientific classroom discourse community, focused on fundamental socioconstructivist characteristics of lessons, including scientific inquiry, oral and written discourse, and academic language development. We collected and analyzed new data from 807 science lessons to develop and expand the instrument's original validity argument beyond the associated professional development program for broader use. We determined the DiISC's factor structure and examined its correlation with the Electronic Quality of Inquiry Protocol. Finally, some items that did not represent DiISC constructs were removed. Thus, the DiISC 2.0 instrument can be used to measure students' opportunities to learn science and provide feedback to teachers on their progress toward building an inclusive scientific classroom discourse community for all students, in particular, historically marginalized groups and multilingual learners.</p> <p>Keywords: classroom observation instrument; evaluation; science education standards; secondary science; validity study</p> <hd id="AN0192956349-2">INTRODUCTION</hd> <p>Educational research measures with strong validity arguments are perennially needed to study science teaching and learning reliably. In this external validation study, we present the <emph>Discourse in Inquiry Science Classrooms</emph>, <emph>Version 2.0</emph> (DiISC 2.0) as a valuable observational instrument for researchers to investigate science teachers' lesson implementation concerning student‐centered meaning‐making. The DiISC 2.0 is grounded in a conceptual framework of a scientific classroom discourse community focused on socio‐constructivist lesson characteristics. Our project aimed to validate a classroom observation instrument to reliably measure secondary science teachers' use of scientific discourse in their inquiry‐based and inclusive instruction. Robust validity and reliability arguments for available instruments are necessary so that researchers can produce consistent, replicable, and generalizable results in their assessment of classroom lessons (American Educational Research Association [AERA], American Psychological Association [APA], and National Council of Measurement in Education [NCME], [<reflink idref="bib3" id="ref1">3</reflink>]).</p> <p>By producing an externally validated instrument, we address the research community's need for a greater variety of science classroom observation instruments. For the <emph>DiISC 2.0</emph> (Supplemental Materials, Part A), we constructed a validity argument for a measure of secondary science teachers' instruction that had initially been developed as a program‐specific research instrument. We accomplished this by generating a new dataset to study early‐ and mid‐career science teachers' instruction (Lewis et al., [<reflink idref="bib33" id="ref2">33</reflink>]) and using Kane's ([<reflink idref="bib21" id="ref3">21</reflink>]) validity framework. Another research team first developed the <emph>Discourse in Inquiry Science Classrooms</emph> (DiISC) (Baker et al., [<reflink idref="bib5" id="ref4">5</reflink>]) for the <emph>Communication in Science Inquiry Project</emph> (CISIP) to measure teachers' construction of science classroom discourse communities as they underwent long‐term professional development (PD) (Baker et al., [<reflink idref="bib4" id="ref5">4</reflink>]; Baker et al., [<reflink idref="bib6" id="ref6">6</reflink>]; Lewis et al., [<reflink idref="bib31" id="ref7">31</reflink>]). Specifically, both DiISC versions measure instructional practices in inquiry, oral and written discourse, academic learning development strategies, and learning principles (LP) (Lewis et al., [<reflink idref="bib31" id="ref8">31</reflink>]).</p> <p>While science education standards are periodically updated, learning theories (e.g., social constructivism, self‐regulation, and metacognition) and empirically supported pedagogical strategies for robust scientific literacy and equity have continued to be foundational for meeting science education reform goals (i.e., <emph>Project 2061</emph>, American Association for the Advancement of Science [AAAS], [<reflink idref="bib2" id="ref9">2</reflink>]). We describe these elements in a model of effective science teaching as a reliable navigational reference for developing a practical and empirically grounded classroom observation instrument. Through external validation, DiISC 2.0 can be used to support US science education reform by investigating secondary science teachers' inquiry‐based instructional practices. However, the first version of the instrument has also had an international audience, having been downloaded over 2200 times by individuals from over 330 institutions in 80 countries (e.g., most often in the United States, Philippines, India, China, Australia, United Kingdom, Germany, and Turkey). If researchers used the original DiISC, they would also need to have performed their own validation, as we did, to produce reliable research results. Since this is the first time anyone else has published a study using the instrument in a different context and provided an external validity argument or a new version, we did so with a new opportunity to study secondary science teachers. Other researchers focused on American schools, as in this study, can now use our validity argument for the <emph>DiISC 2.0</emph> and the updated instrument to research discourse‐rich, inclusive science teaching.</p> <hd id="AN0192956349-3">SCIENCE EDUCATION REFORM EFFORTS IN THE UNITED STATES</hd> <p>Classroom observation instruments are crucial to understanding educational reform because they measure classroom activity and events directly. Science education reform efforts in the United States (US) have been ongoing for over a century since the Committee of Ten (Mackenzie, [<reflink idref="bib34" id="ref10">34</reflink>]), spurred on by the 1957 Sputnik launch that fueled the international space race. A famous reform‐oriented program was the Biological Sciences Curriculum Study (BSCS), now an independent organization, BSCS Science Learning (BSCS, [<reflink idref="bib8" id="ref11">8</reflink>]). In the mid‐1980s, the BSCS generated the 5E inquiry‐based instructional model that started with elementary science curriculum reform efforts (Bybee et al., [<reflink idref="bib9" id="ref12">9</reflink>]; Karplus & Thier, [<reflink idref="bib23" id="ref13">23</reflink>]). The 5E model is rooted in social constructivist learning theory (Vygotsky, [<reflink idref="bib46" id="ref14">46</reflink>]) and the "learning cycle" (Karplus, [<reflink idref="bib22" id="ref15">22</reflink>]; Karplus & Thier, [<reflink idref="bib23" id="ref16">23</reflink>]). In the 1990s, US K‐12 science education reform and inquiry‐based learning were articulated in the <emph>National Science Education Standards</emph> (NSES) (NRC, [<reflink idref="bib39" id="ref17">39</reflink>]). However, the first US NSES rendered the inquiry standards ignorable. More recently, the standards have undergone a 21st‐century update and, under inquiry‐based instruction, focus on scientific and engineering practices in the <emph>Next Generation Science Standards</emph> (NGSS) (Crawford, [<reflink idref="bib13" id="ref18">13</reflink>]). In a three‐dimensional design, integrating science and engineering practices, crosscutting concepts, and disciplinary core ideas in each NGSS performance expectation indicates stronger inclusion of scientific inquiry (NRC, [<reflink idref="bib42" id="ref19">42</reflink>]). Researchers who seek to study specific aspects of the NGSS should reference other instruments, that is, the eight scientific practices (Chen & Terada, [<reflink idref="bib12" id="ref20">12</reflink>]) and engineering design integration in science classrooms (Wheeler et al., [<reflink idref="bib48" id="ref21">48</reflink>]).</p> <hd id="AN0192956349-4">INSTRUMENT EXTERNAL VALIDATION PROJECT: PURPOSE AND RATIONALE</hd> <p>National standards can encourage curricular change at state and local levels and researchers need instruments with strong validity arguments to reliably measure teachers' enacted curriculum and use of inquiry‐based instruction over time. Such instruments that attend to core aspects of effective teaching also provide teachers feedback regarding their PD needs in inclusive inquiry‐based instructional strategies. However, few science education instruments have been developed for this purpose. Lawrenze and Goodyear ([<reflink idref="bib26" id="ref22">26</reflink>]) summarize research instruments in their review of the science education literature on program evaluation, many of which are surveys designed to focus on discipline‐specific topics, the nature of science, or large‐scale assessment data. In an older review, Lawrenze and Thao ([<reflink idref="bib27" id="ref23">27</reflink>]) identify more widely used observational instruments to evaluate science teaching practices holistically, most notably the popular <emph>Reform‐based Teaching Protocol</emph> (Sawada et al., [<reflink idref="bib45" id="ref24">45</reflink>]). In the interest of space, we refer readers to these handbook chapters to explore other instruments. Importantly, Lawrenze and Thao ([<reflink idref="bib27" id="ref25">27</reflink>]) identified the information shortage on the construction and validity of data collection instruments as part of their critique of science education program evaluation. This report summarizes our new external validity evidence to further the DiISC's initial design and validation so that the instrument can be reliably used to observe science classrooms beyond the original associated PD program.</p> <hd id="AN0192956349-5">BACKGROUND: INITIAL INSTRUMENT DEVELOPMENT AND LITERATURE REVIEW</hd> <p>By design, the DiISC measures teachers' use of inclusive, inquiry‐based science instructional strategies as viewed through a conceptual model of a science classroom discourse community that explicitly addresses the nature of science communication and academic language development (ALD) for all students (Figure 1). Initially, the model was developed for the associated CISIP teacher PD program grounded in situated learning (Lave & Wegner, [<reflink idref="bib25" id="ref26">25</reflink>]; Wegner, [<reflink idref="bib47" id="ref27">47</reflink>]). The instrument developers had a strong focus on developing scientific literacy and communication skills through oral discourse (OD) and writing support for students to develop structured, coherent ideas (Kelly et al., [<reflink idref="bib24" id="ref28">24</reflink>]; Lewis et al., [<reflink idref="bib31" id="ref29">31</reflink>]; Rivard & Straw, [<reflink idref="bib44" id="ref30">44</reflink>]). Social constructivist learning theory is foundational to this model, focusing on using scientific inquiry to construct knowledge with others (NRC, [<reflink idref="bib39" id="ref31">39</reflink>]). The CISIP model also emphasizes characteristics of scientific literacy, including text structures and argumentation (Halliday & Martin, [<reflink idref="bib19" id="ref32">19</reflink>]; Osborne, [<reflink idref="bib43" id="ref33">43</reflink>]). The DiISC was also developed to measure use of cognitive LP and instructional practices (NRC, [<reflink idref="bib40" id="ref34">40</reflink>]).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/SSM/01apr26/ssm18325-fig-0001.jpg?ephost1=dGJyMMvl7ESepq84yOvsOLCmsE6epq5Srqa4SK6WxWXS" alt="ssm18325-fig-0001.jpg" title="1 Original professional development theoretical and conceptual framework (from Lewis et al., [31])." /> </p> <p></p> <p>The CISIP PD program focused on teachers' learning of building a science classroom discourse community committed to equity and inclusion to support all students' learning. Numerous items attend to inclusive teaching practices. For example, on the OD scale, there is item OD‐3, "Teacher (or instruction) bridges everyday experiences and scientific discourse," in which observers look for evidence that the teacher is sensitive to students' intersectional identities and discourse (i.e., using topics of interest to all students) and connects every day (e.g., pop culture, alternative literacies (Gee, [<reflink idref="bib17" id="ref35">17</reflink>])) and scientific discourse. Another item, on the ALD scale, ALD‐5, "Teacher addresses multiple levels of academic language proficiency (differentiated instruction and/or assessment)," directs observers to rate the degree to which the teacher provides activities of varying academic linguistic demands, uses assessments that match academic language proficiency, and/or adjusts pedagogy to the student(s)'s language proficiency. Details of the iterative design and construct validity that resulted in the original 36‐item DiISC instrument are included in the Supplemental Materials (Part B).</p> <p>The CISIP model of a scientific classroom discourse community for effective science teaching has continued to be supported by science education priorities. Kelly et al. ([<reflink idref="bib24" id="ref36">24</reflink>]), outline a detailed description of the discussion and priorities of scientific discourse in their handbook chapter on discourse practices in science learning. From a theoretical perspective, Gee ([<reflink idref="bib18" id="ref37">18</reflink>]) and Kelly et al. ([<reflink idref="bib24" id="ref38">24</reflink>]) argue that "studies of scientific practice <emph>in situ</emph> have identified the importance of discourse practices in constructing scientific knowledge" (p. 415). The CISIP model reflects the current stance in science education that discourse is more complex than just language in use but should be interpreted as "situated in ongoing sociocultural practices" (Kelly et al., [<reflink idref="bib24" id="ref39">24</reflink>], p. 415).</p> <hd id="AN0192956349-7">Definitions and description of the DiISC observation instrument scales</hd> <p>All the original DiISC instrument items were drafted based on known effective instructional strategies from the educational research base, field‐tested, and iteratively refined, addressing five aspects of a scientific classroom discourse community (Lewis et al., [<reflink idref="bib31" id="ref40">31</reflink>]). Each latent scale or factor is described as follows. The complete updated instrument is available in the Supplementary Materials in Part A, along with factor loadings table (Appendix, Table A1), and a comparison chart between the original and final DiISC 2.0 items (Appendix, Table A2).</p> <hd id="AN0192956349-8">Inquiry scale</hd> <p>Inquiry‐based education has long been the focus of high‐quality science education, notably beginning with Dewey's emphasis on student‐centered learning, as outlined in <emph>Experience and Education</emph> (Dewey, [<reflink idref="bib14" id="ref41">14</reflink>]) and, more recently, in its inclusion in <emph>Science for All Americans</emph> (American Association for the Advancement of Science, [<reflink idref="bib1" id="ref42">1</reflink>]) and the <emph>NSES</emph> (NRC, [<reflink idref="bib39" id="ref43">39</reflink>]). The Inquiry (I) scale addresses real‐world skills that scientists use, such as modeling and analyzing data, as seen now in the NGSS science practices (SPs) (NRC, [<reflink idref="bib41" id="ref44">41</reflink>]). A foundation of inquiry‐based science instruction is an inclusive learning environment in which students develop and use complex cognitive processes (Windschitl et al., [<reflink idref="bib50" id="ref45">50</reflink>]). The Inquiry scale focuses on lesson aspects that support inquiry and student‐centered instruction such that students engage with scientific phenomena through varying degrees of independence. It also focuses on observable teacher instructional practices found in discourse‐rich inquiry‐based lessons, such as those promoted by the 5E instructional model (Bybee et al., [<reflink idref="bib9" id="ref46">9</reflink>]). Table 1 provides a pivotal example, item I‐5, that addresses explaining phenomena via claims and evidence. The item focuses on the degree to which the teacher provides students with opportunities to make claims, provide evidence, develop explanations, revise their explanations and models using data and logic, and/or make predictions and build models.</p> <p>1 TABLE DiISC scale descriptions and selected item examples.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Scale</th><th align="left">Descriptions</th><th align="left">Example item</th></tr></thead><tbody valign="top"><tr><td align="left">Inquiry</td><td align="left">Measures the degree to which teaching takes place in a student‐centered classroom where students are actively engaged in activities to explore the natural world with varying degrees of investigative independence using scientific practices</td><td align="left">I‐5. Opportunities for later stages of scientific exploration: explaining phenomena via claims and evidence, making predictions, and/or building models.Rubric:0 = no use of data for scientific explanation; 1 = teacher‐led, incidental use of claims and evidence; 2 = students generate scientific explanation and/or models; 3 = includes all of 2 and teacher directs students to evaluate their scientific explanations and revise</td></tr><tr><td align="left">Oral discourse</td><td align="left">Measures the degree to which teachers bridge everyday experiences and scientific discourse by providing students with opportunities to build scientific vocabulary, engage in peer‐to‐peer discussions that lead to scientific explanations, and exploring the nature of scientific communication</td><td align="left">OD‐4. Teacher engages students in discussion that emphasizes the nature of science.Rubric:0 = no discussion of NOS; 1 = teacher transmission of information about NOS; 2 = whole group or small group discussion of NOS; 3 = teacher facilitates in‐depth discussion of the NOS with whole group</td></tr><tr><td align="left">Written discourse</td><td align="left">Measures the degree to which teachers provide students with opportunities to pre‐write, write, and share their writing in order to acquire the language patterns and vocabulary to communicate scientific ideas, use science notebooks, and write in a variety of genres</td><td align="left">WD‐1. Formal writing in a genre that reflects the nature of science.Rubric:0 = no formal writing; 1 = writing is unstructured or simply restated from text; 2 = teacher provides a limited data set to students to write with a purpose; 3 = teacher provides students a clear structure incorporating high level of inquiry, specific audience, and reflects the NOS</td></tr><tr><td align="left">Academic language development</td><td align="left">Measures the degree to which teachers use visual aids, supplemental resource materials, clear instruction throughout the lesson, and lessons that build on students' language and culture. It also measures instruction for student interactions and academic learning strategies and opportunities for students to acquire scientific vocabulary</td><td align="left">ALD‐4. Building lesson on students' language (vernacular or non‐English) OR culture.Rubric.0 = teacher does not incorporate links to language or culture; 1 = minor use of students' language or culture; 2 = teacher bridges students' language and culture consistently through lesson; 3 = lesson is planned and executed using familiar language with culturally relevant links to science content</td></tr><tr><td align="left">Learning principles</td><td align="left">Measures the degree to which the teacher aligns lessons with the CISIP model. This includes providing opportunities for students to assess prior knowledge, make conceptual connections, and engage in metacognition. The teacher also models thinking, establishes community norms, and promotes an academic focus that supports learning science</td><td align="left">LP‐2. Teacher and/or students situate factual knowledge (experiences, ideas, data, and explanations to past lessons and/or real‐world experiences) within a conceptual framework (fact to concept relationship).Rubric:0 = no conceptual framework utilized, just factual information; 1 = teacher provides informal opportunities for students to generate understanding of topics; 2 = teacher provides formal structure for generating understanding of facts within a conceptual framework; 3 = teacher provides opportunities and monitors student understanding</td></tr></tbody></table> </ephtml> </p> <hd id="AN0192956349-9">Oral discourse (OD) scale</hd> <p>The OD scale assesses how instruction scaffolds everyday language, culture, and experiences to scientific discourse to support students' development of scientific literacy (Osborne, [<reflink idref="bib43" id="ref47">43</reflink>]). For example, students can engage with their peers in small group discussions (item OD‐2 teacher promotes peer‐to‐peer discussion) as they develop their understanding of scientific concepts. In addition, the scale includes items to identify if students have experiences in which they can develop their ability to use scientific discourse. Scientific communication in classrooms incorporates a variety of skills that students need to build, such as experiencing and understanding scientific phenomena and being able to read, talk, and write about those phenomenon (Kelly et al., [<reflink idref="bib24" id="ref48">24</reflink>]), or the use of argumentation through speaking and writing, as well as understanding and doing science (Lemke, [<reflink idref="bib30" id="ref49">30</reflink>]; Windschitl et al., [<reflink idref="bib50" id="ref50">50</reflink>]). Item OD‐4 specifically directs observers to determine the extent to which teachers engage students in discussion that emphasizes the nature of science (Table 1 and Supplemental Materials, Part A). Finally, inclusive classroom scientific discourse supports the co‐construction of knowledge where all students can bridge their experiences with learning and develop a science identity (Kelly et al., [<reflink idref="bib24" id="ref51">24</reflink>]; Lee et al., [<reflink idref="bib29" id="ref52">29</reflink>]).</p> <hd id="AN0192956349-10">Written discourse (WD) scale</hd> <p>The WD scale measures the opportunities that teachers provide students to develop their WD skills (i.e., pre‐writing (WD‐2), recursive writing (WD‐3), and peer evaluation (WD‐3)). Importantly, these instructional strategies allow students to develop scientific literacy through written discourse. Some of these strategies include the development of language patterns and vocabulary to communicate scientific ideas in a science classroom discourse community. In sum, through scientific communication expressed through writing (WD‐1), students can build and reflect on their understanding of scientific concepts (Halliday & Martin, [<reflink idref="bib19" id="ref53">19</reflink>]; Lemke, [<reflink idref="bib30" id="ref54">30</reflink>]) (Table 1).</p> <hd id="AN0192956349-11">Academic language development scale</hd> <p>The ALD scale items focus on constructivist teaching that increasingly supports scientific literacy through visual aids (ALD‐3), supplemental resource materials (ALD‐8), and clear instruction (ALD‐2). It also assesses how teachers' lessons connect students' everyday language and culture with learning scientific vocabulary (ALD‐4, Table 1). The ALD scale incorporates multiple practices from Herrell and Jordan ([<reflink idref="bib20" id="ref55">20</reflink>]) that support multilingual learners in their science lessons (Lee et al., [<reflink idref="bib29" id="ref56">29</reflink>]). Additionally, these strategies (e.g., using visuals and gestures) are also supportive of all students learning science.</p> <hd id="AN0192956349-12">Learning principles scale</hd> <p>The LP scale was constructed based on key constructs in <emph>How People Learn</emph> (NRC, [<reflink idref="bib40" id="ref57">40</reflink>]). The LP scale items assess whether students have time to identify their prior knowledge (LP‐1) and reflect upon their learning (LP‐4) (Dori et al., [<reflink idref="bib15" id="ref58">15</reflink>]). Item LP‐2 (Table 1) focuses on students' opportunities to integrate factual knowledge with conceptual frameworks to learn science (NRC, [<reflink idref="bib40" id="ref59">40</reflink>]). A strong connection across the DiISC is that in accessing students' prior knowledge, teachers should provide inquiry‐based science learning opportunities that bridge and draw upon students' cultural funds of knowledge (National Academies of Sciences, Engineering, and Medicine, [<reflink idref="bib38" id="ref60">38</reflink>]). The LP scale also illustrates whether teachers address and illustrate scientific thinking, establish a welcoming classroom culture (LP‐7), and support students throughout learning science in their lessons (Windschitl et al., [<reflink idref="bib50" id="ref61">50</reflink>]). Finally, item LP‐8 assesses how teachers incorporate immediate feedback into instruction as formative assessment (Black & Wiliam, [<reflink idref="bib7" id="ref62">7</reflink>]).</p> <hd id="AN0192956349-13">METHODOLOGY AND MEASUREMENT OF INQUIRY‐BASED INSTRUCTION</hd> <p>To build an external validation argument for the DiISC, we strategically used another validated classroom observation instrument, the 19‐item <emph>Electronic Quality of Inquiry Protocol</emph> (EQUIP), because it also addresses classroom inquiry (Marshall et al., [<reflink idref="bib35" id="ref63">35</reflink>]). While both EQUIP and DiISC measure constructs associated with inquiry‐based instruction, these latent constructs are described by different observed variables. Specifically, DiISC items assess observable instructional strategies that teachers can incorporate into their lessons for inquiry, oral discourse, writing, academic language development, and learning principles; inquiry items are their scale. Similarly, the EQUIP items focus on the observed actions of teachers to determine the degree of student‐centered nature of the lesson in the four areas of instruction, discourse, assessment, and curriculum; the level of inquiry is inferred rather than scored directly. However, the EQUIP instrument does not explicitly address ALD or culturally relevant pedagogies; the DiISC instrument does and can provide measures of inclusive teaching practices that support diverse students and multilingual learners (Lee, [<reflink idref="bib28" id="ref64">28</reflink>]).</p> <hd id="AN0192956349-14">Instrument validity versus validity arguments</hd> <p>Recently, validity as a concept has become an interpretive argument that focuses on how it is used by other researchers in their studies (Messick, [<reflink idref="bib36" id="ref65">36</reflink>], [<reflink idref="bib37" id="ref66">37</reflink>]). Historically, validity theory recognizes discrete validity categories, but contemporary validity theory argues that test validity is a single concept (Messick, [<reflink idref="bib37" id="ref67">37</reflink>]). The idea of validity as an aspect of an instrument has evolved into an argument for how it is used. The current conceptualization of validity stresses the importance of the instrument's intended purpose. Therefore, validity arguments are contingent on multiple aspects of a larger argument (Kane, [<reflink idref="bib21" id="ref68">21</reflink>]) and are no longer considered a static instrument characteristic, but rather an evolving and changing property continually affected by new data collection and analysis. In modern validity involves six to eight elements or aspects, each requiring different evidence types and thresholds, building a preponderance for an argument. Next, we discuss the six aspects of our validity argument: (a) content, (b) substantive, (c) structural, (d) external, (e) generalizability, and (f) consequential validity.</p> <hd id="AN0192956349-15">Content validity</hd> <p>Content validity arguments use artifact analyses to ensure that the content intended to be assessed is assessed. This is akin to traditional "face" validity and typically involves a table of specifications when developing a new measure. When developing a validity argument, we use a domain analysis to ensure that instrument scores involve the content of interest. This is part of a larger validity argument but is a foundation for justifying how any instrument describes a setting, person, or activity. It corresponds with our fundamental theory of measurement.</p> <hd id="AN0192956349-16">Substantive validity</hd> <p>Evidence for the substantive validity argument establishes the underlying processes by which an instrument is administered and that the scores are consistent with the construct validity evidence. This is typically not an observable process as the underlying human processes in scoring or responding to instrument items involve knowledge and attitudes of those individuals being observed or responding to items. In this case, it was important to understand why and how scores on the DiISC were assigned to teacher behaviors. Accordingly, we used think‐aloud interviews (Ericcson & Simon, [<reflink idref="bib16" id="ref69">16</reflink>]), which allow those using the instrument to describe the cognitive processes used to generate ratings on specific items on the instrument. This has implications for interrater reliability and how items are presented to instrument users. Items must correspond with the observed characteristics rather than processes used to make observations.</p> <hd id="AN0192956349-17">Structural validity</hd> <p>Evidence for an instrument's structural validity is based upon the notion that observed scores correspond with unseen characteristics, in our case, inquiry‐based instruction. This is typically explored, established by, and interpreted with factor analyses. Therefore, we factored the instrument to ensure that the number of factors underlying an observation could be interpreted coherently. We determined how some factors were consistent with the measured content, and how the observational tool was used in this project.</p> <hd id="AN0192956349-18">External validity</hd> <p>External validity evidence is based on how well an instrument corresponds with another instrument intended to measure similar characteristics or constructs that already have a high‐quality validity argument. In dichotomous settings, selection ratios and contingency tables are helpful. In our project, we correlated the instrument of interest (DiISC) with an external measure with extant validity argumentation (EQUIP). Because multiple factors were involved, we used a multivariate approach that involved an overall model fit, standard errors of measurement, and significance tests.</p> <hd id="AN0192956349-19">Generalizability validity</hd> <p>Generalizability validity involves evidence that scores can be interpreted over different contexts, formats, and populations (with subpopulations). This involves examining how items are administered, which is important when tests involve one‐way human interaction. It also affects how an observational instrument, such as the DiISC, performs when administered to subgroups of teachers. Finally, the process involves analyzing differential item functioning (DIF), in which factors not associated with the content being assessed change the scores involved in the assessment. For example, the teachers' sex should not cause their scores to change on a measure of anything unrelated to their sex. If it does, then there is clear evidence of potential DIF.</p> <hd id="AN0192956349-20">Consequential validity</hd> <p>The most contentious aspect of a modern validity argument involves how an instrument is used and the consequences associated with the scores. A validity argument makes the case that a measure is appropriate for a particular use, but if used in unintended ways, it may have unintended consequences for those being observed. Consequential validity asks if the developers of an instrument and its validity argument are responsible for how others use it. This debate continues among psychometricians but is beyond this paper's scope.</p> <hd id="AN0192956349-21">Summary</hd> <p>There are six aspects to our validity argument. We focused on five aspects, excluding consequential validity evidence, because: (a) there is no clear consensus on evidence for it, (b) we do not govern how others may use the DiISC, and (c) few studies have used the DiISC to make broad claims that might deviate from its original purpose. This argument‐based approach to validating the use of the DiISC enhances how we can capture inquiry‐based teaching practices, including discourse, academic language development, and learning principles. For a high‐quality validity argument, various types of data and analysis are needed (Kane, [<reflink idref="bib21" id="ref70">21</reflink>]), including recommendations on the appropriate instrument use. This report assesses evidence for the emerging validity argument and provides evidence for those five aspects below.</p> <hd id="AN0192956349-22">VALIDITY EVIDENCE</hd> <p></p> <hd id="AN0192956349-23">Content validity</hd> <p>Although the DiISC was developed within the CISIP project, the items were initially designed using an educational research literature search that included social constructivism and inquiry‐based instruction (Baker et al., [<reflink idref="bib4" id="ref71">4</reflink>]; Lewis et al., [<reflink idref="bib31" id="ref72">31</reflink>]). In addition, to support the content validity of the original DiISC, there is a table and description that includes items and scoring rubrics with the literature and standards in its user's manual (Baker et al., [<reflink idref="bib5" id="ref73">5</reflink>]).</p> <p>To begin the validation process, we reviewed the original instrument developers' development processes, reliability measures, and analysis to establish the content, face, construct, and concurrent validity (Baker et al., [<reflink idref="bib5" id="ref74">5</reflink>]). The development process resulted in a high intraclass correlation, <emph>r</emph> = 0.90, illustrating a strong agreement among raters. The original developers of the DiISC also conducted an EFA using 204 lesson observations of secondary science teachers in their study, which showed five factors, many items of which cross‐loaded or failed to meet basic criteria associated with structural validity. Criterion validity evidence was developed using a correlation between science lesson observation DiISC scores and the "My Science Classroom" survey given to 187 students. The survey measures students' views of the instructional strategies incorporated in science lessons. There was a statistically significant correlation between DiISC scores and the survey (<emph>r</emph> = 0.80, <emph>p</emph> < 0.01). However, without further validation of DiISC scores collected outside of the original context, the original DiISC lacked the detailed validation evidence required for reliable, widespread use. The validity argument for the original DiISC supports the use of the DiISC to observe changes in instruction for participants in the CISIP PD program. Our study's validity argument for the updated DiISC 2.0 supports its use to measure aspects of reform‐based instruction (e.g., inquiry, discourse, learning principles) in U.S. secondary science classrooms. As part of our collection of valid evidence, we conducted several analyses. We add to previous work using the aforementioned aspects of the validity argument; each is discussed next.</p> <hd id="AN0192956349-24">Substantive validity argument</hd> <p>To generate a substantive validity argument, we showed four independent raters the same science lesson video and interviewed them. These rates were the same research group that collected the longitudinal dataset on secondary science teachers' instruction. The lesson for these cognitive interviews was on biological phenotypes and genotypes, and the teacher used worksheets, short animations, and whole‐group instruction. The classroom discourse was structured mainly in an Initiate‐Respond‐Evaluate format, with two notable exceptions: a video and a worksheet with notebooks. The lesson was separated into 11 segments based on different instructional elements to reveal how each researcher approached the task of coding the lesson using the original DiISC instrument. For example, an activity involving students writing in their science notebooks was differentiated from lectures or completing low‐level questions on worksheets. After each segment, the interviewer, who was not a research team member, paused the video and asked open‐ended questions corresponding to inquiry‐based instructional practices. The purpose was to investigate these observers' cognitive processes when assigning quantitative codes to teachers' classroom instruction using the DiISC item rubrics. During each lesson segment, the observers could also request that the video be paused to provide additional comments. Follow‐up questions were used to encourage the researchers to describe their coding decisions further. At the end of the video, each observer was asked about the overall quality of the lesson.</p> <p>The cognitive processes the research team engaged in constructing a substantive validity argument were important to how the DiISC instrument would be used. Accordingly, notes from these interviews were read iteratively and coded using qualitative methods of constant comparison. The DiISC scores produced by the raters were compared with their statements as they assigned item values assessing the teacher's instruction. Codes were collated into themes and themes into assertions. Finally, the case was made that the raters' overall cognitive processes were similar when coding each element of the teacher's behaviors. Findings from the cognitive interviews of raters supported a substantive validity argument. Specifically, the trends or assertions in the raters' thinking were comparable when they applied codes to classroom observations.</p> <hd id="AN0192956349-25">Raters' foci</hd> <p> <emph>Assertion 1</emph>: <emph>each observer's focus differed slightly</emph>. Our first assertion was built from the overall content of the completed interviews about the observed lesson. Each rater found separate aspects of the lesson interesting. One noted the lack of inquiry‐based instruction, issues with school culture, and educational priorities that may have influenced how the teacher taught the lesson, and the lack of metacognitive opportunities. Another noted how the teacher focused mainly on specific concepts, had time management problems, and could have better sequenced the lesson. A third commented that while the teacher accessed students' prior knowledge cursorily, they did not adjust their instruction based on the student's responses. The last rater also found the teacher‐led whole group discourse patterns limiting for the students.</p> <p> <emph>Assertion 2</emph>: <emph>Raters ultimately assigned the same quantitative values to the same items</emph>. Every rater agreed that the lesson was an example of more traditional, direct instruction, and they each assigned low scores on academic language development, opportunities for interactive student communication, and scientific inquiry items. They also agreed that while the visual aids were useful and led to slightly higher, non‐zero scores on the items corresponding to how the teacher used graphics, no rater interpreted the lesson to demonstrate medium or high levels of inquiry‐based instruction. They arrived at their codes from their interpretations of slightly different aspects of the teacher's instruction but reliably converged on the same codes. More information about these assertions is detailed in the Supplemental Materials, Part C.</p> <hd id="AN0192956349-26">Summary of substantive validity argument</hd> <p>The DiISC raters independently observed and attributed many of the same codes in the teacher's lesson. They deviated slightly in identifying problematic teaching behaviors and their ideas about alternative instructional materials and methods the teacher could have used. This was important in categorizing the teacher's use of ALD strategies and the subsequent opportunities for students. However, what is most important is that every rater interpreted the teacher's lesson as demonstrating a low level of inquiry. This conclusion corresponded with strong interrater agreement. At the beginning of the study, the inter‐rater reliability (IRR) was low, being 0.64 in the 2015–2016 school year. As a result, there was another round of calibration before independent observations commenced. However, as the study progressed, the reliabilities increased. In the 2017–2018 academic year, the initial IRR was more than 0.80.</p> <p>The research team observers followed a semestral procedure to calibrate and recalibrate DiISC data collection each year. At the beginning of each academic year, all raters participated in a group calibration in which individuals discussed their independent ratings for at least two and as many as four recorded lessons. After this stage, coders engaged in at least six paired observations, each member being paired twice with every other member, and consensus coding. IRR statistics were calculated and assessed as acceptable before team members worked independently. Because of the care that this team of observers took to calibrate and recalibrate, they could assign similar codes and had little difficulty reaching a consensus about the scores they assigned to describe classroom instruction. This supported a strong and compelling substantive validity argument.</p> <p>It is unreasonable to expect different observers to focus on the same teacher's behaviors when making their interpretations as they code lessons. They will, and did, have variations in their experiences as former science teachers, their current areas of interest, and their understanding of the teacher's lesson instruction. However, this did not significantly change their codes. As a result, it is important that future users of the DiISC instrument also be trained and calibrated, evaluated for interrater reliability, and recalibrated periodically (e.g., at the beginning of each semester).</p> <hd id="AN0192956349-27">Structural validity</hd> <p>We completed an exploratory factor analysis of 807 new secondary science lesson observations using two inquiry‐based observation instruments, the EQUIP and DiISC. As summarized below, the EQUIP scores loaded on two factors, and the DiISC loaded on three. We also provide more details on the factor loading in the Supplemental Materials, Part D.</p> <hd id="AN0192956349-28">EQUIP factors</hd> <p>The EQUIP developers presented a four‐factor solution. In our factor analysis, we identified a best‐fitting, two‐factor solution. We used principal axis factoring as our extraction method and our rotation method was Promax with Kaiser Normalization. Factor 1 was "Curriculum, Instruction, and Assessment" and Factor 2 was "Classroom Discourse." Inquiry‐based teaching was used as the basis of all ratings. Three of the four developer‐determined scales on the EQUIP were more strongly correlated with Factor 1 than Factor 2 in our analysis of science lessons.</p> <p> <emph>Factor 1</emph>: <emph>Curriculum, Instruction, and Assessment</emph>. Items associated with curriculum, instruction, and assessment compose Factor 1; each type is discussed and connected to foundational aspects of inquiry‐based instruction and the practical use of the instrument.</p> <p> <emph>Curriculum</emph>. EQUIP curricular scale items were associated with the factor structure provided by the instrument providers. Three of four items loaded more strongly on Factor 1 than on Factor 2 (with correlations ranging from 0.469 to 0.809)—these three items focused on types of student activities and the degree of active learning that was provided to them, that is, if the lesson: (a) integrated the activities and/or content (C3), (b) the degree of active learning exhibited by the students (C2), which can also be thought as the difference between a "cookbook" verification activity and a student‐driven investigation, and (c) students' opportunities to organize and record information (C4). The fourth item focused on content depth of knowledge (C1) loaded equally on Factors 1 and 2 (0.447 and 0.455).</p> <p> <emph>Instruction</emph>. Four of five instructional items on the EQUIP scale (on the developer‐provided factor structure) loaded higher on Factor 1 than Factor 2, with correlations between 0.690 and 0.860. These items attended to: (a) the teacher's role as a facilitator of learning (I3), (b) the student's role as a learner (I4), (c) the student's opportunity to learn through investigations (I1), and (d) depth of knowledge required in the science lesson (I5). The fifth item (I2) loaded equally well on Factors 1 and 2 (0.535 and 0.568, respectively). This item attended to the order of instruction, which is associated with the 5E inquiry‐based instructional model. The role of scientific explanations in the 5E model is also connected to Factor 2 (Classroom Discourse).</p> <p> <emph>Assessment</emph>. Two of the five EQUIP instructional factor items loaded higher on Factor 1 with correlations between 0.664 and 0.686, respectively. These items attended to the development of conceptual knowledge and critical thinking on the part of the students (A2) and what kinds of knowledge (factual and/or conceptual) were assessed by formal and informal assessments (A4). The other assessment scale items (A1 and A3) loaded equally poorly on Factors 1 and 2 (with correlations ranging from 0.191 to 0.380). Aspects of these items and correlations with Factor 2 (Classroom Discourse), described below, indicated that metacognitive activities required the opportunity to learn and use discursive skills.</p> <p> <emph>Factor 2: Classroom Discourse</emph>. All five EQUIP discourse scale items loaded on Factor 2 with correlations between 0.715 and 0.816. Three of the five items focus on rich, interactive discussion and questioning, and the other two focus on the classroom communication pattern and how students are prompted to justify their reasoning. Four of the five items also loaded on Factor 1 (0.412 to 0.551), reflecting the sociocultural nature and constructivist priorities of learning science.</p> <p> <emph>Assessment</emph>. One EQUIP assessment item (A5) loaded more strongly (0.628) on Factor 2 and focused on the teachers' use of formative assessment and immediate adjustment of instruction. The correlation makes sense as this classroom‐level assessment relies heavily on oral and WD practices.</p> <hd id="AN0192956349-29">DiISC factors</hd> <p>We identified a best‐fitting, three‐factor solution in our factor analysis using 807 DiISC lesson observations (Appendix, Table A1). We then added the coded data from 229 observations of science lessons from the original CISIP project and still found a three‐factor solution. These three factors met the interpretability criterion. We labeled Factor 1 as "Inquiry and Use of Scientific Practices," Factor 2, "Written Discourse;" and Factor 3, "Learning Supported by Oral Discourse and Academic Language Development Strategies." We used principal axis factoring as our extraction method, and our rotation method was Promax with Kaiser Normalization. Items loading below 0.25 were omitted and ordered from largest to smallest factor loading (Appendix, Table A1).</p> <p> <emph>Factor 1: Inquiry and use of scientific practices</emph>. A total of nine items loaded above 0.25 on Factor 1. Inquiry was the main DiISC scale that contributed the most items, five, to Factor 1 in our analysis of measurements of teachers' science lessons compared to the other two factors. These items include an overall rating of the teacher's use of inquiry‐based science lessons (Item #1) and individual items based upon the 5E inquiry‐based model of instruction that also aligns with the NGSS SPs. In addition, four items: (a) #1, the inquiry environment; (b) #2, student‐developed questions for investigation (NGSS SP1); (c) #4, student data collection and analysis (e.g., graphing) (NGSS SP4 and SP5); and (d) #5, students make claims, predictions, and models (NGSS SP6 and SP7) correlated strongly with Factor 1 (0.590 to 0.904). One of the other two Inquiry scale items (#3, students design and plan investigations (NGSS SP3)) loaded on Factors 1 and 2, correlating at 0.426 with Factor 1.</p> <p>Four additional items from other scales loaded on Factor 1. One from the Oral Discourse (OD) scale (#8, the teacher promotes peer‐to‐peer discourse), which reflects social constructivist learning, the underlying theory of learning in science education, correlated more strongly with Factor 1 (0.414) but also loaded on Factor 3. One item from the Academic Language Development (ALD) scale (#24, the teacher provides instruction for student interaction), which mirrors and reinforces the OD item, also correlated with Factor 1 (0.436). Another ALD item (#20, teacher used visual aids and gestures) correlated about equally with Factors 1 and 3 (0.270 and 0.259). A final item from the LP scale (#28, placing facts within a conceptual framework (NRC, [<reflink idref="bib40" id="ref75">40</reflink>])) also correlated with Factors 1 and 3 (0.377 and 0.360). This item assesses instructional practices focusing on constructing scientific knowledge by learning science content via comprehensible ways of organizing facts, such as using a concept map.</p> <p> <emph>Factor 2: Written discourse</emph>. Ten DiISC items loaded above 0.25 on Factor 2. Five of the six items (#12–14, #16–17) from the DiISC Written Discourse (WD) scale strongly correlated with Factor 2, with correlations ranging from 0.343 to 0.596. These items attend to aspects of the scientific writing process ranging from direct instruction of how to write scientifically, engaging in pre‐writing strategies, using science notebooks as a learning tool to more formal writing (e.g., lab reports), and students' opportunity to engage in recursive writing using feedback.</p> <p>As mentioned in the section on Factor 1, one item from the Inquiry scale (#3, students designing and planning investigations) correlated with Factor 2. This correlation makes sense as students are likely to engage in writing to draft their investigatory procedures. Another inquiry‐related item, #6, generating scientific arguments, also loaded on Factor 2 (0.281). Students crafting arguments were often observed during small group work, sometimes when they used their science notebooks and/or wrote rough drafts on small whiteboards. Two items from the ALD scale (#22, teacher addresses multiple levels of academic language; #25, supplemental resources provided to support language development) also moderately correlated with Factor 2 at 0.390 and 0.347, respectively. With increasing numbers of multilingual learners in K‐12 schools, these are vital teaching strategies that all teachers should employ to ensure equitable access to learning science.</p> <p>Finally, three DiISC Learning Principles (LP) scale items were loaded on Factor 2. These were items #31 (teaching self‐monitoring for understanding, 0.274), #32 (opportunities to develop an awareness of learning strengths and challenges, 0.257), and #34 (community norms for discourse, 0.269). These social and reflective self‐monitoring strategies prompt students to engage in writing and sound habits of mind that contribute to their learning in science classes.</p> <p> <emph>Factor 3: Learning supported by oral discourse and academic language development strategies</emph>. The final factor, Factor 3, was comprised of 10 items that only loaded on Factor 3 and another five that also loaded on another factor. The 15 items across three DiISC scales address inclusive teaching strategies that involve discussing science, how teachers support ALD, and means of formative learning assessment. Four OD strategies focusing on questioning and small group discussion (#7, teacher promotes discourse through questioning, 0.413; #8, teacher promotes peer‐to‐peer discussion, 0.282) and scientific discourse and the nature of science (#9, teacher bridges everyday experiences and scientific discourse, 0.414; #11, teacher engages students in discussion that emphasizes the nature of science, 0.257) correlated with Factor 3.</p> <p>Five items from the ALD scale loaded on Factor 3, two of which also cross‐loaded on one of the other two factors. These included items #18 (providing students opportunities to acquire vocabulary, 0.475), #19 (teacher uses clear instruction by modeling expectations, 0.373), #20 (teacher uses visual aids and gestures to communicate with students, 0.259), #21 (building lesson on students' language or culture, 0.296), and #23 (teacher provides direct instruction for using academic learning strategies, 0.393) that also attend to multilingual learners' language‐based learning needs to be successful in their science lessons.</p> <p>Finally, six DiISC LP scale items correlated with Factor 3. Item #26 (accessing students' prior knowledge, 0.257), #28 (teacher and/or students situate factual knowledge within a conceptual framework, 0.360), #29 (teacher provides opportunities for students to review key concepts), and #30 (providing embedded metacognition opportunities, 0.477) and #31 (teaching self‐monitoring for understanding, 0.324). The final item, #36 (teacher uses feedback strategies with an academic focus, 0.318), is a crucial formative assessment activity. This cycle has been empirically shown to improve students' learning (Black & Wiliam, [<reflink idref="bib7" id="ref76">7</reflink>]). Three items (#26, 28, and 30) are directly from <emph>How People Learn</emph> (NRC, [<reflink idref="bib40" id="ref77">40</reflink>]) and are key LP that undergird human cognition. The other three (#29, 31, and 36) are excellent strategies for reinforcing all students' learning.</p> <hd id="AN0192956349-30">Generating theoretically meaningful DiISC scales</hd> <p>Five items were removed from the original DiISC to improve the measurement properties of the DiISC scales to operationalize constructs. Items that did not load on factors above 0.25 were omitted, and items that loaded negatively on factors in uninterpretable ways. As a result, the three‐factor DiISC instrument using all items was correlated with the EQUIP, and evidence of external validity was established. However, several items did not represent the constructs described in our DiISC three‐factor solution. Empirically appraising the underlying factor structure is necessary to establish a validity argument, although the original version of the DiISC was generated with items theoretically supported to measure latent constructs (Worthington & Whittaker, [<reflink idref="bib51" id="ref78">51</reflink>]). In this new effort, we aimed to develop a generalized version with parsimonious scales that measure constructs based on the least necessary information. The updated version also maintains evidence of external validity via factor correlations with the EQUIP. All factor analytic solutions were required to retain statistical relationships between our new version of the DiISC and the EQUIP.</p> <p>We examined the factor analytic solutions to compare how specific items and how they loaded on relative constructs. Five items did not sufficiently represent constructs on which they loaded or problematically cross‐loaded (e.g., positive on one factor and negative on another). We removed these five items (#10, 15, 27, 33, and 35) to determine if the DiISC factors remained statistically related to EQUIP scores. This iterative process ensured that our measure included the essential items to retain validity and complete the structural validity argument without disrupting other elements of the validity argument (i.e., external validity argument).</p> <p>We also followed judgmental criteria to support statistical criteria when removing items from the original version of the DiISC by qualitatively assessing the appropriateness of the content and writing of items. For example, one item, #15, that focuses on academic writing, can be challenging to observe in the classroom without also having access to student writing samples. We also determined that an item might have an issue if it was ambiguously worded or difficult for the classroom observer to score. An example is item #33, which asked observers to rate how well the teacher promoted executive control of learning in their students by offering them choices about what and how they learn. A single observation of a science lesson (i.e., one class period) is insufficient to accurately evaluate what is happening over a longer curricular period.</p> <p>The final version, the DiISC 2.0, retains 31 items from the original 36‐item instrument. The revised instrument was provided to the lead developer of the original DiISC to evaluate the theoretical constructs relevant to inquiry‐based science instruction. She supported the new version as a useful instrument for investigating changes in science teachers' instructional practices. While the items concerning those removed strategies are useful foci for teachers, the scope of the instrument still reflects the original vision and model of a scientific classroom discourse community. For example, item #35 on the original was removed, even though we agree that rubrics and student work exemplars are key to proficient educative assessment practices (Wiggins, [<reflink idref="bib49" id="ref79">49</reflink>]). Unfortunately, the item did not contribute to measuring the latent constructs appropriately in <emph>DiISC 2.0</emph>. As a result, the <emph>DiISC 2.0</emph> instrument only includes measures for teaching practices typically observed as a natural limitation of our structural validity argument. Finalizing it with items that measured constructs from clearly observable instructional practices in our dataset contributes to our validity argument for the science education community's wider use of the instrument.</p> <hd id="AN0192956349-31">External validity</hd> <p>As part of the validation process, we used numerous measures to investigate the correlations between the DiISC and EQUIP. In this validity argument, external validity is defined as the extent to which the measure of interest predicts another external measure of interest with an extant validity argument (Kane, [<reflink idref="bib21" id="ref80">21</reflink>]; Messick, [<reflink idref="bib37" id="ref81">37</reflink>]). Evidence for this external validity argument was generated by correlating the EQUIP and DiISC factor scores from the previously described factor analyses. In our case, the DiISC was the measure of interest and was correlated with an external measure, the EQUIP. Factor scores were constructed for the DiISC and EQUIP using exploratory factor analyses with Barlett regression to extract factor scores (see previous discussion of factor analysis). To establish external validity for the DiISC, its factor scores were correlated with EQUIP factor scores because the EQUIP already had a validity argument. In addition, external validity was established because factors on both instruments correlated with each other, as expected since both instruments measured similar teacher behaviors. Thus, we also established external validity for the DiISC (Kane, [<reflink idref="bib21" id="ref82">21</reflink>]). Specifically, we used a multivariate regression and found that all three factors associated with the DiISC were statistically significantly correlated with the EQUIP scores (Table 2).</p> <p>2 TABLE Multivariate test results.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Factor</th><th align="left">Pillai's trace</th><th align="left">df</th><th align="left">Sig.</th><th align="left">Partial <italic>η</italic><sup>2</sup></th></tr></thead><tbody valign="top"><tr><td align="left">DiISC 1</td><td align="char" char=".">0.17</td><td align="char" char=".">(2802)</td><td align="left">p < 0.01</td><td align="char" char=".">0.17</td></tr><tr><td align="left">DiISC 2</td><td align="char" char=".">0.02</td><td align="char" char=".">(2802)</td><td align="left">p < 0.01</td><td align="char" char=".">0.02</td></tr><tr><td align="left">DiISC 3</td><td align="char" char=".">0.09</td><td align="char" char=".">(2802)</td><td align="left">p < 0.01</td><td align="char" char=".">0.19</td></tr></tbody></table> </ephtml> </p> <p>Each DiISC factor accounted for unique variability in the EQUIP scores at 17%, 2%, and 19%, respectively. Combined, they accounted for 25.2% of the variability in EQUIP scores. In follow‐up analyses, DiISC Factor 1 was statistically correlated with both EQUIP factors (<emph>F</emph> = 159.63, df(1803), <emph>p</emph> < 0.01 and <emph>F</emph> = 49.48, df(1803), <emph>p</emph> < 0.01, respectively), as was DiISC Factor 2 (<emph>F</emph> = 10.69, df (1803), <emph>p</emph> < 0.01 and <emph>F</emph> = 8.46, df(1803), <emph>p</emph> < 0.01, respectively). Only DiISC Factor 3 had mixed results; statistically correlated with EQUIP Factor 2 (<emph>F</emph> = 56.05, df(1803), <emph>p</emph> < 0.01), but not Factor 1 (<emph>F</emph> = 0.71, df(1803), <emph>p</emph> = 0.07). All significant values accounted for 1%–17% of the unique variability in their respective EQUIP factor comparisons. Standardized residual plots supported these overall positive findings. In summary, there was adequate evidence for the external validity of the DiISC measure as it was statistically significantly related to the external measures.</p> <hd id="AN0192956349-32">Generalizability validity</hd> <p>In our validity argument, generalizability validity is defined as how well scores on the measure of interest generalize across populations, settings, or types of tasks (Kane, [<reflink idref="bib21" id="ref83">21</reflink>]; Messick, [<reflink idref="bib37" id="ref84">37</reflink>]). In this case, the measure of interest was the DiISC instrument, specifically its three factors. We speculated on three potential possibilities for DIF. However, the instrument developers indicated that because analyses were conducted over subgroups, DIF issues needed to be sufficiently concerning (Baker et al., [<reflink idref="bib5" id="ref85">5</reflink>]). While many studies and modern validity arguments feature additional demographic information, those data were not collected as part of this project. As such, there was adequate evidence, and a new covariate emerged.</p> <p>As part of a generalizability argument, none of the factors we considered unrelated to measurement should have been associated with scores on the DiISC, that is, the teacher's sex and length of the instructional period observed. Neither sex (<emph>Pillai's Trace</emph> = 0.003, df(3801), <emph>p</emph> = 0.55) nor class length during observation (<emph>Pillai's Trace</emph> = 0.004, df(3801), <emph>p</emph> = 0.38) were statistically related to DiISC scores. This indicated that the DiISC generalizes over time on task because, in our study, time on task was reconceptualized as class length. Similarly, when available, atheoretically related demographic predictions indicated no statistically significant effect. This was one step toward generalizing the range of teachers in this study.</p> <p>Other elements of investigating the generalizability aspect of our validity argument still need to be explored. For example, class sizes did not vary greatly across the sample, so we did not use it as a variable in this study. It is, therefore, a topic for further investigation. Overall, there was limited evidence for the generalizability of the DiISC which necessitates further study. For example, it can safely be used with secondary science teachers without male–female bias. Also, it should be considered that this validation process was part of a larger project that used a representative sample of in‐service teachers who were graduates of an undergraduate‐ or MA‐level science teacher preparation program versus alternatively or emergency‐certified teachers who did not benefit from a formal preservice science teacher education program and mentoring.</p> <hd id="AN0192956349-33">DiISC VERSION 2.0 VALIDITY ARGUMENT</hd> <p>Based upon the validity evidence discussed in the previous section, experience using the original instrument in a non‐PD context, and second‐factor analysis with the five items removed, we have streamlined the DiISC for general use. The new 31‐item version is in the Supplemental Materials, Part A and with an accompanying users' manual via the digital repository link in the references. The removal of these items is warranted through the low correlations and consideration of the instrument's intended use to measure core aspects of a scientific classroom discourse community. The first DiISC was developed as a companion instrument to attend to a broad range of specific facets of a single teacher PD program. The <emph>DiISC 2.0</emph> makes it more widely applicable in studying US secondary science teachers' instructional practices. In addition to researchers, teacher PD providers, including school district science curriculum coordinators, will find that the <emph>DiISC 2.0</emph> provides clear targets for shifting teachers' instructional practices to align with inclusive inquiry‐based teaching practices.</p> <p>The new instrument maintains 86% of the original items and strongly supports the conceptual framework of a scientific classroom discourse community and its measurement; a comparison table is provided in Appendix Table A2 to show the old and new numbering of items on the two versions. The <emph>DiISC 2.0</emph> is appropriate for measuring aspects of reform‐based secondary science instruction through classroom observations in the United States, where the study was conducted in public schools. The original version was appropriate for measuring teacher improvement within the CISIP teacher PD program. It was also developed in the context of, and to observe and validate with, United States secondary‐level science instruction.</p> <hd id="AN0192956349-34">Limitations</hd> <p>The <emph>DiISC 2.0</emph> is not intended for ranking teachers but rather providing information on specific aspects of reform‐based instruction that are currently practiced or need to be improved by providing support for teachers. However, while data generated using this instrument can provide researchers, PD providers, and teachers with robust information about the quality of upper grades inquiry‐based instruction and associated academic language and discourse opportunities provided to students in their science classes, it may not be a reliable instrument for: (a) early childhood and elementary grade level science instruction, (b) all international educational settings, or (c) non‐standards aligned science curriculum and/or programs. It is possible that the <emph>DiISC Version 2.0</emph> would be useful in these learning environments. However, other researchers would need to engage in further validation to support the instrument's use beyond those discussed in this particular validity argument. While the instrument does attend to some aspects of teachers' use of culturally responsive curriculum and inclusive pedagogies, we recommend that researchers use the <emph>DiISC 2.0</emph> in conjunction with other research methods to achieve a more holistic approach, including curricular artifact analysis and interviews, to explore equity‐related research questions.</p> <hd id="AN0192956349-35">CONCLUSION</hd> <p>Our primary conclusion is the argument that the DiISC adequately and usefully measures inquiry‐based instructional practices for particular groups of teachers. It is inherent in this validity argument, as it is in any validity argument, that additional contexts could be considered, in turn adapting the measurement instrument to be used more broadly. However, what is critical is the unified argument in the project's scope. Naturally, there will be inevitable limitations, qualitative considerations of context and content, and alternative learning standards and theories. Still, those are extensions of the existing measurement development processes rather than requirements for every iteration of the validity argument. This paper provides one such implementation, or iteration, of a validity argument, with the myriad limitations and issues of any validity argument. This reflects the larger challenge that psychometricians face as validity transitions from simple requirements to dynamic arguments that change and adapt over time. The DiISC 2.0 instrument provides a useful and reliable research tool across time and iterative science education standards revisions. Furthermore, it is strongly connected to the long‐term vision of scientific literacy for all students and the discourse‐rich, inquiry‐based teaching practices that foster their learning.</p> <hd id="AN0192956349-36">ACKNOWLEDGMENTS</hd> <p>This material is based upon work supported by the National Science Foundation under Grant No. DRL‐0353469 and the Arizona Board of Regents Grants No. ITQ07‐05 and No. wwITQ08‐02 as well as NSF Noyce Program Grant No. 1540797.</p> <hd id="AN0192956349-37">CONFLICT OF INTEREST STATEMENT</hd> <p>We have no conflicts of interest to disclose.</p> <hd id="AN0192956349-38">A APPENDIX</hd> <p>A1 TABLE DiISC factor structure with low correlation items removed.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Pattern matrix<xref ref-type="fn" rid="tfn2" /></th></tr><tr><th align="left" /><th align="left">Factor</th></tr><tr><th align="left">1</th><th align="left">2</th><th align="left">3</th></tr></thead><tbody valign="top"><tr><td align="left"><p>i4</p></td><td align="char" char="."><p>0.904</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>i1</p></td><td align="char" char="."><p>0.863</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>i2</p></td><td align="char" char="."><p>0.734</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>i5</p></td><td align="char" char="."><p>0.590</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>i24</p></td><td align="char" char="."><p>0.436</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>i3</p></td><td align="char" char="."><p>0.426</p></td><td align="char" char="."><p>0.353</p></td><td align="left" /></tr><tr><td align="left"><p>i8</p></td><td align="char" char="."><p>0.414</p></td><td align="left" /><td align="char" char="."><p>0.282</p></td></tr><tr><td align="left"><p>i28</p></td><td align="char" char="."><p>0.377</p></td><td align="left" /><td align="char" char="."><p>0.360</p></td></tr><tr><td align="left"><p>i20</p></td><td align="char" char="."><p>0.270</p></td><td align="left" /><td align="char" char="."><p>0.259</p></td></tr><tr><td align="left"><p>i12</p></td><td align="left" /><td align="char" char="."><p>0.596</p></td><td align="left" /></tr><tr><td align="left"><p>i13</p></td><td align="left" /><td align="char" char="."><p>0.536</p></td><td align="left" /></tr><tr><td align="left"><p>i14</p></td><td align="left" /><td align="char" char="."><p>0.510</p></td><td align="left" /></tr><tr><td align="left"><p>i16</p></td><td align="left" /><td align="char" char="."><p>0.429</p></td><td align="left" /></tr><tr><td align="left"><p>i22</p></td><td align="left" /><td align="char" char="."><p>0.390</p></td><td align="left" /></tr><tr><td align="left"><p>i25</p></td><td align="left" /><td align="char" char="."><p>0.347</p></td><td align="left" /></tr><tr><td align="left"><p>i17</p></td><td align="left" /><td align="char" char="."><p>0.343</p></td><td align="left" /></tr><tr><td align="left"><p>i6</p></td><td align="left" /><td align="char" char="."><p>0.281</p></td><td align="left" /></tr><tr><td align="left"><p>i34</p></td><td align="left" /><td align="char" char="."><p>0.269</p></td><td align="left" /></tr><tr><td align="left"><p>i32</p></td><td align="left" /><td align="char" char="."><p>0.257</p></td><td align="left" /></tr><tr><td align="left"><p>i30</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.477</p></td></tr><tr><td align="left"><p>i18</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.475</p></td></tr><tr><td align="left"><p>i9</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.414</p></td></tr><tr><td align="left"><p>i7</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.413</p></td></tr><tr><td align="left"><p>i23</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.393</p></td></tr><tr><td align="left"><p>i19</p></td><td align="char" char="."><p>0.286</p></td><td align="left" /><td align="char" char="."><p>0.373</p></td></tr><tr><td align="left"><p>i31</p></td><td align="left" /><td align="char" char="."><p>0.274</p></td><td align="char" char="."><p>0.324</p></td></tr><tr><td align="left"><p>i36</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.318</p></td></tr><tr><td align="left"><p>i29</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.307</p></td></tr><tr><td align="left"><p>i21</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.296</p></td></tr><tr><td align="left"><p>i26</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.257</p></td></tr><tr><td align="left"><p>i11</p></td><td align="left" /><td align="left" /><td align="char" char="."><p>0.257</p></td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: Extraction method: Principal axis factoring; Rotation method: Promax with Kaiser normalization.</p> <p>2 Rotation converged in seven iterations.</p> <p>A2 TABLE Cross‐referenced items from DiISC Versions 1 (Baker et al., 2008) and 2 (Lewis et al., 2022).</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Category</th><th align="left">DiISC version 1.0 item number</th><th align="left">DiISC version 2.0 item number</th></tr></thead><tbody valign="top"><tr><td align="left"><p><italic>Inquiry</italic></p></td><td align="char" char="."><p>1</p></td><td align="left"><p>I‐1</p></td></tr><tr><td align="char" char="."><p>2</p></td><td align="left"><p>I‐2</p></td></tr><tr><td align="char" char="."><p>3</p></td><td align="left"><p>I‐3</p></td></tr><tr><td align="char" char="."><p>4</p></td><td align="left"><p>I‐4</p></td></tr><tr><td align="char" char="."><p>5</p></td><td align="left"><p>I‐5</p></td></tr><tr><td align="char" char="."><p>6</p></td><td align="left"><p>I‐6</p></td></tr><tr><td align="left"><p><italic>Oral discourse</italic></p></td><td align="char" char="."><p>7</p></td><td align="left"><p>OD‐1</p></td></tr><tr><td align="char" char="."><p>8</p></td><td align="left"><p>OD‐2</p></td></tr><tr><td align="char" char="."><p>9</p></td><td align="left"><p>OD‐3</p></td></tr><tr><td align="char" char="."><p>10</p></td><td align="left"><p>‐</p></td></tr><tr><td align="char" char="."><p>11</p></td><td align="left"><p>OD‐4</p></td></tr><tr><td align="left"><p><italic>Written discourse</italic></p></td><td align="char" char="."><p>12</p></td><td align="left"><p>WD‐1</p></td></tr><tr><td align="char" char="."><p>13</p></td><td align="left"><p>WD‐2</p></td></tr><tr><td align="char" char="."><p>14</p></td><td align="left"><p>WD‐3</p></td></tr><tr><td align="char" char="."><p>15</p></td><td align="left"><p>‐</p></td></tr><tr><td align="char" char="."><p>16</p></td><td align="left"><p>WD‐4</p></td></tr><tr><td align="char" char="."><p>17</p></td><td align="left"><p>WD‐5</p></td></tr><tr><td align="left"><p><italic>Academic language development</italic></p></td><td align="char" char="."><p>18</p></td><td align="left"><p>ALD‐1</p></td></tr><tr><td align="char" char="."><p>19</p></td><td align="left"><p>ALD‐2</p></td></tr><tr><td align="char" char="."><p>20</p></td><td align="left"><p>ALD‐3</p></td></tr><tr><td align="char" char="."><p>21</p></td><td align="left"><p>ALD‐4</p></td></tr><tr><td align="char" char="."><p>22</p></td><td align="left"><p>ALD‐5</p></td></tr><tr><td align="char" char="."><p>23</p></td><td align="left"><p>ALD‐6</p></td></tr><tr><td align="char" char="."><p>24</p></td><td align="left"><p>ALD‐7</p></td></tr><tr><td align="char" char="."><p>25</p></td><td align="left"><p>ALD‐8</p></td></tr><tr><td align="left"><p><italic>Learning principles</italic></p></td><td align="char" char="."><p>26</p></td><td align="left"><p>LP‐1</p></td></tr><tr><td align="char" char="."><p>27</p></td><td align="left"><p>‐</p></td></tr><tr><td align="char" char="."><p>28</p></td><td align="left"><p>LP‐2</p></td></tr><tr><td align="char" char="."><p>29</p></td><td align="left"><p>LP‐3</p></td></tr><tr><td align="char" char="."><p>30</p></td><td align="left"><p>LP‐4</p></td></tr><tr><td align="char" char="."><p>31</p></td><td align="left"><p>LP‐5</p></td></tr><tr><td align="char" char="."><p>32</p></td><td align="left"><p>LP‐6</p></td></tr><tr><td align="char" char="."><p>33</p></td><td align="left"><p>‐</p></td></tr><tr><td align="char" char="."><p>34</p></td><td align="left"><p>LP‐7</p></td></tr><tr><td align="char" char="."><p>35</p></td><td align="left"><p>‐</p></td></tr><tr><td align="char" char="."><p>36</p></td><td align="left"><p>LP‐8</p></td></tr></tbody></table> </ephtml> </p> <p>GRAPH: Data S1: Supporting Information.</p> <ref id="AN0192956349-39"> <title> Footnotes </title> <blist> <bibl id="bib1" idref="ref42" type="bt">1</bibl> <bibtext> Additionally, schools were not provided more resources to support curricular reform, prolonging science education reform by school districts in school‐level curriculum alignment.</bibtext> </blist> <blist> <bibl id="bib2" idref="ref9" type="bt">2</bibl> <bibtext> Researchers focusing on the Common Core State Standards (CCSS) should not use the DiISC because the CCSS does not include science education standards.</bibtext> </blist> </ref> <ref id="AN0192956349-40"> <title> REFERENCES </title> <blist> <bibtext> American Association for the Advancement of Science (1989). Science for all Americans: A Project 2061 report on literacy goals in science mathematics and technology. https://<ulink href="http://www.project2061.org/publications/sfaa/online/Chap13.htm">www.project2061.org/publications/sfaa/online/Chap13.htm</ulink></bibtext> </blist> <blist> <bibtext> American Association for the Advancement of Science (1993). Benchmarks for Science Literacy. Oxford University Press.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref1" type="bt">3</bibl> <bibtext> American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. American Educational Research Association.</bibtext> </blist> <blist> <bibl id="bib4" idref="ref5" type="bt">4</bibl> <bibtext> Baker, D. R., Lewis, E. B., Purzer, S., Watts, N. B., Uysal, S., Wong, S., Beard, R., & Lang, M. (2009). The Communication in Science Inquiry Project (CISIP): A project to enhance scientific literacy through the creation of science classroom discourse communities. International Journal of Environmental & Science Education, 4(3), 259–274.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref4" type="bt">5</bibl> <bibtext> Baker, D., Beard, R., Bueno‐Watts, N., Lewis, E., Özdemir, G., Perkins, G., Uysal, S., Wong‐Kavas, S., & Yaşar‐Purzer, S. (2008). Discourse in Inquiry Science Classrooms (DiISC): Reference Manual, Technical Report No. 001. The Communication in Science Inquiry Project (CISIP): Arizona State University. https://digitalcommons.unl.edu/teachlearnfacpub/121/</bibtext> </blist> <blist> <bibl id="bib6" idref="ref6" type="bt">6</bibl> <bibtext> Baker, D., Lewis, E., Uysal, S., Purzer, S., Lang, M., & Baker, P. (2011). Using the Communication in Science Inquiry Project professional development model to facilitate learning middle school genetics concepts. Professional Development in Education, 37(3), 453–468.</bibtext> </blist> <blist> <bibl id="bib7" idref="ref62" type="bt">7</bibl> <bibtext> Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5–31.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref11" type="bt">8</bibl> <bibtext> BSCS Science LearningWebsite. (n.d.) Accessed: 10/22/20 https://bscs.org/about/our-story/</bibtext> </blist> <blist> <bibl id="bib9" idref="ref12" type="bt">9</bibl> <bibtext> Bybee, R., Taylor, J., Gardner, A., Van Scotter, P., Powell, J. C., Westbrook, A., & Landes, N. (2006). The BSCS 5E instructional model: Origins and effectiveness. BSCS, 5, 88–98.</bibtext> </blist> <blist> <bibtext> Carrasquillo, A., & Rodriquez, V. (1996). Language minority students in the mainstream classroom. Multilingual Matters Ltd.</bibtext> </blist> <blist> <bibtext> Chamot, A., & O'Malley, J. (1987). The cognitive academic language learning approach: A bridge to the mainstream. TESOL Quarterly, 21, 227–249.</bibtext> </blist> <blist> <bibtext> Chen, Y. C., & Terada, T. (2021). Development and validation of an observation‐based protocol to measure the eight scientific practices of the next generation science standards in K‐12 science classrooms. Journal of Research in Science Teaching, 58(10), 1489–1526.</bibtext> </blist> <blist> <bibtext> Crawford, B. A. (2014). From inquiry to scientific practices in the science classroom. In S. Abell & N. Lederman (Eds.), Handbook of research on science education (Volume 2, pp. 529–556). Routledge.</bibtext> </blist> <blist> <bibtext> Dewey, J. (1938). Experience and education. Simon & Schuster.</bibtext> </blist> <blist> <bibtext> Dori, Y. J., Zemira, M. R., & Baker, D. R. (2018). Discussion. In Y. J. Dori, Z. R. Mevarech, & D. R. Baker (Eds.), Cognition, metacognition, and culture in STEM education. Learning, teaching and assessment (pp. 331–352). Springer.</bibtext> </blist> <blist> <bibtext> Ericcson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. MIT Press.</bibtext> </blist> <blist> <bibtext> Gee, J. P. (2007). Social linguistics and literacies: Ideology in discourses. Routledge.</bibtext> </blist> <blist> <bibtext> Gee, L. P. (2014). An introduction to discourse analysis: Theory and method. Routledge.</bibtext> </blist> <blist> <bibtext> Halliday, M., & Martin, J. (1993). Writing science: Literacy and discursive power. University of Pittsburgh Press.</bibtext> </blist> <blist> <bibtext> Herrell, A. L., & Jordan, M. (2016). 50 strategies for teaching English language learners. Pearson.</bibtext> </blist> <blist> <bibtext> Kane, M. (2013). Validation as a pragmatic, scientific activity. Journal of Educational Measurement, 50(1), 115–122.</bibtext> </blist> <blist> <bibtext> Karplus, R. (2002). A love of discovery: Science education‐the second career of Robert Karplus. Springer Science & Business Media.</bibtext> </blist> <blist> <bibtext> Karplus, R., & Thier, H. D. (1967). A new look at elementary school science: Science curriculum improvement study. Rand McNally.</bibtext> </blist> <blist> <bibtext> Kelly, G. J., Brown, B., & Jiménez‐Aleixandre, M. P. (2023). Discourse practices in science learning. In N. G. Lederman, D. L. Zeidler, & J. S. Lederman (Eds.), Handbook of research on science education, Volume 3 (pp. 413–446). Routledge.</bibtext> </blist> <blist> <bibtext> Lave, J., & Wegner, E. (1992). Situated learning: Legitimate peripheral participation. Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Lawrenz, F., & Goodyear, L. (2023). Review of science education program evaluation. In Handbook of research on science education, Volume 3 (pp. 993–1010). Routledge.</bibtext> </blist> <blist> <bibtext> Lawrenz, F., & Thao, M. (2014). Review of science education program evaluation. In Handbook of research on science education, Volume 2 (pp. 773–788). Routledge.</bibtext> </blist> <blist> <bibtext> Lee, O. (2017). Common core state standards for ELA/literacy and Next Generation Science Standards: Convergences and discrepancies using argument. Educational Researcher, 46(2), 90–102.</bibtext> </blist> <blist> <bibtext> Lee, O., Llosa, L., Grapin, S., Haas, A., & Goggins, M. (2019). Science and language integration with English learners: A conceptual framework guiding instructional materials development. Science Education, 103(2), 317–337.</bibtext> </blist> <blist> <bibtext> Lemke, J. (1990). Talking science. Ablex.</bibtext> </blist> <blist> <bibtext> Lewis, E. B., Baker, D., Bueno Watts, N., & van der Hoeven Kraft, K. (2016). Science teachers' professional growth and the Communication in Science Inquiry Project. In Professional development: Recent advances and future directions (pp. 1–57). Nova Science Publishers, Inc.</bibtext> </blist> <blist> <bibtext> Lewis, E. B., Baker, D., Lucas, L., Tankersley, A., Hasseler, E., Rivero, A., & Helding, B. (2022). Discourse in Inquiry Science Classrooms, Version 2.0 (User's Manual). University of Nebraska‐Lincoln & Arizona State University. https://digitalcommons.unl.edu/teachlearnfacpub/475/</bibtext> </blist> <blist> <bibtext> Lewis, E. B., Rivero, A., Musson, A., Lucas, L., Tankersley, A., & Helding, B. A. (2020). Educating effective science teachers: Preparing and following teachers into the field. In J. Carinci, S. Meyer, & C. Jackson (Eds.), Linking teacher preparation program design and implementation to outcomes for teachers and students (pp. 87–129). Information Age Publishing.</bibtext> </blist> <blist> <bibtext> Mackenzie, J. C. (1894). The report of the Committee of Ten. The School Review, 2(3), 146–155.</bibtext> </blist> <blist> <bibtext> Marshall, J. C., Smart, J., & Horton, R. M. (2010). The design and validation of EQUIP: An instrument to assess inquiry‐ based instruction. International Journal of Science and Mathematics Education, 8, 299–321.</bibtext> </blist> <blist> <bibtext> Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education and MacMillian Publishing Company.</bibtext> </blist> <blist> <bibtext> Messick, S. (1995). Standards of validity and the validity of standards in performance assessments. Educational Measurement: Issues and Practice, 14(4), 4–5.</bibtext> </blist> <blist> <bibtext> National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. The National Academies Press.</bibtext> </blist> <blist> <bibtext> National Research Council. (1996). National Science Education Standards. The National Academy Press.</bibtext> </blist> <blist> <bibtext> National Research Council. (2000). How people learn: Brain, mind, experience, and school. National Academy Press.</bibtext> </blist> <blist> <bibtext> National Research Council. (2012). A framework for K‐12 science education: Practices, cross cutting concepts, and core ideas. The National Academies Press.</bibtext> </blist> <blist> <bibtext> National Research Council. (2013). Next Generation Science Standards: For states, by states. The National Academies Press. https://doi.org/10.17226/18290</bibtext> </blist> <blist> <bibtext> Osborne, J. (2023). Scientific practices and inquiry in the science classroom. In N. Lederman, D. Zeidler, & J. Lederman (Eds.), Handbook of research on science education, volume III (pp. 782–785). Routledge.</bibtext> </blist> <blist> <bibtext> Rivard, L., & Straw, S. (2000). The effect of talking and writing on science learning: An exploratory study. Science Education, 84, 566–593.</bibtext> </blist> <blist> <bibtext> Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253.</bibtext> </blist> <blist> <bibtext> Vygotsky, L. (1978). Mind in society: The development of higher psychological process. Harvard University Press.</bibtext> </blist> <blist> <bibtext> Wegner, E. (1998). Communities of practice. Cambridge University Press.</bibtext> </blist> <blist> <bibtext> Wheeler, L. B., Navy, S. L., Maeng, J. L., & Whitworth, B. A. (2019). Development and validation of the classroom observation protocol for engineering design (COPED). Journal of Research in Science Teaching, 56(9), 1285–1305.</bibtext> </blist> <blist> <bibtext> Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. Jossey‐Bass Publishers.</bibtext> </blist> <blist> <bibtext> Windschitl, M., Thompson, J., & Braaten, M. (2020). Ambitious science teaching. Harvard Education Press.</bibtext> </blist> <blist> <bibtext> Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838.</bibtext> </blist> </ref> <aug> <p>By Elizabeth Lewis; Lyrica Lucas; Brandon Helding; Amy Tankersley; Elizabeth Hasseler; Ana Rivero and Dale Baker</p> <p>Reported by Author; Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib33" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib21" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib31" firstref="ref7"></nolink> <nolink nlid="nl4" bibid="bib34" firstref="ref10"></nolink> <nolink nlid="nl5" bibid="bib23" firstref="ref13"></nolink> <nolink nlid="nl6" bibid="bib46" firstref="ref14"></nolink> <nolink nlid="nl7" bibid="bib22" firstref="ref15"></nolink> <nolink nlid="nl8" bibid="bib39" firstref="ref17"></nolink> <nolink nlid="nl9" bibid="bib13" firstref="ref18"></nolink> <nolink nlid="nl10" bibid="bib42" firstref="ref19"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref20"></nolink> <nolink nlid="nl12" bibid="bib48" firstref="ref21"></nolink> <nolink nlid="nl13" bibid="bib26" firstref="ref22"></nolink> <nolink nlid="nl14" bibid="bib27" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib45" firstref="ref24"></nolink> <nolink nlid="nl16" bibid="bib25" firstref="ref26"></nolink> <nolink nlid="nl17" bibid="bib47" firstref="ref27"></nolink> <nolink nlid="nl18" bibid="bib24" firstref="ref28"></nolink> <nolink nlid="nl19" bibid="bib44" firstref="ref30"></nolink> <nolink nlid="nl20" bibid="bib19" firstref="ref32"></nolink> <nolink nlid="nl21" bibid="bib43" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib40" firstref="ref34"></nolink> <nolink nlid="nl23" bibid="bib17" firstref="ref35"></nolink> <nolink nlid="nl24" bibid="bib18" firstref="ref37"></nolink> <nolink nlid="nl25" bibid="bib14" firstref="ref41"></nolink> <nolink nlid="nl26" bibid="bib41" firstref="ref44"></nolink> <nolink nlid="nl27" bibid="bib50" firstref="ref45"></nolink> <nolink nlid="nl28" bibid="bib30" firstref="ref49"></nolink> <nolink nlid="nl29" bibid="bib29" firstref="ref52"></nolink> <nolink nlid="nl30" bibid="bib20" firstref="ref55"></nolink> <nolink nlid="nl31" bibid="bib15" firstref="ref58"></nolink> <nolink nlid="nl32" bibid="bib38" firstref="ref60"></nolink> <nolink nlid="nl33" bibid="bib35" firstref="ref63"></nolink> <nolink nlid="nl34" bibid="bib28" firstref="ref64"></nolink> <nolink nlid="nl35" bibid="bib36" firstref="ref65"></nolink> <nolink nlid="nl36" bibid="bib37" firstref="ref66"></nolink> <nolink nlid="nl37" bibid="bib16" firstref="ref69"></nolink> <nolink nlid="nl38" bibid="bib51" firstref="ref78"></nolink> <nolink nlid="nl39" bibid="bib49" firstref="ref79"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1502629
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Measuring Scientific Classroom Discourse: The 'DiISC Version 2.0's' Validity and Use in Observing Secondary Science Lessons
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Elizabeth+Lewis%22">Elizabeth Lewis</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-3429-3003">0000-0002-3429-3003</externalLink>)<br /><searchLink fieldCode="AR" term="%22Lyrica+Lucas%22">Lyrica Lucas</searchLink><br /><searchLink fieldCode="AR" term="%22Brandon+Helding%22">Brandon Helding</searchLink><br /><searchLink fieldCode="AR" term="%22Amy+Tankersley%22">Amy Tankersley</searchLink><br /><searchLink fieldCode="AR" term="%22Elizabeth+Hasseler%22">Elizabeth Hasseler</searchLink><br /><searchLink fieldCode="AR" term="%22Ana+Rivero%22">Ana Rivero</searchLink><br /><searchLink fieldCode="AR" term="%22Dale+Baker%22">Dale Baker</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22School+Science+and+Mathematics%22"><i>School Science and Mathematics</i></searchLink>. 2026 126(2):143-158.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 1540797
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Evaluative
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Science+Education%22">Science Education</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Instruction%22">Science Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Science%22">Secondary School Science</searchLink><br /><searchLink fieldCode="DE" term="%22Discussion+%28Teaching+Technique%29%22">Discussion (Teaching Technique)</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Teachers%22">Science Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Inquiry%22">Inquiry</searchLink><br /><searchLink fieldCode="DE" term="%22Discourse+Analysis%22">Discourse Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Measures+%28Individuals%29%22">Measures (Individuals)</searchLink><br /><searchLink fieldCode="DE" term="%22Discourse+Communities%22">Discourse Communities</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Oral+Language%22">Oral Language</searchLink><br /><searchLink fieldCode="DE" term="%22Written+Language%22">Written Language</searchLink><br /><searchLink fieldCode="DE" term="%22Factor+Structure%22">Factor Structure</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Language%22">Academic Language</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/ssm.18325
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0036-6803<br />1949-8594
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: To continue to support long-term, ongoing science curriculum and instruction reform efforts in the United States, there is a significant need to be able to reliably measure teachers' discourse-rich, inquiry-based instruction. In this external validation study, we present the "Discourse in Inquiry Science Classrooms Version 2.0" (DiISC 2.0) as a valuable observational instrument for researchers to investigate multiple aspects of science teachers' lessons. The DiISC 2.0 is grounded in a research-based conceptual framework of a scientific classroom discourse community, focused on fundamental socioconstructivist characteristics of lessons, including scientific inquiry, oral and written discourse, and academic language development. We collected and analyzed new data from 807 science lessons to develop and expand the instrument's original validity argument beyond the associated professional development program for broader use. We determined the DiISC's factor structure and examined its correlation with the "Electronic Quality of Inquiry Protocol." Finally, some items that did not represent DiISC constructs were removed. Thus, the "DiISC 2.0" instrument can be used to measure students' opportunities to learn science and provide feedback to teachers on their progress toward building an inclusive scientific classroom discourse community for all students, in particular, historically marginalized groups and multilingual learners.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1502629
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1502629
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/ssm.18325
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
        StartPage: 143
    Subjects:
      – SubjectFull: Science Education
        Type: general
      – SubjectFull: Science Instruction
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Secondary School Science
        Type: general
      – SubjectFull: Discussion (Teaching Technique)
        Type: general
      – SubjectFull: Science Teachers
        Type: general
      – SubjectFull: Inquiry
        Type: general
      – SubjectFull: Discourse Analysis
        Type: general
      – SubjectFull: Measures (Individuals)
        Type: general
      – SubjectFull: Discourse Communities
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Oral Language
        Type: general
      – SubjectFull: Written Language
        Type: general
      – SubjectFull: Factor Structure
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Academic Language
        Type: general
    Titles:
      – TitleFull: Measuring Scientific Classroom Discourse: The 'DiISC Version 2.0's' Validity and Use in Observing Secondary Science Lessons
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Elizabeth Lewis
      – PersonEntity:
          Name:
            NameFull: Lyrica Lucas
      – PersonEntity:
          Name:
            NameFull: Brandon Helding
      – PersonEntity:
          Name:
            NameFull: Amy Tankersley
      – PersonEntity:
          Name:
            NameFull: Elizabeth Hasseler
      – PersonEntity:
          Name:
            NameFull: Ana Rivero
      – PersonEntity:
          Name:
            NameFull: Dale Baker
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0036-6803
            – Type: issn-electronic
              Value: 1949-8594
          Numbering:
            – Type: volume
              Value: 126
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: School Science and Mathematics
              Type: main
ResultId 1