View in EDS HTML Full Text PDF Full Text

Creating an Instrument to Measure Student Response to Instructional Practices

Saved in:
Bibliographic Details
Title: Creating an Instrument to Measure Student Response to Instructional Practices
Language: English
Authors: DeMonbrun, Matt, Finelli, Cynthia J., Prince, Michael, Borrego, Maura, Shekhar, Prateek, Henderson, Charles, Waters, Cindy
Source: Journal of Engineering Education. Apr 2017 106(2):273-298.
Availability: Wiley Periodicals, Inc. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Peer Reviewed: Y
Page Count: 26
Publication Date: 2017
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: 1347417
1347482
1347580
1347718
1500309
Document Type: Journal Articles
Reports - Research
Tests/Questionnaires
Descriptors: Test Construction, Attitude Measures, Student Attitudes, Student Reaction, Teaching Methods, STEM Education, Factor Analysis, Test Validity
DOI: 10.1002/jee.20162
ISSN: 1069-4730
Abstract: Background: Calls for the reform of education in science, technology, engineering, and mathematics (STEM) have inspired many instructional innovations, some research based. Yet adoption of such instruction has been slow. Research has suggested that students' response may significantly affect an instructor's willingness to adopt different types of instruction. Purpose: We created the Student Response to Instructional Practices (StRIP) instrument to measure the effects of several variables on student response to instructional practices. We discuss the step-by-step process for creating this instrument. Design/Method: The development process had six steps--item generation and construct development, validity testing, implementation, exploratory factor analysis, confirmatory factor analysis, and instrument modification and replication. We discuss pilot testing of the initial instrument, construct development, and validation using exploratory and confirmatory factor analyses. Results: This process produced 47 items measuring three parts of our framework. Types of instruction separated into four factors (interactive, constructive, active, and passive); strategies for using in-class activities into two factors (explanation and facilitation); and student responses to instruction into five factors (value, positivity, participation, distraction, and evaluation). Conclusions: We describe the design process and final results for our instrument, a useful tool for understanding the relationship between type of instruction and students' response.
Abstractor: As Provided
Entry Date: 2020
Accession Number: EJ1254473
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFK_jRFOe3PDjccAOiKZCyMAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDMbUIb1MyvL4q4uJ-AIBEICBmnPfiQ2lV32jS8s5TUMwVgYa8BCg3b5L0c6hXj8Rf5AEF1BReqV-L6xftzwgTjtsTI8e2f7M5Kyca5_oGbvpE-iUXJrVjE86B3zLx3NUREwXJiDmZ239H8aiQSrMphHVz4VuH0U9ZK80eHIaS_gC7h46Gu2pBaCtXo9koD436NPcbu3_KrihG716UIjybdvrpQ5fPeKUobQOpWc=
Text:
  Availability: 1
  Value: <anid>AN0122635335;6m401apr.17;2018Jun29.11:57;v2.2.500</anid> <title id="AN0122635335-1">Creating an Instrument to Measure Student Response to Instructional Practices. </title> <p>Background: Calls for the reform of education in science, technology, engineering, and mathematics (STEM) have inspired many instructional innovations, some research based. Yet adoption of such instruction has been slow. Research has suggested that students' response may significantly affect an instructor's willingness to adopt different types of instruction. Purpose: We created the Student Response to Instructional Practices (StRIP) instrument to measure the effects of several variables on student response to instructional practices. We discuss the step‐by‐step process for creating this instrument. Design/Method: The development process had six steps: item generation and construct development, validity testing, implementation, exploratory factor analysis, confirmatory factor analysis, and instrument modification and replication. We discuss pilot testing of the initial instrument, construct development, and validation using exploratory and confirmatory factor analyses. Results: This process produced 47 items measuring three parts of our framework. Types of instruction separated into four factors (interactive, constructive, active, and passive); strategies for using in‐class activities into two factors (explanation and facilitation); and student responses to instruction into five factors (value, positivity, participation, distraction, and evaluation). Conclusions: We describe the design process and final results for our instrument, a useful tool for understanding the relationship between type of instruction and students' response.</p> <p>active learning; instructional methods; factor analysis; student resistance</p> <p>There have been various calls for the reform of education in science, technology, engineering, and mathematics (STEM), including increasing the number and diversity of students receiving these degrees (AAAS, [<reflink idref="bib1" id="ref1">1</reflink>] ; NAS, NAE, & IOM, [<reflink idref="bib61" id="ref2">61</reflink>] ). These calls for reform have drawn forth many innovations in the types of instruction used in the classroom, several of which are research based (Jamieson & Lohmann, [<reflink idref="bib46" id="ref3">46</reflink>] ; Kuh, [<reflink idref="bib51" id="ref4">51</reflink>] ; Seymour & Hewitt, [<reflink idref="bib79" id="ref5">79</reflink>] ). Many of these research‐based types of instruction fall under the broad definition of active learning, or requiring students to participate in class activities beyond watching an instructor lecture (Felder & Brent, [<reflink idref="bib25" id="ref6">25</reflink>] ); prior research has shown active learning can be especially effective for educating a diverse student body (Prince, [<reflink idref="bib72" id="ref7">72</reflink>] ; Seymour & Hewitt, [<reflink idref="bib79" id="ref8">79</reflink>] ) and for increasing the retention rate of students in STEM programs (Angelo & Cross, [<reflink idref="bib3" id="ref9">3</reflink>] ; Prince & Felder, [<reflink idref="bib74" id="ref10">74</reflink>] ).</p> <p>Despite this literature base, translation of research about innovative types of instruction to instructional practice has been slow (Friedrich, Sellers, & Burstyn, [<reflink idref="bib31" id="ref11">31</reflink>] ; Handelsman et al., [<reflink idref="bib38" id="ref12">38</reflink>] ; Hora, Ferrare, & Oleson, [<reflink idref="bib42" id="ref13">42</reflink>] ; PCAST, [<reflink idref="bib71" id="ref14">71</reflink>] ; Singer, Nielsen, & Schweingruber, [<reflink idref="bib81" id="ref15">81</reflink>] ). Several researchers have identified a number of instructor‐reported barriers that help to explain these slow adoption rates. Among the least researched but most often mentioned barriers is the concern that students will resist, or respond in negative ways (Borrego, Froyd, & Hall, [<reflink idref="bib10" id="ref16">10</reflink>] ; Dancy & Henderson, [<reflink idref="bib21" id="ref17">21</reflink>] ; Finelli, Daly, & Richardson, [<reflink idref="bib26" id="ref18">26</reflink>] ; Froyd, Borrego, Cutler, Prince, & Henderson, [<reflink idref="bib32" id="ref19">32</reflink>] ; Henderson & Dancy, [<reflink idref="bib40" id="ref20">40</reflink>] ; Seidel & Tanner, [<reflink idref="bib78" id="ref21">78</reflink>] ). In actuality, student response to new and different types of instruction can be positive if students are engaged in these activities, view them in a positive light, and see the value in their use (Gauci, Dantas, Williams, & Kemm, [<reflink idref="bib35" id="ref22">35</reflink>] ; Livingstone & Lynch, [<reflink idref="bib56" id="ref23">56</reflink>] ). However, worries about such negative responses can discourage instructors from adopting new and different types of instruction.</p> <p>Research that characterizes the types of student response (both positive and negative) to various types of instruction and identifies strategies for using these types of instruction could help eliminate a key barrier to faculty adopting new instructional practices. And although literature offers a variety of tips for instructors wishing to promote positive response and minimize negative reactions to different types of instruction (e.g. Armstrong, [<reflink idref="bib4" id="ref24">4</reflink>] ; Arum & Roksa, [<reflink idref="bib5" id="ref25">5</reflink>] ; Felder, [<reflink idref="bib24" id="ref26">24</reflink>] ; Johnson, Johnson, & Smith, [<reflink idref="bib47" id="ref27">47</reflink>] ; Lake, [<reflink idref="bib52" id="ref28">52</reflink>] ; Michael, [<reflink idref="bib59" id="ref29">59</reflink>] ; Moffett & Hill, [<reflink idref="bib60" id="ref30">60</reflink>] ; Prince, Borrego, Henderson, Cutler, & Froyd, [<reflink idref="bib73" id="ref31">73</reflink>] ), these suggestions tend to be drawn from personal experience and have yet to be empirically tested. These limitations show the need for additional research in this area. Such research requires an instrument to assess and measure students' responses to different types of instruction and the strategies used (or not used) with each instructional type. Here, we report on development of the Student Response to Instructional Practices (StRIP) instrument to achieve this goal.</p> <p>Rather than focusing on the effects of instructional types, this article describes the development process of the StRIP instrument, which followed accepted approaches for instrument development (e.g., Carberry, Lee, & Ohland, [<reflink idref="bib13" id="ref32">13</reflink>] ; Li, McCoach, Swaminathan, & Tang, [<reflink idref="bib55" id="ref33">55</reflink>] ; Ro, Merson, Lattuca, & Terenzini, [<reflink idref="bib76" id="ref34">76</reflink>] ). The resulting StRIP instrument can be used by researchers and practitioners seeking a tool to study student response to all types of instruction in the classroom, and the framework we have developed attempts to explain the relationship between types of instruction, strategies for using these types of instruction during class, and how students respond.</p> <hd id="AN0122635335-2">Methods</hd> <p>We adapted the development process for the Student Response to Instructional Practices (StRIP) instrument from Hinkin ([<reflink idref="bib41" id="ref35">41</reflink>] ), as shown in Figure [NaN] . The process is iterative, which involves a six‐step approach:</p> <p>(<reflink idref="bib1" id="ref36">1</reflink>)    generating items and developing constructs for the instrument; the process borrows from prior literature on instructional types, student response, and strategies for using in‐class activities,</p> <p>(<reflink idref="bib2" id="ref37">2</reflink>)    testing for validity by observing the engineering classroom, as well as by subjecting the instrument to expert review and cognitive interviewing,</p> <p>(<reflink idref="bib3" id="ref38">3</reflink>)    implementing the instrument,</p> <p>(<reflink idref="bib4" id="ref39">4</reflink>)    conducting an exploratory factor analysis, an important step since there was no previous instrument on student response to instructional practices,</p> <p>(<reflink idref="bib5" id="ref40">5</reflink>)    conducting a confirmatory factor analysis to verify the constructs established in the exploratory factor analysis, and</p> <p>(<reflink idref="bib6" id="ref41">6</reflink>)    modifying the instrument and replicating findings through full instrument administration.</p> <hd id="AN0122635335-3">Step 1: Item Generation and Construct Development</hd> <p>In creating a new instrument, researchers must first generate the items needed to measure the desired construct(s), a process that can be accomplished through deductive or inductive scale development. Given the limited amount of empirical research and absence of a developed framework on students' responses to types of instruction, we chose an inductive approach to item generation (Ironson, Smith, Brannick, Gibson, & Paul, [<reflink idref="bib45" id="ref42">45</reflink>] ). Figure [NaN] represents the framework we developed in order to better understand students' responses to types of instruction; the framework comprises several groups of variables that potentially contribute to student response. Instructors likely influence student response by their choice of instructional strategy (e.g., lecturing or active learning) and how they introduce and manage that strategy in the classroom. We hypothesized that student response depends in part on student characteristics, preferences, expectations, and prior experiences. The framework features characteristics of the course itself and clarifies that a student's reason for taking the course potentially influences his or her response to types of instruction.</p> <p>The three sections of the instrument correspond to the three parts of our framework:</p> <p>types of instruction</p> <p>strategies for using in‐class activities</p> <p>student responses to instruction</p> <hd id="AN0122635335-4">Types of instruction</hd> <p>Because students' responses vary according to the types of instruction experienced by the student, we developed items to capture these instructional types, ranging from traditional lecture to simple and more complex forms of active learning. While trying to characterize these types of instruction by the nature of what occurs during the instruction, such as individual work, group work, and pair and share, we also wanted to frame them around the types of cognitive processes used by students during the activities to understand whether or not certain types of instruction shape students' responses.</p> <p>We modified Chi and Wylie's ([<reflink idref="bib18" id="ref43">18</reflink>] ) interactive‐constructive‐active‐passive (ICAP) model, which classifies instructional activities as interactive, constructive, active, or passive learning processes. Although our modified version uses the same format as the original ICAP, we redefined some of the original terminology to be more consistent with other research on active learning (Felder & Brent, [<reflink idref="bib25" id="ref44">25</reflink>] ; Prince, [<reflink idref="bib72" id="ref45">72</reflink>] ). We made three modifications.</p> <p>First, we sought to differentiate between active and passive types of instruction. Both types involve the individual students' actions (or lack thereof) during the instructional practice. We defined passive instruction as occurring when students are expected to passively receive information from the instructor. Examples include listening to lectures or watching the instructor solve problems on the board. Since our focus for passive instruction was on information received directly from the instructor, we did not include textbooks and other resources when asking students about information sources. We defined active instruction as occurring when students are engaged with the course content in any individual activity. Examples include asking the instructor questions or answering questions posed by the instructor during class.</p> <p>Because there is clear evidence that team and group activities can generate high levels of negative student response (Bacon, Stewart, & Silver, [<reflink idref="bib7" id="ref46">7</reflink>] ; Donohue & Richards, [<reflink idref="bib22" id="ref47">22</reflink>] ; Lake, [<reflink idref="bib52" id="ref48">52</reflink>] ; Oakley, Hanna, Kuzmyn, & Felder, [<reflink idref="bib64" id="ref49">64</reflink>] ; Powell & Kalina, [<reflink idref="bib70" id="ref50">70</reflink>] ), we made a distinction between individual activities and those with two or more students. For the latter we used the term interactive instruction, which is similar to Chi and Wylie's ([<reflink idref="bib18" id="ref51">18</reflink>] ) use of the term. Our conceptualization differs, however, in that we included any interaction students might have with their peers during the semester (including studying or completing homework in groups), while Chi and Wylie ([<reflink idref="bib18" id="ref52">18</reflink>] ) stipulate that the activity must involve students creating knowledge together; for example, students must have a dialogue with other students. Examples of interactive types of instruction include doing hands‐on group activities during class and being graded based on the performance of a group.</p> <p>Finally, some complex types of active learning include elements such as self‐directed learning and ill‐structured problems that have been hypothesized to generate significant student resistance (Hung, Bailey, & Jonassen, [<reflink idref="bib44" id="ref53">44</reflink>] ; Van Barneveld & Strobel, [<reflink idref="bib88" id="ref54">88</reflink>] ; Yadav, Subedi, Lunderberg, & Bunting, [<reflink idref="bib93" id="ref55">93</reflink>] ). These types of activities are defined by learning on one's own (self‐discovery), rather than learning from being told what to do (direct instruction; Chi, [<reflink idref="bib17" id="ref56">17</reflink>] ). Thus, we retained Chi and Wylie's ([<reflink idref="bib18" id="ref57">18</reflink>] ) definition for constructive instruction for these instructional types since they place high expectations on students and represent significant departures from many traditional classes.</p> <p>Altogether, we created 21 items for students to report the frequency of these types of instruction (Table [NaN] ). To gauge their desired frequency, we also asked students whether they wanted more or fewer of these activities in their ideal course. We expected students' responses to a particular type of instructional practice to be based not only on the actual level of use, but also on the difference between the actual and desired levels of use.</p> <p>Factor Loadings for the Ideal Types of Instruction Items</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Instrument item</th><th align="center">Factor</th></tr><tr valign="bottom"><th align="center">1</th><th align="center">2</th><th align="center">3</th><th align="center">4</th></tr><tr valign="top"><td align="left">Factor 1: Interactive</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Solve problems in a group during class</td><td align="center">0.71</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Do hands‐on group activities during class</td><td align="center">0.61</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Discuss concepts with classmates during class</td><td align="center">0.59</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Work in assigned groups to complete homework or other projects</td><td align="center">0.56</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Be graded based on the performance of my group</td><td align="center">0.52</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Study course content with classmates outside of class</td><td align="center">0.42</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Factor 2: Constructive</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Make and justify assumptions when not enough information is provided</td><td align="center" /><td align="center">0.69</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Find additional information not provided by the instructor to complete assignments</td><td align="center" /><td align="center">0.66</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Take initiative for identifying what I need to know</td><td align="center" /><td align="center">0.62</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Brainstorm different possible solutions to a given problem</td><td align="center" /><td align="center">0.58</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Assume responsibility for learning material on my own</td><td align="center" /><td align="center">0.58</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Solve problems that have more than one correct answer</td><td align="center" /><td align="center">0.43</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Factor 3: Active</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Make individual presentations to the class</td><td align="center" /><td align="center" /><td align="center">0.64</td><td align="center" /></tr><tr valign="top"><td align="left">Be graded on my class participation</td><td align="center" /><td align="center" /><td align="center">0.54</td><td align="center" /></tr><tr valign="top"><td align="left">Solve problems individually during class</td><td align="center" /><td align="center" /><td align="center">0.47</td><td align="center" /></tr><tr valign="top"><td align="left">Answer questions posed by the instructor during class</td><td align="center" /><td align="center" /><td align="center">0.46</td><td align="center" /></tr><tr valign="top"><td align="left">Ask the instructor questions during class</td><td align="center" /><td align="center" /><td align="center">0.42</td><td align="center" /></tr><tr valign="top"><td align="left">Preview concepts before class by reading, watching videos, etc.</td><td align="center" /><td align="center" /><td align="center">0.32</td><td align="center" /></tr><tr valign="top"><td align="left">Factor 4: Passive</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Listen to the instructor lecture during class</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.52</td></tr><tr valign="top"><td align="left">Watch the instructor demonstrate how to solve problems</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.57</td></tr><tr valign="top"><td align="left">Get most of the information needed to solve the homework directly from the instructor</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.48</td></tr><tr valign="top"><td align="left">Eigenvalue</td><td align="center">2.08</td><td align="center">2.59</td><td align="center">1.49</td><td align="center">1.16</td></tr><tr valign="top"><td align="left">Percentage of variance</td><td align="center">23%</td><td align="center">29%</td><td align="center">17%</td><td align="center">13%</td></tr><tr valign="top"><td align="left">Construct reliability</td><td align="center">0.73</td><td align="center">0.77</td><td align="center">0.63</td><td align="center">0.59</td></tr></table> </ephtml> </p> <hd id="AN0122635335-5">Strategies for using in‐class activities</hd> <p>While little empirical research has investigated the effectiveness of strategies for using in‐class activities, several authors give advice about how to introduce different types of instruction and minimize negative reactions (Armstrong, [<reflink idref="bib4" id="ref58">4</reflink>] ; Bentley, Kennedy, & Semsar, [<reflink idref="bib9" id="ref59">9</reflink>] ; Moffett & Hill, [<reflink idref="bib60" id="ref60">60</reflink>] ; Van Barneveld & Strobel, [<reflink idref="bib88" id="ref61">88</reflink>] ). We included these strategies in the StRIP instrument to allow more thoughtful analysis of their relative effectiveness.</p> <p>Three themes emerge from the literature on reducing student resistance. First, beginning the course activity with an explanation of its purpose and process and an acknowledgment of its challenges can better prepare students for what is expected of them and why the activity is important (Bacon et al., [<reflink idref="bib7" id="ref62">7</reflink>] ; Yadav, Subedi, Lunderberg, & Bunting, [<reflink idref="bib93" id="ref63">93</reflink>] ), especially if their participation might affect their grade (Donohue & Richards, [<reflink idref="bib22" id="ref64">22</reflink>] ). Indeed, Gaffney, Gaffney, and Beichner's ([<reflink idref="bib34" id="ref65">34</reflink>] ) Pedagogical Expectancy Violation Assessment acknowledges that students' expectations of active learning can fluctuate throughout the semester, and that this fluctuation can affect students' responses to the activities. Second, soliciting student feedback and providing the support needed to successfully complete the activity assists students in achieving their goals (Bentley et al., [<reflink idref="bib9" id="ref66">9</reflink>] ; Yadav et al., [<reflink idref="bib93" id="ref67">93</reflink>] ). Finally, designing appropriately challenging activities ensures that all students can successfully attempt and complete the activity (Donohue & Richards, [<reflink idref="bib22" id="ref68">22</reflink>] ; Van Barneveld & Strobel, [<reflink idref="bib88" id="ref69">88</reflink>] ).</p> <p>We used both the published strategies suggested for using in‐class activities and strategies we observed in our prior research (Shekhar et al., [<reflink idref="bib80" id="ref70">80</reflink>] ) as we developed the strategies for using in‐class activities items on the StRIP instrument. Altogether, we created eight items for students to report how frequently the instructor engaged in the recommended strategies (Table [NaN] ).</p> <p>Factor Loadings for Strategies for Using In‐Class Activities Items</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Instrument item</th><th align="center">Factor</th></tr><tr valign="bottom"><th align="center">1</th><th align="center">2</th></tr><tr valign="top"><td align="left">Factor 1: Explanation</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Clearly explained the purpose of the activity</td><td align="center">0.66</td><td align="center" /></tr><tr valign="top"><td align="left">Discussed how this activity related to my learning</td><td align="center">0.60</td><td align="center" /></tr><tr valign="top"><td align="left">Clearly explained what I was expected to do for the activity</td><td align="center">0.42</td><td align="center" /></tr><tr valign="top"><td align="left">Used activities that were the right difficulty level (not too easy, not too difficult)</td><td align="center">0.38</td><td align="center" /></tr><tr valign="top"><td align="left">Factor 2: Facilitation</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Encouraged students to engage with the activity through his/her demeanor</td><td align="center" /><td align="center">0.81</td></tr><tr valign="top"><td align="left">Walked around the room to assist me or my group with the activity, if needed</td><td align="center" /><td align="center">0.71</td></tr><tr valign="top"><td align="left">Gave me an appropriate amount of time to engage with the activity</td><td align="center" /><td align="center">0.56</td></tr><tr valign="top"><td align="left">Solicited my feedback or that of other students about the activity</td><td align="center" /><td align="center">0.37</td></tr><tr valign="top"><td align="left">Eigenvalue</td><td align="center">1.96</td><td align="center">1.62</td></tr><tr valign="top"><td align="left">Percentage of variance</td><td align="center">49%</td><td align="center">41%</td></tr><tr valign="top"><td align="left">Construct reliability</td><td align="center">0.80</td><td align="center">0.71</td></tr></table> </ephtml> </p> <p>Student responses to instruction To characterize students' responses to types of instruction, we drew upon ideas found in the literature, including the school classroom engagement concept of Fredricks, Blumenfeld, and Paris ([<reflink idref="bib30" id="ref71">30</reflink>] ), Chasteen's ([<reflink idref="bib16" id="ref72">16</reflink>] ) construct of productive engagement, and Weimer's ([<reflink idref="bib90" id="ref73">90</reflink>] ) framework on student resistance. The idea of engagement is often characterized as the responses students have to their experiences at specific moments in time (Lawson & Lawson, [<reflink idref="bib53" id="ref74">53</reflink>] ). Such responses can range from moments of total engagement or flow to more passive moments of boredom or lack of interest (Pekrun & Linnebrink‐Garcia, [<reflink idref="bib68" id="ref75">68</reflink>] ). Hence, we designed our instrument to examine how types of instruction facilitate students' engagement in the classroom, but we also wished to address faculty concerns regarding student resistance to these types of instruction, rather than simply measure boredom or lack of engagement.</p> <p>Previous research has conceptualized three forms of classroom engagement: cognitive engagement (psychological investment in classroom activities), affective‐emotional engagement (social and emotional connections to the classroom), and behavioral engagement (students' behavior in the classroom; Appleton, Christenson, & Furlong, [<reflink idref="bib6" id="ref76">6</reflink>] ; Fredericks et al., [<reflink idref="bib30" id="ref77">30</reflink>] ; Furlong & Christenson, [<reflink idref="bib33" id="ref78">33</reflink>] ; Skinner & Pitzer, [<reflink idref="bib83" id="ref79">83</reflink>] ). To these three forms of engagement we added a fourth concept of evaluation, because of the value instructors place on end‐of‐semester student ratings. We constructed four subscales:</p> <p>value – the degree to which students see the activity as worthwhile (cognitive)</p> <p>positivity – how positive or negative students feel about the activity (affective‐emotional)</p> <p>participation – the extent to which students do or do not participate or demonstrate resistance (behavioral)</p> <p>evaluation – the way students rate the instructor or course at the end of the term</p> <p>Value Chasteen ([<reflink idref="bib16" id="ref80">16</reflink>] ) defines value as a measure of some elements of cognitive engagement that are affected by students' thoughts, beliefs, and expectations. In their review of school engagement, Fredricks et al. ([<reflink idref="bib30" id="ref81">30</reflink>] ) indicated that cognitive engagement stresses students' investment in their learning and incorporated literature on learning and instruction, self‐regulation, and investment in learning. There are several conceptualizations of cognitive engagement, which include a desire to go beyond the typical requirements of a course (Connell & Wellborn, [<reflink idref="bib20" id="ref82">20</reflink>] ; Newmann, Wehlage, & Lamborn, [<reflink idref="bib62" id="ref83">62</reflink>] ; Wehlage, Rutter, Smith, Lesko, & Fernandez, [<reflink idref="bib89" id="ref84">89</reflink>] ) and a self‐regulated motivation to learn and do well in a course (Brophy, [<reflink idref="bib11" id="ref85">11</reflink>] ; Pintrich & De Groot, [<reflink idref="bib69" id="ref86">69</reflink>] ; Zimmerman, [<reflink idref="bib94" id="ref87">94</reflink>] ). In our instrument, value is related to students' investment in their learning. At the high end of the value scale, students understand and accept the rationale for the activity, and they feel the time used for the activity is beneficial. At the other end of the scale, students tend to disagree with the rationale for the activity and feel that time could be better spent doing other things.</p> <p>Positivity Affective‐emotional engagement refers to the affective reactions of students in the classroom, including anxieties, feelings of belongingness, happiness, sadness, interest, and boredom (Connell & Wellborn, [<reflink idref="bib20" id="ref88">20</reflink>] ; Skinner & Belmont, [<reflink idref="bib82" id="ref89">82</reflink>] ). Although a traditional scale of academic emotions (Pekrun, Goetz, Titz, & Perry, [<reflink idref="bib67" id="ref90">67</reflink>] ) measures how students' goals affect their own emotions in the classroom setting (Lee & Smith, [<reflink idref="bib54" id="ref91">54</reflink>] ; Pekrun, Elliot, & Maier, [<reflink idref="bib66" id="ref92">66</reflink>] ; Stipek, [<reflink idref="bib84" id="ref93">84</reflink>] ), the context of our StRIP instrument is different in that it measures students' reactions to the instructor and the course. Thus, we decided to label this factor as positivity to avoid any confusion with the academic emotions scale. At the high end of this scale, students feel positively about the task, instructor, and classroom environment. Students with low positivity respond in a negative way.</p> <p>Participation Because the research on behavioral engagement is considerably broad (Lawson & Lawson, [<reflink idref="bib53" id="ref94">53</reflink>] ) and often captures student behavior outside of the classroom (Finn, Folger, & Cox, [<reflink idref="bib27" id="ref95">27</reflink>] ), we opted to constrain behaviors in our instrument to those exhibited only in the college classroom. Chasteen's ([<reflink idref="bib16" id="ref96">16</reflink>] ) work provided guidance for the positive components of behavioral engagement included in the instrument; we applied Weimer's ([<reflink idref="bib90" id="ref97">90</reflink>] ) framework on student resistance to further distinguish the negative components. Weimer identified three types of resistance, or negative behavioral engagement:</p> <p>Open resistance   On some occasions, students openly object to the approach. They may demonstrate open resistance by complaining, arguing, or objecting, and they generally do so in ways that are not constructive.</p> <p>Passive, nonverbal resistance   Students exhibit an overall lack of enthusiasm as a way to assert their objection to the approach. Students may demonstrate passive, nonverbal resistance by not doing assignments but offering excuses, faking attention, or appearing to take notes while working on material from another class.</p> <p>Partial compliance Students may demonstrate partial compliance by completing a task poorly, half‐heartedly, or quickly, by putting forth minimal effort, or by being preoccupied with procedural details.</p> <p>We labeled this factor participation. The items on our StRIP instrument in the participation subscale represent both these positive and negative components of participation.</p> <p>Evaluation Another significant element of students' responses is evaluation, or how students rate both the overall course and quality of instruction on course evaluation forms. Since student evaluations play a significant role in many instructors' retention, tenure, and promotion reviews, low student ratings are clearly an important response that is likely to influence whether instructors adopt and continue to use various types of instruction in their classes. To capture this element of students' responses, we added items to our StRIP instrument about the quality of the course and its instruction. These items were based on similar items from the Individual Development and Educational Assessment (IDEA) student survey form (Cashin, [<reflink idref="bib14" id="ref98">14</reflink>] , [<reflink idref="bib15" id="ref99">15</reflink>] ).</p> <p>Altogether, we created 15 items for students to report how often they responded in various ways to the types of instruction that were used in their course. These items are listed in Table [NaN] .</p> <p>Factor Loadings for the Student Responses to Instruction Items</p> <p> <ephtml> <table><tr valign="bottom"><th align="left" /><th align="center">Factor</th></tr><tr valign="bottom"><th align="left">Instrument item</th><th align="char" char=".">1</th><th align="char" char=".">2</th><th align="char" char=".">3</th><th align="char" char=".">4</th><th align="char" char=".">5</th></tr><tr valign="top"><td align="left">Factor 1: Value</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I felt the time used for the activity was beneficial</td><td align="center">0.89</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I saw the value in the activity</td><td align="center">0.84</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I felt the effort it took to do the activity was worthwhile</td><td align="center">0.71</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Factor 2: Positivity</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I felt positively towards the instructor</td><td align="center" /><td align="center">0.73</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I felt the instructor had my best interests in mind</td><td align="center" /><td align="center">0.66</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I enjoyed the activity</td><td align="center" /><td align="center">0.57</td><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Factor 3: Participation</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I participated actively (or attempted to)</td><td align="center" /><td align="center" /><td align="center">0.59</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I tried my hardest to do a good job</td><td align="center" /><td align="center" /><td align="center">0.48</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I pretended but did not actually participate (R<sup>a</sup>)</td><td align="center" /><td align="center" /><td align="center">0.46</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I rushed through the activity, giving minimal effort (R<sup>a</sup>)</td><td align="center" /><td align="center" /><td align="center">0.41</td><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Factor 4: Distraction</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">I distracted my peers during the activity</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.63</td><td align="center" /></tr><tr valign="top"><td align="left">I talked with classmates about other topics besides the activity</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.60</td><td align="center" /></tr><tr valign="top"><td align="left">I surfed the internet, checked social media, or did something else instead of doing the activity</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.52</td><td align="center" /></tr><tr valign="top"><td align="left">I pretended but did not actually participate</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.35</td><td align="center" /></tr><tr valign="top"><td align="left">I rushed through the activity, giving minimal effort</td><td align="center" /><td align="center" /><td align="center" /><td align="center">0.34</td><td align="center" /></tr><tr valign="top"><td align="left">Factor 5: Evaluation</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center" /></tr><tr valign="top"><td align="left">Overall, this was an excellent course</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center">0.82</td></tr><tr valign="top"><td align="left">Overall, the instructor was an excellent teacher</td><td align="center" /><td align="center" /><td align="center" /><td align="center" /><td align="center">0.82</td></tr><tr valign="top"><td align="left">Eigenvalue</td><td align="center">2.00</td><td align="center">1.28</td><td align="center">1.70</td><td align="center">2.10</td><td align="center">1.36</td></tr><tr valign="top"><td align="left">Percentage of variance</td><td align="center">67%</td><td align="center">43%</td><td align="center">43%</td><td align="center">27%</td><td align="center">68%</td></tr><tr valign="top"><td align="left">Construct reliability</td><td align="center">0.87</td><td align="center">0.72</td><td align="center">0.77</td><td align="center">0.69</td><td align="center">0.73<sup>b</sup></td></tr></table> </ephtml> </p> <p>1 aItems were reverse‐coded in this factor. bStatistic calculated using the Spearman‐Brown coefficient (Eisinga et al., 23).</p> <hd id="AN0122635335-6">Step 2: Validity Testing</hd> <p>In our second step, testing for validity, we wanted to ensure that the proposed uses for the instrument were appropriate given the context and purposes of our study (AERA, APA, & NCME, [<reflink idref="bib2" id="ref100">2</reflink>] ). Specifically, we developed our StRIP instrument to measure students' responses to types of instruction encountered in the undergraduate engineering classroom. Therefore, the process of establishing evidence for the validity of our measures was achieved in a number of ways: using multiple, mixed‐methods approaches for development and validation (Haynes, Richard, & Kubany, [<reflink idref="bib39" id="ref101">39</reflink>] ); subjecting the instrument to expert review (Nunnally & Bernstein, [<reflink idref="bib63" id="ref102">63</reflink>] ); conducting cognitive interviewing with potential respondents of the instrument (Nunnally & Bernstein, [<reflink idref="bib63" id="ref103">63</reflink>] ); and reporting results to expert reviewers (Hinkin, [<reflink idref="bib41" id="ref104">41</reflink>] ). We especially used classroom observations, expert review, and cognitive interviewing during this validation process. These are all standard practices for establishing validity as according to the Standards for Educational and Psychological Measurement (AERA, APA, & NCME, [<reflink idref="bib2" id="ref105">2</reflink>] ). As indicated by the recursive nature of steps 1 and 2 in Figure [NaN] , this process often led to generating new items and revising factors based on feedback from these various sources.</p> <p>Classroom observations   In addition to our extensive literature review and item development process, we recognized the need to collect more concrete data about students' responses to types of instruction. We conducted classroom observations to inform the instrument development process. During our survey development process, we conducted observations in four large introductory engineering courses, ranging in size from 70 to 150 students, at two large public research universities (Shekhar et al., [<reflink idref="bib80" id="ref106">80</reflink>] ).</p> <p>These observations served three purposes. First, by collecting first‐hand observations of various types of students' responses to instruction, we further confirmed our framework (Figure [NaN] ). Second, we observed strategies for using in‐class activities that were not mentioned in the literature and which we subsequently added to our instrument. Specifically, we included two items from Table [NaN] (“Used activities that were the right difficulty level (not too easy, not too difficult)” and “Walked around the room to assist me or my group with the activity, if needed”) to address strategies observed in the classroom. Finally, we pilot tested the StRIP instrument in some of the same classes we observed; this testing allowed us to study the extent to which students' responses about types of instruction were related to our independent observations. Using these observations as a form of triangulation (Greene, Caracelli, & Graham, [<reflink idref="bib36" id="ref107">36</reflink>] ), we gained confidence in the instrument's ability to measure the underlying factors in our study.</p> <p>Expert review   Following our initial review of the literature, we created a preliminary draft of the StRIP instrument and invited our three‐member advisory board to offer their expert critique. The board included faculty who were experienced in instrument design and psychometrics, types of instruction, and students' responses to different types of instruction. Their feedback aided in refining our instrument. They provided guidance on timing and logistics for implementing the instrument, suggested that we find a framework for our instructional types and include items related to positivity and enjoyment, and recommended we clarify the response scale for the items by incorporating Fraser's ([<reflink idref="bib29" id="ref108">29</reflink>] ) classroom environments frequency scale rather than using a typical Likert scale response.</p> <p>Cognitive interviewing   Following the approach used by Ouimet, Bunnage, Carini, Kuh, and Kennedy ([<reflink idref="bib65" id="ref109">65</reflink>] ), we conducted cognitive interviews (Willis, [<reflink idref="bib91" id="ref110">91</reflink>] ) with 12 undergraduate engineering students at three institutions to confirm that the instrument was well designed for the target audience. We asked students to review each individual item; describe what they thought the item was asking, how they would respond, and how they would arrive at their response; and talk about other issues such as clarity of items and response scales and ease of completion. These cognitive interviews provided assurance that the students' interpretations of the instrument and its individual items were aligned with the intended constructs. Student feedback allowed us to better organize the instrument and reformat some question prompts. Specifically, these students suggested that we move the student responses to instruction section to the front of the instrument, because it allowed them to think broadly about their experiences in class before outlining specific practices in the types of instruction section.</p> <hd id="AN0122635335-7">Step 3: Implementation</hd> <p>Next we pilot tested the draft instrument in two phases. During the first phase, we studied 191 students in four courses from three institutions; during the second phase, we studied an additional 171 students in four courses from three institutions. Across both phases, we administered the instrument to a total of 362 students in eight courses at four institutions. Additional information on the courses in our sample is given in Table [NaN] .</p> <p>Descriptive Statistics for Courses in Pilot Testing of StRIP Instrument ( N = 362)</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Course no.</th><th align="center">n</th><th align="center">Type of analysis</th><th align="center">Course type</th><th align="center">Sample activities</th></tr><tr valign="top"><td align="left">1</td><td align="char" char=".">96</td><td align="left">Exploratory</td><td align="left">Sophomore‐level chemical engineering</td><td align="left">Team‐based work in randomized seating assignments</td></tr><tr valign="top"><td align="left">2</td><td align="char" char=".">34</td><td align="left">Exploratory</td><td align="left">Team‐taught first‐year engineering</td><td align="left">Traditional lecture‐style and pair‐and‐share group work</td></tr><tr valign="top"><td align="left">3</td><td align="char" char=".">34</td><td align="left">Exploratory</td><td align="left">Sophomore‐level thermodynamics</td><td align="left">Multiple instructional practices throughout the semester</td></tr><tr valign="top"><td align="left">4</td><td align="char" char=".">27</td><td align="left">Exploratory</td><td align="left">First‐year chemical engineering</td><td align="left">Primarily traditional instructional strategies</td></tr><tr valign="top"><td align="left">5</td><td align="char" char=".">34</td><td align="left">Confirmatory</td><td align="left">Lower division mechanical engineering</td><td align="left">Students worked on problems individually</td></tr><tr valign="top"><td align="left">6</td><td align="char" char=".">59</td><td align="left">Confirmatory</td><td align="left">Lower division electrical engineering</td><td align="left">Group based problem solving and discussion activities</td></tr><tr valign="top"><td align="left">7</td><td align="char" char=".">24</td><td align="left">Confirmatory</td><td align="left">Junior‐level mechanical engineering</td><td align="left">Multiple instructional practices throughout the semester</td></tr><tr valign="top"><td align="left">8</td><td align="char" char=".">54</td><td align="left">Confirmatory</td><td align="left">First‐year chemical engineering</td><td align="left">Primarily traditional instructional strategies</td></tr></table> </ephtml> </p> <p>We selected courses for our pilot testing through a mix of convenience and purposive sampling (Teddlie & Yu, [<reflink idref="bib86" id="ref111">86</reflink>] ). A member of our research team at each of four institutions chose one or two instructors teaching gateway engineering courses on the basis of their prior knowledge of their instructional methods. All students in those classes were asked to complete the StRIP instrument. Although students were offered an opportunity to opt out of taking the instrument, we are not aware of any students who did so. Therefore, no sample weights were used, because our selection was representative of each course. Only 11 responses had missing or incomplete data on any of the items. Because this number was less than 3% of the total sample and the missing data pattern appeared to be random (Rubin, [<reflink idref="bib77" id="ref112">77</reflink>] ), these surveys were removed from the analyses. We used data from the first phase of pilot testing for an exploratory factor analysis and the second phase for confirming the factors identified in the first phase. All analyses were performed using Stata 13.1 SE software.</p> <hd id="AN0122635335-8">Step 4: Exploratory Factor Analysis</hd> <p>We conducted an exploratory factor analysis (EFA) on the StRIP instrument to identify emergent factors from our first phase of pilot testing and to determine items that might be particularly problematic given low or multiple factor loadings. The EFA included 191 responses to 44 items, giving us a 4:1 ratio of respondents to items, remaining above recommendations for a 3:1 participant‐to‐item ratio (Reise, Waller, & Comrey, [<reflink idref="bib75" id="ref113">75</reflink>] ; Thompson, [<reflink idref="bib87" id="ref114">87</reflink>] ).</p> <p>Because we were studying three categories of variables, we conducted three separate EFAs. Using a common‐factors method and promax oblique rotation (recommended for intercorrelated measures by Worthington and Whittaker, [<reflink idref="bib92" id="ref115">92</reflink>] ), we identified four factors for types of instruction, two factors for strategies for using in‐class activities, and four factors for student responses to instruction (as described subsequently and shown in Table [NaN] , we later split this construct into five factors). The factors and their loadings are also listed in Tables [NaN] , [NaN] , and [NaN] . All factors had eigenvalues above 1.0 (Kaiser, [<reflink idref="bib48" id="ref116">48</reflink>] ), and each EFA model was tested using standard tests of significance (Bartlett's test of sphericity) and sampling adequacy (Kaiser‐Meyer‐Olkin). All models were statistically significant (p < 0.001), indicating that the variables were intercorrelated, and their sampling adequacies were above the 0.60 required for good factor analyses (Tabachnick & Fidell, [<reflink idref="bib85" id="ref117">85</reflink>] ). All items had a loading at or above the threshold of 0.32 (Comrey & Lee, [<reflink idref="bib19" id="ref118">19</reflink>] ), and each construct had a construct reliability above the recommended benchmark of 0.60 (Bagozzi & Yi, [<reflink idref="bib8" id="ref119">8</reflink>] ). For the evaluation construct, we used the Spearman‐Brown coefficient to measure construct reliability, as recommended in previous research (Eisinga, Grotenhuis, & Pelzer, [<reflink idref="bib23" id="ref120">23</reflink>] ).</p> <p>Based on the response loadings in each EFA, we developed a name for each factor to assist in describing the phenomenon captured by the groupings. For types of instruction (Table [NaN] ), we conducted an EFA on both students' ideal types of instruction as well as what they actually experienced in the course. While the factors related to ideal instruction closely aligned to our adaptations of the ICAP framework (Chi & Wylie, [<reflink idref="bib18" id="ref121">18</reflink>] ), those related to actual experience did not. We hypothesize this occurred because, while students tend to think about ideal types of instruction in term of the interactive, constructive, active, and passive categories, the capabilities of an instructor to balance each of these types in actual instruction might be limited. Therefore, we only present the analyses for the ideal types of instruction.</p> <p>For strategies for using in‐class activities (Table [NaN] ), we identified factors including explanation strategies (where the instructor was the main character in the strategy and took the role of explaining the activity) and facilitation strategies (where the instructor facilitated opportunities for students to participate in the strategy). For student responses to instruction (Table [NaN] ), although we initially designed the instrument with four subscales, the EFA resulted in two factors that emerged from the participation factor – student distraction and student participation. Distraction contains items where students distract themselves or peers during the learning process, whereas participation indicates the extent to which students participated in the activity. All five resulting factors and their loadings are presented in Table [NaN] .</p> <hd id="AN0122635335-9">Step 5: Confirmatory Factor Analysis</hd> <p>Given the success of the initial pilot testing, we used data from the second pilot phase of the StRIP instrument to conduct a confirmatory factor analysis (CFA); see Tables [NaN] , [NaN] , and [NaN] to verify the reliability of the factors. The CFA included 171 responses to 44 items giving us a nearly 4:1 ratio of respondents to items, which falls within recommended minimum sample size (Kline, [<reflink idref="bib50" id="ref122">50</reflink>] ). The purpose of the CFA was to test the model identified in the EFA for structural fit to the developed constructs. Recently, researchers have turned to structural equation modeling (SEM) rather than standard factor analysis techniques to conduct a CFA (Martens, [<reflink idref="bib57" id="ref123">57</reflink>] ; Martens & Hasse, [<reflink idref="bib58" id="ref124">58</reflink>] ). Usually, SEM consists of two steps in the model‐building process: testing for the factorial validity of a theoretical construct (first‐order CFA model) and a path analysis to describe the relationship between theoretical constructs. Given our desire to replicate the latent factors of the instrument, as opposed to determining their relationship(s) with other factors, we chose to only conduct a first‐order CFA model (Byrne, [<reflink idref="bib12" id="ref125">12</reflink>] ).</p> <p>Confirmatory Factor Analysis Estimates for Ideal Types of Instruction</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Instrument item</th><th align="center">Standardized factor loadings</th><th align="center">Standard error</th><th align="center">Item reliability (R<sup>2</sup>)</th><th align="center">Construct reliability</th><th align="center">Average variance extracted</th></tr><tr valign="top"><td align="left">Factor 1: Interactive</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.80</td><td align="char" char=".">0.88</td></tr><tr valign="top"><td align="left">Solve problems in a group during class</td><td align="char" char=".">0.72</td><td align="char" char=".">0.05</td><td align="char" char=".">0.74</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Do hands‐on group activities during class</td><td align="char" char=".">0.63</td><td align="char" char=".">0.06</td><td align="char" char=".">0.76</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Discuss concepts with classmates during class</td><td align="char" char=".">0.64</td><td align="char" char=".">0.06</td><td align="char" char=".">0.77</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Work in assigned groups to complete homework or other projects</td><td align="char" char=".">0.70</td><td align="char" char=".">0.05</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Be graded based on the performance of my group</td><td align="char" char=".">0.44</td><td align="char" char=".">0.07</td><td align="char" char=".">0.80</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Study course content with classmates outside of class</td><td align="char" char=".">0.61</td><td align="char" char=".">0.06</td><td align="char" char=".">0.77</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 2: Constructive</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.77</td><td align="char" char=".">0.86</td></tr><tr valign="top"><td align="left">Make and justify assumptions when not enough information is provided</td><td align="char" char=".">0.51</td><td align="char" char=".">0.07</td><td align="char" char=".">0.76</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Find additional information not provided by the instructor to complete assignments</td><td align="char" char=".">0.66</td><td align="char" char=".">0.05</td><td align="char" char=".">0.72</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Take initiative for identifying what I need to know</td><td align="char" char=".">0.60</td><td align="char" char=".">0.06</td><td align="char" char=".">0.73</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Brainstorm different possible solutions to a given problem</td><td align="char" char=".">0.50</td><td align="char" char=".">0.07</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Assume responsibility for learning material on my own</td><td align="char" char=".">0.76</td><td align="char" char=".">0.05</td><td align="char" char=".">0.69</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Solve problems that have more than one correct answer</td><td align="char" char=".">0.50</td><td align="char" char=".">0.07</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 3: Active</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.73</td><td align="char" char=".">0.85</td></tr><tr valign="top"><td align="left">Make individual presentations to the class</td><td align="char" char=".">0.43</td><td align="char" char=".">0.08</td><td align="char" char=".">0.72</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Be graded on my class participation</td><td align="char" char=".">0.42</td><td align="char" char=".">0.08</td><td align="char" char=".">0.73</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Solve problems individually during class</td><td align="char" char=".">0.55</td><td align="char" char=".">0.06</td><td align="char" char=".">0.71</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Answer questions posed by the instructor during class</td><td align="char" char=".">0.74</td><td align="char" char=".">0.04</td><td align="char" char=".">0.67</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Ask the instructor questions during class</td><td align="char" char=".">0.74</td><td align="char" char=".">0.04</td><td align="char" char=".">0.67</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Preview concepts before class by reading, watching videos, etc.</td><td align="char" char=".">0.56</td><td align="char" char=".">0.06</td><td align="char" char=".">0.70</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 4: Passive</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.65</td><td align="char" char=".">0.80</td></tr><tr valign="top"><td align="left">Listen to the instructor lecture during class</td><td align="char" char=".">0.62</td><td align="char" char=".">0.09</td><td align="char" char=".">0.51</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Watch the instructor demonstrate how to solve problems</td><td align="char" char=".">0.60</td><td align="char" char=".">0.08</td><td align="char" char=".">0.52</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Get most of the information needed to solve the homework directly from the instructor</td><td align="char" char=".">0.51</td><td align="char" char=".">0.08</td><td align="char" char=".">0.63</td><td align="left" /><td align="left" /></tr></table> </ephtml> </p> <p>Confirmatory Factor Analysis Estimates for Strategies for Using In‐Class Activities</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Instrument item</th><th align="center">Standardized factor loadings</th><th align="center">Standard error</th><th align="center">Item reliability (R<sup>2</sup>)</th><th align="center">Construct reliability</th><th align="center">Average variance extracted</th></tr><tr valign="top"><td align="left">Factor 1: Explanation</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.80</td><td align="char" char=".">0.91</td></tr><tr valign="top"><td align="left">Clearly explained the purpose of the activity</td><td align="char" char=".">0.82</td><td align="char" char=".">0.05</td><td align="char" char=".">0.68</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Discussed how this activity related to my learning</td><td align="char" char=".">0.63</td><td align="char" char=".">0.04</td><td align="char" char=".">0.79</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Clearly explained what I was expected to do for the activity</td><td align="char" char=".">0.70</td><td align="char" char=".">0.06</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Used activities that were the right difficulty level (not too easy, not too difficult)</td><td align="char" char=".">0.63</td><td align="char" char=".">0.05</td><td align="char" char=".">0.78</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 2: Facilitation</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.71</td><td align="char" char=".">0.83</td></tr><tr valign="top"><td align="left">Encouraged students to engage with the activity through his/her demeanor</td><td align="char" char=".">0.81</td><td align="char" char=".">0.08</td><td align="char" char=".">0.54</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Walked around the room to assist me or my group with the activity, if needed</td><td align="char" char=".">0.70</td><td align="char" char=".">0.08</td><td align="char" char=".">0.61</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Gave me an appropriate amount of time to engage with the activity</td><td align="char" char=".">0.57</td><td align="char" char=".">0.08</td><td align="char" char=".">0.66</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Solicited my feedback or that of other students about the activity</td><td align="char" char=".">0.37</td><td align="char" char=".">0.08</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr></table> </ephtml> </p> <p>Confirmatory Factor Analysis Estimates for Student Responses to Instruction</p> <p> <ephtml> <table><tr valign="bottom"><th align="left">Instrument item</th><th align="center">Standardized factor loadings</th><th align="center">Standard error</th><th align="center">Item reliability (R<sup>2</sup>)</th><th align="center">Construct reliability</th><th align="center">Average variance extracted</th></tr><tr valign="top"><td align="left">Factor 1: Value</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.87</td><td align="char" char=".">0.95</td></tr><tr valign="top"><td align="left">I felt the time used for the activity was beneficial</td><td align="char" char=".">0.71</td><td align="char" char=".">0.04</td><td align="char" char=".">0.89</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I saw the value in the activity</td><td align="char" char=".">0.84</td><td align="char" char=".">0.03</td><td align="char" char=".">0.80</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I felt the effort it took to do the activity was worthwhile</td><td align="char" char=".">0.89</td><td align="char" char=".">0.03</td><td align="char" char=".">0.74</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 2: Positivity</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.72</td><td align="char" char=".">0.86</td></tr><tr valign="top"><td align="left">I felt positively towards the instructor</td><td align="char" char=".">0.66</td><td align="char" char=".">0.07</td><td align="char" char=".">0.64</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I felt the instructor had my best interests in mind</td><td align="char" char=".">0.73</td><td align="char" char=".">0.07</td><td align="char" char=".">0.53</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I enjoyed the activity</td><td align="char" char=".">0.57</td><td align="char" char=".">0.07</td><td align="char" char=".">0.72</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 3: Participation</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.77</td><td align="char" char=".">0.84</td></tr><tr valign="top"><td align="left">I participated actively (or attempted to)</td><td align="char" char=".">0.58</td><td align="char" char=".">0.08</td><td align="char" char=".">0.70</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I tried my hardest to do a good job</td><td align="char" char=".">0.67</td><td align="char" char=".">0.08</td><td align="char" char=".">0.72</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I pretended but did not actually participate</td><td align="char" char=".">0.71</td><td align="char" char=".">0.07</td><td align="char" char=".">0.74</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I rushed through the activity, giving minimal effort</td><td align="char" char=".">0.64</td><td align="char" char=".">0.09</td><td align="char" char=".">0.68</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 4: Distraction</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.73</td><td align="char" char=".">0.85</td></tr><tr valign="top"><td align="left">I distracted my peers during the activity</td><td align="char" char=".">0.58</td><td align="char" char=".">0.08</td><td align="char" char=".">0.75</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I talked with classmates about other topics besides the activity</td><td align="char" char=".">0.39</td><td align="char" char=".">0.05</td><td align="char" char=".">0.68</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I surfed the internet, checked social media, or did something else instead of doing the activity</td><td align="char" char=".">0.65</td><td align="char" char=".">0.06</td><td align="char" char=".">0.65</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I pretended but did not actually participate</td><td align="char" char=".">0.70</td><td align="char" char=".">0.06</td><td align="char" char=".">0.67</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">I rushed through the activity, giving minimal effort</td><td align="char" char=".">0.63</td><td align="char" char=".">0.06</td><td align="char" char=".">0.65</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Factor 5: Evaluation</td><td align="left" /><td align="left" /><td align="left" /><td align="char" char=".">0.72<sup>a</sup></td><td align="char" char=".">0.93</td></tr><tr valign="top"><td align="left">Overall, this was an excellent course</td><td align="char" char=".">0.82</td><td align="char" char=".">0.05</td><td align="char" char=".">0.60</td><td align="left" /><td align="left" /></tr><tr valign="top"><td align="left">Overall, the instructor was an excellent teacher</td><td align="char" char=".">0.82</td><td align="char" char=".">0.05</td><td align="char" char=".">0.60</td><td align="left" /><td align="left" /></tr></table> </ephtml> </p> <p>2 aStatistic calculated using the Spearman‐Brown coefficient (Eisinga et al., 23).</p> <p>The test statistics indicated good overall model fit. The chi‐square statistic for the model was 2.98, falling below the recommended threshold (Kline, [<reflink idref="bib49" id="ref126">49</reflink>] ). The root‐mean‐square error of approximation was 0.06, with the lower bound of our 90% confidence interval at 0.00 and an upper bound at 0.14, suggesting a reasonable fit to the model. The comparative fit index statistic was 0.98, indicating good model fit (Hu & Bentler, [<reflink idref="bib43" id="ref127">43</reflink>] ). Finally, the standardized root‐mean‐square residual was 0.03, considered to be favorable for the model (Hu & Bentler, [<reflink idref="bib43" id="ref128">43</reflink>] ).</p> <p>In addition to the factor loadings, we also display the standard error, item reliability, average variance extracted, and construct reliability of each of the factors in Tables [NaN] , [NaN] , and [NaN] . Item reliabilities ranged from 0.51 to 0.89, which exceed the acceptable value of 0.50 (Hair, Anderson, Tatham, & Black, [<reflink idref="bib37" id="ref129">37</reflink>] ). The average variance extracted for all constructs was well above the threshold value of 0.5 (Fornell & Larcker, [<reflink idref="bib28" id="ref130">28</reflink>] ). Finally, the reliabilities for each construct were above the benchmark of 0.60 (Bagozzi & Yi, [<reflink idref="bib8" id="ref131">8</reflink>] ). As noted in the Exploratory Factor Analysis section, the construct reliability for our two‐item evaluation construct was conducted using the Spearman‐Brown statistic, as is recommended with the use of two‐item scales (Eisinga et al., [<reflink idref="bib23" id="ref132">23</reflink>] ).</p> <hd id="AN0122635335-10">Step 6: Instrument Modification and Replication</hd> <p>After conducting the exploratory and confirmatory factor analyses, we engaged in instrument modification and replication to further strengthen the instrument. In the EFA and CFA, we found that two of the student responses to instruction items loaded strongly on two different factors. These items included “I pretended but did not actually participate” and “I rushed through the activity, giving minimal effort.” We determined this instance of double‐loading to be the result of items being worded as compound statements. The statements “I did not actually participate” and “I gave minimal effort” appeared to relate to the participation factor (the standardized factor loadings from our CFA were 0.71 and 0.64, respectively), while the statements “I pretended to participate” and “I rushed through the activity” appeared to relate to distraction (the standardized factor loadings from our CFA were 0.70 and 0.63, respectively). Therefore, we split these items to create four items to address both of the factors:</p> <p>I did not actually participate in the activity (participation)</p> <p>I gave the activity minimal effort (participation)</p> <p>I pretended to participate in the activity (distraction)</p> <p>I rushed through the activity (distraction)</p> <p>In addition to these changes, while we found that the reliability for the evaluation factor was strong, we chose to add a third item to strengthen the factor: “I would recommend this instructor to other students” (Cashin, [<reflink idref="bib14" id="ref133">14</reflink>] , [<reflink idref="bib15" id="ref134">15</reflink>] ). As our objective for this study was to measure the effects of in‐class exercises, we also modified or removed all instances of out‐of‐class learning from our instrument to represent only those types of instruction that occur during class. Following this modification process, we finalized the StRIP instrument v1.0 (Appendix). This instrument represents our team's efforts to further investigate students' responses to different types of instruction and is ready to be administered as part of our full‐scale study.</p> <hd id="AN0122635335-11">Limitations and Future Research</hd> <p>It is worth noting a few limitations in our instrument development, which we plan to address in our future research. First, the exploratory and confirmatory factor analyses are based on data from eight courses at four institutions. Although the four institutions represent doctoral, baccalaureate, and minority‐serving institutions, our findings are not necessarily generalizable. Furthermore, although our sample sizes appear to meet recommendations from the literature, they are still small and might influence our model fit. In our future research, we plan to expand our data collection methods to more locations to address these issues. We also plan to refine the instrument as needed on the basis of these expanded results.</p> <p>Second, because we asked students how often they reacted in various ways to all activities as a whole, rather than specific types of activities (see Appendix), it is more difficult to relate specific activities to specific student reactions. This decision was a tradeoff for brevity, because students would have to respond to the 17 student response to instruction items for each of the 21 different instructional types listed. We may reconsider expanding this survey in future studies to focus on student response to specific types of instruction.</p> <p>Third, the estimates for our types of instruction models are based solely on ideal types of instruction, because it would be difficult for an instructor to actually cover each of these types of instruction in a semester. However, students still perceive these types of instruction as aligned with one of these four categories: interactive, constructive, active, and passive. Consequently, much of our future research will directly investigate how students feel about these types of instruction in their ideal classroom, whether or not this perception aligns with what they actually experienced in the classroom, and subsequently, how they responded to the use of these types of instruction. Furthermore, we also plan to consider the use of separate constructs for the actual and ideal types of instruction in our future research.</p> <p>Finally, the instrument relies on student self‐reports of instructional practices, instructor strategies, and reactions to active learning. While this limitation is less of a concern for positivity, value, and evaluation items (Table [NaN] ), student reports of their own participation (Table [NaN] ), instructor strategies (Table [NaN] ), and frequency of types of instruction (Table [NaN] ) may be different from those of other students and the instructor. We will note this constraint in all our future research utilizing this instrument, yet we conclude that students' perception of the frequency of these activities is an important key to understanding how they ultimately respond to active learning. Other aspects of our ongoing work describe the preliminary results for our findings on student response to types of instruction and how we are comparing student and instructor responses and working with instructors to interpret their own data in instructional decisions.</p> <hd id="AN0122635335-12">Conclusion</hd> <p>This article has described the design process and pilot results for an instrument to measure student response to instructional practices. Since our focus was on development of the instrument, future analyses will involve a broader administration and more systematic analysis of the instrument across multiple types of courses and institutions. The instrument measures three constructs related to our framework: types of instruction, strategies for using in‐class activities, and student responses to instruction. Although the instrument was developed in the context of required gateway engineering courses, we expect that it may be relevant for a wider variety of STEM contexts, and we encourage other researchers to examine its usefulness in other contexts.</p> <p>We believe there are several practical implications for the use of this instrument in the engineering classroom. First, we described a spectrum of activities in the instrument so instructors can examine the types of instruction currently used in the engineering classroom, and how ideal these activities might be to their students. Second, from our review of the literature, we compiled a list of several strategies for using in‐class activities that instructors may wish to incorporate into their own courses to support student engagement. Third, we provided a list of students' responses to these types of instruction so that instructors can examine how their students respond to these activities and identify behaviors that might indicate students are disengaged during the process. Finally, our overall framework was developed with the hope that researchers and instructors, alike, can utilize this instrument to study multiple classrooms and identify relationships between types of instruction, how each type of instruction is introduced, and how students subsequently respond. For example, do students notice efforts taken by an instructor to explain the purpose of an activity? If not, maybe these efforts need to be more explicit or more frequent. Similarly, a few vocal students can sometimes give the impression that the entire class dislikes active learning. Having results from the instrument can help an instructor understand the views of all students in the class. There is much to be learned about this important area, and we encourage other instructors and researchers to use and build on this instrument in their own work.</p> <hd id="AN0122635335-13">Acknowledgments</hd> <p>This project was funded by the U.S. National Science Foundation through grants 1347417, 1347482, 1347580, 1347718, and 1500309. The opinions are those of the authors and do not necessarily represent the National Science Foundation. The authors would like to thank the instructors and students who agreed to be part of the pilot study, as well as project advisory board members.</p> <hd id="AN0122635335-14">Appendix</hd> <hd id="AN0122635335-15">StRIP Student Instrument</hd> <p>Student Responses to Instruction<sups>a</sups></p> <p>In this course, when the instructor asked you to do an in‐class activity (e.g., solve problems in a group during class or discuss concepts with classmates), how often did you react in the following ways?</p> <p>I did not actually participate in the activity.</p> <p>I gave the activity minimal effort.</p> <p>I felt positively towards the instructor.</p> <p>I tried my hardest to do a good job.</p> <p>I distracted my peers during the activity.</p> <p>I pretended to participate in the activity.</p> <p>I felt the effort it took to do the activity was worthwhile.</p> <p>I participated actively (or attempted to).</p> <p>I talked with classmates about other topics besides the activity.</p> <p>I felt the instructor had my best interests in mind.</p> <p>I saw the value in the activity.</p> <p>I felt the time used for the activity was beneficial.</p> <p>I enjoyed the activity.</p> <p>I surfed the internet, checked social media, or did something else</p> <p>instead of doing the activity.</p> <p>I rushed through the activity.</p> <p>Strategies for Using In‐Class Activities<sups>a</sups></p> <p>In this course, when the instructor asked you to do an in‐class activity (e.g,. solve problems in a group during class or discuss concepts with classmates), how often did the instructor do the following things?</p> <p>Clearly explained what I was expected to do for the activity.</p> <p>Clearly explained the purpose of the activity.</p> <p>Discussed how this activity related to my learning.</p> <p>Solicited my feedback or that of other students about the activity.</p> <p>Used activities that were the right difficulty level (not too easy, not too difficult).</p> <p>Walked around the room to assist me or my group with the activity, if needed.</p> <p>Encouraged students to engage with the activity through his/her demeanor.</p> <p>Gave me an appropriate amount of time to engage with the activity.</p> <p>Course Evaluation<sups>b</sups></p> <p>Please rate your level of agreement with the following items.</p> <p>Overall, this was an excellent course.</p> <p>Overall, the instructor was an excellent teacher.</p> <p>I would recommend this instructor to other students.</p> <p>Types of Instruction</p> <p>For each of the following things, please indicate how often you did each thing in this course<sups>c</sups> and how often you would like to do each in your ideal course.<sups>d</sups></p> <p>Listen to the instructor lecture during class.</p> <p>Brainstorm different possible solutions to a given problem.</p> <p>Find additional information not provided by the instructor to complete assignments.</p> <p>Work in assigned groups to complete homework or other projects.</p> <p>Make individual presentations to the class.</p> <p>Be graded on my class participation.</p> <p>Study course content with classmates outside of class.</p> <p>Assume responsibility for learning material on my own.</p> <p>Discuss concepts with classmates during class.</p> <p>Make and justify assumptions when not enough information is provided.</p> <p>Get most of the information needed to solve the homework directly from the instructor.</p> <p>Be graded based on the performance of my group.</p> <p>Preview concepts before class by reading, watching videos, etc.</p> <p>Solve problems in a group during class.</p> <p>Solve problems individually during class.</p> <p>Answer questions posed by the instructor during class.</p> <p>Ask the instructor questions during class.</p> <p>Take initiative for identifying what I need to know.</p> <p>Watch the instructor demonstrate how to solve problems.</p> <p>Solve problems that have more than one correct answer.</p> <p>Do hands‐on group activities during class.</p> <p> <sups>a</sups>Response options for each item were: 1 = almost never (<10% of the time); 2 = seldom (∼30% of the time); 3 = sometimes (∼50% of the time); 4 = often (∼70% of the time); 5 = very often (>90% of the time). <sups>b</sups>Response options for each item were: 1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree. <sups>c</sups>Response options for each item were: 1 = never; 2 = seldom (1–5 times per semester); 3 = sometimes (5–10 times per semester); 4 = often (once a week); 5 = very often (more than once/week). <sups>d</sups>Response options for each item were: 1 = much less; 2 = slightly less; 3 = about the same; 4 = slightly more; 5 = much more.</p> <ref id="AN0122635335-16"> <title>References</title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext>American Association for the Advancement of Science (AAAS). ( 2010 ). Vision and change: A call to action. Washington, DC : AAAS. <ulink href="http://visionandchange.org/files/2011/03/VC‐Brochure‐V6‐3.pdf">http://visionandchange.org/files/2011/03/VC‐Brochure‐V6‐3.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib2" idref="ref37" type="bt">2</bibl> <bibtext>American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, & NCME). ( 2014 ). Standards for educational and psychological testing. Washington, DC : AERA. </bibtext> </blist> <blist> <bibl id="bib3" idref="ref9" type="bt">3</bibl> <bibtext>Angelo, T. A., & Cross. K. P. ( 1993 ). Classroom assessment techniques. A handbook for college teachers. San Francisco, CA : Jossey‐Bass. </bibtext> </blist> <blist> <bibl id="bib4" idref="ref24" type="bt">4</bibl> <bibtext>Armstrong, J. S. ( 1998 ). Are student ratings of instruction useful? American Psychologist, 53 ( 11 ), 1223 – 1232. <ulink href="http://dx.doi.org/10.1037/0003‐066X.53.11.1223">http://dx.doi.org/10.1037/0003‐066X.53.11.1223</ulink></bibtext> </blist> <blist> <bibl id="bib5" idref="ref25" type="bt">5</bibl> <bibtext>Arum, R., & Roksa, J. ( 2011 ). Academically adrift. Chicago, IL : University of Chicago Press. <ulink href="http://dx.doi.org/10.7208/chicago/9780226028576.001.0001">http://dx.doi.org/10.7208/chicago/9780226028576.001.0001</ulink></bibtext> </blist> <blist> <bibl id="bib6" idref="ref41" type="bt">6</bibl> <bibtext>Appleton, J. J., Christenson, S. L., & Furlong, M. ( 2008 ). Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in Schools, 45, 369 – 386. <ulink href="http://dx.doi.org/10.1002/pits.20303">http://dx.doi.org/10.1002/pits.20303</ulink></bibtext> </blist> <blist> <bibl id="bib7" idref="ref46" type="bt">7</bibl> <bibtext>Bacon, D., Stewart, K., & Silver, W. ( 1999 ). Lessons from the best and worst student team experiences: How a teacher can make the difference. Journal of Management Education, 23 ( 5 ), 467 – 488. <ulink href="http://dx.doi.org/10.1177/105256299902300503">http://dx.doi.org/10.1177/105256299902300503</ulink></bibtext> </blist> <blist> <bibl id="bib8" idref="ref119" type="bt">8</bibl> <bibtext>Bagozzi, R. P., & Yi, Y. ( 1988 ). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16 ( 1 ), 74 – 94. <ulink href="http://dx.doi.org/10.1007/BF02723327">http://dx.doi.org/10.1007/BF02723327</ulink></bibtext> </blist> <blist> <bibl id="bib9" idref="ref59" type="bt">9</bibl> <bibtext>Bentley, F. J., Kennedy, S., & Semsar, K. ( 2011 ). How not to lose your students with concept maps. Journal of College Science Teaching, 41, 61 – 68. <ulink href="http://www.colorado.edu/sei/documents/IPHY/HowNotToLoseStudentsWithConceptMaps%5fJCST2011.pdf">http://www.colorado.edu/sei/documents/IPHY/HowNotToLoseStudentsWithConceptMaps%5fJCST2011.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib10" idref="ref16" type="bt">10</bibl> <bibtext>Borrego, M., Froyd, J. E., & Hall, T. S. ( 2010 ). Diffusion of engineering education innovations: A survey of awareness and adoption rates in US engineering departments. Journal of Engineering Education, 99 ( 3 ), 185 – 207. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2010.tb01056.x">http://dx.doi.org/10.1002/j.2168‐9830.2010.tb01056.x</ulink></bibtext> </blist> <blist> <bibl id="bib11" idref="ref85" type="bt">11</bibl> <bibtext>Brophy, J. E. ( 1987 ). Socializing students' motivation to learn. In M. L. Maehr & D. Kleiber (Eds.), Advances in motivation and achievement: Enhancing motivation (pp. 181 – 210 ). Greenwich, CT : JAI Press. </bibtext> </blist> <blist> <bibl id="bib12" idref="ref125" type="bt">12</bibl> <bibtext>Byrne, B. M. ( 2013 ). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York, NY: Routledge </bibtext> </blist> <blist> <bibl id="bib13" idref="ref32" type="bt">13</bibl> <bibtext>Carberry, A. R., Lee, H. S., & Ohland, M. W. ( 2010 ). Measuring engineering design self‐efficacy. Journal of Engineering Education, 99 ( 1 ), 71 – 79. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2010.tb01043.x">http://dx.doi.org/10.1002/j.2168‐9830.2010.tb01043.x</ulink></bibtext> </blist> <blist> <bibl id="bib14" idref="ref98" type="bt">14</bibl> <bibtext>Cashin, W. E. ( 1988 ). Student ratings of teaching: A summary of the research. IDEA Paper No. 20. Manhattan, KS: Kansas State University, Center for Faculty Evaluation and Development. </bibtext> </blist> <blist> <bibl id="bib15" idref="ref99" type="bt">15</bibl> <bibtext>Cashin, W. E. ( 1990 ). Student ratings of teaching: Recommendations for use. IDEA Paper No. 22. Manhattan, KS: Kansas State University, Center for Faculty Evaluation and Development. </bibtext> </blist> <blist> <bibl id="bib16" idref="ref72" type="bt">16</bibl> <bibtext>Chasteen, S. ( 2014, November 4). Measuring and improving students' engagement [Blog post]. Retrieved January 9, 2016 from <ulink href="http://blog.sciencegeekgirl.com/2014/11/02/measuring‐and‐improving‐students‐engagement/">http://blog.sciencegeekgirl.com/2014/11/02/measuring‐and‐improving‐students‐engagement/</ulink></bibtext> </blist> <blist> <bibl id="bib17" idref="ref56" type="bt">17</bibl> <bibtext>Chi, M. T. H. ( 2009 ). Active‐constructive‐interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73 – 105. <ulink href="http://dx.doi.org/10.1111/j.1756‐8765.2008.01005.x">http://dx.doi.org/10.1111/j.1756‐8765.2008.01005.x</ulink></bibtext> </blist> <blist> <bibl id="bib18" idref="ref43" type="bt">18</bibl> <bibtext>Chi, M. T. H., & Wylie, R. ( 2014 ). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49 ( 4 ), 219 – 243. <ulink href="http://dx.doi.org/10.1080/00461520.2014.965823">http://dx.doi.org/10.1080/00461520.2014.965823</ulink></bibtext> </blist> <blist> <bibl id="bib19" idref="ref118" type="bt">19</bibl> <bibtext>Comrey, A. L., & Lee, H. B. ( 1992 ). A first course in factor analysis ( 2nd ed. ). Hillsdale, NJ : Lawrence Erlbaum. </bibtext> </blist> <blist> <bibl id="bib20" idref="ref82" type="bt">20</bibl> <bibtext>Connell, J. P., & Wellborn, J. G. ( 1991 ). Competence, autonomy, and relatedness: A motivational analysis of self‐system processes. In M. Gunnar & L. A. Sroufe (Eds.), Minnesota Symposium on Child Psychology (Vol. 23). Chicago, IL : University of Chicago Press. </bibtext> </blist> <blist> <bibl id="bib21" idref="ref17" type="bt">21</bibl> <bibtext>Dancy, M. H., & Henderson, C. ( 2012 ). Experiences of new faculty implementing research‐based instructional strategies. Proceedngs of the 2011 Physics Education Research Conference, Melville, NY. <ulink href="http://homepages.wmich.edu/∼chenders/Publications/2011DancyPERCPaper.pdf">http://homepages.wmich.edu/∼chenders/Publications/2011DancyPERCPaper.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib22" idref="ref47" type="bt">22</bibl> <bibtext>Donohue, S., & Richards, L. ( 2009 ). Factors affecting student attitudes toward active learning activities in a graduate engineering statistics course. Paper presented at the 39th ASEE/IEEE Frontiers in Education conference, San Antonio, TX. <ulink href="http://dx.doi.org/10.1109/FIE.2009.5350587">http://dx.doi.org/10.1109/FIE.2009.5350587</ulink></bibtext> </blist> <blist> <bibl id="bib23" idref="ref120" type="bt">23</bibl> <bibtext>Eisinga, R., Grotenhuis, M. T., & Pelzer, B. ( 2013 ). The reliability of a two‐item scale: Pearson, Cronbach, or Spearman‐Brown?. International Journal of Public Health, 58 ( 4 ), 637 – 642. <ulink href="http://dx.doi.org/10.1007/s00038‐012‐0416‐3">http://dx.doi.org/10.1007/s00038‐012‐0416‐3</ulink></bibtext> </blist> <blist> <bibl id="bib24" idref="ref26" type="bt">24</bibl> <bibtext>Felder, R. M. ( 2011 ). Hang in there! Dealing with student resistance to learner‐centered teaching. Chemical Engineering Education, 43 ( 2 ), 131 – 132. <ulink href="http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Columns/HangInThere.pdf">http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Columns/HangInThere.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib25" idref="ref6" type="bt">25</bibl> <bibtext>Felder, R. M., & Brent, R. ( 2009 ). Active learning: An introduction. ASQ Higher Education Brief, 2 ( 4 ), 1 – 5. <ulink href="http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Papers/ALpaper(ASQ).pdf">http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Papers/ALpaper(ASQ).pdf</ulink></bibtext> </blist> <blist> <bibl id="bib26" idref="ref18" type="bt">26</bibl> <bibtext>Finelli, C. J., Daly, S. R., & Richardson, K. M. ( 2014 ). Bridging the research‐to‐practice gap: Designing an institutional change plan using local evidence. Journal of Engineering Education, 103 ( 2 ), 331 – 361. <ulink href="http://dx.doi.org/10.1002/jee.20042">http://dx.doi.org/10.1002/jee.20042</ulink></bibtext> </blist> <blist> <bibl id="bib27" idref="ref95" type="bt">27</bibl> <bibtext>Finn, J. D., Folger, J., & Cox, D. ( 1991 ). Measuring participation among elementary school students. Educational and Psychological Measurement, 50, 393 – 401. <ulink href="http://dx.doi.org/10.1177/0013164491512013">http://dx.doi.org/10.1177/0013164491512013</ulink></bibtext> </blist> <blist> <bibl id="bib28" idref="ref130" type="bt">28</bibl> <bibtext>Fornell, C., & Larcker, D. F. ( 1981 ). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18 ( 1 ), 39 – 50. <ulink href="http://dx.doi.org/10.2307/3151312">http://dx.doi.org/10.2307/3151312</ulink></bibtext> </blist> <blist> <bibl id="bib29" idref="ref108" type="bt">29</bibl> <bibtext>Fraser, B. J. ( 1998 ). Classroom environment instruments: Development, validity and applications. Learning Environments Research, 1, 7 – 33. <ulink href="http://dx.doi.org/10.2307/3151312">http://dx.doi.org/10.2307/3151312</ulink></bibtext> </blist> <blist> <bibl id="bib30" idref="ref71" type="bt">30</bibl> <bibtext>Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. ( 2004 ). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74 ( 1 ), 59 – 109. <ulink href="http://dx.doi.org/10.3102/00346543074001059">http://dx.doi.org/10.3102/00346543074001059</ulink></bibtext> </blist> <blist> <bibl id="bib31" idref="ref11" type="bt">31</bibl> <bibtext>Friedrich, K., Sellers, S., & Burstyn, J. ( 2007 ). Thawing the chilly climate: Inclusive teaching resources for science, technology, engineering, and math. To Improve the Academy: Resources for Faculty, Instructional, and Organizational Development, 26, 133 – 144. </bibtext> </blist> <blist> <bibl id="bib32" idref="ref19" type="bt">32</bibl> <bibtext>Froyd, J., Borrego, M., Cutler, S., Prince, M., & Henderson, C. ( 2013 ). Estimates of use of research‐based instructional strategies in core electrical or computer engineering courses. IEEE Transactions on Education, 56 ( 4 ), 393 – 399. <ulink href="http://dx.doi.org/10.1109/TE.2013.2244602">http://dx.doi.org/10.1109/TE.2013.2244602</ulink></bibtext> </blist> <blist> <bibl id="bib33" idref="ref78" type="bt">33</bibl> <bibtext>Furlong, M. J., & Christenson, S. L. ( 2008 ). Engaging students with school and academics: A relevant construct for all students. Psychology in Schools, 45, 365 – 368. doi: 10.1002/pits.20303 </bibtext> </blist> <blist> <bibl id="bib34" idref="ref65" type="bt">34</bibl> <bibtext>Gaffney, J. D. H., Gaffney, A. L. H., & Beichner, R. J. ( 2010 ). Do they see it coming? Using expectancy violation to gauge the success of pedagogical reforms. Physical Review Special Topics – Physics Education Research, 6 ( 1 ). <ulink href="http://dx.doi.org/10.1103/PhysRevSTPER.6.010102">http://dx.doi.org/10.1103/PhysRevSTPER.6.010102</ulink></bibtext> </blist> <blist> <bibl id="bib35" idref="ref22" type="bt">35</bibl> <bibtext>Gauci, S. A., Dantas, A. M., Williams, D. A., & Kemm, R. E. ( 2009 ). Promoting student‐centered active learning in lectures with a personal response system. Advances in Physiology Education, 33 ( 1 ), 60 – 71. <ulink href="http://dx.doi.org/10.1152/advan.00109.2007">http://dx.doi.org/10.1152/advan.00109.2007</ulink></bibtext> </blist> <blist> <bibl id="bib36" idref="ref107" type="bt">36</bibl> <bibtext>Greene, J. C., Caracelli, V. J., & Graham, W. F. ( 1989 ). Toward a conceptual framework for mixed‐method evaluation designs. Educational evaluation and policy analysis, 11 ( 3 ), 255 – 274. <ulink href="http://dx.doi.org/10.2307/1163620">http://dx.doi.org/10.2307/1163620</ulink></bibtext> </blist> <blist> <bibl id="bib37" idref="ref129" type="bt">37</bibl> <bibtext>Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. ( 1992 ). Multivariate data analysis. New York, NY : Macmillan. </bibtext> </blist> <blist> <bibl id="bib38" idref="ref12" type="bt">38</bibl> <bibtext>Handelsman, J., Ebert‐May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J.,... Wood, W. B. ( 2004 ). Scientific teaching. Science, 304 ( 5670 ), 521 – 522. <ulink href="http://handelsmanlab.sites.yale.edu/sites/default/files/Scientific%20Teaching.pdf">http://handelsmanlab.sites.yale.edu/sites/default/files/Scientific%20Teaching.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib39" idref="ref101" type="bt">39</bibl> <bibtext>Haynes, S. N., Richard, D. C. S., & Kubany, E. S. ( 1995 ). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7 ( 3 ), 238 – 247. <ulink href="http://dx.doi.org/10.1037/1040‐3590.7.3.238">http://dx.doi.org/10.1037/1040‐3590.7.3.238</ulink></bibtext> </blist> <blist> <bibl id="bib40" idref="ref20" type="bt">40</bibl> <bibtext>Henderson, C., & Dancy, M. ( 2007 ). Barriers to the use of research‐based instructional strategies: The influence of both individual and situational characteristics. Physical Review Special Topics – Physics Education Research, 3 ( 2 ), 020102‐020101 to 020102‐020114. <ulink href="http://dx.doi.org/10.1103/PhysRevSTPER.3.020102">http://dx.doi.org/10.1103/PhysRevSTPER.3.020102</ulink></bibtext> </blist> <blist> <bibl id="bib41" idref="ref35" type="bt">41</bibl> <bibtext>Hinkin, T. R. ( 1998 ). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1 ( 1 ), 104 – 121. <ulink href="http://dx.doi.org/10.1177/109442819800100106">http://dx.doi.org/10.1177/109442819800100106</ulink></bibtext> </blist> <blist> <bibl id="bib42" idref="ref13" type="bt">42</bibl> <bibtext>Hora, M. T., Ferrare, J., & Oleson, A. ( 2012 ). Findings from classroom observations of 58 math and science faculty. Madison, WI : University of Wisconsin‐Madison, Wisconsin Center for Education Research. <ulink href="http://ccher.wceruw.org/documents/CCHER%5ffinal%20report%5f2012.pdf">http://ccher.wceruw.org/documents/CCHER%5ffinal%20report%5f2012.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib43" idref="ref127" type="bt">43</bibl> <bibtext>Hu, L., & Bentler, P. M. ( 1999 ). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1 – 55. <ulink href="http://dx.doi.org/10.1080/10705519909540118">http://dx.doi.org/10.1080/10705519909540118</ulink></bibtext> </blist> <blist> <bibl id="bib44" idref="ref53" type="bt">44</bibl> <bibtext>Hung, W., Bailey, J. H., & Jonassen, D. H. ( 2003 ). Exploring the tensions of problem‐based learning: Insights from research. New Directions for Teaching and Learning, 2003 ( 95 ), 13 – 23. <ulink href="http://dx.doi.org/10.1002/tl.108">http://dx.doi.org/10.1002/tl.108</ulink></bibtext> </blist> <blist> <bibl id="bib45" idref="ref42" type="bt">45</bibl> <bibtext>Ironson, G. H., Smith, P. C., Brannick, M. T., Gibson, W. M., & Paul, K. B. ( 1989 ). Construction of a Job in General scale: A comparison of global, composite, and specific measures. Journal of Applied Psychology, 74 ( 2 ), 193. <ulink href="http://dx.doi.org/10.1037/0021‐9010.74.2.193">http://dx.doi.org/10.1037/0021‐9010.74.2.193</ulink></bibtext> </blist> <blist> <bibl id="bib46" idref="ref3" type="bt">46</bibl> <bibtext>Jamieson, L. H., & Lohmann, J. R. ( 2012 ). Innovation with impact: Creating a culture for scholarly and systematic innovation in engineering education. Washington, DC : American Society for Engineering Education. https://<ulink href="http://www.asee.org/member‐resources/reports/Innovation‐with‐Impact/Innovation‐With‐Impact‐Report.pdf">www.asee.org/member‐resources/reports/Innovation‐with‐Impact/Innovation‐With‐Impact‐Report.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib47" idref="ref27" type="bt">47</bibl> <bibtext>Johnson, D. W., Johnson, R. T., & Smith, K. A. ( 1991 ). Active learning: Cooperation in the college classroom. Edina, MN : Interaction. </bibtext> </blist> <blist> <bibl id="bib48" idref="ref116" type="bt">48</bibl> <bibtext>Kaiser, H. F. ( 1958 ). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187 – 200. <ulink href="http://dx.doi.org/10.1007/BF02289233">http://dx.doi.org/10.1007/BF02289233</ulink></bibtext> </blist> <blist> <bibl id="bib49" idref="ref126" type="bt">49</bibl> <bibtext>Kline, R. B. ( 1998 ). Principles and practice of structural equation modeling. New York, NY : The Guilford Press. </bibtext> </blist> <blist> <bibl id="bib50" idref="ref122" type="bt">50</bibl> <bibtext>Kline, R. B. ( 2005 ). Principles and practice of structural equation modeling ( 2nd ed. ). New York, NY : The Guilford Press. </bibtext> </blist> <blist> <bibl id="bib51" idref="ref4" type="bt">51</bibl> <bibtext>Kuh, G. D. ( 2008 ). High‐impact educational practices: What they are, who has access to them, and why they matter. Washington, DC : Association of American Colleges and Universities. <ulink href="http://provost.tufts.edu/celt/files/High‐Impact‐Ed‐Practices1.pdf">http://provost.tufts.edu/celt/files/High‐Impact‐Ed‐Practices1.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib52" idref="ref28" type="bt">52</bibl> <bibtext>Lake, D. ( 2001 ). Student performance and perceptions of a lecture‐based course compared with the same course utilizing group discussion. Physical Therapy, 81, 896 – 902. </bibtext> </blist> <blist> <bibl id="bib53" idref="ref74" type="bt">53</bibl> <bibtext>Lawson, M. A., & Lawson, H. A. ( 2013 ). New conceptual frameworks for student engagement research, policy, and practice. Review of Educational Research, 83 ( 3 ), 432 – 479. <ulink href="http://dx.doi.org/10.3102/00346vv54313480891">http://dx.doi.org/10.3102/00346vv54313480891</ulink></bibtext> </blist> <blist> <bibl id="bib54" idref="ref91" type="bt">54</bibl> <bibtext>Lee, V. E., & Smith, J. B. ( 1995 ). Effects of high school restructuring and size on early gains in achievement and engagement. Sociology of Education, 68, 241 – 270. <ulink href="http://dx.doi.org/10.2307/2112741">http://dx.doi.org/10.2307/2112741</ulink></bibtext> </blist> <blist> <bibl id="bib55" idref="ref33" type="bt">55</bibl> <bibtext>Li, Q., McCoach, D. B., Swaminathan, H., & Tang, J. ( 2008 ). Development of an instrument to measure perspectives of engineering education among college students. Journal of Engineering Education, 97 ( 1 ), 47 – 56. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2008.tb00953.x">http://dx.doi.org/10.1002/j.2168‐9830.2008.tb00953.x</ulink></bibtext> </blist> <blist> <bibl id="bib56" idref="ref23" type="bt">56</bibl> <bibtext>Livingstone, D., & Lynch, K. ( 2000 ). Group project work and student‐centred active learning: Two different experiences. Studies in Higher education, 25 ( 3 ), 325 – 345. <ulink href="http://dx.doi.org/10.1080/713696161">http://dx.doi.org/10.1080/713696161</ulink></bibtext> </blist> <blist> <bibl id="bib57" idref="ref123" type="bt">57</bibl> <bibtext>Martens, M. P. ( 2005 ). The use of structural equation modeling in counseling psychology research. The Counseling Psychologist, 33 ( 3 ), 269 – 298. <ulink href="http://dx.doi.org/10.1177/0011000004272260">http://dx.doi.org/10.1177/0011000004272260</ulink></bibtext> </blist> <blist> <bibl id="bib58" idref="ref124" type="bt">58</bibl> <bibtext>Martens, M. P., & Hasse, R. F. ( 2006 ). Advanced applications of structural equation modeling in counseling psychology research. The Counseling Psychologist, 34, 878 – 911. <ulink href="http://dx.doi.org/10.1177/0011000005283395">http://dx.doi.org/10.1177/0011000005283395</ulink></bibtext> </blist> <blist> <bibl id="bib59" idref="ref29" type="bt">59</bibl> <bibtext>Michael, J. ( 2007 ). Faculty perceptions about barriers to active learning. College Teaching, 54 ( 1 ), 42 – 47. <ulink href="http://dx.doi.org/10.3200/CTCH.55.2.42‐47">http://dx.doi.org/10.3200/CTCH.55.2.42‐47</ulink></bibtext> </blist> <blist> <bibl id="bib60" idref="ref30" type="bt">60</bibl> <bibtext>Moffett, B. S., & Hill, K. B. ( 1997 ). The transition to active learning: A lived experience. Nurse Educator, 22 ( 4 ), 44 – 47. <ulink href="http://dx.doi.org/10.1097/00006223‐199707000‐00015">http://dx.doi.org/10.1097/00006223‐199707000‐00015</ulink></bibtext> </blist> <blist> <bibl id="bib61" idref="ref2" type="bt">61</bibl> <bibtext>National Academy of Sciences, National Academy of Engineering, & Institute of Medicine (NAS, NAE, & IOM). ( 2007 ). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Washington, DC. https://<ulink href="http://www.utsystem.edu/competitive/files/RAGS‐fullreport.pdf">www.utsystem.edu/competitive/files/RAGS‐fullreport.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib62" idref="ref83" type="bt">62</bibl> <bibtext>Newmann, F., Wehlage, G. G., & Lamborn, S. D. ( 1992 ). The significance and sources of student engagement. In F. Newmann (Ed.), Student engagement and achievement in American secondary schools (pp. 11 – 39 ). New York, NY : Teachers College Press. </bibtext> </blist> <blist> <bibl id="bib63" idref="ref102" type="bt">63</bibl> <bibtext>Nunnally, J. C., & Bernstein, I. H. ( 1994 ). The assessment of reliability. Psychometric Theory, 3, 248 – 292. </bibtext> </blist> <blist> <bibl id="bib64" idref="ref49" type="bt">64</bibl> <bibtext>Oakley, B. A., Hanna, D. M., Kuzmyn, A., & Felder, R. M. ( 2007 ). Best practices involving teamwork in the classroom: Results from a survey of 6435 engineering student respondents. IEEE Transactions on Education, 50 ( 3 ), 266 – 272. <ulink href="http://dx.doi.org/10.1109/TE.2007.901982">http://dx.doi.org/10.1109/TE.2007.901982</ulink></bibtext> </blist> <blist> <bibl id="bib65" idref="ref109" type="bt">65</bibl> <bibtext>Ouimet, J. A., Bunnage, J. C., Carini, R. M., Kuh, G. D., & Kennedy, J. ( 2004 ). Using focus groups, expert advice, and cognitive interviews to establish the validity of a college student survey. Research in Higher Education, 45 ( 3 ), 233 – 250. <ulink href="http://dx.doi.org/10.1023/B:RIHE.0000019588.05470.78">http://dx.doi.org/10.1023/B:RIHE.0000019588.05470.78</ulink></bibtext> </blist> <blist> <bibl id="bib66" idref="ref92" type="bt">66</bibl> <bibtext>Pekrun, R., Elliot, A. J., & Maier, M. A. ( 2009 ). Achievement goals and achievement emotions: Testing a model of their joint relations with academic performance. Journal of Educational Psychology, 101 ( 1 ), 115 – 135. </bibtext> </blist> <blist> <bibl id="bib67" idref="ref90" type="bt">67</bibl> <bibtext>Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. ( 2002 ). Academic emotions in students' self‐regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37 ( 2 ), 91 – 105. </bibtext> </blist> <blist> <bibl id="bib68" idref="ref75" type="bt">68</bibl> <bibtext>Pekrun, R., & Linnenbrink‐Garcia, L. ( 2012 ). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 259 – 282 ). New York, NY : Springer. <ulink href="http://dx.doi.org/10.1007/978‐1‐4614‐2018‐7%5f12">http://dx.doi.org/10.1007/978‐1‐4614‐2018‐7%5f12</ulink></bibtext> </blist> <blist> <bibl id="bib69" idref="ref86" type="bt">69</bibl> <bibtext>Pintrich, P. R., & De Groot, E. ( 1990 ). Motivated and self‐regulated learning components of academic performance. Journal of Educational Psychology, 82, 33 – 40. <ulink href="http://rhartshorne.com/fall‐2012/eme6507‐rh/cdisturco/eme6507‐eportfolio/documents/pintrich%20and%20degroodt%201990.pdf">http://rhartshorne.com/fall‐2012/eme6507‐rh/cdisturco/eme6507‐eportfolio/documents/pintrich%20and%20degroodt%201990.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib70" idref="ref50" type="bt">70</bibl> <bibtext>Powell, K. C., & Kalina, C. J. ( 2009 ). Cognitive and social constructivism: Developing tools for an effective classroom. Education, 130 ( 2 ), 241. </bibtext> </blist> <blist> <bibl id="bib71" idref="ref14" type="bt">71</bibl> <bibtext>President's Council of Advisors on Science and Technology (PCAST). ( 2012 ). Report to the president: Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics. Washington, DC : Executive Office of the President. https://<ulink href="http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast‐executive‐report‐final%5f2‐13‐12.pdf">www.whitehouse.gov/sites/default/files/microsites/ostp/pcast‐executive‐report‐final%5f2‐13‐12.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib72" idref="ref7" type="bt">72</bibl> <bibtext>Prince, M. ( 2004 ). Does active learning work? A review of the research. Journal of Engineering Education, 93 ( 3 ), 223 – 232. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2004.tb00809.x">http://dx.doi.org/10.1002/j.2168‐9830.2004.tb00809.x</ulink></bibtext> </blist> <blist> <bibl id="bib73" idref="ref31" type="bt">73</bibl> <bibtext>Prince, M., Borrego, M., Henderson, C., Cutler, S., & Froyd, J. ( 2013 ). Use of research‐based instructional strategies in core chemical engineering courses. Chemical Engineering Education, 47 ( 1 ), 27 – 37. <ulink href="http://homepages.wmich.edu/∼chenders/Publications/2013PrinceCEEUseofRBIS.pdf">http://homepages.wmich.edu/∼chenders/Publications/2013PrinceCEEUseofRBIS.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib74" idref="ref10" type="bt">74</bibl> <bibtext>Prince, M. J., & Felder, R. M. ( 2006 ). Inductive teaching and learning methods: Definitions, comparisons, and research bases. Journal of Engineering Education, 95 ( 2 ), 123 – 138. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2006.tb00884.x">http://dx.doi.org/10.1002/j.2168‐9830.2006.tb00884.x</ulink></bibtext> </blist> <blist> <bibl id="bib75" idref="ref113" type="bt">75</bibl> <bibtext>Reise, S. P., Waller, N. G., & Comrey, A. L. ( 2000 ). Factor analysis and scale revision. Psychological Assessment, 12, 287 – 297. <ulink href="http://dx.doi.org/10.1037/1040‐3590.12.3.287">http://dx.doi.org/10.1037/1040‐3590.12.3.287</ulink></bibtext> </blist> <blist> <bibl id="bib76" idref="ref34" type="bt">76</bibl> <bibtext>Ro, H. K., Merson, D., Lattuca, L. R., & Terenzini, P. T. ( 2015 ). Validity of the contextual competence scale for engineering students. Journal of Engineering Education, 104 ( 1 ), 35 – 54. <ulink href="http://dx.doi.org/10.1002/jee.20062">http://dx.doi.org/10.1002/jee.20062</ulink></bibtext> </blist> <blist> <bibl id="bib77" idref="ref112" type="bt">77</bibl> <bibtext>Rubin, D. B. ( 1976 ). Inference and missing data. Biometrika, 63 ( 3 ), 581 – 592. <ulink href="http://dx.doi.org/10.1093/biomet/63.3.581">http://dx.doi.org/10.1093/biomet/63.3.581</ulink></bibtext> </blist> <blist> <bibl id="bib78" idref="ref21" type="bt">78</bibl> <bibtext>Seidel, S. B., & Tanner, K. D. ( 2013 ). What if students revolt? – Considering student resistance: Origins, options, and opportunities for investigation. CBE‐Life Sciences Education, 12 ( 4 ), 586 – 595. <ulink href="http://dx.doi.org/10.1187/cbe‐13‐09‐0190">http://dx.doi.org/10.1187/cbe‐13‐09‐0190</ulink></bibtext> </blist> <blist> <bibl id="bib79" idref="ref5" type="bt">79</bibl> <bibtext>Seymour, E., & Hewitt, N. ( 1997 ). Talking about leaving. Boulder, CO : Westview Press. </bibtext> </blist> <blist> <bibl id="bib80" idref="ref70" type="bt">80</bibl> <bibtext>Shekhar, P., DeMonbrun, M., Borrego, M., Finelli, C., Prince, M., Henderson, C., & Waters, C. ( 2015 ). Development of an observation protocol to study undergraduate engineering student resistance to active learning. International Journal of Engineering Education, 31 ( 2 ), 597 – 609. <ulink href="http://homepages.wmich.edu/∼chenders/Publications/2015ShekharIJEE.pdf">http://homepages.wmich.edu/∼chenders/Publications/2015ShekharIJEE.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib81" idref="ref15" type="bt">81</bibl> <bibtext>Singer, S. R., Nielsen, N. R., & Schweingruber, H. A. (Eds.). ( 2012 ). Discipline‐based education research: Understanding and improving learning in undergraduate science and engineering. Washington, DC : National Academies Press. <ulink href="http://eduinfo.cchem.berkeley.edu/pdf/NAPunderstandingandimprovinglearning.pdf">http://eduinfo.cchem.berkeley.edu/pdf/NAPunderstandingandimprovinglearning.pdf</ulink></bibtext> </blist> <blist> <bibl id="bib82" idref="ref89" type="bt">82</bibl> <bibtext>Skinner, E. A., & Belmont, M. J. ( 1993 ). Motivation in the classroom: Reciprocal effect of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85, 571 – 581. <ulink href="http://dx.doi.org/10.1037/0022‐0663.85.4.571">http://dx.doi.org/10.1037/0022‐0663.85.4.571</ulink></bibtext> </blist> <blist> <bibl id="bib83" idref="ref79" type="bt">83</bibl> <bibtext>Skinner, E. A., & Pitzer, J. R. ( 2012 ). Developmental dynamics of student engagement, coping, and everyday resilience. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 21 – 44 ). New York, NY : Springer. <ulink href="http://dx.doi.org/10.1007/978‐1‐4614‐2018‐7%5f2">http://dx.doi.org/10.1007/978‐1‐4614‐2018‐7%5f2</ulink></bibtext> </blist> <blist> <bibl id="bib84" idref="ref93" type="bt">84</bibl> <bibtext>Stipek, D. ( 2002 ). Good instruction is motivating. In A. Wigfield & J. Eccles (Eds.), Development of achievement motivation (pp. 309 – 332 ). San Diego, CA : Academic Press. <ulink href="http://dx.doi.org/10.1016/B978‐012750053‐9/50014‐0">http://dx.doi.org/10.1016/B978‐012750053‐9/50014‐0</ulink></bibtext> </blist> <blist> <bibl id="bib85" idref="ref117" type="bt">85</bibl> <bibtext>Tabachnick, B. G., & Fidell, L. S. ( 2001 ). Using multivariate statistics ( 4th ed. ). New York, NY : Harper & Row. </bibtext> </blist> <blist> <bibl id="bib86" idref="ref111" type="bt">86</bibl> <bibtext>Teddlie, C., & Yu, F. ( 2007 ). Mixed methods sampling a typology with examples. Journal of Mixed Methods Research, 1 ( 1 ), 77 – 100. <ulink href="http://dx.doi.org/10.1177/2345678906292430">http://dx.doi.org/10.1177/2345678906292430</ulink></bibtext> </blist> <blist> <bibl id="bib87" idref="ref114" type="bt">87</bibl> <bibtext>Thompson, B. ( 2004 ). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC : American Psychological Association. <ulink href="http://dx.doi.org/10.1037/10694‐000">http://dx.doi.org/10.1037/10694‐000</ulink></bibtext> </blist> <blist> <bibl id="bib88" idref="ref54" type="bt">88</bibl> <bibtext>Van Barneveld, A., & Strobel, J. ( 2011 ). Reports from teaching practice: Experiences and management of tensions encountered with PBL implementations in the early years of undergraduate engineering education. Paper presented at the Research in Engineering Education Symposium, Madrid, Spain. </bibtext> </blist> <blist> <bibl id="bib89" idref="ref84" type="bt">89</bibl> <bibtext>Wehlage, G. G., Rutter, R. A., Smith, G. A., Lesko, N. L., & Fernandez, R. R. ( 1989 ). Reducing the risk: Schools as communities of support. Philadelphia, PA: Farmer Press. </bibtext> </blist> <blist> <bibl id="bib90" idref="ref73" type="bt">90</bibl> <bibtext>Weimer, M. ( 2002 ). Learner‐centered teaching: Five key changes to practice. San Francisco, CA : Jossey‐Bass. </bibtext> </blist> <blist> <bibl id="bib91" idref="ref110" type="bt">91</bibl> <bibtext>Willis, G. B. ( 2004 ). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA : Sage. <ulink href="http://dx.doi.org/10.4135/9781412983655">http://dx.doi.org/10.4135/9781412983655</ulink></bibtext> </blist> <blist> <bibl id="bib92" idref="ref115" type="bt">92</bibl> <bibtext>Worthington, R. L., & Whittaker, T. A. ( 2006 ). Scale development research a content analysis and recommendations for best practices. The Counseling Psychologist, 34 ( 6 ), 806 – 838. <ulink href="http://dx.doi.org/10.1177/0011000006288127">http://dx.doi.org/10.1177/0011000006288127</ulink></bibtext> </blist> <blist> <bibl id="bib93" idref="ref55" type="bt">93</bibl> <bibtext>Yadav, A., Subedi, D., Lunderberg, M., & Bunting, C. ( 2011 ). Problem‐based Learning: Influence on Students' Learning in an Electrical Engineering Course. Journal of Engineering Education, 100, 253 – 280. <ulink href="http://dx.doi.org/10.1002/j.2168‐9830.2011.tb00013.x">http://dx.doi.org/10.1002/j.2168‐9830.2011.tb00013.x</ulink></bibtext> </blist> <blist> <bibl id="bib94" idref="ref87" type="bt">94</bibl> <bibtext>Zimmerman, B. J. ( 1990 ). Self‐regulated learning and academic achievement: An overview. Educational Psychologist, 21, 3 – 17. <ulink href="http://dx.doi.org/10.1207/s15326985ep2501%5f2">http://dx.doi.org/10.1207/s15326985ep2501%5f2</ulink></bibtext> </blist> </ref> <p>Graph: The StRIP instrument development process (adapted from Hinkin, 41 ).</p> <p>Graph: A framework for explaining students' responses to types of instruction. Items with asterisks are measured using other instruments not discussed here.</p> <aug> <p>By Matt DeMonbrun; Cynthia J. Finelli; Michael Prince; Maura Borrego; Prateek Shekhar; Charles Henderson and Cindy Waters</p> <p></p> <p>Matt DeMonbrun is a doctoral candidate in the School of Education at the University of Michigan, 610 East University, Ann Arbor, MI, 48109;.</p> <p>Cynthia J. Finelli is an associate professor of electrical engineering and computer science, associate professor of education, and Director of the Engineering Education Research Program at the University of Michigan, 4227 EECS Building, 1301 Beal Avenue, Ann Arbor, MI, 48109;.</p> <p>Michael Prince is a professor of chemical engineering at Bucknell University, Lewisburg, PA, 17837;.</p> <p>Maura Borrego is an associate professor of mechanical engineering and STEM education at the University of Texas at Austin, 204 E. Dean Keeton Street, Stop C2200, ETC II 5.160, Austin, TX, 78712;.</p> <p>Prateek Shekhar is a post‐doctoral research fellow in the Department of Biomedical Engineering at the University of Michigan, 1101 Beal Avenue, Ann Arbor, MI, 48109;.</p> <p>Charles Henderson is a professor of physics at Western Michigan University, 1903 W. Michigan Ave., Kalamazoo, MI, 49008;.</p> <p>Cindy Waters is an assistant professor of mechanical engineering at North Carolina A&T State University, 1601 E. Market Street, McNair Hall Room 614, Greensboro, NC, 27411;.</p> </aug>
Header DbId: eric
DbLabel: ERIC
An: EJ1254473
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Creating an Instrument to Measure Student Response to Instructional Practices
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22DeMonbrun%2C+Matt%22">DeMonbrun, Matt</searchLink><br /><searchLink fieldCode="AR" term="%22Finelli%2C+Cynthia+J%2E%22">Finelli, Cynthia J.</searchLink><br /><searchLink fieldCode="AR" term="%22Prince%2C+Michael%22">Prince, Michael</searchLink><br /><searchLink fieldCode="AR" term="%22Borrego%2C+Maura%22">Borrego, Maura</searchLink><br /><searchLink fieldCode="AR" term="%22Shekhar%2C+Prateek%22">Shekhar, Prateek</searchLink><br /><searchLink fieldCode="AR" term="%22Henderson%2C+Charles%22">Henderson, Charles</searchLink><br /><searchLink fieldCode="AR" term="%22Waters%2C+Cindy%22">Waters, Cindy</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Engineering+Education%22"><i>Journal of Engineering Education</i></searchLink>. Apr 2017 106(2):273-298.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley Periodicals, Inc. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 26
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2017
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 1347417<br />1347482<br />1347580<br />1347718<br />1500309
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research<br />Tests/Questionnaires
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Attitude+Measures%22">Attitude Measures</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Attitudes%22">Student Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Reaction%22">Student Reaction</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Factor+Analysis%22">Factor Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/jee.20162
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1069-4730
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Calls for the reform of education in science, technology, engineering, and mathematics (STEM) have inspired many instructional innovations, some research based. Yet adoption of such instruction has been slow. Research has suggested that students' response may significantly affect an instructor's willingness to adopt different types of instruction. Purpose: We created the Student Response to Instructional Practices (StRIP) instrument to measure the effects of several variables on student response to instructional practices. We discuss the step-by-step process for creating this instrument. Design/Method: The development process had six steps--item generation and construct development, validity testing, implementation, exploratory factor analysis, confirmatory factor analysis, and instrument modification and replication. We discuss pilot testing of the initial instrument, construct development, and validation using exploratory and confirmatory factor analyses. Results: This process produced 47 items measuring three parts of our framework. Types of instruction separated into four factors (interactive, constructive, active, and passive); strategies for using in-class activities into two factors (explanation and facilitation); and student responses to instruction into five factors (value, positivity, participation, distraction, and evaluation). Conclusions: We describe the design process and final results for our instrument, a useful tool for understanding the relationship between type of instruction and students' response.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1254473
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1254473
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/jee.20162
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 26
        StartPage: 273
    Subjects:
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Attitude Measures
        Type: general
      – SubjectFull: Student Attitudes
        Type: general
      – SubjectFull: Student Reaction
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Factor Analysis
        Type: general
      – SubjectFull: Test Validity
        Type: general
    Titles:
      – TitleFull: Creating an Instrument to Measure Student Response to Instructional Practices
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: DeMonbrun, Matt
      – PersonEntity:
          Name:
            NameFull: Finelli, Cynthia J.
      – PersonEntity:
          Name:
            NameFull: Prince, Michael
      – PersonEntity:
          Name:
            NameFull: Borrego, Maura
      – PersonEntity:
          Name:
            NameFull: Shekhar, Prateek
      – PersonEntity:
          Name:
            NameFull: Henderson, Charles
      – PersonEntity:
          Name:
            NameFull: Waters, Cindy
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2017
          Identifiers:
            – Type: issn-print
              Value: 1069-4730
          Numbering:
            – Type: volume
              Value: 106
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Journal of Engineering Education
              Type: main
ResultId 1