View in EDS HTML Full Text PDF Full Text

Opportunities for Educational Innovations in Authentic Project-Based Learning: Understanding Instructor Perceived Challenges to Design for Adoption

Saved in:
Bibliographic Details
Title: Opportunities for Educational Innovations in Authentic Project-Based Learning: Understanding Instructor Perceived Challenges to Design for Adoption
Language: English
Authors: Rees Lewis, Daniel G. (ORCID 0000-0002-0928-3831), Gerber, Elizabeth M., Carlson, Spencer E., Easterday, Matthew W.
Source: Educational Technology Research and Development. Aug 2019 67(4):953-982.
Availability: Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 30
Publication Date: 2019
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: IIS1530833
IIS1320693
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Educational Innovation, Active Learning, Student Projects, Educational Technology, Teacher Attitudes, Adoption (Ideas), Design, Barriers, College Faculty, College Students, Educational Opportunities
DOI: 10.1007/s11423-019-09673-4
ISSN: 1042-1629
Abstract: Authentic project-based learning (APBL) is a highly effective way for instructors to help students learn disciplinary skills, modes of thinking, and collaborative practices by creating solutions to real-world problems for real users and clients. While educational technology innovations can bolster APBL by making a promising but challenging pedagogy more effective, as with many areas of education instructor adoption is slow. Diffusion of innovations theory predicts that instructors will adopt and maintain their use of innovations if innovations are perceived to, and then do, address their challenges. To guide design of future APBL technologies, we interviewed 47 university APBL instructors about their most significant challenges and inductively analyzed the resulting interview transcripts. APBL instructors reported interrelated challenges of: (a) "scoping," sourcing challenges and balancing the needs of the program, students, and clients; (b) "curriculum" preparation, making the curriculum flexible enough for shifting project problems and codify standards to help students understand how to do quality work; (c) providing "assistance" to teams, including monitoring, and delivering assistance; and (d) "coordinating" a range of stakeholders involved in assisting teams, including co-instructors, clients, and students. To support instructor adoption in APBL, educational technology innovators might communicate existing technology, or create technological innovations, that provide: (a) scoping tools for sourcing projects, and forming teams; (b) authoring tools for sharing and remixing of curricular materials; (c) project management tools for team management and monitoring; and (d) coordination software to manage all APBL stakeholders.
Abstractor: As Provided
Entry Date: 2019
Accession Number: EJ1221426
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFA9RbJswbxo1ElLuoZQy5rAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJMwlrcpznAJW-9MVwIBEICBmutnkkChEiD4akgxVNOIPkqjJbVavqpzlVNtTmudPdYn6TQTZ5FEQO0749Xqs_-1m9UCrRVVNxBkPBRxPWyIvsHAf0rWYZlCdU6LIrWAUsQrxhV14Csveff6sS_NRAU9wOubvPeCnVWO2--Khe5TNC4owJqiP7td-E4C-iWx2QFyDpbTPeVZ2cqdADk9nTwsqjeiQEQ_DX-ZQN0=
Text:
  Availability: 1
  Value: <anid>AN0137339107;etr01aug.19;2019Jul08.03:11;v2.2.500</anid> <title id="AN0137339107-1">Opportunities for educational innovations in authentic project-based learning: understanding instructor perceived challenges to design for adoption </title> <p>Authentic project-based learning (APBL) is a highly effective way for instructors to help students learn disciplinary skills, modes of thinking, and collaborative practices by creating solutions to real-world problems for real users and clients. While educational technology innovations can bolster APBL by making a promising but challenging pedagogy more effective, as with many areas of education instructor adoption is slow. Diffusion of innovations theory predicts that instructors will adopt and maintain their use of innovations if innovations are perceived to, and then do, address their challenges. To guide design of future APBL technologies, we interviewed 47 university APBL instructors about their most significant challenges and inductively analyzed the resulting interview transcripts. APBL instructors reported interrelated challenges of: (a) scoping, sourcing challenges and balancing the needs of the program, students, and clients; (b) curriculum preparation, making the curriculum flexible enough for shifting project problems and codify standards to help students understand how to do quality work; (c) providing assistance to teams, including monitoring, and delivering assistance; and (d) coordinating a range of stakeholders involved in assisting teams, including co-instructors, clients, and students. To support instructor adoption in APBL, educational technology innovators might communicate existing technology, or create technological innovations, that provide: (a) scoping tools for sourcing projects, and forming teams; (b) authoring tools for sharing and remixing of curricular materials; (c) project management tools for team management and monitoring; and (d) coordination software to manage all APBL stakeholders.</p> <p>Keywords: Project-based learning; Diffusion of innovations; Innovation; Qualitative; Interview</p> <p>"When felt needs are met, a faster rate of adoption usually occurs" Rogers, 2003, p. 246.</p> <hd id="AN0137339107-2">Introduction</hd> <p>We as educational technologists often create innovations that are not adopted, despite a plethora of effective educational technology innovations (e.g. Archer et al. [<reflink idref="bib1" id="ref1">1</reflink>]; Tatar et al. [<reflink idref="bib64" id="ref2">64</reflink>]), and an increasing willingness and preparedness by educational institutions to implement technology (Cheung and Slavin [<reflink idref="bib12" id="ref3">12</reflink>]; Johnson et al. [<reflink idref="bib30" id="ref4">30</reflink>]). Diffusion of innovation theory predicts that we can increase the adoption rate of our innovations if instructors perceive our innovations solve their challenges (Rogers [<reflink idref="bib54" id="ref5">54</reflink>]). Educational technologists seek long term classroom adoption, or integration, of educational technology (de Koster et al. [<reflink idref="bib16" id="ref6">16</reflink>]; ISTE [<reflink idref="bib28" id="ref7">28</reflink>])—we want instructors to use technology in regular classroom practices to support learning and maintained use over time. To achieve wide-scale long-term adoption, diffusion of innovations literature stresses that we should understand the challenges instructors perceive in their classroom practices, design technologies that help instructors overcome these challenges, and communicate these benefits (Rogers [<reflink idref="bib54" id="ref8">54</reflink>]; Zaritsky et al. [<reflink idref="bib70" id="ref9">70</reflink>]). Designers of educational technology must balance the needs of multiple stakeholders (Easterday et al. [<reflink idref="bib20" id="ref10">20</reflink>]), particularly those responsible for adopting technology in a classroom; consequently, a sound starting point for promoting adoption of educational innovations is an empirically grounded framework describing instructor perceived challenges to inform design and communication.</p> <p>In this study, we investigate opportunities for educational technology innovators to design for Authentic Project-based Learning (APBL) in higher education. We investigate the challenges instructors experience in APBL—itself an effective innovative pedagogical approach (Frank and Barzilai [<reflink idref="bib23" id="ref11">23</reflink>]; Prince and Felder [<reflink idref="bib47" id="ref12">47</reflink>])—to inform innovations that can be adopted and integrated into existing APBL classrooms; in this paper we do not address the adoption of APBL, but rather the adoption of technology for APBL. We focus on APBL instructors in higher education because they are implementing a demanding state-of-the-art pedagogical approach and have significant autonomy to decide what innovations to adopt (Prince and Felder [<reflink idref="bib47" id="ref13">47</reflink>]). As such we argue APBL is ripe for technological innovation.</p> <p>In APBL, student teams work with real users and clients to create a solution (idea, service, product) to a real-world problem with the opportunity of implementing the solution. Effective APBL mirrors professional practice by having students work in teams using professional practices and problems (Dym et al. [<reflink idref="bib19" id="ref14">19</reflink>]; Shaffer and Resnick [<reflink idref="bib56" id="ref15">56</reflink>]). Such practices include giving students experience with teamwork and iterating on solutions to complex problems. While APBL provides tremendous learning experiences for students, the additional complexity caused by real-world problems (Jonassen and Hung [<reflink idref="bib31" id="ref16">31</reflink>]) could create additional challenges for instructors; technology is well placed to address these challenges.</p> <hd id="AN0137339107-3">Background</hd> <p></p> <hd id="AN0137339107-4">Authentic project-based learning</hd> <p>APBL is a pedagogical approach that prepares students to solve real-world highly ill-structured problems (Dunlap [<reflink idref="bib17" id="ref17">17</reflink>]; Dym et al.[<reflink idref="bib19" id="ref18">19</reflink>]; Frank and Barzilai [<reflink idref="bib23" id="ref19">23</reflink>]; Prince and Felder [<reflink idref="bib47" id="ref20">47</reflink>]). APBL is a type of project-based learning which incorporates a real client, real users, and the opportunity for students to implement their solution. APBL involves addressing a non-trivial project for the client, so involves a "big project" approach (Boling et al. [<reflink idref="bib6" id="ref21">6</reflink>]) in which student teams take on one project spanning the length of the class. We illustrate APBL with a project from an engineering design class: an APBL student team worked with a disability advocacy group (client), on the issue of wheelchair damages in the hold of passenger aircraft (Carlson et al. [<reflink idref="bib8" id="ref22">8</reflink>]; Rees Lewis et al. [<reflink idref="bib50" id="ref23">50</reflink>]). The team designed a number of prototypes that they iteratively tested with wheelchair users and baggage handlers. The team produced a set of graphical stickers that directed baggage handlers to practice safer loading techniques, and a website to distribute the stickers to wheelchair users.</p> <p>In APBL the project is motivated by a driving question that describes a client problem (Gerber [<reflink idref="bib24" id="ref24">24</reflink>]). In other forms of project-based learning, instructors might generate the driving question that can be feasibly answered within the classroom, such as investigating science phenomenon (Krajcik and Czerniak [<reflink idref="bib34" id="ref25">34</reflink>]). Driving questions in APBL encourage students to take into account multiple additional constraints such as user needs, legal, and technological constraints. For example, in the wheelchair breakages team the driving question came from the client: how can we reduce serious damage to wheelchairs in the hold of passenger aircraft? The team had to take into account real user needs (baggage handlers, wheelchair users), and airport policy, federal aviation laws, and the technological challenges of creating solutions. The team activities, included qualitative needs analysis, planning, service design, graphic design, product design, web-design, prototype testing, and client management. This integrating of different skills and dealing with complex highly ill-structured problems is consistent with authentic practitioner problems (Jonassen et al. [<reflink idref="bib32" id="ref26">32</reflink>]).</p> <p>APBL supports learning real-world practices and increases student retention. Compared to lecture-based classes, APBL can better prepare students for professional work and promote greater learning of conceptual knowledge and teamwork (Dutson et al. [<reflink idref="bib18" id="ref27">18</reflink>]; Olds and Miller [<reflink idref="bib44" id="ref28">44</reflink>]; Strobel and Van Barneveld [<reflink idref="bib62" id="ref29">62</reflink>]; Yadav et al. [<reflink idref="bib69" id="ref30">69</reflink>]). In meta–meta-review of APBL in higher education classes in business, medicine, and engineering Prince and Felder ([<reflink idref="bib47" id="ref31">47</reflink>]) found that, compared to lecture-based classes, students performed comparably or better in assessments of content knowledge, and significantly better on assessments of conceptual understanding, problem-solving, and metacognitive skills. Furthermore, APBL can increase undergraduate retention, especially for underrepresented groups (Hoit and Ohland[<reflink idref="bib27" id="ref32">27</reflink>]; Olds and Miller [<reflink idref="bib44" id="ref33">44</reflink>]).</p> <p>While APBL is promising, teaching APL is highly challenging for instructors. APBL problems are highly ill-structured, meaning that the students, instructors, and clients do not know the 'answer' to the question (Jonassen and Hung [<reflink idref="bib31" id="ref34">31</reflink>]). Furthermore, mirroring real-world problem solving, each problem involves multiple unknowns for students and instructors, such as stakeholder needs, legal, technical, or social constraints (Jonassen et al. [<reflink idref="bib32" id="ref35">32</reflink>]; Goel and Pirolli [<reflink idref="bib25" id="ref36">25</reflink>]). Given APBL is simultaneously highly promising and demanding, we argue that this context is ripe for creating and diffusing educational technology innovations. The goal of this work is to help innovators make decisions about what to create. We build on expert opinion (e.g. Dym et al. [<reflink idref="bib19" id="ref37">19</reflink>]) and single-context empirical studies (e.g. Hadim and Esche [<reflink idref="bib26" id="ref38">26</reflink>]), to provide empirical research on what APBL instructors experience as their most pressing challenges.</p> <p>If we want to design and diffuse APBL educational technology innovations, we must understand APBL instructors' perceived challenges and tradeoffs. Therefore, we ask: <emph>What do APBL instructors in higher education perceive as their greatest challenges?</emph></p> <hd id="AN0137339107-5">Diffusion of innovations and uncovering challenges to inform innovation</hd> <p>Innovations in educational research often fail to diffuse widely in education (Borrego et al. [<reflink idref="bib7" id="ref39">7</reflink>]; Ertmer and Simons [<reflink idref="bib22" id="ref40">22</reflink>]). Diffusion efforts are often hampered when instructors do not perceive innovations as useful (Niederhauser and Stoddart [<reflink idref="bib43" id="ref41">43</reflink>]), and the innovations fail to support valued existing instructional practices (Blumenfeld et al. [<reflink idref="bib5" id="ref42">5</reflink>]). Institutional support does not guarantee that teachers will adopt innovations within classrooms (Penuel and Spillane [<reflink idref="bib46" id="ref43">46</reflink>]), especially when innovations do not support instructional practices that instructors value. Even when teachers do adopt a given innovation, adoption may be superficial to appease administrative pressure rather than be fully integrated into important classroom practices (Spillane et al. [<reflink idref="bib60" id="ref44">60</reflink>]; Vanderlinde and van Braak [<reflink idref="bib66" id="ref45">66</reflink>]), so fail to support teaching and learning.</p> <p>Innovators can increase the rate of adoption by first understanding adoptees needs, then designing for—and communicating how the innovation meets—these needs (Rogers [<reflink idref="bib54" id="ref46">54</reflink>]). Rogers' ([<reflink idref="bib54" id="ref47">54</reflink>]) diffusion of innovations theory posits adoption as the process of the adoptee (<reflink idref="bib1" id="ref48">1</reflink>) gaining awareness of the innovation, (<reflink idref="bib2" id="ref49">2</reflink>) deciding that the innovation will meet their needs, (<reflink idref="bib3" id="ref50">3</reflink>) implementing the innovation, and (<reflink idref="bib4" id="ref51">4</reflink>) re-evaluating the innovation positively after implementation (Rogers [<reflink idref="bib54" id="ref52">54</reflink>]). Drawing on the diffusion of innovations we conceptualize a perceived challenge as an unmet need that a stakeholder experiences within a given role. People adopt innovations when they perceive that it offers a relative advantage over the status quo, and is compatible with their needs (Zaritsky et al. [<reflink idref="bib70" id="ref53">70</reflink>]). Research in educational technology is consistent with diffusion of innovations theory: perceived usefulness predicts instructor's adoption of innovations (Matusovich et al. [<reflink idref="bib40" id="ref54">40</reflink>]; Sadaf et al. [<reflink idref="bib55" id="ref55">55</reflink>]). Similarly, practitioner methods advocate for capturing adoptees perceived challenges to inform innovations in areas such as software design (Sutherland and Sutherland [<reflink idref="bib63" id="ref56">63</reflink>]), product design (Beyer and Holtzblatt [<reflink idref="bib2" id="ref57">2</reflink>]), and organization design (Blank and Dorf [<reflink idref="bib4" id="ref58">4</reflink>]).</p> <p>We focus on the adoption of new innovations within ABPL by instructors in higher education, who typically have significant pedagogical autonomy (Matusovich et al. [<reflink idref="bib40" id="ref59">40</reflink>]). We are not focusing on adoption of wholesale pedagogical approaches such as PBL for a whole department, or program. Rather, we focus on opportunities for the adoption of technology and practices that are complementary to the larger state-of-the-art pedagogical approach of ABPL.</p> <p>Innovators must balance the needs of different stakeholders. We are not suggesting that educational technology innovators should only create innovations that meet the needs of instructors. Rather, in creating adoptable educational technology innovators should incorporate instructors' needs, other stakeholder needs, and state-of-the-art knowledge on learning (Easterday et al. [<reflink idref="bib20" id="ref60">20</reflink>], [<reflink idref="bib21" id="ref61">21</reflink>]). Because instructors share interdependent goals and tasks with other stakeholders (Somech [<reflink idref="bib59" id="ref62">59</reflink>]), adopting new instructional innovations impacts numerous stakeholders (Penuel and Spillane [<reflink idref="bib46" id="ref63">46</reflink>]), even in higher education where instructors have relatively high level of autonomy. Consequently, many different stakeholders might impede the adoption of an innovation if their needs are not met, or their constraints are violated. For example, students' negative reaction to technological innovations might cause instructors to abandon their adoption plans (Carlson et al. [<reflink idref="bib9" id="ref64">9</reflink>]). As such, innovators need to consider a range of stakeholders, which might include, but not limited to students, administrators, parents, and peer-teachers. Our goal is to support APBL innovators by helping them to understand one of the stakeholders they need to design for, namely the instructor. APBL instructors are implementing a cutting edge and challenging pedagogical approach to support student learning when less demanding approaches may be available (Jonassen and Hung [<reflink idref="bib31" id="ref65">31</reflink>]). This work seeks to inform technological innovations to supports these cutting-edge instructors.</p> <hd id="AN0137339107-6">Methods</hd> <p>We conducted and analyzed semi-structured interviews to reveal APBL instructors' perceived challenges.</p> <hd id="AN0137339107-7">Participants and contexts</hd> <p>We interviewed 47 instructors teaching 35 APBL classes at 20 universities. Our sample included the following higher education institutions classifications (McCormick and Zhao [<reflink idref="bib41" id="ref66">41</reflink>]), including private (55%) and public (45%); high (between 75 and 90%) undergraduate schools (20%), majority undergraduate schools (55%), and majority graduate schools (25%); 4-year medium schools (20%) and 4-year large schools (80%); located in the Eastern U.S. (35%), the Southern U.S. (15%), the Midwestern U.S. (30%), the Western U.S. (15%), and Australia (5%). All institutions were doctoral research universities with selective admissions and full-time 4-year enrollment status; the institutions in the study have a strong focus on research, with high research spending and many researchers, as shown by the institutions" US Carnegie Classification (McCormick and Zhao [<reflink idref="bib41" id="ref67">41</reflink>]): 85% were R1 'highest research activity' (85%), 10% were R2 'higher research activity', and 5% had no classification as they are non-US universities.</p> <p>Instructors taught APBL cornerstone (<reflink idref="bib5" id="ref68">5</reflink>), capstone (<reflink idref="bib9" id="ref69">9</reflink>), and elective or required (<reflink idref="bib21" id="ref70">21</reflink>) classes. Seventy-eight percent of classes were cross-listed in multiple schools; classes were listed in schools of engineering (<reflink idref="bib29" id="ref71">29</reflink>), natural sciences (<reflink idref="bib8" id="ref72">8</reflink>), education (<reflink idref="bib8" id="ref73">8</reflink>), business (<reflink idref="bib8" id="ref74">8</reflink>), humanities (<reflink idref="bib5" id="ref75">5</reflink>), and communication (<reflink idref="bib6" id="ref76">6</reflink>). The number of students (range 6–80) and instructors (range 1–8) per class varied, with a mean of 29 students (SD = 20.46) and 2.6 instructors (SD = 1.55). There was a mean of 6.3 teams per class (SD = 3.58) and 4.5 students per team (SD = 1.27), and 2.7 teams per instructor (SD = 1.60). Instructors also invited a mean of 3.9 experts per class (Tables 4 and 5 in "Appendix" section).</p> <p>The APBL instructors were research faculty (15%), clinical/teaching faculty (38%), adjunct faculty (30%), and graduate student teachers (17%). Graduate student teachers were included if they planned and led the class. All instructors had a master's degree, and 36% had a doctoral degree. Instructors had between 1 and 15 years of experience teaching APBL, with a mean of 6.5 years (SD = 4.3 years). Fifty-five percent of participants identified as female.</p> <hd id="AN0137339107-8">Data collection</hd> <p>We conducted 47 semi-structured interviews with instructors over a 6-month period. We collected data concurrently with data analysis. Interviews lasted between 18 and 66 min (Mean = 34 min). We recruited participants through snowball sampling by asking participants to recommend other APBL instructors. We conducted snowball sampling as an effective way to access a large number of hard to identify participants' (Biernacki and Waldorf [<reflink idref="bib3" id="ref77">3</reflink>]). We sought to gather data from a wide range of participants teaching many different classes at many institutions as a way to increase the credibility of the findings—what Shenton ([<reflink idref="bib57" id="ref78">57</reflink>]) calls "triangulating via data sources" (p. 66). Our goal was to collect data so we could see which challenges were consistent across a range of APBL instructors (Shenton [<reflink idref="bib57" id="ref79">57</reflink>]). After a participant recommended another APBL instructor to interview, we first conducted an internet search to verify the potential participants' roles and position. We then conducted a screening interview, selecting participants if the screening interviewed showed that instructors: (a) taught classes with a real client who provided the problem; (b) interacted with real users (not students taking the class); (c) involved teams with multiple students, (d) involved one "big project" that spanned the whole class; and (e) students had an opportunity to implement their solution. Internet searches and screening interviews allowed us to establish if potential participants were indeed APBL instructors in higher education; in total, we rejected seven participants because they did not meet the criteria.</p> <p>We took several measures to encourage participant honesty. First, we asked participants if they would be willing to be interviewed (with audio), and informed participants they could withdraw from the study at any time. Second, we did not have any institutional power over participants, such as being their supervisor or gaining agreement from a superior for their participation. We did this to ensure willing participation. Third, we framed our role (truthfully) as APBL researchers, designers, and instructors who are seeking to improve a challenging and emerging pedagogy for instructors and students. We did this to signal our desire as peers to learn from, and support, APBL instructors. Fourth, we sought to establish rapport at the start of the interview by expressing our interest in the participants' experiences and expertise, and to frame the interview as a joint effort to improve APBL.</p> <p>Interviews involved three sections after the initial introductions and rapport building. First, we asked instructors to talk about their APBL practices. We asked instructors to describe their practices week-by-week in the last APBL class they taught. Asking about the last time they taught the class encouraged instructors to talk about concrete events rather than provide abstract and vague descriptions (Weiss [<reflink idref="bib68" id="ref80">68</reflink>]); to help participants offer concrete descriptions of their practices we asked participants to show class artifacts they mentioned (e.g. curriculum, assignments). We gathered concrete descriptions from specific instances and captured specific artifacts to support our ability to capture and produced thicker descriptions of instructor challenges (Shenton [<reflink idref="bib57" id="ref81">57</reflink>]; Weiss [<reflink idref="bib68" id="ref82">68</reflink>]). To uncover trade-offs instructors made, we asked what changes instructors had made to their last class. During our semi-structured interview, we asked follow-up questions on previous responses by asking for more concrete detail, to see artifacts (e.g. curriculum, examples of student work, software use), and by asking how practices related to one another. Whenever participants were willing to share an artifact, we appended the artifacts to the interview transcripts and made a note of when in the transcript the artifacts were referenced. We used the appended artifacts in the transcripts to clarify what the participants were referencing in those transcripts (e.g. "as you can see in week 3 of the syllabus..."). We asked follow-up questions both to help participants be more precise and concrete, as well as to ensure their responses were accurate (Shenton [<reflink idref="bib57" id="ref83">57</reflink>]). At the end of the first interview section, we asked about the class context (e.g. team size, number of instructors).</p> <p>In the second section of the interview, we asked participants to name their biggest challenges as APBL instructors. We framed challenges as the most significant unmet need instructors experience. We encouraged instructors to give us concrete accounts by asking them to describe a specific time they recently experienced each challenge (Weiss [<reflink idref="bib68" id="ref84">68</reflink>]). We also asked follow-up questions about how the challenges related to the practices the instructors described in the first section of the interview. This second round of questions again helped us to encourage response precision, concreteness, and accuracy (Shenton [<reflink idref="bib57" id="ref85">57</reflink>]). As in section one of the interview, in section two we also asked to see any artifacts relevant to the challenges. We then asked what the instructors viewed as possible options for changing their course to mitigate these challenges and the related repercussions; we asked this question to elicit trade-offs and the reason for the current balance between competing needs. This involved the interviewer first re-stating what they understood to be the participant's challenges, looking to reiterate both the practitioner's abstract description of the challenge and the concrete example(s) in as close to the participant's words as possible (Spradley [<reflink idref="bib61" id="ref86">61</reflink>]). If the challenges noted by a participant seemed to align with challenges that had emerged from analysis of other interviews, the interviewer would also describe other instructors' challenges to the participant and elicit if they were the same or different from the challenges the participant experienced. To reduce interviewer bias we asked open questions first (e.g. "What are the biggest challenges you face as an instructor?"; Weiss [<reflink idref="bib68" id="ref87">68</reflink>]), and only used technical terms if introduced by the instructor.</p> <p>Our interview approach involved conducting member checks and triangulating findings across participants (Lincoln and Guba [<reflink idref="bib37" id="ref88">37</reflink>]; Shenton [<reflink idref="bib57" id="ref89">57</reflink>]). We conducted member checks during the interview by reiterating our interpretations of the practices and challenges back to the participants towards the end of the interview. We asked participants to correct, confirm, or add to the interviewer's characterization of the practices and challenges. Furthermore, having instructors discuss trade-offs functioned as a member check as instructors discussed their intentions behind their current practices (Lincoln and Guba [<reflink idref="bib37" id="ref90">37</reflink>]). Discussing trade-offs towards the end of the interviews also functioned as a round of questioning, as participants were asked to re-describe past responses while imagining alternative possibilities. We also triangulated across participants (source triangulation, Lincoln and Guba [<reflink idref="bib37" id="ref91">37</reflink>]) to investigate the extent participants reported experiencing the same challenges. During interviews, when a participant reported a challenge we judged to be similar to other challenges in our analysis, we would describe that challenge to the participant; we asked the participant if they viewed this challenge as different or similar to the challenge they experienced. Asking participants to comment on the similarities between challenges helped us to gather data on the extent that participants reported experiencing common challenges.</p> <p>Third, we asked about the instructor's position, such as university role and teaching experience. We transcribed all interviews (598 pages, mean = 12.7 pages per interview).</p> <hd id="AN0137339107-9">Data analysis</hd> <p>To create an account of APBL instructor challenges, we conducted an iterative deductive and inductive qualitative analysis of interview transcripts (Miles et al. [<reflink idref="bib42" id="ref92">42</reflink>]). We conducted data analysis concurrently with data collection until we reached saturation (Small [<reflink idref="bib58" id="ref93">58</reflink>]) and established inter-rater reliability. To improve the credibility of our findings, throughout our analysis we conducted debriefing sessions after each 4–9 interviews; in debriefing sessions we analyzed with one senior author who did not conduct analysis and one non-author senior researcher. In debriefing sessions, we presented our interpretations of emerging codes with data extracts, allowing us to test our interpretations of the data, and consider alternative interpretations.</p> <p>First, to establish our initial set of codes in our codebook we conducted deductive and inductive coding (Miles et al. [<reflink idref="bib42" id="ref94">42</reflink>]) with our first set of nine interview transcripts. We deductively identified extracts of each transcript in which instructors reported practices, challenges, and trade-offs. We then inductively identified codes within these larger categories to create sets of:</p> <p></p> <ulist> <item> <emph>Practice codes.</emph> We created <emph>practice codes</emph> to understand what practices instructors undertake. We inductively created practices codes that described participant reported actions (e.g. <emph>scoping</emph> to form teams and provide a project).</item> <p></p> <item> <emph>Challenge codes.</emph> We created <emph>challenge codes</emph> to capture APBL instructor's perceived challenges. We inductively created challenge codes that captured what instructors experienced as their most pressing challenges (e.g. the challenge that instructors had <emph>recalling team history</emph>, such as remembering project goals).</item> <p></p> <item> <emph>Trade</emph>-<emph>off codes.</emph> We created <emph>trade</emph>-<emph>off codes</emph> to understand the decisions instructors make between the different challenges. We identified each interview extract in which instructors described either (a) past trade-offs they had made, or (b) trade-offs they perceive, in terms of the challenges they face (e.g. trade-off between having more co-instructors who can help students, and then needing to do more co-instructor management).</item> </ulist> <p>Trade-offs represent the balance between two desirable but simultaneously unachievable outcomes given current resources and technology (Collins [<reflink idref="bib13" id="ref95">13</reflink>]), and so they reveal opportunities for innovations that can help achieve both desirable outcomes.</p> <p>Second, with each new set of 4–9 interviews we iteratively refined the codebook. We conducted further analysis, iteratively adding, deleting, or modifying definitions of codes in our codebook. Iterative rounds of coding meant that we revisited previously analyzed transcripts to make sure that all of the changing codes accurately describe the data extracts (Lincoln and Guba [<reflink idref="bib37" id="ref96">37</reflink>]; Miles et al. [<reflink idref="bib42" id="ref97">42</reflink>]; Shenton [<reflink idref="bib57" id="ref98">57</reflink>]). During this iterative analysis, two of the authors both wrote up and then discussed short memos about the definitions of the codes in relation to the data extracts we coded. The memos, code definitions, and data extracts were presented during the debriefing sessions every 4–9 interviews we analyzed. Over the course of data analysis, we deleted three practices codes and six challenge codes upon re-analysis of the data (e.g. the challenge of marketing APBL courses to promote a class or program). We amended every code in some way after their initial conception. In this further analysis, we also created patterns, building a model of how the codes were related (Miles et al. [<reflink idref="bib42" id="ref99">42</reflink>]); in particular, we associated each challenge code with practice codes based (e.g. the challenge that instructors had <emph>recalling team history</emph>, such as remembering project goals, was associated with instructor accounts of <emph>monitoring teams</emph>).</p> <p>Iteratively coding the data from multiple participants functioned as a check on our biases. As APBL instructors and researchers who had previously focused on providing enough of the right feedback as being the core challenge in APBL; our results show that providing feedback in APBL was far from the central challenge experienced by the APBL instructors in our study.</p> <p>We stopped data collection and analysis when we reached saturation (Small [<reflink idref="bib58" id="ref100">58</reflink>]). Saturation occurs when no new practices, challenges, or trade-offs emerged during analysis of new interview transcripts. We operationalized saturation when no new codes emerged between debriefing sessions with senior researchers; in this case, no new codes emerged from data analysis of the final seven interview transcripts.</p> <p>Once we had established what we perceived to be a robust codebook, we conducted a test of inter-rater reliability. Two authors, one new to the codebook, independently coded the same seven randomly selected transcripts (14% of the data); the Cohen's Kappa measure of inter-rater reliability was 0.824, showing almost perfect agreement. We discussed and resolved any disagreements, and re-coded the remaining transcripts.</p> <p>We summarized our findings in the following narrative description of APBL instructor challenges (Table 1), and instructor trade-offs (Fig. 1).</p> <p>Instructors perceived challenges</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Challenge</p></th><th align="left"><p>% Instructors identifying challenge</p></th></tr></thead><tbody><tr><td align="left"><p>1. Providing assistance: monitoring challenges</p></td><td align="left" /></tr><tr><td align="left"><p> Monitoring teams: high interaction cost</p></td><td char="." align="char"><p>70%</p></td></tr><tr><td align="left"><p> Recalling team history</p></td><td char="." align="char"><p>47%</p></td></tr><tr><td align="left"><p> Individual assessment</p></td><td char="." align="char"><p>21%</p></td></tr><tr><td align="left" colspan="2"><p>2. Providing assistance: delivering assistance challenges</p></td></tr><tr><td align="left"><p> Delivering assistance: high interaction cost</p></td><td char="." align="char"><p>57%</p></td></tr><tr><td align="left"><p> Helping students see value in PBL learning goals and experiences</p></td><td char="." align="char"><p>49%</p></td></tr><tr><td align="left"><p> Supporting teamwork skills</p></td><td char="." align="char"><p>47%</p></td></tr><tr><td align="left" colspan="2"><p>3. Scoping challenges</p></td></tr><tr><td align="left"><p> Balancing needs of stakeholders</p></td><td char="." align="char"><p>26%</p></td></tr><tr><td align="left"><p> Sourcing projects</p></td><td char="." align="char"><p>23%</p></td></tr><tr><td align="left" colspan="2"><p>4. Curriculum challenges</p></td></tr><tr><td align="left"><p> Codifying standards</p></td><td char="." align="char"><p>32%</p></td></tr><tr><td align="left"><p> Creating materials to meet shifting needs</p></td><td char="." align="char"><p>28%</p></td></tr><tr><td align="left" colspan="2"><p>5. Coordinating stakeholders</p></td></tr><tr><td align="left"><p> Coordinating and monitoring co-instructors</p></td><td char="." align="char"><p>60%</p></td></tr><tr><td align="left"><p> Coordinating peer support</p></td><td char="." align="char"><p>43%</p></td></tr><tr><td align="left"><p> Coordinating and monitoring clients</p></td><td char="." align="char"><p>28%</p></td></tr></tbody></table> </ephtml> </p> <p>Graph: Fig. 1APBL model summarizing instructor trade-offs related to their challenges in scoping, delivering assistance, developing curriculum, and coordinating others</p> <hd id="AN0137339107-10">Findings</hd> <p>In the findings, we present the challenges that emerged from our analysis. First, we discuss instructor challenges relating to practices of (<reflink idref="bib1" id="ref101">1</reflink>) monitoring teams to provide assistance and (<reflink idref="bib2" id="ref102">2</reflink>) providing assistance, (<reflink idref="bib3" id="ref103">3</reflink>) scoping a problem and creating project teams, (<reflink idref="bib4" id="ref104">4</reflink>) curriculum design, and (<reflink idref="bib5" id="ref105">5</reflink>) coordinating co-instructors, clients, and students. In Table 1 we show the challenges that emerged from interviews with instructors and the percentage of instructors that identified each challenge; we organized the challenges by the practices that the challenges relate to. Second, we discuss the trade-offs instructors reported experiencing.</p> <hd id="AN0137339107-11">Challenge: monitoring teams to provide assistance</hd> <p>In APBL, student teams encounter a range of unforeseen problems. As a result, instructors must provide just-in-time, tailored instruction to each team. Such instruction involves both monitoring teams to diagnose what challenges they face and then responding with appropriate assistance. We now discuss challenges related to monitoring. In the next section, we discuss challenges to providing assistance.</p> <p>To support learning and performance in APBL, instructors monitor each team to determine their needs. Monitoring involves reviewing student work (e.g. prototypes) and meeting with teams one-on-one. The open-ended nature of APBL challenges makes frequent monitoring necessary. The team freedom means that the instructors can't accurately predict what assistance they need to provide for any given team.</p> <hd id="AN0137339107-12">Monitoring teams: high interaction cost (70%)</hd> <p>Instructors report that they spend a great deal of their total instruction time meeting face-to-face with teams to understand their work, thoughts, and feelings. Instructors reported that there are many possible aspects of each team that they need to know about. These included which users the teams are working with, how the team was working together, how the teams were defining the problem, or how they conducted solution testing. Instructors noted a wide array of aspects of each project they instructors needed to find out about, requiring a lot of face-to-face meeting time. As a Bioengineering instructor teaching a first-year cornerstone class reported:</p> <p>...there are different teams with different approaches, so giving every team individual mentorship is time-consuming. You have to work with every team 1-on-1. Every team has a different requirement.</p> <p>Monitoring is challenging in part because students often do not communicate precisely or concisely so instructors must spend time scaffolding student communication.</p> <hd id="AN0137339107-13">Recalling team history (47%)</hd> <p>Instructors reported needing to remember a team's history to know the effectiveness of past assistance and monitor the team for current problems. Accessing relevant team history, including past actions, thinking, and plans, was difficult because of the complexity of the projects and the number of teams. In a sense, the challenge of recalling team history is an extension of the monitoring challenge; in order to support teams instructors need to remember past important aspects of each team. These aspects include areas they felt the team needed support, and the instructional strategies the instructor previously used to try and support the team (so they don't repeat an ineffective strategy). An Entrepreneurship instructor reflects upon this challenge of recording the previous plans of teams in his elective graduate APBL class to help inform his face-to-face meetings with teams:</p> <p>Did they listen to what I said [before]? Did they pay any attention? Did it produce any change in their behavior? We had, what I thought, were great office hours and then they went off and they did the same thing they've been doing. I guess I just muddle along. I don't have a specific solution... I actually started keeping spreadsheets myself.</p> <p>In this example, the instructor spent time recording his interactions with APBL teams on an Excel spreadsheet to review before meeting with students to help remember and judge the effectiveness of his past assistance.</p> <p>Beyond team actions, thinking, and plans, instructors reported that remembering personal details such as names, faces, and skills of individual students was challenging. One Mechanical Engineering instructor in a class of 30 undergraduate students described initiating coaching interactions with a group of students who he assumed were a team in his class, only to be informed by the students that they were not members of his class. This illustrates the considerable cognitive load experienced by APBL instructors.</p> <hd id="AN0137339107-14">Individual assessment (21%)</hd> <p>Some instructors found it challenging to understand what individual students on the APBL team contributed so that they could assess individual learning, and give individual grades. The majority of work in APBL is submitted as a group, but instructors must give each individual student their own grade and make sure each student is learning. Instructors cannot be present for the vast majority of the teamwork in APBL; consequently instructors might struggle with individual assessment. As one Marketing instructor explained of an undergraduate cornerstone class, students did not want to reveal individual team member contributions to instructors:</p> <p>One of the challenges we have when we do a team design project is what do the students actually contribute? Was it just one kid who worked hard and carried the team or was it truly a group effort?</p> <p>Instructors struggle to assess individual student learning, performance, and effort.</p> <hd id="AN0137339107-15">Challenge: delivering assistance</hd> <p>Monitoring project teams allows instructors to tailor assistance. Instructors give assistance to support better team performance and learning. Instructors reported providing assistance on teams' skills, a wide range of problem-solving content knowledge, and team communication. Instructor assistance involved spoken, written, and nonverbal help.</p> <hd id="AN0137339107-16">Assistance: high interaction cost (57%)</hd> <p>Instructors reported that providing students sufficient assistance to help teams overcome impasses was highly challenging. Beyond understanding what teams are doing and what they are struggling with, instructors then need to help the team understand and overcome their impasses. An instructor from a biomimicry undergraduate APBL class reported that he found it challenging to give frequent assistance, and explained why he needed to give this assistance:</p> <p>Interviewer 1: Why do you have the five sessions [of feedback every 10 weeks]?</p> <p>Instructor: It's to make sure that we're course-correcting as we go. There are short time periods that get through a design at least considering all the processes that they are trying to absorb and learn. If they don't course-correct midway, they're going to chase a blind alley.</p> <p>The instructor 'course-corrects', that is, regularly scaffolds student teams' activity and thinking, to make sure students projects can overcome impasses. Without frequent assistance, teams are unlikely to overcome these impasses, because they are taking on authentic project work that is often approaches the difficulty of real-world professional practice. Providing frequent assistance takes significant instructor time. A digital journalism instructor of an undergraduate elective class talks through their class calendar showing regular face-to-face meetings to assist students:</p> <p>My problems? A lot of face-to-face meetings for the student problems and stuff because I'll have two to three meetings individually with each student team in the course of ten weeks. That's a lot of it. Feedback is just trying to make time and I don't have it. A lot of it is meeting with students... That's not acceptable, but it's what works.</p> <p>Here, the instructor relented to organizing his class to meet with teams multiple times because it was the only way he has found to successfully assist students in APBL, despite the fact this took a considerable amount of time to meet face-to-face.</p> <hd id="AN0137339107-17">Helping students see value in learning goals (49%)</hd> <p>Instructors reported that it is challenging to make students realize the importance of the learning goals of APBL classes. Instructors noted APBL goals and experiences are different from other classes, such as giving students ill-structured challenges where instructors cannot define the 'correct' answer. As a result, instructors find it challenging to persuade students of the value of APBL goals, so students are not receptive to instructor assistance. This is particularly the case with non-technical skills such as teamwork and communication, as this Civil Engineering instructor explains:</p> <p>Sometimes you get students coming in and it's just their circumstances that make it challenging because they either don't think that they need to learn this, because they're just going to be an engineer and they're just going to build bridges... That can be very challenging because we try to get them to think about their social skills.</p> <p>Here the instructor explains that teamwork is not seen as important to learn compared to mathematics; the instructor sees this narrow view of important engineering skills as creating challenges for meeting the goals of their class. Similarly, the computer science instructor in the following extract points out that she struggles to teach students communication skills:</p> <p>I always try to tell them that you can be a great engineer in a lab with an amazing solution, but if you are not able to convey that to stakeholders, people that are not necessarily engineers, then that idea stays with you."</p> <p>In this extract the instructor reports struggling to help students see the value in learning these skills relevant to their future work.</p> <hd id="AN0137339107-18">Supporting teamwork skills (47%)</hd> <p>Instructors reported that it is challenging to help students work well in teams. In APBL students must work together to achieve a shared goal. Instructors stressed how in APBL teams had to undertake a range of activities that taxed students' teamwork skills, such as make decisions about what to do, decide what solution to create, schedule meetings, and divide up work. An entrepreneurship instructor observed that team failure in both her undergraduate and graduate APBL classes is most often due not to the lack of content or understanding of the problem, but to team interactions:</p> <p>So, the content is overwhelming but then the [teamwork] process is what often gets them discouraged or has them completely sign out. When I've seen failures that people get burned out, they get tired, they get discouraged and typically it's because... because of the interactions in their teams.</p> <p>Instructors perceived that teams needed assistance on a variety of teamwork skills because students struggle to make group decisions, manage their time, and create accountability for completing work. Instructors reported experiencing this challenge for both graduates and undergraduates.</p> <p>While some instructors reported dedicating part of their courses to teaching teamwork skills, they often reported being unable to successfully assist teamwork, resulting in projects that created less successful products, and caused students emotional strife. The educational technology instructor laments his limited ability to assist teamwork skills:</p> <p>The students typically have never worked effectively... getting to know how to work together through multiple phases of work that probably no one is an expert in.</p> <p>As a result of the instructor's limited ability to assist teams in improving their teamwork, the instructor is resigned to the possibility of breakdowns in teamwork. The resulting dysfunction in-turn creates a greater need for assistance later in the class.</p> <hd id="AN0137339107-19">Challenge: scoping</hd> <p>Instructors scope challenges to provide teams with projects that they can tackle based on time, experience, and available assistance. Scoping involves defining the problem and setting up key aspects of the project such as clients, users, technology, and outcome measures.</p> <hd id="AN0137339107-20">Balancing needs of stakeholders (26%)</hd> <p>Instructors struggled to scope projects that balance student needs, programs goals, and client needs. In the weeks or months leading up to a class, instructors have to define a project with that client that meets all of these needs. This includes helping clients understand the types of project the students might be capable and interested in solving. Given students are new to the types of challenges that clients face, there is a narrower range of projects that meet both student and client needs. As a mechanical engineering instructor noted:</p> <p>The first challenge that we have is project scoping and chartering... to make sure that they've [the scoping team] picked and identified and scoped the project that fits the [program] objective. It's hard, too, because it has to be something that's technically challenging.</p> <p>In this APBL class, the instructors had to balance the projects' need to be technically challenging enough to meet the project goals, while also incorporating students with a range of different technical skills.</p> <hd id="AN0137339107-21">Sourcing projects (23%)</hd> <p>APBL instructors must create projects for student teams. As these projects include a real problem faced by a real client, some instructors found it challenging to find clients. While scoping projects with clients, instructors must find an overlap between clients' needs and students' learning needs. As noted by a product design instructor, "...the hardest part for me is always finding the client...the real challenge is not a technical one, but it's more of a social one of trying to line up the right client." The social nature of sourcing a client involves the instructor negotiating project expectations, including finding a client who has realistic expectations for student work. Furthermore, instructors reported spending a significant amount of time and effort finding the right person within larger client organizations who could provide viable projects.</p> <hd id="AN0137339107-22">Challenge: curriculum</hd> <p>To support their assistance practices when interacting with teams, instructors create and source relevant material.</p> <hd id="AN0137339107-23">Codifying standards (32%)</hd> <p>APBL Instructors of all levels of experience reported codifying standards by creating assessment material that helps students understand the standards for quality work. APBL involves students taking on activities that are different from much of their previous schoolwork, so instructors sought to support students in understanding the standards they expected. These materials included examples of good work (often annotated) and rubrics. Materials that codify quality work standards require significant testing until they capture all the relevant elements, as this education instructor APBL reported:</p> <p>I think that for a lot of... a lot of things that I like to do are problem project based and I think developing rubrics and assessment is really challenging. It is very time-consuming...</p> <p>As this instructor later noted in the interview, assessment material can help "remove some of the subjectivity" so that students understand what they need to do to produce quality work.</p> <hd id="AN0137339107-24">Creating materials to meet shifting needs (28%)</hd> <p>Instructors reported variation across projects that led to a need for instructors to continually create new curricular material. APBL instructors needed to redesign their curriculum because (a) during the class, projects took unforeseen directions that required new curricular material, and (b) when clients changed, instructors needed to provide different materials for new projects. Instructors also had to change the curriculum if clients' wanted student teams to work on different projects the following year. As one mechanical engineer noted of a capstone class:</p> <p>One of the biggest challenges is always figuring out, for lack of better terms, of the master curriculum that was originally conceived for this program, what parts need to change or be removed or replaced according to the specific domain... [We]massage that curriculum in a way that they think is most appropriate to address the needs of the specific problem domain that we'll be focused on during the coming year.</p> <p>Here the instructors had to redesign the curriculum each year. Later in the interview, the instructor reported that they often changed over fifty percent of the material in the curriculum each year.</p> <hd id="AN0137339107-25">Challenge: coordinating stakeholders</hd> <p>APBL instructors must coordinate different stakeholders types who can give teams extra assistance and provide authentic learning experiences. Instructors increase the assistance available in their classes by including more instructors, expert others (e.g. coaches from industry or relevant experts who give feedback), and coordinating students to give peer support. The classes in this study had a mean of 2.56 instructors per-class (12.3 students per-instructor and 2.7 teams per-instructor), and 3.9 expert visitors per-class. Furthermore, clients are another stakeholder type to coordinate. We report on the instructor challenges related to coordinating co-instructors, clients, and students.</p> <hd id="AN0137339107-26">Coordinating and monitoring co-instructors (60%)</hd> <p>Sixty percent of instructors reported significant challenges in coordinating their co-instructors; in larger courses with multiple co-instructors, instructors report challenges working with their co-instructors, including getting new co-instructors to join and aligning instructors in terms of learning goals and work processes. In the same way that APBL projects are highly challenging so they need a team to address them, teaching ABPL is also challenging so instructors often need a teaching team to support the students. Instructors reported that working in co-instructor teams brought on extra management and collaboration challenges, as illustrated in this extract from an industrial engineering instructor:</p> <p>Well, that is one of the pitches we use to get people [co-instructors] recruited to this. You know because, as you can imagine when you're working in this kind of environment, for a faculty member who's used to preparing their own courses and lectures and assignments and they do everything on their own, they are the master of their own world, being in this environment now you are beholden to others... There's this added layer of effort and work, which personally I think improves the quality of the work, but for many faculty members that are accustomed to operating in their own silo, this is a huge challenge.</p> <p>In this example, the instructor notes that the siloed nature of universities is pertinent; as faculty are not habituated to team-teaching they might struggle to work effectively together.</p> <hd id="AN0137339107-27">Coordinating peer support (43%)</hd> <p>Instructors reported challenges coordinating students helping each other. Coordinating peer support is different from teamwork, as it involves students from one team supporting the students in a different team. Given the need for assistance in APBL, instructors often created opportunities within APBL classes for students to assist each other, such as presentations with peer feedback. However, forty-three percent of instructors reported challenges coordinating peer support such as helping students give each other useful assistance or even maintaining a cordial class atmosphere. In the following extract the instructional design instructor expanded upon this challenge:</p> <p>Sometimes it's a rude comment rather than a question [from a peer]. Sometimes it's an irrelevant question because they fell asleep halfway through. Sometimes it's a brilliant question. If you can find those brilliant questions in the mix that will help, not just the presenting students but everyone else watching, that would help me. It would help them, and it would help me, being able to target the lesson that I want them to get out of it.</p> <p>Here, the instructor notes the challenge or coordinating peer assistance, and her desire to be able to more effectively highlight high-quality peer assistance. This extract is instructive, as it both notes the potential instructors see in student peer support, but that peer support often requires instructor-coordination to be useful. The instructor in the above extract suggested that peer support without instructor coordination might even be harmful at times.</p> <hd id="AN0137339107-28">Coordinating and monitoring clients (28%)</hd> <p>Instructors face challenges knowing if client's needs are being met, as well as supporting and tracking client interactions with teams. Instructors must create and maintain relationships with clients, and help the teams solve the clients' problem. This means that there are multiple teams all communicating with the client(s) over the course of the class; the instructor might not be aware of student-client communications, the client's reactions to theses commutations, or the client's shifting needs. In the following extract the mechanical engineering instructor reported his challenges:</p> <p>The things that take the most time are coordinating with the client and a lot of stuff that happens before the class begins, but then it continues to be just a management challenge during the quarter... It was hard to keep track of all that's going on. That was actually probably the biggest source of pain for me, personally.</p> <p>Given a common goal in APBL classes may be to teach students how to respond to client needs, instructors may need to track client responses to properly provide assistance.</p> <p>In sum, our in-depth analysis revealed thirteen instructor challenges relating to assistance, project scope, curriculum, and coordinating stakeholder practices.</p> <hd id="AN0137339107-29">APBL design challenges and tradeoffs</hd> <p>These challenges do not each exist in isolation, so how do these challenges relate and what tradeoffs do instructors perceive? We now describe a model of instructor reported tradeoffs that arise from the challenges and practices of teaching APBL (Fig. 1; Table 2). Our findings highlight how APBL courses involve a delicate balance between authentic challenges, motivation, and learning. APBL instructors teach a range of authentic skills that prepare students for real projects. Students work on authentic challenges, involving client interactions, and working in teams, which both increase problem complexity. Authentic problems motivate teams, but increase project complexity leading to impasses. Impasses and motivation in-turn drive the need for greater assistance. Furthermore authentic problems lead to scoping challenges.</p> <p>Tradeoffs between instructor challenges</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Challenge</p></th><th align="left"><p>Involves trade off between...</p></th></tr></thead><tbody><tr><td align="left" colspan="2"><p>Scoping</p></td></tr><tr><td align="left"><p> Balancing needs of stakeholders</p></td><td align="left"><p>Choosing an authentic challenge so students learn to work with clients on a real project versus spending less time on assistance and curriculum design</p></td></tr><tr><td align="left" colspan="2"><p>Assistance</p></td></tr><tr><td align="left"><p> High interaction cost of assistance (monitoring and delivering assistance)</p></td><td align="left"><p>Spending time monitoring and delivering assistance to teams versus teams unable to overcome impasses (leading to lower performance & learning)</p></td></tr><tr><td align="left"><p> Recalling team history</p></td><td align="left"><p>Spending time recording team activities versus more time spent monitoring students in face-to-face</p></td></tr><tr><td align="left"><p> Supporting-team working skills</p></td><td align="left"><p>Spending time teaching teamwork versus time on other aspects of APBL and time dealing with team dysfunction</p></td></tr><tr><td align="left"><p> Helping students see value in learning goals</p></td><td align="left"><p>Spending time persuading students of the importance of the goals and APBL experience versus students not focusing on (and learning) important APBL skills (e.g. communication)</p></td></tr><tr><td align="left" colspan="2"><p>Curriculum</p></td></tr><tr><td align="left"><p> Creating materials to meet shifting needs</p></td><td align="left"><p>Spending time making the curriculum versus having less authentic more predictable challenges or offering less tailored curricular support</p></td></tr><tr><td align="left"><p> Codifying standards</p></td><td align="left"><p>Spending time making the curriculum versus teams unable to overcome impasses or spending time assisting teams</p></td></tr><tr><td align="left" colspan="2"><p>Coordination</p></td></tr><tr><td align="left"><p> Coordinating and monitoring peers and co-instructors</p></td><td align="left"><p>Spending time co-ordinating peers and co-instructors (and resulting assistance) versus time the instructor could spend giving assistance (and resulting assistance)</p></td></tr><tr><td align="left"><p> Coordinating and monitoring clients</p></td><td align="left"><p>Spending time co-ordinating clients and resulting authentic projects versus time the instructor could spend giving assistance (and resulting assistance)</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0137339107-30">Scoping tradeoffs</hd> <p>When APBL instructors scope projects they report having to balance different aspects of the project problem (project scope, Fig. 1; Table 2, row A). Using real-world problems can help APBL instructors achieve authentic learning goals, motivate students and attract clients. However, without significant scaffolding, these problems can easily overwhelm students. Instructors attempt to maintain student and client interest while keeping the problem feasible for students. But instructors must consider more than project difficulty for students; if projects are scoped too small they do not provide value to client organizations. The complexity of the challenge and the client, in turn, impacts the amount of assistance that the instructor needs to provide to teams, either personally or by coordinating other stakeholders.</p> <p>Instructors thus face scoping tradeoffs as they balance the learning goals: problem complexity, student abilities, student interest, and additionally client interest. Instructors first tradeoff is between spending a significant amount of time creating the initial problem for teams that matches abilities, interests, and the right complexity level to produce good student learning. Furthermore, more authentic projects that are more aligned to real-world practice require that the instructor spend more time providing assistance or creating curriculum.</p> <hd id="AN0137339107-31">Assistance tradeoffs</hd> <p>Instructors reported that introducing difficult new skills and authentic challenges means teams require significant assistance (assistance, Fig. 1; Table 2, row B). In APBL, students learn teamwork, problem-solving, and communication skills; instructors report that these are often new skills that students struggle to master, or even recognize as important. Instructors monitor student work to identify the many potential impasses in teamwork, problem-solving and communication that students may encounter. Much of the assistance instructors reported was face-to-face coaching, a medium of communication well suited for establishing joint understanding (Daft and Lengel [<reflink idref="bib15" id="ref106">15</reflink>]).</p> <p>Instructors reported facing a number of assistance tradeoffs. The overarching assistance tradeoff reported relates to the high interaction costs of monitoring and delivering assistance. If instructors reduce challenge complexity teams need less assistance or provide less support to teams, then learning will decrease as students either take on a less authentic challenge unaligned with professional practice or will be unable to overcome project impasses. Holding project difficulty constant, instructors are faced with the tradeoff of spending more time assisting teams (at the cost of instructor time) versus leaving teams unassisted when they cannot overcome impasses, which leads to lower student performance and learning. Instructors also face tradeoffs with recalling team history. Given the high cognitive load instructors face in APBL, instructors reported experiencing the challenge of recalling team history in order to offer assistance. Instructors face the tradeoff of spending time recording the team history, versus having to spend more time monitoring teams, such as asking teams to summarize project details during precious meeting time.</p> <p>When offering assistance, instructors also reported challenges supporting teamwork and helping students see the value of APBL learning goals. APBL seeks to teach, and often taxes, student teamwork skills. Instructors face a tradeoff of spending time teaching teamwork, versus using that time to assist students in other ways and also dealing with any fallout from dysfunctional teams that might have been avoided through teamwork training (e.g. mediating team disputes). Instructors also face tradeoffs with helping students see the value of APBL learning goals. Instructors reported balancing time persuading students of the importance of learning goals, versus students having less focus on the importance of these skills (e.g. communication; and using the class time for other aspects of assisting teams. Part of the calculus of balancing teamwork and learning goal tradeoffs is how effective instructors can make time spent teaching teamwork or communicating the importance of the learning goals; some instructors reported having no effective strategies to assist teamwork and so devoting little time to teamwork.</p> <hd id="AN0137339107-32">Curricular tradeoffs</hd> <p>Beyond providing human assistance, instructors also create, source, and augment curricular material to support teams (curricular development, Fig. 1; Table 2, row C). Curricular material must respond to shifts in projects, which in turn are dictated by the scope of the project, client needs, and the unpredictability of real-world problems. Furthermore, instructors must codify standards, in part to reduce their burden to assist teams; because the work in APBL is often new to students, instructors have a lot of work to do to communicate the standards for quality work.</p> <p>Curricular tradeoffs that instructors reported involved choosing between spending time creating, augmenting or sourcing materials or offering less assistance to students. Curriculum materials are closely connected to the project scope; instructors can spend more time tailoring their curriculum to the projects, versus having less authentic and more predictable projects. Instructors must also balance trade-offs between giving assistance to students, versus preparing materials that codify standards allowing teams to produce high-quality work with less assistance.</p> <hd id="AN0137339107-33">Coordination tradeoffs</hd> <p>To expand the assistance provided by co-instructors, peers and outside experts, APBL classes in-turn create coordination challenges (peer, co-instructor, and client coordination, Fig. 1; Table 2, row D). Instructors reported balancing the overhead of coordinating these different stakeholders with the assistance they provide versus offering assistance themselves. This must also take into account learning benefits peers gain from providing assistance. Likewise, instructors reported facing the tradeoff of balancing spending time coordinating clients and the learning benefits from working on authentic projects, versus the learning benefits received from the instructor having more time to assist the teams on less authentic problems.</p> <p>The scoping, assistance, curricular, and coordination tradeoffs discussed here provide direction for future innovation. When people make tradeoffs, they must choose between two or more desirable outcomes that cannot all be achieved given current resources, expertise, and technology (Collins [<reflink idref="bib13" id="ref107">13</reflink>]). As such, uncovering tradeoffs provides opportunities for innovators to meet needs by partly or fully resolving these tradeoffs.</p> <hd id="AN0137339107-34">Discussion</hd> <p>We now discuss the implications of these findings, including opportunities for innovation in APBL. In the discussion we first summarize instructor's perceived challenges and trade-offs, and relevance to the literature. Second, we discuss possible future directions, and connect these future directions to existing research.</p> <hd id="AN0137339107-35">Instructor perceived challenges in APBL</hd> <p>This study presents an empirical account of instructor perceived challenges to teaching APBL classes that can inform educational technology innovators. Previous work on APBL provides expert experience (e.g. Dym et al. [<reflink idref="bib19" id="ref108">19</reflink>]) and single-context empirical studies (e.g. Hadim and Esche [<reflink idref="bib26" id="ref109">26</reflink>]) to inform APBL innovation. Work by Mandala et al. ([<reflink idref="bib39" id="ref110">39</reflink>]) investigated instructor challenges in PBL, but not in ABPL, and found only one overlapping challenge to our findings (coordinating peer support). Here we complement this work by empirically investigating what challenges APBL instructors perceive across multiple institutions and domains. Instructors greatest perceived challenges are providing impasse-driven assistance (both monitoring and delivering assistance), scoping APBL problems, developing curriculum tailored to shifting projects, and coordinating stakeholders (Table 1).</p> <p>APBL instructors spend significant time providing impasse-driven assistance (VanLehn [<reflink idref="bib67" id="ref111">67</reflink>]) to teams as they undertake motivating authentic project work (Dym et al. [<reflink idref="bib19" id="ref112">19</reflink>]; Gerber [<reflink idref="bib24" id="ref113">24</reflink>]; Lepper and Malone [<reflink idref="bib36" id="ref114">36</reflink>]; Shaffer and Resnick [<reflink idref="bib56" id="ref115">56</reflink>]), including both monitoring and delivering assistance. Instructors must <emph>monitor</emph> teams to elicit work, and reports on actions, thoughts, and feelings (Collins and Kapur [<reflink idref="bib14" id="ref116">14</reflink>]). <emph>Monitoring</emph> involves a <emph>high interaction cost</emph> (Daft and Lengel [<reflink idref="bib15" id="ref117">15</reflink>]) because of the face-to-face communication time it takes to: understand what assistance teams need; <emph>recall team history</emph> (e.g. past interactions, assistance offered); and monitor for <emph>individual assessment</emph>. Once instructors identify a need for assistance, there is also a <emph>high interaction cost of delivering assistance.</emph> Instructors spend significant time to communicate assistance so that students understand; provide <emph>assistance on teamwork</emph>; and <emph>help students see value in APBL learning goals and experiences</emph> because APBL experiences differ from other classes (Dym et al. [<reflink idref="bib19" id="ref118">19</reflink>]; Prince and Felder [<reflink idref="bib47" id="ref119">47</reflink>]).</p> <p>To create an appropriate learning challenge, APBL instructors scope projects that <emph>balance the needs of stakeholders</emph> by: supporting course learning goals, matching student interests and skills with projects; forming appropriate teams, remaining within the domain(s) of instructors' expertise (Rees Lewis et al. [<reflink idref="bib49" id="ref120">49</reflink>]). Scoping also requires <emph>sourcing projects</emph> (Reifenberg and Long [<reflink idref="bib52" id="ref121">52</reflink>]), which becomes particularly challenging as it involves real clients that instructors must find and negotiate with before the course.</p> <p>Likewise, instructors must <emph>create curricular materials to meet shifting needs</emph>, even though they cannot predict exactly how these project team needs will shift over the course of the project. Instructors saw <emph>codifying standards</emph> (Reiser [<reflink idref="bib53" id="ref122">53</reflink>]), such as creating or sourcing examples and creating rubrics, as challenging because it required significant time, testing, and skill.</p> <p>APBL instructors often recruit other stakeholders to assist APBL teams, which reduces the burden to provide assistance but in turn introduces <emph>coordination</emph> challenges (Somech [<reflink idref="bib59" id="ref123">59</reflink>]), including <emph>coordinating and monitoring co</emph>-<emph>instructors and experts</emph> and <emph>coordinating peer support.</emph> Furthermore, as classes involved clients (Reifenberg and Long [<reflink idref="bib52" id="ref124">52</reflink>]), <emph>coordinating and monitoring clients</emph> places an extra burden on instructors.</p> <hd id="AN0137339107-36">Design and research implications and future directions</hd> <p>Researchers of APBL have called on innovators to build upon existing progress and produce more scalable innovations (Dym et al. [<reflink idref="bib19" id="ref125">19</reflink>]). The final contribution of this study is to offer an empirically grounded set of design and research directions for innovators that are more likely to be adopted by APBL instructors (Table 3). Our contribution here is to suggest new design and research directions, validate existing directions, or to offer more precision on what existing directions should achieve.</p> <p>Possible APBL innovations and research questions that address instructor challenges</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Innovation...</p></th><th align="left"><p>...helps instructors...</p></th><th align="left"><p>by...</p></th><th align="left"><p>...addressing challenge established in the findings section</p></th></tr></thead><tbody><tr><td align="left" colspan="4"><p>Assistance</p></td></tr><tr><td align="left"><p> 1. Team tracker</p></td><td align="left"><p>Helps monitor teams</p></td><td align="left"><p>By showing which teams are on track to meeting their short-term goals, and which need help (and on what), in real time</p></td><td align="left"><p>Monitoring teams: high interaction cost; recalling team history</p></td></tr><tr><td align="left"><p> 2. Project canvas</p></td><td align="left"><p>Helps record critical team information</p></td><td align="left"><p>By prompting teams to surface most important project decisions</p></td><td align="left"><p>Monitoring teams: high interaction cost; recalling team history</p></td></tr><tr><td align="left"><p> 3. Individual assessment and 360 reviews</p></td><td align="left"><p>Helps assess individual abilities and supports teamwork</p></td><td align="left"><p>By automatically collecting quantitative assessments of individual performance and collaboration</p></td><td align="left"><p>Supporting teamwork skills; Individual assessment</p></td></tr><tr><td align="left"><p> 4. Common problem/solution knowledge base</p></td><td align="left"><p>Helps provide feedback</p></td><td align="left"><p>By allowing instructor to search common problems and link students to existing learning resources to address the problem</p></td><td align="left"><p>Assistance: high interaction cost; Creating materials to meet shifting needs</p></td></tr><tr><td align="left"><p> 5. Agile team communication tools</p></td><td align="left"><p>Helps support teamwork</p></td><td align="left"><p>By scaffolding teamwork routines such as planning, task management, surfacing problem and reflecting on how to improve</p></td><td align="left"><p>Supporting teamwork skills; Recalling team history; Monitoring teams: high interaction cost</p></td></tr><tr><td align="left"><p> 6. Client demos</p></td><td align="left"><p>Helps motivate learning goals</p></td><td align="left"><p>By having <italic>clients</italic> (not instructors) convey why learning different concepts and skills will lead to better real-world outcomes</p></td><td align="left"><p>Helping students see value in learning goals</p></td></tr><tr><td align="left" colspan="4"><p>Scoping</p></td></tr><tr><td align="left"><p> 7. Problem brief library and client database library</p></td><td align="left"><p>Helps create problem brief with clients</p></td><td align="left"><p>By providing ready-made, tested project briefs and list of clients</p></td><td align="left"><p>Balancing needs of stakeholders; sourcing projects</p></td></tr><tr><td align="left"><p> 8. Team sorting systems</p></td><td align="left"><p>Help balance needs of the program, students, and clients</p></td><td align="left"><p>By sorting students into teams</p></td><td align="left"><p>Balancing needs of stakeholders</p></td></tr><tr><td align="left" colspan="4"><p>Curriculum</p></td></tr><tr><td align="left"><p> 9. Curriculum remixing tools</p></td><td align="left"><p>Helps develop curriculum</p></td><td align="left"><p>By allowing them to quickly remix existing curriculum to make a custom guide for their PBL course</p></td><td align="left"><p>Creating materials to meet shifting needs</p></td></tr><tr><td align="left"><p> 10. Library of standard measures</p></td><td align="left"><p>Helps instructors codify standards</p></td><td align="left"><p>By providing examples of good work taken from other students (previous years, or other PBL courses)</p></td><td align="left"><p>Codifying standards</p></td></tr><tr><td align="left" colspan="4"><p>Coordinating co-instructors</p></td></tr><tr><td align="left"><p> 11. Help-queue</p></td><td align="left"><p>Helps manage co-instructors</p></td><td align="left"><p>By tracking outstanding help requests and distributing them amongst co-instructors</p></td><td align="left"><p>Coordinating and monitoring co-instructors; Assistance: high interaction cost</p></td></tr><tr><td align="left"><p> 12. Client dashboard</p></td><td align="left"><p>Helps manage clients</p></td><td align="left"><p>By tracking team-client interactions and client satisfaction with the team</p></td><td align="left"><p>Coordinating and monitoring clients</p></td></tr><tr><td align="left"><p> 13. Peer critique system</p></td><td align="left"><p>Helps manage peer support</p></td><td align="left"><p>By making it easy and desirable for peers to give each other quality feedback</p></td><td align="left"><p>Feedback: high interaction cost; coordinating peer support</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0137339107-37">Innovations in assistance</hd> <p>APBL innovators have the opportunity to create and spread innovations that solve challenges of providing assistance (Table 3, rows 1–6). For example, innovators may support instructors monitoring teams and recalling details about teams through team trackers and canvases. Team tracking software and related routines could help instructors monitor teams by displaying which teams are on track to meeting their short term goals, and which teams need help (on what), in real time. Dashboard or canvas software or processes, which show an overview of critical team information (Kolodner et al. [<reflink idref="bib33" id="ref126">33</reflink>]), could help instructors quickly see the state of the project by prompting teams to surface aspects of the project. In previous work (e.g. Chandra Kruse and Nickerson [<reflink idref="bib10" id="ref127">10</reflink>]; Kolodner et al. [<reflink idref="bib33" id="ref128">33</reflink>]), researchers and designers use canvases as a way to prompt team reflection and communication. Based on these findings we suggest future design and research might also focus on using canvases to help instructors to more quickly monitor many teams to see which teams need help. Such work might combine canvases with dashboards or other technologies that help display team and class-wide information to meet monitoring challenges in APBL. Instructors also noted that monitoring challenges were exacerbated by the fact students struggled to communicate their thinking and activities in instructors. This validates existing approaches for scaffolding teamwork with agile (Zhang et al. [<reflink idref="bib71" id="ref129">71</reflink>]) or shared social regulation of learning (Järvelä and Hadwin [<reflink idref="bib29" id="ref130">29</reflink>]) software related to planning, task management, surfacing problems and reflection. While much of the work on shared social regulation of learning focuses on supporting student reflection and teamwork (see meta-review by Panadero and Järvelä [<reflink idref="bib45" id="ref131">45</reflink>]), this work suggests there is an opportunity to extend this research and design work to support instructor monitoring in APBL.</p> <p>To help instructors overcome the high interaction costs of delivering assistance, innovators might create databases of common team problems and solutions that allow instructors to search common problems and link students to existing curricular materials to address the problem (e.g. case library, see Tawfik and Kolodner [<reflink idref="bib65" id="ref132">65</reflink>]). Existing theory and designs exist for case-libraries for PBL problems in which instructors know the solutions to the problems, and cases are shorter as PBL problems can we solved in few days or hours (Tawfik and Kolodner [<reflink idref="bib65" id="ref133">65</reflink>]). It is an open question how to create the technology and curate case libraries for APBL, where instructors and other stakeholder do not know the solution to the problem, and problems take the entire term to work on. Furthermore, instructors find teaching teamwork particularly challenging in APBL. Consequently, this work (a) validates recent approaches focusing on supporting teamwork through 360 in-team review software and processes, such tools like CATME (Loughry et al. [<reflink idref="bib38" id="ref134">38</reflink>]), and (b) supports the need for future design and research on classroom practices and software that can support teaching and learning of teamwork.</p> <p>Instructors experienced the challenges of convincing students that authentic APBL learning goals and activities were worthwhile to prepare students for real-world projects. Innovators might help instructors motivate the importance of APBL learning goals through external sources that support instructors' claims about the learning goals. For example, tools that connect students to professional coaches or clients so these professionals (not instructors) convey the importance of learning goals. That is, an interesting line of design and research that would meet instructor challenges could focus on how we use technology to (a) more frequently embed professionals with relevant experience and expertise to the APBL problem at hand within the learning environment, and (b) encourage interactions with instructors that would help students understand authentic learning goals in APBL. Such a line of research could be tremendously impactful but remains challenging as professionals and students who can rarely meet face-to-face experience significant communication challenges (Rees Lewis et al. [<reflink idref="bib51" id="ref135">51</reflink>], [<reflink idref="bib48" id="ref136">48</reflink>]).</p> <hd id="AN0137339107-38">Innovations in scoping</hd> <p>To support instructors in sourcing projects that balance the needs of the program, students, and clients, a problem brief/client database could be shared to provide ready-made, tested project briefs and client lists (Table 3, rows 7, 8). To specifically help to balance needs during scoping, sorting systems and processes could create students teams based upon key variables (e.g. interests, skills). Existing work by Chang and Downey ([<reflink idref="bib11" id="ref137">11</reflink>]) presents a sorting system that balances students' skills and preferences with the program's learning goals. Future design and research might look to see how to balance further important scoping constraints, such as client needs, client availability, and project difficulty in relation to student skills and goals. If specific client or problem databases are developed, it is an open question as to (a) what are the key variables that we might need to capture relating to clients, and (b) how might we update and curate such databases to make them continually useful to instructors.</p> <hd id="AN0137339107-39">Innovations in curriculum development</hd> <p>Instructors experience challenges in creating a flexible curriculum that supports multiple projects and creating materials that codify and communicate standards to students (e.g. rubrics or examples; Table 3, rows 9, 10). Innovators could create curriculum sharing and remixing software (a GitHub for curricular material), which could allow instructors to quickly remix existing curriculum to tailor guide for their APBL course. Similarly, to support instructors codifying standards of work, innovators might create curatable databases of standards or examples taken from other students (e.g. Tawfik and Kolodner [<reflink idref="bib65" id="ref138">65</reflink>]). There are several open research and design questions in terms of how we might create an 'open source' GitHub-like curriculum sharing platform and community. For example, there could be a lot of innovative work to encourage instructors to share their curricular material, especially in terms of online community norms, and reputational rewards (Kraut and Resnick [<reflink idref="bib35" id="ref139">35</reflink>]).</p> <hd id="AN0137339107-40">Innovations in coordination</hd> <p>Instructors also experience the challenges of coordinating co-instructors, students, and clients to provide learning assistance to teams (Table 3, rows 11–13). There are numerous areas for novel research and design in terms of supporting the APBL teaching teams and the broader community. For example, to support lead-instructors coordinate their co-instructors, a software queue of instructor work (completed/upcoming) could track outstanding work, such as stating which teams need assistance, and distribute tasks amongst co-instructors. Future work might look to apply and tailor online work distribution technology (e.g. Kraut and Resnick [<reflink idref="bib35" id="ref140">35</reflink>]) to the APBL team-teaching context. To support instructors as they manage peers giving assistance to teams, systems and processes that manage and scaffold peer feedback could make it easy and desirable for peers to give each other quality feedback, and for instructors to manage this feedback and track the feedback students apply to their projects. Finally, to support instructors to coordinate clients, client dashboards and other online communication software could help instructors to track team-client interactions and client satisfaction. In general, there are many open design and research questions for how instructors could coordinate the large networks that emerge in APBL and quickly become unwieldy.</p> <hd id="AN0137339107-41">Limitations and future work</hd> <p>While we only interviewed instructors at selective universities, the significant challenges even at highly resourced institutions demonstrate the many opportunities for APBL educational technology innovations. This work primarily took place within the United States, so we are limited in our ability to generalize to other nations and cultures. We used snowball sampling as an effective way to initially identify and access APBL instructors. However, this does mean we are limited in our ability to claim we have a representative sample. In interviewing many participants teaching different classes at different institutions we are (<reflink idref="bib1" id="ref141">1</reflink>) increasing credibility through triangulating data from a wide range of participants and locations with, while (<reflink idref="bib2" id="ref142">2</reflink>) losing the ability to gain in-depth familiarity with a few classrooms that we might gain in a field study (Shenton [<reflink idref="bib57" id="ref143">57</reflink>]). Furthermore, this paper drew on self-report data, which introduces issues of participant reliability (Weiss [<reflink idref="bib68" id="ref144">68</reflink>]). For example, participants' might be limited ability to access their experiences teaching their APBL classes; we attempted to mitigate this issue by having participants draw upon specific experiences and discuss specific artifacts from their classes.</p> <p>Future work in terms of helping understand the needs in APBL could help innovators gain greater certainty on which innovations instructors consider the most useful. In particular, the current study did not present specific innovations or innovation types for participant review. Future work could address this limitation through a large-scale survey with a representative sample of APBL instructors that asks them to prioritize the challenges and solutions. Furthermore, future work could augment the model we created here to include the perceived challenges of other APBL stakeholders, such as students, administrators and program directors; combining this work with investigations of other APBL stakeholders would help APBL innovators balance the needs of multiple stakeholders when designing for adoption (Easterday et al. [<reflink idref="bib20" id="ref145">20</reflink>], [<reflink idref="bib21" id="ref146">21</reflink>]; Penuel and Spillane [<reflink idref="bib46" id="ref147">46</reflink>]; Rogers [<reflink idref="bib54" id="ref148">54</reflink>]).</p> <hd id="AN0137339107-42">Conclusion</hd> <p>The history of diffusing APBL technological innovations has been disappointing, in part because innovations do not meet the perceived needs of APBL instructors. Previous research has not adequately articulated these perceived needs. This study found that APBL instructors' greatest challenges are: (a) setting up the APBL problem; (b) preparing a curriculum flexible enough for APBL; (c) providing assistance to teams, including monitoring; (d) managing a range of stakeholders, including co-instructors, clients, and students.</p> <p>These challenges arise because of the real-world nature of APBL—once instructors commit to giving students more authentic problems, they must incorporate problems scoped to student ability that produce client value. The unpredictable nature of APBL requires the labor-intensive process of assisting students through impasses and often involves face-to-face coaching one team at a time. Adding extra-instructors and peer activities to create additional assistance raises additional coordination challenges.</p> <p>To better appeal to APBL instructors, technological innovations should make it easier for instructors to set up and manage APBL environments. Innovations should help instructors overcome challenges such as scoping problems, developing a flexible curriculum, providing assistance (including monitoring), and decreasing the difficulty of coordinating stakeholders that assist students. To do so, innovators might design: (a) scoping software databases, tools, and practices for sourcing projects, clients, and forming teams; (b) authoring tools that allow for sharing and remixing of curricular materials, including those that codify standards; (c) project management practices, tools and software that help teams manage themselves while allowing instructors to monitor to deliver assistance; and (d) coordination software and practices that can help instructors track actions taken and manage assistance from stakeholders to teams. With these innovations, we can realize the full potential of APBL in preparing students to take on authentic real-world projects.</p> <hd id="AN0137339107-43">Funding</hd> <p>This work was funded by the National Science Foundation (US) Grant Nos. IIS-1530833 and IIS-1320693.</p> <hd id="AN0137339107-44">Acknowledgements</hd> <p>We thank the members of the Delta Lab, Simone Ispa-Landa, Bruce Sherin, and Christopher Riesbeck for their feedback on data collection and analysis.</p> <hd id="AN0137339107-45">Compliance with ethical standards</hd> <p></p> <hd id="AN0137339107-46">Ethical approval</hd> <p>All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.</p> <hd id="AN0137339107-47">Appendix</hd> <p>See Tables 4 and 5.</p> <p>APBL classes in the sample</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>#</p></th><th align="left"><p>%</p></th></tr></thead><tbody><tr><td align="left" colspan="3"><p>Course type</p></td></tr><tr><td align="left"><p> Elective/required<sup>a</sup></p></td><td char="." align="char"><p>21</p></td><td char="." align="char"><p>60</p></td></tr><tr><td align="left"><p> Capstone—final year class to prepare students about to enter industry. In many programs these are prerequisite for graduating</p></td><td char="." align="char"><p>9</p></td><td char="." align="char"><p>25</p></td></tr><tr><td align="left"><p> Cornerstone—introductory APBL class to discipline</p></td><td char="." align="char"><p>5</p></td><td char="." align="char"><p>14</p></td></tr><tr><td align="left" colspan="3"><p>Term type</p></td></tr><tr><td align="left"><p> Quarter (10 weeks)</p></td><td char="." align="char"><p>15</p></td><td char="." align="char"><p>43</p></td></tr><tr><td align="left"><p> Semester (15 weeks)</p></td><td char="." align="char"><p>20</p></td><td char="." align="char"><p>57</p></td></tr><tr><td align="left" colspan="3"><p>Terms</p></td></tr><tr><td align="left"><p> 1 term</p></td><td char="." align="char"><p>21</p></td><td char="." align="char"><p>60</p></td></tr><tr><td align="left"><p> 2 terms</p></td><td char="." align="char"><p>13</p></td><td char="." align="char"><p>37</p></td></tr><tr><td align="left"><p> 3 terms</p></td><td char="." align="char"><p>1</p></td><td char="." align="char"><p>3</p></td></tr></tbody></table> </ephtml> </p> <p> <sups>a</sups>We collapsed elective and required courses, as many classes were both reported as electives for some students and required for other students</p> <p>APBL Classes in the Sample</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Mode</p></th><th align="left"><p>Min</p></th><th align="left"><p>Max</p></th><th align="left"><p>Mean</p></th><th align="left"><p>SD</p></th></tr></thead><tbody><tr><td align="left"><p>Weeks/course</p></td><td char="." align="char"><p>15</p></td><td char="." align="char"><p>10.0</p></td><td char="." align="char"><p>30.0</p></td><td char="." align="char"><p>18.57</p></td><td char="." align="char"><p>7.9</p></td></tr><tr><td align="left"><p>Instructors/course</p></td><td char="." align="char"><p>1</p></td><td char="." align="char"><p>1.0</p></td><td char="." align="char"><p>8.0</p></td><td char="." align="char"><p>2.6</p></td><td char="." align="char"><p>1.55</p></td></tr><tr><td align="left"><p>Students/course</p></td><td char="." align="char"><p>20</p></td><td char="." align="char"><p>6.0</p></td><td char="." align="char"><p>80.0</p></td><td char="." align="char"><p>29.0</p></td><td char="." align="char"><p>20.46</p></td></tr><tr><td align="left"><p>Students/instructor</p></td><td char="." align="char"><p>10</p></td><td char="." align="char"><p>4.9</p></td><td char="." align="char"><p>40.0</p></td><td char="." align="char"><p>12.3</p></td><td char="." align="char"><p>7.55</p></td></tr><tr><td align="left"><p>Students/team</p></td><td char="." align="char"><p>5</p></td><td char="." align="char"><p>2.0</p></td><td char="." align="char"><p>8.0</p></td><td char="." align="char"><p>4.5</p></td><td char="." align="char"><p>1.27</p></td></tr><tr><td align="left"><p>Teams/course</p></td><td char="." align="char"><p>4</p></td><td char="." align="char"><p>1.0</p></td><td char="." align="char"><p>16.0</p></td><td char="." align="char"><p>6.3</p></td><td char="." align="char"><p>3.58</p></td></tr><tr><td align="left"><p>Teams/instructor</p></td><td char="." align="char"><p>2</p></td><td char="." align="char"><p>1.0</p></td><td char="." align="char"><p>8.0</p></td><td char="." align="char"><p>2.7</p></td><td char="." align="char"><p>1.60</p></td></tr><tr><td align="left"><p>Problems/course*</p></td><td char="." align="char"><p>4</p></td><td char="." align="char"><p>1.0</p></td><td char="." align="char"><p>13.0</p></td><td char="." align="char"><p>4.5</p></td><td char="." align="char"><p>2.62</p></td></tr><tr><td align="left"><p>Expert visitors/course</p></td><td char="." align="char"><p>2</p></td><td char="." align="char"><p>0.0</p></td><td char="." align="char"><p>30.0</p></td><td char="." align="char"><p>3.9</p></td><td char="." align="char"><p>5.01</p></td></tr><tr><td align="left"><p>Students/expert (visitors + instructors)</p></td><td char="." align="char"><p>2</p></td><td char="." align="char"><p>1.5</p></td><td char="." align="char"><p>27.0</p></td><td char="." align="char"><p>6.2</p></td><td char="." align="char"><p>5.55</p></td></tr><tr><td align="left"><p>Teams/expert (visitors + instructors)</p></td><td char="." align="char"><p>2</p></td><td char="." align="char"><p>0.25</p></td><td char="." align="char"><p>7.0</p></td><td char="." align="char"><p>1.4</p></td><td char="." align="char"><p>1.37</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0137339107-48">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0137339107-49"> <title> References </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Archer K, Savage R, Sanghera-Sidhu S, Wood E, Gottardo A, Chen V. Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis. Computers & Education. 2014; 78: 140-149</bibtext> </blist> <blist> <bibl id="bib2" idref="ref49" type="bt">2</bibl> <bibtext> Beyer H, Holtzblatt K. Contextual design: Defining customer-centered systems. 1998: San Franscisco, CA; Morgan Kaufmann</bibtext> </blist> <blist> <bibl id="bib3" idref="ref50" type="bt">3</bibl> <bibtext> Biernacki P, Waldorf D. Snowball sampling: Problems and techniques of chain referral sampling. Sociological Methods & Research. 1981; 10; 2: 141-163</bibtext> </blist> <blist> <bibl id="bib4" idref="ref51" type="bt">4</bibl> <bibtext> Blank S, Dorf B. The startup owner's manual. 2012: Pescadero, CA; K&S Ranch Press</bibtext> </blist> <blist> <bibl id="bib5" idref="ref42" type="bt">5</bibl> <bibtext> Blumenfeld P, Fishman BJ, Krajcik J, Marx RW, Soloway E. Creating usable innovations in systemic reform: Scaling up technology-embedded project-based science in urban schools. Educational Psychologist. 2000; 35; 3: 149-164</bibtext> </blist> <blist> <bibl id="bib6" idref="ref21" type="bt">6</bibl> <bibtext> Boling E, Schwier RA, Gray CM, Smith KM, Campbell K. Studio teaching in higher education: Selected design cases. 2016: Abingdon; Routledge</bibtext> </blist> <blist> <bibl id="bib7" idref="ref39" type="bt">7</bibl> <bibtext> Borrego M, Froyd JE, Hall TS. Diffusion of engineering education innovations: A survey of awareness and adoption rates in US engineering departments. Journal of Engineering Education. 2010; 99; 3: 185-207</bibtext> </blist> <blist> <bibl id="bib8" idref="ref22" type="bt">8</bibl> <bibtext> Carlson, S.E., Maliakal, L.V., Rees Lewis, D.G., Gorson, J., Gerber, E.M., & Easterday, M.W. (2018a). Defining and assessing risk analysis: The key to strategic iteration in real-world problem solving. In Proceedings of the International Conference of the Learning Sciences (ICLS). London: ICLS.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref64" type="bt">9</bibl> <bibtext> Carlson SE, Rees Lewis DG, Gerber EM, Easterday MW. Challenges of peer instruction in an undergraduate student-led learning community: Bi-directional diffusion as a crucial instructional process. Instructional Science. 2018; 46: 405-433</bibtext> </blist> <blist> <bibtext> Chandra Kruse, L., & Nickerson, J. V. (2018). Portraying design essence. Proceedings of the 51st Hawaii International Conference on System Sciences (HICSS 2018). Waikoloa Village, Hawaii, USA: AIS.</bibtext> </blist> <blist> <bibtext> Chang, M. L., & Downey, A. B. (2008, June). A semi-automatic approach for project assignment in a capstone course. In Proceedings of ASEE annual conference and exposition, Pittsburgh, PA. Retrieved from https://peer.asee.org/4116.</bibtext> </blist> <blist> <bibtext> Cheung AC, Slavin RE. The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational research review. 2013; 9: 88-113</bibtext> </blist> <blist> <bibtext> Collins A Vosniadou S, De Corte E, Glaser R, Mandl H. Design issues for learning environments. International perspectives on the design of technology-supported learning environments. 1996: New York, NY; Routledge: 347-361</bibtext> </blist> <blist> <bibtext> Collins A, Kapur M Sawyer RK. Cognitive apprenticeship. The Cambridge handbook of the learning sciences. 2014: Cambridge; Cambridge University Press: 109-127</bibtext> </blist> <blist> <bibtext> Daft RL, Lengel RH. Organizational information requirements, media richness and structural design. Management Science. 1986; 32; 5: 554-571</bibtext> </blist> <blist> <bibtext> de Koster S, Volman M, Kuiper E. Concept-guided development of technology in 'traditional' and 'innovative' schools: Quantitative and qualitative differences in technology integration. Educational Technology Research and Development. 2017; 65; 5: 1325-1344</bibtext> </blist> <blist> <bibtext> Dunlap JC. Problem-based learning and self-efficacy: How a capstone course prepares students for a profession. Educational Technology Research and Development. 2005; 53; 1: 65-83</bibtext> </blist> <blist> <bibtext> Dutson AJ, Todd RH, Magleby SP, Sorensen CD. A review of literature on teaching engineering design through project-oriented capstone courses. Journal of Engineering Education. 1997; 86; 1: 17-28</bibtext> </blist> <blist> <bibtext> Dym CL, Agogino AM, Eris O, Frey DD, Leifer LJ. Engineering design thinking, teaching, and learning. Journal of Engineering Education. 2005; 94; 1: 103-120</bibtext> </blist> <blist> <bibtext> Easterday MW, Rees Lewis DG, Gerber EM. The logic of the theoretical and practical products of design research. Australasian Journal of Educational Technology. 2016</bibtext> </blist> <blist> <bibtext> Easterday MW, Rees Lewis DG, Gerber EM. The logic of design research. Learning: Research and Practice. 2018; 4; 2: 131-160</bibtext> </blist> <blist> <bibtext> Ertmer PA, Simons KD. Jumping the PBL implementation hurdle: Supporting the efforts of K–12 teachers. Interdisciplinary Journal of Problem-based Learning. 2006; 1; 1: 5</bibtext> </blist> <blist> <bibtext> Frank M, Barzilai A. Integrating alternative assessment in a project-based learning course for pre-service science and technology teachers. Assessment & Evaluation in Higher Education. 2004; 29; 1: 41-61</bibtext> </blist> <blist> <bibtext> Gerber E. Design for America: Organizing for civic innovation. Interactions. 2014; 21; 2: 42-47</bibtext> </blist> <blist> <bibtext> Goel V, Pirolli P. The structure of design problem spaces. Cognitive Science. 1992; 16; 3: 395-429</bibtext> </blist> <blist> <bibtext> Hadim, H. A., & Esche, S. K. (2002, November). Enhancing the engineering curriculum through project-based learning. In Proceedings of 32nd annual ASEE/IEEE frontiers in education conference. Boston, MA, USA. Champaign, IL, USA: Stipes Publishing..</bibtext> </blist> <blist> <bibtext> Hoit M, Ohland M. The impact of a discipline-based introduction to engineering course on improving retention. Journal of Engineering Education. 1998; 87; 1: 79-85</bibtext> </blist> <blist> <bibtext> . National education technology standards for students. 2002: Eugene; ISTE</bibtext> </blist> <blist> <bibtext> Järvelä S, Hadwin AF. New frontiers: Regulating learning in CSCL. Educational Psychologist. 2013; 48; 1: 25-39</bibtext> </blist> <blist> <bibtext> Johnson L, Adams Becker S, Estrada V, Freeman A. The NMC Horizon Report: 2015 Higher Education. 2015: Austin, TX; New Media Consortium</bibtext> </blist> <blist> <bibtext> Jonassen DH, Hung W Walker A, Leary H. All problems are not equal: Implications for problem-based learning. Essential readings in problem-based learning. 2015: West Lafayette; Purdue University Press: 17-41</bibtext> </blist> <blist> <bibtext> Jonassen D, Strobel J, Lee CB. Everyday problem solving in engineering: Lessons for engineering educators. Journal of Engineering Education. 2006; 95; 2: 139-151</bibtext> </blist> <blist> <bibtext> Kolodner J, Owensby J, Guzdial M Jonassen DH. Case-based learning aids. Handbook of research on educational communications and technology: A project of the Association for Educational Communications and Technology. 20042: New York; L. Erlbaum: 829-861</bibtext> </blist> <blist> <bibtext> Krajcik JS, Czerniak CM. Teaching science in elementary and middle school classrooms: A project-based approach. 2013: London; Taylor and Francis</bibtext> </blist> <blist> <bibtext> Kraut RE, Resnick P Kraut RE, Resnick P. Encouraging contribution to online communities. Building successful online communities: Evidence-based social design. 2011: Boston, MA; MIT Press: 21-76</bibtext> </blist> <blist> <bibtext> Lepper MR, Malone TW Snow RE, Farr MJ. Intrinsic motivation and instructional effectiveness in computer-based education. Aptitude, learning, and instruction: III. Conative and affective process analyses. 1987: Hillsdale, NJ; Lawrence Erlbaum Associates, In: 255-296</bibtext> </blist> <blist> <bibtext> Lincoln YS, Guba EG. Establishing trustworthiness. Naturalistic inquiry. 1985: London; Sage Publications: 289-331</bibtext> </blist> <blist> <bibtext> Loughry ML, Ohland MW, Woehr DJ. Assessing teamwork skills for assurance of learning using CATME team tools. Journal of Marketing Education. 2014; 36; 1: 5-19</bibtext> </blist> <blist> <bibtext> Mandala M, Schunn C, Dow S, Goldberg M, Pearlman J. Uncovering the practices, challenges, and incentives for engineering design faculty. International Journal of Engineering Education. 2018; 34; 4: 1314-1324</bibtext> </blist> <blist> <bibtext> Matusovich HM, Paretti MC, McNair LD, Hixson C. Faculty motivation: A gateway to transforming engineering education. Journal of Engineering Education. 2014; 103; 2: 302-330</bibtext> </blist> <blist> <bibtext> McCormick AC, Zhao CM. Rethinking and reframing the Carnegie classification. Change: The Magazine of Higher Learning. 2005; 37; 5: 51-57</bibtext> </blist> <blist> <bibtext> Miles MB, Huberman AM, Saldana J. Qualitative data analysis. 2013: London; Sage</bibtext> </blist> <blist> <bibtext> Niederhauser DS, Stoddart T. Teachers' instructional perspectives and use of educational software. Teaching and Teacher Education. 2001; 17; 1: 15-31</bibtext> </blist> <blist> <bibtext> Olds BM, Miller RL. The effect of a first-year integrated engineering curriculum on graduation rates and student satisfaction: A longitudinal study. Journal of Engineering Education. 2004; 93; 1: 23</bibtext> </blist> <blist> <bibtext> Panadero E, Järvelä S. Socially shared regulation of learning: A review. European Psychologist. 2015; 20; 3: 190-203</bibtext> </blist> <blist> <bibtext> Penuel WR, Spillane JP Sawyer K. Learning sciences and policy design and implementation: Key concepts and tools for collaborative engagement. The cambridge handbook of the learning sciences. 2014: New York, NY; Cambridge University Press: 649-667</bibtext> </blist> <blist> <bibtext> Prince MJ, Felder RM. Inductive teaching and learning methods: Definitions, comparisons, and research bases. Journal of Engineering Education. 2006; 95; 2: 123-138</bibtext> </blist> <blist> <bibtext> Rees Lewis DG, Easterday MW, Harburg E, Gerber EM, Riesbeck CK. Overcoming barriers between volunteer professionals advising project-based learning teams with regulation tools. British Journal of Educational Technology. 2017</bibtext> </blist> <blist> <bibtext> Rees Lewis D. G., Gerber E. M., & Easterday. M. W. (2015) Supporting project scoping: The Scoping wheel. In Poster presented at Harvey Mudd Design Workshop IX "Design Thinking in Design Education". Claremont, CA, US.</bibtext> </blist> <blist> <bibtext> Rees Lewis, D.G., Gorson, J., Maliakal, L.V., Carlson, S.E., Riesbeck, C.K., Gerber, E.M., & Easterday, M.W. (2018). Planning to Iterate: Supporting iterative practices for real-world ill-structured problem-solving. In Proceedings of the international conference of the learning sciences (ICLS). London, UK: ICLS.</bibtext> </blist> <blist> <bibtext> Rees Lewis, D., Harburg, E., Gerber, E.M., Easterday, M. (2015). Building help-seeking tools for novice designers. In Proceedings of the 2015 ACM SIGCHI conference on creativity & cognition 2015 (pp. 43–52). New York, NY: ACM.</bibtext> </blist> <blist> <bibtext> Reifenberg S, Long S. Negotiating the client-based capstone experience. International Journal of Teaching and Learning in Higher Education. 2017; 29; 3: 580-588</bibtext> </blist> <blist> <bibtext> Reiser BJ. Scaffolding complex learning: The mechanisms of structuring and problematizing student work. The Journal of the Learning Sciences. 2004; 13; 3: 273-304</bibtext> </blist> <blist> <bibtext> Rogers EM. Diffusion of innovations. 2003: New York, NY; Simon and Schuster</bibtext> </blist> <blist> <bibtext> Sadaf A, Newby TJ, Ertmer PA. An investigation of the factors that influence preservice teachers' intentions and integration of Web 2.0 tools. Educational Technology Research and Development. 2016; 64; 1: 37-64</bibtext> </blist> <blist> <bibtext> Shaffer DW, Resnick M. "Thick" authenticity: New media and authentic learning. Journal of Interactive Learning Research. 1999; 10; 2: 195-215</bibtext> </blist> <blist> <bibtext> Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. Education for information. 2004; 22; 2: 63-75</bibtext> </blist> <blist> <bibtext> Small ML. How many cases do I need? 'On science and the logic of case selection in field-based research. Ethnography. 2009; 10; 1: 5-38</bibtext> </blist> <blist> <bibtext> Somech A. Managing conflict in school teams: The impact of task and goal interdependence on conflict management and team effectiveness. Educational administration quarterly. 2008; 44; 3: 359-390</bibtext> </blist> <blist> <bibtext> Spillane JP, Parise LM, Sherer JZ. Organizational routines as coupling mechanisms: Policy, school administration, and the technical core. American Educational Research Journal. 2011; 48; 3: 586-619</bibtext> </blist> <blist> <bibtext> Spradley JP. Participant observation. 2016: Long Grove, IL; Waveland Press</bibtext> </blist> <blist> <bibtext> Strobel J, Van Barneveld A. When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-based Learning. 2009; 3; 1: 4</bibtext> </blist> <blist> <bibtext> Sutherland J, Sutherland JJ. Scrum: The art of doing twice the work in half the time. 2014: New York, NY; Crown Business</bibtext> </blist> <blist> <bibtext> Tatar D, Roschelle J, Knudsen J, Shechtman N, Kaput J, Hopkins B. Scaling up innovative technology-based mathematics. The Journal of the Learning Sciences. 2008; 17; 2: 248-286</bibtext> </blist> <blist> <bibtext> Tawfik AA, Kolodner JL. Systematizing scaffolding for problem-based learning: A view from case-based reasoning. Interdisciplinary Journal of Problem-Based Learning. 2016; 10; 1: 6</bibtext> </blist> <blist> <bibtext> Vanderlinde R, van Braak J. Implementing an ICT curriculum in a decentralised policy context: Description of ICT practices in three Flemish primary schools. British Journal of Educational Technology. 2010; 41; 6: E139-E141</bibtext> </blist> <blist> <bibtext> VanLehn K Mandl H, Lesgold A. Toward a theory of impasse-driven learning. Learning issues for intelligent tutoring systems. 1988: New York, NY; Springer: 19-41</bibtext> </blist> <blist> <bibtext> Weiss RS. Learning from strangers: The art and method of qualitative interview studies. 1994: New York, NY; Free Press</bibtext> </blist> <blist> <bibtext> Yadav A, Subedi D, Lundeberg MA, Bunting CF. Problem-based learning: Influence on students' learning in an electrical engineering course. Journal of Engineering Education. 2011; 100; 2: 253</bibtext> </blist> <blist> <bibtext> Zaritsky R, Kelly AE, Flowers W, Rogers E, O'Neill P. Clinical design sciences: A view from sister design efforts. Educational Researcher. 2003; 32; 1: 32-34</bibtext> </blist> <blist> <bibtext> Zhang, H., Easterday, M. W., Gerber, E. M., Rees Lewis, D., & Maliakal, L. (2017). Agile research studios: Orchestrating communities of practice to advance research training. In Proceedings of the ACM conference on computer supported cooperative work and social computing, Portland, OR, USA. New York, NY: ACM Press.</bibtext> </blist> </ref> <aug> <p>By Daniel G. Rees Lewis; Elizabeth M. Gerber; Spencer E. Carlson and Matthew W. Easterday</p> <p>Reported by Author; Author; Author; Author</p> <p></p> <p>Daniel G. Rees Lewis is a postdoctoral research fellow in the Delta Lab in the Segal Design Institute at Northwestern University. His research focuses on creating sociotechnical systems for novice designers.</p> <p>Elizabeth M. Gerber is an associate professor of design in the Schools of Engineering and Communication, Director of the Design Research Cluster, at the Northwestern University, and the Faculty Founder of Design for America. Dr. Gerber researches and designs technology and organizations to support innovation.</p> <p>Spencer E. Carlson is a 3rd year Learning Sciences PhD student in the Delta Lab in the School of Education and Social Policy at Northwestern University. His research focuses on designing learning environments for teaching students to solve ill-structured challenges.</p> <p>Matthew W. Easterday is an associate professor in the School of Education and Social Policy at Northwestern University. His research focuses on conducting design-based research on interventions to prepare the next generation of civic innovators.</p> </aug> <nolink nlid="nl1" bibid="bib64" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib12" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib30" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib54" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib16" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib28" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib70" firstref="ref9"></nolink> <nolink nlid="nl8" bibid="bib20" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib23" firstref="ref11"></nolink> <nolink nlid="nl10" bibid="bib47" firstref="ref12"></nolink> <nolink nlid="nl11" bibid="bib19" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib56" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib31" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib17" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib50" firstref="ref23"></nolink> <nolink nlid="nl16" bibid="bib24" firstref="ref24"></nolink> <nolink nlid="nl17" bibid="bib34" firstref="ref25"></nolink> <nolink nlid="nl18" bibid="bib32" firstref="ref26"></nolink> <nolink nlid="nl19" bibid="bib18" firstref="ref27"></nolink> <nolink nlid="nl20" bibid="bib44" firstref="ref28"></nolink> <nolink nlid="nl21" bibid="bib62" firstref="ref29"></nolink> <nolink nlid="nl22" bibid="bib69" firstref="ref30"></nolink> <nolink nlid="nl23" bibid="bib27" firstref="ref32"></nolink> <nolink nlid="nl24" bibid="bib25" firstref="ref36"></nolink> <nolink nlid="nl25" bibid="bib26" firstref="ref38"></nolink> <nolink nlid="nl26" bibid="bib22" firstref="ref40"></nolink> <nolink nlid="nl27" bibid="bib43" firstref="ref41"></nolink> <nolink nlid="nl28" bibid="bib46" firstref="ref43"></nolink> <nolink nlid="nl29" bibid="bib60" firstref="ref44"></nolink> <nolink nlid="nl30" bibid="bib66" firstref="ref45"></nolink> <nolink nlid="nl31" bibid="bib40" firstref="ref54"></nolink> <nolink nlid="nl32" bibid="bib55" firstref="ref55"></nolink> <nolink nlid="nl33" bibid="bib63" firstref="ref56"></nolink> <nolink nlid="nl34" bibid="bib21" firstref="ref61"></nolink> <nolink nlid="nl35" bibid="bib59" firstref="ref62"></nolink> <nolink nlid="nl36" bibid="bib41" firstref="ref66"></nolink> <nolink nlid="nl37" bibid="bib29" firstref="ref71"></nolink> <nolink nlid="nl38" bibid="bib57" firstref="ref78"></nolink> <nolink nlid="nl39" bibid="bib68" firstref="ref80"></nolink> <nolink nlid="nl40" bibid="bib61" firstref="ref86"></nolink> <nolink nlid="nl41" bibid="bib37" firstref="ref88"></nolink> <nolink nlid="nl42" bibid="bib42" firstref="ref92"></nolink> <nolink nlid="nl43" bibid="bib58" firstref="ref93"></nolink> <nolink nlid="nl44" bibid="bib13" firstref="ref95"></nolink> <nolink nlid="nl45" bibid="bib15" firstref="ref106"></nolink> <nolink nlid="nl46" bibid="bib39" firstref="ref110"></nolink> <nolink nlid="nl47" bibid="bib67" firstref="ref111"></nolink> <nolink nlid="nl48" bibid="bib36" firstref="ref114"></nolink> <nolink nlid="nl49" bibid="bib14" firstref="ref116"></nolink> <nolink nlid="nl50" bibid="bib49" firstref="ref120"></nolink> <nolink nlid="nl51" bibid="bib52" firstref="ref121"></nolink> <nolink nlid="nl52" bibid="bib53" firstref="ref122"></nolink> <nolink nlid="nl53" bibid="bib33" firstref="ref126"></nolink> <nolink nlid="nl54" bibid="bib10" firstref="ref127"></nolink> <nolink nlid="nl55" bibid="bib71" firstref="ref129"></nolink> <nolink nlid="nl56" bibid="bib45" firstref="ref131"></nolink> <nolink nlid="nl57" bibid="bib65" firstref="ref132"></nolink> <nolink nlid="nl58" bibid="bib38" firstref="ref134"></nolink> <nolink nlid="nl59" bibid="bib51" firstref="ref135"></nolink> <nolink nlid="nl60" bibid="bib48" firstref="ref136"></nolink> <nolink nlid="nl61" bibid="bib11" firstref="ref137"></nolink> <nolink nlid="nl62" bibid="bib35" firstref="ref139"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1221426
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Opportunities for Educational Innovations in Authentic Project-Based Learning: Understanding Instructor Perceived Challenges to Design for Adoption
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Rees+Lewis%2C+Daniel+G%2E%22">Rees Lewis, Daniel G.</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-0928-3831">0000-0002-0928-3831</externalLink>)<br /><searchLink fieldCode="AR" term="%22Gerber%2C+Elizabeth+M%2E%22">Gerber, Elizabeth M.</searchLink><br /><searchLink fieldCode="AR" term="%22Carlson%2C+Spencer+E%2E%22">Carlson, Spencer E.</searchLink><br /><searchLink fieldCode="AR" term="%22Easterday%2C+Matthew+W%2E%22">Easterday, Matthew W.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Technology+Research+and+Development%22"><i>Educational Technology Research and Development</i></searchLink>. Aug 2019 67(4):953-982.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 30
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2019
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: IIS1530833<br />IIS1320693
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Educational+Innovation%22">Educational Innovation</searchLink><br /><searchLink fieldCode="DE" term="%22Active+Learning%22">Active Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Projects%22">Student Projects</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Technology%22">Educational Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Attitudes%22">Teacher Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Adoption+%28Ideas%29%22">Adoption (Ideas)</searchLink><br /><searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22College+Faculty%22">College Faculty</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Opportunities%22">Educational Opportunities</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11423-019-09673-4
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1042-1629
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Authentic project-based learning (APBL) is a highly effective way for instructors to help students learn disciplinary skills, modes of thinking, and collaborative practices by creating solutions to real-world problems for real users and clients. While educational technology innovations can bolster APBL by making a promising but challenging pedagogy more effective, as with many areas of education instructor adoption is slow. Diffusion of innovations theory predicts that instructors will adopt and maintain their use of innovations if innovations are perceived to, and then do, address their challenges. To guide design of future APBL technologies, we interviewed 47 university APBL instructors about their most significant challenges and inductively analyzed the resulting interview transcripts. APBL instructors reported interrelated challenges of: (a) "scoping," sourcing challenges and balancing the needs of the program, students, and clients; (b) "curriculum" preparation, making the curriculum flexible enough for shifting project problems and codify standards to help students understand how to do quality work; (c) providing "assistance" to teams, including monitoring, and delivering assistance; and (d) "coordinating" a range of stakeholders involved in assisting teams, including co-instructors, clients, and students. To support instructor adoption in APBL, educational technology innovators might communicate existing technology, or create technological innovations, that provide: (a) scoping tools for sourcing projects, and forming teams; (b) authoring tools for sharing and remixing of curricular materials; (c) project management tools for team management and monitoring; and (d) coordination software to manage all APBL stakeholders.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2019
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1221426
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1221426
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11423-019-09673-4
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 30
        StartPage: 953
    Subjects:
      – SubjectFull: Educational Innovation
        Type: general
      – SubjectFull: Active Learning
        Type: general
      – SubjectFull: Student Projects
        Type: general
      – SubjectFull: Educational Technology
        Type: general
      – SubjectFull: Teacher Attitudes
        Type: general
      – SubjectFull: Adoption (Ideas)
        Type: general
      – SubjectFull: Design
        Type: general
      – SubjectFull: Barriers
        Type: general
      – SubjectFull: College Faculty
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Educational Opportunities
        Type: general
    Titles:
      – TitleFull: Opportunities for Educational Innovations in Authentic Project-Based Learning: Understanding Instructor Perceived Challenges to Design for Adoption
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Rees Lewis, Daniel G.
      – PersonEntity:
          Name:
            NameFull: Gerber, Elizabeth M.
      – PersonEntity:
          Name:
            NameFull: Carlson, Spencer E.
      – PersonEntity:
          Name:
            NameFull: Easterday, Matthew W.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2019
          Identifiers:
            – Type: issn-print
              Value: 1042-1629
          Numbering:
            – Type: volume
              Value: 67
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Educational Technology Research and Development
              Type: main
ResultId 1