View in EDS HTML Full Text PDF Full Text

A Learning Analytics Approach towards Understanding Collaborative Inquiry in a Problem-Based Learning Environment

Saved in:
Bibliographic Details
Title: A Learning Analytics Approach towards Understanding Collaborative Inquiry in a Problem-Based Learning Environment
Language: English
Authors: Saleh, Asmalina (ORCID 0000-0001-8178-4238), Phillips, Tanner M., Hmelo-Silver, Cindy E., Glazewski, Krista D., Mott, Bradford W., Lester, James C.
Source: British Journal of Educational Technology. Sep 2022 53(5):1321-1342.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 22
Publication Date: 2022
Sponsoring Agency: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
Contract Number: 1561655
1561486
Document Type: Journal Articles
Reports - Research
Descriptors: Cooperative Learning, Learning Analytics, Pretests Posttests, Problem Based Learning, Teaching Methods, Game Based Learning, Factor Analysis, Brainstorming, Science Instruction, Inquiry, Active Learning, Achievement Gains, Independent Study, Learning Processes, Course Content, Interaction Process Analysis
DOI: 10.1111/bjet.13198
ISSN: 0007-1013
1467-8535
Abstract: This exploratory paper highlights how problem-based learning (PBL) provided the pedagogical framework used to design and interpret learning analytics from "Crystal Island: EcoJourneys," a collaborative game-based learning environment centred on supporting science inquiry. In "Crystal Island: EcoJourneys," students work in teams of four, investigate the problem individually and then utilize a brainstorming board, an in-game PBL whiteboard that structured the collaborative inquiry process. The paper addresses a central question: how can PBL support the interpretation of the observed patterns in individual actions and collaborative interactions in the collaborative game-based learning environment? Drawing on a mixed method approach, we first analyzed students' pre- and post-test results to determine if there were learning gains. We then used principal component analysis (PCA) to describe the patterns in game interaction data and clustered students based on the PCA. Based on the pre- and post-test results and PCA clusters, we used interaction analysis to understand how collaborative interactions unfolded across selected groups. Results showed that students learned the targeted content after engaging with the game-based learning environment. Clusters based on the PCA revealed four main ways of engaging in the game-based learning environment: students engaged in low to moderate self-directed actions with: (1) high and (2) moderate collaborative sense-making actions; (3) low self-directed with low collaborative sense-making actions; and (4) high self-directed actions with low collaborative sense-making actions. Qualitative interaction analysis revealed that a key difference among four groups in each cluster was the nature of verbal student discourse: students in the low to moderate self-directed and high collaborative sense-making cluster actively initiated discussions and integrated information they learned to the problem, whereas students in the other clusters required more support. These findings have implications for designing adaptive support that responds to students' interactions with in-game activities.
Abstractor: As Provided
Entry Date: 2022
Accession Number: EJ1344679
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEKlyK94u8O1Zi4dQse36j2AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDG7o7LsgiKeaVQXtqwIBEICBmyCLXaQVMByWUIR4p80xz3N4jlZXMtsDlYDlMNki9JECTneNx0-zPr5iemSkj4EbH-YfmqCXp5ONkhpuJ7K0K0JBmxJHuljjAR-EM60fgbrYFMKHY5whzL5sF7D3vv1PVILBeWUoWjndKhsEjG9EQhIMOWbIAYfuC1alupGGIxk82XvVG2hLpxsFjQxSuT1BCdR0iEFtulharHTu
Text:
  Availability: 1
  Value: <anid>AN0158528157;58i01sep.22;2022Aug17.08:13;v2.2.500</anid> <title id="AN0158528157-1">A learning analytics approach towards understanding collaborative inquiry in a problem‐based learning environment </title> <p>This exploratory paper highlights how problem‐based learning (PBL) provided the pedagogical framework used to design and interpret learning analytics from Crystal Island: EcoJourneys, a collaborative game‐based learning environment centred on supporting science inquiry. In Crystal Island: EcoJourneys, students work in teams of four, investigate the problem individually and then utilize a brainstorming board, an in‐game PBL whiteboard that structured the collaborative inquiry process. The paper addresses a central question: how can PBL support the interpretation of the observed patterns in individual actions and collaborative interactions in the collaborative game‐based learning environment? Drawing on a mixed method approach, we first analyzed students' pre‐ and post‐test results to determine if there were learning gains. We then used principal component analysis (PCA) to describe the patterns in game interaction data and clustered students based on the PCA. Based on the pre‐ and post‐test results and PCA clusters, we used interaction analysis to understand how collaborative interactions unfolded across selected groups. Results showed that students learned the targeted content after engaging with the game‐based learning environment. Clusters based on the PCA revealed four main ways of engaging in the game‐based learning environment: students engaged in low to moderate self‐directed actions with (<reflink idref="bib1" id="ref1">1</reflink>) high and (<reflink idref="bib2" id="ref2">2</reflink>) moderate collaborative sense‐making actions, (<reflink idref="bib3" id="ref3">3</reflink>) low self‐directed with low collaborative sense‐making actions and (<reflink idref="bib4" id="ref4">4</reflink>) high self‐directed actions with low collaborative sense‐making actions. Qualitative interaction analysis revealed that a key difference among four groups in each cluster was the nature of verbal student discourse: students in the low to moderate self‐directed and high collaborative sense‐making cluster actively initiated discussions and integrated information they learned to the problem, whereas students in the other clusters required more support. These findings have implications for designing adaptive support that responds to students' interactions with in‐game activities. Practitioner notesWhat is already known about this topicLearning analytic methods have been effective for understanding student learning interactions for the purposes of assessment, profiling student behaviour and the effectiveness of interventions.However, the interpretation of analytics from these diverse data sets are not always grounded in theory and challenges of interpreting student data are further compounded in collaborative inquiry settings, where students work in groups to solve a problem.What this paper addsProblem‐based learning as a pedagogical framework allowed for the design to focus on individual and collaborative actions in a game‐based learning environment and, in turn, informed the interpretation of game‐based analytics as it relates to student's self‐directed learning in their individual investigations and collaborative inquiry discussions.The combination of principal component analysis and qualitative interaction analysis was critical in understanding the nuances of student collaborative inquiry.Implications for practice and/or policySelf‐directed actions in individual investigations are critical steps to collaborative inquiry. However, students may need to be encouraged to engage in these actions.Clustering student data can inform which scaffolds can be delivered to support both self‐directed learning and collaborative inquiry interactions.All students can engage in knowledge‐integration discourse, but some students may need more direct support from teachers to achieve this.</p> <p>Keywords: collaboration; game‐based learning; learning analytics; problem‐based learning</p> <hd id="AN0158528157-2">INTRODUCTION</hd> <p>Over the last decade, learning analytics research has developed methods for analyzing student trace data, or data derived from student interactions in online learning environments (Gašević et al., 2016). Learning analytics from game‐based learning environments provide researchers with multiple streams of data about student learning interactions for the purposes of assessment, profiling student behaviour and the effectiveness of interventions (Alonso‐Fernández, Calvo‐Morata, et al., 2019; Emerson et al., 2020; Geden et al., 2020). However, the interpretation of analytics from these data sets are not always grounded in theory, and challenges of interpreting student data are further compounded in collaborative inquiry settings, where students work in groups to solve a problem (Bell et al., 2010; Dillenbourg, 1999; Mangaroska & Giannakos, 2018). Fortunately, sociocultural pedagogical approaches such as problem‐based learning (PBL) can inform the design of game‐based learning environments and support the interpretation of learning analytics from these environments (Saleh et al., 2020).</p> <p>PBL is a student‐centered instructional approach that aims to develop students' individual and collaborative problem‐solving skills (Savery, 2019). Research in computer‐supported environments for PBL has focused on how embedded tools support learning, the role of scaffolds and the overall impact of the learning environment on learning (Kim et al., 2018; Liu et al., 2014). Although PBL can be effective in supporting learning in traditional and computer‐supported environments, there is a need to generate an explanatory learning model that maps individual and collaborative actions in game‐based learning environments to student performances (Alonso‐Fernández, Cano, et al., 2019; Archer & Prinsloo, 2020; Koedinger et al., 2012). An explanatory learner model can highlight how we might infer learner actions based on the patterns in the data (Rosé et al., 2019). Because explanatory learning models require extensive human effort, analytics provide a data‐driven approach to understand the intersection between theory and learning (Liu & Koedinger, 2017). Thus, the goals of this research are to understand how PBL (<reflink idref="bib1" id="ref5">1</reflink>) can support content learning outcomes and (<reflink idref="bib2" id="ref6">2</reflink>) guide the design and interpretation of analytics from game‐based learning environments. Ultimately, we aim to generate an initial explanatory learning model that accounts for the patterns of individual and collaborative interactions (Rosé et al., 2019). Our research questions are: (<reflink idref="bib1" id="ref7">1</reflink>) To what extent did the PBL‐informed game‐based learning environment support content learning? (<reflink idref="bib2" id="ref8">2</reflink>) How did individual and collaborative participation in the problem‐solving process differ among students?</p> <p>We first briefly describe PBL and then highlight how PBL shaped the design of the collaborative game‐based learning environment, Crystal Island: EcoJourneys, which provided a rich problem context for middle school students to learn about ecosystems. We then articulate our mixed method approach to address the research questions, highlighting how quantitative analyses informed our selection of cases for qualitative analysis. Subsequently, we report learning gains and articulated how the combination of the PBL environment and facilitators may have supported student learning before discussing the implications of our work.</p> <hd id="AN0158528157-3">Problem‐based learning</hd> <p>As a pedagogical framework, PBL supports the collaborative inquiry processes among groups of students. In PBL, students work in small groups consisting of four to seven students to solve complex, ill‐structured problems (Jonassen & Hung, 2008). In PBL, students work in small groups and engage in an inquiry process that consists of (<reflink idref="bib1" id="ref9">1</reflink>) understanding the problem scenario, (<reflink idref="bib2" id="ref10">2</reflink>) identifying learning issues (i.e., what the group needs to learn more about to solve the problem), (<reflink idref="bib3" id="ref11">3</reflink>) collecting information and identifying relevant facts, (<reflink idref="bib4" id="ref12">4</reflink>) generating and testing hypotheses and (<reflink idref="bib5" id="ref13">5</reflink>) providing explanations (Hmelo‐Silver, 2004; Tawfik & Kolodner, 2016). To be successful in collaborative learning, students must be able to engage in two essential practices: (<reflink idref="bib1" id="ref14">1</reflink>) self‐directed learning and (<reflink idref="bib2" id="ref15">2</reflink>) collaborative actions such as sharing and negotiating ideas with peers (Barrows, 1983).</p> <p>Because students face challenges linked to individual and collaborative learning processes, PBL provides several ways to support learning (Hmelo‐Silver & Eberbach, 2012; Jonassen, 2011; Kim et al., 2018; Savery, 2019). First, the phases of inquiry help students manage their self‐directed or individual learning process by allowing them to focus on specific activities during each phase such as data collection and analyzing the data (Wijnia et al., 2019). Moreover, students must reflect on their own learning and develop self‐directed learning skills (Barrows, 1983; Hmelo‐Silver, 2004). Second, the facilitator plays a critical role by encouraging group accountability to reasoning processes and ensuring that individual students respond to ideas generated by members in the classroom community (O'Connor & Michaels, 2019). Finally, when engaging with complex problems, groups make their thinking and processes visible, for example, while using a whiteboard, which provides a space for students to co‐construct knowledge and regulate collaboration (Hmelo‐Silver, 2006). At the whiteboard, students record and negotiate evolving ideas, structure their reasoning and prioritize the focus of discussion related to the problems. A typical whiteboard may include the following elements: a space to share facts, ideas, learning issues and action plans (Hmelo‐Silver & Eberbach, 2012). During the inquiry process, students negotiate what ideas need to be on the board and what ought to be removed. In our work, an in‐game PBL whiteboard called the brainstorming board, is the locus of social interactions (Saleh et al., 2020).</p> <hd id="AN0158528157-4">CRYSTAL ISLAND: ECOJOURNEYS</hd> <p>In Crystal Island: EcoJourneys, students take on the role of middle schoolers who are on a cultural exchange trip to Buglas, a fictional island in the Philippines. Students work in groups of four and learn about ecosystems and systems thinking by engaging in problem solving. Students are tasked to engage in a parallel investigation alongside the locals and reason about why fish at a local hatchery are sick. In the game‐based learning environment, students talk to in‐game characters and interact with objects to collect information related to the problem (Figure 1). Students use in‐game tools such as a task‐list, a notebook and chat to communicate with their peers.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep22/bjet13198-fig-0001.jpg?ephost1=dGJyMMvl7ESepq84yOvsOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13198-fig-0001.jpg" title="1 Overview of tools in Crystal Island: EcoJourneys" /> </p> <p></p> <hd id="AN0158528157-6">Using PBL to design and interpret learning analytics</hd> <p>To align our work with principles of PBL, we navigated a tension between structuring a complex problem with multiple necessary related elements while considering what might overburden students (Jonassen & Hung, 2008). Because ecosystems and systems thinking can be a complex phenomenon for middle school students, the design of the problem space was less ill‐structured than in typical PBL problems. For example, rather than engaging in independent investigations using web‐based or database searches as learners would in medical school contexts (Bridges et al., 2012), the middle school students were guided in their investigations by prompts from facilitators and structured in‐game activities that aligned to the PBL inquiry process (Table 1, for more details on the scaffolds, see Saleh et al., 2020). The ill‐structured problem was designed such that there are multiple paths toward a similar conclusion (Yoon et al., 2018). To support the effective interpretations of the game‐based learning analytics using PBL, we identified two key PBL inquiry processes: (<reflink idref="bib1" id="ref16">1</reflink>) self‐directed learning as part of the process of individual investigations and (<reflink idref="bib2" id="ref17">2</reflink>) jointly brainstorming ideas and discussing the information using the brainstorming board. Students in the game engaged in three iterative cycles. Each cycle began with the self‐directed investigation phase and concluded with the brainstorming phase (Table 1).</p> <p>1 TABLESequence of the PBL process in C rystal Island: EcoJourneys</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">PBL phases in game</th><th align="left">Self‐directed and collaborative inquiry practices</th></tr></thead><tbody><tr><td align="left">Phase 1.1 Self‐directed investigation 1</td><td align="left"><list list-type="Bullet"><list-item><p>Orient to the problem (i.e., fish are sick) by meeting in‐game characters</p></list-item><list-item><p>Identify learning issues related to the needs of tilapia fish</p></list-item></list></td></tr><tr><td align="left">Phase 1.2 Collaborative brainstorming board session 1 (BBS1)</td><td align="left"><list list-type="Bullet"><list-item><p>Share initial findings and conceptualize the problem</p></list-item><list-item><p>Discuss and come to consensus about ideas that may not be salient</p></list-item><list-item><p>Complete phase 1 and move to phase 2</p></list-item></list></td></tr><tr><td align="left">Phase 2.1 Self‐directed investigation 2</td><td align="left">Explore and collect more data from the game‐based learning environment to support initial ideas</td></tr><tr><td align="left">Phase 2.2 Collaborative brainstorming board session 2 (BBS2)</td><td align="left"><list list-type="Bullet"><list-item><p>Use the brainstorming board again to communicate findings, negotiate ideas, and consider the evidence</p></list-item><list-item><p>Eliminate ideas that are not salient</p></list-item><list-item><p>Complete phase 2 and move to phase 3</p></list-item></list></td></tr><tr><td align="left">Phase 3.1 Self‐directed investigation 3</td><td align="left"><list list-type="Bullet"><list-item><p>Explore and collect more data from the game‐based learning environment to support initial ideas</p></list-item></list></td></tr><tr><td align="left">Phase 3.2 Collaborative brainstorming board session 3 (BBS3)</td><td align="left"><list list-type="Bullet"><list-item><p>Use the brainstorming board again to discuss new information and connect it to prior data</p></list-item><list-item><p>Finalize a hypothesis</p></list-item></list></td></tr><tr><td align="left">Conclude</td><td align="left"><list list-type="Bullet"><list-item><p>Communicate findings and explanations to other teams</p></list-item></list></td></tr></tbody></table> </ephtml> </p> <p>During the individual investigative phases, the game narrative provided the context for students to collect evidence to address the problem. The individual or the self‐directed learning phase involves several important skills: (<reflink idref="bib1" id="ref18">1</reflink>) enacting metacognitive awareness (i.e., identifying what learners do or do not know), (<reflink idref="bib2" id="ref19">2</reflink>) establishing learning goals and (<reflink idref="bib3" id="ref20">3</reflink>) planning and selecting appropriate learning strategies (Hmelo‐Silver, 2004). To support metacognitive awareness and goal setting, students used an in‐game task list that reminds them what they need to do next (Figure 1). The task list offered students different ways of engaging in the task and helped determine what they do or do not know. To support student use of different learning strategies and knowledge construction, each student in the group was assigned one of four storylines within the game narrative, wherein students met different in‐game characters who shared different perspectives of the problem (Aronson, 2002). As part of their investigations in the game‐based learning environment, students talk to different in‐game characters and explore different parts of the island. All students were introduced to the basic concepts related to the problem. For instance, all students were introduced to the biotic (e.g., tilapia fish and cyanobacteria) and abiotic components (e.g., water and dissolved oxygen) in an aquatic ecosystem. However, students also developed individualized expertise to enable division of labour and group interdependence. For example, one student learned more about water quality, whereas another gathered additional information about dissolved oxygen. As they gathered information, students shared their findings informally using the in‐game chat or more formally as part of the brainstorming board phase.</p> <p>After individual students collected data in their investigation phases, they engaged in brainstorming sessions with their group members. As a semi‐structured collaborative space, the brainstorming board helped groups engage in complex problem‐solving processes by structuring groups' complex inquiry, supporting reasoning practices and keeping collaborative inquiry learning on track (see Figure 2). The board contained five columns, listing ideas relevant to the well‐being of the tilapia: Air, water quality, food, space, and temperature.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep22/bjet13198-fig-0002.jpg?ephost1=dGJyMMvl7ESepq84yOvsOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13198-fig-0002.jpg" title="2 The five ideas and group agreement as indicated by the colour‐coded notes. Green means that all students agree, red means there is at least one disagreement and orange means at least one student indicated that the note may be relevant to the core ideas" /> </p> <p></p> <p>At the board, students placed and sorted notes that they collected in the appropriate idea column. Students can click on these notes to examine detailed information and voted on the relevance of the note to the associated idea. If all students agreed, the note turned green. If there were disagreements, the note turned red. If any students voted 'may be relevant', the note remained orange. The board provided students and facilitators with a visual indicator of the team's current consensus about their collection of notes. Students could also initiate a vote to remove ideas from the board and other members provided explanations on whether they agreed or disagreed to remove the idea from discussion. Students' discussion was facilitated by the design of the brainstorming board and a facilitator who guided students' discussion. Consistent with PBL, the facilitator guided student inquiry by marking information that was relevant to students and prompting students to elaborate their thinking (Resnick et al., 2018; Van de Pol et al., 2010).</p> <p>In summary, PBL provided the theoretical grounding for student interactions in the game‐based learning environment by articulating actions that must be supported as part of self‐directed learning and collaborative problem‐solving. For instance, in‐game actions such as collecting information were assumed to be part of the self‐directed learning process, whereas sharing information mapped on to collaborative inquiry learning. To understand the impact of the design, we investigated these research questions: (<reflink idref="bib1" id="ref21">1</reflink>) To what extent did the PBL‐informed game‐based learning environment support content learning? (<reflink idref="bib2" id="ref22">2</reflink>) How did individual and collaborative participation in the problem‐solving process differ among students?</p> <hd id="AN0158528157-8">METHODS</hd> <p>To answer our research questions, we engaged in a mixed‐method analysis, beginning with quantitative analysis and followed by qualitative analysis. We first conducted mixed ANOVA analysis to understand individual and group ecosystems learning outcomes. Subsequently, we conducted a principal component analysis (PCA) and used the PCA results to perform a cluster analysis using k‐means clustering to explore patterns from the log files of student actions in the game‐based learning environment. Finally, qualitative interaction analysis was conducted to examine student collaboration (Jordan & Henderson, 1995).</p> <hd id="AN0158528157-9">Participants</hd> <p>This study was conducted in a rural school in Midwestern United States, where students participated in nine 55‐minutes classroom sessions. In total, 45 sixth‐grade students (11–12 years old, 23 males, 22 females, all self‐identified) consented, but only 39 had complete data (i.e., no missing log data and pre‐ and post‐tests). We used quasi‐random assignment, allocating students based on factors such as competencies in collaboration and student grades in science, reading and writing. Each group had diverse collaborative and science competencies, and similar reading and writing competencies. Students worked in groups of 4–5 and each group had a trained human facilitator to support small group work. The facilitator used the in‐game chat and delivered prompts to students that supported their collaborative inquiry work. Depending on the needs of the students, the facilitator also engaged in face‐to‐face discussion to clarify confusion (For details of how facilitators scaffolded the learning process, please see Saleh et al., 2020).</p> <hd id="AN0158528157-10">Procedures</hd> <p>Before playing the game, students completed a pre‐test. During the second session, students discussed group norms to define collaboration and signed a group contract. For the next six sessions, students engaged in Crystal Island: EcoJourneys. During the last session, students created written explanations as to why the tilapia were sick and completed a post‐test.</p> <hd id="AN0158528157-11">Sources of data</hd> <p>There were four main sources of data: pre‐ and post‐ content tests, video data of group interactions, written artifacts, and game interaction data. To determine if students learned the targeted content, students took the same pre‐ and post‐test. There were a total of 13 questions measuring ecosystems content understanding in the test. The test consisted of two multiple‐choice questions, five fill‐in‐the‐blank questions, one performance‐based question, three short‐answer and two open‐ended questions. The items were derived from NAEP test banks and prior studies (Hmelo‐Silver et al., 2017; Jordan et al., 2014) and externally validated through review by an ecosystems expert. This external reviewer then evaluated and provided feedback on the structure, language and potential student responses to rule out construct‐irrelevant features and confirm the alignment of items with desired student competencies. The maximum possible score for the test was 42 points. Students scored 1 point for each correct answer whereas short and open‐ended questions were scored based as accurate (<reflink idref="bib2" id="ref23">2</reflink>), partially accurate (<reflink idref="bib1" id="ref24">1</reflink>) or inaccurate (0). The performance‐based item accounted for 15 points (Figure 3). Given that students had limited exposure to these ecosystem processes, we expected that students would score points for demonstrating relationships but would otherwise struggle with identifying the processes. To measure the extent to which the items on the test were interrelated, we used Cronbach's alpha (Cronbach, 1951). Cronbach's alpha for the test was 0.70, an acceptable value indicating the equivalence of the items (Taber, 2018).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep22/bjet13198-fig-0003.jpg?ephost1=dGJyMMvl7ESepq84yOvsOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13198-fig-0003.jpg" title="3 Sample questions from the pre‐ and post‐tests, the model‐based question (Q8) and open‐ended question (Q12)" /> </p> <p></p> <p>Using convenience sampling, video data was collected from 6 out of the 11 groups. Because the students were in a classroom environment and audio data was difficult to capture, groups were chosen based on the best video and audio quality that could be recorded. Audio data was transcribed and will be referred to as "verbal discourse data" to distinguish it from the in‐game chat text (see the game log data). Written artifacts consisted of pen and paper worksheets that guided student inquiry and collaboration and group scientific models that explained why the problem is occurring.</p> <p>Game log file data was collected from all groups who engaged in the game‐based learning environment. There were 12 distinct types of actions captured, grouped into three forms of in‐game actions (Table 2). In the investigation and brainstorming phases, we assume that the viewing and closing of notes meant that students have reviewed or ideally have read the information in their note. A total of over 40,000 individual actions were captured in the log files.</p> <p>2 TABLEOverview of in‐game actions and associated phases</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left">Phases</th><th align="left">In‐game or trace data action</th></tr></thead><tbody><tr><td align="left">Investigation</td><td align="left">View and close list of notes</td></tr><tr><td align="left">View and close task list</td></tr><tr><td align="left">View notes by speaking to in game characters and objects</td></tr><tr><td align="left">Brainstorming</td><td align="left">Close note after viewing detailed information</td></tr><tr><td align="left">Share note</td></tr><tr><td align="left">Delete note</td></tr><tr><td align="left">Vote on note</td></tr><tr><td align="left">Move notes to the appropriate column</td></tr><tr><td align="left">In‐game chat use in both phases</td><td align="left">Receive chat messages</td></tr><tr><td align="left">Send chat messages</td></tr><tr><td align="left">Close chat application after viewing</td></tr></tbody></table> </ephtml> </p> <p>To understand student engagement during the problem‐solving process, we tabulated game summary statistics from students' in‐game log file data and identified two units of analysis, individual and group. For individual engagement, this included the (<reflink idref="bib1" id="ref25">1</reflink>) total time spent talking to in‐game characters which included viewing and reading the information provided, (<reflink idref="bib2" id="ref26">2</reflink>) time spent viewing and reading tasks and (<reflink idref="bib3" id="ref27">3</reflink>) time spent viewing and reading notes when using the brainstorming board (Table 2). We assumed that reading was a largely individual activity although students did read notes aloud to one another.</p> <p>Collaborative participation included group aggregates for (<reflink idref="bib1" id="ref28">1</reflink>) mean time spent on the board, (<reflink idref="bib2" id="ref29">2</reflink>) the mean votes for each note, (<reflink idref="bib3" id="ref30">3</reflink>) time on chat and (<reflink idref="bib4" id="ref31">4</reflink>) number of chat lines. Taken together, these indicators provided an overview of how much time each group spent talking about the ideas presented in the notes, justifying their actions at the board, and the extent to which the voting feature may have triggered these discussions (i.e., collaborative sense‐making). The mean number of votes on each note was an indication of how often students voted for the relevance of the notes to the specific idea. A higher count of votes meant students likely discussed the note, which resulted in the changes in votes. On the other hand, a lower count might have meant that students came to agreement quicker.</p> <hd id="AN0158528157-13">Data analysis</hd> <p>To analyze the data, we used the following stats packages in R: ggplot2 (for visualization, Wickham, 2016), psych (for descriptive statistics, Revelle, 2021) and functions in the stats package: aov, princomp and kmeans (R Core Team, 2021). The bootstrapping was done manually, with random sampling with replacement from data and using princomp() on all bootstrapped samples. Results were then ordered and displayed as quartiles. Below, we specify these analyses in detail.</p> <p>RQ1. To what extent did the PBL‐informed game‐based learning environment support content learning?</p> <p>We hypothesize that students would learn the targeted content and that groups would engage in the game‐based learning environments differently. We conducted a mixed ANOVA, with time as a within‐subject factor (pretest and post‐test), and groups as a between‐subject factor (ID: A to K). The assumptions of sphericity, homogeneity of variances and homogeneity of covariances were not violated. However, the normality assumption was violated for the pretest scores. Student assignment was quasi‐random because of challenges with classroom management. Given that the mixed ANOVA model is robust to slight violations of the normality assumptions and the other assumptions were met, the test was conducted but we interpret the results with caution (Blanca et al., 2017; Kirk, 2013). The pre‐ and post‐test scores were then used to understand group performances.</p> <p>RQ2. How did individual and collaborative engagement in the problem‐solving process differ between students?</p> <p>To explore the relationship between individual and collaborative actions, the frequency of individual actions across different students were tabulated (Table 2). Because of the large quantity of features and possible observations in the data, PCA was used to reduce the complexity of the large student log files and search for general patterns in student activity. PCA extracts the most vital characteristics from the data and compresses the information using principal components, or linear combinations of the original variables (Abdi & Williams, 2010). The first principal component accounts for the highest amount of variance in the data. The second principal component is orthogonal to the first principal component and must have the largest spread of data. Thus, PCA simplifies the data set, allows for an analysis of structures underlying student actions and for variables to load proportionally across multiple components. Because our data consist of actions that are highly correlated (i.e., reading a note, voting on a note), PCA was preferred over factor analysis because it is more robust to highly correlated variables (Joliffe & Morgan, 1992). PCA also selects components based on patterns inherent in the data, agnostic of theory. This approach controlled for biases that can be present in the feature‐selection process due to specialized domain knowledge (Wu et al., 2014). Although PCA with small sample sizes is feasible, it is sensitive to minor changes in the data. Thus, bootstrapping was utilized to ensure convergence to stable factor loadings (Babamoradi et al., 2013).</p> <p>k‐means clustering was then used to create distinct student groups based on the principal components (see Jain, 2010 for brief history of the use of k‐means across multiple disciplines). k‐means was chosen because of the relative simplicity of the underlying algorithm, which makes it easy to understand and is applicable across a variety of data sets, even when data is nonparametric or difficult to interpret (Fix & Hodges, 1989). For this study, the silhouettes measure of cluster homogeneity was used as a measure of cluster quality (Rousseeuw, 1987). The silhouettes value refers to how similar an object is to its own cluster when compared to other clusters. A value of close to 1 means that that the objects are matched well to its associated clusters. In all cases, adding a cluster will increase the silhouette score. When selecting the number of clusters, we settled on four because (a) adding a fifth cluster did not increase the silhouette score by as much as adding a fourth, and (b) a five‐cluster model did not appear to offer any additional substantive insight, but simply split an existing group into two (see Figure S1 in Supporting Information comparing the models). After clustering, the clusters were compared across three types of activities: (<reflink idref="bib1" id="ref32">1</reflink>) individual learning phase actions, (<reflink idref="bib2" id="ref33">2</reflink>) collaborative actions during the brainstorming phase and (<reflink idref="bib3" id="ref34">3</reflink>) in‐game chat actions. For the PCA, only the in‐game chat information is used, and the verbal discourse data is used to augment the PCA analysis, as we describe below.</p> <p>The clustering analysis helped identify contrasting cases for interaction analysis (Jordan & Henderson, 1995). Interaction analysis is a qualitative method that focuses on how knowledge construction can be observed in social activities (Hall & Stevens, 2016). As a method, it allows for repeated analysis of multiple streams of data, such as the audio‐video data captured from 6 out of the 11 groups, written artifacts and log file data, to illuminate how students from the clusters engaged in the problem‐solving process. This process includes viewing all available data, creating a content log or descriptions of what occurred in each group, and triangulating the information with data from the log files (i.e., game analytics). Because the PBL inquiry process defined specific phases of interaction, we examined group interaction by examining audio‐video and in‐game chat data, focusing on the temporal order of talk and how actions (discursive and log‐file actions) contributed to collaborative inquiry. We thematized student actions when using the brainstorming board based on the collaborative problem‐solving and inquiry learning literature, (<reflink idref="bib1" id="ref35">1</reflink>) sharing and sorting notes by associating them with the appropriate idea, (<reflink idref="bib2" id="ref36">2</reflink>) negotiating by voting on the relevance of notes to the ideas, (<reflink idref="bib3" id="ref37">3</reflink>) discussing the content of the notes, (<reflink idref="bib4" id="ref38">4</reflink>) negotiating relevance of the notes and (<reflink idref="bib5" id="ref39">5</reflink>) discussing and/or eliminating irrelevant ideas (Liu et al., 2016; Pedaste et al., 2015). In our discussion of the groups, each student was provided with a unique identifier (i.e., Eagle, Jeepney, Sun and Turtle) and a suffix that identified which team the student worked in. Thus, students in group A will be identified as Eagle‐A and so on.</p> <hd id="AN0158528157-14">RESULTS</hd> <p></p> <hd id="AN0158528157-15">RQ1. To what extent did the PBL‐informed game‐based learning environment support content lear...</hd> <p>A mixed analysis of variance with groups as between‐subjects and time as within‐subjects factors revealed a main effect of time. Students scored significantly better in their post‐tests, <emph>F</emph> (<reflink idref="bib1" id="ref40">1</reflink>, 37) = 13.36, <emph>p</emph> = 0.009 (pre‐test mean = 13.6, SD = 3.92; post‐test mean = 15.64, SD = 3.54), η<subs>p</subs><sups>2</sups> = 0.409. There was neither a significant main effect for groups (<emph>F</emph> (<reflink idref="bib1" id="ref41">1</reflink>, 10) = 0.715, <emph>p</emph> = 0.703, η<subs>p</subs><sups>2</sups> = 0.203), nor an interaction among group and time (<emph>F</emph> (1.10) = 1.26, <emph>p</emph> = 0.297, η<subs>p</subs><sups>2</sups> = 0.311). This suggests that engagement in the game‐based learning environment supported groups in learning ecosystems content. Table 3 provides an overview of pre‐ and post‐test scores and in‐game collaborative actions for each group.</p> <p>3 TABLESummary statistics for all groups' log file data actions and pre‐ and post‐tests</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Group</th><th align="left">Pre‐test mean</th><th align="left">Post‐test mean</th><th align="left">No. of notes</th><th align="left">Individual interactions</th><th align="left">Collaborative interactions</th></tr><tr><th align="left">Mean time spent per note (minutes)</th><th align="left">Total time investigating (minutes)</th><th align="left">Total time at the board (minutes)</th><th align="left">Total votes</th><th align="left">Mean votes per note</th><th align="left">Lines of chat</th><th align="left">Time on chat (minutes)</th></tr></thead><tbody><tr><td align="left">A<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">11.5</td><td align="char" char=".">15.3</td><td align="char" char=".">26</td><td align="char" char=".">0.5</td><td align="char" char=".">54</td><td align="char" char=".">118</td><td align="char" char=".">201</td><td align="char" char=".">9</td><td align="char" char=".">387</td><td align="char" char=".">59</td></tr><tr><td align="left">B<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">13.8</td><td align="char" char=".">16</td><td align="char" char=".">27</td><td align="char" char=".">0.6</td><td align="char" char=".">60</td><td align="char" char=".">75</td><td align="char" char=".">496</td><td align="char" char=".">5</td><td align="char" char=".">985</td><td align="char" char=".">157</td></tr><tr><td align="left">C<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">15</td><td align="char" char=".">19</td><td align="char" char=".">29</td><td align="char" char=".">0.7</td><td align="char" char=".">55</td><td align="char" char=".">109</td><td align="char" char=".">223</td><td align="char" char=".">5</td><td align="char" char=".">413</td><td align="char" char=".">131</td></tr><tr><td align="left">D<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">17.5</td><td align="char" char=".">18.5</td><td align="char" char=".">29</td><td align="char" char=".">0.7</td><td align="char" char=".">69</td><td align="char" char=".">46</td><td align="char" char=".">335</td><td align="char" char=".">7</td><td align="char" char=".">289</td><td align="char" char=".">108</td></tr><tr><td align="left">E<xref ref-type="fn" rid="tfn3" /></td><td align="char" char=".">13.3</td><td align="char" char=".">12</td><td align="char" char=".">26</td><td align="char" char=".">1.2</td><td align="char" char=".">83</td><td align="char" char=".">85</td><td align="char" char=".">237</td><td align="char" char=".">9</td><td align="char" char=".">524</td><td align="char" char=".">111</td></tr><tr><td align="left">F<xref ref-type="fn" rid="tfn3" /></td><td align="char" char=".">15.7</td><td align="char" char=".">15.7</td><td align="char" char=".">26</td><td align="char" char=".">1.3</td><td align="char" char=".">37</td><td align="char" char=".">123</td><td align="char" char=".">308</td><td align="char" char=".">8</td><td align="char" char=".">422</td><td align="char" char=".">154</td></tr><tr><td align="left">G<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">11.8</td><td align="char" char=".">14</td><td align="char" char=".">26</td><td align="char" char=".">1.3</td><td align="char" char=".">77</td><td align="char" char=".">78</td><td align="char" char=".">288</td><td align="char" char=".">11</td><td align="char" char=".">272</td><td align="char" char=".">95</td></tr><tr><td align="left">H<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">13.5</td><td align="char" char=".">14.8</td><td align="char" char=".">29</td><td align="char" char=".">1.4</td><td align="char" char=".">40</td><td align="char" char=".">119</td><td align="char" char=".">143</td><td align="char" char=".">17</td><td align="char" char=".">347</td><td align="char" char=".">69</td></tr><tr><td align="left">I<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">13.5</td><td align="char" char=".">18</td><td align="char" char=".">26</td><td align="char" char=".">1.5</td><td align="char" char=".">48</td><td align="char" char=".">110</td><td align="char" char=".">236</td><td align="char" char=".">12</td><td align="char" char=".">528</td><td align="char" char=".">175</td></tr><tr><td align="left">J<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">12</td><td align="char" char=".">16.3</td><td align="char" char=".">29</td><td align="char" char=".">1.6</td><td align="char" char=".">45</td><td align="char" char=".">107</td><td align="char" char=".">134</td><td align="char" char=".">11</td><td align="char" char=".">180</td><td align="char" char=".">73</td></tr><tr><td align="left">K<xref ref-type="fn" rid="tfn1" /></td><td align="char" char=".">12.3</td><td align="char" char=".">15</td><td align="char" char=".">17</td><td align="char" char=".">2.6</td><td align="char" char=".">28</td><td align="char" char=".">118</td><td align="char" char=".">310</td><td align="char" char=".">20</td><td align="char" char=".">417</td><td align="char" char=".">153</td></tr><tr><td align="left">Grand mean</td><td align="char" char=".">13.4</td><td align="char" char=".">15.6</td><td align="char" char=".">26.6</td><td align="char" char=".">1.2</td><td align="char" char=".">54</td><td align="char" char=".">99</td><td align="char" char=".">265</td><td align="char" char=".">10.4</td><td align="char" char=".">433</td><td align="char" char=".">116.8</td></tr></tbody></table> </ephtml> </p> <p>1 a Above‐average gain band.</p> <ulist> <item>2 b Average gain band.</item> <item>3 c No improvement.</item> </ulist> <p>Based on Table 3, the overall average improvement was 2.2 points (difference between grand mean in the pre‐ and post‐tests). If the improvement of the group was at least 2.2, these groups were in the above‐average improvement band. The above‐average improvement band consisted of seven groups. If the groups had less than 2.2 improvements in their scores, but above the grand mean, they were categorized as the average improvement band (Groups D and H). Finally, if groups had no improvement, they fell into the no improvement band (Groups E and F). It is worth noting that students in Group F (no improvement band) scored higher than the mean in the pre‐test and near the mean of the post‐test, which may indicate that the students may have better content understanding to begin with. On the other hand, two students in Group I scored lower in their post‐test, bringing the group average down. When comparing the groups in the above‐average band to the groups in the average and no improvement bands together, groups in the above‐average improvement band spent <emph>more</emph> time at the board (about 8 minutes) and on chat (10 minutes more, 59 more chat lines), but less time during the investigation phase. However, groups in all bands spent about the same amount of time reading their notes and had similar voting patterns, while using the brainstorming board. The in‐game summary statistics suggests that groups in the above‐average improvement band may be more deliberate in their discussions, as we will highlight later.</p> <hd id="AN0158528157-16">RQ2. How did individual and collaborative participation in the problem‐solving process suppor...</hd> <p>Bootstrapping revealed that although there were differences in the magnitude of loadings across samples, the directionality and general magnitudes of loadings was preserved across samples. Table 4 includes quantiles for the bootstrapped PCA and loadings for the full data.</p> <p>4 TABLEPrinciple component analysis loadings and interquartile ranges from the bootstrap analysis</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Actions</th><th align="left">PC1: collaborative sense‐making</th><th align="left">PC2: self‐directed actions</th></tr><tr><th align="left">Loading based on data</th><th align="left">1st quartile bootstrap</th><th align="left">3rd quartile bootstrap</th><th align="left">Loading based on data</th><th align="left">1st quartile bootstrap</th><th align="left">3rd quartile bootstrap</th></tr></thead><tbody><tr><td align="left">Move notes to idea column</td><td align="char" char=".">0.17</td><td align="char" char=".">0.11</td><td align="char" char=".">0.20</td><td align="char" char=".">0.25</td><td align="char" char=".">0.13</td><td align="char" char=".">0.40</td></tr><tr><td align="left">Close chat application</td><td align="char" char=".">0.06</td><td align="char" char=".">0.03</td><td align="char" char=".">0.11</td><td align="char" char=".">0.46</td><td align="char" char=".">0.33</td><td align="char" char=".">0.51</td></tr><tr><td align="left">Close list of notes</td><td align="char" char=".">0</td><td align="char" char=".">0</td><td align="char" char=".">0</td><td align="char" char=".">0.13</td><td align="char" char=".">0.11</td><td align="char" char=".">0.15</td></tr><tr><td align="left">Close notes after viewing</td><td align="char" char=".">−0.11</td><td align="char" char=".">−0.16</td><td align="char" char=".">−0.05</td><td align="char" char=".">0.73</td><td align="char" char=".">0.56</td><td align="char" char=".">0.72</td></tr><tr><td align="left">Close task list</td><td align="char" char=".">−0.02</td><td align="char" char=".">0.01</td><td align="char" char=".">0.02</td><td align="char" char=".">0.08</td><td align="char" char=".">0.05</td><td align="char" char=".">0.09</td></tr><tr><td align="left">Share notes</td><td align="char" char=".">0</td><td align="char" char=".">0</td><td align="char" char=".">0.02</td><td align="char" char=".">0.01</td><td align="char" char=".">0.01</td><td align="char" char=".">0.01</td></tr><tr><td align="left">Delete idea</td><td align="char" char=".">−0.01</td><td align="char" char=".">−0.01</td><td align="char" char=".">0</td><td align="char" char=".">0.01</td><td align="char" char=".">0</td><td align="char" char=".">0.01</td></tr><tr><td align="left">Move to location</td><td align="char" char=".">−0.01</td><td align="char" char=".">−0.02</td><td align="char" char=".">−0.00</td><td align="char" char=".">0.12</td><td align="char" char=".">0.05</td><td align="char" char=".">0.16</td></tr><tr><td align="left">Receive chat message</td><td align="char" char=".">0.93</td><td align="char" char=".">0.90</td><td align="char" char=".">0.94</td><td align="char" char=".">0.09</td><td align="char" char=".">0.04</td><td align="char" char=".">0.15</td></tr><tr><td align="left">Send chat message</td><td align="char" char=".">0.29</td><td align="char" char=".">0.23</td><td align="char" char=".">0.31</td><td align="char" char=".">0.05</td><td align="char" char=".">0.03</td><td align="char" char=".">0.12</td></tr><tr><td align="left">View notes while investigating</td><td align="char" char=".">0.04</td><td align="char" char=".">0.02</td><td align="char" char=".">0.07</td><td align="char" char=".">0.14</td><td align="char" char=".">0.07</td><td align="char" char=".">0.17</td></tr><tr><td align="left">Voted on notes</td><td align="char" char=".">−0.05</td><td align="char" char=".">−0.07</td><td align="char" char=".">−0.03</td><td align="char" char=".">0.34</td><td align="char" char=".">0.27</td><td align="char" char=".">0.35</td></tr><tr><td align="left">Total variance:</td><td align="char" char=".">56%</td><td align="char" char=".">17%</td></tr></tbody></table> </ephtml> </p> <p>4 Loadings over |0.2| are bolded.</p> <p>The PCA revealed that component 1, which we refer to as <emph>collaborative sense‐making</emph>, accounts for a total 56% of the variance in the data. This component is a combination of receiving and sending chat messages, moving notes to the columns on the brainstorming board, and coincides with moving notes the notes to the board (Table 4). These actions suggest that this component is an indicator of collaborative interactions at the brainstorming board that may centred on sense‐making. The negative loading on reading and the positive loading on sending and receiving chat messages also indicate that students do not have their notes open when they are chatting with their peers. This is likely because in the current design, students must view the detailed notes and then close them before using the chat app to talk to their peers. However, the smaller load on moving notes and the higher loads of sending and receiving messages indicate that students may be discussing the relevance of the notes to the ideas on the board.</p> <p>The second component, which we have named <emph>self‐directed actions</emph> accounts for a total of 17% of the variance is loaded across (<reflink idref="bib1" id="ref42">1</reflink>) actions at the board, which includes closing notes after viewing them (highest load), closing chat application after viewing, voting on notes, and moving notes, as well as (<reflink idref="bib2" id="ref43">2</reflink>) individual actions such as closing the list of notes while investigating, viewing chat messages, moving to locations and speaking to in‐game characters. Because all loadings are positive, this component is most likely a combination of individual activity in the game‐based learning environment. The loadings reveal that PBL provided a useful framework for how in‐game actions can be meaningfully designed and interpreted to account for the contexts of learning and accounting for how individual students interact with different activities and tasks across time and with other students (Han et al., 2021; Zimmermann et al., 2007).</p> <hd id="AN0158528157-17">k‐means clusters</hd> <p>Based on the first two principal components, a k‐means cluster was performed to search for distinct student clusters. Figure 4 illustrates cluster membership by group and student improvement on the post‐test.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep22/bjet13198-fig-0004.jpg?ephost1=dGJyMMvl7ESepq84yOvsOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13198-fig-0004.jpg" title="4 Clusters by principal components. Letters indicate group assignment of individual students. Outlined shapes indicate student membership in the clusters whereas coloured areas indicate their membership in the above‐average (large green area), average (medium blue area) and no improvement (thin red area) bands" /> </p> <p></p> <p>When viewing student actions across principal components, there is a high level of homogeneity within groups despite the data being considered individually. This suggests that each group settles into a set of norms dependent on their group members. Based on the self‐directed actions (SDA) and collaborative sense‐making (CS) principal components, we identified four clusters, which are highly group‐dependent (Figure 4). Students in Clusters 1 and 2 had low to moderately SDA with (<reflink idref="bib1" id="ref44">1</reflink>) high and (<reflink idref="bib2" id="ref45">2</reflink>) moderate CS, whereas students in Cluster 3 had low SDA and low CS. Finally, Cluster 4 consisted of students with high SDA and low CS. We expected groups in the above‐average improvement band to have higher load of collaborative sense‐making, and thus, were curious as to why students in groups in the above‐average improvement band had lower collaborative sense‐making (Figure 4).</p> <hd id="AN0158528157-19">Interaction analysis</hd> <p>To understand this and develop a more nuanced understanding of collaborative participation during the brainstorming session, we conducted qualitative interaction analysis (Jordan & Henderson, 1995). Interaction analysis involves evaluating all corpus of available data to understand how students interacted in their activities (Hall & Stevens, 2016). Out of the six groups that were selected for video capture, two groups were from the no improvement cluster (groups E & F), two groups were from the average improvement cluster (groups D & H), and the last two were from the above‐average improvement cluster (Groups B & G). We first created descriptive logs of student interactions by integrating all available corpus for these six groups. After reviewing the data, we narrowed the analysis to four groups from each cluster:</p> <p></p> <ulist> <item> Group B (above‐average improvement, Cluster 1: low to moderate SDA, high CS),</item> <p></p> <item> Group E (no improvement, Cluster 2: low to moderate SDA, moderate CS),</item> <p></p> <item> Group G (above‐average improvement, Cluster 3: low SDA, low CS) and</item> <p></p> <item> Group H (average improvement, Cluster 4: high SDA, low CS).</item> </ulist> <p>Before discussing the interaction analysis, we briefly unpack the differences between Group H and groups with low to moderate SDA. Compared to the four groups that were chosen for interaction analysis, Group H likely had higher SDA loadings because the students in the group averaged 17 votes on each note, spent 1.4 minutes on the notes. Groups E and G, respectively, spent 1.2 and 1.3 minutes on each note and averaged 9 and 11 votes for each note. On the other hand, the lower SDA loadings for students in Group B could be because one of the students in the group was absent for two days. Group B also averaged 5 votes and spent only 0.6 minutes on each note. Furthermore, Group B had 985 lines of chat compared to Group E, G and H (<reflink idref="bib524" id="ref46">524</reflink>, 272 and 347 lines, respectively). The differences in the use of notes may indicate that the students in Group H (i.e., high SDA) were likely to work independently at the board by individually reading the notes.</p> <p>Despite the apparent differences in SDA and CS loadings, qualitative interaction analysis of these four groups revealed that they engaged in <emph>verbal and text‐based</emph> scientific discussions as a group (see Supporting Information for transcripts and analysis of groups E and H). In all the groups, the facilitator provided prompts to help them with their collaborative sense‐making. A key difference among the groups, however, is the extent to which the students took on responsibility for their learning (Belland, 2011). This was characterized by two observations in group discourse, the initiation and presence of student‐generated questions and nature of problem solving. Below, we present illustrative examples from Group B and G, as they engage in the second brainstorming board session. We chose Group B to contrast with Group G since both groups had similar above‐average scores yet have differences in their collaborative sense‐making (i.e., low vs. high).</p> <p>Students in Group G scored above‐average in their post‐test and were in Cluster 3, with low SDA and low CS. However, similar to the groups that we qualitatively analyzed, these groups often engaged in verbal discussions (Table 5 and excerpts in Supporting Information). These conversations were also facilitator‐led (see Group B for exception). The time spent on verbal discussions may explain why Group G spent only 78 minutes at the board and spent 95 minutes in chat. Based on the video analysis, the students sometimes closed their laptops and used their peers' screen to discuss ideas. In Group G and other groups, students responded to facilitator prompts that focus on explanations, or knowledge integration discourse (King, 1994). This discourse typically involved one student in dialogue with the facilitator, with other students observing and listening in (Table 5, lines 1–9). However, this initial discussion typically allows the students to then consider these concepts further and negotiate the meaning of these notes in relation to the problem (Table 5, lines 11–15). This suggests that facilitator questions play a large role in supporting learning. Comparatively, students in Group B often <emph>initiated</emph> sharing ideas and posing questions (Table 6). Recall that students in Group B scored above‐average in their post‐tests and was in Cluster 4, low to moderate SDA and high CS.</p> <p>5 TABLEFacilitator‐led discussion in Group G with knowledge integration‐type discourse</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" /><th align="left">Speaker</th><th align="left">Verbal discussion</th></tr></thead><tbody><tr><td align="left">1</td><td align="left">Facilitator‐G</td><td align="left">Okay, so I know that there were questions last time about the space</td></tr><tr><td align="left">2</td><td align="left">Sun‐G</td><td align="left">Why is that good, though?</td></tr><tr><td align="left">3</td><td align="left">Facilitator‐G</td><td align="left">Okay, let's talk about it</td></tr><tr><td align="left">4</td><td align="left">Sun‐G</td><td align="left">Because, like, if it's crowded, I don't like it</td></tr><tr><td align="left">5</td><td align="left">Facilitator‐G</td><td align="left">Okay. But what does the note say? Did you open the note?</td></tr><tr><td align="left">6</td><td align="left">Sun‐G</td><td align="left">One second ...</td></tr><tr><td align="left">7</td><td align="left">Facilitator‐G</td><td align="left">You can read it out loud. It's okay</td></tr><tr><td align="left">8</td><td align="left">Sun‐G</td><td align="left">Alright, each of those [reads note] ... tilapia can tolerate overcrowding. So they're the same as before</td></tr><tr><td align="left">9</td><td align="left">Facilitator‐G</td><td align="left">Yeah! Whether they're crowded or not, they're the same. So, does that mean the space is important?</td></tr><tr><td align="left">10</td><td align="left">Sun‐G</td><td align="left">Not really</td></tr><tr><td align="left">11</td><td align="left">Turtle‐G</td><td align="left">It doesn't mean it's not relevant, or like ...</td></tr><tr><td align="left">12</td><td align="left">Sun‐G</td><td align="left">Because, like, they don't need space. They're the same with or without it</td></tr><tr><td align="left">13</td><td align="left">Eagle‐G</td><td align="left">But still, it should go in space because they're talking about how they are healthy their way</td></tr><tr><td align="left">14</td><td align="left">Facilitator‐G</td><td align="left">Alright, but is space even relevant to the fish?</td></tr><tr><td align="left">15</td><td align="left">Eagle‐G</td><td align="left">No</td></tr></tbody></table> </ephtml> </p> <p>6 TABLEStudent‐led discussion in Group B with comprehension and integration‐type discourse</p> <p> <ephtml> <table><thead valign="top"><tr><th align="left" /><th align="left">Speaker</th><th align="left">In‐game chat</th></tr></thead><tbody><tr><td align="left">1</td><td align="left">Jeepney‐B</td><td align="left">Okay so the aerator produces oxygen for the Tilapia</td></tr><tr><td align="left">2</td><td align="left">Sun‐B</td><td align="left">Sure the name is helpful but, is it that important?</td></tr><tr><td align="left">3</td><td align="left">Jeepney‐B</td><td align="left">I think that has everything to do with this</td></tr><tr><td align="left">4</td><td align="left">Sun‐B</td><td align="left">To the story I mean</td></tr><tr><td align="left">5</td><td align="left">Jeepney‐B</td><td align="left">Because if it's not working, that is an issue. And we are trying to solve the issue</td></tr><tr><td align="left">6</td><td align="left">Sun‐B</td><td align="left">Knowing the fishes names?</td></tr><tr><td align="left">7</td><td align="left">Turtle‐B</td><td align="left">?</td></tr><tr><td align="left">8</td><td align="left">Jeepney‐B</td><td align="left">The aerator is the thing providing the Telapia with oxygen‐ i think</td></tr><tr><td align="left">9</td><td align="left">Jeepney‐B</td><td align="left">Since the card mentions that now that it works</td></tr><tr><td align="left">10</td><td align="left">Sun‐B</td><td align="left">Or is it a fish?</td></tr><tr><td align="left">11</td><td align="left">Turtle‐B</td><td align="left">So like a oxygen filter</td></tr><tr><td align="left">12</td><td align="left">Jeepney‐B</td><td align="left">That there are air bubbles providing dissolved oxygen</td></tr><tr><td align="left">13</td><td align="left">Turtle‐B</td><td align="left">But the opposite</td></tr><tr><td align="left">14</td><td align="left">Jeepney‐B</td><td align="left">Click on the card, Sun</td></tr><tr><td align="left">15</td><td align="left">Sun‐B</td><td align="left">Grand wizard, is the Aerators a fish? (Reads note) ooh nvm</td></tr><tr><td align="left">16</td><td align="left">Jeepney‐B</td><td align="left">It is not. Lol</td></tr></tbody></table> </ephtml> </p> <p>Discourse in Group B was typically student‐led, with students generating questions that focus on knowledge integration during each of their brainstorming board session. Students in Group B were comfortable leading discussions with limited or no prompting from Facilitator B. The students also typically began discussions with descriptions of observable phenomenon (lines 1 and 3–4), "what" questions (lines 2 and 4, 15), which is indicative of comprehension‐level discourse (King, 1994). Student discussion was also characterized by knowledge‐integration or "why" questions that focused on making connections from the scientific concepts to the problem (lines 8–9), informed one another where the information could be found (line 14), and provided diverse perspectives and ideas about the topic of discussion (lines 10, 11, 13).</p> <hd id="AN0158528157-20">DISCUSSION</hd> <p>We explored the extent to which a PBL‐informed game‐based learning environment supported content learning and how individual and collaborative participation may differ. Results indicated that although students learned the content, groups that had higher improvement in their post‐tests spent more time collaborating and may have adopted more responsibility for their learning by engaging in productive discourse (Belland, 2011). When factoring the different ways of participating in the game, groups that appeared to be less collaborative based on their in‐game actions typically engaged in more verbal discussions which may include verbal support from their facilitators. Based on our findings, there are three key implications for the use of PBL in the design of collaborative inquiry and in interpreting learning analytics.</p> <p>First, our study illustrates how leveraging PBL in a game‐based learning environment can support student learning outcomes. Notably, because of the limited sample size, we are cautious in our claims and future work will need to determine what factors can predict learning gains. Regardless, it is challenging to implement PBL in the K‐12 classroom because of the amount of instructional support that teachers need to provide (Glazewski & Hmelo‐Silver, 2019). Students appeared to need additional support in collaborative sense‐making, especially in knowledge‐integration and comprehension discourse. For our learning environment to be usable at scale (i.e., without facilitators), students must assume responsibility for their learning. Fortunately, microscripts centred on comprehension and knowledge‐integration questions can be embedded in our chat tool to prompt student conversations (Kollar et al., 2018). Our study also highlights that students may not engage in desired self‐directed learning as they explore the learning environment. In our next iteration, we are implementing an adaptive collaborative problem‐solving system that supports comprehensive and knowledge‐integration discourse and expanding the self‐directed learning process to include individual sense‐making supported by peer interactions.</p> <p>Second, as an explanatory learning model, PBL provides interpretable and actionable results interpretation of learning analytics (Martinez‐Maldonado et al., 2021). This may take the form of fully automating the real‐time evaluation of collaborative analytics using PCA. The clustering of the PCA results indicate four distinct clusters, low to moderate SDA with (<reflink idref="bib1" id="ref47">1</reflink>) high and (<reflink idref="bib2" id="ref48">2</reflink>) moderate CS, (<reflink idref="bib3" id="ref49">3</reflink>) low SDA with low CS and (<reflink idref="bib4" id="ref50">4</reflink>) high SDA with low CS. These clusters can be used to diagnose the quality of individual and collaborative sense‐making, which then allows teachers to support groups. Because verbal support appears be a factor in supporting student learning, groups that focus mainly on individual tasks or engaged in low collaborative sense‐making may require support from the teacher. Such information can be actionable if provided to teachers in real‐time, such as via an informative dashboard (van Leeuwen et al., 2019). Given that the group profiles are somewhat varied, they provide a more nuanced view of learning. This in turn does not privilege normative ideas of what good collaborative learning might look like (Rummel et al., 2016; Wise et al., 2021).</p> <p>Finally, although PBL has often been used with multimedia (Liu et al., 2014; Su & Klein, 2010), these interactions are not centred on an online collaborative problem‐space, which can be messy and challenging to analyze. In this study, we adopt a mixed method approach to better understand student interactions with tools and visualizations of their participation. We found that each analysis provided additional insights into aspects of collaborative inquiry. For example, focusing on pre‐ and post‐test results suggested that all students learned the content, but when factoring in student actions in the game‐based learning environment, students approached the designed tasks differently. Our work therefore demonstrates how complimentary trace data analyses can be used to triangulate findings in a complex learning context.</p> <hd id="AN0158528157-21">CONCLUSION</hd> <p>The use of PBL as a pedagogical framework allowed us to focus on individual and collaborative actions in the game‐based learning environment and provide insights into how to design an adaptive system to support collaborative inquiry. Drawing on the PBL inquiry cycle and interactions at the brainstorming board, we can design with the following parameters in mind: (<reflink idref="bib1" id="ref51">1</reflink>) provide scripts to promote desired actions related to content and collaborative outcomes, (<reflink idref="bib2" id="ref52">2</reflink>) alert teachers about extreme patterns in the data and (<reflink idref="bib3" id="ref53">3</reflink>) provide differential support for group negotiation. Depending on students' progress in their inquiry phases, the system could provide hints related to definitions (initial exploration) or higher‐level inferences (later phases). Our work also suggests that a combination of methods is necessary to understand complex collaborative learning interactions. Given that game‐based analytics of serious games has typically focused on pre‐ and post‐test measures to understand learning gains (Alonso‐Fernández, Cano, et al., 2019), our work contributes to the growing body of literature that aims to leverage learning analytics to understand learning outcomes and processes.</p> <hd id="AN0158528157-22">ACKNOWLEDGMENTS</hd> <p>This research was supported by the National Science Foundation through grants DRL‐1561655 and DRL‐1561486. Any opinions, findings, conclusions or recommendations expressed in this report are those of the authors, and do not necessarily represent the official views, opinions or policy of the National Science Foundation.</p> <hd id="AN0158528157-23">CONFLICT OF INTEREST</hd> <p>There is no potential conflict of interest in this work.</p> <hd id="AN0158528157-24">ETHICS STATEMENT</hd> <p>This study was conducted with the IRB approval of Indiana University.</p> <hd id="AN0158528157-25">DATA AVAILABILITY STATEMENT</hd> <p>Due to human subject protection policies, the study data are not open.</p> <p>GRAPH: Supplementary Material</p> <ref id="AN0158528157-26"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref1" type="bt">1</bibl> <bibtext> Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2, 433 – 459. https://doi.org/10.1002/wics.101</bibtext> </blist> <blist> <bibl id="bib2" idref="ref2" type="bt">2</bibl> <bibtext> Alonso‐Fernández, C., Calvo‐Morata, A., Freire, M., Martínez‐Ortiz, I., & Fernández‐Manjón, B. (2019). Applications of data science to game learning analytics data: A systematic literature review. Computers & Education, 141, 103612. https://doi.org/10.1016/j.compedu.2019.103612</bibtext> </blist> <blist> <bibl id="bib3" idref="ref3" type="bt">3</bibl> <bibtext> Alonso‐Fernández, C., Cano, A. R., Calvo‐Morata, A., Freire, M., Martínez‐Ortiz, I., & Fernández‐Manjón, B. (2019). Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior, 99, 301 – 309. https://doi.org/10.1016/j.chb.2019.05.036</bibtext> </blist> <blist> <bibl id="bib4" idref="ref4" type="bt">4</bibl> <bibtext> Archer, E., & Prinsloo, P. (2020). Speaking the unspoken in learning analytics: Troubling the defaults. Assessment & Evaluation in Higher Education, 45 (6), 888 – 900. https://doi.org/10.1080/02602938.2019.1694863</bibtext> </blist> <blist> <bibl id="bib5" idref="ref13" type="bt">5</bibl> <bibtext> Aronson, E. (2002). Building empathy, compassion, and achievement in the jigsaw classroom. In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors on education (pp. 209 – 225). Academic Press.</bibtext> </blist> <blist> <bibl id="bib6" type="bt">6</bibl> <bibtext> Babamoradi, H., van den Berg, F., & Rinnan, Å. (2013). Bootstrap based confidence limits in principal component analysis—A case study. Chemometrics and Intelligent Laboratory Systems, 120, 97 – 105. https://doi.org/10.1016/j.chemolab.2012.10.007</bibtext> </blist> <blist> <bibl id="bib7" type="bt">7</bibl> <bibtext> Barrows, H. S. (1983). Problem‐based, self‐directed learning. JAMA, 250 (22), 3077 – 3080. https://doi.org/10.1001/jama.1983.03340220045031</bibtext> </blist> <blist> <bibl id="bib8" type="bt">8</bibl> <bibtext> Bell, T., Urhahne, D., Schanze, S., & Ploetzner, R. (2010). Collaborative inquiry learning: Models, tools, and challenges. International Journal of Science Education, 32 (3), 349 – 377. https://doi.org/10.1080/09500690802582241</bibtext> </blist> <blist> <bibl id="bib9" type="bt">9</bibl> <bibtext> Belland, B. R. (2011). Distributed cognition as a lens to understand the effects of Scaffolds: The role of transfer of responsibility. Educational Psychology Review, 23 (4), 577 – 600. https://doi.org/10.1007/s10648‐011‐9176‐5</bibtext> </blist> <blist> <bibtext> Blanca Mena, M. J., Alarcón Postigo, R., Arnau Gras, J., Bono Cabré, R., & Bendayan, R. (2017). Non‐normal data: Is ANOVA still a valid option? Psicothema, 29 (4), 552 – 557.</bibtext> </blist> <blist> <bibtext> Bridges, S., Botelho, M., Green, J. L., & Chau, A. C. (2012). Multimodality in problem‐based learning (PBL): An interactional ethnography. In S. Bridges, C. McGrath, & T. L. Whitehall (Eds.), Problem‐based learning in clinical education (pp. 99 – 120). Springer. https://doi.org/10.1007/978‐94‐007‐2515‐7</bibtext> </blist> <blist> <bibtext> Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297 – 334. https://doi.org/10.1007/BF02310555</bibtext> </blist> <blist> <bibtext> Dillenbourg, P. (1999). Collaborative learning: Cognitive and computational approaches. Advances in learning and instruction series. Elsevier Science, Inc.</bibtext> </blist> <blist> <bibtext> Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51 (5), 1505 – 1526. https://doi.org/10.1111/bjet.12992</bibtext> </blist> <blist> <bibtext> Fix, E., & Hodges, J. L. (1989). Discriminatory analysis. Nonparametric discrimination: Consistency properties. International Statistical Review, 57 (3), 238 – 247.</bibtext> </blist> <blist> <bibtext> Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68 – 84. https://doi.org/10.1016/j.iheduc.2015.10.002</bibtext> </blist> <blist> <bibtext> Geden, M., Emerson, A., Carpenter, D., Rowe, J., Azevedo, R., & Lester, J. (2020). Predictive student modeling in game‐based learning environments with word embedding representations of reflection. International Journal of Artificial Intelligence in Education, 31, 1 – 23.</bibtext> </blist> <blist> <bibtext> Glazewski, K. D., & Hmelo‐Silver, C. E. (2019). Scaffolding and supporting use of information for ambitious learning practices. Information and Learning Sciences, 120 (1/2), 39 – 58. https://doi.org/10.1108/ILS‐08‐2018‐0087</bibtext> </blist> <blist> <bibtext> Hall, R., & Stevens, R. (2016). Interaction analysis approaches to knowledge in use. In A. A. diSessa, M. Levin, & N. J. S. Brown (Eds.), Knowledge and interaction (pp. 88 – 124). Routledge.</bibtext> </blist> <blist> <bibtext> Han, A., Krieger, F., & Greiff, S. (2021). Collaboration analytics need more comprehensive models and methods: An opinion paper. Journal of Learning Analytics, 8 (1), 13 – 29. https://doi.org/10.18608/jla.2021.7288</bibtext> </blist> <blist> <bibtext> Hmelo‐Silver, C. E. (2004). Problem‐based learning: What and how do students learn? Educational Psychology Review, 16 (3), 235 – 266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3</bibtext> </blist> <blist> <bibtext> Hmelo‐Silver, C. E. (2006). Design principles for scaffolding technology‐based inquiry. In A. M. O'Donnell, C. E. Hmelo‐Silver, & G. Erkens (Eds.), Collaborative learning, reasoning, and technology (pp. 147 – 170). Routledge.</bibtext> </blist> <blist> <bibtext> Hmelo‐Silver, C. E., & Eberbach, C. (2012). Learning theories and problem‐based learning. In S. Bridges, C. McGrath, & T. L. Whitehill (Eds.), Problem‐based learning in clinical education. Innovation and change in professional education (Vol. 8, pp. 3 – 17). Springer. https://doi.org/10.1007/978‐94‐007‐2515‐7_1</bibtext> </blist> <blist> <bibtext> Hmelo‐Silver, C. E., Jordan, R., Eberbach, C., & Sinha, S. (2017). Systems learning with a conceptual representation: A quasi‐experimental study. Instructional Science, 45 (1), 53 – 72. https://doi.org/10.1007/s11251‐016‐9392‐y</bibtext> </blist> <blist> <bibtext> Jain, A. K. (2010). Data clustering: 50 years beyond K‐means. Pattern Recognition Letters, 31 (8), 651 – 666. https://doi.org/10.1016/j.patrec.2009.09.011</bibtext> </blist> <blist> <bibtext> Joliffe, I., & Morgan, B. (1992). Principal component analysis and exploratory factor analysis. Statistical Methods in Medical Research, 1 (1), 69 – 95. https://doi.org/10.1177/096228029200100105</bibtext> </blist> <blist> <bibtext> Jonassen, D. (2011). Supporting problem solving in PBL. Interdisciplinary Journal of Problem‐based Learning, 5 (2), 8. https://doi.org/10.7771/1541‐5015.1256</bibtext> </blist> <blist> <bibtext> Jonassen, D. H., & Hung, W. (2008). All problems are not equal: Implications for problem‐based learning. Interdisciplinary Journal of Problem‐based Learning, 2 (2), 4. https://doi.org/10.7771/1541‐5015.1080</bibtext> </blist> <blist> <bibtext> Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. Journal of the Learning Sciences, 4 (1), 39 – 103. https://doi.org/10.1207/s15327809jls0401_2</bibtext> </blist> <blist> <bibtext> Jordan, R. C., Sorensen, A. E., & Hmelo‐Silver, C. E. (2014). A conceptual representation to support ecological systems learning. Natural Sciences Education, 43 (1), 141 – 146.</bibtext> </blist> <blist> <bibtext> Kim, N. J., Belland, B. R., & Walker, A. E. (2018). Effectiveness of computer‐based scaffolding in the context of problem‐based learning for STEM education: Bayesian meta‐analysis. Educational Psychology Review, 30, 397 – 429. https://doi.org/10.1007/s10648‐017‐9419‐1</bibtext> </blist> <blist> <bibtext> King, A. (1994). Guiding knowledge construction in the classroom: Effects of teaching children how to question and how to explain. American Educational Research Journal, 31 (2), 338 – 368. https://doi.org/10.3102/00028312031002338</bibtext> </blist> <blist> <bibtext> Kirk, R. E. (2013). Experimental design. Procedures for the behavioral sciences (4 th ed.). Sage.</bibtext> </blist> <blist> <bibtext> Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The Knowledge‐Learning‐Instruction framework: Bridging the science‐practice chasm to enhance robust student learning. Cognitive Science, 36 (5), 757 – 798. https://doi.org/10.1111/j.1551‐6709.2012.01245.x</bibtext> </blist> <blist> <bibtext> Kollar, I., Wecker, C., & Fischer, F. (2018). Scaffolding and scripting (computer‐supported) collaborative learning. In F. Fischer, C. E. Hmelo‐Silver, S. R. Goldman, & P. Reimann (Eds.), International handbook of the learning sciences (pp. 340 – 350). Routledge.</bibtext> </blist> <blist> <bibtext> Liu, L., Hao, J., von Davier, A. A., Kyllonen, P., & Zapata‐Rivera, J.‐D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real‐world skill development (pp. 344 – 359). IGI Global.</bibtext> </blist> <blist> <bibtext> Liu, M., Horton, L., Lee, J., Kang, J., Rosenblum, J., O'Hair, M., & Lu, C.‐W. (2014). Creating a multimedia enhanced problem‐based learning environment for middle school science: Voices from the developers. Interdisciplinary Journal of Problem‐based Learning, 8 (1), 4. https://doi.org/10.7771/1541‐5015.1422</bibtext> </blist> <blist> <bibtext> Liu, R., & Koedinger, K. R. (2017). Going beyond better data prediction to create explanatory models of educational data. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The Handbook of Learning Analytics (Vol. 1, pp. 69 – 76). Society for Learning Analytics Research. https://doi.org/10.18608/hla17</bibtext> </blist> <blist> <bibtext> Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics‐driven design to enhance learning. IEEE Transactions on Learning Technologies, 12 (4), 516 – 534. https://doi.org/10.1109/TLT.2018.2868673</bibtext> </blist> <blist> <bibtext> Martinez‐Maldonado, R., Gašević, D., Echeverria, V., Fernandez Nieto, G., Swiecki, Z., & Buckingham Shum, S. (2021). What do you mean by collaboration analytics? A conceptual model. Journal of Learning Analytics, 8 (1), 126 – 153. https://doi.org/10.18608/jla.2021.7227</bibtext> </blist> <blist> <bibtext> O'Connor, C., & Michaels, S. (2019). Supporting teachers in taking up productive talk moves: The long road to professional learning at scale. International Journal of Educational Research, 97, 166 – 175. https://doi.org/10.1016/j.ijer.2017.11.003</bibtext> </blist> <blist> <bibtext> Pedaste, M., Mäeots, M., Siiman, L. A., de Jong, T., van Riesen, S. A. N., Kamp, E. T., Manoli, C. C., Zacharia, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry‐based learning: Definitions and the inquiry cycle. Educational Research Review, 14, 47 – 61. https://doi.org/10.1016/j.edurev.2015.02.003</bibtext> </blist> <blist> <bibtext> R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R‐project.org/</bibtext> </blist> <blist> <bibtext> Resnick, L. B., Asterhan, C. S. C. & Clarke, S. N. (2018). Accountable talk: Instructional dialogue that builds the mind. The International Academy of Education (IAE) and the International Bureau of Education (IBE) of the United Nations Educational, Scientific and Cultural Organization (UNESCO).</bibtext> </blist> <blist> <bibtext> Revelle, W. (2021). psych: Procedures for psychological, psychometric, and personality research. Northwestern University.</bibtext> </blist> <blist> <bibtext> Rosé, C. P., McLaughlin, E. A., Liu, R., & Koedinger, K. R. (2019). Explanatory learner models: Why machine learning (alone) is not the answer. British Journal of Educational Technology, 50 (6), 2943 – 2958. https://doi.org/10.1111/bjet.12858</bibtext> </blist> <blist> <bibtext> Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53 – 65. https://doi.org/10.1016/0377‐0427(87)90125‐7</bibtext> </blist> <blist> <bibtext> Rummel, N., Walker, E., & Aleven, V. (2016). Different futures of adaptive collaborative learning support. International Journal of Artificial Intelligence in Education, 26 (2), 784 – 795. https://doi.org/10.1007/s40593‐016‐0102‐3</bibtext> </blist> <blist> <bibtext> Saleh, A., Yuxin, C., Hmelo‐Silver, C. E., Glazewski, K. D., Mott, B. W., & Lester, J. C. (2020). Coordinating scaffolds for collaborative inquiry in a game‐based learning environment. Journal of Research in Science Teaching, 57 (9), 1490 – 1518. https://doi.org/10.1002/tea.21656</bibtext> </blist> <blist> <bibtext> Savery, J. R. (2019). Comparative pedagogical models of problem‐based learning. In M. Moallem, W. Hung, & N. Dabbagh (Eds.), The Wiley handbook of problem‐based learning (pp. 81 – 104). Wiley Blackwell.</bibtext> </blist> <blist> <bibtext> Su, Y., & Klein, J. (2010). Using scaffolds in problem‐based hypermedia. Journal of Educational Multimedia and Hypermedia, 19 (3), 327 – 347.</bibtext> </blist> <blist> <bibtext> Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research instruments in science education. Research in Science Education, 48 (6), 1273 – 1296. https://doi.org/10.1007/s11165‐016‐9602‐2</bibtext> </blist> <blist> <bibtext> Tawfik, A. A., & Kolodner, J. L. (2016). Systematizing scaffolding for problem‐based learning: A view from case‐based reasoning. Interdisciplinary Journal of Problem‐based Learning, 10 (1), 6. https://doi.org/10.7771/1541‐5015.1608</bibtext> </blist> <blist> <bibtext> Van de Pol, J., Volman, M., & Beishuizen, J. (2010). Scaffolding in teacher–student interaction: A decade of research. Educational Psychology Review, 22 (3), 271 – 296. https://doi.org/10.1007/s10648‐010‐9127‐6</bibtext> </blist> <blist> <bibtext> van Leeuwen, A., Rummel, N., & van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer‐Supported Collaborative Learning, 14 (3), 261 – 289. https://doi.org/10.1007/s11412‐019‐09299‐x</bibtext> </blist> <blist> <bibtext> Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer‐Verlag. https://ggplot2.tidyverse.org</bibtext> </blist> <blist> <bibtext> Wijnia, L., Loyens, S. M., & Rikers, R. M. (2019). The problem‐based learning process: An overview of different models. In M. Moallem, W. Hung, & N. Dabbagh (Eds.), The Wiley handbook of problem‐based learning (pp. 273 – 295). Wiley Blackwell.</bibtext> </blist> <blist> <bibtext> Wise, A. F., Sarmiento, J. P., & Boothe, M. B., Jr. (2021). Subversive learning analytics. In LAK21: 11th International Learning Analytics and Knowledge Conference. https://doi.org/10.1145/3448139.3448210</bibtext> </blist> <blist> <bibtext> Wu, X., Zhu, X., Wu, G. Q., & Ding, W. (2014). Data mining with big data. IEEE Transactions on Knowledge and Data Engineering, 26 (1), 97 – 107. https://doi.org/10.1109/TKDE.2013.109</bibtext> </blist> <blist> <bibtext> Yoon, S. A., Goh, S.‐E., & Park, M. (2018). Teaching and learning about complex systems in K–12 science education: A review of empirical studies 1995–2015. Review of Educational Research, 88 (2), 285 – 325. https://doi.org/10.3102/0034654317746090</bibtext> </blist> <blist> <bibtext> Zimmermann, A., Lorenz, A., & Oppermann, R. (2007). An operational definition of context. In B. Kokinov, D. C. Richardson, T. R. Roth‐Berghofer, & L. Vieu (Eds.), Modeling and using context (pp. 558 – 571). Springer.</bibtext> </blist> </ref> <aug> <p>By Asmalina Saleh; Tanner M. Phillips; Cindy E. Hmelo‐Silver; Krista D. Glazewski; Bradford W. Mott and James C. Lester</p> <p>Reported by Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib524" firstref="ref46"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1344679
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Learning Analytics Approach towards Understanding Collaborative Inquiry in a Problem-Based Learning Environment
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Saleh%2C+Asmalina%22">Saleh, Asmalina</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8178-4238">0000-0001-8178-4238</externalLink>)<br /><searchLink fieldCode="AR" term="%22Phillips%2C+Tanner+M%2E%22">Phillips, Tanner M.</searchLink><br /><searchLink fieldCode="AR" term="%22Hmelo-Silver%2C+Cindy+E%2E%22">Hmelo-Silver, Cindy E.</searchLink><br /><searchLink fieldCode="AR" term="%22Glazewski%2C+Krista+D%2E%22">Glazewski, Krista D.</searchLink><br /><searchLink fieldCode="AR" term="%22Mott%2C+Bradford+W%2E%22">Mott, Bradford W.</searchLink><br /><searchLink fieldCode="AR" term="%22Lester%2C+James+C%2E%22">Lester, James C.</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22British+Journal+of+Educational+Technology%22"><i>British Journal of Educational Technology</i></searchLink>. Sep 2022 53(5):1321-1342.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF), Division of Research on Learning in Formal and Informal Settings (DRL)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 1561655<br />1561486
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Cooperative+Learning%22">Cooperative Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Pretests+Posttests%22">Pretests Posttests</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Based+Learning%22">Problem Based Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Game+Based+Learning%22">Game Based Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Factor+Analysis%22">Factor Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Brainstorming%22">Brainstorming</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Instruction%22">Science Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Inquiry%22">Inquiry</searchLink><br /><searchLink fieldCode="DE" term="%22Active+Learning%22">Active Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Gains%22">Achievement Gains</searchLink><br /><searchLink fieldCode="DE" term="%22Independent+Study%22">Independent Study</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Processes%22">Learning Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Course+Content%22">Course Content</searchLink><br /><searchLink fieldCode="DE" term="%22Interaction+Process+Analysis%22">Interaction Process Analysis</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/bjet.13198
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0007-1013<br />1467-8535
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This exploratory paper highlights how problem-based learning (PBL) provided the pedagogical framework used to design and interpret learning analytics from "Crystal Island: EcoJourneys," a collaborative game-based learning environment centred on supporting science inquiry. In "Crystal Island: EcoJourneys," students work in teams of four, investigate the problem individually and then utilize a brainstorming board, an in-game PBL whiteboard that structured the collaborative inquiry process. The paper addresses a central question: how can PBL support the interpretation of the observed patterns in individual actions and collaborative interactions in the collaborative game-based learning environment? Drawing on a mixed method approach, we first analyzed students' pre- and post-test results to determine if there were learning gains. We then used principal component analysis (PCA) to describe the patterns in game interaction data and clustered students based on the PCA. Based on the pre- and post-test results and PCA clusters, we used interaction analysis to understand how collaborative interactions unfolded across selected groups. Results showed that students learned the targeted content after engaging with the game-based learning environment. Clusters based on the PCA revealed four main ways of engaging in the game-based learning environment: students engaged in low to moderate self-directed actions with: (1) high and (2) moderate collaborative sense-making actions; (3) low self-directed with low collaborative sense-making actions; and (4) high self-directed actions with low collaborative sense-making actions. Qualitative interaction analysis revealed that a key difference among four groups in each cluster was the nature of verbal student discourse: students in the low to moderate self-directed and high collaborative sense-making cluster actively initiated discussions and integrated information they learned to the problem, whereas students in the other clusters required more support. These findings have implications for designing adaptive support that responds to students' interactions with in-game activities.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2022
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1344679
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1344679
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/bjet.13198
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 1321
    Subjects:
      – SubjectFull: Cooperative Learning
        Type: general
      – SubjectFull: Learning Analytics
        Type: general
      – SubjectFull: Pretests Posttests
        Type: general
      – SubjectFull: Problem Based Learning
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Game Based Learning
        Type: general
      – SubjectFull: Factor Analysis
        Type: general
      – SubjectFull: Brainstorming
        Type: general
      – SubjectFull: Science Instruction
        Type: general
      – SubjectFull: Inquiry
        Type: general
      – SubjectFull: Active Learning
        Type: general
      – SubjectFull: Achievement Gains
        Type: general
      – SubjectFull: Independent Study
        Type: general
      – SubjectFull: Learning Processes
        Type: general
      – SubjectFull: Course Content
        Type: general
      – SubjectFull: Interaction Process Analysis
        Type: general
    Titles:
      – TitleFull: A Learning Analytics Approach towards Understanding Collaborative Inquiry in a Problem-Based Learning Environment
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Saleh, Asmalina
      – PersonEntity:
          Name:
            NameFull: Phillips, Tanner M.
      – PersonEntity:
          Name:
            NameFull: Hmelo-Silver, Cindy E.
      – PersonEntity:
          Name:
            NameFull: Glazewski, Krista D.
      – PersonEntity:
          Name:
            NameFull: Mott, Bradford W.
      – PersonEntity:
          Name:
            NameFull: Lester, James C.
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 0007-1013
            – Type: issn-electronic
              Value: 1467-8535
          Numbering:
            – Type: volume
              Value: 53
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: British Journal of Educational Technology
              Type: main
ResultId 1