View in EDS HTML Full Text PDF Full Text

Experience Sampling Methodology and Technology: An Approach for Examining Situational, Longitudinal, and Multi-Dimensional Characteristics of Engagement

Saved in:
Bibliographic Details
Title: Experience Sampling Methodology and Technology: An Approach for Examining Situational, Longitudinal, and Multi-Dimensional Characteristics of Engagement
Language: English
Authors: Kui Xie (ORCID 0000-0002-7173-4859), Vanessa W. Vongkulluksn, Benjamin C. Heddy, Zilu Jiang
Source: Educational Technology Research and Development. 2024 72(5):2585-2615.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 31
Publication Date: 2024
Document Type: Journal Articles
Information Analyses
Descriptors: Learner Engagement, Environment, Student Characteristics, Research Methodology, Educational Research, Information Technology, Context Effect, Longitudinal Studies, Multivariate Analysis
DOI: 10.1007/s11423-023-10259-4
ISSN: 1042-1629
1556-6501
Abstract: Engagement has been recognized as one of the most important factors of learning and achievement in academic settings. Research on engagement has been gearing toward a "person-in-context" orientation, where both personal characteristics and contextual features in relation to students' engagement are considered. This orientation allows a more in-depth understanding of how a person embedded within a context engages in a task, and it pays particular attention to the interactions between the person and contextual features. Engagement in context is situational, longitudinal, and multi-dimensional. This in-situ orientation requires a research methodology that is embedded in and responsive to the context where learning occurs. In this paper, we provide a conceptual synthesis of research on academic engagement in proposing a framework of engagement in context. We introduce the affordances of Experience Sampling Methodology (ESM) and provide a review of current technologies in supporting ESM. In addition, we provide example cases of examining engagement using ESM and technology. In these cases, we discuss details about how ESM combines with technologies and statistical approaches in providing insights to educational research, theory, and practice.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1448032
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwG6fNqcaJUYmEm1MmmL4WbQAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDHzOm4m09CQJvoHfXgIBEICBm97jB7ovmvv52M1EFLt7XNgGa0jRfpXUr2qF2sb0djv7mFI3mCvV_XQgyuEGCK_DlLCe1YIAUCBg4B89RpE9R-GrU7Y4LzdGtvbZfHMPkIec_Mend8Mm_3C78CLt4ZVbxNQoisrp1fdH2MUI31Y0yD55g5qoiV9PaKkBA2TGPcRfoW1dEeD_JoS7seV6989oWtoyGdDm-AmvFsdZ
Text:
  Availability: 1
  Value: <anid>AN0180830877;etr01oct.24;2024Nov14.05:15;v2.2.500</anid> <title id="AN0180830877-1">Experience sampling methodology and technology: an approach for examining situational, longitudinal, and multi-dimensional characteristics of engagement </title> <p>Engagement has been recognized as one of the most important factors of learning and achievement in academic settings. Research on engagement has been gearing toward a "person-in-context" orientation, where both personal characteristics and contextual features in relation to students' engagement are considered. This orientation allows a more in-depth understanding of how a person embedded within a context engages in a task, and it pays particular attention to the interactions between the person and contextual features. Engagement in context is situational, longitudinal, and multi-dimensional. This in-situ orientation requires a research methodology that is embedded in and responsive to the context where learning occurs. In this paper, we provide a conceptual synthesis of research on academic engagement in proposing a framework of engagement in context. We introduce the affordances of Experience Sampling Methodology (ESM) and provide a review of current technologies in supporting ESM. In addition, we provide example cases of examining engagement using ESM and technology. In these cases, we discuss details about how ESM combines with technologies and statistical approaches in providing insights to educational research, theory, and practice.</p> <p>Keywords: Experience sampling method; ESM; Engagement in context; Situational engagement; Intensive longitudinal design</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0180830877-2">Introduction</hd> <p>Engagement has been recognized as one of the most important factors of learning and achievement in academic settings (Christenson et al., [<reflink idref="bib17" id="ref1">17</reflink>]). Studies have well documented that engagement has significant and positive relationships with beneficial student outcomes such as learning (Greene, [<reflink idref="bib33" id="ref2">33</reflink>]; Marks, [<reflink idref="bib53" id="ref3">53</reflink>]; Pekrun & Linnenbrink-Garcia, [<reflink idref="bib69" id="ref4">69</reflink>]), motivation (Buhs & Ladd, [<reflink idref="bib12" id="ref5">12</reflink>]; Cleary & Zimmerman, [<reflink idref="bib18" id="ref6">18</reflink>]; Johnson & Sinatra, [<reflink idref="bib42" id="ref7">42</reflink>]), and retention (Fredricks, [<reflink idref="bib28" id="ref8">28</reflink>]; Fredricks et al., [<reflink idref="bib29" id="ref9">29</reflink>]).</p> <p>Recent research on engagement has been gearing toward a "person-in-context" orientation, where engagement should be examined considering both student personal characteristics and contextual features (Schmidt et al., [<reflink idref="bib78" id="ref10">78</reflink>]; Sinatra et al., [<reflink idref="bib81" id="ref11">81</reflink>]). This orientation allows a more in-depth understanding of how a person embedded within context engages in a task, pays particular attention to the interactions between the person and contextual features, and helps researchers to examine the fluctuations of these interactions from moment to moment and from one context to another (Xie et al., [<reflink idref="bib98" id="ref12">98</reflink>]).</p> <p>The in-situ orientation requires a research methodology that is embedded and responsive to the context in which learning occurs. Experience Sampling Methodology (ESM) provides an opportunity for researchers to investigate engagement in context. ESM is a technique used to collect data when people are in an everyday context so that the data reflect what is happening in that moment (Zirkel et al., [<reflink idref="bib102" id="ref13">102</reflink>]). ESM researchers often use digital devices to signal users to respond to short self-report surveys about their thoughts and feelings in the moment (Larson & Csikszentmihalyi, [<reflink idref="bib44" id="ref14">44</reflink>]). Modern technologies, for instance, mobile devices, sensor technologies, server and network infrastructure, have been developed to not only improve educational experiences, but also support educational research to advance our understanding about teaching and learning. ESM combined with modern technologies provide researchers with new tools and methodological approaches helping to advance our understanding of students' engagement in academic settings.</p> <p>The purpose of this conceptual paper is to synthesize and identify characteristics of engagement, and map ESM methodologies, technologies, and statistical approaches to capture the characteristics of engagement in educational research. We first provide a synthesis of engagement from a "person-in-context" orientation, highlighting the unique characteristics of engagement being situational, longitudinal, and multidimensional. We also highlight the need for a research methodology and technology that can fully capture these characteristics to support research on engagement. Second, we introduce the affordances of ESM when combined with technologies. More specifically, we discuss how ESM researchers can collect data in the context and in the moment of engagement, how ESM is an intensive longitudinal methodology, and how ESM researchers can collect multi-modal data. We highlight the role of technologies in facilitating ESM studies. Third, we review statistical approaches for modeling data collected via ESM. Finally, we provide examples of ESM studies in diverse settings (e.g., K-12 vs. higher education, formal vs. informal learning). In these cases, we discuss details about how ESM combined with technologies and statistical approaches can provide insights in advancing engagement research, theory, and practice. We hope to provide a conceptual framework to guide the future engagement research leveraging the affordances of experience-sampling methodology and technologies.</p> <hd id="AN0180830877-3">Engagement in context</hd> <p>Prior studies have adopted a range of terms in describing engagement in educational settings, such as student involvement (Moore et al., [<reflink idref="bib60" id="ref15">60</reflink>]), school engagement (Wang & Eccles, [<reflink idref="bib92" id="ref16">92</reflink>]), learner engagement (Martin & Borup, [<reflink idref="bib54" id="ref17">54</reflink>]), and class participation (Xie et al., [<reflink idref="bib96" id="ref18">96</reflink>]). Therefore, the definition of engagement varies depending on the context in which engagement is discussed. Because in this paper we focus on the in-situ nature of engagement, we adopt a definition that refers to engagement as "the psychological processes that underlie energy, purpose, and durability of human action" (Deci, [<reflink idref="bib26" id="ref19">26</reflink>]). Engagement and motivation are related constructs that are often discussed together, but they differ in focus. Motivation refers to the underlying sources of energy, purpose, and durability, while engagement is the manifestation resulting from motivation (Skinner et al., [<reflink idref="bib85" id="ref20">85</reflink>]; Skinner & Pitzer, [<reflink idref="bib83" id="ref21">83</reflink>]). In this view, motivation can be considered as intent, whereas engagement is the resulting action (Reschly & Christenson, [<reflink idref="bib75" id="ref22">75</reflink>]). In educational settings, engagement encompasses both observable and unobservable qualities of student interactions with learning activities (Deci & Ryan, [<reflink idref="bib27" id="ref23">27</reflink>]). In the following we describe engagement as consisting of three defining characteristics including being situational, longitudinal, and multidimensional.</p> <hd id="AN0180830877-4">Engagement is situational</hd> <p>Educational scholars have positioned engagement as a contextually dependent construct (Skinner et al., [<reflink idref="bib84" id="ref24">84</reflink>]). That is, engagement cannot meaningfully be separated from context as they are dynamically intertwined. This view aligns theoretically with social cognitive theory, which states that individual factors, behaviors, and environmental features dynamically interact in a process of triadic reciprocal causation (Bandura, [<reflink idref="bib5" id="ref25">5</reflink>]).</p> <p>Sinatra and colleagues hypothesized that engagement measurement exists on a continuum from person oriented to context oriented (Sinatra et al., [<reflink idref="bib81" id="ref26">81</reflink>]). Person oriented measurements of engagement focus solely on the psyche of the individual and include physiological instrumentation (e.g., skin response, response time, and trace data). Context oriented measures of engagement focus on the specific context in which engagement occurs (e.g., discourse analysis, observations, teacher ratings). These types of instrumentation cannot be separated from the environment in which they reside and thus are considered context oriented. In between the two ends of the continuum are person-in-context measurements of engagement, which integrate person factors with contextual factors when assessing engagement (e.g., triangulated mix methods data, observations of interaction, and experience sampling methodology). We extend the idea of measuring engagement within context and propose that engagement needs to also be conceptualized within the context in which it occurs. That is, we argue that situatedness is a defining characteristic of engagement in that it is always influenced by the environment, which we call <emph>situational engagement</emph>.</p> <p>We contend that situational engagement is highly sensitive to the environment. Xie and colleagues have provided evidence of this phenomenon by showing that environmental features influence the extent and quality of engagement (Xie etal., [<reflink idref="bib98" id="ref27">98</reflink>]). We conceptualize that engagement in context involves two levels of specificity. The first level involves situational processes that are situated, context-dependent, and task-specific. These processes encompass various features of the environment such as seating location, noise and lighting level, study partners, solo-status, social support, and media support. The second level involves person-level processes that are person-specific. These processes include factors such as self-efficacy and motivational factors (Lu, Xie, & Liu, [<reflink idref="bib49" id="ref28">49</reflink>]; Wang et al., [<reflink idref="bib94" id="ref29">94</reflink>]; Xie, [<reflink idref="bib95" id="ref30">95</reflink>]). These person-level processes while being individual in nature have a reciprocal and synergistic relationship with the environment, and thus they are still contextualized but reside within the individual. It is important to note that some of these variables can be positioned at both the person and the situational levels. For example, situational interest occurs at the situational level, and individual interest occurs at the person level (Hidi & Renninger, [<reflink idref="bib39" id="ref31">39</reflink>]). With this conceptualization, studies can examine engagement at both levels of specificity and consider both personal characteristics and contextual features in relation to students' engagement.</p> <hd id="AN0180830877-5">Engagement is longitudinal</hd> <p>Most research measures engagement prospectively and/or retrospectively (Xie et al., [<reflink idref="bib97" id="ref32">97</reflink>]). However, we argue that engagement does not occur at any 1 time point. Instead, engagement occurs within momentary experiences over time. That is, engagement is continuously and dynamically changing. At each occurrence, engagement is influenced by factors in the past, present, and future. Additionally, engagement can fluctuate from time to time and from one context to another. That is, capturing engagement at 1 time point in a particular context may not be an accurate depiction of engagement. Comparatively, capturing engagement across time points can provide a more elucidated understanding of one's experience (Skinner & Pitzer, [<reflink idref="bib83" id="ref33">83</reflink>]). As engagement experiences build on one another, future engagement experiences are dynamically influenced by past engagement experiences (Gottfried et al., [<reflink idref="bib32" id="ref34">32</reflink>]; Skinner & Belmont, [<reflink idref="bib82" id="ref35">82</reflink>]). Therefore, when modeling learning experiences or processes, prospective and retrospective instrumentation may detect significant changes, but more fine-grained longitudinal examination of engagement may lead to nuanced findings and implications that guide educational practices (Xie et al., [<reflink idref="bib98" id="ref36">98</reflink>]).</p> <hd id="AN0180830877-6">Engagement is multidimensional</hd> <p>Extant research has shown that engagement is multidimensional. While there is disagreement on the number of dimensions of engagement, several have been identified. Fredricks and colleagues ([<reflink idref="bib29" id="ref37">29</reflink>]) operationalized engagement as consisting of three dimensions including behavioral, cognitive, and affective (Fredricks et al., [<reflink idref="bib29" id="ref38">29</reflink>]; Martin et al., [<reflink idref="bib56" id="ref39">56</reflink>]). Behavioral engagement refers to physical learning behaviors including participation, concentration, and effort (Schmidt et al., [<reflink idref="bib78" id="ref40">78</reflink>]). Cognitive engagement is defined as using shallow (e.g., repetition) or deep strategies (e.g., summarizing) when learning (Greene, [<reflink idref="bib33" id="ref41">33</reflink>]). Affective engagement is described as students' emotional reactions to learning such as interest, enjoyment, or frustration (Pekrun & Linnenbrink-Garcia, [<reflink idref="bib69" id="ref42">69</reflink>]). All three dimensions of engagement can have individual and synergistic effects (Heddy et al., [<reflink idref="bib37" id="ref43">37</reflink>]). Additionally, Reeve and Shin ([<reflink idref="bib74" id="ref44">74</reflink>]) provided evidence for a fourth dimension called agentic engagement, which occurs when students interact with the environment to influence the flow of instruction. Furthermore, researchers have described social engagement (e.g., engagement as a function of interacting with others in the environment) as a dimension as well (Linnenbrink-Garcia et al., [<reflink idref="bib48" id="ref45">48</reflink>]; Xie et al., [<reflink idref="bib99" id="ref46">99</reflink>]). Regardless of the number and types of dimensions, existing research clearly points to engagement being a multidimensional construct.</p> <hd id="AN0180830877-7">Affordances of experience sampling methodology and technology</hd> <p>Considering the unique characteristics of engagement, the examination of engagement in context requires a methodology that is responsive to these characteristics. The affordances of experience sampling method (ESM) map well onto (<reflink idref="bib1" id="ref47">1</reflink>) the situational characteristic of engagement by collecting data in the moment and in the context, and (<reflink idref="bib2" id="ref48">2</reflink>) the longitudinal nature of engagement by collecting intensive longitudinal data, and (<reflink idref="bib3" id="ref49">3</reflink>) the multi-dimensional characteristic of engagement by collecting multi-modal data. When combined with modern technologies, the implementation of ESM becomes feasible and manageable in educational settings with (<reflink idref="bib4" id="ref50">4</reflink>) sampling management, and (<reflink idref="bib5" id="ref51">5</reflink>) data management features. Figure 1 illustrates these five major affordances of ESM that correspondence to the characteristics of engagement studies.</p> <p>Graph: Fig. 1 Affordances of experience sampling methodology and technology in engagement studies</p> <hd id="AN0180830877-8">ESM data collection happens in the context and in the moment</hd> <p>In engagement studies, self-reporting has been widely used in as a method to measure students' internal processes such as their emotions, motivation, perceptions, attitudes, and satisfaction that are not easily observed overtly. Traditional self-reports rely on either participants' projection (i.e., data gathered before learning events occur) or their retrospection (i.e., data gathered after learning events occurred). Researchers often ask a participant to respond to contextually dependent questions when the participant is no longer within the relevant context (Xie et al., [<reflink idref="bib97" id="ref52">97</reflink>]). Therefore, concerns have been raised about the validity of responses from self-reports as they rely on participants' long-term memories to reconstruct past-events or experiences (van Berkel et al., [<reflink idref="bib87" id="ref53">87</reflink>]). Experience sampling method (ESM) directly responds to these concerns as ESM surveys catch people in the moment and in the context of doing something while they are still in the proximity of time and space. Therefore, it is much more sensitive in capturing people's thoughts and feelings than traditional self-report measures. While ESM still heavily relies on self-reporting, the data collected using ESM have more ecological validity than those collected based on projections or retrospections (Csikszentmihalyi & Larson, [<reflink idref="bib21" id="ref54">21</reflink>]). For example, Xie and colleagues ([<reflink idref="bib97" id="ref55">97</reflink>]) compared students' self-reports of engagement from prospective, retrospective, and ESM data collection. They found that students reported similar engagement patterns across these various data collection methods. However, students' self-reports of engagement differed in detail with ESM being able to capture more nuanced and accurate data about students' engagement.</p> <p>Because of the in-situ data collection, ESM allows researchers to examine engagement from a situational perspective. Much of the literature have treated engagement as a person-level variable. Those studies examine the relationships between engagement and person-level characteristics such as motivation, emotion, belongingness, and/or outcomes such as grades. In fact, studies that only focus on general or overall engagement do not align well with how engagement occurs in the real world—that is, engagement as it occurs within context. With ESM, studies can collect data when and where engagement occurs. These data can include not only information about engagement, but also the associated contextual information such as the task, the environment, and the timing of engagement. This in-situ approach allows researchers to build more precise models and theories of engagement at the situational level. For example, Schneider et al. ([<reflink idref="bib79" id="ref56">79</reflink>]) examined optimal learning moments "when students experience high levels of challenge, skill, and interest." These in-situ experiences produce positive learning outcomes. Through ESM, they were able to precisely measure how often these optimal learning moments occurred in high-school science classes (Schneider et al., [<reflink idref="bib80" id="ref57">80</reflink>]). In examining behavioral engagement, Xie et al. ([<reflink idref="bib97" id="ref58">97</reflink>], [<reflink idref="bib98" id="ref59">98</reflink>]) provided both temporal and spatial representations of behavioral engagement to precisely map out when and where college students study in out-of-class settings. Martin et al. ([<reflink idref="bib55" id="ref60">55</reflink>]) examined the relationship between prior general versus real-time motivation and engagement of junior high school students. Using ESM, they were able to model these relationships at four levels with different specificity: between lessons, between days, between weeks, and between students.</p> <hd id="AN0180830877-9">ESM as an intensive longitudinal methodology</hd> <p>Engagement is continuous, dynamic, and can fluctuate from time to time and from one context to another. Therefore, the examination of engagement should leverage data which reflects its changing nature. ESM is an intensive longitudinal method that allows the collection of more fine-gained data as the frequency of ESM measures can be much more intensive than traditional self-report repeated measures (Bolger & Laurenceau, [<reflink idref="bib8" id="ref61">8</reflink>]). Compared to models or theories built upon a single measure of engagement representing an abstract concept of engagement in general (Manwaring et al., [<reflink idref="bib51" id="ref62">51</reflink>]), ESM collects multiple and frequent measurements of an individual's engagement over time and across contexts. This intensive longitudinal method provides temporal information about learning engagement and the proximal contexts that are difficult to recall in a single self-report measure (Walls & Schafer, [<reflink idref="bib91" id="ref63">91</reflink>]). Therefore, ESM provides new opportunities for researchers to analyze the dynamic nature of engagement and build models and theories that are situational, concrete, and nuanced. For example, when examining students' performance improvement in a 16-week science curriculum, in a traditional method, this is typically done by comparing pre- and post-test results and identifying <emph>whether</emph> significant changes occur before and after the 16 weeks. On the other hand, ESM and an intensive longitudinal method can model the dynamics in students' daily or weekly experiences and may provide more detailed insights about <emph>how</emph> the changes occur during the 16-week science curriculum, leading to the significant difference between pre- and post-tests. Manwaring and colleagues ([<reflink idref="bib51" id="ref64">51</reflink>]) applied ESM to capture students' day-to-day experiences in a semester-long blended learning course. They explored how much of engagement was a stable characteristic of the student and how much engagement varied by course design elements. Their findings showed that course design and student perception variables had a greater influence on engagement than personal characteristics (Manwaring et al., [<reflink idref="bib51" id="ref65">51</reflink>]).</p> <p>However, the intensive ESM data collections also bring limitations to this method. ESM has been considered a labor-intensive and high-cost research method. Because ESM frequently samples participants and collects data, managing ESM data collection can be quite intense for both the researchers and the participants. On a positive note, enabled by modern mobile and smart technologies, ESM has evolved to include effective and efficient collection and management of intensive longitudinal data (see our discussion about sampling management and data management below). Another potential concern is that ESM and intensive longitudinal methods may cause reactivity—or the potential for the repeated collection procedure to affect or change the participant's experience (Manwaring et al., [<reflink idref="bib51" id="ref66">51</reflink>]). Therefore, researchers may need to consider testing for reactive effects of ESM methodology and controlling for it in the modeling of ESM data (Xie et al., [<reflink idref="bib98" id="ref67">98</reflink>]).</p> <hd id="AN0180830877-10">ESM captures multimodal data</hd> <p>In addition to the intensive frequency of data collection, ESM can collect data from multiple sources simultaneously. Modern mobile and wearable technologies are equipped with various types of smart sensors, for example, position sensors (e.g., GPS, magnetometer, and proximity sensor), motion sensors (e.g., accelerometer, pedometer, gyroscope), environmental sensors (e.g., camera, microphone, ambient light sensor, air-humidity sensor, and thermometer), and biometric sensors (e.g., heart rate sensor, blood pressure sensor). Combining these sensor technologies with self-report method, ESM can capture both subjective experience and objective measures from participants, encompassing in situ physical, psychological, physiological, and contextual information (see Table 1).</p> <p>Table 1 Multimodal data collection through ESM</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Definition</p></th><th align="left"><p>Data collection mechanism</p></th></tr></thead><tbody><tr><td align="left"><p>Physical data</p></td><td align="left"><p>The external behavior records, such as body movement and gestures, activity frequencies, durations, locations, and distances</p></td><td align="left"><p>Sensors in devices</p></td></tr><tr><td align="left"><p>Psychological data</p></td><td align="left"><p>Reflection of mental state during engagement, such as feelings, thoughts, motivation, emotion, or attitude</p></td><td align="left"><p>Self-report</p></td></tr><tr><td align="left"><p>Physiological data</p></td><td align="left"><p>An individual's internal body functions, such as heart rate, body temperatures, blood pressure, breathing respiration, and brain activities.</p></td><td align="left"><p>Sensors in devices</p></td></tr><tr><td align="left"><p>Contextual data</p></td><td align="left"><p>Description the context where engagement occurs, such as how the environment is, whom the participant is with, and what the event is like.</p></td><td align="left"><p>Self-report; Sensors in devices; Retrieved from internet services</p></td></tr></tbody></table> </ephtml> </p> <p>The physical data concerns external behavior records, such as body movement and gestures, activity frequencies, durations, locations, and distances. For example, Sugie ([<reflink idref="bib86" id="ref68">86</reflink>]) used <emph>Survey Droid</emph> to gain objective measures of GPS location to track participants' movement between neighborhood street blocks every 15 min during the daytime. Similarly, utilizing GPS-assisted technologies and timestamp features in the mobile apps, Xie et al. ([<reflink idref="bib97" id="ref69">97</reflink>]) designed the <emph>ESM-Mobile</emph> app to explore college students' behavioral patterns when studying outside of the classroom. Actual study locations were captured with GPS coordinates and further represented with a heat map showing popular study locations on campus. The app recorded students' planned study time and actual study events, which were later interpreted as plan implementation rates.</p> <p>The psychological data refers to the self-reported reflection of mental states during engagement, such as feelings, thoughts, motivation, emotion, or attitude. The data could be obtained from paper forms, web-based surveys, or questions embedded in mobile devices or applications. Those questions could be presented as Likert scale questions. For example, "Please indicate how often you use technology to do the following activities in this class session: Analyze different aspects of a problem or issue. 1 = never, 2 = a few times, 3 = sometimes, 4 = often, 5 = very often" (Vongkulluksn et al., [<reflink idref="bib90" id="ref70">90</reflink>]).</p> <p>The physiological data conveys precise information about an individual's internal body functions, such as heart rate, body temperatures, blood pressure, breathing respiration, and brain activities. The data can be collected through wearable devices and smart mobile applications. For instance, to study participants' eating behaviors during the day, Oh et al. ([<reflink idref="bib66" id="ref71">66</reflink>]) monitored participants' heart rate, glucose level, stress level, and emotions using <emph>Fitbit</emph> devices and <emph>Personicle</emph> on the phone. In Conrad and Newman's ([<reflink idref="bib19" id="ref72">19</reflink>]) ESM study of mind wandering activities during online lectures, they collected electroencephalography (EEG) recordings from EEG caps to observe brain activity. They found an association between mind wandering—as measured by delta, theta, and alpha band activities—and reduced learning scores.</p> <p>The contextual data describes the context where engagement occurs such as, how the environment is, whom the participant is with, and what the event is like. This data could be obtained through indicators of the environment, such as weather and temperature. It could be detected through the sensors probing the immediate surroundings (e.g., audio, proximity, and light sensors). It could also be obtained through self-reporting. As an example, Wang et al. ([<reflink idref="bib93" id="ref73">93</reflink>]) developed the <emph>StudentLife</emph> mobile app to study the impact of the workload on life experience and academic performance. It not only detected students' physical activities but also automatically sensed the surroundings by detecting sounds and inferring whether students were around conversations. Regarding the social surroundings, participants could report the information through survey questions like "Who were you with?", "Who were you talking with?" (Cordier et al., [<reflink idref="bib20" id="ref74">20</reflink>]). Questions regarding social activity could also be triggered automatically when the devices detected incoming calls and text messages (Sugie, [<reflink idref="bib86" id="ref75">86</reflink>]).</p> <p>In addition, details of the event could be presented through journaling with descriptions, images, audios, or videos. In Arnold and Casellas Connors's ([<reflink idref="bib2" id="ref76">2</reflink>]) study about first-year underrepresented students' college transition, they used <emph>Instagram</emph> as a platform to collect visual interpretations of student life. Students were asked to send a 5-min audio diary via email and post Instagram photos with captions. The caption provided the interpretation of the image. Images and audio files complemented each other to offer an illustrated picture of the lived experience of college students, revealing deeper insights and nuances of students' experiences than using standard Likert-scale survey questions.</p> <p>Multimodal data complement each other to reveal multifaceted situational engagement. Physical data are directly related to behavioral engagement as they are external behavior records of participants. These data are typically collected passively through wearable or mobile devices, that is, the data collection does not involve participants' active involvement. Comparing to traditional self-report of behavioral engagement (Schmidt et al., [<reflink idref="bib78" id="ref77">78</reflink>]), physical data from ESM are objective measures of learning behavior and may provide multiple indices of behavioral engagement depending on how these physical data are computed into variables (Xie et al., [<reflink idref="bib98" id="ref78">98</reflink>]). Psychological data through self-reporting may reflect what participants are thinking (e.g., cognitive engagement) and how they are feeling (e.g., emotional engagement). These internal psychological processes can be externalized by simply asking participants questions related to their cognition and emotion (Greene, [<reflink idref="bib33" id="ref79">33</reflink>]). Yet the self-report approach, even in ESM, is distracting and requires participants to stop what they are doing and answer a set of questions. It is also subjective and relies on participants' memory and honesty. On the other hand, physiological data measure participants' internal body functions that are often associated with their psychological processes, therefore, they can also be used as measures of cognitive and emotional engagement (AlZoubi et al., [<reflink idref="bib1" id="ref80">1</reflink>]). Physiological data collections are passive and objective. In order to build reliable and valid measures of cognitive and emotional engagement using physiological data, computational models need to be developed and validated. In our review of the current literature, these models are built and tested primarily using the learning analytics approach with data collected online or in laboratory settings (Baker et al., [<reflink idref="bib4" id="ref81">4</reflink>]; Calvo & D'Mello, [<reflink idref="bib13" id="ref82">13</reflink>]; D'Mello and Graesser, [<reflink idref="bib24" id="ref83">24</reflink>]). Future research needs to look into building computational models of cognitive and emotional engagement using physiological data or a combination of psychological and physiological data in ESM.</p> <hd id="AN0180830877-11">Contrasting ESM with traditional self-report method</hd> <p>Table 2 summarizes major distinctions between traditional self-report method and experience sampling method. In addition to the timing of data collection, ESM differs from traditional self-report method in terms of the frequency of data collection, signal method, data format, data quality, and research orientation and focus.</p> <p>Table 2 Comparison between traditional self-report method and ESM</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Traditional self-report method</p></th><th align="left"><p>Experience sampling method</p></th></tr></thead><tbody><tr><td align="left"><p>When</p></td><td align="left"><p>Before or after the event</p></td><td align="left"><p>In the moment of the event</p></td></tr><tr><td align="left"><p>Where</p></td><td align="left"><p>Out of the event context</p></td><td align="left"><p>In the event context</p></td></tr><tr><td align="left"><p>Frequency</p></td><td align="left"><p>One or a few times</p></td><td align="left"><p>Intensive repeated measures</p></td></tr><tr><td align="left"><p>Signal method</p></td><td align="left"><p>In-person or email</p></td><td align="left"><p>Signal devices (beep machine; or mobile phones)</p></td></tr><tr><td align="left"><p>Data format</p></td><td align="left"><p>Longer comprehensive self-report surveys</p></td><td align="left"><p>Short self-report surveys,</p><p>Multimodal data (physical, physiological, psychological data)</p></td></tr><tr><td align="left"><p>Quality of responses</p></td><td align="left"><p>Participants being able to spend a larger amount of time in response to a longer set of questionnaires; Variables assessed with multiple items with higher psychometric quality; Self-reporting out of context therefore lower ecological validity.</p></td><td align="left"><p>Participants responding to short surveys quickly; Within context data collection therefore better ecological validity. Short surveys with a few items (or single question) therefore lower psychometric quality. Frequent surveys may cause response fatigue.</p></td></tr><tr><td align="left"><p>Research orientation</p></td><td align="left"><p>Based on individuals' projection or retrospection (research at person level)</p></td><td align="left"><p>Based on individual's in-situ experiences (research at situational level)</p></td></tr><tr><td align="left"><p>Research focus</p></td><td align="left"><p>Between-subject</p></td><td align="left"><p>Within-subject across contexts and time</p></td></tr></tbody></table> </ephtml> </p> <p>While ESM has many advantages over the traditional self-report method, researchers should use these methods in complement to each other, rather than replacing one with another. For example, while ESM collects data in-situ and has better ecological validity, ESM surveys are usually very short and possibly with single-item measures. Otherwise, long ESM surveys can cause response fatigue. Compared to traditional self-report surveys, ESM surveys may have lower psychometric quality (e.g., reliability). We suggest researchers to adopt or combine these research methods according to the specific research aims, orientation, and focus.</p> <hd id="AN0180830877-12">ESM sampling approaches and data characteristics</hd> <p>There are four sampling approaches in ESM. Random sampling assesses participants randomly within a certain time frame. Fixed sampling collects data from participants at set time points or in a fixed interval. Event-based sampling triggers data collection during or after specific events, and context-aware sampling occurs according to the specific context or environment. Characteristics related to time interval, number of data points, and sampling accuracy give rise to the unique data structures collected via ESM (Table 3).</p> <p>Table 3 Characteristics of ESM data</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Prompt schedule</p></th><th align="left"><p>Time interval</p></th><th align="left"><p>Number of data points</p></th><th align="left"><p>Sampling accuracy</p></th></tr></thead><tbody><tr><td align="left"><p>Fixed sampling</p></td><td align="left"><p>Assessments at fixed intervals or fixed schedules</p></td><td align="left"><p>Fixed</p></td><td align="left"><p>Fixed</p></td><td align="left"><p>Low</p></td></tr><tr><td align="left"><p>Random sampling</p></td><td align="left"><p>Assessments at random intervals or random points within a time frame</p></td><td align="left"><p>Varying</p></td><td align="left"><p>Fixed/Varying</p></td><td align="left"><p>Low</p></td></tr><tr><td align="left"><p>Event-based sampling</p></td><td align="left"><p>Assessments triggered during/after relevant activities</p></td><td align="left"><p>Varying</p></td><td align="left"><p>Varying</p></td><td align="left"><p>High</p></td></tr><tr><td align="left"><p>Context-aware sampling</p></td><td align="left"><p>Assessments when participants are detected to engage in relevant contexts</p></td><td align="left"><p>Varying</p></td><td align="left"><p>Varying</p></td><td align="left"><p>High</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0180830877-13">Time intervals</hd> <p>One defining characteristic of ESM is repeated data collection. The four ESM sampling approaches differ from each other on the time interval characteristics of data collected. Fixed sampling is the most rigid of all ESM sampling types, assessing participants at specific set times. Data from fixed sampling has a fixed interval, such as an interval of 24 h when participants are assessed at the same time every day. In contrast, the other three sampling approaches result in varying time intervals, where the elapsed time between adjacent time points differ both within participants and across different participants. In random sampling, participants are randomly sampled within a specific period until a set number is reached. Depending on when the participants respond, the time intervals would vary. In both event-based and context-aware sampling, participants are prompted to answer ESM surveys when engaging in relevant activities or contexts, which may occur inconsistently through time. These three sampling approaches produce "imbalanced" data where varying distances between measurements give rise to special analytical considerations (Geiser, [<reflink idref="bib30" id="ref84">30</reflink>]).</p> <hd id="AN0180830877-14">Number of data points</hd> <p>ESM produces intensive longitudinal datasets that include a large number of data points. These datasets are especially useful for examining trends of psychological characteristics across different contexts and time spans. In a fixed sampling design, the researcher typically pre-set the desired number of data points to be collected. Therefore, the expected or maximum number of data points is consistent among participants. In a random sampling design, the number of data points can vary or be fixed. In one random sampling design, participants are prompted a set number of time per time period without regard for whether the survey prompt is answered (e.g., van Berkel et al., [<reflink idref="bib88" id="ref85">88</reflink>]; Broda, [<reflink idref="bib11" id="ref86">11</reflink>]). In this case, the number of data points may vary widely depending on whether the participants answered the notification when prompted. Another random sampling design may involve prompting participants until a desired number is reached, therefore resulting a fixed number of data points. In an event-based or a context-aware sampling design, the number of data points likely varies depending on how many times participants engage in relevant activities or contexts. For example, in one of our previous studies on college students' out-of-classroom studying, students pre-scheduled likely times they will study and were subsequently prompted during those times (Xie et al., [<reflink idref="bib98" id="ref87">98</reflink>]). Thus, the number of prompts varied widely among participants, with some scheduling a few study sessions while others scheduling a numerous amount. Event-based and context aware, as well as some random sampling designs thus give rise to an additional complexity related to the imbalanced nature of ESM data. That is, the data is inconsistent in both the time interval between data points as well as the number of data points across participants.</p> <hd id="AN0180830877-15">Sampling accuracy and missing data</hd> <p>Another key consideration is the sampling accuracy of ESM sampling approaches. Because ESM data collection aims to collect data in authentic contexts, ESM prompts are designed to "catch" participants as they are engaging in relevant activities. Different scheduling approaches offer various timing mechanisms, but the ultimate goal is to maximize the number of data points collected from participants as they engage in activities of interest. Xie et al. ([<reflink idref="bib97" id="ref88">97</reflink>]) distinguished between response rate and sampling accuracy. Response rate refers to the percentage with which participants replied to ESM prompts. In comparison, sampling accuracy refers to participants' indication that they are engaging in relevant activities when prompted. The two are distinguishable in many research settings when the goal is to examine participants engaging in certain activities—such as studying for college classes or using a cellphone. Research on out-of-classroom studying has shown that event-based sampling and fixed sampling yielded about the same response rate, but event-based sampling garnered a higher sampling accuracy (Xie et al., [<reflink idref="bib97" id="ref89">97</reflink>]). This may be especially true when mobile technology is leveraged for participants to schedule their own ESM prompts. Another study found that response rate was higher for event-based sampling (probing usage after cellphone is unlocked), compared to random and fixed sampling (van Berkel et al., [<reflink idref="bib88" id="ref90">88</reflink>]). Relatedly, whereas ESM may inherently produce missing data due to non-response or inaccurate sampling, event-based and context-aware sampling may reduce missingness by probing participants less often with more fidelity.</p> <hd id="AN0180830877-16">Technology support ESM sampling management</hd> <p>Sampling efficiency and accuracy have been the major challenges of ESM studies. Early ESM studies were not efficient in terms of sampling management. Participants were required to keep track of the schedules of sampling times on their own or to fill out a booklet of paper forms when signaled. The burdensome procedures required substantial effort and time from researchers to signal the participants, such as calling the paging devices or calling participants through phones at home. The advent of technologies has improved sampling efficiency and accuracy. It not only integrates the signaling and data collection seamlessly, but also makes data collection less obtrusive through automatically sensing.</p> <hd id="AN0180830877-17">Technology-enhanced signaling approaches</hd> <p>Various technological approaches have been used to support ESM signal prompts. First, email is a convenient and reliable way of sending survey requests to participants. Combined with online survey services (e.g., <emph>Qualtrics</emph> or <emph>Survey Monkey</emph>), emails can effectively serve as a channel for delivering ESM surveys.</p> <p>Second, with the widespread use of mobile phones, text messaging or short message services (SMS) became another popular ESM methods. Participants are typically familiar with the existing phone interface, reducing the burden of learning unconventional devices. Researchers can send text messages directly or through commercial text message platforms to prompt participants to answer ESM surveys at pre-determined times. Leveraging the commercial web-based platform <emph>SurveySignal</emph>, Lawson ([<reflink idref="bib46" id="ref91">46</reflink>]) signaled students via text messages to complete an assessment for sexist and major-related experiences in the previous hour during four timeframes within a day. Xie et al. ([<reflink idref="bib97" id="ref92">97</reflink>]) used <emph>Remind</emph>, a web-based text messaging platform, to send text reminders with Qualtrics survey links to participants. Students received prompts 3 times a day for 5 days asking whether they were studying or not and report their engagement levels.</p> <p>Third, using mobile apps to deliver reminders and present prompts has been one of the common trends of ESM design. Some researchers used social apps that are familiar to participants in their daily life. Martínez-Sierra et al. ([<reflink idref="bib57" id="ref93">57</reflink>]) and Savadova ([<reflink idref="bib77" id="ref94">77</reflink>]) used <emph>WhatsApp</emph> to send prompts and gather audio reflections, video recordings, and pictures of the context. Participants can interact with WhatsApp during relevant events, such as after a class session; or they can respond to WhatsApp at random times when they receive messages from the researchers. Another approach is to utilize available specialized ESM mobile applications to prompt participants, such as <emph>Paco</emph> (Broda, [<reflink idref="bib11" id="ref95">11</reflink>]), <emph>MovisensXS</emph> (Rottweiler et al., [<reflink idref="bib76" id="ref96">76</reflink>]) and <emph>Life Data</emph> (North, [<reflink idref="bib64" id="ref97">64</reflink>]). Signals can be scheduled randomly, at fixed times, or during relevant events. Questions are typically imbedded in the mobile apps. To improve response rate, some studies improved the prompt response mechanism, such as pushing pop-up messages to the phone screen or integrating the prompt with the phone's unlocking features. Chan et al. ([<reflink idref="bib14" id="ref98">14</reflink>]) used <emph>Quedge</emph>t platform to present prompts at pre-determined times. Zhang ([<reflink idref="bib101" id="ref99">101</reflink>]) designed <emph>LogIn</emph>, a single slide-unlock journaling tool to support health and wellness measures. Participants unlock the phone screens by clicking the slide measure to provide answers to survey questions.</p> <p>In addition, diverse smart mobile devices opened new means for participants to interact with the prompts, such as iPod Touch (Cordier et al., [<reflink idref="bib20" id="ref100">20</reflink>]; Muenks et al., [<reflink idref="bib61" id="ref101">61</reflink>]), iPad (Chang & Taxer, [<reflink idref="bib15" id="ref102">15</reflink>]; Xie et al., [<reflink idref="bib97" id="ref103">97</reflink>]), Fitbit (Oh et al., [<reflink idref="bib66" id="ref104">66</reflink>]), and smartwatches and glasses (Hernandez et al., [<reflink idref="bib38" id="ref105">38</reflink>]). Some researchers also combined more than one device to collect ESM data. For example, one study used <emph>Pebble</emph> smartwatches as signaling devices and other devices as survey delivering tools (Hochbein et al., [<reflink idref="bib40" id="ref106">40</reflink>]). Another study used <emph>Fitbit</emph> to collect physiological data and mobile apps on smart phones to collect participants' responses to survey prompts (Oh et al., [<reflink idref="bib66" id="ref107">66</reflink>]).</p> <p>There are several existing systematic reviews that have provided analyses of ESM software and their functionalities (see details in de Vries et al., [<reflink idref="bib25" id="ref108">25</reflink>]; Pejovic et al., [<reflink idref="bib68" id="ref109">68</reflink>]; van Berkel et al., [<reflink idref="bib87" id="ref110">87</reflink>]). As technology continues to evolve, ESM can be further integrated into the learning environment to seamlessly interface with students in assessing their engagement in the moment. For example, van Berkel et al. ([<reflink idref="bib87" id="ref111">87</reflink>]) pointed out that the use of chat-bots can be a potential tool for future ESM data collections although it has not been explored widely. The advantage of using a chat-bot is that it has been installed within existing common communication applications, reducing the need to register participants. Bemmann et al. ([<reflink idref="bib7" id="ref112">7</reflink>]) found that participants involved in a chat-bot study reported fewer missed notifications and preferred chat-bot ESM tools rather than apps. These chat-bots powered by natural language interpretation can adjust the follow-up interview questions, which made the participants feel like they were involved in a conversation with a live person.</p> <hd id="AN0180830877-18">Making event-based and context-aware sampling feasible</hd> <p>Event-based sampling approach requires participants to respond to ESM data collection triggered by specific events. This can be difficult to achieve because researchers may not be able to predict when the event would occur, especially in dealing with events that cannot be pre-scheduled (e.g., do a survey every time when a participant drinks water) or that are personalized (e.g., each participant has their own event schedule).</p> <p>Technology makes event-based sampling more feasible. One way to achieve event-based sampling is to simply ask the participants to tell us their schedules when relevant events will likely happen. In our previous study, we developed an ESM-Mobile approach called <emph>Study Scheduler</emph> available as an iOS and Android app. Students set up their own study events in a mobile app calendar and the app prompts them to answer ESM surveys during those events. Through the app calendar, we asked participants to tell us when the events would occur, so that we can prompt them with ESM surveys during likely times they will be engaging in relevant activities (Xie et al., [<reflink idref="bib97" id="ref113">97</reflink>]).</p> <p>Another way to achieve event-based sampling is through smart technologies. The recent development of algorithm modeling techniques has allowed researchers to incorporate machine learning to infer about the information extracted from sensors and past data collected through ESM. In doing so, ESM will be able to predict or detect when the event is occurring and initiate ESM data collection without user intervention. Pejovic and Musolesi ([<reflink idref="bib67" id="ref114">67</reflink>]) used <emph>SampleMe</emph> to collect users' survey data (activity, emotion, and sentiment for interruption) and sensor data (location, time, accelerameter, and company) to develop an intelligent prompting mechanism, <emph>InterruptMe</emph>. <emph>InterruptMe</emph> can build personalized classifiers for opportune interruptions, leading to an increased user satisfaction and faster response time. Similar approaches can be found in Oh et al.'s ([<reflink idref="bib66" id="ref115">66</reflink>]) study of food journaling. They extracted multi-modal data from <emph>Fitbi</emph>t and <emph>Personicle</emph> and built a model to classify daily activites into non-eating moments and eat moments by using heartrate and stepcounts. <emph>Personicle</emph> created an event-triggered ESM when recognizing an eating moment and asked participants to make a verbal food journal entry that was later converted into text via <emph>Google Voice</emph> API. The eating context information could be recorded comprehensively, including the stress level, glucose level, emotion, weather, location, company, previous activities, as well as the food eaten and its correspondent nutritional information. The automatic process reduced the journaling burdens from participants and improved the accuracy of food journaling.</p> <p>In addition, the combination of mobile devices and sensor technologies makes it ideal to capture contextual information. As participants carry the mobile devices around with them, the prompts will be automatically triggered when the context changes or corresponds to a pre-set condition—an application of context-aware sampling. For example, Markkannen et al. ([<reflink idref="bib52" id="ref116">52</reflink>]) utilized location data to study workspace usage. Participants were asked to carry their mobile phones during the study. Questions were prompted when the device sensed the movement of the phones, which indicated that the participants were changing location in the space and switching between tasks. The ESM questions further inquire participants about the location of the space, the task partners, and the affordance of the spaces.</p> <hd id="AN0180830877-19">Technology supports ESM data management</hd> <p>Due to intensive nature of ESM, data management in ESM studies can be challenging. Traditional ESM studies with paper-based forms are labor-intensive since all data needed to be entered by hand or scanned for analysis, resulting in much human error (Barret & Barret, [<reflink idref="bib6" id="ref117">6</reflink>]). Functionalities of the Internet can ease data management burdens, reduce the logistical issues and enhance the ecological validity of responses. Surveys can be delivered through emails, text messages, or mobile apps. Web-based surveys using online survey platforms (e.g., <emph>Qualitrics</emph> and <emph>SurveyMonkey</emph>) or commercial ESM survey services (e.g., <emph>SurveySignal</emph>) have revolutionized data administration in many ways. The technology enables extensive data collection since it is low-cost and accessible to participants who have access to the Internet. The large volume of responses can be returned in a minute and stored automatically without manual data entry in web-based databases (Hardre et al., [<reflink idref="bib34" id="ref118">34</reflink>]). This also improves the precision of data entry and reduces human error. Each data entry could be labeled with a precise timestamp, such as the local prompting time, reply time, and the signal numbers (Hofmann & Patel, [<reflink idref="bib41" id="ref119">41</reflink>]). When compared to paper-based survey administration, web-based survey methods showed no difference on psychometric quality indicators such as internal consistency of subscales, positive-response bias, strength of interscale correlations (Hardre et al., [<reflink idref="bib35" id="ref120">35</reflink>]), and the overall quality of completeness, coherence, and correctness (Hardre et al., [<reflink idref="bib36" id="ref121">36</reflink>]). Additionally, technology survey administration allows researchers to control the response time intervals. Researchers can specify a time window for participants to respond to the survey. The surveys can be configured to expire after a set period, thus improving data quality (van Berkel et al., [<reflink idref="bib87" id="ref122">87</reflink>]). Importantly, this strategy helps researchers to address compliance issues. Researchers can identify if the participants backfill survey questions through submission timestamps and remove inaccurate answers (Arnold & Rohn, [<reflink idref="bib3" id="ref123">3</reflink>]).</p> <p>While smart devices have increased the efficiency of data collection, the complex signaling process, automatic sensing, and computing may stress the resources of devices. To reduce resource usage, researchers have leveraged cloud infrastructure to offload large-scale data to backend servers, which enables the implementation of more accurate algorithms to compute and interpret data patterns (Lane et al., [<reflink idref="bib43" id="ref124">43</reflink>]).</p> <hd id="AN0180830877-20">Statistical approaches for modeling ESM data</hd> <p>As previously discussed, data collected via ESM is often voluminous and imbalanced (Vongkulluksn & Xie, [<reflink idref="bib89" id="ref125">89</reflink>]). Special types of statistical modeling are needed to handle this unique data structure in order to maximize its potential for scientific discovery. In this section, we will describe statistical approaches that are well-suited to handle complex data from each type of ESM sampling. The affordances and limitations of statistical modeling approaches will be enumerated, such as those for hierarchical linear modeling (Rabe-Hesketh & Skrondal, [<reflink idref="bib72" id="ref126">72</reflink>]; Raudenbush & Bryk, [<reflink idref="bib73" id="ref127">73</reflink>]), latent profile analysis (Muthén, [<reflink idref="bib63" id="ref128">63</reflink>]; Nylund-Gibson & Choi, [<reflink idref="bib65" id="ref129">65</reflink>]), as well as single-level and multilevel latent state-trait structural equation modeling (SEM; Geiser, [<reflink idref="bib30" id="ref130">30</reflink>]). While these modeling approaches can be used with data other than those collected via ESM, we will describe how they can be used to answer specific research questions with the nuance that ESM data provides. Further, we will match characteristics of these modeling approaches with the nature of complexity inherent within fixed, random, event-based, and context-aware ESM sampling.</p> <p>We focus on two broad strands of questions related to engagement: situational and longitudinal (Table 4). We note that this is certainly not an exhaustive list of statistical modeling approaches that can be used to examine engagement with ESM data. Rather, we use this list to demonstrate the range of research questions that could be answered utilizing this methodology.</p> <p>Table 4 Statistical modeling considerations</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Descriptive</p></th><th align="left"><p>Predictive</p></th><th align="left"><p>Stability vs. Volatility</p></th></tr></thead><tbody><tr><td align="left" colspan="4"><p>Situational strand of ESM studies</p></td></tr><tr><td align="left"><p> Focal research question</p></td><td align="left"><p>What is the characteristic of the situational engagement?</p></td><td align="left"><p>What are the factors associated with situational engagement?</p></td><td align="left"><p>What is the decomposition of stability versus volatility in engagement?</p></td></tr><tr><td align="left"><p> Analytical modeling</p></td><td align="left"><p>Momentary Latent Profile Analysis</p></td><td align="left"><p>Multilevel HLM Regression</p></td><td align="left"><p>Multilevel Latent State-Trait Modeling</p></td></tr><tr><td align="left"><p> Measurement assumptions</p></td><td align="left" colspan="3"><p>Measurement invariance is assumed; Engagement is assumed not to increase or decrease systematically over time.</p></td></tr><tr><td align="left"><p> Data characteristics</p></td><td align="left" colspan="3"><p>Varying time intervals and time points acceptable, missing data handled implicitly through multi-level modeling.</p></td></tr><tr><td align="left" colspan="4"><p>Longitudinal strand of ESM studies</p></td></tr><tr><td align="left"><p> Focal research question</p></td><td align="left"><p>What is the characteristic of the change in engagement over time?</p></td><td align="left"><p>What are the factors associated with the change trajectory of engagement?</p></td><td align="left"><p>To what extent is the stability of engagement maintained over time?</p></td></tr><tr><td align="left"><p> Analytical modeling</p></td><td align="left"><p>Latent Curve Modeling; Latent Growth Class Analysis</p></td><td align="left"><p>Conditional Latent Curve Modeling; Growth Mixture Modeling</p></td><td align="left"><p>Autoregressive Cross Lagged Modeling</p></td></tr><tr><td align="left"><p> Measurement assumptions</p></td><td align="left" colspan="3"><p>Measurement invariance can be tested; Systematic increase or decrease over time can be modeled and examined.</p></td></tr><tr><td align="left"><p> Data characteristics</p></td><td align="left" colspan="3"><p>Need consistent time intervals and number of time points, missing data can be handled via estimation methods</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0180830877-21">Situational strand of ESM studies</hd> <p>In education research, <emph>situational</emph> research questions probe the ways in which students' behaviors, cognition, and motivation may differ across contexts depending on learning tasks and environments (Lavigne & Vallerand, [<reflink idref="bib45" id="ref131">45</reflink>]; Xie et al., [<reflink idref="bib98" id="ref132">98</reflink>]). ESM data is especially well-suited to answer these questions because data can be collected in the moment as students are engaging in focal learning tasks, and collected alongside crucial contextual information.</p> <hd id="AN0180830877-22">Descriptive</hd> <p>The first focal research question under the situational strand of ESM studies is descriptive: What are the characteristics of situational engagement? Beside running descriptive statistics depicting the central tendency and dispersion characteristics (e.g., mean and standard deviation), ESM data is particularly well-suited for examining how different dimensions of engagement co-occur together. For example, researchers can examine patterns of association among behavioral, cognitive, affective, and social engagement within situationally-specific data, and link these patterns to contextual features of the learning environment. This requires a person-centered approach which goes beyond variable-centered associations and teases apart naturally occurring combinations of engagement dimensions. Latent Profile Analysis (LPA) is a statistical modeling method that uses observed scores to identify groups of people who have similar characteristics (Muthén, [<reflink idref="bib63" id="ref133">63</reflink>]; Nylund-Gibson & Choi, [<reflink idref="bib65" id="ref134">65</reflink>]). Essentially, the LPA model specifies that the observed continuous indicators such as engagement scores are measures of the underlying latent profile. Several types of indices guide model selection of the number of underlying groups, including information criteria, likelihood ratio tests, and indices indicating, differentiation of profiles. After a model is selected—that is, the number of underlying groups is determined—individuals are classified into their most likely profile membership. Conditional means and variances of the indicators are estimated and used to describe characteristics of each profile. Recent developments in Multilevel Latent Profile Analysis (MLPA, Makikangas et al., [<reflink idref="bib50" id="ref135">50</reflink>]) can also extend LPA modeling specifications to nested data, such as time points nested within person like data derived from ESM. MLPA can discern Level-2 groupings, such as grouping Level-2 classes based on each person's frequency of Level-1 profile membership.</p> <hd id="AN0180830877-23">Predictive</hd> <p>The second focal research question under the situational strand of ESM studies is predictive: What are factors impacting situational engagement? This question probes variabilities in factors that impact engagement in various learning contexts. As previously noted, this type of data has an inherent hierarchical structure, with observations nested within students (Rabe-Hesketh & Skrondal, [<reflink idref="bib72" id="ref136">72</reflink>]; Raudenbush & Bryk, [<reflink idref="bib73" id="ref137">73</reflink>]). Regression within the Hierarchical Linear Modeling (HLM) is well-suited to discern how independent variables predict an outcome variable within nested data. HLM regression accounts for within-individual dependence by representing observations as: (<reflink idref="bib1" id="ref138">1</reflink>) a random effect representing differences between the person-level mean and the sample mean and (<reflink idref="bib2" id="ref139">2</reflink>) a within-person residual representing deviations of each observed score and the person-level mean. For example, focal factors of engagement could be examined for their association with situational engagement, taking into account how engagement scores from one student may be more similar across observations compared to those between different students.</p> <hd id="AN0180830877-24">Stability vs. volatility</hd> <p>Another key question under the situational strand of ESM studies is the relative stability versus volatility of a psychological construct. For engagement, stability refers to how each student tends to be engaged in a certain way in school or in a certain class over time, whereas volatility refers to how the student has a specific response to a learning task (Vongkulluksn & Xie, [<reflink idref="bib89" id="ref140">89</reflink>]). That is, there is an element within each measure of engagement that is contributed by how students typically engage in academic activities and another element that is contributed by students' reaction to a particular learning task. The key driving question is: What is the decomposition of stability versus volatility in engagement? We can recast this question as the extent to which engagement exists at the "trait" versus "state" levels (Curran & Bauer, [<reflink idref="bib22" id="ref141">22</reflink>]; Geiser, [<reflink idref="bib30" id="ref142">30</reflink>]). Latent State-Trait (LST) structural equation models explicitly addresses stability versus variability in longitudinal data. LST models specify that each observed score is a function of a latent trait variable representing person-level effects, a latent state variable representing event-level residual fluctuations, and a residual term representing measurement error. LST models can leverage multiple indicators of engagement, specifying engagement as consisting of one underlying trait (Singletrait-Multistate, STMS) or multiple underlying traits with each item representing one trait (Multitrait-Multistate, MTMS). Additionally, a multilevel specification is possible so that imbalanced ESM data can be modeled (Geiser, [<reflink idref="bib30" id="ref143">30</reflink>]). In multilevel LST specification, the latent state factor and error residuals are estimated at level 1 (event level) and the latent trait factors are estimated as random intercept parameters at level 2 (person level) similar to HLM regression.</p> <hd id="AN0180830877-25">Relative advantages and disadvantages</hd> <p>All three situational sub-strands use analytic strategies which leverage multilevel modeling. Since situational research questions focus on each specific instance of engagement, these modeling techniques are able to narrow the scope to focus on individual time points nested within individual students. Multilevel modeling is advantageous for ESM data because imbalanced data such as those collected via random, event-based, and context-aware sampling can be implicitly handled via the specification of random effect at the individual level (Geiser et al., [<reflink idref="bib31" id="ref144">31</reflink>]). Each observed score does not need to be modeled explicitly, making models more compact and straightforward. Similarly, missing data can be handled with the same random effect specification. With the intensive longitudinal design of ESM, this implicit handling of missing data is a major advantage. The main disadvantage of the multilevel specification is the lack of the ability to test for measurement invariance. In single-level analysis, assumptions regarding the time invariance of intercepts, factor loadings, and residual variances can be tested. However, time invariance assumptions of these parameters allow for the more compact multilevel specification. The imbalanced nature of data that is well suited for these analytic strategies also make testing for measurement invariance difficult. Additionally, situational analytic strategies assumes that there is no systematic decrease or increase in engagement (e.g., growth) over time. The change in engagement over time is not explicitly modeled or examined, which is the main research focus in the longitudinal strand below.</p> <hd id="AN0180830877-26">Longitudinal strand of ESM Studies</hd> <p>Another strand of ESM studies addresses <emph>longitudinal</emph> research questions, focusing on the changing processes of learning-related variables. Longitudinal research questions related to engagement can help map out how students' engagement may change over time as a result of maturation, seasonal changes, or instructional intervention (Xie et al., [<reflink idref="bib100" id="ref145">100</reflink>]).</p> <hd id="AN0180830877-27">Descriptive</hd> <p>Similar to the situational strand, the first focal research question under the longitudinal strand of ESM studies is descriptive: What is the characteristic of the change in engagement over time? An analytical approach that is particularly well-suited to examining change over time is Latent Curve Modeling (LCM; Bollen and Curran, [<reflink idref="bib9" id="ref146">9</reflink>]). An LCM model of engagement would estimate how engagement changes over time for each student and compare the change trajectories across students. Repeated measures of engagement are inferred to be a result of an underlying latent change trajectory. Each observed score is posited to be a function of a latent intercept, a latent slope for each time increment, and a residual term. All factor loadings from each latent intercept to observed scores are fixed at 1 and all factor loadings from each latent slope is coded according to the time increment (e.g., 1, 2, 3, etc.). LCM is thus able to describe the change trajectories of engagement over time in terms of the starting level of engagement at time = 0 and how much it increases or decreases over time. Alternative specifications other than a linear trajectory are available, including a quadratic trajectory with changing slope over time and a piecewise function with slope changes at specified cut points. These alternative specifications of form allow for more flexible modeling of the changes in engagement over time. While LCM is a variable-centered approach to describing the change process, Latent Class Growth Analysis (LCGA) is a person-centered approach which envisions the change process as person-specific (Muthén & Muthén, [<reflink idref="bib62" id="ref147">62</reflink>]). Similar to LPA, LCGA specifies that an underlying latent class contributes to the observed repeated measures. Different change trajectories are estimated for each class along with class probabilities. Just as in LPA, the posterior probabilities of latent class membership for each student can be computed so that we may characterize the engagement trajectories of those in each class. LGCA is thus another approach to describe the change trajectory of engagement captured in longitudinal data, such as those collected via ESM.</p> <hd id="AN0180830877-28">Predictive</hd> <p>The second focal research question under the longitudinal strand is predictive: What are factors associated with the change trajectory of engagement? One analytical method is the conditional LCM that is an extension of the general LCM that is unconditional—a variable-centered approach. While an unconditional LCM model has no predictors other than time, a conditional LCM model adds either time invariant or time varying covariates of the change trajectory (Bollen & Curran, [<reflink idref="bib9" id="ref148">9</reflink>]). Time invariant covariates are specified as predictors of the latent intercept and slope terms. Students who are characterized by different levels of the time invariant covariate thus have different intercepts and slopes for their growth trajectories, similar to an interaction effect in multiple regression (Curran et al., [<reflink idref="bib23" id="ref149">23</reflink>]). Time varying covariates are most simply specified as predictors of each contemporaneous observed measures of engagement. The latent intercept and slope factors thus represent the change trajectory of engagement adjusted for the covariate. For the person-centered approach, general growth mixture modeling (GGMM) presents and extension of LCGA with a model that estimates the mean growth curve and growth factor variances for each class, as well as assesses the association between predictors and the latent trajectory classes (Muthén & Muthén, [<reflink idref="bib62" id="ref150">62</reflink>]). As such, memberships in the latent trajectory classes are regressed on predictors as a multinomial logistic regression model, where odds of membership in a specific latent class can be compared to the reference group conditioned on values of the predictor. Both conditional LCM and GGMM are extensions of more descriptive methods in which predictors can be assessed for their association with change trajectories in engagement.</p> <hd id="AN0180830877-29">Stability vs. volatility</hd> <p>The third focal research question under the longitudinal strand addresses stability versus volatility: To what extent is the stability of engagement maintained over time? In many naturalistic contexts, engagement measured at one time point may have carry over effects to the next adjacent time point (Geiser, [<reflink idref="bib30" id="ref151">30</reflink>]; Vongkulluksn & Xie, [<reflink idref="bib89" id="ref152">89</reflink>]). Further, engagement may be relatively stable over short periods of time when students are engaged in similar tasks, but may be characterized by increasing levels of volatility as time duration increases (Prenoveau, [<reflink idref="bib71" id="ref153">71</reflink>]). In these measurement situations, the latent state-trait model with autoregressions (LST-AR) can decompose latent "trait" and "state" factors as well as model patterns of stability over time (Prenoveau, [<reflink idref="bib71" id="ref154">71</reflink>]). The LST-AR model specification is similar to a single-level LST model, with the addition of autoregressive paths from the latent state factor of one time point to the next time point. The autoregressive paths are in effect regression coefficients of engagement at time t on engagement at time t-1, after controlling for the stable "trait" factor. Interpretations based on the autoregressive paths reveals how the state-level engagement is dependent on previous engagement. Researchers can also examine how the total variance of an observed measure of engagement can be explained by the latent trait factor, the autoregressive pathway from time t-1, and the portion left unexplained. Additionally, we can examine how engagement "propagates" down the time lag (e.g., the influence of engagement at time t-2 on engagement at time t) using the path tracing rule by multiplying the two autoregressive paths (Prenoveau, [<reflink idref="bib71" id="ref155">71</reflink>], p. 740). These analyses help researchers understand how engagement measured at one time point affects another, revealing the extent to which the stability in engagement is maintained over time.</p> <hd id="AN0180830877-30">Relative advantages and disadvantages</hd> <p>Statistical methods aligned with the longitudinal strand intentionally map how engagement changes over time. This is a distinct advantage over the situational lens, which assume that there are no systematic changes over time. ESM studies which aim to investigate engagement over a prolonged period of time could benefit from longitudinal analyses since the assumption of no systematic change in engagement may not be tenable (Vongkulluksn & Xie, [<reflink idref="bib89" id="ref156">89</reflink>]). Relatedly, longitudinal research questions are particularly dependent upon whether properties of engagement measures are stable over time—or measurement invariance (Meredith, [<reflink idref="bib58" id="ref157">58</reflink>]; Millsap, [<reflink idref="bib59" id="ref158">59</reflink>]). At the very least, assumptions regarding the structure and equality of factor loadings (configural and weak invariance) should be met in order to make meaningful comparisons of engagement across time points. Longitudinal latent variable modeling methods typically subsume measurement invariance testing as a preliminary analytic step. On the other hand, a balanced data structure with equal time intervals and number of time points is required for measurement invariance testing and longitudinal latent variable modeling. Using the ESM approach, only the fixed sampling schedule would reasonably provide this type of data structure. This limitation hinders possibilities for longitudinal research in many ESM research contexts.</p> <hd id="AN0180830877-31">Empirical examples of ESM studies in diverse contexts</hd> <p>We present recent studies that utilized various technologies and statistical methods to illustrate how ESM has been used to examine engagement. These studies were conducted in diverse contexts including high school vs. college and classroom vs. out-of-classroom settings.</p> <hd id="AN0180830877-32">A study of high school students' engagement in science classrooms</hd> <p>Schmidt et al. ([<reflink idref="bib78" id="ref159">78</reflink>]) used ESM to examine high school students' engagement while engaging in learning tasks in science classrooms. Data was collected from 244 students in 12 classrooms in integrated science, biology, chemistry, and physics. During five consecutive days in the data collection period, students were prompted to respond to an ESM survey at two random times during the focal class. A vibrating pager was used to signal students when they should complete a brief paper-based form with survey questions. After receiving the signal, students completed a paper form answering questions about the activities they are engaging in and their level of engagement. This data collection strategy is a variation of the fixed sampling approach because students were prompted during a fixed window of time every day to respond about the same science class. In total, 4136 ESM responses were collected with a 91% response rate.</p> <p>The study answered the following research questions: (<reflink idref="bib1" id="ref160">1</reflink>) What types of momentary profiles characterize students' engagement in science? (<reflink idref="bib2" id="ref161">2</reflink>) In what ways are particular science learning activities related to students' momentary engagement profiles? (<reflink idref="bib3" id="ref162">3</reflink>) In what ways are student choices during instruction related to students' momentary engagement profiles? These research questions fall within the descriptive and predictive strands in our classification, respectively. Schmidt and associates ([<reflink idref="bib78" id="ref163">78</reflink>]) utilized two-step cluster analysis to examine naturally occurring profiles of different engagement dimensions within each moment—or "Momentary Engagement Profiles" (MEPs). The study identified six profiles. In addition to the universally low, medium, and high profiles, the study also found: (<reflink idref="bib1" id="ref164">1</reflink>) the Reluctant profile with moderate behavioral engagement but low cognitive and affective engagement; (<reflink idref="bib2" id="ref165">2</reflink>) the Pleasurable profile with high affective engagement but moderate to low cognitive and behavioral engagement; and (<reflink idref="bib3" id="ref166">3</reflink>) the Rational profile with low affective, moderate behavioral, and high cognitive engagement. These results gave us a better picture of what engagement looks like in situationally specific ways. The authors also used a series of logistic regression modeled within the multi-level HLM framework to probe how perceptions of choice are related to MEPs. Results showed that choice characteristics mattered for engagement. For example, students who were offered the choice of who to work with were predicted to be 1.4 times more likely to have universally low engagement compared to no choice. In contrast, students who were offered the choice of materials were more likely to have pleasurable engagement and less likely to report universally low engagement. This study descriptively illustrated the characteristics of momentary engagement and identified provision of choice as a key instructional predictor of MEPs.</p> <p>This study investigated engagement in a structured environment of the classroom. An affordance of this context is the proximity to writing materials so that it is convenient for students to fill out and turn in paper-based surveys. Therefore, the pager works well as a technological tool for ESM data collection in this context since it only needs to provide a signal for students to fill out a survey form. Although not detailed in the paper, we inferred that the data collected from this study have consistent intervals between time points and similar number of data points across students (e.g., balanced data structure). The authors posited situational research questions in the study, utilizing multi-level modeling methods. This type of data structure collected from a structured learning environment is also well-suited for answering longitudinal research questions. For example, this data could be used to probe how students' engagement systematically develops over the course of a lesson. Overall, this study was able to examine students' engagement in-the-moment and in context as they're participating in science class activities. This nuanced analysis of students' changing engagement patterns can be cross-validated with cross-sectional studies which examined the overall associations between engagement and instructional features such as choice provision.</p> <hd id="AN0180830877-33">Studies of college students' out-of-classroom learning</hd> <p>When a study aims to examine engagement in an unstructured learning environment, it requires ESM data collection strategies that are different from those used in a structured learning environment. Importantly, when ESM data collection takes place outside a set schedule and context, technological innovations are needed to deliver ESM prompts and surveys in a convenient and accessible manner. In a series of related studies, our team employed an <emph>ESM-Mobile</emph> approach to collect data from college students as they study outside the classroom. In our iOS and Android mobile app called <emph>Study Scheduler</emph>, students plan study events in a calendar interface. The app prompts students during those prescheduled times and asks "Are you studying?" If students indicate that they are studying as planned, they will then be directed to a short ESM survey on their engagement. The ESM survey can be tailored to include any number of self-report items according to the specific research aims. In our 2019 study (Xie et al., [<reflink idref="bib98" id="ref167">98</reflink>]), we collected data from 52 pre-service teacher education students during 2 weeks in the middle and 2 weeks at the end of the semester. The ESM survey included 10 self-reported items measuring study location, reasons for study, cognitive engagement, and task-specific self-efficacy. The Study Scheduler app also collected students' GPS location data, which we used to triangulate the study location students reported on the survey. In total, 299 study events were created and 170 ESM surveys were collected. This ESM data collection represents an event-based sampling strategy. Students planned study events at varying intervals and frequencies, resulting in an imbalanced data structure.</p> <p>Our 2019 study examined the following research questions: (<reflink idref="bib1" id="ref168">1</reflink>) How do contextual features of planned study events influence students' behavioral engagement in studying activities? Do contextual features moderate the relationship between students' self-efficacy and behavioral engagement?; and (<reflink idref="bib2" id="ref169">2</reflink>) How do contextual features of actual study events influence students' in-the-moment cognitive engagement? Do contextual features moderate the relationship between self-efficacy and cognitive engagement? These research questions fall within the situational-predictive strand in our classification. The study employed effects decomposition and multiple regression analyses within the HLM framework (Raudenbush & Bryk, [<reflink idref="bib73" id="ref170">73</reflink>]). Results showed that self-efficacy was a significant predictor of both behavioral and cognitive engagement. Person-level self-efficacy (i.e., students' mean level of self-efficacy across all learning tasks) was found to be a more consistent predictor of engagement compared to event-level self-efficacy (i.e., event-level deviations from person-level mean). Contextual factors were also found to be influential. For example, students who specified that they were studying because of a deadline had a higher association between self-efficacy and shallow cognitive engagement. That is, students with the same level of self-efficacy used more shallow strategies while studying to meet a deadline. This research probed important student- and context-related factors of engagement.</p> <p>In another study employing the <emph>ESM-Mobile</emph> approach, we collected data from 57 undergraduate students in a learning and motivation strategies course (Vongkulluksn & Xie, [<reflink idref="bib89" id="ref171">89</reflink>]). In total, 802 events were scheduled and 327 ESM surveys were ultimately completed. In this study, we answered the following research questions: (<reflink idref="bib1" id="ref172">1</reflink>) How much variance in engagement is at the trait-level vs. state-level?; and (<reflink idref="bib2" id="ref173">2</reflink>) Does the variance decomposition of engagement differ for students with low vs. high intrinsic motivation? These research questions fall within the situational—stability vs. volatility strand. Multilevel, Latent State-Trait modeling was used as the analytic strategy. Results show that all three items used to measure behavioral and cognitive engagement had a relatively higher amount of variance at the trait level. That is, observed scores are relatively more attributable to inter-individual rather than intra-individual differences. Scores varied more across different students compared to the variation over time within each student. Specifically, 42.7 to 47% of engagement across items were stable over time (trait-level variance), whereas 7.3 to 24.4% varied across occasions (state-level variance). The cognitive engagement item "I am aware of what material I did or did not understand" had the lowest variance at the state level. Additionally, comparing across students with low versus high intrinsic motivation, students with high motivation showed more stability in behavioral engagement and more variability in cognitive engagement compared to those in the low motivation group. One interpretation is students who are more motivated may exhibit more varied cognitive engagement patterns across learning tasks. This research demonstrated how engagement can be decomposed into trait- versus state-levels, and such decomposition may differ for students with different characteristics. These results can be combined with results from traditional survey studies which have investigated the relationship between intrinsic motivation and engagement. Such cross-validation would provide the nuanced insights that not only are levels of intrinsic motivation and engagement associated, intrinsic motivation levels may also influence how much engagement varies over time.</p> <hd id="AN0180830877-34">Conclusion and future directions</hd> <p>To summarize, students' engagement has unique characteristics that are situational, longitudinal, and multidimensional. ESM has affordances that are intrinsically correspondent to these characteristics of engagement. Importantly, modern technologies enhance ESM data collection, sampling accuracy and efficiency, and data management capacity. ESM provides researchers with new instrumentations for data collection and related analytical approaches that help advance our understanding of engagement, making it possible to build comprehensive and nuanced theories of engagement in context. While we consider engagement as particularly suited to ESM methods due to their situational, longitudinal, and multi-dimensional nature, we note that other psychological factors such as interest, attribution, and self-efficacy may consist of both situational and longitudinal aspects which may be examined in nuance with ESM data. Therefore, ESM data collection methods and methodological considerations enumerated here may be used more broadly to bring fine-grained understanding of psychological processes, which in turn can inform educational practices and improve student learning in context-specific manners.</p> <p>Engagement has been an important topic in educational technology research. Studies in the field of educational technology often examine students' engagement in technology-enriched environments, such as online and blended learning (Borup et al., [<reflink idref="bib10" id="ref174">10</reflink>]), computer-supported collaborative learning (Li et al., [<reflink idref="bib47" id="ref175">47</reflink>]), and game-based learning (Chen et al., [<reflink idref="bib16" id="ref176">16</reflink>]). Our conceptual framework of engagement being situational, longitudinal, and multi-dimensional may provide a comprehensive approach to the examination of engagement in these complex environments, and also provide insights on how to design these technological environments to support engagement and learning. In addition, current educational technology research has focused on analyzing the intensive data generated by learning technology systems—learning analytics. Learning analytics provide valuable insights about students' "online" engagement with and within technology systems. ESM methodology and technologies complement with learning analytics to provide similar intensive and situated data about students' "offline" engagement—learning without the mediation of technology (Xie et al., [<reflink idref="bib98" id="ref177">98</reflink>]). Both learning analytics and ESM offer new sources of data and analytical approaches that allow for new research questions and explorations that we could not do without them. Therefore, ESM can be an important contribution to educational technology research and development.</p> <p>There are several limitations of our paper. First, while we offer several statistical approaches to analyze ESM data, the list of approaches is intended to be illustrative, but not exhaustive. Other statistical methods could be explored in future discussions. Second, while we introduced several statistical approaches to utilize ESM data, our discussions were limited as prior studies had not cross-validated findings on student engagement across learning environments with various levels of situated complexity. For example, Pekrun & Marsh ([<reflink idref="bib70" id="ref178">70</reflink>]) posited four levels of situated complexity with respect to variations of time and context (i.e., same time/same context, different times/same context, same time/different contexts, different times/different contexts). Future research can extend this line of inquiry by addressing how ESM approaches may be able to identify nuances in learning engagement across variations of time and context. Third, we elaborated two exemplar ESM studies to showcase how ESM and associated statistical methods can be applied in diverse contexts (e.g., high school vs. college and classroom vs. out-of-classroom settings). These examples are limited to the existing ESM research that used the statistical techniques that we highlighted in the article. They both were based on the self-report data, yet do not represent the full scale of affordances of ESM. For instance, they did not use multimodal data that ESM could offer. In addition, our discussion is constrained with the current development of ESM methodology and technologies. With modern technologies continuing to evolve, future research may develop newer and more robust ESM approaches that continue to contribute to our knowledge about student engagement and human learning.</p> <hd id="AN0180830877-35">Acknowledgements</hd> <p>The study reported in this paper is based upon work in the <emph>Influence of Contextual Features on Learning Engagement in Out-of-Classroom Settings for Academically At-Risk College Students</emph> project supported by the Spencer Foundation project. The conclusions and recommendations expressed in this article do not necessarily reflect the views of the Spencer Foundation.</p> <hd id="AN0180830877-36">Declarations</hd> <p></p> <hd id="AN0180830877-37">Conflict of interest</hd> <p>Kui Xie declares that he has no conflict of interest. Vanessa W. Vongkulluksn declares that she has no conflict of interest. Benjamin C. Heddy declares that he has no conflict of interest. Zilu Jiang declares that she has no conflict of interest.</p> <hd id="AN0180830877-38">Ethical approval</hd> <p>All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.</p> <hd id="AN0180830877-39">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0180830877-40"> <title> References </title> <blist> <bibl id="bib1" idref="ref47" type="bt">1</bibl> <bibtext> AlZoubi O, D'Mello SK, Calvo RA. Detecting naturalistic expressions of nonbasic affect using physiological signals. IEEE Transactions on Affective Computing. 2012; 3; 3: 298-310. 10.1109/T-AFFC.2012.4</bibtext> </blist> <blist> <bibl id="bib2" idref="ref48" type="bt">2</bibl> <bibtext> Arnold KD, Casellas Connors I. The 'anti-Instagram': Using visual methods to study the college experiences of underrepresented students. Visual Studies. 2021. 10.1080/1472586X.2021.1893612</bibtext> </blist> <blist> <bibl id="bib3" idref="ref49" type="bt">3</bibl> <bibtext> Arnold KD, Rohn KCKennedy E, Qian Y. Internet-enabled experience sampling: Innovations in signaling and data sources. Advances in educational technologies and instructional design. 2020; IGI Global: 124-151</bibtext> </blist> <blist> <bibl id="bib4" idref="ref50" type="bt">4</bibl> <bibtext> Baker RS, D'Mello SK, Rodrigo MMT, Graesser AC. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies. 2010; 68; 4: 223-241. 10.1016/j.ijhcs.2009.12.003</bibtext> </blist> <blist> <bibl id="bib5" idref="ref25" type="bt">5</bibl> <bibtext> Bandura A. Human agency in social cognitive theory. American Psychologist. 1989; 44; 9: 1175-1184. 10.1037/0003-066X.44.9.1175</bibtext> </blist> <blist> <bibl id="bib6" idref="ref117" type="bt">6</bibl> <bibtext> Barrett LF, Barrett DJ. An introduction to computerized experience sampling in psychology. Social Science Computer Review. 2001; 19; 2: 175-185. 10.1177/089443930101900204</bibtext> </blist> <blist> <bibl id="bib7" idref="ref112" type="bt">7</bibl> <bibtext> Bemmann, F, Schoedel, R, van Berkel, N, & Buschek, D. (2021). Chatbots for experience sampling-initial opportunities and challenges. Joint Proceedings of the ACM IUI 2021 Workshops.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref61" type="bt">8</bibl> <bibtext> Bolger N, Laurenceau JP. Intensive longitudinal methods: An introduction to diary and experience sampling research. 2013; Guilford Press</bibtext> </blist> <blist> <bibl id="bib9" idref="ref146" type="bt">9</bibl> <bibtext> Bollen KA, Curran PJ. Latent curve models: A structural equation perspective. 2006; Wiley-Interscience</bibtext> </blist> <blist> <bibtext> Borup J, Graham CR, West RE, Archambault L, Spring KJ. Academic communities of engagement: An expansive lens for examining support structures in blended and online learning. Educational Technology Research and Development. 2020; 68: 807-832. 10.1007/s11423-020-09744-x</bibtext> </blist> <blist> <bibtext> Broda M. Using multilevel models to explore predictors of high school students' nonresponse in experience sampling method (ESM) studies. Social Science Computer Review. 2017; 35; 6: 733-750. 10.1177/0894439316667049</bibtext> </blist> <blist> <bibtext> Buhs ES, Ladd GW. Peer rejection as antecedent of young children's school adjustment: An examination of mediating processes. Developmental Psychology. 2001; 37; 4: 550-560. 10.1037/0012-1649.37.4.550</bibtext> </blist> <blist> <bibtext> Calvo RA, D'Mello S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing. 2010; 1; 1: 18-37. 10.1109/T-AFFC.2010.1</bibtext> </blist> <blist> <bibtext> Chan L, Swain VD, Kelley C, de Barbaro K, Abowd GD, Wilcox L. Students' experiences with ecological momentary assessment tools to report on emotional well-being. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2018; 2; 1: 1-20. 10.1145/3191735</bibtext> </blist> <blist> <bibtext> Chang ML, Taxer J. Teacher emotion regulation strategies in response to classroom misbehavior. Teachers and Teaching. 2021; 27; 5: 353-369. 10.1080/13540602.2020.1740198</bibtext> </blist> <blist> <bibtext> Chen CH, Law V, Huang K. The roles of engagement and competition on learner's performance and motivation in game-based science learning. Educational Technology Research and Development. 2019; 67: 1003-1024. 10.1007/s11423-019-09670-7</bibtext> </blist> <blist> <bibtext> Christenson S, Reschly AL, Wylie C. Handbook of research on student engagement. 2012; Springer. 10.1007/978-1-4614-2018-7</bibtext> </blist> <blist> <bibtext> Cleary TJ, Zimmerman BJChristenson SL, Reschly AL, Wylie C. A cyclical self-regulatory account of student engagement: Theoretical foundations and applications. Handbook of research on student engagement. 2012; Springer: 237-257. 10.1007/978-1-4614-2018-7_11</bibtext> </blist> <blist> <bibtext> Conrad C, Newman A. Measuring mind wandering during online lectures assessed with eeg. Frontiers in Human Neuroscience. 2021; 15: 697532. 10.3389/fnhum.2021.697532</bibtext> </blist> <blist> <bibtext> Cordier R, Brown N, Chen YW, Wilkes-Gillan S, Falkmer T. Piloting the use of experience sampling method to investigate the everyday social experiences of children with Asperger syndrome/high functioning autism. Developmental Neurorehabilitation. 2014. 10.3109/17518423.2014.915244</bibtext> </blist> <blist> <bibtext> Csikszentmihalyi M, Larson R. Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease. 1987; 175; 9: 526-536. 10.1097/00005053-198709000-00004</bibtext> </blist> <blist> <bibtext> Curran PJ, Bauer DJ. The disaggregation of within-person and between-person effects in longitudinal models of change. Annual Review of Psychology. 2011; 62; 1: 583-619. 10.1146/annurev.psych.093008.100356</bibtext> </blist> <blist> <bibtext> Curran PJ, Bauer DJ, Willoughby MT. Testing main effects and interactions in latent curve analysis. Psychological Methods. 2004; 9; 2: 220-237. 10.1037/1082-989X.9.2.220</bibtext> </blist> <blist> <bibtext> D'Mello S, Graesser A. Dynamics of affective states during complex learning. Learning and Instruction. 2012; 22; 2: 145-157. 10.1016/j.learninstruc.2011.10.001</bibtext> </blist> <blist> <bibtext> de Vries LP, Baselmans BML, Bartels M. Smartphone-based ecological momentary assessment of well-being: A systematic review and recommendations for future studies. Journal of Happiness Studies. 2021; 22; 5: 2361-2408. 10.1007/s10902-020-00324-7</bibtext> </blist> <blist> <bibtext> Deci, E. L. (1992). On the nature and functions of motivation theories. Psychological Science, 3(3), 167–171.</bibtext> </blist> <blist> <bibtext> Deci E, Ryan R. The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry. 2000; 11; 4: 227-268. 10.1207/S15327965PLI1104_01</bibtext> </blist> <blist> <bibtext> Fredricks JA. Engagement in school and out-of-school contexts: A multidimensional view of engagement. Theory into Practice. 2011; 50; 4: 327-335. 10.1080/00405841.2011.607401</bibtext> </blist> <blist> <bibtext> Fredricks JA, Blumenfeld PC, Paris AH. School engagement: Potential of the concept, state of the evidence. Review of Educational Research. 2004; 74; 1: 59-109. 10.3102/00346543074001059</bibtext> </blist> <blist> <bibtext> Geiser C. Longitudinal structural equation modeling with Mplus: A latent state-trait perspective. 2020; The Guilford Press</bibtext> </blist> <blist> <bibtext> Geiser C, Bishop J, Lockhart G, Shiffman S, Grenard JL. Analyzing latent state-trait and multiple-indicator latent growth curve models as multilevel structural equation models. Frontiers in Psychology. 2013. 10.3389/fpsyg.2013.00975</bibtext> </blist> <blist> <bibtext> Gottfried AE, Fleming JS, Gottfried AW. Continuity of academic intrinsic motivation from childhood through late adolescence: A longitudinal study. Journal of Educational Psychology. 2001; 93: 3-13. 10.1037/0022-0663.93.1.3</bibtext> </blist> <blist> <bibtext> Greene BA. Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist. 2015; 50; 1: 14-30. 10.1080/00461520.2014.989230</bibtext> </blist> <blist> <bibtext> Hardré PL, Xie K, Ly C. Production and data management issues for digital questionnaire administration. Performance Improvement. 2005; 44; 5: 33-39. 10.1002/pfi.4140440509</bibtext> </blist> <blist> <bibtext> Hardré PL, Crowson HM, Xie K. Differential effects of web-based and paper-based administration of questionnaire research instruments in authentic contexts-of-use. Journal of Educational Computing Research. 2010; 42; 1: 103-133. 10.2190/EC.42.1.e</bibtext> </blist> <blist> <bibtext> Hardré PL, Crowson HM, Xie K. Examining contexts-of-use for web-based and paper-based questionnaires. Educational and Psychological Measurement. 2012; 72; 6: 1015-1038. 10.1177/0013164412451977</bibtext> </blist> <blist> <bibtext> Heddy BC, Nelson KG, Husman J, Cheng KC, Goldman JA, Chancey JB. The relationship between perceived instrumentality, interest and transformative experiences in online engineering. Educational Psychology. 2021; 41; 1: 63-78. 10.1080/01443410.2019.1600662</bibtext> </blist> <blist> <bibtext> Hernandez, J, McDuff, D, Infante, C, Maes, P, Quigley, K, & Picard, R. (2016). Wearable ESM: Differences in the experience sampling method across wearable devices. Proceedings of the 18th International conference on human-computer interaction with mobile devices and services. https://doi.org/10.1145/2935334.2935340</bibtext> </blist> <blist> <bibtext> Hidi S, Renninger KA. The four-phase model of interest development. Educational Psychologist. 2006; 41; 2: 111-127. 10.1207/s15326985ep4102_4</bibtext> </blist> <blist> <bibtext> Hochbein C, Dever BV, White G, Mayger L, Gallagher E. Confronting methodological challenges in studying school leader time use through technological advancements: A pilot study. Educational Management Administration & Leadership. 2018; 46; 4: 659-678. 10.1177/1741143217700281</bibtext> </blist> <blist> <bibtext> Hofmann W, Patel PV. SurveySignal: A convenient solution for experience sampling research using participants' own smartphones. Social Science Computer Review. 2015; 33; 2: 235-253. 10.1177/0894439314525117</bibtext> </blist> <blist> <bibtext> Johnson ML, Sinatra GM. Use of task-value instructional inductions for facilitating engagement and conceptual change. Contemporary Educational Psychology. 2013; 38; 1: 51-63. 10.1016/j.cedpsych.2012.09.003</bibtext> </blist> <blist> <bibtext> Lane N, Miluzzo E, Lu H, Peebles D, Choudhury T, Campbell A. A survey of mobile phone sensing. IEEE Communications Magazine. 2010; 48; 9: 140-150. 10.1109/MCOM.2010.5560598</bibtext> </blist> <blist> <bibtext> Larson R, Csikszentmihalyi M. The experience sampling method. New Directions for Methodology of Social & Behavioral Science. 1983; 15: 41-56</bibtext> </blist> <blist> <bibtext> Lavigne GL, Vallerand RJ. The dynamic processes of influence between contextual and situational motivation: A test of the hierarchical model in a science education setting: Motivation toward science. Journal of Applied Social Psychology. 2010; 40; 9: 2343-2359. 10.1111/j.1559-1816.2010.00661.x</bibtext> </blist> <blist> <bibtext> Lawson KM. Women's daily performance, enjoyment, and comfort in male-dominated majors: The role of social interactions in classes. Research in Higher Education. 2021; 62; 4: 478-497. 10.1007/s11162-020-09609-5</bibtext> </blist> <blist> <bibtext> Li Y, Chen K, Su Y, Yue X. Do social regulation strategies predict learning engagement and learning outcomes? A study of English language learners in wiki-supported literature circles activities. Educational Technology Research and Development. 2021; 69: 917-943. 10.1007/s11423-020-09934-7</bibtext> </blist> <blist> <bibtext> Linnenbrink-Garcia L, Rogat TK, Koskey KLK. Affect and engagement during small group instruction. Contemporary Educational Psychology. 2011; 36; 1: 13-24. 10.1016/j.cedpsych.2010.09.001</bibtext> </blist> <blist> <bibtext> Lu G, Xie K, Liu Q. What influences student situational engagement in smart classrooms: Perception of the learning environment and students' motivation. British Journal of Educational Technology. 2022. 10.1111/bjet.13204</bibtext> </blist> <blist> <bibtext> Mäkikangas A, Tolvanen A, Aunola K, Feldt T, Mauno S, Kinnunen U. Multilevel latent profile analysis with covariates: Identifying job characteristics profiles in hierarchical data as an example. Organizational Research Methods. 2018; 21; 4: 931-954. 10.1177/1094428118760690</bibtext> </blist> <blist> <bibtext> Manwaring KC, Larsen R, Graham CR, Henrie CR, Halverson LR. Investigating student engagement in blended learning settings using experience sampling and structural equation modeling. The Internet and Higher Education. 2017; 35: 21-33. 10.1016/j.iheduc.2017.06.002</bibtext> </blist> <blist> <bibtext> Markkanen P, van Berkel N, Visuri A, LeSaint A, Ferreira D, Herneoja A. Exploring work environment usage behaviour through context-aware mobile methods. Blucher Design Proceedings. 2019. 10.5151/proceedings-ecaadesigradi2019_330</bibtext> </blist> <blist> <bibtext> Marks HM. Student engagement in instructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal. 2000; 37; 1: 153-184. 10.3102/00028312037001153</bibtext> </blist> <blist> <bibtext> Martin F, Borup J. Online learner engagement: Conceptual definitions, research themes, and supportive practices. Educational Psychologist. 2022; 57; 3: 162-177. 10.1080/00461520.2022.2089147</bibtext> </blist> <blist> <bibtext> Martin AJ, Mansour M, Malmberg LE. What factors influence students' real-time motivation and engagement? An experience sampling study of high school students using mobile technology. Educational Psychology. 2020; 40; 9: 1113-1135. 10.1080/01443410.2018.1545997</bibtext> </blist> <blist> <bibtext> Martin F, Xie K, Bolliger DU. Engaging learners in the emergency transition to online learning during the COVID-19 pandemic. Journal of Research on Technology in Education. 2022; 54; sup1: S1-S13. 10.1080/15391523.2021.1991703</bibtext> </blist> <blist> <bibtext> Martínez-Sierra G, Arellano-García Y, Hernández-Moreno A, Nava-Guzmán C. Daily emotional experiences of a high school mathematics teacher in the classroom: A qualitative experience-sampling method. International Journal of Science and Mathematics Education. 2019; 17; 3: 591-611. 10.1007/s10763-018-9879-x</bibtext> </blist> <blist> <bibtext> Meredith W. Measurement invariance, factor analysis and factorial invariance. Psychometrika. 1993; 58; 4: 525-543. 10.1007/BF02294825</bibtext> </blist> <blist> <bibtext> Millsap RE. Statistical approaches to measurement invariance. 2012; Routledge. 10.4324/9780203821961</bibtext> </blist> <blist> <bibtext> Moore J, Lovell CD, McGann T, Wyrick J. Why involvement matters: A review of research on student involvement in the collegiate setting. College Student Affairs Journal. 1998; 17; 2: 4-17</bibtext> </blist> <blist> <bibtext> Muenks K, Canning EA, LaCosse J, Green DJ, Zirkel S, Garcia JA, Murphy MC. Does my professor think my ability can change? Students' perceptions of their STEM professors' mindset beliefs predict their psychological vulnerability, engagement, and performance in class. Journal of Experimental Psychology: General. 2020; 149; 11: 2119-2144. 10.1037/xge0000763</bibtext> </blist> <blist> <bibtext> Muthen B, Muthen LK. Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research. 2000; 24; 6: 882-891. 10.1111/j.1530-0277.2000.tb02070.x</bibtext> </blist> <blist> <bibtext> Muthén B. Statistical and substantive checking in growth mixture modeling: Comment on bauer and curran. Psychological Methods. 2003; 8; 3: 369-377. 10.1037/1082-989X.8.3.369</bibtext> </blist> <blist> <bibtext> North, M. (2019). Student motivation in education: An intervention in real-time. Undergraduate Honors Theses. https://digscholarship.unco.edu/honors/28.</bibtext> </blist> <blist> <bibtext> Nylund-Gibson K, Choi AY. Ten frequently asked questions about latent class analysis. Translational Issues in Psychological Science. 2018; 4; 4: 440-461. 10.1037/tps0000176</bibtext> </blist> <blist> <bibtext> Oh, H, Nguyen, J, Soundararajan, S, & Jain, R. (2018). Multimodal food journaling. Proceedings of the 3rd international workshop on multimedia for personal health and health care, pp. 39–47. https://doi.org/10.1145/3264996.3265000.</bibtext> </blist> <blist> <bibtext> Pejovic, V, Musolesi, M. (2014) InterruptMe: Designing intelligent prompting mechanisms for pervasive applications. Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous, & computing. pp. 897–908. https://doi.org/10.1145/2632048.2632062.</bibtext> </blist> <blist> <bibtext> Pejovic V, Lathia N, Mascolo C, Musolesi MTkalčič M, De Carolis B, de Gemmis M, Odić A, Košir A. Mobile-based experience sampling for behaviour research. Emotions and personality in personalized services: Models, evaluation and applications. 2016; Springer International Publishing: 141-161</bibtext> </blist> <blist> <bibtext> Pekrun R, Linnenbrink-Garcia LChristenson SL, Reschly AL, Wylie C. Academic emotions and student engagement. Handbook of research on student engagement. 2012; Springer: 259-282. 10.1007/978-1-4614-2018-7_12</bibtext> </blist> <blist> <bibtext> Pekrun, R, & Marsh, H. W. (2022). Research on situated motivation and emotion: Progress and open problems. Learning and Instruction, 81, 101664.</bibtext> </blist> <blist> <bibtext> Prenoveau JM. Specifying and interpreting latent state–trait models with autoregression: An illustration. Structural Equation Modeling: A Multidisciplinary Journal. 2016; 23; 5: 731-749. 10.1080/10705511.2016.1186550</bibtext> </blist> <blist> <bibtext> Rabe-Hesketh S, Skrondal A. Multilevel and longitudinal modeling using stata, volumes i and ii, third edition. 20123; Stata Press</bibtext> </blist> <blist> <bibtext> Raudenbush SW, Bryk AS. Hierarchical linear models: Applications and data analysis methods. 20022; SAGE Publications, Inc.</bibtext> </blist> <blist> <bibtext> Reeve J, Shin SH. How teachers can support students' agentic engagement. Theory into Practice. 2020; 59; 2: 150-161. 10.1080/00405841.2019.1702451</bibtext> </blist> <blist> <bibtext> Reschly AL, Christenson SLChristenson SL, Reschly AL, Wylie C. Jingle, jangle, and conceptual haziness: Evolution and future directions of the engagement construct. Handbook of research on student engagement. 2012; Springer: 3-19. 10.1007/978-1-4614-2018-7_1</bibtext> </blist> <blist> <bibtext> Rottweiler AL, Taxer JL, Nett UE. Context matters in the effectiveness of emotion regulation strategies. AERA Open. 2018; 4; 2: 233285841877884. 10.1177/2332858418778849</bibtext> </blist> <blist> <bibtext> Savadova S. A living journals approach for the remote study of young children's digital practices in Azerbaijan. Global Studies of Childhood. 2021. 10.1177/20436106211034179</bibtext> </blist> <blist> <bibtext> Schmidt JA, Rosenberg JM, Beymer PN. A person-in‐context approach to student engagement in science: Examining learning activities and choice. Journal of Research in Science Teaching. 2018; 55; 1: 19-43. 10.1002/tea.21409</bibtext> </blist> <blist> <bibtext> Schneider B, Krajcik J, Lavonen J, Salmela-Aro K, Broda M, Spicer J, Viljaranta J. Investigating optimal learning moments in US and Finnish science classes. Journal of Research in Science Teaching. 2016; 53; 3: 400-421. 10.1002/tea.21306</bibtext> </blist> <blist> <bibtext> Schneider B, Krajcik J, Lavonen J, Salmela-Aro K. Learning science: The value of crafting engagement in science environments. 2020; Yale University Press. 10.2307/j.ctvwcjfk1</bibtext> </blist> <blist> <bibtext> Sinatra GM, Heddy BC, Lombardi D. The challenges of defining and measuring student engagement in science. Educational Psychologist. 2015; 50; 1: 1-13. 10.1080/00461520.2014.1002924</bibtext> </blist> <blist> <bibtext> Skinner EA, Belmont MJ. Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology. 1993; 85: 571-581. 10.1037/0022-0663.85.4.571</bibtext> </blist> <blist> <bibtext> Skinner EA, Pitzer JRChristenson SL, Reschly AL, Wylie C. Developmental dynamics of student engagement, coping, and everyday resilience. Handbook of research on student engagement. 2012; Springer: 21-44. 10.1007/978-1-4614-2018-7_2</bibtext> </blist> <blist> <bibtext> Skinner EA, Furrer C, Marchand G, Kinderman T. Engagement and disaffection in the classroom: Part of a larger motivational dynamic?. Journal of Educational Psychology. 2008; 100: 765-781. 10.1037/a0012840</bibtext> </blist> <blist> <bibtext> Skinner EA, Kindermann TA, Connell JP, Wellborn JGWentzel K, Wigfield A. Engagement as an organizational construct in the dynamics of motivational development. Handbook of motivation at school. 2009; Erlbaum: 223-245</bibtext> </blist> <blist> <bibtext> Sugie NF. Utilizing smartphones to study disadvantaged and hard-to-reach groups. Sociological Methods & Research. 2018; 47; 3: 458-491. 10.1177/0049124115626176</bibtext> </blist> <blist> <bibtext> van Berkel N, Ferreira D, Kostakos V. The experience sampling method on mobile devices. ACM Computing Surveys. 2018; 50; 6: 1-40. 10.1145/3123988</bibtext> </blist> <blist> <bibtext> van Berkel N, Goncalves J, Lovén L, Ferreira D, Hosio S, Kostakos V. Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. International Journal of Human-Computer Studies. 2019; 125: 118-128. 10.1016/j.ijhcs.2018.12.002</bibtext> </blist> <blist> <bibtext> Vongkulluksn VW, Xie K. Multilevel latent state-trait models with experience sampling data: An illustrative case of examining situational engagement. Open Education Studies. 2022; 4; 1: 252-272. 10.1515/edu-2022-0016</bibtext> </blist> <blist> <bibtext> Vongkulluksn VW, Lu L, Nelson MJ, Xie K. Cognitive engagement with technology scale: A validation study. Educational Technology Research and Development. 2022; 70; 2: 419-445. 10.1007/s11423-022-10098-9</bibtext> </blist> <blist> <bibtext> Walls TA, Schafer JL. Models for intensive longitudinal data. 2006; Oxford University Press. 10.1093/acprof:oso/9780195173444.001.0001</bibtext> </blist> <blist> <bibtext> Wang MT, Eccles JS. School context, achievement motivation, and academic engagement: A longitudinal study of school engagement using a multidimensional perspective. Learning and Instruction. 2013; 28: 12-23. 10.1016/j.learninstruc.2013.04.002</bibtext> </blist> <blist> <bibtext> Wang, R, Chen, F, Chen, Z, Li, T, Harari, G, Tignor, S, Zhou, X, Ben-Zeev, D, Campbell, A. T. StudentLife: Assessing mental health, academic performance and behavioral trends of college students using smartphones. Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous, & computing (2014). pp. 3–14. https://doi.org/10.1145/2632048.2632054.</bibtext> </blist> <blist> <bibtext> Wang J, Xie K, Liu Q, Long T, Lu G. Examining the effect of seat location on students' real-time social interactions in a smart classroom using experience sampling method. Journal of Computers in Education. 2022. 10.1007/s40692-022-00229-9</bibtext> </blist> <blist> <bibtext> Xie K. Projecting learner engagement in remote contexts using empathic design. Educational Technology Research and Development. 2021; 69; 1: 81-85. 10.1007/s11423-020-09898-8</bibtext> </blist> <blist> <bibtext> Xie K, Debacker TK, Ferguson C. Extending the traditional classroom through online discussion: The role of student motivation. Journal of Educational Computing Research. 2006; 34; 1: 67-89. 10.2190/7BAK-EGAH-3MH1-K7C6</bibtext> </blist> <blist> <bibtext> Xie K, Heddy BC, Greene BA. Affordances of using mobile technology to support experience-sampling method in examining college students' engagement. Computers & Education. 2019; 128: 183-198. 10.1016/j.compedu.2018.09.020</bibtext> </blist> <blist> <bibtext> Xie K, Heddy BC, Vongkulluksn VW. Examining engagement in context using experience-sampling method with mobile technology. Contemporary Educational Psychology. 2019; 59: 101788. 10.1016/j.cedpsych.2019.101788</bibtext> </blist> <blist> <bibtext> Xie K, Vongkulluksn V, Lu L, Cheng SL. A person-centered approach to examining high-school students' motivation, engagement and academic performance. Contemporary Educational Psychology. 2020; 62: 1-13. 10.1016/j.cedpsych.2020.101877</bibtext> </blist> <blist> <bibtext> Xie K, Vongkulluksn VW, Cheng SL, Jiang Z. Examining high-school students' motivation change through a person-centered approach. Journal of Educational Psychology. 2022; 114; 1: 89-107. 10.1037/edu0000507</bibtext> </blist> <blist> <bibtext> Zhang, X, Pina, L. R, & Fogarty, J. (2016). Examining unlock journaling with diaries and reminders for in situ self-report in health and wellness. Proceedings of the 2016 CHI conference on human factors in computing systems, pp. 5658–5664. https://doi.org/10.1145/2858036.2858360.</bibtext> </blist> <blist> <bibtext> Zirkel S, Garcia JA, Murphy MC. Experience-sampling research methods and their potential for education research. Educational Researcher. 2015; 44; 1: 7-16. 10.3102/0013189X14566879</bibtext> </blist> </ref> <aug> <p>By Kui Xie; Vanessa W. Vongkulluksn; Benjamin C. Heddy and Zilu Jiang</p> <p>Reported by Author; Author; Author; Author</p> <p></p> <p>Kui Xie is Red Cedar Professor and Chair of Department of Counseling, Educational Psychology and Special Education in the College of Education at Michigan State University. His research investigates areas related to motivation and engagement in digital learning, K-12 technology integration and teacher professional development, technology intervention and learning environment, learning analytics and research methods.</p> <p>Vanessa W. Vongkulluksn is an Assistant Professor in the Department of Educational Psychology, Leadership, and Higher Education at University of Nevada, Las Vegas (UNLV). She is a faculty associate of the UNLV Office of Learning Analytics. Her research utilizes a variety of statistical methods to examine factors associated with student learning and motivation in technology-integrated contexts. Her work identifies actionable changes in learning environment designs that match the needs and affordances of modern classrooms.</p> <p>Benjamin Heddy is an Associate Professor of Educational Psychology at the University of Oklahoma. He is the director of the Motivation, Out-of-school, Value, and Engagement (MOVE) research group. Dr. Heddy investigates engagement and motivation, especially in out-of-school contexts. Additionally, he explores the mechanisms of motivated change including knowledge revision, attitude and emotion change, and perceptual change.</p> <p>Zilu Jiang is a Ph.D. Candidate at Learning Technologies in Department of Educational Studies, College of Education and Human Ecology at The Ohio State University. Currently, she works as a Research Assistant at the Research Laboratory for Digital Learning. Her academic interest includes learner's motivation and engagement in the digital environment, technology integration in teaching and learning, and technology-enhanced language learning.</p> </aug> <nolink nlid="nl1" bibid="bib17" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib33" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib53" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib69" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib12" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib18" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib42" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib28" firstref="ref8"></nolink> <nolink nlid="nl9" bibid="bib29" firstref="ref9"></nolink> <nolink nlid="nl10" bibid="bib78" firstref="ref10"></nolink> <nolink nlid="nl11" bibid="bib81" firstref="ref11"></nolink> <nolink nlid="nl12" bibid="bib98" firstref="ref12"></nolink> <nolink nlid="nl13" bibid="bib102" firstref="ref13"></nolink> <nolink nlid="nl14" bibid="bib44" firstref="ref14"></nolink> <nolink nlid="nl15" bibid="bib60" firstref="ref15"></nolink> <nolink nlid="nl16" bibid="bib92" firstref="ref16"></nolink> <nolink nlid="nl17" bibid="bib54" firstref="ref17"></nolink> <nolink nlid="nl18" bibid="bib96" firstref="ref18"></nolink> <nolink nlid="nl19" bibid="bib26" firstref="ref19"></nolink> <nolink nlid="nl20" bibid="bib85" firstref="ref20"></nolink> <nolink nlid="nl21" bibid="bib83" firstref="ref21"></nolink> <nolink nlid="nl22" bibid="bib75" firstref="ref22"></nolink> <nolink nlid="nl23" bibid="bib27" firstref="ref23"></nolink> <nolink nlid="nl24" bibid="bib84" firstref="ref24"></nolink> <nolink nlid="nl25" bibid="bib49" firstref="ref28"></nolink> <nolink nlid="nl26" bibid="bib94" firstref="ref29"></nolink> <nolink nlid="nl27" bibid="bib95" firstref="ref30"></nolink> <nolink nlid="nl28" bibid="bib39" firstref="ref31"></nolink> <nolink nlid="nl29" bibid="bib97" firstref="ref32"></nolink> <nolink nlid="nl30" bibid="bib32" firstref="ref34"></nolink> <nolink nlid="nl31" bibid="bib82" firstref="ref35"></nolink> <nolink nlid="nl32" bibid="bib56" firstref="ref39"></nolink> <nolink nlid="nl33" bibid="bib37" firstref="ref43"></nolink> <nolink nlid="nl34" bibid="bib74" firstref="ref44"></nolink> <nolink nlid="nl35" bibid="bib48" firstref="ref45"></nolink> <nolink nlid="nl36" bibid="bib99" firstref="ref46"></nolink> <nolink nlid="nl37" bibid="bib87" firstref="ref53"></nolink> <nolink nlid="nl38" bibid="bib21" firstref="ref54"></nolink> <nolink nlid="nl39" bibid="bib79" firstref="ref56"></nolink> <nolink nlid="nl40" bibid="bib80" firstref="ref57"></nolink> <nolink nlid="nl41" bibid="bib55" firstref="ref60"></nolink> <nolink nlid="nl42" bibid="bib51" firstref="ref62"></nolink> <nolink nlid="nl43" bibid="bib91" firstref="ref63"></nolink> <nolink nlid="nl44" bibid="bib86" firstref="ref68"></nolink> <nolink nlid="nl45" bibid="bib90" firstref="ref70"></nolink> <nolink nlid="nl46" bibid="bib66" firstref="ref71"></nolink> <nolink nlid="nl47" bibid="bib19" firstref="ref72"></nolink> <nolink nlid="nl48" bibid="bib93" firstref="ref73"></nolink> <nolink nlid="nl49" bibid="bib20" firstref="ref74"></nolink> <nolink nlid="nl50" bibid="bib13" firstref="ref82"></nolink> <nolink nlid="nl51" bibid="bib24" firstref="ref83"></nolink> <nolink nlid="nl52" bibid="bib30" firstref="ref84"></nolink> <nolink nlid="nl53" bibid="bib88" firstref="ref85"></nolink> <nolink nlid="nl54" bibid="bib11" firstref="ref86"></nolink> <nolink nlid="nl55" bibid="bib46" firstref="ref91"></nolink> <nolink nlid="nl56" bibid="bib57" firstref="ref93"></nolink> <nolink nlid="nl57" bibid="bib77" firstref="ref94"></nolink> <nolink nlid="nl58" bibid="bib76" firstref="ref96"></nolink> <nolink nlid="nl59" bibid="bib64" firstref="ref97"></nolink> <nolink nlid="nl60" bibid="bib14" firstref="ref98"></nolink> <nolink nlid="nl61" bibid="bib101" firstref="ref99"></nolink> <nolink nlid="nl62" bibid="bib61" firstref="ref101"></nolink> <nolink nlid="nl63" bibid="bib15" firstref="ref102"></nolink> <nolink nlid="nl64" bibid="bib38" firstref="ref105"></nolink> <nolink nlid="nl65" bibid="bib40" firstref="ref106"></nolink> <nolink nlid="nl66" bibid="bib25" firstref="ref108"></nolink> <nolink nlid="nl67" bibid="bib68" firstref="ref109"></nolink> <nolink nlid="nl68" bibid="bib67" firstref="ref114"></nolink> <nolink nlid="nl69" bibid="bib52" firstref="ref116"></nolink> <nolink nlid="nl70" bibid="bib34" firstref="ref118"></nolink> <nolink nlid="nl71" bibid="bib41" firstref="ref119"></nolink> <nolink nlid="nl72" bibid="bib35" firstref="ref120"></nolink> <nolink nlid="nl73" bibid="bib36" firstref="ref121"></nolink> <nolink nlid="nl74" bibid="bib43" firstref="ref124"></nolink> <nolink nlid="nl75" bibid="bib89" firstref="ref125"></nolink> <nolink nlid="nl76" bibid="bib72" firstref="ref126"></nolink> <nolink nlid="nl77" bibid="bib73" firstref="ref127"></nolink> <nolink nlid="nl78" bibid="bib63" firstref="ref128"></nolink> <nolink nlid="nl79" bibid="bib65" firstref="ref129"></nolink> <nolink nlid="nl80" bibid="bib45" firstref="ref131"></nolink> <nolink nlid="nl81" bibid="bib50" firstref="ref135"></nolink> <nolink nlid="nl82" bibid="bib22" firstref="ref141"></nolink> <nolink nlid="nl83" bibid="bib31" firstref="ref144"></nolink> <nolink nlid="nl84" bibid="bib100" firstref="ref145"></nolink> <nolink nlid="nl85" bibid="bib62" firstref="ref147"></nolink> <nolink nlid="nl86" bibid="bib23" firstref="ref149"></nolink> <nolink nlid="nl87" bibid="bib71" firstref="ref153"></nolink> <nolink nlid="nl88" bibid="bib58" firstref="ref157"></nolink> <nolink nlid="nl89" bibid="bib59" firstref="ref158"></nolink> <nolink nlid="nl90" bibid="bib10" firstref="ref174"></nolink> <nolink nlid="nl91" bibid="bib47" firstref="ref175"></nolink> <nolink nlid="nl92" bibid="bib16" firstref="ref176"></nolink> <nolink nlid="nl93" bibid="bib70" firstref="ref178"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1448032
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Experience Sampling Methodology and Technology: An Approach for Examining Situational, Longitudinal, and Multi-Dimensional Characteristics of Engagement
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Kui+Xie%22">Kui Xie</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0002-7173-4859">0000-0002-7173-4859</externalLink>)<br /><searchLink fieldCode="AR" term="%22Vanessa+W%2E+Vongkulluksn%22">Vanessa W. Vongkulluksn</searchLink><br /><searchLink fieldCode="AR" term="%22Benjamin+C%2E+Heddy%22">Benjamin C. Heddy</searchLink><br /><searchLink fieldCode="AR" term="%22Zilu+Jiang%22">Zilu Jiang</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Technology+Research+and+Development%22"><i>Educational Technology Research and Development</i></searchLink>. 2024 72(5):2585-2615.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 31
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Information Analyses
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Learner+Engagement%22">Learner Engagement</searchLink><br /><searchLink fieldCode="DE" term="%22Environment%22">Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Research%22">Educational Research</searchLink><br /><searchLink fieldCode="DE" term="%22Information+Technology%22">Information Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Context+Effect%22">Context Effect</searchLink><br /><searchLink fieldCode="DE" term="%22Longitudinal+Studies%22">Longitudinal Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Multivariate+Analysis%22">Multivariate Analysis</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11423-023-10259-4
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1042-1629<br />1556-6501
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Engagement has been recognized as one of the most important factors of learning and achievement in academic settings. Research on engagement has been gearing toward a "person-in-context" orientation, where both personal characteristics and contextual features in relation to students' engagement are considered. This orientation allows a more in-depth understanding of how a person embedded within a context engages in a task, and it pays particular attention to the interactions between the person and contextual features. Engagement in context is situational, longitudinal, and multi-dimensional. This in-situ orientation requires a research methodology that is embedded in and responsive to the context where learning occurs. In this paper, we provide a conceptual synthesis of research on academic engagement in proposing a framework of engagement in context. We introduce the affordances of Experience Sampling Methodology (ESM) and provide a review of current technologies in supporting ESM. In addition, we provide example cases of examining engagement using ESM and technology. In these cases, we discuss details about how ESM combines with technologies and statistical approaches in providing insights to educational research, theory, and practice.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1448032
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1448032
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11423-023-10259-4
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 31
        StartPage: 2585
    Subjects:
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Environment
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Research Methodology
        Type: general
      – SubjectFull: Educational Research
        Type: general
      – SubjectFull: Information Technology
        Type: general
      – SubjectFull: Context Effect
        Type: general
      – SubjectFull: Longitudinal Studies
        Type: general
      – SubjectFull: Multivariate Analysis
        Type: general
    Titles:
      – TitleFull: Experience Sampling Methodology and Technology: An Approach for Examining Situational, Longitudinal, and Multi-Dimensional Characteristics of Engagement
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kui Xie
      – PersonEntity:
          Name:
            NameFull: Vanessa W. Vongkulluksn
      – PersonEntity:
          Name:
            NameFull: Benjamin C. Heddy
      – PersonEntity:
          Name:
            NameFull: Zilu Jiang
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1042-1629
            – Type: issn-electronic
              Value: 1556-6501
          Numbering:
            – Type: volume
              Value: 72
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Educational Technology Research and Development
              Type: main
ResultId 1