View in EDS HTML Full Text PDF Full Text

Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection

Saved in:
Bibliographic Details
Title: Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection
Language: English
Authors: Joseph T. Coyne (ORCID 0000-0001-6461-0148), Laura Jamison, Kaylin Strong, Ciara Sibley, Cyrus Foroughi, Sarah Melick
Source: Cognitive Research: Principles and Implications. 2025 10.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 16
Publication Date: 2025
Sponsoring Agency: Office of the Secretary of Defense (OSD) (DOD)
Office of Naval Research (ONR) (DOD)
Document Type: Journal Articles
Reports - Research
Descriptors: High Stakes Tests, Construct Validity, Military Training, Aviation Education, Spatial Ability, Attention, Air Transportation, Armed Forces, Personnel Selection
DOI: 10.1186/s41235-025-00660-3
ISSN: 2365-7464
Abstract: This paper looks at how process-based spatial ability and attention measures taken within a high-stakes battery used to select pilots in the US Navy compare to lab-based measures of the same constructs. Process-based measures typically function by having individuals perform either a novel task or perform a task with novel stimuli. However, applicants often spend time practicing the tasks prior to taking the battery. A group of 307 Naval Flight Students participated in the study, in which they took several spatial ability, attention and general processing measures. One of the spatial tasks used in the study was the same as the spatial task in the Navy's pilot selection battery, which all of the participants had taken. All of the lab spatial ability measures including the one used in the selection battery were highly correlated and loaded onto the same spatial ability factor. However, the high-stakes spatial subtest was not correlated with any of the lab spatial measures including the same test administered in the lab. The lab spatial ability data was also correlated with training outcomes whereas the high-stakes process spatial and attention measures were not. The high-stakes attention measure was weakly correlated with some of the general processing measures. The pattern of results suggest that familiarity with the spatial and attention tasks in the high-stakes environment may be negating those tests ability to measure the constructs they were designed to measure, and also reducing their effectiveness to predict training performance.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1480870
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGrTv_-gxXb1Z6ajlvmR1hlAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDOvVuvvdU1-438ADigIBEICBm4pAso_JmDR4wye_arVVDSmw_FYgxksCWe1HvcGUnfeFdZM5PXAl_xZr2Y27T4aaQ0aHTVM3DsUH_v1wmERMBCTTQyreaTxTk_AU0AlYs27UhDn1zD5fik8eKbUmMuI3ApR-GQIzriLyO-iALvyzUwlz7dHTlVOkDO4Y-miIHk00qMzyXwyYZ-EavCkGAzmJgj7x8s1n78ayU32j
Text:
  Availability: 1
  Value: <anid>AN0187435214;[k1e6]20aug.25;2025Aug22.05:59;v2.2.500</anid> <title id="AN0187435214-1">Process-based measures in high-stakes testing: practical implications for construct validity within military aviation selection </title> <p>This paper looks at how process-based spatial ability and attention measures taken within a high-stakes battery used to select pilots in the US Navy compare to lab-based measures of the same constructs. Process-based measures typically function by having individuals perform either a novel task or perform a task with novel stimuli. However, applicants often spend time practicing the tasks prior to taking the battery. A group of 307 Naval Flight Students participated in the study, in which they took several spatial ability, attention and general processing measures. One of the spatial tasks used in the study was the same as the spatial task in the Navy's pilot selection battery, which all of the participants had taken. All of the lab spatial ability measures including the one used in the selection battery were highly correlated and loaded onto the same spatial ability factor. However, the high-stakes spatial subtest was not correlated with any of the lab spatial measures including the same test administered in the lab. The lab spatial ability data was also correlated with training outcomes whereas the high-stakes process spatial and attention measures were not. The high-stakes attention measure was weakly correlated with some of the general processing measures. The pattern of results suggest that familiarity with the spatial and attention tasks in the high-stakes environment may be negating those tests ability to measure the constructs they were designed to measure, and also reducing their effectiveness to predict training performance. Statement of significance: This paper addresses an increasingly difficult challenge the Navy is facing within aviation selection, in that applicants are highly motivated and have access to unofficial replicas of the Navy's test battery. The challenge is specific to the process-based measures such as spatial ability and attention that rely on some degree of novelty to work. When applicants practice these types of tests they can practice to the test, memorize items, and learn strategies which impact the test's ability to measure the cognitive construct it was designed to measure as well as reduces its ability to predict flight training outcomes. This is particularly problematic as the unofficial test preparation software can replicate a new test within days. While the data presented here are limited to spatial ability and attention within military pilot selection it applies to a much broader community of researchers. Anyone developing a high-stakes test with a large and motivated applicant pool may also see their process-based measures perform differently in a high-stakes environment than a low stakes laboratory one in which participants are naïve to the tasks they are taking. The extent to which practice can alter the effectiveness of high-stakes test performance is an important one. The results of the paper suggest that test developers should assume participants are practiced and assess the extent to which practice on process-based measure impacts the tasks ability to measure the construct of interest and predict performance.</p> <p>Keywords: Psychology and Cognitive Sciences Psychology</p> <hd id="AN0187435214-2">Introduction</hd> <p>Each year approximately 6000 applicants take the Aviation Selection Test Battery (ASTB) to determine whether they have the aptitude to become a pilot or flight officer in the US Navy, Marine Corps and Coast Guard. One of the unique challenges faced in this selection procedure is that selection decisions are not based upon flight experience, but rather an individual's aptitude to learn to fly. Those applicants are competing to be one of about 1100 pilots who the Navy trains each year (Prasad-Rao et al., [<reflink idref="bib34" id="ref1">34</reflink>]). The majority of applicants who take the ASTB do not achieve qualifying scores. Given the consequences of the battery, the ASTB is a high-stakes test particularly for those individuals who have a strong desire to pursue an aviation career within the military.</p> <p>Beyond the individual's consequences for performing on the battery, the effectiveness of the test is critical to the Navy. The U.S. Government Accountability Office (GAO, [<reflink idref="bib21" id="ref2">21</reflink>]) estimated it cost the military between $3-$11 million and approximately 5 years to train a single fighter pilot. The combined attrition rate for all phases of flight training is between 15 and 20% for Naval Aviators (NAMI, [<reflink idref="bib31" id="ref3">31</reflink>]). The cost of attrition varies by platform (e.g., helicopter, tilt–rotor, and strike) and the stage of training (Prasad-Rao et al., [<reflink idref="bib34" id="ref4">34</reflink>]). By the time a pilot reaches primary flight training the cost is $250 k per individual. Those costs quickly escalate as training moves to more expensive airframes with increased fuel and maintenance costs. It is estimated that without the ASTB the rate of attrition would be double (Arnold, [<reflink idref="bib2" id="ref5">2</reflink>]). Therefore, ensuring the battery retains and improves its predictive validity is critical to the Navy.</p> <p>Like most aptitude assessments the ASTB is a combination of fluid and crystalized intelligence, or process and content knowledge, respectively. The content measures include: math, reading, mechanical, aviation and nautical knowledge. The process measures are all part of the Performance Based Measures portion of the test which contains ability tests (e.g., attention, psycho-motor, and spatial ability). The performance tests were developed to better reflect the "dynamic environment inside the cockpit" which traditional pen and paper tests can miss (Walker et al., [<reflink idref="bib40" id="ref6">40</reflink>]). This paper focuses on the measurement of process related abilities, specifically spatial ability and attention as they are measured within the ASTB. These constructs are also used by the US Air Force in their pilot selection battery the Test of Basic Aviation Skills (TBAS). The overall batteries are similar and the test of spatial ability is the same. Beyond pilot selection, the US military in general is interested in measurement of these constructs for determining eligibility for enlisted jobs. Attention and spatial ability were two of the three new predictor constructs that the National Research Council recommended to the US Army for consideration in enlisted selection and classification (NRC, [<reflink idref="bib32" id="ref7">32</reflink>]). These constructs are seen to have predictive validity, do not require special equipment, and can add incremental validity to selection batteries which are largely comprised of measures of crystalized intelligence (NRC, [<reflink idref="bib32" id="ref8">32</reflink>]).</p> <p>Ackerman ([<reflink idref="bib1" id="ref9">1</reflink>]) discusses two related problems associated with process measures in intelligence testing, familiarity and learnability. Process related measures involve either a novel process, novel stimuli, or both. Ackerman argues that an individual's level of familiarity with the stimuli can impact their performance on the test, and therefore process measures can be confounded when individuals vary in their familiarity. The second problem he discusses is learnability, the degree to which performance will improve with repeated exposure to the task. Ackerman points to the reduction in correlations between initial exposure and final exposure as the number of exposures increases over time.</p> <p>The issues raised by Ackerman ([<reflink idref="bib1" id="ref10">1</reflink>]) are amplified in high-stakes environments such as the ASTB. There are a number of unofficial resources available to ASTB applicants which recreate subtests within the battery for applicants to practice. This paper includes the first large scale assessment of a new spatial ability measure, the Terrain Orientation Task, which we will refer to as the terrain task throughout this paper. The terrain task was added to the ASTB in September 2023, and within three weeks a practice version of the task was available in an unofficial test prep app. The stimuli in the practice version are different from the ASTB; however, the unofficial version allows applicants to practice the process. An assumption should be made that motivated and informed test takers will have direct exposure to portions of the ASTB prior to having ever taken the exam. A challenge for administrators of high-stakes tests is that there is currently no way of measuring an individual's familiarity with the test. This familiarity potentially reduces the battery's effectiveness by adding unknown variance. What is more problematic is that there are strategies to some process tests within the ASTB which may negate or reduce its ability to measure the construct it was designed to measure.</p> <p>A review on spatial ability or even the use of spatial ability within aviation selection are outside the scope of this paper. Lohman's ([<reflink idref="bib27" id="ref11">27</reflink>], p. 1) definition of spatial ability "the ability to generate, retain, retrieve, and transform well-structured visual images whose properties include location, size, distance, direction, separation, connection, shape, pattern, and movement" is a widely accepted one. Spatial ability is a multifaceted-construct, one in which there is some disagreement not only on what the components are but also which are the most important in careers such as aviation (e.g., Baron & Rose, [<reflink idref="bib3" id="ref12">3</reflink>]). Measures of spatial ability have been included in military aviation selection since World War I (Damos, [<reflink idref="bib16" id="ref13">16</reflink>]). This paper focuses on how spatial ability is currently measured, and how high-stakes testing influences the effectiveness of those measures.</p> <p>The Direction Orientation Task (Fig. 3) is a spatial ability test designed by the US Air Force (Carretta, [<reflink idref="bib9" id="ref14">9</reflink>]), and is currently still a part of their pilot selection battery. Baron and Rose ([<reflink idref="bib3" id="ref15">3</reflink>]) found that the Direction Orientation Task significantly predicted pilot training outcomes in the Air Force. The test was adopted by the Navy, and included in the ASTB in 2013. The Direction Orientation Task was initially predictive of training outcomes for the Navy, however, more recent analysis demonstrated that the test had lost its incremental validity over time (Coyne et al., [<reflink idref="bib14" id="ref16">14</reflink>]). Prior to its inclusion in the ASTB, Momen ([<reflink idref="bib30" id="ref17">30</reflink>]) looked at retest performance of the Direction Orientation Task and found a significant improvement in performance across the two attempts on consecutive days. However, the assumption was made that practice effects would decrease over time as applicants would not be allowed immediately retake the ASTB. The Navy and Air Force both limit re-testing such that a minimum number of weeks must pass between test attempts. This testing policy is meant to limit the practice effects from exposure to the official test. However, this policy does not account for the fact that applicants have ways to unofficially practice between test administrations.</p> <p>Each Direction Orientation Task trial has two images, one which shows the heading of an aerial vehicle (12 possible orientations based upon 30 degree heading increments) and the other image corresponds to a camera from that vehicle depicting four parking lots surrounding a building. The four parking lots are located at the four cardinal directions (North, East, South and West). The trial asks the individual to select one of the four parking lots (e.g., West). The position of that parking lot within the image depends on the heading of the aircraft. When the vehicles heading is one of the cardinal directions the parking lot on the top of the second image is the same as the heading. One of the limitations of the Direction Orientation Task is there are only 48 test items, individuals see every item, and the items are easy to replicate for practice purposes. Attempts to develop a more difficult version of the task, revealed additional concerns regarding non-spatial strategies (Keiser et al., [<reflink idref="bib24" id="ref18">24</reflink>], Coyne et al., [<reflink idref="bib13" id="ref19">13</reflink>]). When surveyed on strategies used after completing the newer more difficult versions of the Direction Orientation Task, it was revealed that there was a negative correlation with the degree to which participants reported using a spatial strategy and performance, and a strong positive correlation between the use of math strategies and performance. These modified Direction Orientation Tasks were never included in the ASTB; however, these findings are problematic as they demonstrate a challenge with spatial ability tests which are amplified in a high-stakes testing. The concern is that effective non-spatial strategies are shared in online forums, and then adopted by a large subset of the population. A simple mathematical solution to the task is to compute the difference between the heading and the target parking lot, in Fig. 3 the heading 120 degrees is 60 degrees away from the target parking lot (south/180) and the response parking lot's position is at 60 degrees. Another non-spatial solution is to draw a compass and then rotate it to match the heading of the vehicle. The Direction Orientation Task likely added incremental validity initially as it measured spatial ability, however as applicants became familiar with the items and aware of non-spatial strategies the test lost its effectiveness.</p> <p>The Terrain Orientation Task (Fig. 1) was developed as a potential replacement for the Direction Orientation Task. The Terrain Orientation Task addresses a number of limitations of the Direction Orientation Task. The Terrain task has an unlimited number of items a single map can serve as 12 items and there is an infinite pool of potential maps. More importantly the task does not have a mathematical solution similar to the Direction Orientation Task. The task does not provide any numerical values (no heading and no target position). Rather the problem most be solved by comparing differences in orientation between the two images. The Terrain Orientation Task is a terrain association task which provides a reference map and a "camera view" of the same terrain in a different orientation. An initial evaluation of the task found that it was significantly correlated with initial training performance in a group of Naval Flight Students (Coyne et al., [<reflink idref="bib15" id="ref20">15</reflink>]). The Terrain Task was designed to capture the same spatial ability variance as the Direction Orientation Task, however in the initial evaluation performance on Terrain Orientation Task (lab) and scores on the Direction Orientation Task taken within the ASTB (high-stakes) were not correlated. The present study addresses whether these two process tests measure the same construct.</p> <p>Graph: Fig. 1 Sample trial for the Terrain Orientation Task (correct answer is 30 degrees)</p> <p>The first goal of this work was to evaluate whether a new measure of spatial ability, the Terrain Orientation Task, could predict training performance for Naval Flight Students. Previous research in a small sample had demonstrated that Terrain Task could predict grades in ground school. However, that sample was small (<emph>N</emph> = 114) and the study did not compare the Terrain Orientation Task to other spatial ability measures. The current study is comprised of a larger sample of flight students and includes several measures of spatial ability. All of the flight students would have taken the ASTB, and would thus have familiarity with the Direction Orientation Task. This study took place prior to the Terrain Orientation Task being included in the ASTB so this was the students first exposure to the new test. It is hypothesized that the Terrain Task, and other spatial ability measures would predict training outcomes for the flight students. However, given the participants familiarity with Direction Orientation Task, it was hypothesized that the Direction Orientation Task in the study would not predict training outcomes or significantly correlate with the other spatial ability measures.</p> <p>Spatial ability is only one of several process-based constructs measured in the ASTB.</p> <p>A second objective of the study was to assess whether new measures of attention control could predict training outcomes. The ASTB includes a Dichotic Listening Task which is intended to measure selective attention. The task has individuals attend to information in one ear (e.g., responding to numbers and ignoring letters) while ignoring information simultaneously presented in the other year. The ear to which individuals must attend switches throughout the task. This type of task was first developed by Gopher and Kahneman ([<reflink idref="bib20" id="ref21">20</reflink>]) and was used in selection of Israeli pilots. Similar to the DOT, the authors are aware of strategies on the Dichotic Listening Task which could eliminate or reduce the attentional demands of the task.</p> <p>Attention control is "a domain general cognitive ability that plays a role in most controlled mental operations" (Burgoyne et al., [<reflink idref="bib6" id="ref22">6</reflink>]). Research in individual differences in attention control has received increased interest since a pair of recent papers identified measurement issues related to response time difference scores which limited the reliability of traditional measures (Hedge et al., [<reflink idref="bib22" id="ref23">22</reflink>], Draheim et al., [<reflink idref="bib17" id="ref24">17</reflink>]). These newer measures focus on accuracy and have been shown to reliably measure the construct (Draheim et al., [<reflink idref="bib18" id="ref25">18</reflink>]). Burgoyne et al. ([<reflink idref="bib7" id="ref26">7</reflink>]) described attention control as a "bottleneck that constrains performance across cognitive domains." Failures to control one's attention can impede an individual's ability to learn new information, and can hinder performance within complex tasks such as flying.</p> <p>Recently, several of these measures of attention control have been shown to predict training performance in Navy Air Traffic Controllers (Burgoyne et al., [<reflink idref="bib7" id="ref27">7</reflink>]; Coyne et al., [<reflink idref="bib15" id="ref28">15</reflink>]). Initial research on attention control with Navy flight students collected prior to the COVID-19 pandemic found that several longer measures of attention control predicted flight training outcomes (Burgoyne et al., [<reflink idref="bib7" id="ref29">7</reflink>]). However, these tests which took over 30 min of time and were considered too long to be included in the Navy's selection battery. Researchers at Georgia Institute of Technology focused on the development of three shorter measures of attention control, specifically double conflict versions of the Stroop, Flanker, and Simon tasks, in which conflict could be present in both the stimulus and the response (Burgoyne et al., [<reflink idref="bib6" id="ref30">6</reflink>]). An evaluation of these three new tests revealed that the Stroop and Flanker tests were predictive of flight training outcomes in a group of Navy flight students; however, the Simon test was not (Coyne et al., [<reflink idref="bib15" id="ref31">15</reflink>]). This study continues to examine the Stroop and Flanker tasks for potential use within the ASTB. The second hypothesis is that these two new measures of attention control will correlate with other domain general cognitive processing (e.g., working memory and fluid intelligence tasks) and predict training outcomes. It is expected that the familiarity with the high-stakes Dichotic Listening Task will reduce its capability to capture the same domain general variance that the new attention tasks capture.</p> <hd id="AN0187435214-3">Method</hd> <p></p> <hd id="AN0187435214-4">Sample</hd> <p>Data were collected from 307 NFS between 2021 and 2022. The mean age was 23.5 (SD = 2.4) and 58 of the students were female. The students were all Naval Officers, the majority of whom were training to be Naval Aviators, the remaining students were training to be Naval Flight Officers. All of the students had taken the ASTB and are part of a down selected sample. The students all participated in the study prior to beginning their flight training. All of the students were waiting to begin the Naval Introductory Flight Evaluation (NIFE) which is the Navy's introductory training and has a ground school component as well as smaller flight component. Aviators and Flight Officers take the introductory training together following the same syllabus. All participants consented to the research study, and consented to have their lab data linked to both their ASTB data and training outcome data. Participants were all told their performance in the lab would not be used in any decisions with respect to their own careers, but would be used to inform the design of future versions of the ASTB. The study was approved by the primary author's institutional review board.</p> <p>It is worth noting that this sample was representative of the population of interest, highly motivated Naval Officers training to become Naval Aviators or Naval Flight Officers. In some studies on military personnel, corrections for range restriction are applied because the full population is not adequately represented and reported correlations may be underestimated since only selected personnel are evaluated (Carretta & Ree, [<reflink idref="bib10" id="ref32">10</reflink>]). However, for the current study, we are specifically interested in the latent structure of these measures in a highly motivated population where participants are likely to have practiced and prepared for the test. Our aim is to estimate the predictive capacity of these measures only for individuals who are already flight students. Therefore, we do not apply any range restriction corrections using other populations (e.g., all ASTB test takers) since the raw sample correlations represent an accurate cross section of our population of interest.</p> <hd id="AN0187435214-5">Procedure</hd> <p>The test protocol was approximately 90 min. Participants came to the lab in groups of 10 and completed the tasks in the same order. The tasks were programmed in C#, XOJO, and E-Prime. A wrapper was programmed in C# that forced the participants to start each task in order allowing them to move through the tasks at their own pace. Participants completed a brief demographic survey followed by a series of spatial ability tasks, the terrain task, Spatial Apperception Task, the Direction Orientation Task, Cube Comparison and Paper Folding. After the spatial tasks participants completed the Mental Counters Task, Stroop and Flanker Double conflict tasks, and the Raven's Advanced Progressive Matrices.</p> <hd id="AN0187435214-6">Terrain orientation task/terrain task</hd> <p>The Terrain Orientation Task (Fig. 1) is designed to measure terrain association that is the ability to identify landmarks and use their position relative to other objects to determine orientation. Individuals are told that the reference map on the left is always oriented with North at the top. They are told that the camera map is an aerial view of the same area but from a vehicle traveling on a different heading. The goal is to determine the vehicles direction based upon changes in landmark/feature positions across the two images. For example in the sample trial in Fig. 1 there is a single East/West spanning bridge that crosses the river. The image on is now at a 30 degree angle with the camera map appearing as if it has rotated 30 degrees counter clockwise. The direction of travel in this map is 30 degrees. The version used in this study was identical to that in Coyne et. al. 2024, it had 18 practice items with feedback and then 24 trials without feedback. The practice feedback always showed the correct response direction, and the first 9 trials had some additional animation where a helicopter icon on the reference map rotates in 15 degree increments while simultaneously showing how the camera view would change with that rotation. The version in this paper (unlimited time and computer-generated images) is different from the version that is currently part of the ASTB (per item time limit and satellite images).</p> <hd id="AN0187435214-7">Spatial apperception task/airplane attitude</hd> <p>The Spatial Apperception Task (name used by the US military), which we will refer to as the Airplane Attitude Task throughout this paper (Fig. 2) shows the out the window view of an aircraft at a specific attitude flying toward varying terrain features (e.g., water). The objective is to identify which of several aircraft in different attitudes (roll, pitch, and yaw) over terrain would create that out the window perspective. A version of this test was originally used in aviation selection during WWII, and is still currently being used by the US Army for measuring spatial ability in aviation applicants. The version used in this study used items from the previous version of the ASTB (last used in 2013). Participants had 10 min to answer as many trials as they could (up to 35 items).</p> <p>Graph: Fig. 2 Sample trial from the spatial apperception/airplane attitude task (correct answer is B)</p> <hd id="AN0187435214-8">Direction orientation task</hd> <p>The Direction Orientation Task (Fig. 3) is designed to measure orientation, each of the 48 trials show two images, one with a map showing a vehicle's heading and the other represents the view from the vehicle's forward-facing camera. The objective is to identify one of four parking lots in the camera view. The parking lots are at the cardinal positions. Each trial asks participants to "image" a specific parking lot (e.g., South) by clicking on that lot. If the vehicle is traveling a cardinal direction e.g., West then the West parking lot is at the top of the camera view. The sample trial in Fig. 3. has the vehicle heading 120 degrees in this case there are two parking lots on the top. As mentioned earlier a mathematical solution could be applied to the trial in Fig. 3 in which you compute the difference between the heading and the target lot (180–120 = 60). The target lot is an additional 60 degrees from the heading which corresponds to the correct lot's 2 O'clock position in the camera view. Participants had 12 practice trials with feedback, followed by 48 trials without feedback. The 48 trails equate to every combination of heading (<reflink idref="bib12" id="ref33">12</reflink>) and target parking lot (<reflink idref="bib4" id="ref34">4</reflink>). There was unlimited time to complete the test; however, participants were instructed to answer as quickly and accurately as possible.</p> <p>Graph: Fig. 3 Sample trial from the direction orientation task (correct answer is 2 o'clock position)</p> <hd id="AN0187435214-9">Cube comparison and paper folding</hd> <p>Following the three aviation specific spatial ability tasks participants completed two additional general spatial ability tasks. The cube comparison and paper folding tasks were included to have a more diverse set of spatial ability measures (Elkstrom et al., [<reflink idref="bib19" id="ref35">19</reflink>]). The paper folding task was completed first. In the paper folding task participants are shown an image in which sheet of paper is folded along different lines, after a series of folds a hole is punched through the paper. Participants were tasked with determining where the hole(s) would appear once the paper is unfolded. Each item had five possible response choices. There were two blocks of ten trials and participants had three minutes to answer as many trials in each block as they could.</p> <p>The cube comparison task displays two cubes in which three sides are visible. Each side has a unique letter, symbol, or number on it and participants must determine if the two cubes are the same (but in a different orientation) or different. There were two blocks with 21 trials each and a three minute time limit for each block.</p> <hd id="AN0187435214-10">Mental counters</hd> <p>The Mental Counters task, is a working memory task which is part of the Navy's Armed Services Vocational Aptitude Battery which is used to select and classify enlisted recruits was also included in the study. Mental counters is an updating task in which individuals maintain 3 numbers in working memory. Updating tasks are ones which a stimulus must be monitored for new information which must then be encoded and replace the old information. Each trial contains a sequence of boxes flashing above or below three lines in the center of the screen. When a box flashes above the line the individual adds one to that number when it flashes below they subtract one from that position. Each position starts with an initial value of five. A trial is scored as correct when all three numbers are correctly reported. There was a total of 32 trials. The speed and length of the sequences varied across trials (all participants saw the same trials); however, the test has become an official test and specifics on timing and duration are considered proprietary (Fig. 4).</p> <p>Graph: Fig. 4 a Sample trial from the Cube Comparison task. The two cubes are different. b Sample trial from the Paper Folding task. The correct response is D</p> <hd id="AN0187435214-11">Stroop and flanker double conflict</hd> <p>Beyond spatial ability the study also included several other ability measures. These include the Stroop and Flanker double conflict tasks (Burgoyne et al., 2024) which are measures of attention control and had previously demonstrated predictive validity with flight training (Coyne et al., [<reflink idref="bib15" id="ref36">15</reflink>]). The Stroop double conflict task (see Fig. 5A) has the participant match the font color of the word on the top with the name of the color below. Both the stimulus and response can either be congruent (font color matches the word) or incongruent. Participants have a 30 s practice to answer as many items correctly as they can. Participants receive a point for each correct response and lose a point for each incorrect response. Following the 30 s practice there is a timed 90 s scored portion. Feedback is provided for every trial in both sessions. The Flanker double conflict task (see Fig. 5B) is similar to the Stroop task in that both the stimulus and response portion can be either congruent (all 5 arrows in the same direction) or incongruent (the outside arrows are in a different direction from the center). The goal of the task is to match the direction of the outside arrow on the top with the direction of the middle arrow in the response. Practice duration, scoring, and feedback are the same as the Stroop task.</p> <p>Graph: Fig. 5 a Sample trial from the Stroop Double Conflict Task. Objective is to match the font color on the top to the name of the word below (correct answer is the word blue on the bottom left). b Sample trial from the Flanker Double Conflict Task. Objective is to match the direction of the outside arrow on the top with direction of the middle arrow on the bottom (correct answer is the box on the bottom left)</p> <hd id="AN0187435214-12">Ravens advanced progressive matrices</hd> <p>Additionally, participants were presented with the odd number items from Raven's Advanced Progressive Matrices, an established measure of fluid intelligence (Raven & Raven, [<reflink idref="bib35" id="ref37">35</reflink>]). Participants had 10 min to solve up to 18 items from the Raven's test.</p> <hd id="AN0187435214-13">Navy selection and training outcome data</hd> <p></p> <hd id="AN0187435214-14">Aviation selection test battery (ASTB)</hd> <p>Data for tasks listed in the procedure were collected in a lab setting. The lab data was also linked with participants official ASTB (high-stakes) data which they took to qualify for aviation training. Applicants are allowed to take the ASTB up to three times with the most recent attempt being the official score utilized when determining an applicant's eligibility. All analyses use the final official score. The ASTB subtests scores used in the analysis include the Direction Orientation Task total correct and the Dichotic Listening total correct. No individual subtest of the ASTB is used to make selection decisions. Decisions are based upon three composite scores, which include combinations of the different crystalized and process-based subtests, however only two are relevant to the training being predicted in this paper. The first composite used in this paper is the Academic Qualification Rating (AQR) which we will refer to as the academic composite. The academic composite is designed to maximize prediction of the academic portion of the initial training, known as the Naval Introductory Flight Evaluation. The second composite is the Primary Flight Aptitude Rating (PFAR) or primary composite, is designed to predict training performance during Primary Flight Training, which is where pilots are trained to learn basic flying skills. Both the composites are stanines, a normalized scale that ranges from 1 to 9.</p> <hd id="AN0187435214-15">Naval introductory flight evaluation (NIFE)</hd> <p>All Naval Flight Students complete the Naval Introductory Flight Evaluation (NIFE), which is comprised of four phases. Performance on the ASTB is not meant to predict the first and final phases which involve components such as medical clearances, and water survival. For purposes of this analysis the outcomes of interest are from phase two of the introductory training which is a condensed three-week academic phase in which they must pass exams in five subjects. The introductory training grade is the average score from the 5 exams. The third phase of the introductory training requires students to complete a series of flights in a Cessna 172. Introductory flight score is the normalized grade students receive from their instructors during the flight portion of this initial training.</p> <hd id="AN0187435214-16">Primary flight training</hd> <p>Students who successfully complete the introductory training move on to primary flight training. At this point in training the aviators and flight officers are separated. For purposes of this analysis only data for the Student Naval Aviators are considered, as there are only a small number of Student Flight Officers. Additionally, data from a subset of students were excluded from the analysis as they were part of an experimental flight training program called "Avenger," which restructured the training, their access to instructors and provided additional access to virtual reality simulators. Students in this program were not assigned a grade for primary otherwise known as the Navy Standard Score (NSS). The standard score is a representation of an individuals' score during primary flight training relative to the average. The standard score has a center of 50 and is normalized within training groups to allow for an easier comparison across individuals trained at different locations. For Primary Flight training the standard score used in the analysis is based upon flight performance.</p> <hd id="AN0187435214-17">Results</hd> <p>The total sample included 307 officers. In each variable used in the analysis, some missing data was present. The highest rate of missingness was on the Flanker (19%) followed by Folding (14%) and Stroop (12%); all other variables had < 10% missingness. There were some issues with some of the software not loading correctly, particularly with the E-Prime software. There were also some issues linking student's lab data with their official ASTB data. With respect to the training outcome data, introductory grade and introductory flight are only available for students who complete the academic phase of the introductory training. Primary flight training data used in the analysis only included students who complete the introductory training, were Student Naval Aviators, and completed the traditional version of Primary. In two tasks, Flanker, and Stroop, some participants experienced difficulties in administration and received extremely low scores causing them to be below chance. To identify these individuals and correct their scores to be representative of their true ability level, we used a random forest based multivariate outlier detection method along with predictive mean matching replacement (Chandola et al., [<reflink idref="bib11" id="ref38">11</reflink>]; Wright & Ziegler, [<reflink idref="bib41" id="ref39">41</reflink>]). This technique was applied to all of the attention measures and three scores on the Dichotic Listening Task were corrected, four scores on the Stroop were corrected, and two scores on the Flanker were corrected. See Table 1 for sample size and descriptive statistics including both the maximum observed and the maximum possible for each scale after applying this correction. The reliabilities are also shown in Table 1. For all of the scales except Stroop and Flanker we report α using the Kuder–Richardson Formula 20 for dichotomous items. For Stroop and Flanker we report estimates using multilevel reliability because, within person, there are varying amounts of repeated measurements in a random order. Two values are reported, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>R</mi><mtext>KRN</mtext></msub></math> </ephtml> and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>R</mi><mtext>CN</mtext></msub></math> </ephtml> , respectively. The first value ( <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>R</mi><mtext>KRN</mtext></msub></math> </ephtml> ) represents the reliability of between-person mean scores on the assessment while the second value ( <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>R</mi><mrow><mi mathvariant="italic">CN</mi></mrow></msub></math> </ephtml> ) represents the reliability of within-person item level scores (Shrout & Lane, [<reflink idref="bib37" id="ref40">37</reflink>]). For these values, we can follow the general guidelines for interpreting an intraclass correlation to assess these values of reliability. This means that values greater than 0.90 indicate excellent reliability (Koo & Li, [<reflink idref="bib25" id="ref41">25</reflink>]).</p> <p>Table 1 Descriptive statistics all scales</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Item</p></th><th align="left"><p>N</p></th><th align="left"><p>Mean</p></th><th align="left"><p>Max (obs)</p></th><th align="left"><p>Max (pos)</p></th><th align="left"><p>SD</p></th><th align="left"><p>Skew</p></th><th align="left"><p>Reliability</p></th></tr></thead><tbody><tr><td align="left"><p>Airplane attitude</p></td><td align="left"><p>307</p></td><td char="." align="char"><p>13.75</p></td><td align="left"><p>34</p></td><td align="left"><p>35</p></td><td char="." align="char"><p>11.44</p></td><td align="left"><p>− 0.44</p></td><td align="left"><p>0.83</p></td></tr><tr><td align="left"><p>Terrain task</p></td><td align="left"><p>300</p></td><td char="." align="char"><p>15.54</p></td><td align="left"><p>24</p></td><td align="left"><p>24</p></td><td char="." align="char"><p>4.57</p></td><td align="left"><p>− 0.59</p></td><td align="left"><p>0.76</p></td></tr><tr><td align="left"><p>Lab direction</p></td><td align="left"><p>306</p></td><td char="." align="char"><p>40.34</p></td><td align="left"><p>48</p></td><td align="left"><p>48</p></td><td char="." align="char"><p>9.42</p></td><td align="left"><p>− 1.83</p></td><td align="left"><p>0.95</p></td></tr><tr><td align="left"><p>Folding</p></td><td align="left"><p>264</p></td><td char="." align="char"><p>6.83</p></td><td align="left"><p>20</p></td><td align="left"><p>20</p></td><td char="." align="char"><p>6.72</p></td><td align="left"><p>− 0.79</p></td><td align="left"><p>0.78</p></td></tr><tr><td align="left"><p>Cube</p></td><td align="left"><p>306</p></td><td char="." align="char"><p>14.08</p></td><td align="left"><p>42</p></td><td align="left"><p>42</p></td><td char="." align="char"><p>11.56</p></td><td align="left"><p>0.00</p></td><td align="left"><p>0.85</p></td></tr><tr><td align="left"><p>Stroop</p></td><td align="left"><p>269</p></td><td char="." align="char"><p>37.23</p></td><td align="left"><p>63</p></td><td align="left"><p>NA</p></td><td char="." align="char"><p>13.29</p></td><td align="left"><p>− 1.04</p></td><td align="left"><p>0.96, 0.99<sup>1</sup></p></td></tr><tr><td align="left"><p>Flanker</p></td><td align="left"><p>248</p></td><td char="." align="char"><p>37.66</p></td><td align="left"><p>83</p></td><td align="left"><p>NA</p></td><td char="." align="char"><p>15.81</p></td><td align="left"><p>− 0.23</p></td><td align="left"><p>0.97, 0.99<sup>1</sup></p></td></tr><tr><td align="left"><p>Counters</p></td><td align="left"><p>307</p></td><td char="." align="char"><p>20.56</p></td><td align="left"><p>32</p></td><td align="left"><p>32</p></td><td char="." align="char"><p>5.81</p></td><td align="left"><p>− 0.64</p></td><td align="left"><p>0.82</p></td></tr><tr><td align="left"><p>Ravens</p></td><td align="left"><p>278</p></td><td char="." align="char"><p>9.76</p></td><td align="left"><p>17</p></td><td align="left"><p>18</p></td><td char="." align="char"><p>3.10</p></td><td align="left"><p>− 0.26</p></td><td align="left"><p>0.72</p></td></tr><tr><td align="left"><p>ASTB direction</p></td><td align="left"><p>302</p></td><td char="." align="char"><p>43.38</p></td><td align="left"><p>48</p></td><td align="left"><p>48</p></td><td char="." align="char"><p>4.70</p></td><td align="left"><p>− 1.93</p></td><td align="left"><p>0.82</p></td></tr><tr><td align="left"><p>ASTB dichotic</p></td><td align="left"><p>302</p></td><td char="." align="char"><p>37.88</p></td><td align="left"><p>44</p></td><td align="left"><p>44</p></td><td char="." align="char"><p>5.15</p></td><td align="left"><p>− 1.46</p></td><td align="left"><p>0.82<sup>2</sup></p></td></tr><tr><td align="left"><p>Introductory grade</p></td><td align="left"><p>289</p></td><td char="." align="char"><p>93.06</p></td><td align="left"><p>100</p></td><td align="left"><p>100</p></td><td char="." align="char"><p>4.0</p></td><td align="left"><p>−.057</p></td><td align="left" /></tr><tr><td align="left"><p>Introductory Flight Score</p></td><td align="left"><p>283</p></td><td char="." align="char"><p>1.10</p></td><td align="left"><p>1.30</p></td><td align="left"><p>1.35</p></td><td char="." align="char"><p>0.06</p></td><td align="left"><p>1.21</p></td><td align="left" /></tr><tr><td align="left"><p>Primary Score</p></td><td align="left"><p>163</p></td><td char="." align="char"><p>52.08</p></td><td align="left"><p>80</p></td><td align="left"><p>80</p></td><td char="." align="char"><p>11.35</p></td><td align="left"><p>0.19</p></td><td align="left" /></tr></tbody></table> </ephtml> </p> <p> <sups>1</sups>Reliability is α for all scales except Stroop and Flanker which are reported using multilevel reliability <sups>2</sups>Reliability for DLT was estimated using only a portion of the sample (<emph>n</emph> = 169)</p> <p>Table 2 shows the correlations between all 9 scales using pairwise deletion for missingness; Spearman's correlation was used to reduce non-normality bias (Bishara & Hittner, [<reflink idref="bib4" id="ref42">4</reflink>]). The correlation between the academic composite and Introductory Training Grade was positive, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo stretchy="false">(</mo><mn>283</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0.47</mn><mo>,</mo><mi>p</mi><mo><</mo><mo>.</mo><mn>001</mn></mrow></math> </ephtml> . We do not provide correlations between the academic composite and any of the subtests as this is proprietary information; however, the academic composite is designed to predict success in ground school and includes the ASTB's academic subtests (i.e., Math, Verbal, Aviation and Nautical Information) as well as some of the process-based subtests. The correlation between the primary composite and Primary Score was positive <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>r</mi><mo stretchy="false">(</mo><mn>160</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0.26</mn><mo>,</mo><mi>p</mi><mo><</mo><mo>.</mo><mn>001</mn></mrow></math> </ephtml> . In Appendix A we have included the correlation between all 9 scales using the same method noted above but instead using the first attempt on the ASTB rather than the final. We include this table for reference; however, since the Navy uses the final attempt on the ASTB for selection purposes we focus on those scores for the purposes of this study.</p> <p>Table 2 Spearman correlations for all variables</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Airplane Attitude</p></th><th align="left"><p>Terrain Task</p></th><th align="left"><p>Lab Direction</p></th><th align="left"><p>Folding</p></th><th align="left"><p>Cube</p></th><th align="left"><p>Stroop</p></th><th align="left"><p>Flanker</p></th><th align="left"><p>Counters</p></th><th align="left"><p>Ravens</p></th><th align="left"><p>ASTB Direction</p></th><th align="left"><p>ASTB Dichotic</p></th><th align="left"><p>Introductory Grade</p></th><th align="left"><p>Flight Score</p></th><th align="left"><p>Primary Score</p></th></tr></thead><tbody><tr><td align="left"><p>Airplane Attitude</p></td><td char="." align="char"><p>–</p></td><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Terrain Task</p></td><td char="." align="char"><p>0.51***</p></td><td align="left"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Lab Direction</p></td><td char="." align="char"><p>0.52***</p></td><td align="left"><p>0.42***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Folding</p></td><td char="." align="char"><p>0.35***</p></td><td align="left"><p>0.35***</p></td><td char="." align="char"><p>0.33***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Cube</p></td><td char="." align="char"><p>0.26***</p></td><td align="left"><p>0.30***</p></td><td char="." align="char"><p>0.27***</p></td><td char="." align="char"><p>0.31***</p></td><td char="." align="char"><p>–</p></td><td align="left" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Stroop</p></td><td char="." align="char"><p>0.13*</p></td><td align="left"><p>0.25***</p></td><td char="." align="char"><p>0.14*</p></td><td char="." align="char"><p>0.32***</p></td><td char="." align="char"><p>0.21***</p></td><td align="left"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Flanker</p></td><td char="." align="char"><p>0.26***</p></td><td align="left"><p>0.09</p></td><td char="." align="char"><p>0.31***</p></td><td char="." align="char"><p>0.34***</p></td><td char="." align="char"><p>0.20**</p></td><td align="left"><p>0.33***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Counters</p></td><td char="." align="char"><p>0.21***</p></td><td align="left"><p>0.27***</p></td><td char="." align="char"><p>0.18**</p></td><td char="." align="char"><p>0.38***</p></td><td char="." align="char"><p>0.20***</p></td><td align="left"><p>0.39***</p></td><td char="." align="char"><p>0.33***</p></td><td char="." align="char"><p>–</p></td><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Ravens</p></td><td char="." align="char"><p>0.21***</p></td><td align="left"><p>0.30***</p></td><td char="." align="char"><p>0.24***</p></td><td char="." align="char"><p>0.47***</p></td><td char="." align="char"><p>0.22***</p></td><td align="left"><p>0.38***</p></td><td char="." align="char"><p>0.37***</p></td><td char="." align="char"><p>0.41***</p></td><td align="left"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>ASTB Direction</p></td><td char="." align="char"><p>0.02</p></td><td align="left"><p> − 0.01</p></td><td char="." align="char"><p>0.03</p></td><td char="." align="char"><p> − 0.08</p></td><td char="." align="char"><p> − 0.06</p></td><td align="left"><p>0.04</p></td><td char="." align="char"><p>0.01</p></td><td char="." align="char"><p> − 0.07</p></td><td align="left"><p> − 0.14*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>ASTB Dichotic</p></td><td char="." align="char"><p> − 0.05</p></td><td align="left"><p>0</p></td><td char="." align="char"><p>0.07</p></td><td char="." align="char"><p>0.06</p></td><td char="." align="char"><p>0.01</p></td><td align="left"><p>0.15*</p></td><td char="." align="char"><p>0.19**</p></td><td char="." align="char"><p>0.12*</p></td><td align="left"><p>0.13*</p></td><td char="." align="char"><p>0.05</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Introductory Grade</p></td><td char="." align="char"><p>0.25***</p></td><td align="left"><p>0.17**</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.23***</p></td><td char="." align="char"><p>0.25***</p></td><td align="left"><p>0.22***</p></td><td char="." align="char"><p>0.29***</p></td><td char="." align="char"><p>0.18**</p></td><td align="left"><p>0.22***</p></td><td char="." align="char"><p> − 0.03</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Introductory Flight Score</p></td><td char="." align="char"><p>0.39***</p></td><td align="left"><p>0.24***</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.14*</p></td><td char="." align="char"><p>0.11</p></td><td align="left"><p>0</p></td><td char="." align="char"><p>0.20**</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>0.06</p></td><td char="." align="char"><p>0.02</p></td><td char="." align="char"><p> − 0.07</p></td><td char="." align="char"><p>0.29***</p></td><td char="." align="char"><p>–</p></td><td align="left" /></tr><tr><td align="left"><p>Primary Score</p></td><td char="." align="char"><p>0.35***</p></td><td align="left"><p>0.15</p></td><td char="." align="char"><p>0.18*</p></td><td char="." align="char"><p>0.15</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>0.19*</p></td><td char="." align="char"><p>0.25**</p></td><td char="." align="char"><p>0.17*</p></td><td align="left"><p>0.15</p></td><td char="." align="char"><p> − 0.03</p></td><td char="." align="char"><p>0.03</p></td><td char="." align="char"><p>0.45***</p></td><td char="." align="char"><p>0.49***</p></td><td align="left"><p>–</p></td></tr></tbody></table> </ephtml> </p> <p> <sups> <emph>*</emph> </sups> <emph> p</emph> < <emph>.</emph>05; <emph>** p</emph> < <emph>.</emph>01; <emph>***</emph> < <emph>.</emph>001</p> <p>To identify factor structure differences among these scales when using high stakes ASTB Direction Orientation Task data as opposed to lab Direction Orientation Task data, we conducted Exploratory Factor Analyses (EFAs) using an oblique rotation to allow for inter-factor correlation estimation. EFAs were conducted on two samples, each sample included the personnel with both high stakes and lab data (<emph>n</emph> = 302), and the lab sample additionally included personnel with only lab data (<emph>n</emph> = 5). This resulted in a high-stakes sample with 302 personnel and a lab sample with complete Navy training data with 306 personnel. Full Information Maximum Likelihood (FIML) was used to account for missing data. We first applied the Hull method (Lorenzo-Seva et al., [<reflink idref="bib28" id="ref43">28</reflink>]), the empirical Kaiser criterion (EKC) (Braeken & Van Assen, [<reflink idref="bib5" id="ref44">5</reflink>]), and the Comparison Data method (CD) (Ruscio & Roche, [<reflink idref="bib36" id="ref45">36</reflink>]) to estimate dimensionality. The results of these three analyses, as well as an evaluation of eigenvalues, were used to determine the number of factors present in each set of data before evaluating the fit of an EFA model.</p> <hd id="AN0187435214-18">Lab exploratory factor analyses</hd> <p>Using the lab data (<emph>n</emph> = 306), the EKC method indicated a one factor model would fit these data the best, while the Hull and CD methods indicated a two factor model would fit these data the best. The eigenvalues for these data contained two values above 1 (3.45 and 1.30) and all remaining values were below 1. Given these results, we assessed the model fit and structure of a two factor EFA model using an oblique rotation. Figure 6 shows the EFA model using the lab data.</p> <p>Graph: Fig. 6 Two factor EFA model using spatial ability measures as well as DOT collected in lab</p> <p>Results indicated that the two factor model fit these data well, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>χ</mi></mrow><mn>2</mn></msup><mfenced close=")" open="("><mn>19</mn></mfenced><mo>=</mo><mn>31.88</mn><mo>,</mo><mi>p</mi><mo><</mo><mo>.</mo><mn>05</mn><mo>,</mo><mtext>RMSEA</mtext><mo>=</mo><mn>0.05</mn><mo>,</mo><mtext>TLI</mtext><mo>=</mo><mn>0.96</mn><mo>,</mo><mtext>CFI</mtext><mo>=</mo><mn>0.98</mn></mrow></math> </ephtml> . In this model, cross loadings were below 0.20 and considerably lower than their respective dominant loadings, indicating that there was no construct contamination (Howard & Henderson, [<reflink idref="bib23" id="ref46">23</reflink>]). While there are no recognized standard cutoff values for what constitutes high or low standardized dominant loadings (Peterson, [<reflink idref="bib33" id="ref47">33</reflink>]), values above 0.40 are generally considered as high dominant loadings (Comrey & Lee, [<reflink idref="bib12" id="ref48">12</reflink>]). Variables with loadings below 0.40, therefore, are generally considered to not be a good representation of the factor. However, provided their cross loadings are not high, they still appropriately load on to their dominant factor.</p> <p>The first factor was comprised of Ravens, Counters, Flanker, Folding, Cube, and Stroop, where the standardized dominant loadings were all above 0.40, meaning that they load strongly on to this factor, with the exception of the Cube test which had a loading of 0.32. This indicated that the Cube test did not strongly load on to the first factor. The Cube test had a cross loading of 0.18, meaning that it still appropriately loads on to the first factor, but is not a good representation of the first factor. The second factor (correlated to the first at 0.50) was comprised of the Airplane Attitude Task, Terrain Task, and the lab-based Direction Orientation Task, all with dominant loadings above 0.40 and cross loadings below 0.10.</p> <p>Communalities represent the amount of variance explained in each variable by the factors, where higher communality values are preferable. Similar to factor loadings, there is no standard value which constitutes a good communality value. For the purposes of this study, we considered communality values below 0.30 to be poor fit (MacCallum et al., [<reflink idref="bib29" id="ref49">29</reflink>]; Sovey et al., [<reflink idref="bib39" id="ref50">39</reflink>]). The two factor EFA model using only lab data had communality values above 0.30 for all variables except for the Cube test which had a communality of 0.19. This indicated that this factor structure did not account well for the variation in the Cube test.</p> <hd id="AN0187435214-19">Lab and high-stakes exploratory factor analyses</hd> <p>We then analyzed the factor structure using ASTB (high-stakes data) and the lab data. This sample included all officers with high-stakes data (<emph>n</emph> = 302). Using the ASTB and lab data, the EKC method indicated a one factor model would fit these data the best, while the Hull and CD methods indicated a two factor model would fit these data the best. The eigenvalues using the ASTB and lab data contained two values above 1 (3.47 and 1.35), one value very close to 1 (1.11), with all remaining values below 1. Given these results, we assessed the model fit and structure of a two factor EFA model using an oblique rotation. Figure 7 shows the results of this second EFA.</p> <p>Graph: Fig. 7 Two factor EFA model using spatial ability tasks and high stakes DOT data</p> <p>Results showed that the two factor model fit these data well, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msup><mrow><mi>χ</mi></mrow><mn>2</mn></msup><mfenced close=")" open="("><mn>34</mn></mfenced><mo>=</mo><mn>56.98</mn><mo>,</mo><mi>p</mi><mo><</mo><mo>.</mo><mn>01</mn><mo>,</mo><mtext>RMSEA</mtext><mo>=</mo><mn>0.05</mn><mo>,</mo><mtext>TLI</mtext><mo>=</mo><mn>0.94</mn><mo>,</mo><mtext>CFI</mtext><mo>=</mo><mn>0.96</mn></mrow></math> </ephtml> . In this model, all cross loadings were below 0.20 and considerably lower than their respective dominant loadings, with the exception of the high-stakes Direction Orientation. The high-stakes Direction Orientation Task had a low cross loading of − 0.013; however, its dominant loading was also very low (0.06). Even though these values were close, because the dominant loading was so low we do not interpret this as construct contamination. The first factor was comprised of Ravens, Counters, Flanker, Folding, Stroop, Cube, and Dichotic Listening, all with dominant standardized loadings above 0.40, with the exception of the Dichotic Listening Task which had a loading of 0.22, and Cube which had a loading of 0.30. Although the Dichotic Listening Task had a low dominant loading, its cross loading was not comparatively high and was also negative (− 0.11) meaning that the Dichotic Listening Task appropriately loaded on to the first factor and is negatively associated with the second factor. The Cube test had a higher cross loading, 0.19, which, in conjunction with its lower dominant loading, could indicate construct contamination and that it is not representative of the first factor.</p> <p>The second factor (correlated at 0.50 to the first) was comprised of Airplane Attitude Task, Terrain Task, Lab Direction Orientation Task, and high-stakes Direction Orientation Task. All variables in the second factor, with the exception of the high-stakes Direction Orientation Task, had high dominant loadings (above 0.40) and low cross loadings (< 0.10). The Lab Direction Task had a dominant standardized loading (0.62); however, the high-stakes Direction Task had a very low dominant standardized loading (0.06). This means that the Lab Direction Orientation was a good representation of this factor while the high-stakes Direction Orientation was not.</p> <p>The communality values for all tests in this model were above 0.30 except for three values. The Cube had a communality of 0.18, the Dichotic Listening had a value of 0.04, and the high-stakes Direction Orientation Task had a value of 0.003. This means that this factor structure did not account well for the variation within these tests. Given these values, we would not consider these variables to be included in the overall factor model. Notably, the communality value for the Lab Direction Orientation Task in this model was 0.44. The factor structure for the variables collected in lab did not change when introducing high-stakes data; however, model fit for high-stakes Direction Orientation Task is poor.</p> <p>All together, these results present strong evidence that the latent factor assessed using high-stakes Direction Orientation Task responses is not psychometrically equivalent to that of the Direction Orientation Task responses collected in lab. Additionally, the Dichotic Listening Task does not fit well into the overall factor model.</p> <hd id="AN0187435214-20">Discussion</hd> <p>The pattern of results did not match our first hypothesis. The expectation was that Direction Orientation Task in the lab, and the one taken as part of the ASTB would perform similarly; however, the two tests were not even correlated with each other. Direction Orientation Task in the lab loaded onto a spatial factor with the two other aviation selection spatial ability tests (Terrain Task and Airplane Attitude Task). All three of these lab-based spatial ability measures were correlated with training outcomes, whereas the high-stakes Direction Orientation Task was not. While unexpected, these results still provide some support to our hypothesis that the Direction Orientation Task, when taken within the ASTB, may no longer be measuring spatial ability. The authors are aware that some portion of test applicants may be using scrap paper to assist in completing the Direction Orientation Task within the high-stakes environment. Participants in the low stakes testing environment would not have been able to use this strategy.</p> <p>The authors included the Cube and Folding Tasks as measures of spatial ability and expected them to load on the spatial factor. However, both measures loaded on the general ability factor. The Cube task loaded poorly on the factor; however, the Paper Folding task was a good fit. One possibility is that individuals may have been using a more analytic approach to the Paper Folding task. Indeed, Burte et al. ([<reflink idref="bib8" id="ref51">8</reflink>]) suggests that non-spatial strategies such as an analytical approach of counting folds and holes are also used when solving items in the task. They suggest that the task is not purely a measure of spatial visualization. This may be why the task loaded on the general factor and demonstrates some of the challenges with process-based measures.</p> <p>The results for the new attention control measures did match our hypothesis. The new attention control measures were significantly correlated with some of the training outcomes, whereas the attention measure in the high-stakes environment, the Dichotic Listening Task, was not. The ASTB Dichotic Listening Task was correlated with the attention measures and did load on to the same factor in the EFA; however, the structural fit was poor and indicated that some variation in the listening task was not explained by the factor structure.</p> <p>Some individuals who take the official ASTB are highly familiar and practiced with portions of the test like the Direction Orientation Task and Dichotic Listening Task even if it is their first official test attempt. In preparing for the official test some individuals learn ways to leverage non-spatial solutions to Direction Orientation Task prior to ever having taken the ASTB. In most cases individuals participating in the study had taken the official ASTB over a year prior to the lab-based study. The familiarity that they acquired in preparation for the ASTB may have been lost over that time. The lack of familiarity may explain why Direction Orientation Task in the lab appears to assess spatial ability (and predict training outcomes), whereas Direction Orientation Task scores from the ASTB do not. The results also support the concerns Ackerman ([<reflink idref="bib1" id="ref52">1</reflink>]) raised about process-based measures.</p> <p>One explanation for these results is that process-based measures function by having individuals perform either a novel operation or use novel stimuli (Ackerman, [<reflink idref="bib1" id="ref53">1</reflink>]). The degree to which practice increases test specific skills, versus an increase in the construct being assessed reflects bias with the measure. While there is a growing body of the literature on effects of re-testing in high-stakes environments (e.g., Lievens et al., [<reflink idref="bib26" id="ref54">26</reflink>]; Sibley, [<reflink idref="bib38" id="ref55">38</reflink>]), there has not been a sufficient look into the impact of practice which occurs before even the initial test. The sophistication of online resources available to potential aviation applicants as evidenced by their ability to quickly generate unofficial versions of subtests such as Terrain Task within weeks of it appearing suggest that simply looking at re-testing is not sufficient. Examining how repeated exposure and practice can introduce measurement bias and impact the effectiveness of a test should be a concern within high-stakes tests such as the ASTB.</p> <p>One of the additional goals of the study was to further evaluate the new Terrain Task. The data from this study provide some additional evidence for the effectiveness of the Terrain Task as a potential replacement to Direction Orientation. However, the correlations of the Direction Orientation Task in the lab with the same training outcomes limit the value of these results when making decisions about future versions of the ASTB. As the US Navy considers replacing the Direction Orientation Task as a measure of spatial ability, it should look beyond assessing test–retest reliability for future spatial ability measures. An assumption should be made that some applicants will have had practice with any new test prior to even their initial score on the ASTB. It is likely that practice on the Direction Orientation Task increases test specific skills not relevant to the spatial ability construct, especially given that non-spatial strategies have been shared for the task. While the Terrain Task is a more flexible test given the potential for unlimited number of items, it is unclear how practice on the Terrain Task might change the test's effectiveness. With respect to the Direction Orientation Task, participants can practice on both the process being measured, as well as with the exact stimuli they will see in the official high-stakes test. The Terrain Task being evaluated within the ASTB has multiple forms, and a large item bank, and while applicants can practice unofficial versions, they do not have access to the actual test items or maps. Thus, when practicing unofficial versions of Terrain Task applicants are being exposed to the process of solving terrain association problems not the test stimuli. Further methods which remove the visual-spatial comparison between the two maps in the Terrain Task have not been identified. These key differences between the tests could allow the Terrain Task to retain its effectiveness even with practice; however, this is an empirical question which has not yet been evaluated.</p> <p>To a lesser extent this paper also looked at measures of attention, another process-based intelligence measure that is currently assessed within the ASTB. The ASTB relies on the Dichotic Listening Task to measure attention. There are some practical concerns and potential vulnerabilities with this test which potentially hinder its validity. The specifics of these vulnerabilities cannot be shared as the test is still in active use and disseminating that information could impact the Dichotic Listening subtest's effectiveness. The Dichotic Listening Task was previously included in the Air Force pilot selection battery but has subsequently been dropped.</p> <p>The study also found additional evidence on the effectiveness of two new attention control measures in predicting flight training outcomes. However, as with all the lab-based measures there was the benefit of the participants in this study being naïve to the task and stimulus. The extent to which these tests depend on a lack of familiarity with the stimuli will ultimately determine whether they are good candidates for use in aviation selection. One disadvantage these double conflict tasks have compared to the Terrain Task is that, if they are replicated by motivated participants, it will allow participants to practice with both the stimulus and the process. The scoring of these tasks, however, does not have a fixed celling as participants are given as many trials as they can answer within the time limit. As with Terrain Task the durability of these tasks under repeated practice needs to be further evaluated.</p> <p>The concerns raised by this paper extend beyond the ASTB and apply more generally to any high-stakes situations in which processed based tests may be used to measure individual differences. The ASTB's composite scores which are comprised of both crystalized and process-based measures still predict training outcomes. Test developers routinely evaluate and replace crystalized test items to ensure the test retains its reliability and effectiveness. High-stakes test developers should take a deeper look at all process-based measures. Learnability and familiarity are issues beyond that which can be evaluated solely by investigating a test's reliability with repeat exposure. Results obtained in lab settings such as those found in this paper must be cautiously considered as performance with a "naïve" sample may not reflect performance once the test is part of an official high-stakes battery. The challenge is that while attention control and spatial ability are important constructs in predicting training outcomes, their effective measurement may hinge on a naivety that can no longer be guaranteed in high-stakes testing. The assumption should be that motivated applicants will replicate any new test that is introduced and be heavily practiced on those tests prior to taking the official battery. This assumption runs counter to the approach test developers have taken in the past. Further research on repeated and deliberate practice in process related measures needs to be done if these measures are going to be used effectively in high-stakes testing environments in the future.</p> <hd id="AN0187435214-21">Acknowledgements</hd> <p>We would like to thank the Naval Strike Group Training and Assessment Program, led by Natalie Steinhauser, and the Manpower, Personnel, Training and Education Information Sciences Program, led by LCDR Michael Natali. Both programs are from the Office of Naval Research and supported different components of this research.</p> <hd id="AN0187435214-22">Author contributions</hd> <p>JC developed the Terrain Orientation Task, wrote all software for data collection, performed data analysis, and led effort to write original draft. LJ performed most of the data analysis and significantly contributed to the results and discussion sections. KS oversaw all data collection and contributed to writing and editing the manuscript. CS contributed to experimental design, interpretation of results and editing. CF funding acquisition, conceptualization of investigation and editing. SM contributed to data collection management, interpretation of results and editing.</p> <hd id="AN0187435214-23">Funding</hd> <p>This research was funded by the Office of Secretary of Defense and the Office of Naval Research.</p> <hd id="AN0187435214-24">Availability of data and materials</hd> <p>The official Navy data used in this study are owned by two different Navy Commands, and restrictions apply to the availability of these data. The data from the laboratory tasks are available from the corresponding author upon reasonable request.</p> <hd id="AN0187435214-25">Declarations</hd> <p></p> <hd id="AN0187435214-26">Competing interests</hd> <p>The authors declare they have no competing interests.</p> <hd id="AN0187435214-27">Ethics approval and consent to participate</hd> <p>This research was approved by the Naval Research Laboratory's Institutional Review Board (NRL-2021.0003). All participants provided informed consent prior to completing the experiment and signed a data release form allowing researchers to link training outcomes and selection test data to experiment data.</p> <hd id="AN0187435214-28">Consent for publication</hd> <p>Not applicable.</p> <hd id="AN0187435214-29">Appendix A</hd> <p></p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" colspan="15"><p>Table: Spearman Correlations for All Variables on First Attempt</p></th></tr><tr><th align="left" /><th align="left"><p>Airplane attitude</p></th><th align="left"><p>Terrain task</p></th><th align="left"><p>Lab direction</p></th><th align="left"><p>Folding</p></th><th align="left"><p>Cube</p></th><th align="left"><p>Stroop</p></th><th align="left"><p>Flanker</p></th><th align="left"><p>Counters</p></th><th align="left"><p>Ravens</p></th><th align="left"><p>ASTB direction</p></th><th align="left"><p>ASTB dichotic</p></th><th align="left"><p>Introductory Grade</p></th><th align="left"><p>Flight score</p></th><th align="left"><p>Primary score</p></th></tr></thead><tbody><tr><td align="left"><p>Airplane Attitude</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Terrain Task</p></td><td char="." align="char"><p>0.51***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Lab Direction</p></td><td char="." align="char"><p>0.52***</p></td><td char="." align="char"><p>0.42***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Folding</p></td><td char="." align="char"><p>0.35***</p></td><td char="." align="char"><p>0.35***</p></td><td char="." align="char"><p>0.33***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Cube</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.30***</p></td><td char="." align="char"><p>0.27***</p></td><td char="." align="char"><p>0.31***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Stroop</p></td><td char="." align="char"><p>0.13*</p></td><td char="." align="char"><p>0.25***</p></td><td char="." align="char"><p>0.14*</p></td><td char="." align="char"><p>0.32***</p></td><td char="." align="char"><p>0.22***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Flanker</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>0.31***</p></td><td char="." align="char"><p>0.34***</p></td><td char="." align="char"><p>0.20**</p></td><td char="." align="char"><p>0.35***</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Counters</p></td><td char="." align="char"><p>0.21***</p></td><td char="." align="char"><p>0.27***</p></td><td char="." align="char"><p>0.18**</p></td><td char="." align="char"><p>0.38***</p></td><td char="." align="char"><p>0.20***</p></td><td char="." align="char"><p>0.39***</p></td><td char="." align="char"><p>0.33***</p></td><td char="." align="char"><p>–</p></td><td align="left" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Ravens</p></td><td char="." align="char"><p>0.21***</p></td><td char="." align="char"><p>0.30***</p></td><td char="." align="char"><p>0.24***</p></td><td char="." align="char"><p>0.47***</p></td><td char="." align="char"><p>0.22***</p></td><td char="." align="char"><p>0.38***</p></td><td char="." align="char"><p>0.37***</p></td><td char="." align="char"><p>0.41***</p></td><td align="left"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>ASTB Direction</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>0.11</p></td><td char="." align="char"><p>0.12*</p></td><td char="." align="char"><p>0.07</p></td><td char="." align="char"><p>0.04</p></td><td char="." align="char"><p>0.12</p></td><td char="." align="char"><p>0.13</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>− 0.04</p></td><td char="." align="char"><p>− </p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>ASTB Dichotic</p></td><td char="." align="char"><p>0.03</p></td><td char="." align="char"><p>0.05</p></td><td char="." align="char"><p>0.17**</p></td><td char="." align="char"><p>0.06</p></td><td char="." align="char"><p>0.10</p></td><td char="." align="char"><p>0.20**</p></td><td char="." align="char"><p>0.30**</p></td><td char="." align="char"><p>0.17**</p></td><td align="left"><p>0.15*</p></td><td char="." align="char"><p>0.11*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Introductory Grade</p></td><td char="." align="char"><p>0.25***</p></td><td char="." align="char"><p>0.17**</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.23***</p></td><td char="." align="char"><p>0.25***</p></td><td char="." align="char"><p>0.23***</p></td><td char="." align="char"><p>0.29***</p></td><td char="." align="char"><p>0.18**</p></td><td align="left"><p>0.22***</p></td><td char="." align="char"><p>0.07</p></td><td char="." align="char"><p>0.14*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char" /><td align="left" /></tr><tr><td align="left"><p>Introductory Flight Score</p></td><td char="." align="char"><p>0.39***</p></td><td char="." align="char"><p>0.24***</p></td><td char="." align="char"><p>0.26***</p></td><td char="." align="char"><p>0.14*</p></td><td char="." align="char"><p>0.11</p></td><td char="." align="char"><p>0.00</p></td><td char="." align="char"><p>0.20**</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>0.06</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>− 0.06</p></td><td char="." align="char"><p>0.29***</p></td><td char="." align="char"><p>–</p></td><td align="left" /></tr><tr><td align="left"><p>Primary Score</p></td><td char="." align="char"><p>0.35***</p></td><td char="." align="char"><p>0.15</p></td><td char="." align="char"><p>0.18*</p></td><td char="." align="char"><p>0.15</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>0.19*</p></td><td char="." align="char"><p>0.24**</p></td><td char="." align="char"><p>0.17*</p></td><td align="left"><p>0.15</p></td><td char="." align="char"><p>− 0.03</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>0.45***</p></td><td char="." align="char"><p>0.49***</p></td><td align="left"><p>–</p></td></tr></tbody></table> </ephtml> </p> <p>*<emph>p</emph> < .05; **<emph>p</emph> < .01; *** < .001</p> <hd id="AN0187435214-30">Abbreviations</hd> <p></p> <p>• AQR</p> <p></p> <ulist> <item> Academic qualification rating</item> <p></p> </ulist> <p>• ASTB</p> <p></p> <ulist> <item> Aviation selection test battery</item> <p></p> </ulist> <p>• EFA</p> <p></p> <ulist> <item> Exploratory factor analysis</item> <p></p> </ulist> <p>• NIFE</p> <p></p> <ulist> <item> Naval flight introductory evaluation</item> <p></p> </ulist> <p>• NFS</p> <p></p> <ulist> <item> Naval flight students</item> <p></p> </ulist> <p>• NSS</p> <p></p> <ulist> <item> Navy Standard Score</item> <p></p> </ulist> <p>• PFAR</p> <p></p> <ulist> <item> Pilot flight aptitude rating</item> </ulist> <hd id="AN0187435214-31">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0187435214-32"> <title> References </title> <blist> <bibl id="bib1" idref="ref9" type="bt">1</bibl> <bibtext> Ackerman PL. Intelligence process vs. content and academic performance: A trip through a house of mirrors. Journal of Intelligence. 2022; 10; 4: 128-147. 36547515. 9782628</bibtext> </blist> <blist> <bibl id="bib2" idref="ref5" type="bt">2</bibl> <bibtext> Arnold, R. D. (2002). The utility of the aviation selection test battery (ASTB) in the reduction of naval aviation training attrition. In Naval Aerospace Medical Institute, technical memorandum dated 10 March 2002, Pensacola, FL.</bibtext> </blist> <blist> <bibl id="bib3" idref="ref12" type="bt">3</bibl> <bibtext> Barron LG, Rose MR. Relative validity of distinct spatial abilities: An example with implications for diversity. International Journal of Selection and Assessment. 2013; 21; 4: 400-406</bibtext> </blist> <blist> <bibl id="bib4" idref="ref34" type="bt">4</bibl> <bibtext> Bishara AJ, Hittner JB. Reducing bias and error in the correlation coefficient due to nonnormality. Educational and Psychological Measurement. 2015; 75; 5: 785-804. 29795841</bibtext> </blist> <blist> <bibl id="bib5" idref="ref44" type="bt">5</bibl> <bibtext> Braeken J, Van Assen MA. An empirical kaiser criterion. Psychological Methods. 2017; 22; 3: 450. 27031883</bibtext> </blist> <blist> <bibl id="bib6" idref="ref22" type="bt">6</bibl> <bibtext> Burgoyne AP, Tsukahara JS, Mashburn CA, Pak R, Engle RW. Nature and measurement of attention control. Journal of Experimental Psychology: General. 2023; 152; 8: 2369. 37079831</bibtext> </blist> <blist> <bibl id="bib7" idref="ref26" type="bt">7</bibl> <bibtext> Burgoyne AP, Mashburn CA, Tsukahara JS, Pak R, Coyne JT, Foroughi C, Sibley C, Drollinger SM, Engle RW. Attention control measures improve the prediction of performance in navy trainees. International Journal of Selection and Assessment. 2025; 33e12510</bibtext> </blist> <blist> <bibl id="bib8" idref="ref51" type="bt">8</bibl> <bibtext> Burte H, Gardony AL, Hutton A, Taylor HA. Knowing when to fold'em: Problem attributes and strategy differences in the Paper Folding test. Personality and Individual Differences. 2019; 146: 171-181</bibtext> </blist> <blist> <bibl id="bib9" idref="ref14" type="bt">9</bibl> <bibtext> Carretta, T. R. (2005). Development and validation of the test of basic aviation skills (TBAS). Air Force Research Laboratory research report (AFRL/HTR-2005–0172). Retrieved October 7, 2024 from https://apps.dtic.mil/sti/tr/pdf/ADA442563.pdf</bibtext> </blist> <blist> <bibtext> Carretta TR, Ree MJ. Correction for range restriction: Lessons from 20 research scenarios. Military Psychology. 2022; 34; 5: 551-569. 38536384. 10069334</bibtext> </blist> <blist> <bibtext> Chandola V, Banerjee A, Kumar V. Anomaly detection: A survey. ACM Computing Surveys. 2009; 41; 3: 1-58</bibtext> </blist> <blist> <bibtext> Comrey AL, Lee HB. A first course in factor analysis. 2013; Psychology Press</bibtext> </blist> <blist> <bibtext> Coyne, J. T, Brown, N. L, Foroughi, C. K, Sibley, C, Sexauer, E, & Rovira, E. (2020). The use of non-spatial strategies in the direction orientation task. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 64, No. 1, pp. 802–806). SAGE Publications.</bibtext> </blist> <blist> <bibtext> Coyne JT, Dollinger S, Brown N, Foroughi C, Sibley C, Phillips H. Limitations of current spatial ability testing for military aviators. Military Psychology. 2022; 34; 1: 33-46. 38536255</bibtext> </blist> <blist> <bibtext> Coyne, J. T, Draheim, C, Sibley, S, Armendariz, N, Melick, S, Foroughi, C, Burgoyne, A. & Engle, E. (2024) Evaluation of new measures of spatial ability and attention control for selection of naval flight students. In Paper presented at the Applied Human Factors and Ergonomics International, Honolulu, HI</bibtext> </blist> <blist> <bibtext> Damos, D. L. (2011). A summary of the technical pilot selection literature, AFCAPS-FR-2011-0009. Randolph AFB, TX: Air Force Personnel Center, Strategic Research and Assessment Branch.</bibtext> </blist> <blist> <bibtext> Draheim C, Mashburn CA, Martin JD, Engle RW. Reaction time in differential and developmental research: A review and commentary on the problems and alternatives. Psychological Bulletin. 2019; 145; 5: 508. 30896187</bibtext> </blist> <blist> <bibtext> Draheim C, Tsukahara JS, Martin JD, Mashburn CA, Engle RW. A toolbox approach to improving the measurement of attention control. Journal of Experimental Psychology: General. 2021; 150; 2: 242. 32700925</bibtext> </blist> <blist> <bibtext> Ekstrom RB, French JW, Harman HH, Dermen D. Manual for kit of factor-referenced cognitive tests. 1976; Educational Testing Service</bibtext> </blist> <blist> <bibtext> Gopher D, Kahneman D. Individual differences in attention and the prediction of flight criteria. Perceptual and Motor Skills. 1971; 33: 1335-1342. 5160058</bibtext> </blist> <blist> <bibtext> Government Accountability Office (GAO) (2018). Military Pilots: The View from 30,000 Feet. Retrieved December, 17, 2024 from https://<ulink href="http://www.gao.gov/blog/2018/08/16/military-pilots-the-view-from-30000-feet">www.gao.gov/blog/2018/08/16/military-pilots-the-view-from-30000-feet</ulink></bibtext> </blist> <blist> <bibtext> Hedge C, Powell G, Sumner P. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods. 2018; 50: 1166-1186. 10.3758/s13428-017-0935-1. 28726177</bibtext> </blist> <blist> <bibtext> Howard MC, Henderson J. A review of exploratory factor analysis in tourism and hospitality research: Identifying current practices and avenues for improvement. Journal of Business Research. 2023; 154113328</bibtext> </blist> <blist> <bibtext> Keiser, H. N, Moclaire, C. M, King, K. M, Brown, N. L, Foroughi, C. K, Sibley, C, & Coyne, J. T. (2019). Updating the direction orientation task: An aviation selection tool. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 63, No. 1, pp. 1414–1418). SAGE Publications.</bibtext> </blist> <blist> <bibtext> Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016; 15; 2: 155-163. 27330520. 4913118</bibtext> </blist> <blist> <bibtext> Lievens F, Reeve CL, Heggestad ED. An examination of psychometric bias due to retesting on cognitive ability tests in selection settings. Journal of Applied Psychology. 2007; 92; 6: 1672. 18020804</bibtext> </blist> <blist> <bibtext> Lohman, D. F. (1993). Spatial Ability and G. In Paper presented at the first spearman seminar. University of Plymouth</bibtext> </blist> <blist> <bibtext> Lorenzo-Seva U, Timmerman ME, Kiers HA. The hull method for selecting the number of common factors. Multivariate Behavioral Research. 2011; 46; 2: 340-364. 26741331</bibtext> </blist> <blist> <bibtext> MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor analysis. Psychological Methods. 1999; 4; 1: 84</bibtext> </blist> <blist> <bibtext> Momen N. The effects of alternative input devices and repeated exposures on the test of basic aviation skills (TBAS) performance. Military Medicine. 2009; 174; 12: 1282-1286. 20055069</bibtext> </blist> <blist> <bibtext> Naval Aerospace Medical Institute (NAMI), (2024) ASTB FAQ. Retrieved December, 17, 2024 from https://<ulink href="http://www.med.navy.mil/Navy-Medicine-Operational-Training-Command/Naval-Aerospace-Medical-Institute/ASTB-FAQ/">www.med.navy.mil/Navy-Medicine-Operational-Training-Command/Naval-Aerospace-Medical-Institute/ASTB-FAQ/</ulink></bibtext> </blist> <blist> <bibtext> National Research Council. Measuring human capabilities: An agenda for basic research on the assessment of individual and group performance potential for military accession. 2015; National Academy Press</bibtext> </blist> <blist> <bibtext> Peterson RA. A meta-analysis of variance accounted for and factor loadings in exploratory factor analysis. Marketing Letters. 2000; 11: 261-275</bibtext> </blist> <blist> <bibtext> Prasad-Rao J, Pinon Fischer OJ, Rowe NC, Williams JR, Puranik TG, Mavris DN, Natali MW, Tindall MJ, Atkinson BW. Attrition risk and aircraft suitability prediction in US Navy pilot training using machine learning. Aerospace. 2023; 10; 4379</bibtext> </blist> <blist> <bibtext> Raven J, Raven JMcCallum RS. Raven progressive matrices. Handbook of nonverbal assessment. 2003; Springer</bibtext> </blist> <blist> <bibtext> Ruscio J, Roche B. Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychological Assessment. 2012; 24; 2: 282. 21966933</bibtext> </blist> <blist> <bibtext> Shrout P, Lane SPMehl MR, Connor TS. Psychometrics. Handbook of research methods for studying daily life. 2012; Guilford Press: 302-320</bibtext> </blist> <blist> <bibtext> Sibley, C. (2024). High-stakes testing and second chances: from data to models (Order No. 31242034). Available from ProQuest Dissertations & Theses Global; Publicly Available Content Database. (3090764246). Retrieved February, 1, 2025. https://<ulink href="http://www.proquest.com/dissertations-theses/high-stakes-testing-second-chances-data-models/docview/3090764246/se-2">www.proquest.com/dissertations-theses/high-stakes-testing-second-chances-data-models/docview/3090764246/se-2</ulink></bibtext> </blist> <blist> <bibtext> Sovey S, Osman K, Mohd-Matore ME. Exploratory and confirmatory factor analysis for disposition levels of computational thinking instrument among secondary school students. European Journal of Educational Research. 2022; 11; 2: 639-652</bibtext> </blist> <blist> <bibtext> Walker, P. B, Olde, B. A, & Olson, T. M. (2007). Improving aviator selection using the performance-based measurement battery (PBMB). In Poster presented at the proceedings of the human systems integration symposium, Annapolis, Maryland. Retrieved from https://<ulink href="http://www.researchgate.net/publication/266910869%5fImproving%5fAviator%5fSelection%5fUsing%5fthe%5fPerformance-Based%5fMeasurement%5fBattery%5fPBMB">www.researchgate.net/publication/266910869%5fImproving%5fAviator%5fSelection%5fUsing%5fthe%5fPerformance-Based%5fMeasurement%5fBattery%5fPBMB</ulink></bibtext> </blist> <blist> <bibtext> Wright, M. N. & Ziegler, A. (2015). Ranger: A fast implementation of random forests for high dimensional data in c++ and r. arXiv:1508.04409</bibtext> </blist> </ref> <aug> <p>By Joseph T. Coyne; Laura Jamison; Kaylin Strong; Ciara Sibley; Cyrus Foroughi and Sarah Melick</p> <p>Reported by Author; Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib34" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib21" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib31" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib40" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib32" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib27" firstref="ref11"></nolink> <nolink nlid="nl7" bibid="bib16" firstref="ref13"></nolink> <nolink nlid="nl8" bibid="bib14" firstref="ref16"></nolink> <nolink nlid="nl9" bibid="bib30" firstref="ref17"></nolink> <nolink nlid="nl10" bibid="bib24" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib13" firstref="ref19"></nolink> <nolink nlid="nl12" bibid="bib15" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib20" firstref="ref21"></nolink> <nolink nlid="nl14" bibid="bib22" firstref="ref23"></nolink> <nolink nlid="nl15" bibid="bib17" firstref="ref24"></nolink> <nolink nlid="nl16" bibid="bib18" firstref="ref25"></nolink> <nolink nlid="nl17" bibid="bib10" firstref="ref32"></nolink> <nolink nlid="nl18" bibid="bib12" firstref="ref33"></nolink> <nolink nlid="nl19" bibid="bib19" firstref="ref35"></nolink> <nolink nlid="nl20" bibid="bib35" firstref="ref37"></nolink> <nolink nlid="nl21" bibid="bib11" firstref="ref38"></nolink> <nolink nlid="nl22" bibid="bib41" firstref="ref39"></nolink> <nolink nlid="nl23" bibid="bib37" firstref="ref40"></nolink> <nolink nlid="nl24" bibid="bib25" firstref="ref41"></nolink> <nolink nlid="nl25" bibid="bib28" firstref="ref43"></nolink> <nolink nlid="nl26" bibid="bib36" firstref="ref45"></nolink> <nolink nlid="nl27" bibid="bib23" firstref="ref46"></nolink> <nolink nlid="nl28" bibid="bib33" firstref="ref47"></nolink> <nolink nlid="nl29" bibid="bib29" firstref="ref49"></nolink> <nolink nlid="nl30" bibid="bib39" firstref="ref50"></nolink> <nolink nlid="nl31" bibid="bib26" firstref="ref54"></nolink> <nolink nlid="nl32" bibid="bib38" firstref="ref55"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1480870
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Joseph+T%2E+Coyne%22">Joseph T. Coyne</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-6461-0148">0000-0001-6461-0148</externalLink>)<br /><searchLink fieldCode="AR" term="%22Laura+Jamison%22">Laura Jamison</searchLink><br /><searchLink fieldCode="AR" term="%22Kaylin+Strong%22">Kaylin Strong</searchLink><br /><searchLink fieldCode="AR" term="%22Ciara+Sibley%22">Ciara Sibley</searchLink><br /><searchLink fieldCode="AR" term="%22Cyrus+Foroughi%22">Cyrus Foroughi</searchLink><br /><searchLink fieldCode="AR" term="%22Sarah+Melick%22">Sarah Melick</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Cognitive+Research%3A+Principles+and+Implications%22"><i>Cognitive Research: Principles and Implications</i></searchLink>. 2025 10.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 16
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Office of the Secretary of Defense (OSD) (DOD)<br />Office of Naval Research (ONR) (DOD)
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22High+Stakes+Tests%22">High Stakes Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Construct+Validity%22">Construct Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Military+Training%22">Military Training</searchLink><br /><searchLink fieldCode="DE" term="%22Aviation+Education%22">Aviation Education</searchLink><br /><searchLink fieldCode="DE" term="%22Spatial+Ability%22">Spatial Ability</searchLink><br /><searchLink fieldCode="DE" term="%22Attention%22">Attention</searchLink><br /><searchLink fieldCode="DE" term="%22Air+Transportation%22">Air Transportation</searchLink><br /><searchLink fieldCode="DE" term="%22Armed+Forces%22">Armed Forces</searchLink><br /><searchLink fieldCode="DE" term="%22Personnel+Selection%22">Personnel Selection</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1186/s41235-025-00660-3
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2365-7464
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This paper looks at how process-based spatial ability and attention measures taken within a high-stakes battery used to select pilots in the US Navy compare to lab-based measures of the same constructs. Process-based measures typically function by having individuals perform either a novel task or perform a task with novel stimuli. However, applicants often spend time practicing the tasks prior to taking the battery. A group of 307 Naval Flight Students participated in the study, in which they took several spatial ability, attention and general processing measures. One of the spatial tasks used in the study was the same as the spatial task in the Navy's pilot selection battery, which all of the participants had taken. All of the lab spatial ability measures including the one used in the selection battery were highly correlated and loaded onto the same spatial ability factor. However, the high-stakes spatial subtest was not correlated with any of the lab spatial measures including the same test administered in the lab. The lab spatial ability data was also correlated with training outcomes whereas the high-stakes process spatial and attention measures were not. The high-stakes attention measure was weakly correlated with some of the general processing measures. The pattern of results suggest that familiarity with the spatial and attention tasks in the high-stakes environment may be negating those tests ability to measure the constructs they were designed to measure, and also reducing their effectiveness to predict training performance.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1480870
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1480870
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1186/s41235-025-00660-3
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 16
    Subjects:
      – SubjectFull: High Stakes Tests
        Type: general
      – SubjectFull: Construct Validity
        Type: general
      – SubjectFull: Military Training
        Type: general
      – SubjectFull: Aviation Education
        Type: general
      – SubjectFull: Spatial Ability
        Type: general
      – SubjectFull: Attention
        Type: general
      – SubjectFull: Air Transportation
        Type: general
      – SubjectFull: Armed Forces
        Type: general
      – SubjectFull: Personnel Selection
        Type: general
    Titles:
      – TitleFull: Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Joseph T. Coyne
      – PersonEntity:
          Name:
            NameFull: Laura Jamison
      – PersonEntity:
          Name:
            NameFull: Kaylin Strong
      – PersonEntity:
          Name:
            NameFull: Ciara Sibley
      – PersonEntity:
          Name:
            NameFull: Cyrus Foroughi
      – PersonEntity:
          Name:
            NameFull: Sarah Melick
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2365-7464
          Numbering:
            – Type: volume
              Value: 10
          Titles:
            – TitleFull: Cognitive Research: Principles and Implications
              Type: main
ResultId 1