Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection
Saved in:
| Title: | Process-Based Measures in High-Stakes Testing: Practical Implications for Construct Validity within Military Aviation Selection |
|---|---|
| Language: | English |
| Authors: | Joseph T. Coyne (ORCID |
| Source: | Cognitive Research: Principles and Implications. 2025 10. |
| Availability: | Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/ |
| Peer Reviewed: | Y |
| Page Count: | 16 |
| Publication Date: | 2025 |
| Sponsoring Agency: | Office of the Secretary of Defense (OSD) (DOD) Office of Naval Research (ONR) (DOD) |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | High Stakes Tests, Construct Validity, Military Training, Aviation Education, Spatial Ability, Attention, Air Transportation, Armed Forces, Personnel Selection |
| DOI: | 10.1186/s41235-025-00660-3 |
| ISSN: | 2365-7464 |
| Abstract: | This paper looks at how process-based spatial ability and attention measures taken within a high-stakes battery used to select pilots in the US Navy compare to lab-based measures of the same constructs. Process-based measures typically function by having individuals perform either a novel task or perform a task with novel stimuli. However, applicants often spend time practicing the tasks prior to taking the battery. A group of 307 Naval Flight Students participated in the study, in which they took several spatial ability, attention and general processing measures. One of the spatial tasks used in the study was the same as the spatial task in the Navy's pilot selection battery, which all of the participants had taken. All of the lab spatial ability measures including the one used in the selection battery were highly correlated and loaded onto the same spatial ability factor. However, the high-stakes spatial subtest was not correlated with any of the lab spatial measures including the same test administered in the lab. The lab spatial ability data was also correlated with training outcomes whereas the high-stakes process spatial and attention measures were not. The high-stakes attention measure was weakly correlated with some of the general processing measures. The pattern of results suggest that familiarity with the spatial and attention tasks in the high-stakes environment may be negating those tests ability to measure the constructs they were designed to measure, and also reducing their effectiveness to predict training performance. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1480870 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | This paper looks at how process-based spatial ability and attention measures taken within a high-stakes battery used to select pilots in the US Navy compare to lab-based measures of the same constructs. Process-based measures typically function by having individuals perform either a novel task or perform a task with novel stimuli. However, applicants often spend time practicing the tasks prior to taking the battery. A group of 307 Naval Flight Students participated in the study, in which they took several spatial ability, attention and general processing measures. One of the spatial tasks used in the study was the same as the spatial task in the Navy's pilot selection battery, which all of the participants had taken. All of the lab spatial ability measures including the one used in the selection battery were highly correlated and loaded onto the same spatial ability factor. However, the high-stakes spatial subtest was not correlated with any of the lab spatial measures including the same test administered in the lab. The lab spatial ability data was also correlated with training outcomes whereas the high-stakes process spatial and attention measures were not. The high-stakes attention measure was weakly correlated with some of the general processing measures. The pattern of results suggest that familiarity with the spatial and attention tasks in the high-stakes environment may be negating those tests ability to measure the constructs they were designed to measure, and also reducing their effectiveness to predict training performance. |
|---|---|
| ISSN: | 2365-7464 |
| DOI: | 10.1186/s41235-025-00660-3 |
