A Monte Carlo Simulation Investigating the Validity and Reliability of Ability Estimation in Item Response Theory with Speeded Computer Adaptive Tests
Saved in:
| Title: | A Monte Carlo Simulation Investigating the Validity and Reliability of Ability Estimation in Item Response Theory with Speeded Computer Adaptive Tests |
|---|---|
| Language: | English |
| Authors: | Schmitt, T. A., Sass, D. A., Sullivan, J. R., Walker, C. M. |
| Source: | International Journal of Testing. 2010 10(3):230-261. |
| Availability: | Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 32 |
| Publication Date: | 2010 |
| Document Type: | Journal Articles Reports - Evaluative |
| Descriptors: | Monte Carlo Methods, Simulation, Computer Assisted Testing, Adaptive Testing, Timed Tests, Item Response Theory, Validity, Reliability, Computation, Ability, Bias, Error of Measurement |
| DOI: | 10.1080/15305058.2010.488098 |
| ISSN: | 1530-5058 |
| Abstract: | Imposed time limits on computer adaptive tests (CATs) can result in examinees having difficulty completing all items, thus compromising the validity and reliability of ability estimates. In this study, the effects of speededness were explored in a simulated CAT environment by varying examinee response patterns to end-of-test items. Expectedly, ability estimates became increasingly negatively biased as the CAT became more speeded, with the bias magnitude depending on the speededness condition. Ability estimates for higher-ability examinees were also influenced more than estimates for low- and middle-ability examinees with the realistic item pool. This finding is likely linked to test operational characteristics since these results were not replicated with an ideal item pool. In any case, the CAT is relatively robust to the speededness assumption, assuming that only a few items are speeded. Results also suggested that if test developers score the test at the point of speededness, ability estimates remain unbiased. This study's findings help inform test developers and test takers about the effects of test speededness within a CAT environment. Additional implications for educators and researchers are discussed. (Contains 2 tables, 3 figures and 2 notes.) |
| Abstractor: | As Provided |
| Number of References: | 89 |
| Entry Date: | 2010 |
| Accession Number: | EJ893543 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Imposed time limits on computer adaptive tests (CATs) can result in examinees having difficulty completing all items, thus compromising the validity and reliability of ability estimates. In this study, the effects of speededness were explored in a simulated CAT environment by varying examinee response patterns to end-of-test items. Expectedly, ability estimates became increasingly negatively biased as the CAT became more speeded, with the bias magnitude depending on the speededness condition. Ability estimates for higher-ability examinees were also influenced more than estimates for low- and middle-ability examinees with the realistic item pool. This finding is likely linked to test operational characteristics since these results were not replicated with an ideal item pool. In any case, the CAT is relatively robust to the speededness assumption, assuming that only a few items are speeded. Results also suggested that if test developers score the test at the point of speededness, ability estimates remain unbiased. This study's findings help inform test developers and test takers about the effects of test speededness within a CAT environment. Additional implications for educators and researchers are discussed. (Contains 2 tables, 3 figures and 2 notes.) |
|---|---|
| ISSN: | 1530-5058 |
| DOI: | 10.1080/15305058.2010.488098 |
