Addressing Current Methodological Challenges in ILSA's Transition to Adaptive Testing
Saved in:
| Title: | Addressing Current Methodological Challenges in ILSA's Transition to Adaptive Testing |
|---|---|
| Language: | English |
| Authors: | Montserrat Beatriz Valdivia Medinaceli |
| Source: | ProQuest LLC. 2023Ph.D. Dissertation, Indiana University. |
| Availability: | ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com/en-US/products/dissertations/individuals.shtml |
| Peer Reviewed: | N |
| Page Count: | 177 |
| Publication Date: | 2023 |
| Document Type: | Dissertations/Theses - Doctoral Dissertations |
| Descriptors: | International Assessment, Achievement Tests, Adaptive Testing, Test Items, Test Bias, Test Construction, Item Response Theory, Error of Measurement |
| ISBN: | 979-83-8060-745-2 |
| Abstract: | My dissertation examines three current challenges of international large-scale assessments (ILSAs) associated with the transition from linear testing to an adaptive testing design. ILSAs are important for making comparisons among populations and informing countries about the quality of their educational systems. ILSA's results inform policymakers and foreign aid decisions, serve as a data source for secondary research, and inform the public, in general, about the state of educational systems around the world. However, as more developing economies participate in ILSAs, the cultural comparability of the test becomes a pressing design issue. Specifically, educational systems at the ends of the continuum are not well measured because items are either too difficult for low-performing countries or too easy for high-performing countries. Multistage testing (MST) and group adaptive testing (GAT) tailor test difficulty to examinees' proficiency. MST and GAT have been implemented to better measure proficiency at the ends of its continuum, especially for low- and high-performing countries. However, research gaps remain, given the challenges of implementing an adaptive design in ILSAs. My dissertation is a multiple-manuscript approach that addresses three separate and important challenges concerning ILSAs' measurement quality in the transition toward adaptive designs. The first paper aims to study the impact of item bias in country-level proficiency estimates when an MST is used. The second paper takes a critical stance in the transition to adaptive testing design when there is a mismatch between item difficulty and proficiency distribution of countries at the ends of the proficiency continuum. The third paper evaluates the performance of the root mean square deviation (RMSD) when the item local independence assumption of item response theory (IRT) models is violated. Findings reveal the significance of an adequate item pool that matches participants across the proficiency continuum. Effective routing mechanisms, like merit plus probabilistic misrouting, are essential, with bias-free items in early MST stages to mitigate bias effects on routing probabilities. Lastly, findings revealed that the RMSD cannot detect item bias when testlet effects are present. These insights serve as valuable guidance for practitioners and researchers aiming to enhance the fairness and comparability of ILSAs amid the growth of heterogeneous participants. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.] |
| Abstractor: | As Provided |
| Entry Date: | 2024 |
| Access URL: | https://gateway.proquest.com/openurl?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&res_dat=xri:pqm&rft_dat=xri:pqdiss:30687539 |
| Accession Number: | ED639715 |
| Database: | ERIC |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED639715 AccessLevel: 3 PubType: Dissertation/ Thesis PubTypeId: dissertation PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Addressing Current Methodological Challenges in ILSA's Transition to Adaptive Testing – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Montserrat+Beatriz+Valdivia+Medinaceli%22">Montserrat Beatriz Valdivia Medinaceli</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22ProQuest+LLC%22"><i>ProQuest LLC</i></searchLink>. 2023Ph.D. Dissertation, Indiana University. – Name: Avail Label: Availability Group: Avail Data: ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com/en-US/products/dissertations/individuals.shtml – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: N – Name: Pages Label: Page Count Group: Src Data: 177 – Name: DatePubCY Label: Publication Date Group: Date Data: 2023 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Dissertations/Theses - Doctoral Dissertations – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Adaptive+Testing%22">Adaptive Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink> – Name: ISBN Label: ISBN Group: ISBN Data: 979-83-8060-745-2 – Name: Abstract Label: Abstract Group: Ab Data: My dissertation examines three current challenges of international large-scale assessments (ILSAs) associated with the transition from linear testing to an adaptive testing design. ILSAs are important for making comparisons among populations and informing countries about the quality of their educational systems. ILSA's results inform policymakers and foreign aid decisions, serve as a data source for secondary research, and inform the public, in general, about the state of educational systems around the world. However, as more developing economies participate in ILSAs, the cultural comparability of the test becomes a pressing design issue. Specifically, educational systems at the ends of the continuum are not well measured because items are either too difficult for low-performing countries or too easy for high-performing countries. Multistage testing (MST) and group adaptive testing (GAT) tailor test difficulty to examinees' proficiency. MST and GAT have been implemented to better measure proficiency at the ends of its continuum, especially for low- and high-performing countries. However, research gaps remain, given the challenges of implementing an adaptive design in ILSAs. My dissertation is a multiple-manuscript approach that addresses three separate and important challenges concerning ILSAs' measurement quality in the transition toward adaptive designs. The first paper aims to study the impact of item bias in country-level proficiency estimates when an MST is used. The second paper takes a critical stance in the transition to adaptive testing design when there is a mismatch between item difficulty and proficiency distribution of countries at the ends of the proficiency continuum. The third paper evaluates the performance of the root mean square deviation (RMSD) when the item local independence assumption of item response theory (IRT) models is violated. Findings reveal the significance of an adequate item pool that matches participants across the proficiency continuum. Effective routing mechanisms, like merit plus probabilistic misrouting, are essential, with bias-free items in early MST stages to mitigate bias effects on routing probabilities. Lastly, findings revealed that the RMSD cannot detect item bias when testlet effects are present. These insights serve as valuable guidance for practitioners and researchers aiming to enhance the fairness and comparability of ILSAs amid the growth of heterogeneous participants. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.] – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2024 – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="https://gateway.proquest.com/openurl?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&res_dat=xri:pqm&rft_dat=xri:pqdiss:30687539" linkWindow="_blank">http://gateway.proquest.com/openurl?url_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:dissertation&res_dat=xri:pqm&rft_dat=xri:pqdiss:30687539</link> – Name: AN Label: Accession Number Group: ID Data: ED639715 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED639715 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 177 Subjects: – SubjectFull: International Assessment Type: general – SubjectFull: Achievement Tests Type: general – SubjectFull: Adaptive Testing Type: general – SubjectFull: Test Items Type: general – SubjectFull: Test Bias Type: general – SubjectFull: Test Construction Type: general – SubjectFull: Item Response Theory Type: general – SubjectFull: Error of Measurement Type: general Titles: – TitleFull: Addressing Current Methodological Challenges in ILSA's Transition to Adaptive Testing Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Montserrat Beatriz Valdivia Medinaceli IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2023 Identifiers: – Type: isbn-print Value: 979-83-8060-745-2 Titles: – TitleFull: ProQuest LLC Type: main |
| ResultId | 1 |
