Forced-Choice Ranking Models for Raters' Ranking Data
Saved in:
| Title: | Forced-Choice Ranking Models for Raters' Ranking Data |
|---|---|
| Language: | English |
| Authors: | Hung, Su-Pin, Huang, Hung-Yu (ORCID |
| Source: | Journal of Educational and Behavioral Statistics. Oct 2022 47(5):603-634. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 32 |
| Publication Date: | 2022 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Evaluation Methods, Rating Scales, Item Analysis, Preferences, Responses, Evaluators, Decision Making, Models, Probability, Item Response Theory, Simulation, Sample Size, Creativity, Correlation |
| DOI: | 10.3102/10769986221104207 |
| ISSN: | 1076-9986 1935-1054 |
| Abstract: | To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters' leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models. |
| Abstractor: | As Provided |
| Entry Date: | 2022 |
| Accession Number: | EJ1350511 |
| Database: | ERIC |
| Abstract: | To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters' leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models. |
|---|---|
| ISSN: | 1076-9986 1935-1054 |
| DOI: | 10.3102/10769986221104207 |
