View in EDS

Forced-Choice Ranking Models for Raters' Ranking Data

Saved in:
Bibliographic Details
Title: Forced-Choice Ranking Models for Raters' Ranking Data
Language: English
Authors: Hung, Su-Pin, Huang, Hung-Yu (ORCID 0000-0001-6244-1950)
Source: Journal of Educational and Behavioral Statistics. Oct 2022 47(5):603-634.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 32
Publication Date: 2022
Document Type: Journal Articles
Reports - Research
Descriptors: Evaluation Methods, Rating Scales, Item Analysis, Preferences, Responses, Evaluators, Decision Making, Models, Probability, Item Response Theory, Simulation, Sample Size, Creativity, Correlation
DOI: 10.3102/10769986221104207
ISSN: 1076-9986
1935-1054
Abstract: To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters' leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models.
Abstractor: As Provided
Entry Date: 2022
Accession Number: EJ1350511
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1350511
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Forced-Choice Ranking Models for Raters' Ranking Data
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hung%2C+Su-Pin%22">Hung, Su-Pin</searchLink><br /><searchLink fieldCode="AR" term="%22Huang%2C+Hung-Yu%22">Huang, Hung-Yu</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6244-1950">0000-0001-6244-1950</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+and+Behavioral+Statistics%22"><i>Journal of Educational and Behavioral Statistics</i></searchLink>. Oct 2022 47(5):603-634.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 32
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Analysis%22">Item Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Preferences%22">Preferences</searchLink><br /><searchLink fieldCode="DE" term="%22Responses%22">Responses</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Probability%22">Probability</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Response+Theory%22">Item Response Theory</searchLink><br /><searchLink fieldCode="DE" term="%22Simulation%22">Simulation</searchLink><br /><searchLink fieldCode="DE" term="%22Sample+Size%22">Sample Size</searchLink><br /><searchLink fieldCode="DE" term="%22Creativity%22">Creativity</searchLink><br /><searchLink fieldCode="DE" term="%22Correlation%22">Correlation</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.3102/10769986221104207
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1076-9986<br />1935-1054
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters' leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2022
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1350511
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1350511
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.3102/10769986221104207
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 603
    Subjects:
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Rating Scales
        Type: general
      – SubjectFull: Item Analysis
        Type: general
      – SubjectFull: Preferences
        Type: general
      – SubjectFull: Responses
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Probability
        Type: general
      – SubjectFull: Item Response Theory
        Type: general
      – SubjectFull: Simulation
        Type: general
      – SubjectFull: Sample Size
        Type: general
      – SubjectFull: Creativity
        Type: general
      – SubjectFull: Correlation
        Type: general
    Titles:
      – TitleFull: Forced-Choice Ranking Models for Raters' Ranking Data
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hung, Su-Pin
      – PersonEntity:
          Name:
            NameFull: Huang, Hung-Yu
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 10
              Type: published
              Y: 2022
          Identifiers:
            – Type: issn-print
              Value: 1076-9986
            – Type: issn-electronic
              Value: 1935-1054
          Numbering:
            – Type: volume
              Value: 47
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Journal of Educational and Behavioral Statistics
              Type: main
ResultId 1