View in EDS

How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828

Saved in:
Bibliographic Details
Title: How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828
Language: English
Authors: D. Betsy McCoach (ORCID 0000-0001-9063-6835), Anthony J. Gambino (ORCID 0000-0001-5820-1997), Scott J. Peters (ORCID 0000-0003-2459-3384), Daniel Long (ORCID 0000-0003-1887-2993), Del Siegle (ORCID 0000-0001-5579-9217), Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2023.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 158
Publication Date: 2023
Document Type: Reports - Research
Descriptors: Rating Scales, Student Evaluation, Academically Gifted, Ability Identification, Bias, Effect Size, Cutting Scores, Evaluative Thinking, Reliability, Cognitive Measurement, Intelligence Tests, Children
Assessment and Survey Identifiers: Wechsler Intelligence Scale for Children
Abstract: Teacher rating scales (TRS) are often used to make service eligibility decisions for exceptional learners. Although TRS are regularly used to identify student exceptionalism either as part of an informal nomination process or through behavioral rating scales, there is little research documenting the between-teacher variance in teacher ratings or the consequences of such rater dependence. To evaluate the possible benefits or disadvantages of using TRS as part of a gifted identification process, we examined the student-, teacher-, and school-level variance in TRS controlling for student ability and achievement to determine the unique information, consistency, and potential bias in TRS. Between 10% and 25% of a students' TRS score can be attributed to the teacher doing the rating, and between-teacher standard deviations represent an effect size of one-third to one-half standard deviation unit. Our results suggest that TRS are not easily comparable across teachers, making it impossible to set a cut score for admission into a program (or for further screening) that functions equitably across teachers.
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED672317
Database: ERIC
Description
Abstract:Teacher rating scales (TRS) are often used to make service eligibility decisions for exceptional learners. Although TRS are regularly used to identify student exceptionalism either as part of an informal nomination process or through behavioral rating scales, there is little research documenting the between-teacher variance in teacher ratings or the consequences of such rater dependence. To evaluate the possible benefits or disadvantages of using TRS as part of a gifted identification process, we examined the student-, teacher-, and school-level variance in TRS controlling for student ability and achievement to determine the unique information, consistency, and potential bias in TRS. Between 10% and 25% of a students' TRS score can be attributed to the teacher doing the rating, and between-teacher standard deviations represent an effect size of one-third to one-half standard deviation unit. Our results suggest that TRS are not easily comparable across teachers, making it impossible to set a cut score for admission into a program (or for further screening) that functions equitably across teachers.