View in EDS

Improving the Design of Evaluations That Include Students, Teachers, and Schools: An Empirical Investigation of Key Design Parameters

Saved in:
Bibliographic Details
Title: Improving the Design of Evaluations That Include Students, Teachers, and Schools: An Empirical Investigation of Key Design Parameters
Language: English
Authors: Dea Mulolli (ORCID 0000-0001-9345-8587), E. C. Hedberg (ORCID 0000-0003-0679-0720), Megan Bogia (ORCID 0000-0002-0177-3721), Jessaca Spybrook (ORCID 0000-0003-1768-6187), Tiffany Berglund, Fatih Unlu, Isaac M. Opper (ORCID 0000-0002-2554-6646)
Source: AERA Open. 2025 11(1).
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Peer Reviewed: Y
Page Count: 20
Publication Date: 2025
Sponsoring Agency: National Science Foundation (NSF)
Contract Number: 2000388
Document Type: Journal Articles
Reports - Research
Education Level: Early Childhood Education
Elementary Education
Grade 3
Primary Education
Grade 4
Intermediate Grades
Grade 5
Middle Schools
Grade 6
Grade 7
Junior High Schools
Secondary Education
Grade 8
Descriptors: Grade 3, Grade 4, Grade 5, Grade 6, Grade 7, Grade 8, Educational Environment, Demography, Reading Achievement, Mathematics Achievement, Science Achievement, Statistical Analysis, Evaluation Methods, Environmental Influences, Material Development, Randomized Controlled Trials, Students, Teachers, Schools
Geographic Terms: North Carolina, Kentucky, Michigan, Maryland
ISSN: 2332-8584
Abstract: The learning environment is a complex environment for conducting evaluations. Students, teachers, and schools all play a role in shaping the environment, which can lead to differences in student achievement. Hence designing an evaluation with the capacity to detect the effectiveness of an educational intervention on student achievement requires careful planning. A key part of the planning is a statistical power analysis that considers the multilevel nature of the design. This study provides empirical estimates of design parameters necessary for planning adequately powered cluster randomized trials that include the student, teacher, and school level and are focused on reading, mathematics, or science achievement. The sample in our study includes administrative state datasets from Michigan, North Carolina, Kentucky, and Maryland for grades 3 through 8. The results showed that, with few exceptions, the variance in student test scores is larger between teachers within schools than between schools.
Abstractor: As Provided
Notes: https://www.openicpsr.org/openicpsr/project/215781/version/V1/view
Entry Date: 2026
Accession Number: EJ1494762
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1494762
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1494762
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Improving the Design of Evaluations That Include Students, Teachers, and Schools: An Empirical Investigation of Key Design Parameters
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Dea+Mulolli%22">Dea Mulolli</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9345-8587">0000-0001-9345-8587</externalLink>)<br /><searchLink fieldCode="AR" term="%22E%2E+C%2E+Hedberg%22">E. C. Hedberg</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0679-0720">0000-0003-0679-0720</externalLink>)<br /><searchLink fieldCode="AR" term="%22Megan+Bogia%22">Megan Bogia</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-0177-3721">0000-0002-0177-3721</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jessaca+Spybrook%22">Jessaca Spybrook</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1768-6187">0000-0003-1768-6187</externalLink>)<br /><searchLink fieldCode="AR" term="%22Tiffany+Berglund%22">Tiffany Berglund</searchLink><br /><searchLink fieldCode="AR" term="%22Fatih+Unlu%22">Fatih Unlu</searchLink><br /><searchLink fieldCode="AR" term="%22Isaac+M%2E+Opper%22">Isaac M. Opper</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2554-6646">0000-0002-2554-6646</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22AERA+Open%22"><i>AERA Open</i></searchLink>. 2025 11(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 20
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 2000388
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+3%22">Grade 3</searchLink><br /><searchLink fieldCode="EL" term="%22Primary+Education%22">Primary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="EL" term="%22Intermediate+Grades%22">Intermediate Grades</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+8%22">Grade 8</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Grade+3%22">Grade 3</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+4%22">Grade 4</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+5%22">Grade 5</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+6%22">Grade 6</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+7%22">Grade 7</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+8%22">Grade 8</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Environment%22">Educational Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Demography%22">Demography</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Achievement%22">Reading Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Achievement%22">Mathematics Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Achievement%22">Science Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Analysis%22">Statistical Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Environmental+Influences%22">Environmental Influences</searchLink><br /><searchLink fieldCode="DE" term="%22Material+Development%22">Material Development</searchLink><br /><searchLink fieldCode="DE" term="%22Randomized+Controlled+Trials%22">Randomized Controlled Trials</searchLink><br /><searchLink fieldCode="DE" term="%22Students%22">Students</searchLink><br /><searchLink fieldCode="DE" term="%22Teachers%22">Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Schools%22">Schools</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22North+Carolina%22">North Carolina</searchLink><br /><searchLink fieldCode="DE" term="%22Kentucky%22">Kentucky</searchLink><br /><searchLink fieldCode="DE" term="%22Michigan%22">Michigan</searchLink><br /><searchLink fieldCode="DE" term="%22Maryland%22">Maryland</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2332-8584
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The learning environment is a complex environment for conducting evaluations. Students, teachers, and schools all play a role in shaping the environment, which can lead to differences in student achievement. Hence designing an evaluation with the capacity to detect the effectiveness of an educational intervention on student achievement requires careful planning. A key part of the planning is a statistical power analysis that considers the multilevel nature of the design. This study provides empirical estimates of design parameters necessary for planning adequately powered cluster randomized trials that include the student, teacher, and school level and are focused on reading, mathematics, or science achievement. The sample in our study includes administrative state datasets from Michigan, North Carolina, Kentucky, and Maryland for grades 3 through 8. The results showed that, with few exceptions, the variance in student test scores is larger between teachers within schools than between schools.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://www.openicpsr.org/openicpsr/project/215781/version/V1/view
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1494762
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1494762
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
    Subjects:
      – SubjectFull: Grade 3
        Type: general
      – SubjectFull: Grade 4
        Type: general
      – SubjectFull: Grade 5
        Type: general
      – SubjectFull: Grade 6
        Type: general
      – SubjectFull: Grade 7
        Type: general
      – SubjectFull: Grade 8
        Type: general
      – SubjectFull: Educational Environment
        Type: general
      – SubjectFull: Demography
        Type: general
      – SubjectFull: Reading Achievement
        Type: general
      – SubjectFull: Mathematics Achievement
        Type: general
      – SubjectFull: Science Achievement
        Type: general
      – SubjectFull: Statistical Analysis
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Environmental Influences
        Type: general
      – SubjectFull: Material Development
        Type: general
      – SubjectFull: Randomized Controlled Trials
        Type: general
      – SubjectFull: Students
        Type: general
      – SubjectFull: Teachers
        Type: general
      – SubjectFull: Schools
        Type: general
      – SubjectFull: North Carolina
        Type: general
      – SubjectFull: Kentucky
        Type: general
      – SubjectFull: Michigan
        Type: general
      – SubjectFull: Maryland
        Type: general
    Titles:
      – TitleFull: Improving the Design of Evaluations That Include Students, Teachers, and Schools: An Empirical Investigation of Key Design Parameters
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Dea Mulolli
      – PersonEntity:
          Name:
            NameFull: E. C. Hedberg
      – PersonEntity:
          Name:
            NameFull: Megan Bogia
      – PersonEntity:
          Name:
            NameFull: Jessaca Spybrook
      – PersonEntity:
          Name:
            NameFull: Tiffany Berglund
      – PersonEntity:
          Name:
            NameFull: Fatih Unlu
      – PersonEntity:
          Name:
            NameFull: Isaac M. Opper
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2332-8584
          Numbering:
            – Type: volume
              Value: 11
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: AERA Open
              Type: main
ResultId 1