View in EDS

Bias Risks in ILSA Related to Non-Participation: Evidence from a Longitudinal Large-Scale Survey in Germany (PISA Plus)

Saved in:
Bibliographic Details
Title: Bias Risks in ILSA Related to Non-Participation: Evidence from a Longitudinal Large-Scale Survey in Germany (PISA Plus)
Language: English
Authors: Sabine Meinck (ORCID 0000-0003-2261-4751), Jörg-Henrik Heine, Julia Mang, Gabriel Nagy
Source: Educational Assessment, Evaluation and Accountability. 2024 36(3):385-404.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 20
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Bias, Risk, Foreign Countries, International Assessment, Achievement Tests, Secondary School Students, Measurement, Student Participation
Geographic Terms: Germany
Assessment and Survey Identifiers: Program for International Student Assessment
DOI: 10.1007/s11092-023-09422-5
ISSN: 1874-8597
Abstract: This study uses evidence from a longitudinal survey (PISA Plus, Germany) to examine the potential of bias in international large-scale assessments (ILSAs). In PISA Plus, participation was mandatory at the first measurement point, but voluntary at the second measurement point. The study provides evidence for relevant selection bias regarding student competencies and background variables when participation is voluntary. Sample dropout at the second measurement point was related to characteristics such as family background, achievement in mathematics, reading and science, and other student and school demographic variables at both the student and school levels, with lower performing students and those with less favorable background characteristics having higher dropout frequencies, from which higher dropout probabilities of such students can be inferred. We further contrast the possibilities for addressing non-response through weight adjustments in longitudinal surveys with those in cross-sectional surveys. Considering our results, we evaluate and confirm the validity and appropriateness of strict participation rate requirements in ILSAs. Likely magnitudes of bias in cross-sectional studies in varying scenarios are illustrated. Accordingly, if combined participation rates drop below 70%, a difference of at least one-fifth of a standard deviation in an achievement score between non-respondents and participants leads to relevant bias. When participation drops below 50%, even a very small difference (one-tenth of a standard deviation) will cause non-negligible bias. Finally, we conclude that the stringent participation rate requirements established in most ILSAs are fully valid, reasonable, and important since they ensure a relatively low risk of biased results.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1438351
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1438351
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Bias Risks in ILSA Related to Non-Participation: Evidence from a Longitudinal Large-Scale Survey in Germany (PISA Plus)
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Sabine+Meinck%22">Sabine Meinck</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-2261-4751">0000-0003-2261-4751</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jörg-Henrik+Heine%22">Jörg-Henrik Heine</searchLink><br /><searchLink fieldCode="AR" term="%22Julia+Mang%22">Julia Mang</searchLink><br /><searchLink fieldCode="AR" term="%22Gabriel+Nagy%22">Gabriel Nagy</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Assessment%2C+Evaluation+and+Accountability%22"><i>Educational Assessment, Evaluation and Accountability</i></searchLink>. 2024 36(3):385-404.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 20
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Risk%22">Risk</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink><br /><searchLink fieldCode="DE" term="%22International+Assessment%22">International Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Tests%22">Achievement Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Students%22">Secondary School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Measurement%22">Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Participation%22">Student Participation</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Germany%22">Germany</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Program+for+International+Student+Assessment%22">Program for International Student Assessment</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11092-023-09422-5
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1874-8597
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study uses evidence from a longitudinal survey (PISA Plus, Germany) to examine the potential of bias in international large-scale assessments (ILSAs). In PISA Plus, participation was mandatory at the first measurement point, but voluntary at the second measurement point. The study provides evidence for relevant selection bias regarding student competencies and background variables when participation is voluntary. Sample dropout at the second measurement point was related to characteristics such as family background, achievement in mathematics, reading and science, and other student and school demographic variables at both the student and school levels, with lower performing students and those with less favorable background characteristics having higher dropout frequencies, from which higher dropout probabilities of such students can be inferred. We further contrast the possibilities for addressing non-response through weight adjustments in longitudinal surveys with those in cross-sectional surveys. Considering our results, we evaluate and confirm the validity and appropriateness of strict participation rate requirements in ILSAs. Likely magnitudes of bias in cross-sectional studies in varying scenarios are illustrated. Accordingly, if combined participation rates drop below 70%, a difference of at least one-fifth of a standard deviation in an achievement score between non-respondents and participants leads to relevant bias. When participation drops below 50%, even a very small difference (one-tenth of a standard deviation) will cause non-negligible bias. Finally, we conclude that the stringent participation rate requirements established in most ILSAs are fully valid, reasonable, and important since they ensure a relatively low risk of biased results.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1438351
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1438351
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11092-023-09422-5
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 20
        StartPage: 385
    Subjects:
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Risk
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: International Assessment
        Type: general
      – SubjectFull: Achievement Tests
        Type: general
      – SubjectFull: Secondary School Students
        Type: general
      – SubjectFull: Measurement
        Type: general
      – SubjectFull: Student Participation
        Type: general
      – SubjectFull: Germany
        Type: general
      – SubjectFull: Program for International Student Assessment
        Type: general
    Titles:
      – TitleFull: Bias Risks in ILSA Related to Non-Participation: Evidence from a Longitudinal Large-Scale Survey in Germany (PISA Plus)
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Sabine Meinck
      – PersonEntity:
          Name:
            NameFull: Jörg-Henrik Heine
      – PersonEntity:
          Name:
            NameFull: Julia Mang
      – PersonEntity:
          Name:
            NameFull: Gabriel Nagy
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 1874-8597
          Numbering:
            – Type: volume
              Value: 36
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Educational Assessment, Evaluation and Accountability
              Type: main
ResultId 1