An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs
Saved in:
| Title: | An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs |
|---|---|
| Language: | English |
| Authors: | Joyce M. W. Moonen-van Loon, Jeroen Donkers (ORCID |
| Source: | Practical Assessment, Research & Evaluation. 2025 30. |
| Availability: | University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/ |
| Peer Reviewed: | Y |
| Page Count: | 18 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Programming Languages, Reliability, Evaluation Methods, Student Evaluation, Generalization, Graphs, Error of Measurement, Work Environment, Multivariate Analysis |
| ISSN: | 1531-7714 |
| Abstract: | The reliability of assessment tools is critical for accurately monitoring student performance in various educational contexts. When multiple assessments are combined to form an overall evaluation, each assessment serves as a data point contributing to the student's performance within a broader educational framework. Determining composite reliability in such cases can be complex, particularly in naturalistic, unbalanced datasets in nested designs, which are common in programmatic and workplace-based assessments, where students are evaluated on unique, practical occasions. This paper introduces the compositeReliabilityInNestedDesigns package in R, designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package produces extensive Generalizability and Decision study results with graphical interpretations. Composite reliability incorporates weights and covariance to integrate results across assessment tools. Weights can be optimized to minimize standard error of measurement or maximize reliability. Overall, the package's flexible use and optimization empowers assessment tailoring and robust insights into student performance. The approach is suitable for programmatic assessment. The package facilitates reliable, comprehensive evaluation across diverse assessments. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1478205 |
| Database: | ERIC |
| Abstract: | The reliability of assessment tools is critical for accurately monitoring student performance in various educational contexts. When multiple assessments are combined to form an overall evaluation, each assessment serves as a data point contributing to the student's performance within a broader educational framework. Determining composite reliability in such cases can be complex, particularly in naturalistic, unbalanced datasets in nested designs, which are common in programmatic and workplace-based assessments, where students are evaluated on unique, practical occasions. This paper introduces the compositeReliabilityInNestedDesigns package in R, designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package produces extensive Generalizability and Decision study results with graphical interpretations. Composite reliability incorporates weights and covariance to integrate results across assessment tools. Weights can be optimized to minimize standard error of measurement or maximize reliability. Overall, the package's flexible use and optimization empowers assessment tailoring and robust insights into student performance. The approach is suitable for programmatic assessment. The package facilitates reliable, comprehensive evaluation across diverse assessments. |
|---|---|
| ISSN: | 1531-7714 |
