UC Berkeley

Motivated by the recent discourses on approaches to combine generalizability theory (GT) and item response theory (IRT), this study suggests an approach that answers to some of the issues raised by previous research on combining GT and IRT. The main idea of the proposed approach is to recognize that IRT models can be written in terms of a latent continuous response and that a classic IRT model can be modeled directly using standard GT with items as a fixed facet. Once this is recognized, treating items as a random facet or considering other facets such as raters, become relatively straightforward extensions. The resulting models are logistic mixed models that contain the parameter of interest: the variance components needed for the generalizability and dependability coefficients. The models can be estimated by straightforward ML estimation. Extensive simulation studies were conducted to evaluate the performance of the proposed approach under various measurement situations. The results suggested that the proposed approach gives overall accurate results, in particular, when estimating the generalizability coefficients. The use of the proposed method is illustrated with large-scale data sets from classroom assessment (the Carbon Cycle 2008-2009 data set) and standardized tests (the PISA 2009 U.S. Science data set). The empirical results were presented and interpreted in the context of the G study and the D study in GT as well as a Wright Map in IRT. The results demonstrated the flexibility of the proposed approach with respect to incorporating extra complications in measurement situations (e.g., multidimensionality, polytomous responses) and further explanatory variables (e.g., rater facet).