UC Berkeley

This dissertation encompasses three papers that touch on the topics of the definition of measurement, the possibility of ordinal measurement and the application of an ordinal psychometric model.

The first paper, A Pragmatic Perspective on Measurement, addresses the issue of the definition of measurement, presenting a conceptualization of the practice of measurement from the perspective of the Pragmatic tradition in philosophy (Bacon, 2012; James, 1907/1995; Rorty, 1999). In the spirit that Pragmatic approach, this definition is put forward as a tool aimed at gauging measurement claims in terms of their usefulness, such that it can contribute to a better understanding between researchers, practitioners and users of measurement. The paper discusses central ideas of the Pragmatic tradition and reviews the main measurement traditions in the social sciences before making a case for a Pragmatic Perspective of Measurement and ordering a definition of measurement based on that approach. The proposed definition attempts to achieve this by bringing to the foreground a conception of measurement according to which: (a) order and classification are part of measurement as well as quantification, (b) the model of the attribute that underlies a measurement is central to designing and interpreting measurements, (c) attributes need not be considered as natural kinds or universals, and (d) the purpose for which the measurement was developed informs us about the scope of its utility, both to judge its success as well as limiting the inferences that can be supported by it.

The second paper, Categorization, Ordering and Quantification: Selecting a Latent Variable Model by Comparing Latent Structures, is a joint work with Ronli Diakow that proposes a model selection framework for identifying the kind of latent structure— classificatory, ordinal, or quantitative—that best describes a dataset. The framework and its rationale for successive comparison of models outlined in this paper offers a blue-print to directly addressing issues that so far are largely thought to only be examinable under the representational theory of measurement, namely, the empirical identification of ordinal and quantitative structure in an attribute. The possibility of analyzing them under a latent variable framework would allow the critical examination of the assumptions regarding the latent structure that can be supported based on the data and, more generally, to question and revise our assumptions regarding the structure that we ascribe to the relevant attributes that we study.

The last paper builds on the previous papers—which present the arguments that measurement can be ordinal, and that it is possible to identify cases when a model that assumes an ordinal latent structure is better suited to a dataset— by introducing The Ordered Mixture Linear Logistic Test Model (OM-LLTM), an explanatory item response theory model conceived for ordinal measurement. The OM-LLTM is a model suited to take advantage of the cases when we have a theory that describes a relevant attribute in terms of a set of ordered performance levels, and we construct our assessment instruments according to that theory. The OM-LLTM assumes respondents are grouped in ordered latent classes where the probability of correctly answering an assessment task is a function not only of the class membership of the respondent, but also of item features that—according to the theory—determine the difficulty of the task. This model is a combination of the Linear Logistic Test Model (LLTM; Fischer, 1973) and Ordered Latent Class Analysis (OLCA; Croon, 1990). The OM-LLTM can also be considered an ordered extension of the mixture LLTM developed by Mislevy and Verhelst (1990).

The combination of these two models will allow researchers and practitioners to model student proficiency according to explanatory (De Boeck & Wilson, 2004) models expressed through the LLTM part of the model, while providing simple and interpretable results in terms of ranked performance groups, through the OLCA part of the model. Accordingly, the OM-LLTM offers both a simple, coarser, interpretation of the respondent classes according to overall proficiency and also an explanatory interpretation in terms of the specific item features; where the former interpretation lends itself for use in context where summative assessments are needed and the latter is more appropriate when diagnostic information is required.