UC Irvine

This dissertation is composed of projects on three aspects of gathering and learning from data in the social sciences: drawing representative samples, taking valid measurement, and making warranted inductive inferences. Chapter one studies the challenge of drawing representative human samples. It is well documented that most samples used by studies in psychology-related fields are composed of Euro-American undergraduate students. Most writers agree that this is a serious problem for the generalizability of study results, but little improvement has occurred. By tracing the history of sampling, I identify the scientific and statistical rationale of sampling as a method of induction. I explain how the design-based approach, where to sample representatively is to sample randomly, became dominant. I show that this approach faces too many practical challenges within the social sciences to be useful as a guiding framework. In its place, I argue that the model-based approach, initially disfavoured for its theoretical shortcomings, is a better framework for the social sciences, because it allows the systematic integration of multiple imperfect samples. Instead of relying on one general framework to provide `in-principle' justifications to all cases, the model-based approach allows context and background knowledge to inform practice. Chapter two discusses measurement validity in the social sciences. Through an examination of the historical evolution of measurement and validity theories and the relationship between them, I argue that we should reject the view that measurement should be about an attribute that exists in the world in some robust sense, and that a pronouncement of measurement validity is a vindication of such an existence. First, I argue that this view, while attractive, has numerous theoretical difficulties and practical limitations. Next, I show that a rejection of this view, exemplified by the modern argument-based approach validity, presents a better perspetive in analyzing complex measurement problems in non-laboratory contexts. I conclude by pointing out that, based on the argument advanced in this paper, we should be more skeptical of ontological claims made on the basis of valid testing alone. Chapter three studies the problem of induction in the context of statistical learning theory. I examine a claim in the literature that the Vapnik-Chervonenkis (VC) theorem, which specifies conditions under which a problem is machine-learnable, offers a response to the problem of induction. I prove that the problem of when this learnability condition applies in general is uncomputable. Hence this solution strategy fails. If statistical learning theory is trustworthy at all, the justification of this trust must be in parallel with other inductive methodologies and, consequently, subject to the same challenges. I conclude this dissertation by arguing that a naturalistic, practice-first approach to philosophy of social science must pay attention not only to how a scientific method works in theory, but also to how it has been changed to accommodate resource-limited contexts.