Pragmatic Social Measurement

This is a draft book manuscript about measuring social science concepts. This book aims to fill a niche in the range of social science methods texts that is (in the author’s opinion) both important and largely vacant. That niche is defined by two important distinctions: between measurement and other kinds of inference (both population and causal), and between pragmatic and representational measurement (Hand, 1996).

The vast majority of applied statistics texts have historically been focused on population inference: making claims about populations on the basis of samples from those populations. There is a growing collection of good applied texts on causal inference written in the last fifteen years. There is not, to my knowledge, even one applied text on the practical challenges of measurement as such, even though there are lots of texts that cover statistical models that are often used for measurement. There is one recent book on some of the conceptual issues with the kind of measurement I will be exploring (Goertz, 2020) as well as some older classics (eg Blalock, 1982).

Few applied statistics textbooks cover measurement at all, they tend to assume that the origin of the numbers which will be the object of analysis is a solved problem. Others cover representational measurement methods–eg survey sampling design or ecological inference–where we are aiming to quantitatively measure a well-defined attribute of the world, how many units are there of a given type, with more or less adequate data. What is missing is coverage of the methodological issues involved in pragmatic measurement, where we are to some extent inventing—or at least conceptualising—the target concept of measurement at the same time that we conduct the measurement. A great deal of social science measurement is of this type and it presents its own distinctive challenges. These are the subject of this book.

The set of existing textbooks with which the book most closely overlaps are those that cover multivariate summary methods (eg principle components analysis) and latent variable models (eg factor analysis). These are also a subject of this book, but are motivated differently than they are elsewhere. Here, latent variable models and related summary methods are pragmatic methods used for exploratory measurement in the absence of better data that would allow you to reliably define the target concept that you wish to measure. Ideally we would have relevant data that could more definitively link indicators to the concepts that we want to measure.

The first half of this book covers the various kinds of “supervision” that we can use to explicitly specify the connections between the target concepts we want to measure and the indicators that we have actually collected. In the second half, we turn to the “unsupervised” methods that are often used when researchers are a bit less clear about what they want to measure, or otherwise lack the data necessary to use supervised methods. Covering supervised measurement before unsupervised measurement makes it clearer that the “fancier” models involved in the latter replace substantive information about the relationships between indicators and the target concept that we wish to measure with the (often naive) hope that the target concept is the thing that maximises explained variation in the set of indicators.