The importance of measurement in psychology

Gunter Trendler (whose work on measurement in psychology we discussed last year) writes:

I’m aware that theoretical work on measurement is perceived by most psychologists as merely “philosophical” and therefore ignored.

Recently I’ve stumbled over a publication [Measuring Unobserved Variables in Educational Inequality Research: Mathematics and Language ‘Abilities’ in Early Childhood, by Alejandra Rodriguez] which investigates the problem of measuring “unobservable variables” empirically and may therefore attract more attention than purely theoretical argumentation.

I’ve also attached a paper from 2008 [Rationalization and Cognitive Dissonance: Do Choices Affect or Reflect Preferences?, by Keith Chen] that I’ve recently read and which, I believe, illustrates quite well what consequences ignoring nominal measurement may have.

I don’t have anything to add here. I glanced at the two linked articles but have not tried to read them in detail. I’m posting here because I think measurement in statistics is underrated so it’s good to be reminded of its importance.

12 thoughts on “The importance of measurement in psychology

    • On page 13 they write

      ” . . . Stevens’ rules prohibit calculations of means, variances and correlations with nominal variables . . . If we take this prohibition seriously, Zumbo and Kroc note that this means that one could not use a binary predictor in a linear regression model, within which the mean and variance of the nominal variable are intermediate calculations.”

      I think this is confusing the model — linear regression with binary predictor — with the method of calculating estimates. I think what Zumbo and Kroc allude to are the basic analytical solutions for minimizing the sum of squares[1]. But if I write down the likelihood function for the model e.g. in R (or in Stan) and numerically find the maximum, means and/or variances of the predictor variable are never calculated.

      To present the rest of the argument (that the writer in the essay is making), they continue

      “However, one could use a binary predictor in a Student’s t test (which is calculated in such a way as to not require these intermediate calculations). And yet, a Student’s t test is directly equivalent to a simple linear regression with a binary predictor . . . To prohibit one procedure and not the other is clearly nonsensical.”

      I think this is only nonsensical if one considers the specific method of calculating the estimates. So I don’t think it really follows that one couldn’t use a binary predictor variable in linear regression and would have to instead use the t-test. Perhaps slavish following of Steven’s rules would lead to one not being able to use the aforementioned analytical solutions for the linear model, which would be indeed nonsensical, but that still feels like a strawman argument to me, or at least something besides the point.

      Footnote:
      1: E.g. what’s on the Wikipedia page: https://en.wikipedia.org/wiki/Simple_linear_regression#Fitting_the_regression_line

  1. Thank you for this post. I think the issues raised are really important and ones that seem to be poorly understood. To simply a very complex topic, social scientist (not just psychologists) often assume that the phenomena they are studying is subject to measurement without considering what is required for measurement. On the one extreme are the physicists who argued that measurement requires physical concatenation. On the other extreme is Stevens who believed that numbers could be assigned to any set of objects and “measurements” taken as long as we understood whether the mapping of numbers to objects had cardinalitly, interval scale, ordinal or nominal. Parametric statistics were to not to be used for ordinal or nominal data. In the middle are the representational measurement theorists (RMT) that have worked out a number of axioms that allow measurement when the axioms hold. The thing that ought to concern everyone, is that social scientists seem to just ignore the problem. Even if you take Stevens view, you can only use parametric statistics with interval or cardinal data, which is the exception and not the rule in social science. While the RTM would allow broader use of parametric statistics, it requires the axioms to be tested, and they rarely are. So, we have this huge problem sitting at the center of all social science, i.e., “Are we really measuring anything? Can we use parametric statistics?” And, it seems like it is just being ignored.

    • The problem with Stevens’ awful 1946 paper is that it framed the issue as one of “permissible statistics”, which is completely absurd as Lord illustrated in “On the statistical treatment of football numbers”. The numbers don’t know where they came from. The question of quantifiability is about the nature of psychological constructs and what kind of theories of such constructs can be developed (and tested!). I don’t think any healthy science would see the basic nature of what is studied a matter of statistical technicalities!

      • I was unaware of Lord’s critique until you raised it. As far as I can tell, it is nonsense. Lord imagines taking nominal data putting those numbers in a vending machine and then taking the numbers out of the vending machine and seeing if parametric statistics can be used on the numbers. Of course they can. What does that prove. The natural numbers have the necessary properties for parametric statistics. Stevens point was that in assigning numbers to observations, we can’t always assume that we have cardinality or an interval scale. For instance, we have a race, did the person who came in first do twice of X as the person who came in second. Such statements have no meaning. Of course, the data being in numerical form is subject to parametric statistics, but is the underlying variable subject to measurement. I think a “healthy science” should care about “statistical technicalities” like the question: Are we measuring anything at all or just assigning numbers to observations and assuming without justification that the “construct” has the properties of the set of real numbers? Lord’s rhetoric can’t waive that serious problem away.

        • I said, “The natural numbers have the necessary properties for parametric statistics.”

          I realize this is wrong. I should have said bounded set of real numbers.

        • That wasn’t my point nor was it Lord’s, but seeing how almost everybody has misunderstood Lord for 70 years I maybe should have seen this coming.

          Lord’s point was that “impermissible statistics” can be quite useful, just look at grade point averages which can be very informative. Stevens’ rules for “permissible statistics” aren’t without exemptions (as he recognizes himself – which makes you wonder what the point of his paper was supposed to be).

          The problem with Stevens’ paper is that he doesn’t treat the issue of measurability seriously. For him the problem with using parametric statistics to analyse ordinal variables is that levels may be separated by unequal intervals but he doesn’t really consider the possibility that there might not be any intervals!

        • So, I couldn’t find Lord’s paper, and only found summaries of it. I will assume you are right about my characterization of Lord being wrong. But, then I don’t understand the point. Your example of GPAs is unhelpful to your cause. That we use GPA doesn’t mean that they are useful. Suppose student 1 has a 4.0 GPA and student 2 has a 3.5. You think, student 1 is better. Then, I tell you student 1 went to Bob’s Backyard University and student 2 went to MIT. You then say, “Gee, maybe student 2 is better.” I then tell you that student student 1 studied quantum mechanics and student 2 studied basket weaving. And on and on. Parametric statistics are a tool to help summarize data without lose of information. If the parameter hides everything that is important about the data, then what’s the point. Sure we use parametric stats all the time when we shouldn’t and it doesn’t always cause catastrophes, but that is not a very scientific defense of applying parametric statistics to ordinal data.

        • Knowing the GPA can tell you quite a bit about what combinations of grades are possible and, since the numerical values assigned to them tend to correlate, what combinations are probable, if that’s useful to know then they are informative (but I wouldn’t necessarily recommend them for comparing people across programs or institutions).

          The point I was trying to make was that if statistics work for some purpose it’s ridiculous to think of them as not “permissible”. If I ask people to rate something (like an attitude) on a Likert-scale, assign numbers to the responses and successfully use it to predict something using a parametric statistical model, that’s fine and there is no reason that should be impermissible (but I could not interpret this finding in the same way as with a measure predicting the outcome).

        • Sure. If “works” means, predicts something, then sure, we don’t need to worry about whether our model maps to the world. But, I don’t think that is what scientists believe they are doing. They want to describe the underlying structures that generate the phenomena. I am not agreeing with Steven’s way of dealing with measurement, but the act of measurement makes certain assumptions about the phenomena being measured and that’s a problem if no one is checking those assumptions.

        • “…but the act of measurement makes certain assumptions about the phenomena being measured and that’s a problem if no one is checking those assumptions”.

          I couldn’t agree more. The real damage Stevens did was to make up a ridiculous definition of measurement and turn the issue into a question of which statistics are “permissible”, offering up obviously invalid rules. Psychology has subsequently discarded the rules, kept the definition and forgotten the question – and that’s the problem.

  2. “I think a “healthy science” should care about “statistical technicalities” like the question: Are we measuring anything at all or just assigning numbers to observations and assuming without justification that the “construct” has the properties of the set of real numbers? ”

    Or worse yet, assuming the “construct” has a property that can be accurately modeled by the properties of integers.

Leave a Reply to Steve Cancel reply

Your email address will not be published. Required fields are marked *