Statistics is associated with random numbers: normal distributions, probability distributions more generally, random sampling, randomized experimentation.
But I don’t think these are the most important parts of statistics.
I thought about this when rereading this post that I wrote awhile ago but happened to appear yesterday. Here’s the relevant bit:
In his self-experimentation, Seth lived the contradiction between the two tenets of evidence-based medicine: (1) Try everything, measure everything, record everything; and (2) Make general recommendations based on statistical evidence rather than anecdotes.
Seth’s ideas were extremely evidence-based in that they were based on data that he gathered himself or that people personally sent in to him, and he did use the statistical evidence of his self-measurements, but he did not put in much effort to reduce, control, or adjust for biases in his measurements, nor did he systematically gather data on multiple people.
I think that these are the most important parts of statistics:
(a) to reduce, control, or adjust for biases and variation in measurement, and
(b) to systematically gather data on multiple cases.
This all should be obvious, but I don’t think it comes out clearly in textbooks, including my own. We get distracted by the shiny mathematical objects.
And, yes, random sampling and randomized experimentation are important, as is statistical inference in all its mathematical glory—our BDA book is full of math—, but you want those sweet, sweet measurements as your starting point.
“(a) to reduce, control, or adjust for biases and variation in measurement”
And investigating the amount to which these might be deemed successful for any given inferential target? (Quantitatively and/or qualitatively).
Andrew wrote
“I thought about this when rereading this post that I wrote awhile ago but happened to appear yesterday.”
Today is November 24, 2023 and yesterday is November 23, 2023 but, I believe the article appeared on November 20, 2023.
> Today is November 24, 2023 and yesterday is November 23, 2023.
However, three days ago “yesterday” was November 20. Maybe that’s when Andrew wrote that?
A small precondition for (a) is to recognize measurements that just aren’t feasible (in Seth’s case, the mood and the pH level of his body under Shangri-La), and the limitations of systematic strategies in bodies that aren’t arbitrage-free. Much of our metabolism is determined by genes, we have very limited agency over it
I wish every intro stats course, and maybe upper-level ones in an upper-level way, would address the question of what constitutes statistical data and how they differ from information in general. Attention could be given to the selection process (which aspects of a case/situation/etc. can be extracted for statistical use), the standardization process, and of course the question of how well the measurements in use represent the conceptual factors we want them to represent. This could lead to informed consideration of the relationship between large-n evidence and case or lab studies (that Cochrane mask meta-analysis!), techniques to constrain uncertainty derived from multiple sources of evidence, and so on. Actually, I don’t have a clue why this isn’t normal practice.
Thanks for your post on this, Andrew.
How do these important things differentiate from science? Are you arguing they are not part of science per se but really part of statistics?
Psyoskeptic
Statistics, which has sometimes been called “the logic of science,” is a part of science. Or, one might say, a part of engineering. So, yes, I’d say that statistical measurement is part of science and also part of engineering.
«Statistics, which has sometimes been called “the logic of science,” is a part of science. Or, one might say, a part of engineering.»
That is a question of semantics, and that can be vacuous or enlightening to some degree.
Part of the issue is whether mathematics is “science”, or a philosophical tool used by scientists, and whether “statistics” is a branch of mathematics or uses mathematics.
My impression of the conventional wisdom is that “science” means “natural philosophy” and so “mathematics” is usually not considered “science” in that sense being “metaphysical” rather than “natural”, and that “statistics” is a branch of mathematics, because it can be formalized as mathematics.
But the “statistics” used by researchers is not quite “mathematics” it is “applied mathematics” which usually means “engineering”, which uses mathematics but is not “metaphysical” but “physical” (even if not “natural”), so practical philosophy rather than natural or metaphysical philosophy.
Note: I also think that there is another category of “discipline”, like sociology or economics, which is developed systematically but not rigorously because the “population” changes relatively rapidly over time and is “reflexive”.
Andrew, regarding (a) to reduce, control, or adjust for biases and variation in measurement:
what I took away from your textbooks and McElreath is that the reduction of complexity lies at the heart of what we do, so I don’t think you did such a bad job in getting this across (especially in Regression and Other Stories, but probably my prior was that flat…)
«the reduction of complexity lies at the heart of what we do»
I like Penrose’s point that “science” is a (usually somewhat lossy I think) compression scheme. Inasmuch statistics deals with ergodic sources compression is possible. I would not recommend doing “statistics” on non-ergodic sources.
«Statistics is associated with random numbers: normal distributions, probability distributions more generally, random sampling, randomized experimentation»
Actually the census, trade numbers, etc. are also statistics, and they don’t involve probability considerations; the average monthly imports of cars are as much a statistical measure as the average of a probability distribution.
I am sure that our blogger knows very well that statistics includes both “algebraic” (not involving probability but “simple” counting) statistics and “stochastic” (involving probability, usually related to sampling) statistics but I have been surprised by how many statisticians forget that the median of a population and the median of a sample are both “statistics” but in very different ways if one interprets the sample as a proxy for the population (that is the median of a sample as such does not involve probability, it is its interpretation as a guess for that of the population that involves probability). It seems so
And yet one can go through the entire MA program in statistics, the way I did at Columbia, and never encounter potential outcomes, poststratification, hierarchical models, measurement error models, etc. I learned how to maximize a few likelihood functions, so there is that. To be clear, the courses I took were not useless, but they were inadequate to do anything interesting. I had to learn the good stuff from Stan people and Andrew’s books.
I think it is either a problem with MA program in biostats/stats at Columbia, or a problem of MA in stats, or a problem of MA, or all the above. The purpose of a stats MA is orthogonal to educating good data scientists (for sure, Bayesian Data Analysis and Regression and Other Stories are gems).
As a applied biostatistician, I think in short, there is a huge gap between what we learn from class and what is in reality.
There is a need for a very basic, but not easy, textbook. The first chapter should be about language. What does error mean? Does it mean mistake? And a sample- is that like a blood sample? What is a random sample? What does random mean? And a censored sample – is this something to do with government interference? What does replication mean? If we get ten people to weigh a single apple, is that ten replicates? Or do we need ten apples?
The rest of the book should be about data and measurements. If we are measuring daphnia reproduction at different temperatures, should we have ten daphnia (per temperature) in a single flask, or should we have one daphnia in each of ten flasks. And if we want to measure pesticide residues in wheat on a ship from Canada that’s just arrived at Southampton, how should we take the samples. Ten 5g samples from different parts of the ship, or 20 or 30 samples, or a single sample from the surface in each hold. Should we mix the samples together before sending to the lab or ask the lab to analyse each sample separately? And so on and so on and so on.
Peter:
Yes, a statistical sample is like a blood sample. It’s a subset of a larger population of interest.
«Yes, a statistical sample is like a blood sample. It’s a subset of a larger population of interest.»
This use of “subset” I guess is a slip and informal but it made me think and I feel a bit uncomfortable with it, because as obvious from other posts one of my pet peeves (like for NN Taleb) is the difference between arithmetic and stochastic contexts:
* The average of the first 10 integer numbers is a statistical measure.
* The first 3 integer numbers is a subset of the first 10 integer numbers. Is that a “sample” of the first 10 integer numbers, however unrepresentative?
* If the first 3 integer numbers are a “sample” of the first 10 integer numbers then the average of the first 3 integer numbers is an estimator of the average of the first 10 integer numbers. Uhm?
But a blood sample is not merely a given subset of a person’s blood, it is a sample in more common intuitive sense.
My feeling is that “given” in “given subset” is what makes me uncomfortable because the notion of “sample” we intuitively use is that illustrated by the usual “urn extraction” examples of introductory statistics (of which taking a blood sample is sort an analogous), where there is a supposedly ergodic process involved in the selection of the members of the sample.
In an urn extraction example the extracted members of the sample are in a proper sense a subset of the contents of the urns, but I would say only “ex post”; before extraction we cannot say which subset will be extracted, and that (uncertainty, it being not “given”, “odds”) is what to me makes samples in the usual sense are “stochastic”.
So I am making some fuss here over the casual use of “subset” to illustrate the difference between “ex ante” “given” and “ex-ante” “uncertain” subsets, something that I guess many people are intuitively familiar with. Subjectivists will not bet over “given” subsets but may bet over “uncertain” samples, I guess.
Blissex:
No, “subset” was not a slip; it’s what I intended to say. A sample is a subset. In your example, {1,2,3} is a subset, and a sample, of {1,2,3,4,5,6,7,8,9,10}. It is not a representative sample.
A blood sample is some blood that the nurse takes from a vein in my arm. It’s a subset of the set of all the blood in my body. Is it a representative sample? I suppose it is, but I can’t really be sure, as I don’t know enough about blood circulation and how that works. I think it’s considered as an approximately representative sample because the circulation of the blood represents a sort of physical randomization, just as the use of coin flipping or die rolling for the selection of random numbers from a list is based on a physical randomization process.