John Christie writes:
I thought you might find this paper interesting to blog about: “Informal versus formal judgment of statistical models: The case of normality assumptions”
They conclude, based on people’s accuracy of judgment of population normality from sample, that using formal tests of normality are better than informal judgments.
My reply: All silly because who cares about the normal assump at all. With rare exceptions, I don’t care about normality.
Christie responded:
I agree it’s silly. But unfortunately it’s published in a top psychology journal and the main people who will pay most attention to it and use it are those who need better education on the subject.
The fundamental way the tests are used to confirm the null is much worse, assuming one cares about power. And the fact that it got published at all is most perplexing. It causes one to pause realizing that progress on understanding the statistics a field uses can’t be policed by a field that doesn’t understand them.
I dunno. A lot of bad statistics papers are published each year, including in top statistics journals! We just have to accept that an inevitable byproduct of telling thousands of researchers that they need to publish papers, is that some confused papers will be published!
I agree that nobody should really care that much about normality in linear regression. I am just curious to see what you think about joint normality in, say, hyper parameters in hierarchical models, or in other similar situations…
That question reminds me of a paper referenced in a recent-ish post https://statmodeling.stat.columbia.edu/2023/10/11/difference-in-differences-whats-the-difference/ where it’s claimed that an analysis based on a hierarchical model – that assumes that the true effect in different settings is normally distributed around some value – offers “increased statistical efficiency”. However, even after having been “pooled toward the common mean” the distribution of parameters is very skewed with most close to zero and a few much higher. (There is a link to a histogram on my comment at the bottom of that post.)
Concerns about normality may not be justified if your primary focus in the regression modeling is just on estimating means and their confidence intervals. However, if you are interested in making prediction intervals for individual outcomes or tolerance intervals for some proportion of individual outcomes, then the normality assumption is a serious concern.
Brian:
Indeed, as I say in the linked post, “Normality and equal variance are typically minor concerns, unless you’re using the model to make predictions for individual data points.”
Val:
Yes, the choice of model for the varying parameters can make a difference. In this 1990 paper we used a mixture of three normals which enabled an adaptive form of partial pooling. Had we used a single normal, the data would’ve been partially pooled toward the population mean, which would’ve been wrong. In a followup paper in 1994, we added a predictor that captured the three components and we were able to use a normal error for these varying parameters.
Perhaps of interest?
Should we test the model assumptions before running a model-based test?
M. Iqbal Shamsudheen, Christian Hennig
https://arxiv.org/abs/1908.02218
I have only skimmed that paper, but I think the major flaw of the presentation is that when one is deciding between two alternative tests, they should be testing the same null hypothesis distribution (or alternatively, have the same causal estimand). The main example is presented between a two-sample t-test and a Wicoxon-Mann-Whitney test, which do not meet this criteria, although it is commonly misunderstand that the latter is a test of means. The result of such a combined procedure results in both a confounding of the underlying model and the test result that is being shown.
The null hypothesis is not the same, and this is discussed explicitly in the paper. Maybe you missed that from “only skimming”. Note that a major aspect of the paper is review and discussion of existing literature, and this example comes up a lot in the literature, therefore it’s prominent in the paper.
Thanks for mentioning this. Now published here: https://jdssv.org/index.php/jdssv/article/view/73
A major issue with the normality assumption paper discussed in Andrew’s posting is that real data are never normally distributed, so the skill to tell apart truly normal data from non-normal data isn’t relevant in practice anyway. Methods that assume normality typically work well for some non-normal distributions and not so well for others (particularly with outliers), and the really important skill is to tell apart deviations from normality that matter from those that don’t matter much (it’s really a gradual thing of course but whether to apply a certain normality-based method or not is a binary decision).
In this regard, also formal normality tests such as Shapiro-Wilk, KS etc. don’t really solve the right problem.
That said, the school of thought that says we shouldn’t use formal normality tests but rather data visualisation to make decisions about what method to use will have a hard time to find evidence that this works better; and obviously it’ll depend on the person who does it how well it works.
Shapir-Wilks and KS and etc solve a problem which is very relevant for testing computational methods of pseudo-random number generation. In those cases the normality assumption is supposed to hold EXACTLY (or to crazy good approximations, where only trillions of numbers will show the deviations). So they have a role to play, but it’s virtually never from real world data, always synthetic computational numbers.
Even though I have my issues with that paper (see other comment above), it isn’t silly at all to investigate whether people can make good data analytic decisions based on data visualisation. Even though the decision whether data are normal or not is rather irrelevant, the decision whether there are issues in the data that have the potential to lead the methods that we’re using astray is is very important, I’d say.
+ 1 million for this: “…an inevitable byproduct of telling thousands of researchers that they need to publish papers, is that some confused papers will be published”.
Imagine if scientists only attempted to publish, and editors only agreed to publish, when a paper really had something to new *and* important (*and* credible) to say!
Imagine all the people
Living life in peace!
You…You may say I’m a dreamer
But I’m not the only one
I hope someday you’ll join us
And the world will be as one
+1 (Just plus one since someone’s already done my patented overexaggeration bit (yes, I still think the LLM idea is parlor trick level stupidity).)
Somewhat vaguely related to this part of this thread is this.
“Scientists worry that ill-informed use of artificial intelligence is driving a deluge of unreliable or useless research.”
https://www.nature.com/articles/d41586-023-03817-6
[email protected]
The deluge was already ongoing, so it is more that there is a new “tint” to the deluge.
Already back in 2016 or so you could tell the main problem was going to be overfitting via repeated (cross-)validation leaking info about the target into the hyperparameters. Also using random train/val splits rather than splitting by the date the data was generated.
The former is easily fixed by using a train/val/test split, where you only check the skill on the test data once at the very end. But the temptation is overwhelming to run it again if you don’t like the results.
The ancient tech of assessing skill on the ability to predict *new* data generated after the model was published deals cleanly with both issues.
This is interesting though:
https://www.sciencedirect.com/science/article/pii/S2468502X21000371
Who is to say the session doesn’t also affect the radiologist’s diagnosis? Eg, changing a setting like increasing the contrast, or lowering the temperature in the room, if they suspected covid.
I’d also have to look closer at their methods of choosing hyperparameters and splitting the data though.
I’d rather live in a world where original research gets published on arxiv and where the role of journals is to to publish review articles and to provide discussion on arxiv papers they want to highlight.
I also disagree that research should be new or important. If somebody is willing to spend their time on a topic it’s important for at least n=1 people, right?
As a stats instructor, I really like that they are formally studying how students read normality plots. That’s really valuable data to study, even if i disagree about their conclusions to rely on p-values to test normality!
A few thoughts from my own experiences in no particular order … The first studies focus mostly on distributions that have strong positive kurtosis, but not so big skewness. My experience teaching is that undergrads find kurtosos way, way harder to understand and evaluate than skewness (which they are pretty good at detecting).
I think QQ plots that add an error band help students a lot, like in R car::qqPlot implementation
I think that their operational definition of normality violations being “the Mann Whitney U is more powerful than t test” as too subtle a criterion. They want ppl to try and catch skewness and kurtosis values between 1 and 2 sometimes! In practice, that’s not going to make a big impact on using a parametric rest in most cases.
That undergrads are like more than 80% accurate with plots is fantastic! Getting undergrads to be 80% accurate on any content is a triumph
“I think that their operational definition of normality violations being “the Mann Whitney U is more powerful than t test” as too subtle a criterion.” Note in particular that there are many clearly non-normal distributions for which the t-test is more powerful than Mann-Whitney U; in fact in some instances that power difference between the two is bigger than for normal distributions themselves. See our paper mentioned above https://jdssv.org/index.php/jdssv/article/view/73 for an example.
Is any PDF assumption “true”? If you fit PDF model to some data; and then use that parametric fit, whatever it be; and then use that model to say that some observation is ‘rare’ or is ‘consistent’ — that is to attach a so-called probability to some region around or beyond or enclosed by the data; or for that matter if you use the PDF — whatever it be — to quantity uncertainty in certain measurement data, those data being used to fit some explanatory or physical model; because you wish to ultimately make uncertainty statements about the predictions of that model ; say you fit a Keplerian orbit using angular observations via Gauss’s procedure for instance; and you assume your angular data are drawn from some clever fit to some empirical histogram; and your interest then proceeds to the resulting interesting distribution of the output of that processes, which may well look like the profile of Felix the Cat or anything under the sun; etc; Say you do all this with cleverly fitted and estimated empirical PDFS on the input side; and you make all manner of precise statements about the distributions of values on the output side — all tied of course to the precise description of the variability you have made so precise on the input side. Say you do all this. Is this any different than supposing that the Gauss distribution is appropriate in such and such a context; when it may be more or less of an idealization; grossly wrong or only moderately so? All the intense computer simulations in the world to produce some “more accurate” probability distribution (more appropriate than a Gaussian say) to describe some error variability or otherwise unmodeled or unmodelable variability; which are then piped into the simulation to produce a supposedly “more accurate” distributional description of the variability in some secondary or tertiary quantity of interest are — no more or less than the “naive” Gaussian only as accurate — as the saying goes — as the assumptions on which they rest.