Federico D’Atri writes:
I am a PhD student in Neuroscience and Cognitive Science at the University of Trieste.
As an exercise, I was trying to replicate the simulation results in Measurement error and the replication crisis, however, what remains unclear to me is how introducing measurement error, even when selecting for statistical significance, can lead to higher estimates compared to ideal conditions in over 50% of cases.
Imagine the case of two variables, x and y, with a certain degree of true correlation. Introducing measurement noise to y will produce a new variable, y’, which has a diminished true correlation with x. The distribution of the correlation coefficient calculated on a sample is known and depends both on the sample size and the true correlation. If we have a reduced true correlation (due to noise), even when selecting for statistical significance (and hence truncating the distribution of sample correlation coefficients), shouldn’t we find that the correlation in the noise-free case is higher in the majority of the cases?
In your article, there are three graphs. I’ve managed to reproduce the first and second graphs, and I understand that increasing the sample size decreases the proportion of studies where the absolute effect size is greater when noise is added. In the article’s second graph, however, it seems that even for small sample sizes, the majority of the time the effect is larger in the “ideal study” scenario when selecting for statistical significance. The third graph, while correctly representing the monotonic decreasing trend of the proportion, seems to contradict the second graph regarding small samples. Even though the effect might be larger, I don’t think that introducing noise would result in an effect size estimate that’s larger than without noise more than 50% of the time, given the reduced true correlation.
I ran some simulations and the only scenario in which this happens is when considering correlations very close to zero. By adding noise, thus reducing the true correlation, it becomes “easier” to obtain large, statistically significant correlations of the opposite sign. I might be missing something or making a blatant error, but I can’t see how, even when selecting for statistical significance and for small sample sizes, once we select the effect sign consistent with the true correlation, that adding noise could result in larger effects than without it over 50% of the time.
Here is a Word document where my argument is better formalized. Additionally, I’ve included two files here and here with the R code used for plotting the exact distribution of the correlation coefficients and for the simulations used to reproduce the plots from you article.
I responded that I didn’t remember exactly what we did in the paper, but I did have some R code which we used to make our graphs. It’s possible that we made a mistake somewhere or that we described our results in a confused way.
D’Atri responded:
The specific statement that I think could be wrong is this: “For small N, the reverse can be true. Of statistically significant effects observed after error, a majority could be greater than in the “ideal” setting when N is small”.
The simulations I did are slightly different, I added measurement error only on y and not on x, but the result would be the same (if you add measurement error the true correlation get smaller in magnitude).
I’ve reviewed the latter section of the code, particularly the segment related to generating the final graph. It appears there’s a problem: the approach used selects the 100 most substantial “noisy” correlations. By only using the larger noisy correlations, it neglects a fair comparison with the “non-noisy” ones (I added a comment in the line where I think lies the problem).
To address this, I’ve adapted your code while preserving its original structure. Specifically, I’ve chosen to include all effects whenever either of the two correlations (with or without added measurement noise) demonstrates statistical significance. Given the structure of our data, in particular the small sample sizes ranging from 25 to 300, the low true correlation values, and the fact that selecting for statistical significance yields few statistically significant correlations, I believed it was good to increase the simulation count significantly. This adjustment provide a more reliable proportion estimate. With these changes, the graph now closely resembles what one would achieve using your initial code, but without any selection filters based on significance or size.
Furthermore, I’ve added another segment to the code. This new portion employs a package allowing for the generation of random data with a specific foundational structure (in our case, a predetermined degree of correlation).
One thing I definitely agree with you is the need to minimize measurement error as much as possible and the detrimental effects of selecting based on statistical significance. From my perspective, the presence of a greater measurement error amplifies the tendencies towards poor research practices, post-hoc hypotheses, and the relentless pursuit of statistically significant effects where there is mainly noise.
I have not had a chance to study this in detail, so I’m posting the above discussion and code to share with others for now. The topic is important and worth thinking about.
P.S. We looked at it in more detail! See here for update.
Seems like a good example of the value of rigorous reproducibility for simulation papers, like fully automated generation of figures & results.
Someone finds what looks like an error in your paper, one which would be meaningful but ultimately still minor and non-fatal. The paper was long enough and involved enough that… present-you doesn’t know immediately if they are right or if past-you was right. You have some graphs and some R code and… doubtless you could figure it out if you sat down and spent 2 hours rereading everything to reload it into your brain and then carefully thinking about it and writing new simulation code to double-check and consider variants and so on – but actually, that’s a ton of very high-quality time, you aren’t nearly as interested in the topic as when you wrote the paper (the paper having to a considerable extent exhausted your interest/patience in the topic), and you have so many other things to do, like professional obligations or trying to get your blood pressure down at the gym or working on your current hot thing, and wedging in the necessary 2 hours begins to look much more daunting and… maybe you’ll just ignore it, or get busy coming up with a quick excuse, or punt it to them or make a blog post instead of actually figuring it out.
From the outside, it looks bad – someone went to all this trouble to find an error in your work and politely contact you about it and let you explain or correct it with just a trivial bit of effort on your part – but from the inside, to the merely human author, it feels like a lot more work than that!
For reproducibility, it really needs to be reproducible and checkable by a stranger with little time or energy to spare, because even the author will soon enough be that stranger. (A lesson often learned by programmers the hard way.)
+1
Based on the setwd-function call… Andrew uses linux? That’s neat.
No, sorry, I just use a Mac. My collaborator Aki uses Linux, though, so that’s something. And back in 1980 I used straight-up Unix.
Didn’t you use Windows too a while back?
Ah, well, but that’s something too.
related to the issue of reproducibility:
Anna Laurinavichyute, Himanshu Yadav, and Shravan Vasishth. Share the code, not just the data: A case study of the reproducibility of JML articles published under the open data policy. Journal of Memory and Language, 125, 2022.
https://www.sciencedirect.com/science/article/pii/S0749596X22000195?via%3Dihub
Allow me to add a shameless plug for my own paper on this(*): https://journals.sagepub.com/doi/full/10.1177/09567976221140828
(*) Well, I had the idea; of course, I roped in a grad student to do the hard work!
Four out of 14 articles reproducible, pretty similar to what we found. This seems to be the ballpark range of reproducibility across a diverse range of disciplines. I hope that computer science / computational linguistics is doing better (but I doubt it, people are people).
We told you so 5 years ago but you dismissed our logical argument. Now we know your main claim is based on a mistake. What are you going to do? I suggest retraction.
https://replicationindex.com/2017/02/23/random-measurement-error-and-the-replication-crisis-a-statistical-analysis/
Ulrich:
1. We read your note about our paper, where you that measurement error always attenuates effect sizes. We agree that this is true in expectation, and said so in our paper. What we pointed out was that this should not be used to assume automatically that individual studies would have produced larger effects had they somehow been conducted error free. In a simple demonstration we simulated data and estimated regression coefficients – first “error free” and then after adding random measurement error to x and y. In large samples, the “error free” estimate was always larger. In small samples, with non-trivial frequency, the “error free” estimate was smaller.
We would not want researchers to interpret statements about effect sizes always being smaller in expectation as automatically applying to individual studies. That sort of thinking leads to researchers having the mistaken believe that whatever effect size they happened to observe (no matter how extreme) would have been even larger had they been able to do the study with less noise. Our paper cautioned against this error.
2. Regarding the topic of my post above: Eric and I have been corresponding with Federico on this. If we have made a mistake, I’ll send a correction note to the journal, as I’ve done in the other cases where I’ve been made aware of errors in my published papers. Right now I think that we did not have an error; there was just some ambiguity in our claims which led to confusion. Any ambiguity is our fault. Once this is all cleared up, I’ll post on it. Whether it’s worth a correction, I’m not sure, as lots of papers have ambiguity.
3. In any case, I think there’s room for more research on this topic, moving beyond naive views of attenuation that have confused a generation of practitioners. It’s good for everyone involved that you and others are looking critically at our work.
Whatever you say. It is really very simple.
1. Random error can deflate and inflate effect sizes. In an individual study we don’t know the direction. Wow, let’s publish that in Science.
2. On average, effect size estimates are unbiased when studies are randomly selected from the pool of all studies. Hurray, a Nature publication.
3. When we select for significance, we inflate effect sizes and the inflation is proportional to the power of a study. I think Ioannidis already published this before he went on to make the crazy claim that most published results are false positives. But hey, maybe it can be worked into a new PNAS paper.
4. Random measurement error attenuates standardized effect sizes. Always, but random sampling error can counteract this (see 1). In the same study with the same sampling error, the more reliable measure will always show a stronger effect size than the less reliable measure. No paradox, no reversals, just plain attenuation. Of course, we can also do something about it if we use a measure with known psychometric properties and dis-attenuate for unreliability which is common practice in psychology meta-analysis. You might want to check out some of those.
“Practitioners” were not confused. They often ignored – and still ignore – that selection for significance and p-hacking inflates effect sizes. That was highlighted by Ioannidis in 2004. Your paper didn’t shed a new light on the problem and it had zero impact on practices in psychology. It remains true that unbiased effect size estimates in meta-analysis are attenuated by random measurement error and that ignores the problem of systematic measurement error.
The real work that needs to be done is to validate measures and to correct for invalidity of measures in the estimation of true effect sizes.
https://www.jstor.org/stable/25662607
Ulrich:
You seem to have a lot of sarcasm in your post. Eric and I thought it was a good idea for us to publish our paper in Science because there is such widespread confusion among applied researchers (here’s yet another example) who seem to think that noisy studies are ok, who think that, if a noisy study yields a “statistically significant” result, that they’d expect to see something even stronger if the study were less noisy. That’s a mistake, as we explain in our paper.
I can well believe that you, like many other sophisticated methodologists, were already aware of this point. But we’re not only publishing for you! Other people read journals too.
You write that our paper “didn’t shed a new light on the problem.” I think it did! But that’s a judgment call. Considering that you still don’t seem to fully understand our paper, I don’t think you’re the best judge of whether it sheds new light. But I accept that Eric and I should take some of the blame for this. Sometimes it’s hard to anticipate, when we write a paper, what later confusions will arise.
You write that our paper “zero impact on practices in psychology.” I doubt it had zero impact. According to Google, it’s been cited 616 times. Most of my papers have not been cited 616 times! OK, not all of those citations are in psychology, some must be from myself or my collaborators, some of the citations are not practitioners, etc etc . . . but I’m very skeptical of a claim of zero impact.
In any case, as I wrote above, I agree that there’s room for more research on this topic, and you should definitely do the work that’s important and publish it Science, Nature, PNAS, or wherever else is appropriate
In my own work I try to take reproducibility very seriously, but I have to admit that despite trying really hard, it can happen that from one day to the next, a project can suddenly stop compiling, becoming effectively irreproducible. You need to be a really rigorous person to be able to actually produce a truly reproducible piece of work. I don’t think many people can reach that level of sophistication. Certainly I keep falling short despite my best efforts.
Probably someone like Bob Carpenter can achieve it. I downloaded and compiled his prob theory book in a single shot, no errors. Wow.
Andrew, trying one more time. You seem to confuse random sampling error in small samples with random measurement error in the dependent variable. The Heckman quote is the questionable claim that significant results in small samples imply a strong POPULATION effect size because a study with small samples and small effect sizes has low power and is likely to return a non-significant result.
This statement ignores that selection for significance inflates OBSERVED effect sizes and can produce significance when the OBSERVED effect size is inflated. A replication would show that the next study produces a disappointing result. As we point out, confidence intervals are a simple solution to avoiding the misinterpretation of OBSERVED effect sizes in small samples. That was discussed in psychology in the 1990s (Cohen, 1994). All of this has nothing to do with measurement error. Are you sure you are using the term correctly and understand the effect of random variance in a variable? I don’t doubt your intelligence or knowledge of statistics, but measurement error is a psychometric concept. Please distinguish clearly between random sampling error and random measurement error.
>measurement error is a psychometric concept
umm… physicists and engineers and biologists and such around the world would disagree. Perhaps “psychometric measurement error is a psychometric concept” but then not having read this paper I don’t know whether Gelman is discussing anything psychometric specific or is this just a general discussion of “measurement error”… Measurement error is more or less well defined when an “ideal” measurement would be x but because of difficulty with measurement “instruments” the actual measurement is x + err for some value of err.
From the article
Yet, statistical significance
conveys very little information when
measurements are noisy. In noisy research
settings, poor measurement can contribute
to exaggerated estimates of effect size
The article makes it clear that it discusses psychometric concepts like reliability and random measurement error in measures. The language is sloppy, but “noise research settings” translates into studies with large sampling error and poor measurement translates into “low reliability”.
The claim is that low reliability can exaggerate effect size estimates in studies with large sampling error (low N, between-subject designs). This claim goes against the common assumption that random measurement error attenuates estimates of the population effect size. That is the key point of this paper. I claim it is wrong and Federico D’Atri’s work explains how Loken and Gelman may have arrived at this false conclusion. Maybe Andrew can take the time to investigate this and see whether D’Afri and our work are wrong. But statements like “I have not had a chance to study this in detail” and ” Right now I think that we did not have an error” are not very helpful. Just look into it and tell us what you find. We did so 5 years ago and Andrew ignored us. I am glad that we are revisiting this question and that Andrew cannot simply ignore the issue anymore. Time will tell whether unreliable measures can inflate effect sizes in studies with large sampling error. The ball is now clearly in Andrew’s court.
Ulrich:
Again, we did not ignore you several years ago. We read what you wrote at the time and disagreed, as mentioned in point 1 here.
Ulrich,
Your comments on this post make it sound like you’re upset because Andrew published something that is trivially true and widely known, yet at the same time false because he didn’t understand some important concepts, and on the one hand “Practitioners” were not confused in the first place, but on the other hand “they often ignored – and still ignore – that selection for significance and p-hacking inflates affect sizes.” At no point in your posts do I see anything that represents an attempt to move anyone’s understanding forward, you are simply trying to bully Andrew into admitting that his work, which is incorrect-but-correct, widely-known-but-often-ignored, cited-600+-times-but-has-zero-influence, is deficient in some way.
You clearly have a beef with Andrew, but you might consider the fact that for those of us who don’t know you at all you are starting to come across as a bit of a crank.
No. I am upset because we found a mistake in his paper and he dismissed it and now somebody found an error in his code and he thinks that maybe there is a problem. Maybe he should take the time, check the code, and then tell us whether there is a mistake or not. As somebody who blocks about errors in science, I hold him to a higher standard and expect him to correct his own mistakes swiftly and without playing games. Wouldn’t you expect the same?
and yes, I am a crank because the whole game of academia is a sham. Why should I respect Andrew, if he is not willing to take criticism of his work seriously?
Ulrich:
1. For reasons mentioned in point 1 of my above comment, I do not think you found a mistake in our paper. Loken and I did not “dismiss” your comments; we read your comments and disagreed with them.
2. Someone sent me a question about that paper, I responded to him right away with my code, and then when I didn’t have the energy to look into the question right away, I posted it on the blog for all to see. Since then, my coauthor has become involved in the discussion and I’m pretty sure we did not make a mistake; it’s just a communication issue. In any case, it’s both insulting and ludicrous of you to say that I’m acting slowly or “playing games.”
3. No need for you to respect me! It’s all about the work, not the people. I appreciate that you openly post your disagreements here and elsewhere. Students and researchers can read our published work and go from there. People can even ask us directly, as with the student who emailed me the question featured in the above post.
4. I do not think what we are doing is a sham. The questions we’re discussing address real issues, such as the award-winning economist’s overestimates of the effects of early childhood intervention, and many other policy-relevant topics.
5. You say that I’m not willing to take criticism of my work seriously. That’s absolutely ridiculous. You sent me criticism, and Loken and I took it seriously. Federico D’Atri sent me criticism, and again we took it seriously. What I really shouldn’t be doing is wasting time responding to blog comments, but I guess I can’t resist.
Here is a dropbox link with all the files related to our criticism of the Loken and Gelman article, in case somebody else wants to dig deeper into this.
https://www.dropbox.com/scl/fo/kjkiftzymggvzvkfaa5op/h?rlkey=wjniswecfj5vrkjwn0qz22ueg&dl=0
I took a quick look. One thing I saw was this: “There is general consensus that random measurement error introduces noise in the relationship between two variables that makes it harder to detect a signal.” Oh? In the very next post on this blog, https://statmodeling.stat.columbia.edu/2023/10/30/on-a-really-bad-paper-on-birth-month-and-autism/ , we have a nice example of a statistically significant ‘signal’ that appears to be completely due to noise. Perhaps you would argue that the nose in question isn’t “random measurement error”, it’s “small-sample error”, but if that’s your argument then I hope you realize that exactly the same phenomenon can and does occur if the noise is due to what you would agree is measurement error.
But I think your main point is that whatever _genuine_ signal is present, noise can only attenuate it. I think that’s true if one is very careful to define terms. IN EXPECTATION noise will attenuate the signal. But in any individual realization the noise can lead to anything happening, including overwhelming the signal or exaggerating it or whatever. If a poker player plays twenty hands in a four-way game, and wins nine of them, she’s probably the best poker player of the bunch… but she’s probably not going to win 9 of the next 20 games. There’s a popular saying, “it’s better to be lucky than good.” When I have a string of successes that involve some luck, I like to give my own version: “it’s better to be lucky and good.”
I doubt you disagree with anything I’ve said here. Indeed I suspect that if I looked through all the stuff in that dropbox folder I would find that you have nice illustrations of these points. And I don’t think anything here contradicts anything Andrew said in the Science paper, either. So I don’t know what you’re going on about.
It took some time to refresh my memory and read the old discussion I had with Andrew about this issue.
https://statmodeling.stat.columbia.edu/2017/02/11/measurement-error-replication-crisis/#comment-420573
Rickard and I figured out that one key point of confusion is the type of effect size we are talking about.
I assume that Andrew thinks in terms of unstandardized effect sizes; e.g., how many percentage points of error variance are in an opinion poll. As a psychologists, I am used to thinking about standardized effect sizes because mean differences on a Likert scale have little meaning.
This is important because measurement error first of all adds error variance to the variance in a measure. If we care about the unstandardized effect size, it is NOT attenuated by random measurement error because the measurement error variance is simply added to the sampling error variance. There is just more uncertainty about the real effect size. The source of the error (sample or measure) is not very interesting.
Now things are different when we talk about standardized effect size estimates like a correlation coefficient, r.
The article seems to suggest that it also talks about those when it discusses disattenuation for unreliability.
“Measurement error and other sources of uncontrolled variation in scientific research therefore
add noise. The latter is typically an attenuating factor, as acknowledged in various scientific disciplines. Spearman (2)
famously derived a formula for the attenuation of observed correlations due to unreliable reliable measurement”
Spearman’s formula is used to DISATTANUATE correlation coefficients, which are STANDARDIZED effect sizes. It would make no sense to apply this formula to unstandardized effect sizes because they are not ATTENUATED by random measurement error. The reason why we need to disattenuate standardized coefficient is that standardized coefficients divide the effect size by the variance or standard deviation and one portion of this variance is random error variance. So, standardized effect sizes are by definition always underestimating the true effect size. That was our point, but we later realized that things are a bit different for unstandardized effect sizes.
Now to the issue of selection for significance. Selection for significance will always inflate effect size estimates, but the amount differs depending on the power of a study. Of course, the power of a study is not dependent on the effect size that we compute from the results. Thus, standardized or unstandardized effect sizes are equally effected by selection for significance. Both will be inflated by selecting only studies in which sampling error produced a stronger effect than the population effect size. However, the error variance is constant, same sample size, same measure. What is moving around is the observed signal in each study. The main reason less reliable measures lead to more inflation is that they lower power and require more bias to get significance. But it also true that it was harder to get a significant result because the unreliable measure underestimated the true effect size which is the cause of the low power and the need to inflate the effect size estimate to get significance.
A simple solution to all of this is to report results with confidence intervals or if you are Bayesian with credibility intervals. The large amount of error will result in wide intervals and it is immediately clear that the point estimate is a wild guess. Cohen (1994) suggested as much and APA even has reporting of CI in its manual, although it is not always followed.
I think all of my points are simple facts about statistics. Andrews problem seems to be that he hates any form of conventional statistics and is trying to find flaws even where they do not exist. For example, this statement about t-values is absurd.
“It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an
estimate is statistically significant, the signal-to-noise level is high.”
Particle physics uses z > 5. Does that mean they do not know what they are doing?
He also focusses too much on single studies. Single studies are not very informative, even when sample sizes are large. Effect sizes are often underestimated because measures have a lot of measurement error. To deal with these problems we need meta-analyses that correct for publication bias. Unstandardized effect size estimates (e.g., average reaction times) will not be affected by random measurement error. However, standardized effect sizes will be underestimated and need to be corrected for unreliability. However, as most measures in psychology have decent reliability, the estimates are often not that different.
If you followed my expose of the issue, you might also find the following conclusion from Andrew’s article confusing if not misleading.
The consequences for scientific replication are obvious. Many published effects
are overstated and future studies, powered by the expectation that the effects can be
replicated, might be destined to fail before they even begin. We would all run faster
without a backpack on our backs. But when it comes to surprising research findings
from small studies, measurement error (or other uncontrolled variation) should not be
invoked automatically to suggest that effects are even larger. ”
Measurement error cannot be invoked to suggest that effects are even larger when we are talking about unstandardized effect sizes. They are not deflated by unreliable measures. When we are talking about standardized effect sizes, the standardized effect size is inflated by selection for significance and deflated by random measurement error. Ignoring one factor and focusing on the other is wrong, but I don’t think the sentence makes it clear that we are talking about two opposing biases.
I am looking forward to your response or responses by anybody else. I am all for open discussion and open science.
Best, Uli
Particle physics uses z>5 because physicists got burned a few times by using z>3 or other such levels. But using z>5 automatically means the sample size is not extremely small. But this special case does not disprove or even discount the main point of the sentence, which I quote here again: “It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an
estimate is statistically significant, the signal-to-noise level is high.”
Do you really disagree with that? Well, maybe; let’s break it into its components:
1. Do you agree that it’s a mistake to conclude that _just_ because an estimate is statistically significant, the signal-to-noise level is high? (If you think this is not a mistake, please address the autism example from the post following this one.)
2. Do you agree that this is a “common mistake”? I suppose this depends on the definition of “common”. Would you at least agree that it is not rare?
And, moving on to the conclusion of Andrew’s article, let’s give it the same treatment:
1. Many published effects are overstated. Agree or disagree?
2. future studies, powered by the expectation that the effects can be replicated, might be destined to fail before they even begin. Agree or disagree?
3. When it comes to surprising research findings from small studies, measurement error (or other uncontrolled variation) should not be invoked automatically to suggest that effects are even larger [than the central estimate obtained by the researchers].
You’re going to disagree with at least one of those, but I honestly can’t guess which one. Or…well, it pretty much has to be #3. I can’t see a reasonable person arguing the contrary of #1, even a few years ago but especially now that everyone is familiar with the “replication crisis”. And if you agree that published effects are often overstated, then you’ve gotta agree that many studies that attempt to replicate those effects are doomed to fail: such studies will be underpowered or will be attempting to replicate an effect that doesn’t even exist.
So you’re going argue that measurement error _should_ be invoked automatically to suggest that surprising effects from small studies are likely understated. There’s nothing else in that paragraph so if you think that paragraph is misleading then you must find this part misleading or confusing. So, OK, let’s hear your argument for why a surprising effect estimate from a small study should be assumed to be an underestimate.
Phil, I’ll play this guessing game too… My guess is that Ulrich will say that in the limit of large sample size the measurement errors is not important and a statistically significant result is an indicator of a true effect, particularly for abs(t) greater than 5 kinds of levels. He will argue that such levels of significance DO indicate high signal to noise ratios.
I’m not sure he’s wrong per se, I just don’t know how much relevant examples he can come up with particularly in psychology. How many effects have been studied in tens of thousands of people with 5sigma results?
Phil, disappointing.
The main interesting point in my comment that arose from Rickards and my investigation of this topic was the important distinction between standardized and unstandardized effect sizes. No comment on this? Why?
Now to the criticism of t-values not inferences from p < .05 statements.
"“It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an
estimate is statistically significant, the signal-to-noise level is high.”
IT IS A COMMON MISTAKE TO TAKE T-RATIO AS A MEASURE OF STRENGTH OF EVIDENCE.
First of all, nobody looks at the t-values. Everybody looks just at the claim that p is smaller than .05. That is a big problem, but the sentence claims something different. It claims that the signal to noise ratio (effect size over sampling error) is not a measure of the strength of evidence. Now there was no reason to drop such a stupid claim into an article about measurement error and the replication crisis, but it is in there and it is wrong. Of course, a larger signal to noise ratio provides stronger evidence against the nil-hypothesis. You really want to argue with this. Then take it up with particle physics and tell them that using sigma5 is wrong. Personally, I am now rejecting the null-hypothesis that you are an unbiased participant in this discussion interested in understanding the relationship between random measurement error and replicability.
Ullrich,
“… but the sentence [“…just because an estimate is statistically significant, the signal-to-noise level is high” from Andrew’s paper] claims something different. It claims that the signal to noise ratio (effect size over sampling error) is not a measure of the strength of evidence. Now there was no reason to drop such a stupid claim into an article about measurement error and the replication crisis, but it is in there and it is wrong. Of course, a larger signal to noise ratio provides stronger evidence against the nil-hypothesis. You really want to argue with this.”
Andrew is NOT claiming that the signal-to-noise ratio is not a measure of the strength of evidence. He is claiming that a statistically significant estimate does not imply a high signal-to-noise ratio.
I really don’t understand you argument.
Hey, the good news is that I’ve heard that our article had “zero impact on practices in psychology,” so no damage has been done! What we should really be worried about are all the articles and books I’ve published that have had impacts, and all the errors from those that I’ve injected into the scientific literature. When you think about it that way, it’s kinda scary, indeed would’ve made a good Halloween post!
Uh, this is pretty obviously true. The t-ratio is just not a sufficient statistic for the signal to noise ratio. Furthermore, the t-ratio and classical significance tests don’t say anything about the signal to noise ratio; they are constructed to control the type i error rate. Frequentist estimation theory categorically cannot make statements about whether the true signal to noise ratio is probably high or probably low, it can only provide procedures that asymptotically guarantee some error rate within a class of problems.
Consider a control and treatment group with n observations. Control group has mean mu_c, treatment group has mean mu_t, control group has sample standard deviation sd_c, treatment group has sample standard deviation sd_t. You can compute a t-value like normal t_0.
Consider a second set with N >> n observations. The new control group also has Control group has mean mu_c and sample standard deviation sd_c. The new treatment group also has sample standard deviation sd_t, but mean mu_t2 = mu_t – (mu_c – mu_t)(sqrt(n / N) – 1). The new t-value will be exactly t_0, but the inferred maximum likelihood or bayesian signal to noise ratio will be much lower. As N goes to infinity, the t-value stays exactly the same, but signal to noise ratio/standardized effect size goes to zero. This is because the t-value does not measure the signal to noise ratio.
That says, in context, the statement does not even need a technical defense.
The point of the article is that a less noisy, larger sample study is better and strong evidence, even with statistical significance filter. That seems obviously unobjectionable, and I don’t understand why it makes you so upset.
I’m going to assume that what you mean is that random measurement error attenuates the expectation of the measured standardized effect size. But if you hold the absolute effect size constant and let measurement error grow, variance in the sample correlation coefficient or measured standardized effect size increases. If you combine that with “hunting” for statistical significance, you get a systematic inflation effect in the literature. That’s the point.
Perhaps another reason why typical standardized effect size measures standardized using data from the same setting are a bad idea.
This recent paper from the field of accounting seems relevant:
The first set of simulations illustrates the conventional wisdom that measurement error biases against rejecting the null. In the second set of simulations, we allow the measurement error (in X) to be correlated with a noisy control variable (Z). This mirrors the empirical application discussed below […] Our simulations show that, when measurement error in the variable of interest is correlated with a control variable, measurement error can bias the coefficient on the variable of interest away from zero and lead to false positives. The intuition for this result is that when measurement error in the variable of interest is correlated with a control variable, the regression places weight on the variable of interest that would otherwise be placed on the control variable.
In our third set of simulations, we show that high-dimensional fixed effects can exacerbate measurement error bias and lead the researcher to estimate a causal effect when none exists.
https://link.springer.com/article/10.1007/s11142-023-09754-z
Oh boy, so many statistics experts and nobody seems to understand t-values.
Look at the formula. The numerator is the effect size. The bigger the differences from zero, the bigger is the t-value. That is the signal!
Now look at the denominator. It becomes smaller as N increases. That is the noise and it is reduced by increasing sample sizes.
Now compute the ratio and there you have it, the t-value is the signal to noise ratio.
Now, we may say, but wait, it depends on the degrees of freedom, but now we are talking 2.20 to 1.96, which is small potatoes. Clearly a t-value of 3, 4, 5, or 20, is larger than a t-value of 2, and the chance that the null-hypothesis is true becomes smaller and smaller. If that is not strength of evidence, I think statisticians should stick to numbers and not try to use words like strength or evidence.
Some people seem to need a tutorial on t-values.
The numerator is the effect size. The bigger the differences from zero, the bigger is the t-value. That is the signal!
Now look at the denominator. It becomes smaller as N increases. That is the noise and it is reduced by increasing sample sizes.
Now compute the ratio and there you have it, the t-value is the signal to noise ratio.
Now, we may say, but wait, it depends on the degrees of freedom, but now we are talking 2.20 to 1.96, which is small potatoes. Clearly a t-value of 3, 4, 5, or 20, is larger than a t-value of 2, and the chance that the null-hypothesis is true becomes smaller and smaller. If that is not strength of evidence, I think statisticians should stick to numbers and not try to use words like strength or evidence.
But the “chance that the null-hypothesis is true” was and is zero, regardless of the t-value. I don’t need the degrees of freedom to know that.
No, when people usually talk about noise, they do not not mix estimation/finite-sample error and process variance/measurement error in that way. The denominator of the t-ratio is monotonic with the noise, but decreases with sample size while noise does not. For example; standardized effect sizes like cohen’s d are a popular form of signal-to-noise ratio; the denominator is the pooled standard deviation, which does not go to zero as n increases. If your sample size is high enough, your t-value can get arbitrarily high with an arbitrarily small cohen’s d.
Of course, this is semantics; you can define things however you want, but I’m just telling you that what you write here goes against how the term is usually used and is less analogous to its origins in signal detection engineering.
Obviously, if the assumptions of the t-test hold, the t-value is strength of evidence AGAINST THE NULL HYPOTHESIS. The problems comes when we’re not talking about the null hypothesis but trying to make inferences about the actual magnitude of the effect, normalized or not. The introduction to the article under discussion sets the topic pretty clearly
“A common view is that any study finding an effect under noisy conditions provides evidence that the underlying effect is particularly strong and robust.”
Well, you quoted the whole sentence, so not really. But if your only objection only applies to half of a single sentence, then I’m afraid before you start discussing science and statistics you need to learn how to read. You can raise an objection to literally anything that way. For example
Ludicrous! Obviously statisticians need to use words or no one will understand them! Silly Ulrich!
I understand why you want to remain anonymous. It would be embarrassing to post your nonsense with your name.
Cohen’s d is a standardized effect size measure that compares an effect in the units of the measure to the naturally occurring true variance in the dependent variable. For example, Cohen’s d of 1.4 for the difference in height of men and women is obtained by dividing the height difference of 10cm by the standard deviation of actual differences in height 7cm.
Are you calling the naturally occurring differences in height noise?
Noise, or error variance is introduced by (a) unreliability in the measurement of height (negligible in this case) and sampling error (the difference in a small sample will not be the same as in the population).
Now you seem to be confusing effect size with strength of evidence. Strength of evidence is about hypothesis testing. And in large samples, we can have strong evidence that a small effect size is not zero. Whether that difference is important (practically significant) is a different question, but we should not confuse effect size estimation with hypothesis testing.
In my opinion, “strength of evidence” is a vague term, and what it refers to is obviously context dependent. The article talks about effect sizes from the beginning to the end, even in the sentence you quoted.
“It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an estimate is statistically significant, the signal-to-noise level is high”
What it seems like to me is that it was pointed out to you that it’s actually true that t-ratio =/> high signal-to-noise level, so you then decide to claim “my objection is to the first part of the sentence and I made that very clear.” Well first of all, it wasn’t made clear, but more importantly it transparently makes no sense at all to object to half of a sentence. I feel like now you’re trying to make it seem like “strength of evidence” has a very specific definition referring to null hypothesis significance testing in order to make the half-sentence nitpick seem less ludicrous.
No, but I’m pretty sure when you compute cohen’s you divide by the measured standard deviation which does may include measurement error and “natural variation”. For example, here this blog post says that “Random measurement error does clearly influence standardized effect size estimates because the standard deviation is used to standardized effect sizes.”
https://replicationindex.com/2018/05/03/confused-about-statistics-read-more-cohen-and-less-loken-gelman/#:~:text=Similarly%20Cohen's%20d%20is%20a,used%20to%20standardized%20effect%20sizes.
Wait a minute–you wrote this! What exactly are you trying to pull here, trying to make it seem like noise/measurement error does not make it into the denominator of cohen’s d, while writing elsewhere that it does? Could it be that you’re picking and choosing your semantics in order to not look like you’re wrong on the internet?
Anyways, the point is that if absolute effect size is .01 and measurement error is 1.0, I’d consider it noisy and have a high signal to noise ratio, even though as n goes to infinity, the t-ratio will still get arbitrarily large. Similarly, in signal processing, a “noisy channel” has a low signal to noise ratio in terms of power ratio, even though you can still transfer information through it if you allow arbitrary amounts of time for replication and error correction. You can disagree with these semantics if you want, but in a common sense, the full sentence you said is “absurd” seems correct in context for a typical sense of “signal-to-noise”.
Sorry, clipped it wrong. The full relevant quote from (haha) your blog post is
So it does seem like noise is in the denominator of cohen’s d according to your post, even though here you claim it’s only “natural variation” which you are saying is not noise
To revise my earlier statement; it’s entirely context dependent. If I’m interested in the effect of childhood protein consumption on adult height, then yeah, intrinsic variation in height is noise. Just like if I’m studying neutrino interactions, the cosmic microwave background is noise, while if I’m studying the age of the universe, it’s signal. In the most literal possible sense, the leakage radiation from my microwave is “noise” for my WiFi antenna as I try to load this post, but is signal to the engineers at mitsubishi testing of microwave leakage is within regulated limits. There’s no special properties that makes something “noise” or not. It’s just something that adds uncertainty to the thing you’re trying to measure, whether it’s literal measurement error or researcher skill or orthogonal variation in intermediate quantities
Ulrich:
You write of the t-value that “the numerator is the effect size.” That is incorrect. The numerator is the estimated effect size. This can make a big difference when an estimate is noisy. For more on this point, see this letter I published in the Annals of Surgery (which, I’m happy to say, did not try to charge me $300 for the privilege).
Glad you joined the discussion.
Yes, it is the ESTIMATED effect size or the OBSERVED effect size. It is not the population effect size.
This clarification doesn’t change the interpretation of t-values as the signal (signal!!!) in the sample (not the population) whether the population effect size is zero or not.
So, can you know explain why you think “t-ratios are not a measure of strength of evidence?
Take Cohen’s d of .4 as an example. With se = .4, we get a t-ratio of 1. With se = .04, we get a t-ratio of 10. I would say a t-value of 10 provides stronger evidence that the population effect size is not zero than a t-value of 1. What would you say?
This seems to be a common issue in so much writing about “effect size” (perhaps at least by psychologists): it doesn’t keep track of the difference between estimand and estimate… People just write d to mean something that is an estimate. Hats on please!
Dean,
Yes, and it affects practice (or, at least, practical recommendations), as we can see for example with Heckman etc promoting their biased point estimates and then piping them into cost-benefit analyses. I fear that one problem of the focus on causal identification is people forgetting about uncertainty. Or, more precisely, thinking that if they have obtained an estimate that’s 2 standard errors away from zero, that retrospectively they can then think that noise is not a problem with their study (or, even worse, to think that if their study had been less noisy, that their estimate would’ve been even higher).
The above blog discussion is just reminding me of why Eric and I felt the need to write our paper in the first place!
Hi Dean,
do you know psychologists? Do you read their articles?
1. Psychologists have been told by people like Cohen to care about effect sizes. Many articles report some effect size estimate (meaning a sample statistic, not sure whether that translates into estimand or estimate). They are well aware that this value refers to the sample and not to the population.
2. Psychologists have also been told to report confidence intervals, nowadays they often do, but not always.
3. The real issue is that nobody really cares about these values. They are not used for a priori power analyses or to build theories that make quantitative predictions. In practice, the key finding is still p < .05. That is the real problem.
So, I think your claim about psychologists is false. The problem is not that they cannot distinguish between a sample statistic and the effect size in the population. The real problem is that they do not really care about effect sizes.
Yes, I have some interactions with psychologists. My initial training in statistics during my PhD was in a psychology department before I realized how bad the training was and jumped to the stat department.
In my comment here I was mainly thinking about psych methods writing, where I often see no distinction between estimand (population quantity) and estimate. A good example is almost all discussion of various “alternatives” to Cohen’s d. Here’s one haphazardly-selected example: https://psycnet.apa.org/fulltext/2017-10871-001.html
Pek & Flora say a bunch of helpful things here, but repeatedly apparently define “effect size” as an estimate, making it really difficult to reason about why one might have subtly different estimators for the same quantity. It also makes it harder to characterize how you end up biased for the true effect size after selection, which (agreed) is a big problem.
My sense is this is a ubiquitous problem; however, of course that doesn’t mean this is the only problem in empirical psychology research!
Rik, my objection is to the first part of the sentence and I made that very clear.
IT IS A COMMON MISTAKE TO TAKE T-RATIO AS A MEASURE OF STRENGTH OF EVIDENCE.
First, it is not a common mistake because nobody cares about the t-value. Everybody only cares about t > 2, which is p < .05
Second, t-values provide quantitative information about the strength of evidence against the null-hypothesis. A t-value of 20 makes it much less likely that the null-hypothesis is true than a t-value of 2.
“A t-value of 20 makes it much less likely that the null-hypothesis is true than a t-value of 2.”…I would agree with this, if you add “for a given sample size.”
You claim I don’t want to get into a discussion of standardized vs unstandardized effect sizes. You are correct.
The reason I got into this discussion is because you were coming across as a crank and I thought maybe you’re not a crank and would be able to usefully change the way you present your ideas. Unfortunately that doesn’t seem to be the case.
You made the fairly strong claim that the concluding paragraph of Gelman et al.’s Science paper is ‘confusing if not misleading.’ And yet, as I pointed out, that paragraph makes only three assertions and I don’t find any of them confusing or misleading. I asked which one you think is confusing or misleading and you haven’t answered. Instead you are dodging the question by trying to get into a discussion of standardized vs unstandardized effect sizes. The fact that Gelman et al. did not frame their work the way you think they should have framed it bothers you, and you may be right that if they had framed it your way it would be better in some way, but even if that’s the case it does not mean they got something wrong or were confusing or misleading. Their final paragraph is very short; you claim it’s confusing or misleading; but if you can’t even try to articulate what you think they got wrong in it then I really don’t see what you’re trying to achieve here.
I’m going to try to resist the temptation to respond again here. Andrew can fight his own battle if he thinks it’s worth fighting.
Phil
“A t-value of 20 makes it much less likely that the null-hypothesis is true than a t-value of 2.”…I would agree with this, if you add “for a given sample size.”
Give me one sample size scenario, where t = 20 with small N provides less evidence than t = 2 with larger N. Are you thinking about 1 df? LOL
It is scary that even simple deductive facts are controversial. No wonder practitioners don’t listen to the “stats gods”. There are too many gods and nobody knows who is the real god. Meanwhile, p < .05 rules.
“Give me one sample size scenario, where t = 20 with small N provides less evidence than t = 2 with larger N.”
Surely this can be true even when both examples have large N? Take the following scenario: You have two groups with 500 observations each. In group 1, there is one observation each with value 1, 2, 3, etc. all the way up to 500. In group 2, there is one observation each with value 1, 2, 3, etc. all the way up to 499, with the final observation equal to 5000.
Calculate a Welsh t.test in R with “t.test(rep(0:1,each=500),c(1:500,1:499,5000))” and you get a t-value of -38.653.
Compare this to a scenario where the two groups have normally distributed observations, with some small difference in means so that the t-value becomes 2 for a similar test between them.
I would consider the latter example to provide more evidence against the null than the former. Because the former is just the result of one observation – all other observations are exactly equal between the groups – it may not fulfill the assumptions of the test, and it may be the result of measurement error – perhaps the final observation has an extra zero.
Oops, I seem to have wrongly remembered the formula for the t.test function here. You can ignore my previous response. Although it seems there could be similar situations where there would be some such difference.
How about this:
Suppose you have four observations (or some other small number) on two yes / no questions, scored 0 and 1, and that your participants who answered yes on one question always answered yes on the second, and vice versa. Suppose now that you conduct a Pearson correlation. This will yield a correlation of 1, with a corresponding t-value equal to infinity (well above 20).
Next, suppose you do the same with two variables that are bivariate normally distributed for a lot of observations, and receive a t-value of 2.
I would consider the latter case better evidence against the null, because the assumptions of the test are better fulfilled, and because when you have discrete observations in the first case, you increase the chance that the observations will fall out exactly equal (corresponding to r = 1.00) even if there is not an underlying perfect linear relationship between them.
So to get back to the initial point made by Federico, that it’s hard to understand how correlations between measurements made with error could exceed (in absolute values) correlations between measurements made without error in MORE THAN half the cases.
I concur with Federico. As an argument “ad absurdum”, if the measurements of x and y were made entirely of noise, their average correlation would be zero, half of observed correlation values would be positive and half negative, regardless of sample size. So regardless of the true value of rho(x,y) it seems impossible that MORE THAN half observed rs could exceed that true value.
To Ulrich’s argument about sampling error and measurement error. Attenuation due to measurement error pushes observed rs toward 0 in expectation, but there is no telling if in a given sample (especially if N is small) measurement error has reduced or increased r (as compared with an r on measurements made without error).
I did a quick simulation with rho=0.15, x and y normally distributed, N=20, 100 replications. The observed mean r was 0.143, sd 0.222, range -0.363 to 0.518. Then I added measurement error, normally distributed, with a quarter of the variance, so that reliability would be 0.8. Now observed mean r’ was 0.113 (pretty much exactly what Spearman would predict), sd 0.223, range -0.504 to 0.594. In 41 instances (out of 100 the correlation of imprecise measurements exceeded that of precise mesurements.
“So to get back to the initial point made by Federico, that it’s hard to understand how correlations between measurements made with error could exceed (in absolute values) correlations between measurements made without error in MORE THAN half the cases. I concur with Federico.”
Sure, so do I. But once you apply a filter, such as a filter for ‘statistical significance’ or for ‘publishability’ then the picture is completely different. If there is some small effect at the level of the population, and your estimate of it is very noisy, then your estimate is very likely to be larger in magnitude than the true effect. See this for example https://statmodeling.stat.columbia.edu/2023/09/07/this-is-what-power-06-looks-like-visualized-by-art-owen/ or this https://statmodeling.stat.columbia.edu/2014/11/17/power-06-looks-like-get-used/
Sorry about the delayed response.
I agree with you, I had missed the part about publication bias (the figure caption doesn’t mention it, but it’s in the text somewhere).
Somebody
At least we are getting somewhere.
1. Yes, Cohen’s D divides the unstandardized effect size by the standard deviation to provide a comparable measure of effect size when DV are measured in different units. Height differences are dependent on the measurement in meters or inches or millimeters. After standardization, the unit no longer matters and the standardized effect size is the same.
2. Cohen’s d is a ‘noisy’ measure of the true effect size for several reasons.
1. Most important, in small samples the estimate of the mean difference (effect) size is influenced by sampling error.
2. When the SD is estimated from the sample SD, it is a biased measure of the population SD, although the difference is small in reasonably large samples.
3. When the measure is unreliable, random noise is added to the SD and it is wider and the standardized effect size is biased downwards. I t seems we agree on this one based on your last comment.
4. When the population variance (SD) is not relevant and we are only interested in within-person changes over time, we should not use Cohen’s d. There is dz or for the IAT D (the within-person SD).
The most important point here is that we seem to agree on #3. Noise increases the SD and attenuates Cohens’d and other standardized effect size estimates like r. Therefore it is perfectly reasonable to estimate the amount of random error variance and to correct for the attenuation or to suggest that the standardized effect size would have been stronger with a more reliable measure. It follows logically from the contribution of random noise to the SD. Agreed?
In a hypothetical alternate universe with no selection for statistical significance; kind of? We are talking about finite-sample estimators. Estimators have a sampling distribution. Adding extra noise lowers the expectation of the observed standardized effect size and yes you can bias correct this and move it upwards. However, the variance can be extremely large, so I wouldn’t really safely conclude either way.
Now in the real world, the results you actually see are obviously selected, for literal statistical significance and other stuff, so no, not in reality.
I am not sure you know the actual literature.
Few meta-analyses correct for unreliability because they use measures with unknown reliability (ouch).
The correction for unreliability is usually done in correlational studies with large samples where sampling error is not a big issue. In these studies selection for significance is also not an issue.
It is really a problem when armchair statisticians write methodological criticism based on some stereotype of actual researchers without knowing actual research practices in a field.
It is just wrong to claim that in the real word results are obviously selected for significance and selection for significance is a different bias. If there is evidence for selection for significance, the correction should be for the selection for significance. Not correcting for unreliability and for selection for significance because they cancel out is really not a sound approach.
At no point did I that you should not bias correct. Bias correction may or may not be a good idea depending on the situation. In some cases biased estimators are better
https://en.wikipedia.org/wiki/Stein%27s_example#Formal_statement
but it depends. But I didn’t say not to bias correct for unreliability. I haven’t studied the variance of spearman’s disattenuated estimator, but if I had to guess, I’d say it’s probably a good idea.
I do still think that results are selected for significance. I’m surprised you think otherwise. There is quantifiable evidence for it.
https://normaldeviate.wordpress.com/2012/08/16/p-values-gone-wild-and-multiscale-madness/
In any case, selection for significance is the topic of the original discussion.
You don’t seem to know my work. Sometimes there is selection for significance and sometimes there isn’t.
It is an empirical question and it is wrong to have a prior that all research is biased.
https://replicationindex.com/2021/01/19/personalized-p-values/
Ulrich
I can say that I am not familiar with your work. The link is an interesting read and I am probably much more open to your approach than many on this blog. I’m willing to look at p values and even try to use them more appropriately – though I prefer confidence intervals (much to the horror of many on this blog). But one thing about your approach does concern me – the counting of tests and computations based on these seems to move away from a decision theoretic framework that I think is essential. All false discoveries are not created equal. Somewhere the seriousness of the study – the costs associated with wrong or misleading conclusions is important. Counting tests and p value distributions can be interesting to look at, but moves us away from careful consideration of what is at stake in a study. Somehow, based on a researcher’s record, applying a p value threshold of 0.02 rather 0.05 doesn’t seem like the appropriate way to examine these costs. Indeed, no threshold approach is likely to be a suitable way to consider the costs associated with mistaken conclusions from a study.
I believe reputation should matter. Certainly a researcher’s track record is important. But I don’t want to “excuse” a researcher’s bad conduct due to evidence that they have a large number of hypothesis tests that don’t exhibit selection for significance. I don’t think it is a good idea to reduce the issue to a numerical exercise. Is that what you are suggesting?
I agree that using additional information, including individual information about the researcher history, can further refine your interpretation about a particular work. That doesn’t seem to contradict anything written above, or in the original article under discussion.
Dale Lehman,
nothing is perfect and there are many issues with this approach. The most common criticism is that the automatic extraction cannot distinguish crucial theoretical tests and other tests that are reported (e.g., Manipulation Checks). Ideally, all articles would be hand-coded and we could even weight tests based on cost-benefits although most psychology research has little costs or benefits as there is usually no direct application of the results (e.g., what is the cost-benefit analysis for personality predictors of life-satisfaction?). Consider the work a proof of concept that we could go beyond p < .05 and consider the strength (or lack thereoff) of evidence that researchers provide for their claims. Also, CI are related to z-scores. A z-score of 2 implies p = .05 which implies that the lower bound of the CI is exactly at zero. So, it is clear that the study provides no meaningful information about effect sizes and cannot be used to claim practical importance. It just warrants "further research is warranted."
Thanks for taking the time to check it out.
“most psychology research has little costs or benefits as there is usually no direct application of the results”
That is a striking statement. I believe Andrew posted some time back about the value (or lack thereof) of social science research. If most psychology research has no direct application, I do have to wonder why anyone bothers.
it is a proto(type) science. Mostly we are building a science that eventually has some benefits. Before you ax psychology, you might take an ax to English lit and the humanities. It is not always about $$$.
“most psychology research has little costs or benefits as there is usually no direct application of the results”
This is the normal condition of basic research init? It seeks to find and confirm fundamental relationships. It’s only after those relationships are established that applied methods can be derived from them.
IMO the most blatant failing of much social science “research” is that it seeks a simple method to skip all the hard work of finding and confirming actual fundamental relationships and tries to jump straight to applications with idiot crap 5-yr-olds-in-the-sand-box ideas like “mentioning old people or disabilities makes people more sympathetic”. To even call it “science” is laughable, it’s just ridiculous crap perpetuated by incompetent fools.
Forblogs:
Indeed, and a key part of the junk science, Ivy League to NPR to Gladwell to Freaknomics to Ted to Psychological Science to National Academy of Sciences pipeline is the role of “statistical significance” in fostering overconfidence. That’s a big reason why Loken and I wrote our paper, that it’s so frustrating to see people doing crap science without thinking about measurement at all, and then thinking that, just because they achieved “statistical significance,” that they have no problem with signal-to-noise, even to the extent of thinking that had they done better measurements their estimated effects would’ve been even higher! The comments by Schimmack on this thread, not to mention the writings of Heckman and others regarding effects of social interventions, demonstrate that this mathematically simple idea is conceptually so subtle that there are quantitative researchers in psychology and economics who still don’t get the point. They’ve been sucked in by the allure of statistical significance.
Regarding your last sentence: I assume that Heckman is not a “fool,” but he seems stuck in a fixed idea, a bit of overconfidence buttressed by an illusion of rigor.
Andrew.
Your overconfidence is startling. You don’t read or think. You are just writing in the illusion that you know everything better than everybody else. The Loken and Gelman paper is at best confusing and probably based on an error in the R-Code. Where are we with your effort to investigate this problem and telling us what is going on with the discovery that started this post? How much time does a brilliant mind like you need to figure it out?
The problem of selection for significance has been discussed for a long time. Your paper claimed that the contribution of random measurement error to the replication crisis has been overlooked. Yet, even if we buy your simulation results (a BIG IF), there is no empirical evidence that low reliability is a major contributor to replication failures. Unlike you I have actually studied the replication crisis empirically and know a lot more about the actual causes of replication failures than you do. You just have written a flippant comment on Leek and Jager’s article. Fast and sloppy, just like some of the people you like to criticize.
I won’t defend Andrew, as he is perfectly capable of doing that himself. But I find your attitude startling and off-putting. Frankly, it makes it difficult to see through your diatribes and assertions to identify any valid points you might be making. Sarcasm about “brilliant” minds, claims about knowing “a lot more” and accusations of “fast and sloppy” don’t convince me of anything other than your own overconfidence. Just telling you how it strikes one reader who doesn’t have any stake in this particular battle.
Dale:
Occasionally I’ve felt the need to ban a commenter, if they are posting things that are some combination of too abusive and too off-topic. Usually, though, I follow the strategy of responding carefully to the specific points being made. Even if the commenter I’m responding with is a troll, or just someone too stubborn to learn, I hope that my direct responses will be useful to others who are reading the comment thread. In addition, by directly responding to comments, we’re modeling good internet behavior. At some point, such threads go to what Phil refers to as “garbage time,” but sometimes we can make some progress before then. At the very least, sometimes this can reveal places where our original explanations could’ve been clearer.
Not trying to convince you. I am happy to wait for Andrew to present the results of his investigation of the alleged error in their simulation code.
It has been 5 days since Andrew shared this concern with us. Let’s see when we get an answer.
“Posted on October 29, 2023 9:55 AM by Andrew”
Ulrich:
For chrissake, it’s not “an investigation.” Dude send me an email, I replied, we’ve had an email exchange, we asked him to write something we can post. Maybe it’ll appear soon, maybe not. The short answer is that we all agree there was no mistake in our simulations nor was there a mistake in our paper; the new post will just clarify some things. In the meantime, you can rest assured that our original paper was just fine. But don’t worry, this all has “zero impact on practices in psychology,” as you so charmingly put it.
Ulrich:
1. What Dale said. Try removing the abuse, the contradictory statements, the unsupported claims, the mind-reading, the all-caps, etc.
2. My comment, coauthored with Keith O’Rourke, on Jager and Leek’s paper was published in 2013, and it’s neither flippant nor sloppy. As to “fast,” that’s a relative term; all I can say is that Keith and I took the writing of our comment seriously.
If he removes the abuse, the contradictory statements, the unsupported claims, etc., will there be anything left? That sounds like pure sarcasm but it’s a genuine question.
I actually think there is/was content in the issues, but the animosity got in the way of my being able to see it. I really don’t care whether Andrew’s article years ago added to Ulrich’s understanding or what he thought was well known – I do care about detecting signals amidst noise, significance filters, and whether/how we might correct for selection biases. But rather than state the issues, I got lost in the diatribe about whether an old article did or did not present new information, whether an old discussion was handled correctly, and unfounded and inappropriate hubris about qualifications to express opinions. Whatever issues there are got buried under all that, at least for me.
Phil:
I think that his specific issues are: (1) he thinks that the term “measurement error” should be reserved for measurement error in the predictor, whereas we use the term to also include measurement error in the outcome, (2) he is bothered that we’re discussing the measurement-error issue in the context of selection on statistical significance. With regard to point 1, we prefer the more general term, because measurement error happens all over. With regard to point 2, we’re addressing selection on statistical significance and we make that very clear in our paper. So, yeah, there’s some content in his issues, but not a lot.
I think I can understand his sputtering, though. Loken and I published something he didn’t like. He wrote us a note saying why we didn’t like it. We read his note and did not find it convincing. Meanwhile, this paper he doesn’t like has been cited all over. He’s frustrated and he’s lashing out. Not helpful behavior, but understandable in some way. He’s essentially screaming at the TV, just as I do when I hear Malcolm Gladwell talking about airplanes taking off in a stiff tailwind or whatever. From Schimmack’s point of view, Loken and I are just Gladwells with Ph.D.’s.
Ulrich:
For chrissake, it’s not “an investigation.” Dude send me an email, I replied, we’ve had an email exchange, we asked him to write something we can post. Maybe it’ll appear soon, maybe not. The short answer is that we all agree there was no mistake in our simulations nor was there a mistake in our paper; the new post will just clarify some things. In the meantime, you can rest assured that our original paper was just fine. But don’t worry, this all has “zero impact on practices in psychology,” as you so charmingly put it.
Andrew:
Here I point out in detail why there is an error in your simulation unless your claim is “when we select the most extreme estimates on variable 1 and compare it to the matching values on variable 2, the values on variable 1 are higher. Sure, but you are doing the selection for significance here. We can flip the variables and condition on the reliable measure and suddenly it produces higher estimates.”
Well, here it is. Tell me where I am wrong.
Ulrich:
https://replicationindex.com/2023/11/04/random-measurement-error-and-the-replication-crisis/
You can also ignore it and be Reviewer 2 when we submit it to a journal.
I think you make some good points in your post. But isn’t the point of the article by Loken and Gelman that significant results with errors should not be interpreted as if they would automatically yield stronger effect sizes with less error – a reasoning I feel one can see not too unfrequently. So the whole point is to look at what would happen with the measure without error when the one with error yields significant results. Then there is no point to condition on the significance of the measure without error, because you do not have access to that measure in the study. In short, I do not see any contradiction between your and Loken and Gelman’s results, and I think there is a reason to point out, as they do, that with selection for significance, there is no guarantee that published estimates from measures with errors will even be underestimating the effect size most of the time. One could certainly always clarify things further, and develop the reasoning around different situations, but I just don’t see the contradiction you seem to see.
significant results with errors
we are past these casual statements. What “error” and how does “error” influence effect size estimates and when do people who find inflated effect size estimates in small noisy samples say, the effect size would have been even bigger. Even if you find some of these articles, the real problem is clearly that selection for significance inflates effect sizes independent of the reliability of the measure. This just has nothing to do with random MEASUREMENT error.
The simulation is clearly a mistake as I show because the conditioning of significance biases the results of the simulation. It is not a fair comparison to compare an unreliable measure conditioned for significance with a reliable measure not conditioned for significance. Apples and Oranges.
In a real experiment, performed by a non-omniscient researcher, the amount of error in the measurements is unknown so there is no way to make significance estimates which don’t include effects of the errors. Therefore the Lorken-Gelman simulation seems realistic to me.
JimV
Not sure what discipline you are, but in psychology the amount of random measurement error in a variable is often well-known. Reliability estimates for commonly used measures are pretty much the only psychometric information that is commonly available. Cronbach’s alpha = X. Rather than estimating the standard deviation from a small sample, one could also use estimates from larger samples if the populations are similar. Sample size is also known. The only real unknown is the unstandardized population effect size.
Moreover, this comment has nothing to do with Loken and Gelman’s simulation mistake that simulated different amounts of random measurement error and claimed a paradox reversal based on the mistake of conditioning for significance on one variable, but not on the other.
I posted polls on Twitter/X and in Facebook discussion group. I also asked ChatGPT.
Which lab is more likely to publish a bigger (more biased) effect size estimate? Two labs are studying the same effect, say r = .15 in the population between two variables. Both labs use the same small sample size of N = 50. Both labs only publish when they get a statistically significant result, p < .05 (two-tailed). Both labs use a perfectly reliable preditor variable (e.g., height). The only difference between the two labs is that LAB A uses a more reliable measure (Cronbach's alpha = .8) than LAB B (Cronbach's alpha = .5).
2/3 got it wrong. They answered consistent with Loken and Gelman.
ChatGPT got it right, but ignored selection for significance.
My simulation of this scenario shows that LAB A produces stronger (more biased) correlations.
https://replicationindex.com/2023/11/04/random-measurement-error-and-the-replication-crisis/#Extension1
Ulrich:
Which experiment produces higher effect size estimates, after selection on statistical significance, depends entirely on the standard error of the estimate, which depends on all the inputs to the model. The short answer is that in some settings measurement error decreases that standard error (thus decreasing the estimated effect sizes under selection) and in other settings measurement error increases that standard error (thus increasing the estimated effect sizes under selection). This is completely consistent with our published article.
The two key parameters are how much measurement error there is in the predictor and how much measurement error there is in the outcome. Real-world studies, including in psychology, can include a lot of measurement error in both. Below is some R code, for anyone who’s interested.
Just to keep it simple, I just conditioned on the estimate being more than 2 se’s from 0 and also being positive. Allowing negative and statistically significant results and reporting the absolute value would give similar results.
options(digits=3) set.seed(123) meas_err_once <- function(n, s.x1, s.y1) { # Define predictor x0 and outcome y0 under ideal measrurements x0 <- rnorm(n, 0, 1) y0 <- rnorm(n, 0.15*x0, sqrt(1 - 0.15^2)) # Fit the linear regression fit0 <- lm(y0 ~ x0) # Add some independent measurement error to x0 and y0 x1 <- rnorm(n, x0, s.x1) y1 <- rnorm(n, y0, s.y1) # Fit the linear regression fit1 <- lm(y1 ~ x1) # Return the estimates and se's c(coef(fit0)["x0"], se.coef(fit0)["x0"], coef(fit1)["x1"], se.coef(fit1)["x1"]) } # Loop it meas_err_loop <- function(n, s.x1, s.y1, n_loop=10000) { sim <- array(NA, c(n_loop,4)) colnames(sim) <- c("coef0", "se0", "coef1", "se1") for (loop in 1:n_loop) { sim[loop,] <- meas_err_once(n, s.x1, s.y1) } # Summaries cat("Sd of extra measurment error in x and in y:", s.x1, s.y1, "\n") cat("Averages over", n_loop, "simulations:\n") print(colMeans(sim)) coef1 <- sim[,"coef1"] signif0 <- sim[,"coef0"] > 2*sim[,"se0"] signif1 <- sim[,"coef1"] > 2*sim[,"se1"] cat("Power under each experiment:", mean(signif0), mean(signif1), "\n") cat("Avg estimate under each experiment with selection:", mean(sim[signif0,"coef0"]), mean(sim[signif1,"coef1"]), "\n") } # Run it with essentially zero extra measurement error meas_err_loop(n=50, s.x1=0.1, s.y1=0.1) # Substantial measurement error meas_err_loop(n=50, s.x1=1, s.y1=1) # More measurement error in x than in y meas_err_loop(n=50, s.x1=1, s.y1=0.5) # More measurement error in y than in x meas_err_loop(n=50, s.x1=0.5, s.y1=1)And the output. For each, experiment 0 is the baseline (so, the same in each simulation below) and experiment 1 includes extra measurement error in x and y:
You are adding measurement error but the effect size is not changing. Why.
You changed the sim from correlation as effect sizes (standardized coefficient) to unstandardized regression coefficients. As I show in my blog post it makes a difference. Can we agree that the reversal does not happen with correlations.
Ulrich:
In my simulations, the effect sizes are fixed. The simulations 0 are the same each time; the simulations 1 change with different levels of measurement error. It’s fine for you to do research on correlations. Eric and I just wrote this one paper; as I wrote somewhere in the above comment thread, there’s lots of room for more work in this area, so go for it.
Andrew.
You simulated correlations in your paper. Check your own code. cor(x,y)
Otherwise fig. 2 would not show consistent underestimating.
D’Atri responded:
The specific statement that I think could be wrong is this: “For small N, the reverse can be true. Of statistically significant effects observed after error, a majority could be greater than in the “ideal” setting when N is small”.
I took the R code that Andrew pointed to above (https://statmodeling.stat.columbia.edu/wp-content/uploads/2023/10/measurement_loken.txt), loaded it into R-Studio and ran it.
I’m not very familiar with R or R-Studio so I may have made some mistakes. But, let’s assume that I didn’t make a mistake. At the end of the run, two pdf files were created. They matched the first two figures of the paper. In addition, a plot was displayed that looked somewhat like figure 3 of the paper. It similar axes (x=axis labelled “sample size” and a range from 0 to 3000, y-axis goes from 0.0 to 0.5 and is labelled “Prop where error for greater”) and a similar curve. However, that curve did not match figure 3. It began at 0.5 (well, about 0.48 really) versus beginning at 0.58 in the figure. It looks much like a rescaled version of figure 3. The curve generated by that code matched D’Atri’s statement quoted above. And, whoever set the plot y-axis range seemed to agree with D’Atri.
It seems to me, a guy who took a statistics course about a half-century ago, that D’Atri’s observation is correct. That is, I think that considering a universe of many highly-noisy, small-N studies, at most 50% of that universe would have effects greater than the ideal. And, that is what plot 3 of Andrew’s code shows.
Perhaps the y-axis of the plot in the paper was mislabeled.
I just scanned the paper again. I see the statement that D’Atri references is not part of the text of the paper itself; rather it is set off with the figures and references (correctly) the data shown in the graph in the third figure. But that graph is probably wrong and, if so, the error in the graph could have caused someone to write the incorrect sentence that D’Atri is troubled by.
Bob76
> I just scanned the paper again. I see the statement that D’Atri references is not part of the text of the paper itself; rather it is set off with the figures and references (correctly) the data shown in the graph in the third figure. But that graph is probably wrong and, if so, the error in the graph could have caused someone to write the incorrect sentence that D’Atri is troubled by.
The following is part of the main text: “In a low-noise setting, the theoretical re-sults of Hausman and others correctly show that measurement error will attenuate coefficient estimates. But we can demonstrate with a simple exercise that the opposite occurs in the presence of high noise and selection on statistical significance.”
If I have understood the discussion it’s not correct to say that the opposite occurs (i.e. measurement error will amplify coefficient estimates) in the presence of high noise and selection on statistical estimate – at least compared to the alternative where the measurement error is removed but the selection on statistical significance remains.
Correct. There is an error in fug 3 and in the claim of a reversal
Maybe I am confused. But I think that figure 2 of the paper clearly shows that the opposite occurs.
This exchange is not the first time that Andrew has made the point that Hausman’s “Iron Rule” is a little rusty in the low SNR regime.
D’Atri’s concern was with the assertion that crept in w.r.t. figure 3 that “the opposite” could occur more than 50% of the time. In the low SNR, small N regime, you are more or less flipping a coin to determine whether the “normal” case (attenuation) or the opposite occurs. Worst case, this becomes a 50:50 proposition.
Ulrich’s statement below that “the claim of a reversal is incorrect” is incorrect. The error is that the claim that reversals can occur more than 50% of the time.
Bob76
> Maybe I am confused. But I think that figure 2 of the paper clearly shows that the opposite occurs.
Then figure 1 clearly shows that the opposite occurs as well with large sample size: it happens at least four times in that chart.
Really a reply to Carlos above.
Carlos states, “Then figure 1 clearly shows that the opposite occurs as well with large sample size: it happens at least four times in that chart.”
Correct. And, figure 3, with the y-axis properly labelled, shows that the bigger N is, the less likely it is that the opposite will occur.
Bob76
An aside. The line plotted in figure 3 is jagged and irregular. That is a consequence of the relatively small number of cases considered (1000). Increase that to 100,000 and the line becomes pretty smooth. See
https://drive.google.com/file/d/1oIw4iail9P7t7wGdnavCF3zpVYf7GS7U/view?usp=sharing
It doesn’t seem correct then to say that “In a low-noise setting [something happens but] the opposite occurs in the presence of high noise” if the same thing can happen and the difference is a matter of degree (and there is no reversal of anything nor any meaningful “opposition”).
Thank you. Despite your limited r skills you got the correct result. There is no reversal.
Andrew
You ignore everything I say. Maybe you can respond to this question by somebody neutral?
Perhaps the y-axis of the plot in the paper was mislabeled
This discussion is troubling and yet symptomatic of applied statistics in social sciences:
Terminology in statistical practice is often fuzzy or ambiguous, so much so that even statisticians have difficulties communicating.
Furthermore, language contributes to confusion. For example, I read the statement
“It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an
estimate is statistically significant, the signal-to-noise level is high.”
to mean
“It is a common mistake to take a t-ratio as a measure of strength of evidence”
and I am still unclear, after the many exchanges, if Andrew actually meant to say that.
When two competent statisticians have difficulties communicating clearly the vast majority of ordinary psychologists are likely to be left either utterly dumbfounded or, worst, falsely presuming they comprehend the issues on the strength of eloquence, not statistics.
P. S. I personally do not mind to be called stupid or ignorant (I’m often both) but I do mind when others imply I have bad intentions, as Ulrich seems to do with Andrew.
I get your point. Maybe you also understand my frustration that I did spend probably close to 100 hours on this with co-workers and Andrew blows me off as if I am an idiot.
It seems to me there may be a genuine misunderstanding or a genuine disagreement, although that might be because I do not understand the issues fully. In any case semantics and terminology play a crucial role.
Andrew does not seem the type to ignore factual errors in his work, though I know him only by his past posts. He might be involved in too many things at once, though, and it is then easy to develop tunnel vision.
My own experience with pointing out factual errors in four papers is far, far, far worse than what is indicated by the exchange above.
This includes a widely cited paper on adolescent self-harm that proclaimed
“Notably, 43.1% of SA/SI visits were for children aged 5 to younger than 11 years”
— an assertion that is absurd but that was reported by numerous news outlets (CNN, USA Today, The Atlantic).
It turns out the authors were counting head injuries as self-harm — see The Importance of Being Inquisitive (https://theshoresofacademia.blogspot.com/2020/04/the-importance-of-being-inquisitive.html).
Three years later it is still uncorrected and some of the exchange prior to being ghosted was astonishingly unpleasant (I skipped that part in the report).
The issue here is not of great importance, although Andrew might disagree.
Interpreting point estimates in small samples is wrong.
Claiming that the real effects would be larger when we have a lot of uncertainty is also wrong.
Given rhe small significance of the claim that a reversal can occur Andrew could just agree that there is no paradox effect of attenuation by measurement error in small samples after conditioning on significance.
But first he has to realize that that claim is an error.
If you take “strength of evidence” to be “strength of evidence against the null hypothesis” and assume that the assumptions of the t-test are met, then by definition the t-ratio is a measure of strength of evidence. The problems are
1. The assumptions of the t-value are not necessarily met. Given the meta-data available about replication rates, discontinuities in published papers at the standard p=0.05 threshold, it’s plainly obvious that they *often* aren’t met. Hence, interpretation of the t-ratio needs to take that into account, which requires contextual information which is not in the data presented in the paper itself.
2. Evidence “against the null hypothesis” is not the only kind of evidence people are interested in. People are interested in making inferences about the magnitude of the “true effect.” The t-ratio does not necessarily provide any information about this.
If a statement is difficult to interpret, it’s usually better to add more context than to remove it. Forgive me if I’m wrong, but when people zoom in on a statement like this and remove the context, it sounds like they want an answer to the normative question “is reporting and using t-ratios good?” The t-ratio just is what it is; a quantity that follows a particular distribution under some assumptions. It just implies only what it implies; if it’s extremely outside the bulk of that distribution, it suggests that one or more of those assumptions may be false. Which assumptions are likely to be more or less false and whether or not that falsehood is interesting cannot be given a generic answer. Hence, what you can conclude *just* from the t-ratio or significance is limited.
You ignore that t-values are the best predictor of replication failures. t > 5 implies replication failure rate of < 20%. No context needed. Just t-value
Even in the naive case of perfect t-test assumptions and perfect replications, where a successful replication is defined as seeing statistical significance with the same analysis in a follow up study, the replication success rate depends on both the t-ratio and the distribution of powers the initial studies are drawn from. The latter can be estimated from the broader literature, but cannot be estimated just from the data presented in one paper. When you give a specific number I’m assuming you’re referring to this
https://replicationindex.com/2021/04/25/z-curve-an-even-better-p-curve/
I haven’t read it in detail (it looks cool, I’m sure you’re doing it right), but it looks like a model fitted to an empirical population of input studies that estimates the power distribution. That is, it is formed from other pieces of evidence that are external to the t-ratio of a particular study, but which are part of the circumstances that form the setting for the t-ratio under consideration, and in terms of which those t-ratios can be fully understood and assessed. In other words, it is
https://brainly.ph/question/9323667
It really feels like you want everything to be wrong, and are picking and choosing interpretations and definitions so as to make them wrong
I am happy to play a betting game with you.
We bet on outcomes of replication studies.
I bet on success for t > 5 and failure for t < 3.
I will earn a lot of money
Well, that depends. Even with even odds, you won’t make money if I get to choose the studies we bet on.
First of all, a t-distribution with 1 degree of freedom is a cauchy distribution, where P(t 5, and submit them to vixra.
Not letting me choose the studies is choosing a context
Markdown swallowed my comment
Well, that depends. Even with even odds, you won’t make money if I get to choose the studies we bet on.
First of all, a t-distribution with 1 degree of freedom is a cauchy distribution, where P(t less than 5) ~= 0.9372, which isn’t necessarily that strong even for decently powered studies.
But anyways, if I were to choose studies, I’d study the difference in means between rnorm(2, 0, 1) and a second realization of rnorm(2, 0, 1) and submit statistically significant iterations to vixra.
Speaking of vixra, how would you interpret the t-ratio from confidence calculation VI here?
https://vixra.org/abs/2301.0157
I did read the full text of the letter and I do think the statement
“It is a common mistake to take a t-ratio as a measure of strength of evidence and conclude that just because an estimate is statistically significant, the signal-to-noise level is high.”
is likely to confuse or mislead much of the intended audience.
My view is based a bit paradoxically on an agreement with Andrew, and a disagreement with Ulrich, in one crucial area: the need to explicate seemingly trivial issues in elementary stats to social science researchers (and to do so in prestigious journals). That includes warning against presuming that “any study finding an effect under noisy conditions provides evidence that the underlying effect is particularly strong and robust.”
I do think Ulrich is unfair in dismissing the relevance of this effort as I am convinced that the vast majority of researchers in some areas of social sciences do not comprehend elementary statistics.
Yet it is precisely due to my agreement with Andrew on this that I hold him to a high standard when he writes for social science researchers — the need to communicate clearly and precisely is all the more greater.
One note on the utility of psychological research:
I’m not a psychologist but I became interested in the conundrum of adolescent suicide doubling within a decade (2007-2017). While I think this may be linked to increases in childhood trauma of parental suicides and overdoses, causality is difficult to prove and there are other factors, such as the doubling of adolescent depression and anxiety.
I am not a statistician but I have a sufficiently solid math background to tell that much of the relevant research is flawed in its misuse of statistics, some of it outright celebrated while having no scientific value.
I mention this because there are areas where psychological research (and so also applied statistics) truly matter: there are a thousand more teens killing themselves every year nowadays than there were not so long ago.