Thomas Langkamp writes:
I posted a question on the binomial test on the cross-validated forum. Surprisingly nombody seems to have an answer or even a suggestion. Maybe you can share your thoughts?
Here’s Langkamp’s question:
How to calculate the Signal-to-Noise-Ratio (SNR) for the classic / frequentist binomial test? For example for the z-Test of proportions the empirical z-value or SNR is:
SNR = z
= Signal / Noise
= Cohens g / Standard error of proportions
= (p – p0) / Square Root of (p · (1 – p) / n)
with p as the observed proportion and with p0 as the reference value, sometimes called Pi. with n as the number of observationsSo can I use this SNR also for the Binomial Test if n is small and or if p does not equal 0.5? If so, why do e.g. SPSS or JASP do not report this value? At least for big n it would make sense to report it, as the binomial distribution aproaches the z-distribution.
And here’s my reply:
There’s no precise definition of signal-to-noise ratio, but the way that I think about it, the numerator (the size of the “signal”) should represent some population of effects, not the effect in a single study. Trying to make design calculations using the effect in a single study creates all sorts of problems of the sort discussed here. So instead of using p-p0 in the numerator, I’d use an average absolute value of effect size, or root mean squared effect size, or something like that, in a hypothetical population of studies. You won’t know this value, but I’ll just say that it’s probably much smaller than you think it is.
And that brings us to the answer to the question, why doesn’t statistical software report the signal-to-noise ratio? It’s because this ratio depends on the numerator, which you don’t know—and for studies where statistics is relevant, you can’t estimate it well from the available data either. So best to just report the estimate and standard error, and if you want to take an estimated “signal” from somewhere and make a signal-to-noise ratio calculation, you can do so.
Thanks Andrew Gelman. I rephrased the question a couple of weeks ago to make it more clear, that it is geared towards the binomial test. Ho to calculate the SNR for the binomial test specifically – more specifically for small n and / or p unequal 0.5?
Here’s an alternative idea.
Take a prior for f the frequency of success, let’s use something like beta(a,b) where a,b are the “effective success and failure counts”.
Take a sample X which is a sequence of 0 and 1 values for failure,success.
Calculate a posterior for f which is p(f). This is conjugate to the beta prior so you can do it in closed form, or you can approximate if you didn’t use a beta prior.
Now, call the “signal” the difference between the prior standard deviation s1 and the posterior standard deviation s2. That is, this is the “uncertainty reduction” that the sample gave us.
Call the “noise” the remaining standard deviation in the posterior.
(s1-s2)/s2 = s1/s2 – 1
is then the signal/noise ratio under this construct.
Thank you Daniel Lakeland. This sounds like a valid solution.
However… what I try to accomplish is to explain my second-semester psychology students why we have sample-SNRs all over the place (t, z, F, Chi²) but not for the binomial test. And my students are trained in frequentist statistics only (time constraints and me not feeling comfortable teaching bayesian concepts yet). I should have added this goal over on crossvalidated from the get go, I guess. I will add it now.