The above-titled article is from 2014. Key passage:
Hypothesis testing and p-values are so compelling in that they fit in so well with the Popperian model in which science advances via refutation of hypotheses. For both theoretical and practical reasons I am supportive of a (modified) Popperian philosophy of science in which models are advanced and then refuted. But a necessary part of falsificationism is that the models being rejected are worthy of consideration. If a group of researchers in some scientific field develops an interesting scientific model with predictive power, then I think it very appropriate to use this model for inference and to check it rigorously, eventually abandoning it and replacing it with something better if it fails to make accurate predictions in a definitive series of experiments. This is the form of hypothesis testing and falsification that is valuable to me. In common practice, however, the “null hypothesis” is a straw man that exists only to be rejected. In this case, I am typically much more interested in the size of the effect, its persistence, and how it varies across different situations. I would like to reserve hypothesis testing for the exploration of serious hypotheses and not as in indirect form of statistical inference that typically has the effect of reducing scientific explorations to yes/no conclusions.
The logical followup is that article I wrote the other day, Before data analysis: Additional recommendations for designing experiments to learn about the world.
But the real reason I’m bringing up this old paper is to link to this fun discussion revolving around how the article never appeared in the journal that invited it, because I found out they wanted to charge me $300 to publish it, and I preferred to just post it for free. (OK, not completely free; it does cost something to maintain these sites, but the cost is orders of magnitude less than $300 for 115 kilobytes of content.)
Considering 1) research that suggests that you can possibly find “statistically significant” findings concerning the most absurd and impossible things (e.g. see Bem, 2011; Simmons et al., 2011) by using p-values and a few Questionable Research Practices, 2) the possibility that certain research findings might be useful and desirable for certain people with certain motives, and 3) that scientific papers make some scientific publishers lots of money I now view the “p-value” as a “power-value”.
The use of the “p-value” or “power-value” might provide certain people with the power to present scientific evidence for certain things which may come in handy, and with the power to publish lots of papers considering lots of things that may or may not be valid and may or may not be related to lots of other things which are possibly likely not being investigated due to focusing on the “p-value” instead of, for instance, the effect size.
The $300 charge was in 2014 dollars so that, considering inflation and time value of money, what would be the equivalent current fee? What is, in fact, the current fee? Once these numbers are available, perhaps there should be a reconsideration? If nothing else, once knowing these numbers, a contributor can use them as some sort of bargaining chip.
Would paying the $300 get your article more readership and credibility with those readers?
A:
Publishing an article in more than one place would, I assume, only increase its readership. On the other hand, there are other things I could do with my $300; also it just annoyed me that they asked me to write an article for them, I did them the favor of doing so, and then they wanted to charge me a fee for it. It’s like . . . oh, I dunno, suppose you went to the local park to volunteer on clean-up day, and you spent a few hours picking up trash or whatever, and then you went home and found that the city had billed you $300 for your participation. You’d be kind of annoyed, no? I get that that the Ecological Society of America needs to raise money to support its endeavors, and that the executives of John Wiley & Sons need money to pay for their kids’ college tuitions or whatever, so if they want to charge for their journal, I get that. But asking me to write an article for free, and then to pay them for it . . . that’s a step too far!
I wonder every now and then why I had several data-analyses courses during my “Research Master in Behavioural Science”-education and not one concerning logic or reasoning or argumentation.
I just checked my diploma supplement to verify and I can’t find any course listed there that seems to have to do with logic, reasoning, or argumentation. Or maybe it was part of the course “Basic principles of Behavioural Science” that is listed there. I simply can’t remember being taught any of this explicitly, but perhaps I am remembering it all incorrectly.
I find it more and more puzzling the more I think about it.
To quote Katzko (2006) “(…) sound reasoning is as much a part of a scientist’s methodological toolkit as are procedures for data collection and analysis.” (p. 210).
I fear the possible emphasis on “statistical significance” and “p-values” in publishing may have also resulted in an emphasis on data-analysis and statistics in curricula. Possibly at both places at the cost of (sound) logic and reasoning and argumentation.
This relates to a thing I’ve been thinking about hypothesis testing for a while that I haven’t fleshed out particularly well yet. The core problem comes from the propositional logic formulation of material implication. If p then q. The problem is that there’s no copula between p and q. This sort of fundamental axiom allows for too much space between null and alternative hypotheses, and it’s impossible to test the relationship between antecedent and consequent in an automated way. “If the grass is green, the moon is made of cheese.” is formally valid as a proposition.
I should provide citations for this, but like I said, it’s not a fully formed idea. The pattern I have seen sometimes is something along the lines of:
NULL = if p then q
ALTERNATIVE = if ~p then q
*does research/study/data analysis*
Finding: p is false, therefore we believe that ~p implies q (or at least justifies further research funding). And of course, the press doesn’t help because they report something insane like ~p proves ~q beyond a shadow of a doubt.
To follow through with my ridiculous example it looks like this:
NULL: If the grass is green, the moon is made of cheese
ALTERNATIVE: if the grass is not green, the moon is made of cheese
Finding: the grass is brown, therefore brown grass causes the moon to be made of cheese
Reporters: grass color linked to edible moon rocks!
There’s everything wrong with this, obviously, but if you break it down into its component parts, the absurdity gets covered over rather quickly.
You have an observed phenomenon that you believe is real (moon exists, uncertainty about its makeup, probably a citation from a previous study that had nothing to do with cheese). You have what you think is a reasonable binary construction of null and alternative hypotheses (grass is green or not green). And you have a series of what seem like direct inferences based on a material implication of the null and alternative.
The fact that there are additional problems with buried assumptions about the observed phenomenon and accepting the alternative rather than failing to reject the null make this kind of example more realistic and less absurd.
I’m not trying to suggest that people are intentionally clownish. It’s very easy to get lost in the weeds when the probable truth value of p is complicated! And the rest is following the formula you learned in grad school about the chain of reasoning around p. The problem isn’t hypothesis testing here. The problem is the chain of logic around it. Inductive propositional calculus isn’t a strong enough system to drive the inferences we’re looking for. In other words, there is no chain.
Anyway, like I said, it’s not a particularly well-formed idea. My point is just that there’s more going on than just hypothesis testing. The bigger context that testing happens in is fundamentally problematic for the kinds of certainty that people crave.
Yes “set up your design and data collection to measure what you want to learn about” this is construct operationalization . Once you did the analysis one should consider what to do with the findings. This is action operationalization. Obviously communication of findings is a necessary steps. All these dimensions should be considered. They can be part of the design stage if your focus is on information quality.
A key lacuna is that statistical significance and p-values dependent on hot the findings are communicated. If for example verbal claims are used, in contrast to parametrized claims, methods like S-type error assessment are needed.
For more on these aspects of statistical analysis see https://sites.google.com/site/datainfoq
“This is the form of hypothesis testing and falsification that is valuable to me.” [falsifying “steel-man” null hypotheses]
Could you spell out how one would go about this, with user friendly (as simple as e.g. lmer og bme) software? I.e. falsifying a custom “steel-man” null hypothesis?
Say e.g. my null model is y ~ 5 + 1.2*x, based on previous research, instead of y ~ 0 + 0*x. I’d know how to test this against another competing model, using something like likelihood ratios or AIC. Or, I guess you could calculate the likelihood, perhaps fetch it from a bayesian model using mcmc.
But how would you simply falsify it, using p-values from off the shelf tools?
Hopefully i’m missing something obvious here. But perhaps one of the reasons it doesn’t get done more often is the practical difficulties of doing it?
Wouldn’t the frequentist answer be to calculate a p-value using the F statistic setting beta_0=5 and beta_1=1.2?
https://www.utstat.toronto.edu/~brunner/oldclass/302f14/lectures/302f14GeneralLinearTestWithR.pdf
Jens:
I agree there’s no general formula for constructing statistical models that embody scientific hypotheses. I prefer not to use the term “steel-man” for reasons discussed here; in any case, to see examples of the sorts of models I’m talking about, you can take a look at my applied research papers.
Good statistical workflow can be challenging—this is the main reason we write our books, to develop the ideas of statistical workflow and make them accessible to others—and, yeah, that difficulty is surely a big reason why it isn’t done more.
On the other hand, researchers will routinely put a year or more into conducting a study, gathering data, performing statistical analysis, and writing it all up, so I do think there’s room in this process for statistical tools that better fit to scientific models of interest.
There are many historical examples of people doing this. Eg, look up the 1919 eclipse observations: https://en.m.wikipedia.org/wiki/Eddington_experiment
The old theory predicted x = 0.87, the new one predicted x = 1.75. Check ref 2 from the wikipedia page to see the comparison between lines fit to the data and predicted by both theories.
Actually doing this comparison is the most trivial part of the whole exercise. The hard parts (where you will also find the most errors) are deriving predictions from the theories then going out and collecting the required data cleanly. Ie, logical and “systematic” error.
Those are the steps people are failing at absurdly high levels. They only derive some very vague prediction like x will increase/decrease y (which will be consistent with many possible explanations), collect some loosely related and perhaps confounded proxy for what they want to measure, and then test a strawman of no difference that was predicted by no theory.
The end result is a kind of bizarro science where:
1) Collective opinion determines the conclusions vs vice versa
2) The incentive is to have vague and useless theories vs precise predictions
3) Collecting better data makes it *easier* for the theory to avoid rejection rather than excluding a wider set of theories
I like the idea of getting your article reviewed and then backing out just before publication by my paying the APC.
However, I fear that if this became popular, the model would change. Authors would have to pay the APC up front and it would be refunded if the article was rejected.
The next stage, of course, would be for that refund to be less than 100%. I’ve already seen reports of journals testing the waters there with a nonrefundable “submission fee”, which comes down to the same thing. Apparently none of this goes to pay reviewers, though.
Nick:
No, you’re missing the point of my story! They asked me to write that article, and I wrote it. Their review, if there was any, didn’t provide any service for me. If I’d realized that they were expecting me to pay, I never would’ve sent them the article in the first place.
It’s quite common for journals in economics, finance, and accounting to charge submission fees. Often something like $50 but the American Economic Review charges $300 for non-members, the Journal of Accounting & Economics $650, and the Journal of Accounting Research $750 (except for authors of a small number of countries for which they charge less).
And our funders don’t reimburse these fees (contrary to APC). So it’s those in a well-funded department who can even afford to submit to these journals…