Someone pointed me to this Vision Statement by Chris Chambers, a psychology professor who would like to edit Psychological Science, a journal that just a few years ago was notorious for publishing really bad junk science. Not as bad as PNAS at its worst, perhaps, but pretty bad. Especially because they didn’t just publish junk, they actively promoted it. Indeed, as late as 2021 the Association for Psychological Science was promoting the ridiculous “lucky golf ball” paper they’d published back in the bad old days of 2010.
So it does seem that the Association for Psychological Science and its journals are ripe for a new vision.
See here for further background.
Chambers has a 12-point action plan. It’s full of details about Accountable Replications and Exploratory Reports and all sorts of other things that I don’t really know about, so if you’re interested I recommend you just follow the link and take a look for yourself.
My personal recommendation is that authors when responding to criticism not be allowed to claim that the discovery of errors “does not change the conclusion of the paper.” Or, if authors want to make that claim, they should be required to make it before publication, a kind of declaration of results independence. Something like this: “The authors attest that they believe their results so strongly that, no matter what errors are found in their data or analysis, they will not change their beliefs about the results.” Just get it out of the way already; this will save everyone lots of time that might otherwise be spent reading the paper.
There’s a difference between saying “no matter what errors are found in the data, we won’t change our beliefs” when publishing and saying something like “there are errors in the data, but correcting them changes the magnitude of the main result from 3.0 to 2.9, so we still stand by our conclusions” after reanalysis.
Yes, but at the very least, the paper should go back through some sort of peer review, so that difference can be carefully scrutinized.
Adede:
In the case I’m thinking of, the t statistics went from 5.03 and 11.14 to 1.8 and 3.3.
Then perhaps “does not change the conclusion of the paper” is not the problem, but rather “does change the conclusion of the paper but we’re going to say it doesn’t and the editors are going to let that fly for some reason” is the real problem.
> Something like this: “The authors attest that they believe their results so strongly that, no matter what errors are found in their data or analysis, they will not change their beliefs about the results.” Just get it out of the way already; this will save everyone lots of time that might otherwise be spent reading the paper.
Or just publish the claim like linguists used to back in the day. Why bother with the experiment? Saves money and time. I am now with Chomsky on this; consult your intuition about what you believe is true, then write a paper about it. Why not? I’m not being facetious.
I assume the problem is that this does not generalize well to other people or even to one’s actual use. Benson Mates (in “On the Verification of Statements about Ordinary Language”, 165) makes your point in the other direction (bad idea in psychology → bad idea in linguistics):
Sebastian, I see the intuition-based methodology in linguistics as an instance of developing informative priors based on expert opinion. When syntacticians or semanticists have an intuition that this or that construction is good or not good or questionable, they are expressing an expert opinion As we know from expert opinion elicitation in Bayes (see the book Uncertain Judgements), there is bound to be disagreement between experts, so I am not bothered by the fact that Austin and Ryle don’t agree on something.
Here’s my argument against using empirical data in linguistics: if the empirical data and the statistical analyses reported don’t support the claim made in a paper, why bother reporting the data? Just present your claim, and do what Gelman suggests, begin the sentence with “I believe that”. The absence of empirical support holds, in my experience, in about 60-90% of published papers; it is truly rare to actually read a paper in which the statistical claim is actually supported by the data, assuming we take statistical theory seriously; sure, the p-values are below 0.05, but that’s after dozens of statistical tests and dozens not even reported—I have seen papers with 250+ statistical tests and an alpha of 0.05. The question I ask in such cases is: why not just report your belief, without the scaffolding of fictional empirical support? I’m actually OK with that.
Some 20+ years ago I moved away from intuition-based linguistics to empirical linguistics, not realizing that the intuition-based approach has something to say for it—it’s basically a process of informative prior generation.
There is a strong case to be made for expert linguists’ intuitions being sharper and more informed than those of the lay person, so that asking a lay subject for their judgement about this or that sentence and hoping that the average of n subjects’ judgements will give you something useful and interpretable is a pipe dream. This certainly holds for subtle phenomena like negative polarity licensing, where you really need a sharply honed intuition to figure out what is good and what is not. Or weak crossover; even anaphora resolution in some cases is subtle and difficult to unpack without expertise.
We take empirical data way too seriously. It would be OK if we did this [data collection+analysis] right, but we generally don’t, so we shouldn’t.
Shravan:
Indeed, pure theory can be good. The big problem is when researchers claim there’s empirical evidence when there isn’t. If people would just say, “I believe in ghosts” or “I believe that UFOs are space aliens” or “I believe in ESP” or “I believe that beautiful parents are more likely to have girls” or “I believe that himmicanes….,” without pretending they hae any good evidence, then I think the scientific enterprise would be much better off.