Harvey Motulsky writes:
You applied the term “forking paths” to refer to a form of researcher degrees of freedom.
It seems to me [Motulsky] that the name “forking paths” has not really taken off, partly because the publication has a very broad title, and partly because “garden of forking paths” is only memorable to someone who has read Borges.
But it occurred to me that a better term might be “multiple outs.” That is a term used by magicians to describe a trick where what the audience sees would be an incredible coincidence (without magic) had the trick ended as planned. But in fact the magician chose among multiple endings based on what happened along the way.
Here is one example (well the first part; it ends much more cleverly; skip the first minute).
I see what he’s getting at but I don’t like “multiple outs,” as to me it implies intentionality and misdirection (given the association with magicians). My point with forking paths is that it doesn’t have to happen on purpose!
Also I recommend this recent article, Forking paths and workflow in statistical practice and communication.
Strong agree – not only is it important to be able to refer to this as non-intentional because it can happen both ways, but it’s important because you want to be able to point to research practices that create problems with results separately from pointing to intentional misbehavior.
Having taught both biostatistics and a (separate!) course on “science and pseudoscience”, the latter of which included the concept of “multiple outs” in magic, astrology, palm reading, etc., I appreciate the connection Harvey Motulsky draws between that concept and “forking paths”– they are related.
I would add, though, that, while magicians are conscious of the role of multiple outs in their performances, there are “honest” readers (i.e. people who genuinely believe they are psychic or whatever) who rely on the “multiple out” to be successful, and don’t realize what they are doing. It is very easy for readers or their customers to find, via “free interpretation” and selective attention, a vindication of a non-specific claim in the variety of what goes on in the world. Jeane Dixon’s prediction that Richard Nixon would be elected in 1960, and that the new president would either die or be assassinated in office but not “necessarily” in the first term, was hailed as a startling success by the gullible. (I don’t know whether Dixon herself was a charlatan or an honest crank.)
So “multiple out” does not require intentional misbehavior. Nonetheless, I agree with Andrew that the “garden of forking paths” better reflects the sequence of decisions concerning design, analysis, etc., that face the researcher.
Gregory:
And the situation is not improved when purportedly skeptical journalists promote wacky ideas like magic radios and UFOs as space aliens. There’s also cross-promotion, for example the book by a fraud-tainted researcher that was endorsed by the professional skeptic and the big-ideas popularizer. At some point, excessive credulity can itself be a form of misbehavior, especially given that nowadays excessive credulity can be used to gain not just fame and fortune but also political power.
> better reflects the sequence of decisions concerning design, analysis, etc., that face the researcher
Agreed, and I think this is another virtue of the phrase “forking paths”—that it encourages researchers to realize that the path they followed is not necessarily the only one. As has been discussed on this blog for some time, one problem with how statistics is applied in the sciences is that most scientists have very limited exposure to statistics. They are often trained in a limited set of analyses and are channeled into doing what their advisor tells them or what is common in their literature. In the end, many scientists may not realize that, in doing so, they are implicitly making important choices about how they will eventually come to interpret their data.
The garden of forked paths is a purely philosophical explanation of real problems in the same sense that the “four humors” theory of disease is a purely philosophical explication of real diseases.
What can be advanced based on philosophy can be ignored based on philosophy.
Anon:
That’s kind of insulting of you to say that! I recommend you start by reading my paper with Loken. If after reading it you still think our idea is crap, that’s fine, you can explain your feelings more in detail. I don’t see what is gained to analogizing it to an archaic medical theory.
Regarding what “can be ignored based on philosophy,” I recommend my paper with Shalizi, where we write, “We are interested in philosophy and think it is important for statistical practice – if nothing else, we believe that strictures derived from philosophy can inhibit research
progress.”
Andrew, I’ve read both several times over the years.
The “garden of forked paths” is a philosophically derived claim that is merely consistent with the existence of real problems: just like the four humors theory is a philosophically derived theory merely consistent with real diseases (if you have fever that means you have to much “blood/warmth” humor and so on).
In your mind you’ve nailed the source of the problems since, after all, the problems are very real and the philosophical perspective makes perfect sense to you. But that doesn’t mean you have. I invite you to prove “the garden of forked paths” in a way that doesn’t succumb to a correlation-is-causation fallacy.
I wasn’t knocking philosophy, but merely pointing out that when your evidence is it makes philosophical sense to you, we’re free to ignore it with no more explication than it doesn’t make philosophical sense to me. And that’s why it’s not being adopted more.
I’m with you Andrew. I needed the above explanation of ‘multiple outs’ to understand it, but always found ‘forking paths’ intuitive and memorable – this despite not knowing where it came from until today. (The ‘garden of…’ clearly implied some cultural reference but doesn’t seem necessary, or does understanding that reference add something?)
The term “forking paths” is intuitive and it sounds nice. I had never heard of “multiple outs,” and pausing before reading the explanation I thought this must somehow correspond to baseball; I couldn’t figure out how multiple outs would map onto researcher degrees of freedom.
I also agree that the problem of forking paths in practice is that they’re typically taken *unintentionally,* not as deliberate misdirection.
The statistical ‘forking paths’ is the first in the list of popular culture references to the Wikipedia entry on Borge’s’ story – so unless you put it there yourself Andrew maybe some people are taking it up!
The solution here is for more people to read Borges.
Your explanation (and that of the other commentators) of why “forking paths” is better than “multiple outs” is persuasive. Although maybe it would make sense to teach using both analogies/terms.