In a post entitled, “Just make it stop! When will we say that further research isn’t needed?,” Dorothy Bishop writes:
On Friday we had a great presentation from Lottie Anstee who told us about her Masters project on handedness and musicality. There have been various studies on this topic over the years, some claiming that left-handers have superior musical skills, but samples have been small and results have been mixed. Lottie described a study with an impressive sample size (nearly 3000 children aged 10-18 years) whose musical abilities were evaluated on a detailed music assessment battery that included self-report and perceptual evaluations. The result was convincingly null, with no handedness effect on musicality.
Interesting. Handedness is a funny area of research: the idea of correlation between handedness and various abilities makes a lot of intuitive sense; we are exposed to lots of anecdotal evidence on lefthandedness; the topic lends itself to theorizing; it’s cheap to gather data on handedness, but there are not a lot of good public data sources. Put all this together and you get a literature full of suggestive but shaky findings based on noisy measurements and small samples. We discuss an example in pages 220-222 of Active Statistics.
What I’m saying is, it’s plausible to me that there could be interesting systematic differences between lefties and righties (beyond differences in ability to write, draw, throw, etc., with the non-dominant hand), and it’s also plausible that there are no other stable differences between these groups of people.
Bishop continues the story:
What happened next was what always happens in my experience when someone reports a null result. The audience made helpful suggestions for reasons why the result had not been positive and suggested modifications of the sampling, measures or analysis that might be worth trying . . . perhaps a more nuanced measure would reveal an association? Should the focus be on skilled musicians rather than schoolchildren? Maybe it would be worth looking at nonlinear rather than linear associations? And even though the music assessment was pretty comprehensive, maybe it missed some key factor – amount of music instruction, or experience of specific instruments.
After a bit of to and fro, I [Bishop] asked the question that always bothers me. What evidence would we need to convince us that there is really no association between musicality and handedness? The earliest study that Lottie reviewed was from 1922, so we’ve had over 100 years to study this topic. Shouldn’t there be some kind of stop rule?
Bishop summarizes:
My own view is that further investigation of this association would prove fruitless. In part, this is because I think the old literature (and to some extent the current literature!) on factors associated with handedness is at particular risk of bias, so even the messy results from a meta-analysis are likely to be over-optimistic. . . .
I [Bishop] suspect that most of the exciting ideas about associations between handedness and cognitive or personality traits are built on shaky foundations, and would not replicate if tested in well-powered, preregistered studies. But somehow, the idea that there is some kind of association remains alive, even if we have a well-designed study that gives a null result.
I agree 100% on that last point regarding meta-analysis.
And she concludes:
I hope that as preregistration becomes more normative, we may see more null results getting published, and learn to appreciate their value. But I wonder just what it takes to get people to conclude that a research seam has been mined to the point of exhaustion.
It’s an interesting question. One useful way to look at it, I think, is to recognize that each of us can make our own decision of when to give up on a field. Bishop can give up on studies of handedness and behavior, but this shouldn’t stop others from continuing to study it. Major funders can stop paying for research on this topic, but researchers can continue to work on it using their own funds, etc. Even cold fusion still gets some funding—it’s the whole high-risk, high-reward thing. In answer to Bishop’s question, “Shouldn’t there be some kind of stop rule?,” my response is that each of us has our own stop rule, and that’s a good thing.
To put it another way, we should be open to the possibility that a research area is a dead end. The history of science is full of wrong turns, many of which have happened during our professional lifetimes.
P.S. As a fun bonus, follow the link to see Bishop expressing disappointment with the theories of Noam Chomsky:
As someone who works on child language disorders, I [Bishop] have tried many times to read Chomsky in order to appreciate the insights that he is so often credited with. I regret to say that, over the years, I have come to the conclusion that, far from enhancing our understanding of language acquisition, his ideas have led to stagnation, as linguists have gone through increasingly uncomfortable contortions to relate facts about children’s language to his theories.
Good stuff, including some debate in the comment section.
There is something sinister about left handed research
-1 too obv
A bit gauche
all these comments are a bit too far out in left field.
You’re right
As a researcher, the key element is wanting to make a difference and having several options. If you have a real problem or set of problems, you can quickly discard hypotheses that seem unpromising. I recently did a two year ~$1M project and concluded that the direction wasn’t going anywhere. Not that it couldn’t exactly, but that it wasn’t. So I finished. At the same time, I was doing something else that looks to have a real impact, and I transitioned that to other people. Great, two things I don’t need to do anymore! One will make a positive impact.
If there is really exactly zero correlation between handedness and musical skill, that would be the biggest discovery in the history of psychology/sociology (whatever label this falls under).
It would be like proving ghosts exist, or similar, and have foundational ramifications for fields diverse as history and physics.
Anon:
In the human sciences there are no true zeros (see page 960 here). What’s possible is not that the correlation between handedness and X is zero but that the correlation is typically near zero, that its sign and magnitude vary a lot across settings and over time, and that it can’t be measured with much accuracy.
The world is full of effects that exist but are too small and variable and hard to measure to be interesting, at least not without further theory.
Of course, it is often shorthand. But the statistical test (they are undoubtedly using) is for the pedantic interpretation.
And I am not sure this distinction was recognized in the OP, ie:
There are also the two meanings of “null” causing further confusion, that have been conflated (but not by Fisher!) as far back as 1940:
https://upload.wikimedia.org/wikipedia/commons/a/a9/Origins_Of_Hybrid_Hypothesis_Testing.png
Along these lines, rather than the issue being framed as “yes there’s a correlation”/”no there isn’t”, it would be more useful for results such as Antsee’s to be presented as “the credible range for the effect size under these conditions is -x to +y”. This leaves room for the legitimate belief (consistent with the data) that there is some effect, identifies that the effect is probably too small to be interesting, and shows that even if it were interesting it would take an even larger study to distinguish it from zero. “There’s no effect” does none of this.
It’s easy to do a study and get a null result: just use a small sample. Tightly constrained effect sizes are a much larger contribution to the scientific knowledge base, whether or not the constraint includes zero.
John:
That makes sense. But then researchers need to understand that a “credible range” constructed via a non-regularization approach (for example, a least squares estimate +/- 2 standard errors) can often (a) include values that are not at all credible, and (b) exclude values that are credible.
For a simple example, recall the beauty-and-sex-ratio paper that estimated beautiful parents to be 8 percentage points more likely to have girls, with a standard error of something like 3 percentage points. The standard 95% credible range (or Bayesian interval with weak prior) then goes from 2 percentage points to 14 percentage points, a range which in this case is entirely composed of incredible values. From existing knowledge (see here), any credible value for this parameter would have an absolute value of much less than 0.5 percentage points.
This is the huge problem pointed out by Meehl in his famous 1967 paper. It inverts the entire logic of science along with the incentive structure (yielding what I call “bizarro science”):
Paul Meehl (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115. https://meehl.umn.edu/sites/meehl.umn.edu/files/files/074theorytestingparadox.pdf
The “null hypothesis” (hypothesis to be nullified) must correspond to *your hypothesis* for science to work.
I think that rather than a statement during a seminar that some research topic is hopeless or dead, what happens is that fewer people choose to study that topic; it slowly fades away.
More speculative: Since it’s the more astute people who are first to realize that the topic is pointless and abandon it, the average quality of research on that topic goes down — a feedback loop accelerating the decline. (However, there seem to be a lot of pointless topics and low-quality work whose longevity amazes me.)
More Meehl:
Paul Meehl (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834. https://meehl.umn.edu/sites/meehl.umn.edu/files/files/113theoreticalrisks.pdf
cf. the quote about Chomsky at the end. Let’s hope that era of pointless research is coming to an end!
I was a linguistic, but I was never a Chomskyan. My first NSF grant proposal, circa 1990, was rejected for, I kid you not, being “too European” (i.e., not Chomskyan). Nevertheless, Chomsky’s ideas permeated the whole field, including those with whom he waged the so-called Linguistic Wars (this field is so messed up that referees ask you to remove citations from people who disagree with Chomsky).
The crux of the problem is Chomsky’s distinction between competence and performance. It basically makes the field anti-empirical. During my last linguistics talk (at NYU circa 2000), I was talking about using subword speech recognizers for exploratory data analysis, and two professors were so offended they wouldn’t let me finish until I admitted what people say or write has no bearing on linguistic theory. It’s the only talk I ever walked out of!
So I abandoned the whole field. It’s a shame, because it’s a great subject.
Chomsky wasn’t the only problem. I worked mainly in semantics, which let me largely avoid the devout Chomskyans. The problem is that field was permeated by equally crackpot Frege/Russell/Tarski-style logical positivist theories of meaning (e.g., Lewis, Kripke, and Montague on possible worlds, etc.). You’d think linguists would be up on the full arc of the Linguistic Turn in philosophical semantics, but sadly they weren’t in the early 00s. I was told by a famous semanticist (Barbara Partee) that I could work on problems with real meaning, like metaphor, but only while on sabbatical after tenure, because nobody would take it seriously.
I often wonder where the field’s gotten to. Probably not far. I was at a Berkeley LLM meeting where a Chomskyan took the success of LLMs to imply they’d learned Chomsky’s theory of language! You really can’t make this stuff up.
Yes, Chomsky seemed anti-empirical to me. He described how children learn language, and it did not match my own experiences. What were his views based on? It is hard to tell.
The blog is from 2012. I assume that LLMs have changed everyone’s view about how language works. Everyone except Chomsky.
We knew a lot from psycholinguistics, so it didn’t change my idea of how language works. For example, my job talk in 1989 was about how humans incorporate all channels of information (phonetics, phonology, morphology, syntax, semantics, pragmatics, contextual knowledge, world knowledge, etc.) simultaneously and in real time and how we could start thinking about computer programs that could do the same thing. I was working in a paradigm of logical constraint solving then, but I was still thinking about how humans process language (this was still the “computational linguistics” era of NLP). The attention architecture underlying most LLMs is the closest thing I know to simulating how people process language, except perhaps the RNN technology they replaced.
I don’t think there’s any consensus among linguists (many different subfields), philosophers, cognitive scientists, etc. concerning the implications of LLMs for an understanding of how language works. I don’t think anyone (including Chomsky) disputes that they are impressive feat of engineering. And I think they demonstrate that a lot more is available via “text” that anyone thought possible before. But the crux of the issue for many has always been whether the operations that LLMs perform are or are not like those underlying language acquisition and use in human beings. It’s a promising field of inquiry, but I think it’s pretty speculative at this point.
I don’t know. Linguistics is a pretty big field, and while Chomsky and his followers dominated certain areas, there were always alternative approaches (Labovian sociolinguistics, cignutive linguiistics, corpus linguistics, not to mention fields like computational linguistics, psycholinguistics, etc.) Again, not really my dield, but my understanding is that the Language Wars was an in-house civil war, so of course there were some commonalities.
I think the problem is that Chomsky just had an extremely narrow research project that for some reason got an extreme amount of attention from proponents and critics alike. He was never particularly interested in most aspects of language: I haven’t studied his work extensively (and not in decades) but I think his main focus was on what aspect of the language system had to be encoded on the human genome because it couldn’t be learned in any other way. That has to be extremely small, and if you take a liberal approach to the project, narrowing it down through empirical research counts as success, not a failure.
Chomsky had an irritating way of saying linguists shouldn’t be concerned with anything other than the part of the language system encoded on the genome, which strikes me as silly. But if you situate his statements in relation to his research project, they at least make some sense, even if it turns out that language-specific part of the genome shrinks to nothing.
(I’m not denying your experiences rightly turned you off the field: it just might be you got unlikely in that you wound up in a theoretical linguistics circle instead of computational linguistics)
There are also Chomsky-favoring linguists who do empirical validation of Chomskyan theories (acceptability rating studies or the like). Sometimes called Empirical Syntax. It’s a bit of a shaky field though; e.g., Likert scales (1-7) are modeled as being generated from a Gaussian distribution, underpowered studies, arguing for the null after getting a p-value larger than 0.05, confirmation bias where every researcher magically always finds evidence for their favorite position, heavy p-hacking, non-reproducible analyses, the usual stuff that happens in psychology. The empirical turn in linguistics seems to me to have become a way to anoint one’s a priori beliefs by decorating them with some numerical values. A nice takedown of some MIT semanticist’s attempt to find an interaction is discussed in this paper (which Bob will understand the contents of, it’s about antecedent contained deletion, which is right up his street):
Gibson, E., Piantadosi, S. T., & Levy, R. (2017). Post hoc analysis decisions drive the reported reading time effects in Hackl, Koster-Hale & Varvoutis (2012). Journal of Semantics, 34(3), 539-546.
This kind of thing happens surprisingly often: a (psycho)linguistics paper makes a claim based on data, but closer inspection reveals that it’s a nothingburger. Which raises the question: weren’t we better off using Chomsky’s original methodology of using intuition to come up with empirical generalizations?
Paul and Ringo don’t appear to be more or less musical than John and George. Albert King was not inferior to Freddie King, and Hendrix confounds the question more. Beethoven, Chopin, Mozart, and Rachmaninov were pretty good as well. Where did the question about handedness and musicality arise? My mother was a lefty and suffered much correction in school; she did learn to live in a righty world as a seamstress and taylor. Unavailability of left handed sewing machines was a problem for her. I knew a lefty soldier who felt disadvantaged by the military only providing rightie M16s. Perhaps if we had lefty violins and french horns our results might differ.
Perhaps the effect is limited to the historical period in which folk were forced to be right-handed?
There was once a theory that the human brain consists of a lizard brain inside an ape brain inside a human brain, with an underpinning of neurological justification based upon brain architecture. But as more was learned about that architecture, the theory became less and less tenable, until finally, poof! And now there is a frozen corpus of psychology literature based upon that theory that nobody reads anymore. So theories at least CAN die.
But handedness is different. Lateralization of brain architecture strongly suggests that handedness SHOULD matter in various types of performance. Why shouldn’t it matter that the language half of your brain either does or does not have to convince the opposite, non-language hand to write the sentences down? The burden of proof has shifted to proving the negative, a common trait of zombie ideas.
Oops, not “opposite, non-language hand” but rather the non-language hand controlled by the opposite half of the brain.
Lateralization of brain architecture strongly suggests that handedness SHOULD matter in various types of performance.
Ironically, from what I’ve read, much of the thinking on the outcomes of “lateralization of brain architecture” seems a bit zombie-ish.
Has anyone taken a survey of world-class orchestra musicians?
If you survey mathematicians and physicists, I believe you’ll find a much higher prevalence of lefties than in the general population. I work at an institute full of math and physics professor types and the prevalence of lefties is so high that it’s really noticeable. I’m guessing at least double the population prevalence of lefties.
Has anyone ever established a convincing math-music connection? I’m decent at math but terrible at music. Mitzi’s left handed and her right hand is much more intelligent than my left hand. Most instruments require two hands. Mitzi’s also great at the piano. Is it because she’s left handed or because her father was a world-class orchestra musician and she was surrounded by talented musicians growing up? Or both?
Bob:
1. Yes, that could be with left-handers and math; it’s just tricky to know because of data quality. There could be some careful studies out there that Bishop wasn’t familiar with.
2. I do think it’s a well-established result that righties are much more right dominant and lefties are much more mixed handed. It’s my understanding that left-handedness is typically not the opposite of right-handedness but rather a sign of less lateralization. I seem to recall that animals don’t show much of a dominant hand preference.
If I were to take a survey of all 200 or so math-adjacent scientists working here at Flatiron Institute and find that 20% of them are left-handed, how convincing is that? The two-sided p-value assuming a binomial null at the population rate (10%) is something like 10^-5. Maybe we just have a left-handed hiring bias?
Allow me to rephrase my original response to this question.
To gauge the effect of handedness on musicality don’t we have to acknowledge that we live in a right handed world? The outcome of a study on this topic might differ if there were equal numbers of left handed tools like my mother’s wished for left handed sewing machine or the soldier’s left handed M16 (Remington does make lefty guns for civilians.) A world with numbers of left handed violins, guitars, french horns, etc has to be created before the question can be answered.
Simply looking for correlations between handedness and musical talent seems very weak science. Much is known about the underlying neurobiology and good science involves studying plausible mechanisms underlying putative correlations before looking for said correlations. If there aren’t any such plausible mechanisms, resources shouldn’t be wasted simply scanning for correlations. That’s fine in marketing research but it’s very far from serious science.