Dorothy Bishop followed up on a story reported in the (London) Daily Telegraph as “Severe autism can be reversed, groundbreaking study suggests”—the headline was later changed to “Autism can be reversed, scientists discover” and then to “‘Autism can be reversed’, scientists claim.”
Bishop supplies some background:
The article, by D’Adamo et al., was published in a special issue of an MDPI journal, the Journal of Personalized Medicine. The special issue “A Personalized Medicine Approach to the Diagnosis and Management of Autism Spectrum Disorder: Beyond Genetic Syndromes” appears largely to be a vehicle for papers by the guest editor Richard E. Frye, who co-authored 3/4 editorials, 3/9 articles and 1/1 review in this collection. He was editor but not an author on the paper by D’Adamo et al, which is categorised as a “case report”.
Essentially, this is a description of development of a pair of non-identical twin girls who were diagnosed with autism at 20 months of age, along with various gastric and motor conditions, and were subsequently subjected to a remarkable list of interventions, viz:
· Parents worked with an autism parent coach who informed them about the link between “total allostatic load” and developmental disorders, a concept promoted by the Documenting Hope Project that one of the authors is affiliated with
· Parents accessed resources, including free webinars, through Epidemic Answers
· Parents took part in a parent forum called “Healing Together”
· Applied Behavior Analysis (ABA) from 22 mo to 33 mo
· Speech Therapy starting at 24 mo
· Rigorous diet and nutrition intervention eliminating sources of glutamate, following Reduced Excitatory Inflammatory Diet
· A strict gluten-free, casein-free diet that was low in sugar and additives
· Dietary supplements, including omega-3 fatty acids, multivitamins, vitamin D, carnitine, 5-methyltetrahydrofolate and “bio-individualized homeopathic remedies”
· Family consulted a naturopathic doctor who used IntellxxDNA genomics tool to recommend diet rich in tryptophan, vitamins B12, B6 and folate, betaine and choline, lion’s mane mushroom and resolvins, as well as some dietary variants specific to each twin.
· Neuro-sensory motor reflex integration given by an occupational therapist
· Environmental evaluation of the home for air quality, mould and moisture, culminating in a visit by a Building Biology Environmental Consultant who identified possible water damage.
· Cranial osteopathy, as recommended by a developmental optometrist.When first diagnosed, the girls were given an assessment called the Autism Treatment Evaluation Checklist, which was repeated after 18 months. Scores declined (i.e. improved) for both girls, with one twin improving substantially, and the other less so.
I haven’t been able to find any norms for the ATEC, but there’s a bit of data on trajectories in a study by Mahapatra et al (2018) and this shows that scores normally improve between the age of 2 and 3 years. We may also note that the ATEC is completed by a parent, and so may be influenced by parental expectations of improvement.
On this blog we sometimes talk about “bad science”—this would be work that generally follows the methods of science but has some serious flaws in design, measurement, or analysis, with the most common problem being measurements that are too noisy to allow any realistic effects to be discerned from the data—and we talk about “junk science,” which could be defined as bad science as a general practice. That is, we say that researchers are doing junk science when they set up a research program that allows them to produce a stream of bad science.
For example, the notorious beauty-and-sex-ratio paper is bad science—it’s a study that had no chance of revealing anything useful, it was just an exercise in noise-mining that then was published and hyped—, and I’d describe the series of papers on the general topic published by that researcher as being junk science, in that they were a systematic production of bad science studies. Similarly, the paper on governors’ life spans was bad science; the general approach of drawing big claims from statistical significance in regression discontinuity analysis is a particular form of junk science.
Beyond bad science and junk science is what we might call “fake science”: reports that have the superficial form of science but do not involve any aspects of scientific inquiry. This could include fake data, plagiarism, or completely empty word salad, or various versions of data misrepresentation, politically motivated hackwork, or extreme p-hacking.
Based on Bishop’s post and a look at the paper itself, I’d characterize this autism paper as fake science: It’s written up like a scientific paper and it’s published in a journal that at first glance looks like a scientific paper, it uses scientific jargon (“A parent-driven, multidisciplinary, therapeutic intervention involving a variety of licensed clinicians focusing primarily on addressing environmental and modifiable lifestyle factors”), but ultimately it does not seem to be a serious scientific investigation. But if you want to be more charitable you could just call it bad science, for the usual reason that it makes big claims that are not supported by the data at hand, and you could call it junk science, in that it’s part of this iffy journal issue that Bishop discusses.
As Bishop writes:
The proliferation of anecdotal reports in the literature gives ammunition to those who wish to promote all kinds of unevidenced treatments: they can point to these “peer reviewed” papers as evidence of scientific respectability. . . . every practitioner with a dubious product to sell will see this journal as a perfect outlet for their promotional material.
The biggest mystery to me is how this ended up in the Daily Telegraph. How did their reporter hear about this little paper published in an obscure journal? I guess that someone involved with the paper or the journal has an excellent publicity department.
Finally, I’m surprised that Dorothy Bishop, after reading the paper, “found it hard to believe just how flawed it turned out to be.” Bishop is a longstanding observer and critic of junk science. She’s seen so much bad stuff, that I’m surprised that she’s so surprised!
I agree with her that this sort of thing is horrible, both in itself and in how it degrades the general world of science and science communication. If people want to come up with theories, tell stories, and share anecdotes, they should do so without cloaking it in the language of science.
Isn’t “curing autism” technically just getting a psychiatrist to change their diagnosis? Like 99% of “autism” diagnoses would have been called something else (or nothing at all) 30 years ago.
Eg, say we have lots of money and want to sell access to a chatbot that cures autism. Can we start lobbying people involved in those NDA protected DSM-V update meetings to tweak the definition to be more specific, then show the inverse correlation between bot use and autism cases?
The really severe cases 1% of cases won’t even be able to interact with the bot, so also comparing autism rates in bot users vs not will show a huge benefit.
The bot could also have a catchphrase, like “BaZing!” that is likely to be repeated during psychiatric evaluations. This would make effective blinding impossible. Proper education (marketing/incentives) for the mental health providers can tip this opportunity for bias in the desired direction.
Where in the peer reviewed, statistical significance, literature is the quality control to prevent this?
I was thinking about the aspects of this work that make it fake science, including
(a) lack of theory (i.e., no connection between outcomes/interventions and any kind of general causal mechanisms by which they are supposed to operate);
(b) poor measurement (i.e., operationalizing “autism” using this ATEC measure and then equating “cure” with a reduction in symptoms according to that measure);
(c) lack of meaningful comparison (i.e., no norms or controls against which to compare outcomes); and
(d) poor sample selection (not just that it is a small sample, but also one that may involve many idiosyncracies that distinguish these participants from any broader population).
Of course, there’s plenty more flaws to point out with this specific work, as Bishop documents, but what struck me was how much of modern machine learning exhibits these same issues. Often, rather than justify a new architecture or training regime with some kind of general mathematical/computation theory, the authors seemingly throw stuff at the wall to see what sticks (a). Also, the reliance on “benchmarks” means that progress is assessed solely by performance on those benchmarks with little regard to how well common benchmarks operationalize some construct of interest (b). I guess machine learning involves lots of comparisons between competing models, so (c) may not be an issue as long as you find the chosen benchmarks to be appropriate. But I think that (d) remains an issue, because many benchmark data sets are convenience samples that do not conform to the properties of more general applied settings (see, e.g., issues with facial recognition on populations who are not Asian or Caucasian men, though perhaps there has been more recent progress on that particular front).
Anyway, I guess if I have a broader point to make here it is that technical sophistication alone doesn’t immunize someone against committing fake science.
Frye’s ‘research’ has been reported on already several years ago, e.g. https://forbetterscience.com/2021/11/15/gesundheit-israeli-scientists-treat-autism-with-stem-cells/
I also found an impressive 24 Pubpeer threads on his name.
“The biggest mystery to me is how this ended up in the Daily Telegraph.” This is one reason why the threshold for being criticised in StatsChat is some sort of mass media publication. There’s lots of bad/junk/fake science out there, as well as stuff that is fine for consenting scientists in private but not suitable for mass publication. It takes some sort of positive effort to end up in the newspapers.
In my experience the serious British papers are worse at science than the serious US papers, and The Telegraph is especially bad. A lot of the worst science stories in the NZ press are syndicated from the Telegraph.
If you search for the paper’s name on X, you will get some tweets from a few days before the Telegraph’s piece. So maybe (just maybe) it was a case of a journalist getting material from social media.
Uh, guys. Autism is a _political_ issue nowadays. So the right-wing media is on bad science with results the right likes like flies on what flies like.
So the bad science is done because there’s people who like it, and the trash media reports on it for the same reason.
Today’s news has it that a really nicely done study has convincingly demonstrated that aluminum in vaccines causes no problems whatsoever in children. And RFK Jr. is raving screaming mad and insisting the journal retract it. The journal refused.
https://www.msn.com/en-us/public-safety-and-emergencies/health-and-safety-alerts/aluminum-in-vaccines-study-won-t-be-retracted-journal-says/ar-AA1KkHMB
Speaking of right-wing stupidity, Coke has agreed to replace the corn sweetener it uses in US-sold Coke with cane sugar. Which is exactly as bad for you as corn sweetener. Well, that’s not as bad as the insistence on using animal-fat oils for deep frying instead of vegetable oils; that one really does make the fast food even worse for you.
The intersection between two of the major roads that define the roughly 1 km by 1 km by 1 km triangle I live at one corner thereof is a rather unique and funky neigborhood that features two classes of restaurants:
1) The type where you take your life in your hands by walking in the door, and
2) The type where you take your life savings in your hands by walking in the door
A TV special discussed this neighborhood the other day, and since it was TV, it only talked about famous people/things related to the neighborhood, so the “top 20” restaurants were all of type 2. One of which was a Tonkatsu place that uses animal fat oils for frying.
For photographs of a funky corner of this neighborhood (Arakicho*) see my photos starting at:
https://pbase.com/davidjl/image/175627860
Click “next” at the upper right until you get to the end, then stop.
(The shrine and pond at the end of the set are nice.)
*: https://en.wikipedia.org/wiki/Arakich%C5%8D
Its a standard case of relaxed definition -> create a crisis of cases. But this one has become a great boon for politicians as well.
Once an intervention is devised that allows enough stakeholders to take a cut, this will be reversed. At the same time the intervention is introduced there will also be something like a blood test (or maybe MRI). Then the thresholds will get more stringent, then two positive tests required, etc. Eventually, the rates will have stabilized at a much lower level and everyone can feel progress has been made.
If you look closely you’ll see this pattern is practically a cliche.
“I find the Law of Fives to be more and more manifest the harder I look.”
Pick one. It isn’t all, but this is very common. It works great if you only look at superficial level.
Yes this is getting out of hand and it wouldn’t surprise me if we start to see a more formalized structure around what already is the case less formally (in STEM subjects anyway) namely that there is a perceived stable of quality locations that pukka researchers aim to publish in and review for, and the vast amounts of trash (eminently ignorable if advancing science is your aim) are ignored.
In time some of this junk stuff will die to the extent that funders, invested industry and governments with some pretension to research quality withdraw support and focus on quality, but it won’t go away completely.
This is happening already. The Wellcome Trust and EU have set up journals where researchers they fund can publish without open access charges, the implication being that if they are comfortable awarding grants to researchers they are likely to be comfortable with the quality of the resulting research (still has to pass peer review). Dorothy Bishop has written about this in fact (Google: “New publishing models will only work if authors embrace them”). Some journals have been shut down because their publisher were increasingly shamed by the junk/disreputable practices…
There is a new Forensic Scientometrics (FoSci) initiative (read their Forensic Scientometrics (FoSci) Paris Declaration by Googling), that aims to consolidate efforts to identify and exposing junk – Dorothy Bishop is a signatory.
IMO opinion some of the issues around difficulties in reproducing published research (again in STEM subjects) would be eased if there was a publishing model where authors could update their papers (e.g. with “Methodological Adenda”) with information about new reagents, methods that replace those described in their original paper but no longer work (e.g. because a manufacturer has changed the coatings on the multiwell plates used in the original paper). That’s a different issue that the one raised here but it’s part of the same story..
You had me at “cranial osteopathy, as recommended by a developmental optometrist.”
Those dietary restrictions remind me of a guy on the M60 bus, 20 years ago, loudly telling his date why autism rates were so high: “Too much sugar and not enough discipline!” No idea what his professional background was.