An international correspondent who would like to remain anonymous writes:
I just read your blog post on autism junk science. I am a researcher focusing on autism. The post reminded me of two studies published, not in MDPI journals, but in JAMA series, claiming extraordinarily high diagnostic accuracy for autism.
The first study reports 100% sensitivity and specificity in differentiating autistic individuals from typically developing controls using retinal photographs and deep learning algorithms. As background information, the current gold standard diagnostic tool (ADOS) requires approximately two hours with a certified assessor, and even then, diagnosis remains challenging due to autism’s heterogeneity. A widely used screening tool for autism, social responsiveness scale, showed a sensitivity of ~0.85 and specificity of ~0.75. If the retinal photo were a valid diagnostic method, it would be ground-breaking and change every aspects of diagnosis of autism. However, a claim of 100% diagnosis accuracy from anyone (including a super-intelligent AI) would seem unrealistic to me, as the “diagnosis criteria” per se are still under debate. Also, the perfect alignment results appear spurious, as it is likely that there were some cases of misclassification of autism. A perfect alignment would require replication of human errors on these cases. Such perfect prediction, to me, may indicate complete case segregation on some hidden factors, like camera differences between the two groups, differences in background brightness, etc. In this study, autistic participants were recruited from a single center during a specific period, while controls were retrospectively selected from multiple centers. Though, more specific information on this “hidden factor” was not available and could only be speculative.
The second study, by the same research group, claims near-perfect diagnostic accuracy (area under curve > 0.99) using video recordings of a specific task combined with deep learning algorithms. While they reported standardized conditions for task performance and recording, the exceptional metrics remain concerning from a statistical perspective.
It is kind of annoying that when doing literature review on this topic, we generally have no good reason to omit these studies. However, inclusion of these studies appears to suggest that the issue has been solved, which is definitely not the case. I would appreciate your thoughts on should we think of or deal with such perfect metrics that passed peer review in respected journals.
I agree that these sorts of claims are hard to believe. Given that autism is a spectrum, how could you even hope to get 100% accuracy in diagnosis? Also, there’s this weird disconnect: in the abstract they claim perfect diagnosis on 958 participants, but then the discussion says, “The findings of this study suggest that retinal photographs may serve as a viable candidate for an objective method to screen for ASD.” If they can get perfect classification, this is big news, no? Not just “may suggest . . . a viable candidate . . .”
That said, the data from the study are available from the authors [ok, just kind of available; see this comment], so you could take a look yourself. The paper is from 2023 and it’s been cited a few times, for example here, where they say, “Recent studies, such as Kim et al. [60], propose that retinal photographs could serve as an objective screening tool for autism and may assist in assessing symptom severity.” It’s that disconnect again: it’s a citable study but I guess nobody, even the authors, really believes the 100% claim. Could be a good project for a machine learning student to look into.
Also, it’s kinda funny that “this study included 1890 eyes of 958 participants.” I guess their sample included a few pirates?
I’ve seen similar things with published studies of AirBnb pricing. Two peer-reviewed published studies each showed AUC > .99 for models predicting nightly prices as functions of various attributes of the properties. Of course, that is absurd. If a single entity were setting such prices, then a model with that kind of accuracy would be possible – it would mean that the algorithm used to set prices had been discovered. But given that thousands of hosts set prices for their properties (a number of which probably have little idea of why they priced the way they have), I would not want a model to achieve that kind of accuracy.
Unfortunately, the standards for tourism research seem to be even worse than other fields (social psych?). The studies in question were not well documented so it was hard to determine what happened. But one appeared to use the geocoordinates of each property as features in the predictive model, a clearly overfit model. The other had an elaborate scheme of creating a dozen different pricing tiers and fitting predictive models to each tier – which seems like a severe case of target leakage. The fact that peer reviewers and editors did not notice the near-perfect predictions as a red flag is not a testament to the quality of academic publishing.
I’d like to take a look at the data from the study in the post, but the link provided seems to be broken.
Dale,
I don’t know what’s up with that data link. From the URL it looks like it’s a link that automatically expires. I went back to the paper and found the data sharing statement which I will copy here as I fear it will also disappear:
So, it seems the data are only partly available. You need to contact the authors with a “reasonable request.”
I did find links to the data from the link in the email Andrew posted (it was Andrew’s link below that that did not work for me). However, given that I know nothing about this subject, and I am daunted by 19 level neural networks, and much of the material is translated from Korean, I won’t pretend to be able to analyze this study (I’m looking at the first study reference in the post). I did notice, however, that the AUC was 1.00 (confidence interval [1.0, 1.0]!) for diagnosis but 0.74 for severity of illness. The latter seems more reasonable. It makes me wonder how the diagnosis is determined. It wasn’t clear to me from the paper (which I admittedly do not understand) whether the initial diagnosis was done independently of the imaging or even how it was done. So, that seems like a suspicious step. If someone can parse through the paper or supplementary material to shed any light on this, I’d be most interested.
From what I gather, the autism and non-autism samples are not from a common sample. I suspect there is something unique about the autism sample that is not present in the non-autism sample that the machine learning process is picking up on, rather than “informative” (despite their efforts to remove “noninformative areas”).
^what Will said. I read this study when it came out, and if I recall correctly one group had existing retinal photos that were collected retrospectively and the other were prospectively photographed. My guess is their algorithm was unintentionally trained to detect which pictures came from the prospective camera/operator/observational conditions. As I recall they had perfect discrimination between the two samples but not very good discrimination for mild v. severe cases within the autistic sample. If anyone actually believes the algorithm is picking up something real between all autism and non-autism, but not able to distinguish mild from severe cases, I have a bridge to sell you. This is clearly a very well-trained model for predicting which data collection methodology was used and not for predicting autism.
People need to start ignoring all performance claims not based on a “lockbox” holdout. Even better you should publish paper A with you model, then collect new data, then report skill on that.
Cross validation is not to be trusted (its fine as a preliminary tuning step though), and frankly its unclear to me how this doesn’t become immediately obvious to researchers using ML.
I don’t comprehend at all how test A can be said to be better at a specific diagnosis than test B, when test B is referred to as “the gold standard” precisely because it’s the best test available, and test B is stated to be flawed.
The argument implies a test C that is better than test B and at least equivalent to test A. I don’t understand what that can be, other than “I know it when I see it.” Which doesn’t obviously rise to the level of scientific validity.
“Also, it’s kinda funny that “this study included 1890 eyes of 958 participants.” I guess their sample included a few pirates?”
Well, I don’t find this surprising. Retinal photographs are difficult to take: it requires the patient to hold constant gaze while a bright light is being beamed directly into his or her eyes, and sometimes the beam moves. If you’ve ever had your retinas photographed, you likely have had the experience of having to do one or more retakes. And sometimes, after several retakes, the prudent course is to just give up!
Of course, this doesn’t exclude the possibility that there may have been some in the study who have lost an eye as well. But I think procedure failure would be much more common than that.
Yes I just had this done, and I think that it would be very difficult for some people, possibly people with high sensitivity to light. It really doesn’t say much if anything about how they handled having two eyes from the same person or how strongly associated the measures for two eyes are.
We often exclude these case control or multi-gate (people with the condition identified from a different source than people without) studies from diagnostic test accuracy reviews. As the commenter notes, there’s a substantial risk of bias in these studies due to all kinds of “hidden factors”, as they put it.
So yes, I agree these kind of studies are very annoying but disagree that we “generally have no good reason to omit these studies” in a review.
As an aside, let’s avoid pirate jokes about people who have lost an eye please. (Although, as another commenter notes, this won’t be the primary reason for missing data here)