Researchers have taken photographs of children’s retinas and screened them using a deep learning AI algorithm to diagnose autism with 100% accuracy. The findings support using AI as an objective screening tool for early diagnosis, especially when access to a specialist child psychiatrist is limited.
AI-screened eye pics diagnose childhood autism with 100% accuracy::undefined
A convolutional neural network, a deep learning algorithm, was trained using 85% of the retinal images and symptom severity test scores to construct models to screen for ASD and ASD symptom severity. The remaining 15% of images were retained for testing.
It correctly identified 100% of the testing images. So it’s accurate.
100% accuracy is troublesome. Literally statistics 101 stuff, they tell you in no uncertain terms, never, never trust 100% accuracy.
You can be certain to some value of p. That number is never 0. .001 is suspicious as fuck, but doable. .05 is great if you have a decent sample size.
They had fewer than 1000 participants.
I just don’t trust it. Neither should they. Neither should you. Not at least until someone else recreates the experiments and also finds this AI to be 100% accurate.
What they’re saying, as far as I can tell, is that after training the model on 85% of the dataset, the model predicted whether a participant had an ASD diagnosis (as a binary choice) 100% correctly for the remaining 15%. I don’t think this is unheard of, but I’ll agree that a replication would be nice to eliminate systemic errors. If the images from the ASD and TD sets were taken with different cameras, for instance, that could introduce an invisible difference in the datasets that an AI could converge on. I would expect them to control for stuff like that, though.
I would expect them to control for stuff like that, though.
What was the problem with that male vs female deep-learning test a few years ago?
That all the males were earlier in the day, so the sun angle in the background was a certain direction, while all the females were later in the day, so the sun was in a different angle? And so it turned out that the deep-learning AI was just trained on the window in the background?
100% accuracy almost certainly means this kind of effect happened. No one gets perfect, all good tests should be at least a “little bit” shoddy.
Yeah, exactly. They’re reporting findings. Saying that it worked in 100% of the cases they tested is not making a claim that it will work in 100% of all cases ever. But if they had 30 images and it classified all 30 images correctly, then that’s 100%.
The article headline is what’s misleading. First, it’s poorly written - “AI-screened eye PICS DIAGNOSE childhood autism.” The pics do not diagnose the autism, so the subject of the verb is wrong. But even if it were rephrased, stating that the AI system diagnoses autism itself is a stretch. The AI system correctly identified individuals previously diagnosed with autism based on eye pictures.
This is an interesting but limited finding that suggests AI systems may be capable of serving as one diagnostic tool for autism, based on one experiment in which they performed well. Anything more than that is overstating the findings of the study.
They talk about collecting the images - the two populations of images were collected separately. It’s probably not 100% of the difference, but it might have been enough to push it up to 100%
You mean like the infamous AI model for detecting skin cancers that they figured out was simply detecting if there’s a ruler in the photo because in all of the data fed into it the skin cancer photos had rulers and the control images did not
You need to report two numbers for a classifier, though. I can create a classifier that catches all cases of autism just by saying that everybody has autism. You also need a false positive rate.
For ASD screening on the test set of images, the AI could pick out the children with an ASD diagnosis with a mean area under the receiver operating characteristic (AUROC) curve of 1.00. AUROC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUROC of 0.0; one whose predictions are 100% correct has an AUROC of 1.0, indicating that the AI’s predictions in the current study were 100% correct. There was no notable decrease in the mean AUROC, even when 95% of the least important areas of the image – those not including the optic disc – were removed.
They at least define how they get the 100% value, but I’m not an AIologist so I can’t tell if it is reasonable.
It correctly identified 100% of the testing images. So it’s accurate.
100% accuracy is troublesome. Literally statistics 101 stuff, they tell you in no uncertain terms, never, never trust 100% accuracy.
You can be certain to some value of p. That number is never 0. .001 is suspicious as fuck, but doable. .05 is great if you have a decent sample size.
They had fewer than 1000 participants.
I just don’t trust it. Neither should they. Neither should you. Not at least until someone else recreates the experiments and also finds this AI to be 100% accurate.
What they’re saying, as far as I can tell, is that after training the model on 85% of the dataset, the model predicted whether a participant had an ASD diagnosis (as a binary choice) 100% correctly for the remaining 15%. I don’t think this is unheard of, but I’ll agree that a replication would be nice to eliminate systemic errors. If the images from the ASD and TD sets were taken with different cameras, for instance, that could introduce an invisible difference in the datasets that an AI could converge on. I would expect them to control for stuff like that, though.
What was the problem with that male vs female deep-learning test a few years ago?
That all the males were earlier in the day, so the sun angle in the background was a certain direction, while all the females were later in the day, so the sun was in a different angle? And so it turned out that the deep-learning AI was just trained on the window in the background?
100% accuracy almost certainly means this kind of effect happened. No one gets perfect, all good tests should be at least a “little bit” shoddy.
Definitely possible, but we’ll have to wait for some sort of replication (or lack of) to see, I guess.
Yeah, exactly. They’re reporting findings. Saying that it worked in 100% of the cases they tested is not making a claim that it will work in 100% of all cases ever. But if they had 30 images and it classified all 30 images correctly, then that’s 100%.
The article headline is what’s misleading. First, it’s poorly written - “AI-screened eye PICS DIAGNOSE childhood autism.” The pics do not diagnose the autism, so the subject of the verb is wrong. But even if it were rephrased, stating that the AI system diagnoses autism itself is a stretch. The AI system correctly identified individuals previously diagnosed with autism based on eye pictures.
This is an interesting but limited finding that suggests AI systems may be capable of serving as one diagnostic tool for autism, based on one experiment in which they performed well. Anything more than that is overstating the findings of the study.
They talk about collecting the images - the two populations of images were collected separately. It’s probably not 100% of the difference, but it might have been enough to push it up to 100%
You mean like the infamous AI model for detecting skin cancers that they figured out was simply detecting if there’s a ruler in the photo because in all of the data fed into it the skin cancer photos had rulers and the control images did not
You need to report two numbers for a classifier, though. I can create a classifier that catches all cases of autism just by saying that everybody has autism. You also need a false positive rate.
True, but as far as I can tell the AUROC measure they refer to incorporates both.
Yup, you’re right, good catch 🙂
Then somebody’s lying with creative application of 100% accuracy rates.
The confidence interval of the sequence you describe is not 100%
From TFA:
They at least define how they get the 100% value, but I’m not an AIologist so I can’t tell if it is reasonable.
Yeah, from the way they wrote, it sounds to me they indirectly trained on the test set