Millions of patients could be at risk because of “poor quality” and “exaggerated” studies that claim artificial intelligence (AI) is as good as – or better than – human experts at interpreting medical images, according to an analysis published today.
Writing in The BMJ, researchers from Imperial College London say their review of published studies over the past ten years, comparing the performance of a deep learning algorithm in medical imaging with expert clinicians, found methods and risk of bias that have not been examined.
The team found two eligible randomised clinical trials and 81 non-randomised studies. Of the non-randomised studies, nine were prospective and six were tested in a “real world” clinical setting.
They found that the average number of human experts in the comparator group was four, while access to raw data and code, which allows independent scrutiny of results, was severely limited.
Out of the 81 non-randomised studies, 58 were categorised as high risk of bias and the team also highlighted often poor adherence to recognised reporting standards.
Sixty-one of the studies stated that performance of AI was at least comparable to, or better than, that of clinicians, and 31 said further prospective studies or trials were needed.
While the team says there were some limitations, such as the possibility of missed studies and the focus on deep learning medical imaging studies, nevertheless, they say “many arguably exaggerated claims exist about equivalence with (or superiority over) clinicians, which presents a potential risk for patient safety and population health at the societal level”.
They also warned that overpromising language “leaves studies susceptible to being misinterpreted by the media and the public, and as a result the possible provision of inappropriate care that does not necessarily align with patients’ best interests”.
The researchers conclude by saying that having a high quality and transparently reported evidence base will maximise patient safety.
Nagendran M, Chen Y, Lovejoy CA et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies in medical imaging. BMJ 26 March 2020.
Leave a Reply