Machine learning models for diagnosing COVID-19 are not suitable for clinical use because of biases, methodological flaws, lack of reproducibility and “Frankenstein datasets”, researchers warned last night.
Researchers at Cambridge University, UK, carried out a systematic review of scientific manuscripts, published between 1 January and 3 October 2020, which described machine learning models that claimed to diagnose or prognosticate for COVID-19 from chest radiographs (CXR) and computed tomography (CT) images. Most of the papers had not been peer-reviewed.
They identified 2,212 studies, 415 of which were included after initial screening and, after quality screening, 62 studies were used in the systematic review, which is published in the latest edition of Nature Machine Intelligence.
None of the 62 models was of potential clinical use, they say.
Joint senior author Dr James Rudd, from Cambridge’s Department of Medicine said the early studies show promise, but “they suffer from a high prevalence of deficiencies in methodology and reporting, with none of the literature we reviewed reaching the threshold of robustness and reproducibility essential to support use in clinical practice”.
First author Dr Michael Roberts from Cambridge’s Department of Applied Mathematics and Theoretical Physics said any machine learning algorithm is only as good as the data it is trained on.
“Especially for a brand-new disease like COVID-19, it’s vital that the training data is as diverse as possible because, as we’ve seen throughout this pandemic, there are many different factors that affect what the disease looks like and how it behaves,” he said.
The research team report that many of the studies were hampered by issues with poor quality data, poor application of machine learning methodology, poor reproducibility, and biases in study design.
They point to several training datasets that used images from children for their “non-COVID-19” data and images from adults for their COVID-19 data.
“However, since children are far less likely to get COVID-19 than adults, all the machine learning model could usefully do was to tell the difference between children and adults, since including images from children made the model highly biased,” said Dr Roberts.
Many of the machine learning models were trained on sample datasets that were too small to be effective and many studies did not specify where their data had come from. Researchers also found that the models were trained and tested on the same data, or they were based on publicly available “Frankenstein datasets” that had evolved and merged over time, making it impossible to reproduce the initial results.
The researchers also say a lack of involvement from radiologists and clinicians was another problem.
“Whether you’re using machine learning to predict the weather or how a disease might progress, it’s so important to make sure that different specialists are working together and speaking the same language, so the right problems can be focused on,” added Dr Roberts.
However, the researchers say machine learning could be a powerful tool in combating the pandemic.
Roberts M, Driggs D, Thorpe M et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nature Machine Intelligence 15 March 2021.
Leave a Reply