Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy
Abstract
Background
Coronavirus disease 2019 (COVID-19) has widely spread all over the world since the beginning of 2020. It is desirable to develop automatic and accurate detection of COVID-19 using chest CT.
Purpose
To develop a fully automatic framework to detect COVID-19 using chest CT and evaluate its performance.
Materials and Methods
In this retrospective and multicenter study, a deep learning model, the COVID-19 detection neural network (COVNet), was developed to extract visual features from volumetric chest CT scans for the detection of COVID-19. CT scans of community-acquired pneumonia (CAP) and other non-pneumonia abnormalities were included to test the robustness of the model. The datasets were collected from six hospitals between August 2016 and February 2020. Diagnostic performance was assessed with the area under the receiver operating characteristic curve, sensitivity, and specificity.
Results
The collected dataset consisted of 4352 chest CT scans from 3322 patients. The average patient age (±standard deviation) was 49 years ± 15, and there were slightly more men than women (1838 vs 1484, respectively; P = .29). The per-scan sensitivity and specificity for detecting COVID-19 in the independent test set was 90% (95% confidence interval [CI]: 83%, 94%; 114 of 127 scans) and 96% (95% CI: 93%, 98%; 294 of 307 scans), respectively, with an area under the receiver operating characteristic curve of 0.96 (P < .001). The per-scan sensitivity and specificity for detecting CAP in the independent test set was 87% (152 of 175 scans) and 92% (239 of 259 scans), respectively, with an area under the receiver operating characteristic curve of 0.95 (95% CI: 0.93, 0.97).
Conclusion
A deep learning model can accurately detect coronavirus 2019 and differentiate it from community-acquired pneumonia and other lung conditions.
© RSNA, 2020
Summary
Deep learning detected coronavirus disease 2019 (COVID-19) and distinguished it from community-acquired pneumonia and other non-pneumonia lung diseases using chest CT.
Key Results
■ A deep learning method was able to identify coronavirus disease 2019 on chest CT scans (area under the receiver operating characteristic curve, 0.96).
■ A deep learning method was able to identify community-acquired pneumonia on chest CT scans (area under the receiver operating characteristic curve, 0.95).
■ There was overlap in the chest CT imaging findings of all viral pneumonias with other chest diseases that encourages a multidisciplinary approach to the final diagnosis used for patient treatment.
Introduction
Coronavirus disease 2019 (COVID-19) has widely spread all over the world since the beginning of 2020. It is highly contagious and may lead to acute respiratory distress or multiple organ failure in severe cases (1–4). On January 30, 2020, the outbreak was declared a “public health emergency of international concern” by the World Health Organization.
The disease is typically confirmed with reverse-transcription polymerase chain reaction (RT-PCR) testing. However, it has been reported that the sensitivity of RT-PCR might not be high enough for the purpose of early detection and treatment of patients presumed to have COVID-19 (5,6). CT, as a noninvasive imaging approach, can depict certain characteristic manifestations in the lung associated with COVID-19 (7,8). Therefore, CT could serve as an effective way for early screening and diagnosis of COVID-19. Despite its advantages, CT may demonstrate similar imaging features between COVID-19 and other types of pneumonia, thus making it difficult to differentiate.
Recently, artificial intelligence using deep learning technology has demonstrated great success in the medical imaging domain due to its high capability of feature extraction (9–11). Specifically, deep learning was applied to detect and differentiate bacterial and viral pneumonia on pediatric chest radiographs (12,13). Attempts have also been made to detect various imaging features of chest CT (14,15).
Herein, we propose a three-dimensional deep learning framework to detect COVID-19 using chest CT, referred to as the COVID-19 detection neural network (COVNet). Scans of community-acquired pneumonia (CAP) and other non-pneumonia lung disease were included to test the robustness of the model.
Materials and Methods
Patients
This retrospective study was approved by the ethics committees of the participating hospitals. The need to obtain written informed consent for participation in this study was waived. Our study includes 4563 three-dimensional (3D) volumetric chest CT scans from 3506 patients acquired at six medical centers between August 16, 2016, and February 17, 2020 (Fig 1). The exclusion criteria included (a) contrast material–enhanced CT scans and (b) scans with slice thickness greater than 3 mm. After applying the exclusion criteria, 4352 3D chest CT scans from 3322 patients were selected for this study (Fig 1). The average patient age (±standard deviation) was 49 years ± 15, and there were slightly more men than women (1838 vs 1484, respectively; P = .29). CT scans with multiple reconstruction kernels at the same imaging session or acquired at multiple time points were included. Of the 4352 scans in the final dataset, 1292 (30%) were obtained for COVID-19, 1735 (40%) for CAP, and 1325 (30%) for non-pneumonia abnormalities. All cases of COVID-19 were confirmed as positive with RT-PCR and were acquired from December 31, 2019, to February 17, 2020.
Figure 1: Flow diagram. A dataset of 3506 patients with chest CT scans was collected. After exclusion, 3322 eligible patients were included for model development and evaluation. CT scans were extracted from Digital Imaging and Communications in Medicine (DICOM) files. The dataset was split into a training set (to train the model) and an independent testing set at the patient level. A supervised deep learning framework (COVNet) was developed to detect coronavirus 2019 (COVID-19) and community-acquired pneumonia. The predictive performance of the model was evaluated by using an independent testing set. COVNet = COVID-19 detection neural network.
The median period from the symptom onset to the first chest CT examination was 7 days (range, 0–20 days). The most common symptoms were fever (81%, 381 of 468 patients) and cough (66%, 309 of 468 patients). For each patient, one or multiple CT examinations were performed at several time points during the course of the disease (average number of CT examinations per patient, 1.8; range, 1–6).
Patients with CAP and other non-pneumonia abnormalities were randomly selected from the participating hospitals between August 16, 2016, and February 17, 2020. The admission distribution of the patients with CAP was as follows: outpatient, 46% (713 of 1551); inpatient, 38% (588 of 1551); emergency, 3% (40 of 1551); and physical examination, 3% (46 of 1551). Of the 1551 patients with CAP, 210 received laboratory confirmation of the origin: 112 were confirmed to be bacterial culture positive and 98 were confirmed to be negative (65 mycoplasma and 31 viral pneumonia).
Patients with other non-pneumonia abnormalities had no lung disease (n = 510) or had lung nodules (n = 552), chronic inflammation (n = 264), chronic obstructive pulmonary disease (n = 115), or other diseases (n = 14, including pneumothorax and tracheal diverticulum).
Patients with CAP and other non-pneumonia abnormalities were randomly selected from the participating hospitals between August 16, 2016, and February 17, 2020. All scans were randomly split with a ratio of 9:1 into a training set and an independent testing set at the patient level. The training dataset was then further split for training the model and internal validation (10% of the samples). The independent testing set was not used for training and internal validation. The patient demographic statistics are summarized in Table 1. The patient disease statistics are summarized in Table 2.
|
|
Imaging Protocol
CT scans were obtained with equipment from different manufacturers by using standard imaging protocols. Each volumetric scan contains 51–1094 CT slices with a varying slice thickness of 0.5–3 mm. The reconstruction matrix was 512 × 512 pixels with in-plane pixel spatial resolution ranging from 0.29 × 0.29 mm2 to 0.98 × 0.98 mm2. Please refer to Table E1 (online) for details.
Deep Learning Model
We developed a 3D deep learning framework for the detection of COVID-19, referred to as COVNet (Fig 2). It is able to extract both two-dimensional local and 3D global representative features. The COVNet framework consists of RestNet50 (16) as the backbone, which takes a series of CT slices as input and generates features for the corresponding slices. The extracted features from all slices are then combined by using a max-pooling operation. The final feature map is fed to a fully connected layer and softmax activation function to generate a probability score for each type (COVID-19, CAP, and non-pneumonia).
Figure 2: Coronavirus disease 2019 (COVID-19) detection neural network (COVNet) architecture. COVNet is a convolutional neural network that uses ResNet50 as the backbone. It takes as input a series of CT slices and generates a classification prediction of the CT image. The convolutional neural network features from each slice of the CT series are combined by a max-pooling operation and the resulting feature map is fed to a fully connected layer to generate a probability score for each class. CAP = community-acquired pneumonia.
Specifically, given a 3D CT scan, we first preprocess it and extract the lung region as the region of interest by using a U-net (17) based segmentation method. The preprocessed image is then passed to our COVNet for the predictions. The code to reproduce the results in this study is available at https://github.com/bkong999/COVNet.git. Please refer to Appendix E1 (online) for the details.
Statistical Analysis
The performance of our trained deep learning model was evaluated by using an independent testing set, which was not used during model development. Statistical analysis was performed with using R software (version 3.6.3). Analysis of variance tests and χ2 tests were used to compare differences among different groups for continuous and dichotomous variables, respectively. Two-sided P < .05 was considered to indicate a statistically significant difference. The sensitivity and specificity for both COVID-19 and CAP were calculated. The receiver operating characteristic curve was plotted and the area under the curve was calculated with the 95% confidence intervals (CIs) based on the method used by DeLong et al (18).
Results
Study Population Characteristics
Table 1 lists the study population characteristics for the training and independent testing datasets. There are slightly more male patients than female patients across COVID-19, CAP, and non-pneumonia groups for both the training dataset (percentage of male patients: COVID-19, 53% [212 of 400 patients]; CAP, 56% [782 of 1396 patients]; non-pneumonia, 55% [645 of 1173 patients]; P = .55) and the independent testing dataset (percentage of male patients: COVID-19, 52% [35 of 68 patients]; CAP, 62% [96 of 155 patients]; non-pneumonia, 52% [68 of 130 patients]; P = .17). Patients in the COVID-19 and CAP groups were older than those in the non-pneumonia group for the training dataset (mean age: COVID-19, 53 years; CAP, 51 years; non-pneumonia, 41 years; P < .001) and the independent testing dataset (mean age: COVID-19, 52 years; CAP, 53 years; non-pneumonia, 41 years; P < .001).
Model Performance
Once the deep learning model is trained, the processing time for a new testing examination is very fast. The average processing time for each CT examination was 4.51 seconds with a workstation (GPU NVIDIA Quadro M4000 [Nvidia, Santa Clara, Calif], 8GB, RAM 16GB, and Intel Xeon Processor E5–1620 v4 @3.5GHz).
The performance of COVNet in the detection of COVID-19 and CAP is summarized in Table 3. The sensitivity and specificity for COVID-19 are 90% (114 of 127 scans; 95% CI: 83%, 94%; P < .001) and 96% (294 of 307 scans; 95% CI: 93%, 98%; P < .001), respectively. For the detection of CAP, our deep learning model yielded a sensitivity of 87% (152 of 175 scans; 95% CI: 81%, 91%; P < .001) and a specificity of 92% (239 of 259 scans; 95% CI, 88%, 95%; P < .001). The receiver operating characteristic curves are shown in Figure 3. The corresponding areas under the receiver operating characteristic curves for COVID-19 and CAP are 0.96 (95% CI: 0.94, 0.99) and 0.95 (95% CI: 0.93, 0.97), respectively.
|
Figure 3a: Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic curve of the algorithm (black curve) on the independent testing set for (a) coronavirus disease 2019 (COVID-19), with area under the receiver operating characteristic curve (AUC) of 0.96 (P < .001), (b) community-acquired pneumonia, with area under the receiver operating characteristic curve of 0.95 (P < .001), and (c) non-pneumonia, with area under the receiver operating characteristic curve of 0.98 (P < 0.001). Gray region indicates the 95% confidence interval.
Figure 3b: Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic curve of the algorithm (black curve) on the independent testing set for (a) coronavirus disease 2019 (COVID-19), with area under the receiver operating characteristic curve (AUC) of 0.96 (P < .001), (b) community-acquired pneumonia, with area under the receiver operating characteristic curve of 0.95 (P < .001), and (c) non-pneumonia, with area under the receiver operating characteristic curve of 0.98 (P < 0.001). Gray region indicates the 95% confidence interval.
Figure 3c: Receiver operating characteristic curves of the model. Each plot illustrates the receiver operating characteristic curve of the algorithm (black curve) on the independent testing set for (a) coronavirus disease 2019 (COVID-19), with area under the receiver operating characteristic curve (AUC) of 0.96 (P < .001), (b) community-acquired pneumonia, with area under the receiver operating characteristic curve of 0.95 (P < .001), and (c) non-pneumonia, with area under the receiver operating characteristic curve of 0.98 (P < 0.001). Gray region indicates the 95% confidence interval.
To improve the interpretability of our model, we adopted the gradient-weighted class activation mapping, or Grad-CAM, method (19), to visualize the important regions leading to the decision of the deep learning model. Such a localization map is fully generated by the model without additional manual annotation. Figure 4 shows the heatmap to the suspected regions for the examples of COVID, CAP, and non-pneumonia abnormalities. These heatmaps illustrate that our algorithm paid most attention to the abnormal regions while ignoring the normal-like regions as shown in the non-pneumonia abnormality example.
Figure 4: Representative examples of attention heatmaps generated by using the gradient-weighted class activation mapping, or Grad-CAM, method for, A, coronavirus disease 2019 (COVID-19), B, community-acquired pneumonia, and, C, non-pneumonia. Heatmaps are standard Jet colormaps and overlapped on the original image. The red color highlights the activation region associated with the predicted class.
In addition, Figure 5 shows one representative example of a case of CAP that was misclassified as COVID-19 and Figure 6 shows another example of a COVID-19 case that was misclassified as CAP. The consecutive slices around the abnormality are shown from left to right on each row. It seems that these are very challenging cases. It might be useful to include the history of exposure to further improve the accuracy.
Figure 5: Representative example of a case of community-acquired pneumonia misclassified as coronavirus disease 19. The consecutive slices around the abnormality are shown from left to right.
Figure 6: Representative example of case of coronavirus disease 2019 misclassified as community-acquired pneumonia. The consecutive slices around the abnormality are shown from left to right.
Discussion
In this study, we designed and evaluated a three-dimensional deep learning model for detecting coronavirus disease 2019 (COVID-19) from chest CT scans. On an independent testing data set, we showed that this model achieved high sensitivity (90% [95% confidence interval [CI]: 83%, 94%]) and high specificity (96% [95% CI: 93%, 98%]) in the detection of COVID-19. The areas under the receiver operating characteristic curves for COVID-19 and community-acquired pneumonia were 0.96 (95% CI: 0.94, 0.99) and 0.95 (95% CI: 0.93, 0.97), respectively.
In this study, a 3D deep learning framework was proposed for the detection of COVID-19. This framework can extract both two-dimensional local and 3D global representative features. Deep learning has achieved superior performance in the field of radiology (10–13). Previous studies have successfully applied deep learning techniques to detect pneumonia on pediatric chest radiographs and further to differentiate viral and bacterial pneumonia on two-dimensional pediatric chest radiographs (12,13). We were able to collect a large number of CT scans from multiple hospitals, which included 1292 COVID-19 CT scans. More importantly, 1735 CAP and 1325 non-pneumonia CT scans were also collected as the control groups in this study to ensure the detection robustness considering that certain similar imaging features may be observed in COVID-19 and other types of lung diseases.
COVID-19 has widely spread all over the world since the first case was detected at the end of 2019. Early diagnosis of the disease is important for treatment and to isolate the patients to prevent the virus spread. RT-PCR is considered the reference standard; however, it has been reported that chest CT could be used as a reliable and rapid approach for screening of COVID-19 (5,6). The abnormal CT findings in COVID-19 have been recently reported to include ground-glass opacification, consolidation, bilateral involvement, and peripheral and diffuse distribution (1–5,7,8).
This study has several limitations. First, COVID-19 is caused by a corona virus and may have similar imaging characteristics as pneumonia caused by other types of viruses. However, due to the lack of laboratory confirmation of the origin for each of these cases, we were not able to select other viral pneumonias for comparison in this study. Instead, we randomly selected CAP from August 2016 to February 2020. We propose that our sampling method was sufficient for the typical distribution of various subtypes of CAP. These cases should have included non–COVID-19 viral pneumonias (eg, influenza virus), bacterial pneumonia, and organizing pneumonia from any cause. It would be desirable to test the performance of COVNet in distinguishing COVID-19 from other viral pneumonias that have real-time polymerase chain reaction confirmation of the viral agent in a future study. Second, a disadvantage of all deep learning methods is the lack of transparency and interpretability (eg, it is impossible to determine what imaging features are being used to determine the output). While we used a heatmap to visualize the important regions in the scans leading to the decision of the algorithm, heatmaps are still not sufficient to visualize what unique features are used by the model to distinguish between COVID-19 and CAP. Third, there is a great deal of overlap in how the lung responds to various insults and a significant amount of overlap in the presentation of many diseases in the lung that depend on host factors (eg, age, drug reactivity, immune status, underlying comorbidities). No one method will be able to differentiate all lung diseases based simply on the imaging appearance on chest CT scans. A multidisciplinary approach for this is recommended. Furthermore, although a large amount of data were collected in this study, the test set came from the same hospitals as the training set. We plan to collect additional CT scans from different centers to evaluate its performance in the near future. Finally, this study focused on whether one scan is COVID-19; it did not address the categorization of the disease into different severities. As a next step, it would be important to not only predict the presence of COVID-19, but also the severity degree to further help monitor and treat patients.
In conclusion, a robust deep learning model is developed to differentiate coronavirus disease 2019 (COVID-19) and community-acquired pneumonia (CAP) from chest CT scans. These results demonstrate that a machine learning approach using a convolutional networks model has the ability to distinguish COVID-19 from CAP.
Author Contributions
Author contributions: Guarantors of integrity of entire study, L.L., L.Q., Y.Y., G.W., Jun Xia; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, Y.Y., X.W., J.B., Y.L., G.W., Jun Xia; clinical studies, L.L., L.Q., D.L., G.W., Jun Xia; experimental studies, Y.Y., X.W., B.K., J.B., Y.L., Z.F., G.W.; statistical analysis, Y.Y., X.W., Y.L., G.W.; and manuscript editing, L.L., L.Q., Y.Y., X.W., B.K., J.B., Y.L., Q.S., K.C., G.W., Jun Xia
References
- 1. . Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. Lancet 2020;395(10223):507–513. Crossref, Medline, Google Scholar
- 2. . Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China. JAMA 2020 Feb 7 [Epub ahead of print]. Crossref, Google Scholar
- 3. . Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus-Infected Pneumonia. N Engl J Med 2020 Jan 29 [Epub ahead of print]. Crossref, Google Scholar
- 4. . First Case of 2019 Novel Coronavirus in the United States. N Engl J Med 2020;382(10):929–936. Crossref, Medline, Google Scholar
- 5. . Correlation of Chest CT and RT-PCR Testing in Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology 2020 Feb 26:200642 [Epub ahead of print]. Link, Google Scholar
- 6. . Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR. Radiology 2020 Feb 19:200432 [Epub ahead of print]. Link, Google Scholar
- 7. . CT Imaging Features of 2019 Novel Coronavirus (2019-nCoV). Radiology 2020;295(1):202–207. Link, Google Scholar
- 8. . Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020;395(10223):497–506 [Published correction appears in Lancet 2020;395(10223):496.]. Crossref, Medline, Google Scholar
- 9. . A Multi-modality Network for Cardiomyopathy Death Risk Prediction with CMR Images and Clinical Information. In: Shen D, Liu T, Peters TM, , eds. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science, vol 11765. Cham, Switzerland: Springer, 2019; 577–585. Crossref, Google Scholar
- 10. . Learning tree-structured representation for 3D coronary artery segmentation. Comput Med Imaging Graph 2020;80:101688. Crossref, Medline, Google Scholar
- 11. . Precise diagnosis of intracranial hemorrhage and subtypes using a three-dimensional joint convolutional and recurrent neural network. Eur Radiol 2019;29(11):6191–6201. Crossref, Medline, Google Scholar
- 12. . Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell 2018;172(5):1122–1131.e9. Crossref, Medline, Google Scholar
- 13. . Visualization and interpretation of convolutional neural network predictions in detecting pneumonia in pediatric chest radiographs. Appl Sci (Basel) 2018;8(10):1715. Crossref, Medline, Google Scholar
- 14. . Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution computed tomography. Invest Radiol 2015;50(4):261–267. Crossref, Medline, Google Scholar
- 15. . Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging 2016;35(5):1207–1216. Crossref, Medline, Google Scholar
- 16. . Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. Google Scholar
- 17. . U-net: Convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A, eds. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Cham, Switzerland: Springer, 2015; 234–241. https://doi.org/10.1007/978-3-319-24574-4_28. Crossref, Google Scholar
- 18. . Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44(3):837–845. Crossref, Medline, Google Scholar
- 19. . Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, 2017. Google Scholar
Article History
Received: Mar 7 2020Revision requested: Mar 12 2020
Revision received: Mar 13 2020
Accepted: Mar 19 2020
Published online: Mar 19 2020
Published in print: Aug 2020
