Next Article in Journal
Buffy Coat Processing Impacts on Monocytes’ Capacity to Present Lipid Antigens
Previous Article in Journal
A Brief Review of Duchenne Muscular Dystrophy Treatment Options, with an Emphasis on Two Novel Strategies
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Biomedicines
Volume 11
Issue 3
10.3390/biomedicines11030831
biomedicines-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Rapid Assessment of COVID-19 Mortality Risk with GASS Classifiers

by
Salvatore Greco
1,2,†,
Alessandro Salatiello
3,†,
Nicolò Fabbri
4,
Fabrizio Riguzzi
5,
Emanuele Locorotondo
6,
Riccardo Spaggiari
1,
Alfredo De Giorgi
7,* and
Angelina Passaro
1,8,9
1
Department of Translational Medicine, University of Ferrara, Via Luigi Borsari 46, 44121 Ferrara, Italy
2
Department of Internal Medicine, Ospedale del Delta, Via Valle Oppio 2, 44023 Ferrara, Italy
3
Section for Computational Sensomotorics, Department of Cognitive Neurology, Centre for Integrative Neuroscience & Hertie Institute for Clinical Brain Research, University Clinic Tübingen, Otfried-Müller-Straße 25, 72076 Tübingen, Germany
4
Department of General Surgery, Ospedale del Delta, Via Valle Oppio 2, 44023 Ferrara, Italy
5
Department of Mathematics and Informatics, Via Nicolò Machiavelli 30, 44121 Ferrara, Italy
6
Radiology Department, University Radiology Unit, Hospital of Ferrara Arcispedale Sant’Anna, Via Aldo Moro 8, 44124 Ferrara, Italy
7
Clinica Medica Unit, Azienda Ospedaliero-Universitaria S. Anna of Ferrara, Via Aldo Moro 8, 44124 Ferrara, Italy
8
Medical Department, University Hospital of Ferrara Arcispedale Sant’Anna, Via A. Moro 8, 44124 Ferrara, Italy
9
Research and Innovation Section, University Hospital of Ferrara Arcispedale Sant’Anna, Via A. Moro 8, 44124 Ferrara, Italy
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Biomedicines 2023, 11(3), 831; https://doi.org/10.3390/biomedicines11030831
Submission received: 20 December 2022 / Revised: 10 February 2023 / Accepted: 3 March 2023 / Published: 9 March 2023
(This article belongs to the Topic Health Informatics and Epidemiological Data Analysis in COVID-19 Based Internet of Medical Things (HIEDA-COVID19-IOMT))
Browse Figures
Versions Notes

Abstract

:
Risk prediction models are fundamental to effectively triage incoming COVID-19 patients. However, current triaging methods often have poor predictive performance, are based on variables that are expensive to measure, and often lead to hard-to-interpret decisions. We introduce two new classification methods that can predict COVID-19 mortality risk from the automatic analysis of routine clinical variables with high accuracy and interpretability. SVM22-GASS and Clinical-GASS classifiers leverage machine learning methods and clinical expertise, respectively. Both were developed using a derivation cohort of 499 patients from the first wave of the pandemic and were validated with an independent validation cohort of 250 patients from the second pandemic phase. The Clinical-GASS classifier is a threshold-based classifier that leverages the General Assessment of SARS-CoV-2 Severity (GASS) score, a COVID-19-specific clinical score that recently showed its effectiveness in predicting the COVID-19 mortality risk. The SVM22-GASS model is a binary classifier that non-linearly processes clinical data using a Support Vector Machine (SVM). In this study, we show that SMV22-GASS was able to predict the mortality risk of the validation cohort with an AUC of 0.87 and an accuracy of 0.88, better than most scores previously developed. Similarly, the Clinical-GASS classifier predicted the mortality risk of the validation cohort with an AUC of 0.77 and an accuracy of 0.78, on par with other established and emerging machine-learning-based methods. Our results demonstrate the feasibility of accurate COVID-19 mortality risk prediction using only routine clinical variables, readily collected in the early stages of hospital admission.

1. Introduction

In late 2019, a new member of the coronavirus family, named Severe Acute Respiratory Syndrome CoronaVirus-2 (SARS-CoV-2), emerged in the Chinese province of Hubei [1] and rapidly spread worldwide, causing the first pandemic by a coronavirus. As of January 2023, the virus has caused more than 670 million cases of infection, and has led to over 6.7 million deaths in more than 200 countries [2].
COVID-19 (COronaVIrus Disease-19), the clinical manifestation of SARS-CoV-2 infection, is undoubtedly the most important global health concern this century has witnessed and has been taking a heavy toll in terms of human, financial, and social resources. Moreover, the appearance of new SARS-CoV-2 strains and the lagging vaccination campaigns hinder the objective of reaching herd immunity, implying that such issues will likely persist in the coming months. Hospitals are particularly affected, especially during periods of increased disease transmission, when the number of patients that simultaneously need treatment increases exponentially. Such waves of patients can quickly deplete hospital resources if not dealt with optimally.
In these circumstances, it is fundamental to estimate the amount of resources incoming COVID-19 patients will likely require during their hospital stay; ideally, this assessment would be made in the early phases of hospital admission. This would ensure that high-risk patients have access to adequate resources and that, in general, hospital resources are not misdirected. General risk scores and comorbidity indices, such as the Charlson Comorbidity Index (CCI) [3], have been shown to be helpful in estimating the prognosis and thus the level of care that COVID-19 patients require [4,5]. Similarly, pneumonia-specific risk scores such as the CURB-6 [6] and the CURB-65 [7] scores have been shown to have even greater prognostic ability [8,9]. Nevertheless, both kinds of risk scores tend to underperform clinical risk scores that are specifically targeted to COVID-19 patients [10,11,12]. This is in part due to the remarkable variability across patients of the SARS-CoV-2 manifestations.
The SARS-CoV-2 infection manifests itself heterogeneously: symptoms can range from being completely absent (up to 33% of total cases [13]) to being critical (up to 5% of total cases [14]), in which case they include respiratory failure, high fever, hyperinflammation, and multiorgan dysfunction. The percentage of critical cases increases substantially in the cohorts of hospitalized COVID-19 patients, reaching peaks close to 20% [15]; such percentages are particularly high in older patients with coexisting morbid conditions and cardiovascular diseases.
To move beyond the limitations of general risk scores, in this work, we introduce two COVID-19-specific classifiers: the Clinical-GASS and the SVM22-GASS (GASS = General Assessment of SARS-CoV-2 Severity; SVM = Support Vector Machine). Both classifiers identify high-risk patients by predicting their 30-day mortality outcome and were developed and validated using independent derivation and validation cohorts. The SVM22-GASS classifier is based on a SVM model and is built in a fully data-driven fashion exploiting state-of-the-art machine learning methods. The Clinical-GASS builds on a COVID-19 specific risk score we recently introduced and which proved effective at stratifying the population of hospitalized COVID-19 patients: the GASS score [10].

2. Materials and Methods

2.1. Study Design and Participant Recruitment

This is an observational cohort study, developed in the two hospitals of Ferrara’s territory dedicated to COVID inpatients, “Arcispedale S.Anna” in Cona (Fe) and “Ospedale del Delta” in Lagosanto (Fe). The province of Ferrara is geographically located in the Eastern part of the Emilia-Romagna region of Italy, with a population of approximately 350,000 inhabitants, and it is characterized by a high presence of elderly subjects (~26% of total population is aged >65 years, and nearly 1% >90 years).
This study analyzes the data of two independent cohorts of COVID-19 patients: a derivation cohort—used to develop the classifiers—and a validation cohort—used to validate them. In the machine learning community, such data are commonly referred to as training and test set, respectively. The derivation cohort comprises data of 499 patients recruited between March 2020 and June 2020, which have been already partially described and analyzed in our previous study [10]. The validation cohort comprises data of 250 patients recruited between September 2020 and March 2021; in this period, 450 patients were admitted to our departments but the data relative to 200 subjects were not complete for all the relevant variables and were thus discarded.
The collected variables consist of demographic, anamnestic, and laboratory data. The anamnestic data comprise: previous history of smoking, hypertension, ischemic heart disease, heart failure, chronic kidney disease, stroke or TIA (Transient Ischemic Attack), peripheral arterial disease (PAD), chronic obstructive pulmonary disease (COPD), hepatopathy, cancer, dementia, and diabetes. The laboratory data comprise: white blood cell count (WBC), lymphocyte count, creatinine, C Reactive Protein (CRP), procalcitonin, fibrinogen, D-dimer, isoamylase, alanine transferase (ALT), creatine phosphokinase (CPK), lactic dehydrogenase (LDH), ferritin, brain natriuretic peptide (BNP), and HS Tropoin I (HS TnI). All the variables were retrieved from the patients’ electronic health records with their consent. Baseline symptoms and vital signs were added to their records and used to calculate both their CCI score (Charlson Comorbidity Index), and the GASS score. Note that the variables on which the CCI and GASS scores are based are defined and explained in previous studies [3,10], and are reported in the Supplementary Tables S1 and S2.
SARS-CoV-2 infection was confirmed with the reverse transcriptase-polymerase chain reaction (RT-PCR) test. The exclusion criteria were the negativity of the swab tests to viral detection and age (patients younger than 18 years were excluded). All patients signed an informed consent designed specifically for the purpose of this study; in case the patients were unable to sign, the consent was obtained from their legal representatives.
Importantly, all data were collected at the first hospital visit or blood examination.

2.2. Descriptive Analyses

We evaluated the differences between subjects in terms of the three major COVID-19 outcomes (1. IoC = intensification of care, meant as the need for non-invasive mechanical ventilation or for endotracheal intubation; 2. in-hospital death; 3. 30-day death). The observation period was protracted until the 30th day after hospital admission for those patients who survived the hospitalization: this was possible using records of the local registry office linked to the hospital information system. Patients needing IoC can be recognized by the acronym “IoCp” while, for those who did not undergo IoC, we chose the acronym “nIoCp”. The groups of patients who survived or died after a period of observation of 30 days are presented with the acronyms “30-ddp” (30-day deceased patients) and “30-dsp” (30-day survived patients), respectively.

2.3. Statistical Analysis

Data analyses were performed by using SPSS 26.0 software (IBM SPSS Statistics, IBM Corporation) and MATLAB (MATLAB 2020a, The MathWorks, Natick, MA, USA). The normal distribution of the continuous variables was analyzed using Kolmogorov–Smirnov and Shapiro–Wilk tests. Variables not normally distributed were analyzed using non-parametric tests. Categorical variables were summarized by using frequencies and percentages, and continuous data were presented as median (interquartile range, IQR). The Mann–Whitney U test was used for continuous variables, while the χ2 test was used for categorical variables. Variables with a p value < 0.05 in the univariate analyses were used to perform multivariate logistic regression analyses. All p values < 0.05 are considered statistically significant.

2.4. The Clinical-GASS Classifier

In our previous study [10], we defined a new clinical score, the GASS score, and discussed its ability to stratify the population of hospitalized COVID-19 patients into groups with significantly different 30-day mortality risk. Such a stratification has the potential to improve patient care by signaling to the hospital personnel the patients who are likely to need a mild (GASS < 6), moderate (6 ≤ GASS ≤ 10), or high (GASS > 10) level of care. However, clinical personnel might also benefit from knowing whether the patient at hand has a low or a high risk of dying and is likely to require specialized treatment such as the admission to the intensive care unit. This can be accomplished by building a binary classifier that predicts, for example, the 30-day mortality outcome. This is the primary outcome measure we aim to predict with this, and the other classifiers considered in the following sections. To assess the potential of the GASS score to be used for such a binary classification task, we built a simple threshold-based classifier. The classifier—named Clinical-GASS—was designed by simply computing the optimal Receiver Operating Characteristic (ROC) threshold on the training set, in terms of the cost/benefit ratio [16]; equal costs were assumed for the misclassification of survivors and non-survivors. Patients with a GASS score below such a threshold are classified as likely survivors while patients with a GASS score above such a threshold are considered as likely non-survivors. A similar approach was used to build the Charlson Comorbidity Index (CCI) classifier from the CCI score. The CCI is a clinical score that is often used as a baseline method to estimate the mortality risk in the general population.

2.5. The SVM22-GASS Classifier

Both the Clinical-GASS and the CCI classifiers are built leveraging medical expertise and thus might have a strong appeal to clinicians: they are likely to have full understanding of how the classifiers work, to trust their decisions, and to thus be willing to use them. However, both classifiers only use a limited number of features (Clinical-GASS: n = 11, CCI classifier: n = 19) and combine them with only sums and other simple operations defined by piecewise constant functions. For these reasons, both classifiers are potentially unable to fully exploit useful features contained in the patients’ health records and their non-linear interactions, which might be predictive of survival outcome.
To address this issue, we used a Support-Vector-Machine-based classifier (SVM) with Radial Basis Function (RBF) kernel [17]. An RBF-SVM is a binary non-linear classification method that classifies data by non-linearly mapping the feature vectors into an infinite dimensional space in which data from different groups can be easily separated by a hyperplane. In brief, data 𝑥 are classified by the decision function 𝑓(𝑥), which is given by
f ( x ) = ∑ i = 1 N α i k   x i , x
Here, N is the number of support vectors, 𝛼𝑖 are weighting coefficients, and 𝑘( , ) is the kernel function. In our case, we chose the Gaussian Radial Basis Function:
k ( y , z ) = e x p − | | y − z | | 2 2 σ 2
Therefore, the decision function f(x) becomes:
f ( x ) = ∑ i = 1 N α i e x p − | | x i − x | | 2 2 σ 2
The SVM classifier was trained using only the data contained in the training set, was tested on the independent test set, and was compared against the CCI and GASS classifiers. In the remainder of this article, we will refer to the SVM classifier trained on our training set as SVM22-GASS.

2.5.1. Data Preprocessing

Before training the SVM classifier, we made sure to have sound training data. This was accomplished in two steps. First, we excluded from the analysis the variables that were measured in less than 50% of the patients. Secondly, we imputed the remaining missing values with the weighted average of the values of the three most similar instances, with weights inversely proportional to the distances from these. As a distance measure, we used the Euclidean distance for continuous features and the Hamming distance for binary ones.

2.5.2. Data Augmentation

Learning a classifier from imbalanced training data such as ours (% survivors = 75.7 %) is a challenging task and can lead to poor sensitivity to the minority class. To deal with this issue, we balanced the training set using the Synthetic Minority Over-sampling Technique (SMOTE). We used this specific method because of its proved effectiveness and simplicity [18]. In brief, this method works by generating synthetic instances of the minority class by linearly combining samples of the minority class. Specifically, for each minority class sample s one can generate h synthetic samples by first randomly sampling h of the k nearest neighbors, and then randomly perturbing s along the directions of the difference vectors between s and the h samples. In this algorithm, k is a hyperparameter, while h depends on the number of synthetic samples to generate.

2.5.3. Feature Selection

To remove potentially redundant and uninformative features, while reducing the computational cost of training the SVM model, we performed feature selection; this was accomplished by learning a regularized logistic regression model with LASSO penalty [19,20]. The regularization parameter λLASSO, which determines the regularization strength and thus the number of selected features, was chosen by minimizing the ten-fold cross-validation deviance. Note that we chose this specific feature selection method due to its well known cost-effectiveness, its ability to deal with models with both continuous and categorical variables, and its strong performance exhibited in a previous study [10].

2.5.4. Parameter Training

To learn the model’s parameters, we minimized the hinge loss using the Limited-memory Broyden–Fletcher–Goldfarb–Shanno (LBFGS) solver [21]. Importantly, to speed up training and reduce memory requirements, we approximated the Gaussian kernel using the Fastfood [22] random feature expansion method. To deal with potential residual classification biases due to the class imbalance, we adjusted the SVM classification threshold [23]. Specifically, we used the SVM classification scores to compute the optimal operating point of the ROC curve and used the resulting threshold to classify the data.

2.5.5. Hyperparameters Optimization

Before learning the final model’s parameters, to preselect a class of SVM models suitable for the dataset at hand, we performed hyperparameter optimization [24] using Bayesian optimization [25]. Specifically, with this procedure we found the kernel scale σ2, the regularization strength λSVM, and the dimensionality of the random features space δ, that minimize the five-fold cross-validation hinge loss. Note that we used Bayesian optimization rather than simpler methods such as random or grid search because this method tends to be less sensitive to the choice of the hyperparameter ranges. Furthermore, it generally requires a lower expected number of calls to the cost function, which reduces the computational cost.

2.5.6. Model Evaluation

To evaluate the quality of the model’s predictions, we computed the Area Under the ROC Curve (AUC), accuracy, sensitivity, and specificity on the independent test set. The SVM’s performance is then compared to that of the other classifiers of interest, namely, the CCI classifier and the GASS classifier. As a baseline model, we also considered the majority classifier, which assigns the majority class to every instance in the dataset, regardless of its features. Importantly, the majority classifier is patient agnostic: it does not use the data associated with a specific patient to determine the risk class. Thus, comparing our classifiers to this baseline model allows us to quickly assess whether they are able to competently extract useful information from patient data, rather than only exploiting the class distribution of the training set.

2.5.7. Model Interpretation

As it is often the case with most high-dimensional and non-linear machine learning models, also in our case, once the SVM model is fully trained, it is challenging to gain an intuitive understanding of the mechanism underlying the classifier’s decision process. To shed some light on such a mechanism, we estimated the variable importance (VI) [26] of each input feature of the SVM model, by computing the model reliance. In brief, such a method assigns high reliance to the features that, when perturbed. lead to a strong decrease in classification accuracy. Furthermore, to assess the discriminative power of the most important features selected with this method, we trained a reduced SVM model to classify survivors and non-survivors using only the top-3 features [27] and measured the decrease in classification performance.

2.6. Guidelines and Ethical Approval

For the compilation of this manuscript, we followed STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines for reporting observational studies. The local Ethics Committee “Comitato Etico Indipendente di Area Vasta Emilia Centro (CE-AVEC)” approved the protocol of this study; the protocol code is 712/2020/Oss/AOUFe.

3. Results

3.1. Descriptive Analyses

The distribution of males and females in the overall population was quite homogeneous (54.8% males vs. 45.2% females), while in the group of patients who underwent intensification of care (IoCp) there was a higher percentage of males (65.8% vs. 34.2%; p = 0.021). The median age of the overall population was 72 years (IQR 58–82 years) and in-hospital mortality was recorded for 62 patients (24.8%)
The subjects who died within the hospitalization period (deceased) were significantly older than those who were discharged (median age 82 vs. 67 years; p < 0.001). This was true also for the group of patients who died within 30 days (30-ddp) compared with those who survived within the same period (30-dsp) (median age 81 vs. 68 years; p < 0.001).
Not surprisingly, we found differences between the IoCp and NIoCp groups in terms of length of stay (21 vs. 10 days; p < 0.001); significant differences were also found between the 30-ddp and 30-dsp groups (9 days vs. 13 days; p < 0.001).
The GASS scores were higher in the IoCp group compared to NIoCp (9 vs. 7 points; p < 0.001); as expected, similar differences were found between deceased and discharged patients (11 vs. 6 points; p < 0.001) but also between 30-ddp and 30-dsp (11 vs. 7 points; p < 0.001).
The patients with worse disease outcomes presented with a greater load of comorbidities (CCI 3.3 ± 2.7 vs. 1.6 ± 2.3 points, p < 0.001 between deceased and discharged and 3.0 ± 2.6 vs. 1.7 ± 2.4 points, p = 0.001 between 30-ddp and 30-dsp). As for vital parameters, IoCp and NIoCp groups differed in terms of respiratory rate, RR (22 bpm, IQR 20–30 vs. 20 bpm, IQR 16–22; p < 0.001) and PaO2/FiO2 ratio at the first visit (300, IQR 230-352 vs. 229, IQR 155-295; p < 0.001). Deceased and discharged patients differed in terms of systolic blood pressure (SBP), diastolic blood pressure (DBP) and RR; similarly, 30-ddp and 30-dsp differed for SBP, DBP and RR.
The laboratory findings were also heterogeneously distributed between groups: IoCp and NIoCp differed for serum C Reactive Protein (CRP), procalcitonin, D-Dimer, isoamylase, alanine transferase (ALT), lactic dehydrogenase (LDH) and ferritin. Differences between deceased and discharged patients related to serum lymphocyte count, creatinine, CRP, procalcitonin, D-Dimer, LDH, BNP and high sensitivity troponin I (HS TnI).
Similar differences were also found between 30-ddp and 30-dsp: these groups differed for serum creatinine, CRP, procalcitonin, D-Dimer (1.15 mg/L FEU, IQR 0.76–2.00 vs. 0.88 mg/L FEU, IQR 0.48–2.04; p = 0.044), LDH, BNP and HS TnI (42 ng/mL, IQR 19–101 vs. 9 ng/mL, IQR 4–22; p < 0.001).
All data concerning the characteristics of the studied population can be found in Table 1; extended results are provided only for those variables found to be important in the multivariate analyses later.
Differences between groups were also found in terms of comorbidities, and all the results are summarized in Table 2. Patients of the IoCp group were more often smokers; in the group of deceased subjects we found more diagnoses of hypertension, ischemic heart disease, heart failure, chronic kidney disease, history of stroke or transient ischemic attack (TIA), peripheral arterial disease (PAD), chronic obstructive pulmonary disease (COPD), localized or hematological cancer, dementia and diabetes with organ damage.
Comparison analyses between groups were also performed for each item considered in the Clinical-GASS score and they can be found in the Supplementary Tables S3–S5. All variables found to be significantly different between groups in the univariate analyses were entered into multivariate analyses.
In the logistic regression analyses concerning the need for intensification of care, the strongest predictive variable was the PaO2/FiO2 ratio: both ratios <100 and 100–199 showed to be somehow protective against the intensification of care (OR 0.11, 95% CI 0.02–0.72; p = 0.021 and OR 0.07, 95% CI 0.02–0.27; p < 0.001, respectively), while the male sex and a higher respiratory rate (>30 breaths per minute) seemed to be predictive for a higher need for intensification of care (OR 3.56, 95% CI 1.20–10.56; p = 0.022 and OR 4.76, 95% CI 1.00–22.96; p = 0.050).
Concerning the in-hospital mortality, age and PaO2/FiO2 ratio >300 were the only variables able to adequately predict this outcome (OR 1.07, 95% CI 1.02–1.12; p = 0.007 and OR 0.35, 95% CI 0.13–0.92; p = 0.034, respectively); moreover, age was particularly able at predicting death within a period of 30 days (OR 1.06, 95% CI 1.01–1.12; p = 0.027), together with serum D-dimer between 0.5 and 2.0 mg/L FEU (OR 4.22, 95% CI 1.28–13.90; p = 0.018); serum troponin I (HS TnI) < 20 ng/mL was, instead, protective against 30-day death (OR 0.23, 95% CI 0.06–0.80; p = 0.022).
The Clinical-GASS score was calculated for each patient, and this allowed us to predict the relative risk of in-hospital and 30-day death based on the analyses of this cohort of subjects. We developed an open access web tool to help clinicians identify the patients at higher risk of bad outcomes by COVID-19, simply filling out the form with all the variables needed. This tool can be found at the following link: https://ml.unife.it/GASS.html and it is available for free consultation.
Furthermore, we performed Receiver Operating Characteristic (ROC) curve analyses, in order to evaluate the predictive power of the GASS score, compared to the Charlson Comorbidity Index, which has been also recently tested in COVID-19 inpatients [28]. The main results of the classification analysis are reported in Figure 1.

3.2. The CCI Classifier

The optimal ROC threshold for the Charlson Comorbidity Index classifier was 11. This means that the optimal way to classify COVID-19 patients based on the CCI alone is to consider as high-risk all the patients with a CCI greater than or equal to this value. However, this classifier performed poorly on the test set, with an Area Under the Curve (AUC) of 0.66 and an accuracy of 0.76 (Figure 1D, grey bars). The poor performance of this classifier is attributable to the complete inability to correctly identify non-survivors (sensitivity = 0), which makes it effectively equivalent to the naïve majority classifier (black bars). We refer the reader to Section 2.4 for further details about the CCI classifier.

3.3. The Clinical-GASS Classifier

The optimal ROC threshold for the Clinical-GASS classifier was 13, which lies, as expected, within the range of values of the high mortality risk class (GASS > 10). Overall, the Clinical-GASS displayed satisfying test performance, with an AUC of 0.77 and an accuracy of 0.78 (Figure 1D, orange bars). Importantly, the Clinical-GASS classifier performed better than both the majority and the CCI classifiers, due to its improved ability to identify non-survivors (sensitivity = 0.31). We refer the reader to Section 2.4 for further details about the Clinical-GASS classifier.

3.4. The SVM22-GASS Classifier

The minimum-deviance ten-fold cross-validation regularization strength λLASSO was equal to 4.9 × 10−3. The corresponding LASSO logistic regression model fitted with such a parameter selected a total of 38 features, which are reported in Figure 1A. The best cross-validated hyperparameters of the RBF-SVM classifier trained using such features were σ2 = 774.16, λSVM = 1.4 × 10−6, and δ = 7318. The classifier trained with such hyperparameters achieved nearly optimal performance on the training set (AUC ≈ 1, accuracy ≈ 1, Figure 1C—blue bars), which means that the decision hyperplane is able to completely separate survivors from non-survivors in the transformed feature space. Importantly, the SVM classifier achieved very good performance also on the independent test set, with an AUC of 0.87 and an accuracy of 0.88. The ability to classify COVID-19 patients of the SVM classifier is thus superior to that of all the other considered classifiers. Such an improvement is largely attributable to a markedly better ability to correctly identify non-survivors (sensitivity = 0.61). We refer the reader to Section 2.5 for further details about the SVM22-GASS classifier.

Model Interpretation and Reduced Models

The SVM classifier just described predicts the most likely 30-day mortality outcome of COVID-19 patients by non-linearly processing a selection of 38 features that characterize them. To understand whether some of these features are more important than others, we computed the model reliance [17]. The results of this analysis are reported in Figure 2A, where we plotted the model reliance of the full SVM model, on all the 38 features. The features are sorted from top to bottom in decreasing order of reliance. According to this analysis, 8 of the 38 features appear to be particularly informative to the model, as its predictions change drastically when such features are perturbed. Specifically, the eight most important features of the full SVM classifier are White Blood Cell Count (WBC), Lymphocyte Count (LYM), Brain Natriuretic Peptide (BNP), Creatine Phosphokinase (CPK), Lactate Dehydrogenase (LDH), Fibrinogen (FIBR), PaO2/FiO2 Ratio (PFR), and High-Sensitivity Troponin I (TnI).
To understand how such features differ between survivors and non-survivors, and whether such differences are statistically significant, we performed Mann–Whitney U tests, using a Bonferroni-adjusted significance value of 0.05 (Figure 2B). Out of the 8 most important features, only 2 do not appear to be significantly different between survivors and non-survivors (CPK and fibrinogen, p value > 0.05). Medians, 95% bootstrap confidence intervals, and interquartile ranges of the top eight features, for the 30-day survival (green) and death (red) group can be found in the Figure 2, panel B.
The fact that the full SVM model appears to rely mostly on a subset of the full feature set of 38 features, raises the question of whether a reduced model, which has only access to the most informative features, can also accurately predict the COVID-19 outcome. To address this question, we trained a reduced RBF-SVM model to predict the COVID-19 outcome using only the three most informative features, that is White Blood Cell Count (WBC), Lymphocyte Count (LYM), and Brain Natriuretic Peptide (BNP). Choosing exactly three features allows us to visualize the entire feature space in which the classifier operates, and to plot the resulting decision surface.
As an additional baseline, we also trained another RBF-SVM model that had access to only the three least informative features (namely, Creatinine (CRE), Procalcitonin (PCT), and D-dimer (XDP)). The models, which we refer to as SVM-TOP3 and SVM-BOT3 respectively, were trained following the same steps adopted for the full SVM model. The results of this analysis are reported in Figure 3. As expected, both reduced models exhibited worse training and test performance than the full SVM model (Figure 3B,C). Nevertheless, while the SVM-BOT3 classifier exhibited overall poor test performance, comparable to that of the CCI classifier (AUC = 0.7, accuracy = 0.7—Figure 3C), the SVM-TOP3 classifier performed surprisingly well. As a matter of fact, the SVM-TOP3 had better test performance than the Clinical-GASS classifier (AUC = 0.83, accuracy = 0.83—Figure 3C), despite using only three features.
The resulting decision surface (Figure 3A) is, as expected, highly non-linear. The 3D plot of the feature space confirms that, generally, non-survivors seem to have higher serum BNP and lymphocyte count. Interestingly, the white blood cells count appears to become strongly predictive of a non-survival outcome, only for very high values (WBC > 10 × 103 n/mmc), especially when associated with high values of lymphocytes (LYM > 9 × 102 n/mmc).

4. Discussion

Predicting the mortality risk of hospitalized COVID-19 patients could go a long way to optimal allocation of hospital resources. Popular clinical scores are potentially useful in this regard, but have proven to be less accurate than COVID-19-specific methods [10,29]. In this work we have introduced and validated two new methods for predicting 30-day mortality risk: the Clinical-GASS and the SVM22-GASS. Overall, our results show that both methods are reliable and can thus be used to effectively triage incoming COVID-19 patients using only readily available variables.

4.1. Clinical-GASS Score and Risk Factors

The Clinical-GASS score is based on 11 variables easily available after the first visit at the Emergency Room, with the execution of both a venous and an arterial blood sample. The need for reliable and easy-to-use tools, able to predict the outcomes in COVID-19 inpatients is day by day more urgent because of the scarcity of human and financial resources, already under strain from the beginning of the pandemic.
The retrospective analyses from the first 499 patients hospitalized with COVID-19 to our hospitals in the territory of Ferrara, allowed us to notice a strong relationship between the Clinical-GASS score and mortality by COVID-19 (both in-hospital and 30-day rates); moreover, a slight relationship with the need for intensification of care was observed and confirmed only in the population with a Clinical-GASS score lower than 10 points; in the current study, this was true only for extreme values of the score (<5 or >10 points).
In our cohort of patients, the in-hospital mortality rate was 24.8%, while on the 30th day of observation, it was 23.6%. Our findings showed how patients who underwent intensification of care, needing non-invasive mechanical ventilation or endotracheal intubation, were more often males, with a greater Clinical-GASS score (p < 0.001) and worse respiratory performances (higher respiratory rates and lower PaO2/FiO2 ratios). As for laboratory abnormalities, those patients had higher serum inflammatory markers (CRP and ferritin), higher organ damage markers (isoamylase, ALT, LDH, HS TnI), and marked pro-coagulative status (higher serum D-Dimer).
Such findings are consistent with previous studies: male sex has been already linked to poorer chances of survival from COVID-19 and with increased risk of admission to the Intensive Care Units (ICUs) [30]; similarly, the PaO2/FiO2 ratio was already shown to be independently associated with worse COVID-19 outcomes [29,31]. Furthermore, recent studies showed how COVID-19 inpatients present commonly with laboratory abnormalities and the role of inflammatory, and organ-damage markers has been also consistently reported [32,33]. Our analyses also confirmed the role of age. Early Chinese studies showed how the elderly population was bound to encounter worse COVID-19 outcomes [34,35], and such findings were later confirmed in studies with European [36,37] and American patients [38]. Interestingly, the laboratory findings and respiratory performances reported in these studies were similar to those of our IoC group. Additionally, in accordance with previous studies that showed how low levels of blood pressure at hospital admission are often associated with worse COVID-19 outcomes [39], we observed higher blood pressure (both systolic and diastolic) in the survivor group. Finally, the negative role of comorbidities (especially cardiovascular) and smoking [40] is also well established [41,42].

4.2. Alternative Clinical Scores

Popular clinical scores of general applicability, such as the Charlson Comorbidity Score (CCI) [4], the modified Elixhauser Index (mEI) [43], the National Early Warning Score 2 (NEWS2) [44] and the CURB models [8,9] have been used during the current pandemic to predict COVID-19 outcomes with discrete success. However, they tend to underperform COVID-19-specific clinical scores [10,29].
The first COVID-19-specific scores were developed in China [11,45], during the first wave of the pandemic. Further scores for prediction of COVID outcomes were later developed all over the world [46,47,48], and data from more heterogeneous datasets were collected in order to reduce potential selection bias stemming from sampling from a restricted population of subjects. A case in point is represented by the 4C score, developed by Knight et al. [12]: such a score has quickly become popular due to its vast derivation and validation cohorts and its ease of use. However, the predictive performance of the score (validation AUC = 0.77), appears to lag behind that of SVM22-GASS introduced in this work (validation AUC = 0.87, accuracy = 0.88), and to be comparable with that of the Clinical-GASS (validation AUC = 0.77, accuracy = 0.78). Similar arguments can be made with regard to the recently developed “Piacenza score” (validation AUC = 0.78, accuracy = 0.55).
Importantly, the SVM22-GASS bases its prediction on the analysis of clinical variables that can be quickly and inexpensively retrieved early on during hospital admission. This is a significant advantage over other popular multivariate methods that rely on the manual analysis of medical imaging results (e.g., [29]), and yet achieve comparable predictive performance (validation AUC = 0.88). Like most machine-learning-based classifiers, the SVM22-GASS, the 4C and the Piacenza scores are purely data-driven. This makes them carry an intrinsic limitation represented by the fact that often it can be hard to give a “clinical” sense to the weight assigned to each feature considered in the development of the score. For this reason, in our first work, we decided to develop the GASS score, an informatic tool with the ability to accurately predict COVID-19 outcomes while keeping a clinical sense. The 11 features were chosen, in fact, based on both the strength of the associations with the outcomes and the clinical importance established for each of them in the current literature.

4.3. Interpreting the SVM22-GASS Classifier

The SVM22-GASS predicts the 30-day mortality outcome by nonlinearly processing 38 patient features. Our VARIABLE importance analysis isolated eight variables as the most important for the model: White Blood Cell Count (WBC), Lymphocyte Count (LYM), Brain Natriuretic Peptide (BNP), Creatine Phosphokinase (CPK), Lactate Dehydrogenase (LDH), Fibrinogen (FIBR), PaO2/FiO2 Ratio, and (PFR), and High-Sensitivity Troponin I (TnI). Interestingly, half of them, (namely, LYM, BNP, PFR, and TnI) are also part of the Clinical-GASS score, which confirms their strong information content and predictive power. Consistently, serum BNP has been recently found to be significantly elevated in critically ill COVID patients in a recent meta-analysis [49]; whether or not this peptide can help discriminate high-risk COVID-19 patients remains unclear and it merits further investigation. Similarly, high WBC values, high LDH values, and low LYM values [15] have also been recently linked to high mortality risk.
On the other hand, variables such as age, sex, respiratory rate, and serum D-Dimer (XDP) do not appear to significantly influence the predictions of the SVM22-GASS classifier. This finding might appear surprising, as their association with poor COVID-19 outcome is well established. For example, age is among the predominant risk factors for developing the severe form of the disease, due to the immuno-senescence and all the physiological modifications related to it [35]. Similarly, the serum levels of D-Dimer (XDP) were found to be strictly associated with COVID mortality [50]. Furthermore, the respiratory rate is a useful indicator of potential respiratory dysfunctions. Nevertheless, the fact that the SVM22-GASS does not rely on such variables, does not mean that they provide no information about the mortality risk; it merely means that other variables, those that carry greater weight, are more informative and/or less noisy.
To validate the results of the variable importance analysis, we trained two additional classifiers: the SVM-TOP3—which had only access to the three most important features White Blood Cell Count (WBC), Lymphocyte Count (LYM), Brain Natriuretic Peptide (BNP)—and the SVM-BOT3—which had only access to the three least important features. Interestingly, we observed only a moderate performance decrease of the SVM-TOP3 with respect to the full SVM22-GASS model (AUC: 0.83 vs. 0.87, accuracy: 0.83 vs. 0.88). This suggests that SVM-TOP3 is also a competent predictor and can be used in those cases in which one does not have access to all of the 38 variables for a given patient.

4.4. Potential Limitations

In this observational cohort study, we used independent derivation and validation cohorts to develop and validate our COVID-19-specific classifiers and risk score. However, both cohorts were recruited from the same two hospitals in the Italian Province of Ferrara and included only participants of Caucasian ethnicity. This might lead to overestimating the performance of our methods. Additionally, the sample size we used, albeit comparable to that of many other related studies (e.g., see [48]), is still too small to allow us to draw any final conclusion concerning the generalizability of our approaches.
To deal with these issues, future work will focus on undertaking a larger and multicentered study to validate our approaches on a larger and more heterogeneous cohort.
Additionally, some of the patients included in this study had missing values that were imputed using a weighted nearest neighbor method. Imputation is not an easy ask and might potentially lead to inflated performance measures. However, in real world setting it often is a necessary evil that comes with analyzing large and heterogenous datasets such as ours.
Furthermore, in the current study we did not use information about medical treatments, as this was missing. This implies that we cannot determine whether and how drug prescriptions altered the prognostic trajectories of our patients.
Finally, the fact that our feature selection step selected a subset of the 38 variables, suggests that some of the original variables are correlated. Future studies will investigate these correlations in depth to provide further insight into the disease and potentially reduce the number of variables to collect.

5. Conclusions

This work introduced two reliable classification approaches for the rapid assessment of mortality risk of COVID-19 patients. Importantly, both approaches rely on the automatic analysis of only routine clinical variables that can be readily acquired during hospital admission. The classifiers were developed and validated with independent derivation and validation cohorts and exhibited classification performance comparable to that of popular approaches that leverage information obtained with expensive medical imaging methods (e.g., chest radiography).
Our results prove that the classifiers have the potential to facilitate the triaging of incoming COVID-19 patients, thereby optimizing the allocation of hospital resources.
Clearly, predictive scores, in general, should not be taken into consideration as the only tools able to evaluate patients and their complexity. All scoring systems should be considered as an addition to clinical reasoning, laboratory, and instrumental examinations, besides experts’ opinion. Clinicians must always remember that any decision must be taken only after having the complete picture of their patients.
In conclusion, we are confident that future work will be able to further validate the proposed methods, using larger and more heterogeneous validation cohorts and we will focus on designing suitable strategies to assess them in hospital settings.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/biomedicines11030831/s1, Table S1: GASS score items; Table S2: Charlson Comorbidity Index; Table S3: GASS score items – differences between groups in terms of Intensification of Care; Table S4: GASS score items – differences between groups in terms of in-hospital mortality; Table S5: GASS score items – differences between groups in terms of 30-day mortality.

Author Contributions

S.G. and A.S.: the conception and design of the study, acquisition, analysis, interpretation of data and drafting the article. N.F., R.S. and E.L.: acquisition and analysis of data. F.R.: the development of the web tool, revising the article for important intellectual content. A.D.G.: interpretation of data and revision of the article for important intellectual content. A.P.: the conception and design of the study, drafting the article and revising it critically for important intellectual content and final approval of the version to be submitted. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the local Ethics Committee “Comitato Etico Indipendente di Area Vasta Emilia Centro (CE-AVEC)”; the protocol code is 712/2020/Oss/AOUFe. ClinicalTrials.gov Identifier: NCT05162534.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The datasets generated and analyzed during the current study are available from the corresponding author upon reasonable request.

Availability of Data and Material

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Acknowledgments

Special thanks to the directors of the medical departments where our research took place: Giovanni Zuliani, director of the Department of University Internal Medicine in the “Arcispedale S. Anna” in Cona (Fe); Roberto Manfredini, director of the Department of Clinical Medicine in the “Arcispedale S. Anna” in Cona (Fe); and Stefano Parini, director of the Department of Internal Medicine in the “Ospedale del Delta” in Lagosanto (Fe). We would like to dedicate our sincere thanks to all the members of the medical staff who are still participating in the fight against the virus, to all the extremely skillful and valuable nursing staff members as well as our assistance personnel, every day working on the frontline. Finally: the authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting A.S. List of Collaborators: Outcome SARS-CoV-2 Study Group. Herewith is the list of collaborators who indirectly contributed to the realization of this study. Alessandro Bella: MD; Beatrice Bonsi, MD; Andrea D’Amuri, MD; Michele Polastri, MD; Stefania Bonazzi, MD; Francesco Luciani, MD; Gaetano Lo Coco, MD; Enrico Giorgini, MD; Patrizia Guasti, MD; Chiara Marina Semprini, MD; Chiara Pistolesi, MD; Alessandro Scopa, MD; Grazia Vestita, MD; Dario Pedrini, MD; Sara Morrone, MD; Alessandra Violi, MD; Mariarosaria Lopreiato, MD; Valeria Fortunato, MD; Paola Chessa, MD; Flavia Albonetti, MD; Carlotta Braga, MD; Federico Cotti, MD; Alessandro Greatti, MD; Paola Guindani, MD; Caterina Savriè, MD; Chiara Sicuro, MD; Luigi Friio, MD; Stefano Fantasia, MD; Benedetta Garuti, MD. All participants belong to the Department of Translational Medicine of the University of Ferrara, Ferrara, Italy.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. The-nCoV Outbreak Joint Field Epidemiology Investigation Team; Li, Q. An Outbreak of NCIP (2019-NCoV) Infection in China—Wuhan, Hubei Province, 2019–2020. China CDC Wkly. 2020, 2, 79–80. [Google Scholar] [CrossRef]
  2. COVID—Coronavirus Statistics—Worldometer. Available online: worldometers.info (accessed on 31 January 2023).
  3. Charlson, M.E.; Pompei, P.; Ales, K.L.; MacKenzie, R. Charlson Comorbidity Index. J. Chronic Dis. 1987, 40, 373–383. [Google Scholar] [CrossRef]
  4. Tuty Kuswardhani, R.A.; Henrina, J.; Pranata, R.; Anthonius Lim, M.; Lawrensia, S.; Suastika, K. Charlson Comorbidity Index and a Composite of Poor Outcomes in COVID-19 Patients: A Systematic Review and Meta-Analysis. Diabetes Metab. Syndr. Clin. Res. Rev. 2020, 14, 2103–2109. [Google Scholar] [CrossRef] [PubMed]
  5. Christensen, D.M.; Strange, J.E.; Gislason, G.; Torp-Pedersen, C.; Gerds, T.; Fosbøl, E.; Phelps, M. Charlson Comorbidity Index Score and Risk of Severe Outcome and Death in Danish COVID-19 Patients. J. Gen. Intern. Med. 2020, 35, 2801–2803. [Google Scholar] [CrossRef] [PubMed]
  6. Neill, A.M.; Martin, I.R.; Weir, R.; Anderson, R.; Chereshsky, A.; Epton, M.J.; Jackson, R.; Schousboe, M.; Frampton, C.; Hutton, S.; et al. Community Acquired Pneumonia: Aetiology and Usefulness of Severity Criteria on Admission. Thorax 1996, 51, 1010–1016. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Jones, B.E.; Jones, J.; Bewick, T.; Lim, W.S.; Aronsky, D.; Brown, S.M.; Boersma, W.G.; van der Eerden, M.M.; Dean, N.C. CURB-65 Pneumonia Severity Assessment Adapted for Electronic Decision Support. Chest 2011, 140, 156–163. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Nguyen, Y.; Corre, F.; Honsel, V.; Curac, S.; Zarrouk, V.; Fantin, B.; Galy, A. Applicability of the CURB-65 Pneumonia Severity Score for Outpatient Treatment of COVID-19. J. Infect. 2020, 81, e96–e98. [Google Scholar] [CrossRef]
  9. Satici, C.; Demirkol, M.A.; Sargin Altunok, E.; Gursoy, B.; Alkan, M.; Kamat, S.; Demirok, B.; Surmeli, C.D.; Calik, M.; Cavus, Z.; et al. Performance of Pneumonia Severity Index and CURB-65 in Predicting 30-Day Mortality in Patients with COVID-19. Int. J. Infect. Dis. 2020, 98, 84–89. [Google Scholar] [CrossRef]
  10. Greco, S.; Salatiello, A.; Fabbri, N.; Passaro, A. Early Prediction of COVID-19 Outcome: Contrasting Clinical Scores and Computational Intelligence Methods. In Studies in Computational Intelligence; Springer: Cham, Switzerland, 2022; Volume 963, pp. 403–423. [Google Scholar]
  11. Yuan, Y.; Sun, C.; Tang, X.; Cheng, C.; Mombaerts, L.; Wang, M.; Hu, T.; Sun, C.; Guo, Y.; Li, X.; et al. Development and Validation of a Prognostic Risk Score System for COVID-19 Inpatients: A Multi-Center Retrospective Study in China. Engineering 2020, 8, 116–121. [Google Scholar] [CrossRef]
  12. Knight, S.; Ho, A.; Pius, R.; Buchan, I.; Carson, G.; Drake, T.; Dunning, J.; Fairfield, C.; Gamble, C.; Green, C.; et al. Risk Stratification of Patients Admitted to Hospital with Covid-19 Using the ISARIC WHO Clinical Characterisation Protocol: Development and Validation of the 4C Mortality Score. Br. Med. J. 2020, 370, m3339. [Google Scholar] [CrossRef]
  13. Yang, F.; Ma, D. The Proportion of SARS-CoV-2 Infections That Are Asymptomatic. Ann. Intern. Med. 2021, 174, 1343–1344. [Google Scholar] [CrossRef] [PubMed]
  14. Wu, Z.; McGoogan, J.M. Characteristics of and Important Lessons From the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72 314 Cases From the Chinese Center for Disease Control and Prevention. JAMA 2020, 323, 1239–1242. [Google Scholar] [CrossRef] [PubMed]
  15. Zhou, F.; Yu, T.; Du, R.; Fan, G.; Liu, Y.; Liu, Z.; Xiang, J.; Wang, Y.; Song, B.; Gu, X.; et al. Clinical Course and Risk Factors for Mortality of Adult Inpatients with COVID-19 in Wuhan, China: A Retrospective Cohort Study. Lancet 2020, 395, 1054–1062. [Google Scholar] [CrossRef] [PubMed]
  16. Metz, C.E. Basic Principles of ROC Analysis. Semin. Nucl. Med. 1978, 8, 283–298. [Google Scholar] [CrossRef]
  17. Cortes, C.; Vapnik, V. Support-Vector Networks. Mach. Learn. 1995, 20, 273–297. [Google Scholar] [CrossRef]
  18. Chawla, N.V.; Bowyer, K.W.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic Minority Over-Sampling Technique. J. Artif. Intell. Res. 2002, 16, 321–357. [Google Scholar] [CrossRef]
  19. Fonti, V.; Belitser, E. Feature Selection Using LASSO; VU Amsterdam: Amsterdam, The Netherlands, 2017. [Google Scholar]
  20. Tibshirani, R. Regression Shrinkage and Selection Via the Lasso. J. R. Stat. Soc. Ser. B 1996, 58, 267–288. [Google Scholar] [CrossRef]
  21. Byrd, R.H.; Lu, P.; Nocedal, J.; Zhu, C. A Limited Memory Algorithm for Bound Constrained Optimization. SIAM J. Sci. Comput. 1995, 16, 1190–1208. [Google Scholar] [CrossRef]
  22. Le, Q.; Sarlós, T.; Smola, A. Fastfood—Approximating Kernel Expansions in Loglinear Time. In Proceedings of the 30th International Conference on Machine Learning, ICML, Atlanta, GA, USA, 16 June–21 June 2013; pp. 1281–1289. [Google Scholar]
  23. Fawcett, T. An Introduction to ROC Analysis. Pattern Recognit. Lett. 2006, 27, 861–874. [Google Scholar] [CrossRef]
  24. Feurer, M.; Hutter, F. Hyperparameter Optimization. In Automated Machine Learning; The Springer Series on Challenges in Machine Learning; Hutter, F., Kotthoff, L., Vanschoren, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2019; pp. 3–33. [Google Scholar]
  25. Snoek, J.; Larochelle, H.; Adams, R.P. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2012; Volume 4, pp. 2951–2959. [Google Scholar]
  26. Wei, P.; Lu, Z.; Song, J. Variable Importance Analysis: A Comprehensive Review. Reliab. Eng. Syst. Saf. 2015, 142, 399–432. [Google Scholar] [CrossRef]
  27. Fisher, A.; Rudin, C.; Dominici, F. All Models Are Wrong, but Many Are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously. J. Mach. Learn. Res. 2019, 20, 1–81. [Google Scholar]
  28. Varol, Y.; Hakoglu, B.; Kadri Cirak, A.; Polat, G.; Komurcuoglu, B.; Akkol, B.; Atasoy, C.; Bayramic, E.; Balci, G.; Ataman, S.; et al. The Impact of Charlson Comorbidity Index on Mortality from SARS-CoV-2 Virus Infection and A Novel COVID-19 Mortality Index: CoLACD. Int. J. Clin. Pract. 2021, 75, e13858. [Google Scholar] [CrossRef]
  29. Munera, N.; Garcia-Gallo, E.; Gonzalez, Á.; Zea, J.; Fuentes, Y.V.; Serrano, C.; Ruiz-Cuartas, A.; Rodriguez, A.; Reyes, L.F. A Novel Model to Predict Severe COVID-19 and Mortality Using an Artificial Intelligence Algorithm to Interpret Chest Radiographs and Clinical Variables. ERJ Open Res. 2022, 8, 00010–02022. [Google Scholar] [CrossRef]
  30. Peckham, H.; de Gruijter, N.M.; Raine, C.; Radziszewska, A.; Ciurtin, C.; Wedderburn, L.R.; Rosser, E.C.; Webb, K.; Deakin, C.T. Male Sex Identified by Global COVID-19 Meta-Analysis as a Risk Factor for Death and ITU Admission. Nat. Commun. 2020, 11, 6317. [Google Scholar] [CrossRef]
  31. Zinellu, A.; De Vito, A.; Scano, V.; Paliogiannis, P.; Fiore, V.; Madeddu, G.; Maida, I.; Zinellu, E.; Mangoni, A.A.; Arru, L.B.; et al. The The PaO2/FiO2 Ratio on Admission Is Independently Associated with Prolonged Hospitalization in COVID-19 Patients. J. Infect. Dev. Ctries. 2021, 15, 353–359. [Google Scholar] [CrossRef] [PubMed]
  32. Kermali, M.; Khalsa, R.K.; Pillai, K.; Ismail, Z.; Harky, A. The Role of Biomarkers in Diagnosis of COVID-19—A Systematic Review. Life Sci. 2020, 254, 117788. [Google Scholar] [CrossRef]
  33. Lippi, G.; Plebani, M. Laboratory Abnormalities in Patients with COVID-2019 Infection. Clin. Chem. Lab. Med. 2020, 58, 1131–1134. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Wu, C.; Chen, X.; Cai, Y.; Xia, J.; Zhou, X.; Xu, S.; Huang, H.; Zhang, L.; Zhou, X.; Du, C.; et al. Risk Factors Associated with Acute Respiratory Distress Syndrome and Death in Patients with Coronavirus Disease 2019 Pneumonia in Wuhan, China. JAMA Intern. Med. 2020, 180, 934–943. [Google Scholar] [CrossRef] [Green Version]
  35. Lee, J.Y.; Nam, B.-H.; Kim, M.; Hwang, J.; Kim, J.Y.; Hyun, M.; Kim, H.A.; Cho, C.-H. A Risk Scoring System to Predict Progression to Severe Pneumonia in Patients with Covid-19. Sci. Rep. 2022, 12, 5390. [Google Scholar] [CrossRef]
  36. Salje, H.; Kiem, C.T.; Lefrancq, N.; Courtejoie, N.; Bosetti, P.; Paireau, J.; Andronico, A.; Hozé, N.; Richet, J.; Dubost, C.L.; et al. Estimating the Burden of SARS-CoV-2 in France. Science 2020, 369, 208–211. [Google Scholar] [CrossRef]
  37. Onder, G.; Rezza, G.; Brusaferro, S. Case-Fatality Rate and Characteristics of Patients Dying in Relation to COVID-19 in Italy. JAMA-J. Am. Med. Assoc. 2020, 323, 1775–1776. [Google Scholar] [CrossRef]
  38. McMichael, T.M.; Currie, D.W.; Clark, S.; Pogosjans, S.; Kay, M.; Schwartz, N.G.; Lewis, J.; Baer, A.; Kawakami, V.; Lukoff, M.D.; et al. Epidemiology of Covid-19 in a Long-Term Care Facility in King County, Washington. N. Engl. J. Med. 2020, 382, 2005–2011. [Google Scholar] [CrossRef]
  39. Lanzani, C.; Simonini, M.; Arcidiacono, T.; Messaggio, E.; Bucci, R.; Betti, P.; Avino, M.; Magni, G.; Maggioni, C.; Conte, C.; et al. Role of Blood Pressure Dysregulation on Kidney and Mortality Outcomes in COVID-19. Kidney, Blood Pressure and Mortality in SARS-CoV-2 Infection. J. Nephrol. 2021, 34, 305–314. [Google Scholar] [CrossRef] [PubMed]
  40. Patanavanich, R.; Glantz, S.A. Smoking Is Associated with COVID-19 Progression: A Meta-Analysis. Nicotine Tob. Res. 2020, 22, 1653–1656. [Google Scholar] [CrossRef]
  41. Ejaz, H.; Alsrhani, A.; Zafar, A.; Javed, H.; Junaid, K.; Abdalla, A.E.; Abosalif, K.O.A.; Ahmed, Z.; Younas, S. COVID-19 and Comorbidities: Deleterious Impact on Infected Patients. J. Infect. Public Health 2020, 13, 1833–1839. [Google Scholar] [CrossRef] [PubMed]
  42. Gold, M.S.; Sehayek, D.; Gabrielli, S.; Zhang, X.; McCusker, C.; Ben-Shoshan, M. COVID-19 and Comorbidities: A Systematic Review and Meta-Analysis. Postgrad. Med. 2020, 132, 749–755. [Google Scholar] [CrossRef]
  43. De Giorgi, A.; Fabbian, F.; Greco, S.; Di Simone, E.; De Giorgio, R.; Passaro, A.; Zuliani, G.; Manfredini, R. Prediction of In-Hospital Mortality of Patients with SARS-CoV-2 Infection by Comorbidity Indexes: An Italian Internal Medicine Single Center Study. Eur. Rev. Med. Pharmacol. Sci. 2020, 24, 10258–10266. [Google Scholar] [CrossRef] [PubMed]
  44. Tuncer, G.; Surme, S.; Bayramlar, O.F.; Karanalbant, H.K.; Copur, B.; Yazla, M.; Zerdali, E.; Nakir, I.Y.; Cinar, A.R.K.; Buyukyazgan, A.; et al. National Early Warning Score 2 and Laboratory Predictors Correlate with Clinical Deterioration in Hospitalized Patients with COVID-19. Biomark. Med. 2021, 15, 807–820. [Google Scholar] [CrossRef]
  45. Liang, W.; Liang, H.; Ou, L.; Chen, B.; Chen, A.; Li, C.; Li, Y.; Guan, W.; Sang, L.; Lu, J.; et al. Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19. JAMA Intern. Med. 2020, 180, 1081. [Google Scholar] [CrossRef]
  46. Zhao, Z.; Chen, A.; Hou, W.; Graham, J.M.; Li, H.; Richman, P.S.; Thode, H.C.; Singer, A.J.; Duong, T.Q. Prediction Model and Risk Scores of ICU Admission and Mortality in COVID-19. PLoS ONE 2020, 15, e0236618. [Google Scholar] [CrossRef] [PubMed]
  47. López-Escobar, A.; Madurga, R.; Castellano, J.M.; Velázquez, S.; Suárez Del Villar, R.; Menéndez, J.; Peixoto, A.; Jimeno, S.; Ventura, P.S.; Ruiz de Aguiar, S. Risk Score for Predicting In-Hospital Mortality in COVID-19 (Rim Score). Diagnostics 2021, 11, 596. [Google Scholar] [CrossRef] [PubMed]
  48. Halasz, G.; Sperti, M.; Villani, M.; Michelucci, U.; Agostoni, P.; Biagi, A.; Rossi, L.; Botti, A.; Mari, C.; Maccarini, M.; et al. A Machine Learning Approach for Mortality Prediction in COVID-19 Pneumonia: Development and Evaluation of the Piacenza Score. J. Med. Internet Res. 2021, 23, e29058. [Google Scholar] [CrossRef] [PubMed]
  49. Sorrentino, S.; Cacia, M.; Leo, I.; Polimeni, A.; Sabatino, J.; Spaccarotella, C.A.M.; Mongiardo, A.; De Rosa, S.; Indolfi, C. B-Type Natriuretic Peptide as Biomarker of COVID-19 Disease Severity—A Meta-Analysis. J. Clin. Med. 2020, 9, 2957. [Google Scholar] [CrossRef] [PubMed]
  50. Soni, M.; Gopalakrishnan, R.; Vaishya, R.; Prabu, P. D-Dimer Level Is a Useful Predictor for Mortality in Patients with COVID-19: Analysis of 483 Cases. Diabetes Metab. Syndr. Clin. Res. Rev. 2020, 14, 2245–2249. [Google Scholar] [CrossRef]
Biomedicines 11 00831 g001 550
Figure 1. Summary prediction performance. (A) Receiver Operating Characteristic (ROC) curves of the considered classifiers, computed on the training set (majority classifier: black; Charlson Comorbidity Index (CCI) classifier: grey; GASS classifier: orange; SVM classifier: blue). Filled circles indicate the optimal operating points. Note that the classification threshold of the majority classifier is fixed; thus, the corresponding ROC curve collapses to a single point. (B) Corresponding curves computed on the independent test set. Filled circles represent the optimal operating points of the training ROC curves. (C) Considered classification performance measures, computed on the training set; values are rounded to the nearest hundredths; AUC: Area Under the ROC Curve. (D) Corresponding measures computed on the test set. Note that the majority class in both the training and test set is the survival class, which was arbitrarily associated to a negative test result; therefore, by always predicting survival, the majority classifier has a sensitivity of 0 and a specificity of 1.
Figure 1. Summary prediction performance. (A) Receiver Operating Characteristic (ROC) curves of the considered classifiers, computed on the training set (majority classifier: black; Charlson Comorbidity Index (CCI) classifier: grey; GASS classifier: orange; SVM classifier: blue). Filled circles indicate the optimal operating points. Note that the classification threshold of the majority classifier is fixed; thus, the corresponding ROC curve collapses to a single point. (B) Corresponding curves computed on the independent test set. Filled circles represent the optimal operating points of the training ROC curves. (C) Considered classification performance measures, computed on the training set; values are rounded to the nearest hundredths; AUC: Area Under the ROC Curve. (D) Corresponding measures computed on the test set. Note that the majority class in both the training and test set is the survival class, which was arbitrarily associated to a negative test result; therefore, by always predicting survival, the majority classifier has a sensitivity of 0 and a specificity of 1.
Biomedicines 11 00831 g001
Biomedicines 11 00831 g002 550
Figure 2. Understanding the full SVM model. (A) Model reliance of the full SVM model on all the 38 features its predictions are based on. Features are sorted (from top to bottom) in descending order of importance. (B) Medians, 95% bootstrap confidence intervals, and interquartile ranges of the top eight features, for the 30-day survival (green) and death (red) group. Black dots represent medians; box heights represent confidence intervals; thin lines represent interquartile ranges. Asterisks indicate significant (Bonferroni-adjusted p value ≤ 0.05) differences between groups (Mann–Whitney U test).
Figure 2. Understanding the full SVM model. (A) Model reliance of the full SVM model on all the 38 features its predictions are based on. Features are sorted (from top to bottom) in descending order of importance. (B) Medians, 95% bootstrap confidence intervals, and interquartile ranges of the top eight features, for the 30-day survival (green) and death (red) group. Black dots represent medians; box heights represent confidence intervals; thin lines represent interquartile ranges. Asterisks indicate significant (Bonferroni-adjusted p value ≤ 0.05) differences between groups (Mann–Whitney U test).
Biomedicines 11 00831 g002
Biomedicines 11 00831 g003 550
Figure 3. Reduced models and decision surface. (A) Decision surface of the SVM-TOP3 model, which only uses the top-3 features. The surface divides the feature space into two regions: a survival region (green) and a death region (red). Small green circles represent 30-day survivors, while small red circles represent 30-day non-survivors; misclassified instances are marked with a black dot. The corresponding large circles represent the group medians. Note: to facilitate the visualization of the decision surface, we have only plotted the data within a cuboid delimited by the 20th and 80th percentiles. (B) Classification performance measures of the full SVM model and of the reduced SVM models, computed on the training set. The SVM-TOP3 model only uses the top-3 features, while the SVM-BOT3 model only uses the bottom-3 features; values are rounded to the nearest hundredths. (C) Corresponding measures computed on the test set.
Figure 3. Reduced models and decision surface. (A) Decision surface of the SVM-TOP3 model, which only uses the top-3 features. The surface divides the feature space into two regions: a survival region (green) and a death region (red). Small green circles represent 30-day survivors, while small red circles represent 30-day non-survivors; misclassified instances are marked with a black dot. The corresponding large circles represent the group medians. Note: to facilitate the visualization of the decision surface, we have only plotted the data within a cuboid delimited by the 20th and 80th percentiles. (B) Classification performance measures of the full SVM model and of the reduced SVM models, computed on the training set. The SVM-TOP3 model only uses the top-3 features, while the SVM-BOT3 model only uses the bottom-3 features; values are rounded to the nearest hundredths. (C) Corresponding measures computed on the test set.
Biomedicines 11 00831 g003
Table
Table 1. Characteristics of population and differences between groups.
Table 1. Characteristics of population and differences between groups.
Cohort Characteristics All Patients (N = 250) IoCp (N = 76) NIoCp (N = 174) p-Value Deceased (N = 62) Discharged (N = 188) p-Value 30-ddp (N = 59) 30-dsp (N = 191) p-Value
Males, n (%) 137 (54.8) 50 (65.8) 87 (50.0) 0.021 31 (50.0) 106 (56.4) 0.38 31 (52.5) 109 (57.1) 0.20
Females, n (%) 113 (45.2) 26 (34.2) 87 (50.0) 31 (50.0) 82 (43.6) 28 (47.5) 82 (42.9)
Age, years 72 (58–82) 72 (62–80) 72 (53–82) 0.26 82 (76–86) 67 (53–78) <0.001 81 (73–86) 68 (54–79) <0.001
Length of stay, days 12 (7–22) 21 (12–40) 10 (6–16) <0.001 10 (6–20) 12 (7–23) 0.16 9 (6–16) 13 (8–24) <0.001
GASS score, points 7 (5–11) 9 (7–11) 7 (4–10) <0.001 11 (9–13) 6 (4–9) <0.001 11 (8–13) 7 (4–9) <0.001
CCI, points ± SD 2.0 ± 2.5 2.1 ± 2.5 2.0 ± 2.5 0.69 3.3 ± 2.7 1.6 ± 2.3 <0.001 3.0 ± 2.6 1.7 ± 2.4 0.001
Vital Signs and Parameters
SBP, mmHg 130 (120–140) 130 (120–150) 130 (115–140) 0.47 120 (110–140) 130 (120–140) 0.024 120 (110–130) 130 (120–140) 0.005
DBP, mmHg 70 (70–80) 70 (60–80) 70 (70–80) 0.08 70 (60–80) 75 (70–80) 0.019 70 (60–80) 75 (70–80) 0.003
HR, bpm 86 (77–99) 86 (75–100) 87 (78–99) 0.39 88 (76–100) 86 (78–99) 0.94 88 (74–100) 86 (78–99) 0.84
RR, apm 20 (18–23) 22 (20–30) 20 (16–22) <0.001 22 (18–28) 20 (16–22) <0.001 21 (18–28) 20 (16–22) 0.016
PaO2/FiO2 ratio 300 (230–352) 229 (155–295) 310 (261–355) <0.001 266 (200–304) 310 (244–355) 0.09 269 (193–311) 308 (245–352) 0.22
Laboratory Findings
WBC (n/mmc) 6395 (4815–9375) 6970 (5315–9795) 6210 (4750–9280) 0.33 7600 (5380–10905) 6140 (4750–9010) 0.07 6280 (5060–10030) 6420 (4795–9320) 0.84
Lymphocytes (n/mmc) 1000 (740–1390) 990 (635–1325) 1000 (760–1470) 0.33 870 (640–1085) 1050 (778–1483) 0.008 905 (658–1278) 1030 (770–1465) 0.14
Creatinine (mg/dl) 0.94 (0.77–1.28) 0.96 (0.78–1.35) 0.94 (0.77–1.24) 0.78 1.34 (0.82–2.01) 0.91 (0.75–1.16) <0.001 1.31 (0.88–1.83) 0.91 (0.75–1.17) <0.001
CRP (mg/dl) 5.82 (1.98–10.92) 8.57 (1.94–18.00) 5.21 (1.98–9.52) 0.038 8.49 (3.10–13.71) 4.92 (1.74–9.87) 0.007 8.49 (2.87–12.88) 4.98 (1.83–10.12) 0.025
Procalcitonin (ng/mL) 0.15 (0.05–0.51) 0.25 (0.08–0.73) 0.11 (0.05–0.43) 0.018 0.54 (0.20–1.75) 0.09 (0.05–0.26) <0.001 0.49 (0.17–1.96) 0.10 (0.05–0.27) <0.001
Fibrinogen (mg/dl) 509 (441–659) 555 (438–689) 501 (441–643) 0.27 572 (431–677) 502 (441–642) 0.37 550 (428–674) 506 (441–643) 0.91
D-Dimer (mg/L FEU) 0.93 (0.54–2.04) 1.12 (0.74–1.99) 0.88 (0.47–2.04) 0.039 1.20 (0.77–2.30) 0.85 (0.47–1.80) 0.008 1.15 (0.76–2.00) 0.88 (0.48–2.04) 0.044
Isoamylase (U/L) 27 (20–44) 36 (25–100) 26 (18–41) 0.008 32 (18–51) 27 (21–44) 0.92 36 (20–45) 27 (20–43) 0.63
ALT (U/L) 21 (14–37) 26 (18–49) 19 (12–33) 0.003 18 (11–29) 87 (52–193) 0.06 19 (12–34) 21 (14–37) 0.29
CPK (U/L) 87 (51–203) 111 (54–224) 83 (46–169) 0.14 93 (50–220) 280 (211–361) 0.82 122 (59–356) 86 (50–167) 0.12
LDH (U/L) 286 (219–380) 390 (274–466) 271 (210–332) <0.001 328 (250–436) 304 (125–593) 0.015 332 (265–434) 276 (210–358) 0.001
Ferritin (ng/mL) 327 (142–637) 506 (294–922) 275 (120–565) 0.006 364 (180–767) 304 (125–593) 0.19 439 (264–736) 293 (121–610) 0.07
BNP (pg/mL) 76 (36–185) 80 (43–168) 75 (33–195) 0.90 197 (94–410) 60 (30–126) <0.001 209 (87–377) 62 (30–132) <0.001
HS TnI (ng/mL) 14 (5–34) 15 (7–47) 13 (5–29) 0.13 41 (16–85) 9 (4–21) <0.001 42 (19–101) 9 (4–22) <0.001
Data are reported as median (interquartile range IQR) unless otherwise specified; IoCp, patients who underwent intensification of care; nIoCp, patients who did not undergo intensification of care; 30-ddp, patients deceased after 30 days; 30-dsp, patients surviving after 30 days; GASS, General Assessment of SARS-CoV-2 patients; SD, Standard Deviation; CCI, Charlson Comorbidity Index; SBP, Systolic Blood Pressure; DBP, Diastolic Blood Pressure; HR, Heart Rate; RR, Respiratory Rate; WBC, White Blood Cells; CRP, C Reactive Protein; ALT, Alanine Transferase; CPK, Creatine Phosphokinase; LDH, Lactic Dehydrogenase; BNP, Brain Natriuretic Peptide; HS TnI, High Sensitivity Troponin I.
Table
Table 2. Comorbidities and differences between groups.
Table 2. Comorbidities and differences between groups.
Comorbidity All Patients (N = 250) IoCp (N = 76) NIoCp (N = 174) p-Value Deceased (N = 62) Discharged (N = 188) p-Value 30-ddp (N = 59) 30-dsp (N = 191) p-Value
Smoking habit 53 (21.2) 25 (32.9) 28 (16.1) 0.003 17 (27.4) 36 (19.1) 0.17 14 (23.7) 39 (20.4) 0.59
Hypertension 141 (56.4) 49 (64.5) 92 (52.9) 0.09 45 (72.6) 96 (51.1) 0.003 45 (76.3) 96 (50.3) <0.001
Ischemic heart disease 25 (10.0) 8 (10.5) 17 (9.8) 0.86 11 (17.7) 14 (7.4) 0.019 9 (15.3) 16 (8.4) 0.12
Heart failure 18 (7.2) 5 (6.6) 13 (7.5) 0.80 8 (12.9) 10 (5.3) 0.045 6 (10.2) 12 (6.3) 0.31
Chronic kidney disease (moderate or severe) 35 (14.0) 10 (13.2) 25 (14.4) 0.80 15 (24.2) 20 (10.6) 0.008 13 (22.0) 22 (11.5) 0.042
Stroke or TIA 33 (13.2) 8 (10.5) 25 (14.4) 0.41 17 (27.4) 16 (8.5) <0.001 15 (25.4) 18 (9.4) 0.002
PAD 16 (6.4) 3 (3.9) 13 (7.5) 0.30 9 (14.5) 7 (3.7) 0.003 8 (13.6) 8 (4.2) 0.010
COPD 26 (10.4) 9 (11.8) 17 (9.8) 0.62 13 (21.0) 13 (6.9) 0.002 10 (16.9) 16 (8.4) 0.06
Mild hepatopathy 8 (3.2) 0 (0) 8 (4.6) 0.06 1 (1.6) 7 (3.7) 0.41 1 (1.7) 7 (3.7) 0.45
Moderate or severe hepatopathy 3 (1.2) 1 (1.3) 2 (1.1) 0.91 3 (4.8) 0 (0) 0.002 3 (5.1) 0 (0) 0.002
Localized or hematological cancer 49 (19.6) 17 (22.4) 32 (18.4) 0.47 19 (30.6) 30 (15.9) 0.012 19 (32.2) 30 (15.7) 0.005
Metastatic cancer 11 (4.4) 3 (3.9) 8 (4.6) 0.82 2 (3.2) 9 (4.8) 0.60 1 (1.7) 10 (5.2) 0.25
Dementia 47 (18.8) 13 (17.1) 34 (19.5) 0.65 22 (35.5) 25 (13.3) <0.001 20 (33.8) 27 (14.1) 0.001
Diabetes without organ damage 23 (9.2) 7 (9.2) 16 (9.2) 1.00 7 (11.3) 16 (8.5) 0.51 8 (13.6) 15 (7.9) 0.19
Diabetes with organ damage 26 (10.4) 9 (11.8) 17 (9.8) 0.62 11 (17.7) 15 (8.0) 0.029 9 (15.3) 17 (8.9) 0.16
Data are reported as number of subjects (percentage); IoCp, patients who underwent intensification of care; nIoCp, patients who did not undergo intensification of care; 30-ddp, patients deceased after 30 days; 30-dsp, patients surviving after 30 days; TIA, Transient Ischemic Attack; PAD, Peripheral Arterial Disease; COPD, Chronic Obstructive Pulmonary Disease.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Greco, S.; Salatiello, A.; Fabbri, N.; Riguzzi, F.; Locorotondo, E.; Spaggiari, R.; De Giorgi, A.; Passaro, A. Rapid Assessment of COVID-19 Mortality Risk with GASS Classifiers. Biomedicines 2023, 11, 831. https://doi.org/10.3390/biomedicines11030831

AMA Style

Greco S, Salatiello A, Fabbri N, Riguzzi F, Locorotondo E, Spaggiari R, De Giorgi A, Passaro A. Rapid Assessment of COVID-19 Mortality Risk with GASS Classifiers. Biomedicines. 2023; 11(3):831. https://doi.org/10.3390/biomedicines11030831

Chicago/Turabian Style

Greco, Salvatore, Alessandro Salatiello, Nicolò Fabbri, Fabrizio Riguzzi, Emanuele Locorotondo, Riccardo Spaggiari, Alfredo De Giorgi, and Angelina Passaro. 2023. "Rapid Assessment of COVID-19 Mortality Risk with GASS Classifiers" Biomedicines 11, no. 3: 831. https://doi.org/10.3390/biomedicines11030831

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop