Introduction

The severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) infection, first reported in China in December 2019, swiftly spread to other countries, resulting in the Coronavirus Disease-19 (COVID-19) pandemic [1]. Many countries implemented several measures in the past two years to slow the spread of COVID-19 and avoid fatalities, such as strict lockdowns, physical distancing and home/institutional isolation, yet the pandemic continues [2]. As of March 9, 2022, > 448 million cases of COVID-19 including > 6 million deaths, have been reported worldwide [3].

The lack of definitive medical therapy for COVID-19 led to a search for an effective and safe vaccine to prevent transmission and a substantial magnitude of research and funding were devoted to vaccine development [4, 5]. The first anti-COVID-19 vaccine was made available to the public in a matter of months in December 2020 [6]. Since then, several COVID-19 vaccines have been approved and many countries are undertaking mass vaccination drives. As of March 9, 2022, 35 COVID-19 vaccines have been approved across the globe. Sputnik V (Gamaelya), Spikevax (Moderna), Vaxzevria (Oxford/AstraZeneca), Comirnaty (Pfizer/BioNTech), Ad26.CoV2.S (Johnson & Johnson), and Covilo (Sinopharm) are among those COVID-19 vaccines that have received approval in most nations. There are also more than 190 vaccine candidates under different phases of clinical development. Moreover, eleven vaccines have been granted emergency approval status by WHO [7].

As of March 9, 2022, around 11 billion vaccine doses have been administered worldwide. The top four nations where the highest number of doses of COVID-19 vaccines have been given include China (3.17 billion), India (1.79 billion), the United States (0.56 billion), and Indonesia (0.36 billion) [8]. Most vaccines were given emergency use authorization by regulatory authorities [9].

Though all of these vaccines have been found to be efficacious, there is no direct evidence of their comparative efficacies due to a lack of head-to-head clinical trials between vaccines. Therefore, this network meta-analysis was performed to combine phase 3 data of all the published randomized controlled trials to determine the comparative efficacy of COVID-19 vaccines. It aims to answer the natural clinical question of which among the competing existing COVID-19 vaccines shows the maximum protection against symptomatic COVID-19 infection and to comment on their comparative efficacies. This network meta-analysis is expected to provide insight and a piece of valuable evidence-based information to the healthcare community on the same and help taking appropriately informed decisions by the stakeholders.

Methods

The protocol of this systematic review and network meta-analysis was registered in PROSPERO on February 28, 2022 (registration number CRD42022313196). Reporting of this network meta-analysis slightly differs from the protocol with an intent to include updated information and tool for analysis for improving the quality of the work. First, the literature search was performed in Medline, Scopus, Embase and Cochrane library to include recent studies as well. Second, the latest version of ‘Cochrane Risk of Bias tool’ was used for the assessment of the risk of bias for the included studies. Results of this systematic review and network meta-analysis is being reported following ‘Preferred reporting items for systematic review and meta-analysis (PRISMA)-NMA extension 2015 guidelines.

Search strategy

A comprehensive search was made in Medline (via PubMed), Embase and Cochrane Library databases using search terms ‘Covid-19’, ‘SARS-CoV-2’, ‘severe acute respiratory syndrome coronavirus 2’, in combination with ‘vaccines’ and ‘phase 3 clinical trial’. Two review authors (SK and VM) independently reviewed the studies for inclusion/exclusion. Any disagreement between the two was resolved by discussion and/or involving the third review author (DS). Details of the search strategy for PubMed are described in the supplementary material.

Eligibility criteria

Published original research articles of phase 3 randomized controlled trials reporting the efficacy of COVID-19 vaccine(s) in comparison to other COVID-19 vaccines or placebo in adult healthy or diseased participants with chronic stable comorbid conditions were eligible for inclusion. In this network meta-analysis adults are considered as individuals of age ≥ 18 years of age and/or those described as adults in the respective studies. Phase 3 randomized controlled clinical trials were chosen for this network meta-analysis as these trials involve a large enough number of participants and are considered to generate the most reliable evidence of efficacy for the intervention of interest. Articles published till September 05, 2022 were included in the study.

Study outcome(s)

The outcome measure was vaccine efficacy defined as the protection conferred by COVID-19 vaccines against laboratory (nucleic acid amplification i.e. polymerase chain reaction test)-confirmed symptomatic SARS-CoV-2 infection following completion of the dosing regimen. It was measured as a relative risk by using raw data i.e., no. of events (laboratory-confirmed symptomatic COVID-19) in the vaccine and placebo groups in the per-protocol population.

Data extraction and synthesis

Two independent review authors screened the search records based on their titles, abstracts, and full texts. Data were extracted from the included studies on the efficacy of the COVID-19 vaccine using a prespecified format. The data items extracted, include study-related parameters (study author, year of publication, study design, level of blinding, study duration), population-related parameters (age, number of participants receiving intervention, number of participants receiving control), intervention-related parameters (name of intervention/drug, type of intervention, dose/dosage, route of administration), comparator related parameters (name of control, route of administration), outcome-related parameters (primary efficacy outcome(s), number of participants achieving primary efficacy outcomes). Raw data from the included individual studies were used in this network meta-analysis.

Risk of bias and quality assessment

The risk of bias in the included studies was assessed by ‘The Cochrane Risk of Bias tool version 2 (RoB 2)’ by two independent reviewers. Any disagreement was resolved by discussion and/or involvement of a third author. RoB2 tool includes a randomization process, deviations from the intended interventions, missing outcome data, measurement of the outcome and selection of the reported results as components for an assessment of the quality of the included studies [10].

Assessment of inconsistency and heterogeneity

Assessment of inconsistency was not applicable as there was no randomized controlled trial comparing two COVID-19 vaccines as direct evidence. Heterogeneity was assessed using Cochrane Q statistics and I2 test.

Statistical analysis.

Network meta-analysis was performed according to PRISMA-NMA extension 2015. Raw data (number of events) from the included individual studies were used to estimate treatment effects (risk ratio). This was followed by a network meta-analysis with a random effect model by estimating indirect evidence between two interventions. The statistical analysis was done using software ‘R’ studio version 4.1.2 and ‘netmeta’ (v 2.1–0)’ package which is based on the frequentist method. The geometry of the treatment network was assessed by the ‘netgraph.netmeta’ function of netmeta package version 2.0–1 of ‘R’ software. ‘P-scores’ were used to rank vaccine efficacy which is based on point estimates and the standard errors and was calculated using ‘netrank {netmeta}’ function. ‘P-score’ (ranges from ‘0’ to ‘1’) indicates, the mean extent of certainty that a treatment is better than the competing treatments. In other words, it shows ‘how better an intervention is relative to all others (a higher score indicates a better rank)’. Rankogram and league table (which indicate the probability of each treatment being at each possible rank) and all pairwise comparisons (direct and indirect) respectively, were also obtained. The asymmetry of included studies (in terms of small and large) was assessed using comparison-adjusted funnel plot.

Results

A total of 460 items were found after a database search using relevant terms. After excluding non-relevant studies a total of 17 studies were included in the systematic review and network meta-analysis [Fig. 1]. These 17 studies [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27] included 16 vaccines. Characteristics of the studies included are shown in Table 1 (Refer to Supplementary Table 1 for details). The inclusion criteria for the population were studies involving adult human participants. One of the studies of search results included individuals of age more than 16 years considering them as adults. The study was included in the network meta-analysis to avoid the loss of a sufficient magnitude of the sample. Sixteen COVID-19 vaccines were compared with a placebo in these included studies. None of the included studies compared two COVID-19 vaccines directly and therefore geometry of the network graph obtained is of star shape [Fig. 2]. A total of 361,386 participants was included in this network meta-analysis of which 204,390 received an intervention and 156,996 a placebo [Supplementary Table 1].

Fig. 1
figure 1

Prisma flow chart showing inclusion and exclusion of studies

Full size image
Table 1 Characteristics of various randomised controlled trials comparing efficacy of COVID-19 vaccines with that of placebo, included in network meta-analysis
Full size table
Fig. 2
figure 2

Network plot of studies included in the network meta-analysis. Line width represents inverse standard error of random effect model comparing two treatments. The numerals on the lines represent the number of studies between the respective two treatment arms

Full size image

Forest plots for comparative efficacy of direct and indirect evidence of COVID-19 vaccines are shown in Figs. 3 & 4, respectively, and in the form of a league table [Supplementary Table 2]. The ranking of individual vaccines for the study outcome against all competing vaccines is shown as a ‘P’ score [Table 2] and a Rankogram plot [Supplementary Fig. 1]. The highest ‘P’ score is for BNT126b2 and the least for the CVnCoV SARS-Cov-2 vaccine.

Fig. 3
figure 3

Forest plot of intervention (COVID-19 vaccines) in comparison to placebo in various clinical trials included in the network meta-analysis. RR relative risk, CI confidence interval

Full size image
Fig. 4
figure 4

Forest plot of indirect comparison of different pairs of various COVID-19 vaccines through a network meta-analysis. RR relative risk, 95%-CI 95% confidence interval

Full size image
Table 2 ‘P’ score of various vaccines: used to rank treatment in network meta-analysis
Full size table

The Cochrane ‘Q’ statistics were found to be 2.73, which was not significant (p = 0.25). Variability between designs was shown to be ‘zero’ as there is no direct comparison between vaccines. I2 value was 26.8% denoting the absence of much heterogeneity among the studies [Supplementary Fig. 2].

RoB2 tool assesses the risk of bias under five major domains namely ‘bias arising from randomisation process (D1)’, ‘bias due to deviation from intended intervention (D2)’, ‘bias due to missing outcome data (D3)’, ‘bias in measurement of the outcome (D4)’ and ‘bias in selection of the reported results (D5)’. All of the studies used the per-protocol method for data analysis instead of intention-to-treat, therefore, had some concern under the D2 domain except for one study (Khairullin B, et al. 2022) which was of low risk under this domain. One study (Khairullin B, et al. 2022) has some concerns under D3, D4 and D5 domains. One study (Fadlyana E, et al. 2021) had some concerns under the D5 domain. Study by Hager KJ, et al. 2022 had some concerns under D1 domain. All studies had low risk under the remaining domains. The overall risk of bias assessment revealed ‘some concern’. [Supplementary Fig. 3].

Comparison adjusted funnel plot suggests a minimal small study effect [Supplementary Fig. 4]. However, because of a smaller number of included studies the power of the tests of asymmetry may decrease and, therefore, results should be interpreted cautiously.

Discussion

We performed this network meta-analysis in an effort to find the most efficacious vaccine against SARS-CoV-2 infection as none of the COVID-19 vaccines has been compared directly till now. Though two network meta-analyses on COVID-19 vaccines have been performed, in the current one we included the highest number of COVID-19 vaccines whose phase 3 clinical trial data have been published.

The added information from this network meta-analysis has been obtained at the cost of additional assumptions of transitivity and consistency. However, the characteristics of the included studies are sufficiently akin to allow the pooling of the data. The consistency of the network could not be assessed due to the lack of information on any direct study comparing two vaccines. Since there is no clinically relevant heterogeneity, it was not explored further.

To date, one recent meta-analysis reported the results of phase 3 clinical trials of five vaccines [28]. The overall vaccine efficacy to prevent SARS-CoV-2 infection obtained in this study was 89.1% (95% CI 85.6–92.6%) against the placebo. BNT162b2 mRNA COVID- 19 vaccine (Pfizer-BioNTech) and mRNA-1273 vaccine (Moderna) showed the highest summary vaccine efficacy (98.1% & 91.2% respectively). However, using the Network meta-analysis techniques, multiple treatment interventions can be compared indirectly without performing comparative clinical trials of different therapy interventions in a single statistical analysis, unlike the conventional meta-analysis, in which only two interventions can be compared that are directly reported/used in randomized clinical trials [29].

In one of the earlier network metanalyses, indirect comparison of seven COVID-19 was performed based on the results of phase 3 randomized controlled trials to find the vaccine with the highest probability of reducing symptomatic COVID-19. The vaccine with the highest probability of efficacy was BNT162b2-Corminarty vaccine with a P-score of 90.6%), followed by mRNA-1273-Spikevax, Gam-COVID-Vac-Sputnik-V, NVX-CoV2373-Novavax, Ad26.COV2.S-Janssen, ChAdOx1 nCoV-19-Vaxzevria, and CoronaVac (P-scores 88.8%, 76.5%, 50.7%, 37.7%, 28.5% and 25.1%, respectively) [30]. Similarly, in another network meta-analysis, published in November 2021, nine COVID-19 vaccines were compared indirectly, and the candidate vaccine with the greatest likelihood of being most effective against symptomatic SARS-CoV-2 infection was BNT162b2, followed by the mRNA-1273 vaccine, Gam-COVID-Vac, NVX-CoV23730, CoronaVac, BN02, WIV04, ChAdOx1 and Ad26.COV2.S (P-scores 0.952, 0.843, 0.782, 0.700, 0.570, 0.428, 0.327, 0.199 and 0. 0.198, respectively) [31].

In line with these previously published network meta-analyses, our results indicate BNT126b2 as a COVID-19 vaccine conferring the maximum reduction in the risk to contract symptomatic SARS-CoV-2 infection in comparison to placebo with a relative risk of 0.05 followed by mRNA1273 and rAd26 & rAd5, both of which had a relative risk of 0.08 against placebo. Further, the indirect evidence, generated by the present network meta-analysis, does not indicate a significant difference in the efficacy of BNT126b2 in comparison to that of mRNA-1273, rAd26 + rAd5 and NVX-CoV2373. On the other hand, the BNT126b2 vaccine was found to be of significantly greater efficacy compared to SCB-2019, ChAdOx1 nCoV-19, Sinopharm, Covaxin and Coronvac, to prevent symptomatic SARS-CoV-2 infection. BNT126b2 vaccine had the highest probability to hold the rank ‘1’ in the rankogram and also had the highest ‘P’ score. While interpreting the results, it is important to remember that risk of bias in all the included studies is of ‘some concern’, as stated earlier.

One of the possible explanations for results suggesting that mRNA technology-based COVID-19 vaccines could be the most efficacious against symptomatic SARS-CoV-2 infection is helping the immune cells remember the infection for a long time apart from generating a good neutralizing antibody response, leading to longer-lasting immunity against Coronavirus.

Thus the present piece of work through network meta-analysis shows BNT126b2, amongst all the included vaccines, to provide the highest magnitude of protection against contracting symptomatic SARS-CoV-2 infection followed by mRNA-1273, rAd26 + rAd5 and NVX-CoV2373 vaccines.

Strengths of the study

To the best of our knowledge, the highest number of COVID-19 vaccines and randomized controlled phase 3 trial data are used in this network meta-analysis to generate an evidence of the efficacy of COVID-19 vaccines amongst themselves in absence of any direct comparison. This network meta-analysis focused on one outcome only, at a time, to not compromise the quality of the analysis. There is no publication restriction on behalf of authors.

Limitations of the study

The assessment of consistency was not possible as there was no study to compare two or more vaccines directly. So, our network meta-analysis is based on indirect comparison only, amongst the included vaccines. Though the vaccines have been compared only on the basis of their efficacies, a number of other factors such as dose and frequency of administration, safety, and cost-effectiveness have an important role in the selection of vaccines. Most of the included studies have similar efficacy outcomes, however, they are different in terms of timelines of their conduction during the pandemic, the number of doses of vaccines and the follow-up period for assessment of efficacy. The grading for the certainty of evidence has not been performed so the conclusion from the study should be interpreted cautiously. This network meta-analysis included the efficacy outcome from included studies irrespective of these factors. Further, the efficacy of these vaccines against several new strains and variants of concern (delta, omicron, etc.) is either unknown or limited; hence any statement about comparability in relation to specific strains cannot be made.

Conclusion & recommendation

This network meta-analysis finds that BNT162b2 followed by mRNA-1273, rAd26&rAd5, and NVX-Cov2373 to confer the highest protection against symptomatic COVID-19 compared to others. Besides, they were not found significantly different from each other. In this light, a wise decision should be taken while choosing a particular vaccine against COVID-19 before considering the indirect evidence(s) generated through network meta-analyses.