Developing a rapid evidence response to COVID-19: The collaborative approach of Saskatchewan, Canada
Abstract
Introduction
The COVID-19 Evidence Support Team (CEST) was a provincial initiative that combined the support of policymakers, researchers, and clinical practitioners to initiate a new learning health cycle (LHS) in response to the pandemic. The primary aim of CEST was to produce and sustain the best available COVID-19 evidence to facilitate decision-making in Saskatchewan, Canada. To achieve this objective, four provincial organizations partnered to establish a single, data-driven system.
Methods
The CEST partnership was driven by COVID-19 questions from Emergency Operational Committee (EOC) of the Saskatchewan Health Authority. CEST included three processes: (a) clarifying the nature and priority of COVID-19 policy and clinical questions; (b) providing Rapid Reviews (RRR) and Evidence Search Reports (ESR); and (c) seeking the requestors' evaluation of the product. A web-based repository, including a dashboard and database, was designed to house ESRs and RRRs and offered a common platform for clinicians, academics, leaders, and policymakers to find COVID-19 evidence.
Results
In CEST's first year, 114 clinical and policy questions have been posed resulting in 135 ESRs and 108 RRRs. While most questions (41.3%) originated with the EOC, several other teams were assembled to address a myriad of questions related to areas such as long-term care, public health and prevention, infectious diseases, personal protective equipment, vulnerable populations, and Indigenous health. Initial challenges were mobilization of diverse partners and teams, remote work, lack of public access, and quality of emerging COVID-19 literature. Current challenges indicate the need for institutional commitment for CEST sustainability. Despite these challenges, the CEST provided the Saskatchewan LHS with a template for successful collaboration.
Conclusions
The urgency of COVID-19 pandemic and the implementation of the CEST served to catalyze collaboration between different levels of a Saskatchewan LHS.
1 INTRODUCTION
When SARS-CoV-2 caused the infectious disease COVID-19, the resultant worldwide pandemic1 caused many countries around the world to introduce widespread public health measures to contain the impacts of coronavirus.2 Similarly, the provincial government of Saskatchewan, Canada, declared a state of emergency to control anticipated negative impacts on public health and health service delivery.3, 4 Simultaneously, there was a high demand for rapid and ongoing scientific information related to COVID-19 to support leaders' decision-making.5, 6 Previous experience from outbreaks has demonstrated that reliable and timely scientific information is vital for informed decision-making.7, 8 For instance, lessons learned from the Ebola disease outbreak confirmed that lack of real-time information can considerably affect the decisions of frontline responders which placed public health and healthcare providers at serious risk.8 In response to this need for rapid and valid COVID-19 information in Saskatchewan, the COVID-19 Evidence Support Team (CEST) was founded at the request of the Emergency Operating Committee (EOC). To meet this demand, a partnership between the Saskatchewan Health Authority (SHA), College of Medicine at the University of Saskatchewan (USASK), Health Quality Council (HQC), and the Ministry of Health (MoH) was established. In this report, we will describe how the four provincial institutions collaborated to (a) respond to the need for a single reliable COVID-19 evidence database, (b) maintain a continuously updated source of COVID-19 evidence, (c) facilitate COVID-19 decision-making by providing this evidence to administrators, clinicians, and policymakers, and (d) initiate a Learning Health System in response to a difficult and fast-moving crisis.
2 SETTING
Saskatchewan is a Canadian province with a population of 1 178 681.9 The MoH and the SHA are the two principal organizations responsible for Saskatchewan healthcare direction, vision, and delivery.10 While the MoH has the overall responsibility for legislating, planning, and overseeing health delivery, the SHA and its affiliated agencies are accountable for health service provision.10 SHA was established in December 2017 through a transition from 12 former Regional Health Authorities to a single provincial health authority.11
Saskatchewan has a history of innovation and large system transformation in health reform, including the introduction of Canada's first provincial hospital insurance programme in 1947 and implementation of systems thinking in more recent years.12 For the last two decades, the Saskatchewan government has made significant investment toward realizing the vision of a patient-oriented rapid Learning Health System (LHS) to structure a cycle of continuous research, learning at all levels, and ongoing quality improvement.13, 14 At the center of the Saskatchewan LHS is the systematic production of evidence to create insight for the best practice (Evidence >> Insight >> Action).13 To achieve this vision, Saskatchewan has been implementing a wide range of strategies to provide timely evidence for decision-makers at all levels to enhance healthcare outcomes.13 Aligned with this practice, the CEST initiative was an early response to the need for producing and sustaining rapid reliable COVID-19 evidence to inform decision-making. Like many developed jurisdictions, the Saskatchewan healthcare system faces the challenges of an aging population and increased rates of chronic disease,15 two risk factors for severe symptoms or death from COVID-19.16, 17 Evidence-informed decisions enable us to deliver the best practice for Saskatchewan's diverse population, including the most vulnerable.
3 METHODS
3.1 Goals and strategies
As the COVID-19 situation unfolded, the SHA established the EOC to oversee the pandemic response. The EOC quickly identified a need for evidence support to address clinical and policy-related questions. To assist with this issue, the EOC engaged the first author (G.G.), a clinician-scientist, who has a cross-appointment with SHA and USASK, to create a single reliable, evidence-based system in response to their COVID-19 inquiries. Under the first author's supervision, the CEST oversight committee was established and has been a central contact point for the key users of the CEST, the EOC and, later, the PHICC (Public Health Incident Command Center).
The CEST initiative had three key objectives: (a) the rapid production of the best evidence for facilitating COVID-related decision-making, (b) the establishment of a single electronic platform (a database, dashboard, and repository) for systematic sharing of the updated COVID-19 reviews, and (c) the initiation of a Learning Health System by constant exchange between reliable evidence, policy, and practice.
3.2 People and processes
The CEST's development required clear communication between partners and the alignment of resources for a consistent evidence synthesis approach. To that end, an oversight committee of key representatives from the SHA, College of Medicine USASK, and HQC was established to engage the essential participants. The MoH representation was added later to connect with the provincial government. Each organization brought unique knowledge and services necessary for the CEST functionality. For instance, HQC, as the provincial promoter of healthcare quality improvement, lent their technical expertise to assist the development of CEST processes. Lay members of the public were not involved when CEST was being developed due to time and financial constraints. Early in the Saskatchewan state of emergency, when the uncertainty of COVID-19 seemed at its peak, the oversight committee met three times weekly to discuss emerging issues such as the development of the repository, standardizing review processes, and assessing evidence quality.
There were three distinct processes to coordinate and sustain CEST's knowledge translation system: (a) clarifying the nature and priority of questions; (b) providing Rapid Reviews; and (c) seeking the evaluation of the product by the requestors. First, an Intake Prioritization Process and feedback loop determined which questions would be accepted and the timeline for response. Early in the process, the need for strategic prioritization to optimize the allocation of our limited resources became evident. To help with prioritization, the CEST chairperson was embedded in daily EOC meetings and met weekly with the physician pandemic lead to identify urgent questions.
After identifying a question, the second process involved a team of one or more clinician experts (clinical, epidemiologic, or other relevant experts), one or more members of the library team and one or more Health Sciences masters or PhD trained researchers, who together refined the questions to searchable inquiries. Working in pairs, Librarians then used a combination of subject headings and keyword terms to produce a comprehensive ESR based on published, pre-print, and grey literature. Researcher staff from USASK and SHA then synthesized the literature into narrative RRRs that were further refined by the clinician expert(s) on the team. This rapid searching and reviewing of published and grey literature produced the EOC's review summaries. The research intake process and feedback loop are outlined in Figure 1.
Early on, the oversight committee identified the need for teams to evaluate the quality of the synthesized evidence. However, we found that traditional systematic review grading tools were not appropriate because of the rapid turnaround time and the emerging nature of the literature. Later, we developed a descriptive assessment of the number, design, and settings of the studies to provide users with additional background for the review.
Finally, a web-based repository was situated on SHA library website and contained a dashboard to provide a single electronic platform for easy access to the database that housed full-text ESRs and RRRs https://saskhealthauthority.libguides.com/covid-19/repository/home (Figure 2). The dashboard was generated from the database and offered a snapshot of number of questions, their status (search in progress, review in progress, completed, update in progress, updated review, canceled, review delayed), the overall process, and additional documents provided by team members. Both the dashboard and repository were designed to be user-friendly to facilitate the finding, reuse, and citation of reviews. Figure 3 demonstrates an example of repository functionality for a COVID-19 hot topic, Face Masks, and the questions around this topic. Clicking on each question directs the user to the full-text review and related information. For instance, clicking the first question “What is the evidence for the effectiveness of universal mask use by the public? [Update]” provides useful information such as document type (ESR or RRR), the team who posed the question, key findings, and etc. (see Figure 4).
For some topics, an ESR was conducted but a RRR was not produced if the evidence base for a review was inadequate. The team made a concerted effort to obtain formal (an evaluation form) and informal feedback (verbal communication to first author; see Data S1) from the end users to provide the best result possible.
3.3 Technology and measures
Technology played a major role in launching the CEST project. WebEx was used to coordinate the search meetings between the librarians and the researchers, the librarians and the clinical experts, and meetings with EOC. Mural, a visual workspace, enabled collaboration between the oversight committee to develop the repository. Through several virtual meetings, Mural enabled anonymous critique of the system functions and the visualization of the Intake Prioritization Process. In addition, Redcap software and automated scripts facilitated the implementation of the dashboard from the DBTextworks software which houses all of the data. DBTextworks, a flexible database management system that can be customized for non-IT staff, has been used by SHA Library to organize the policy database and has search capabilities for text-based databases.18 AnDI (Andornot Discovery Interface), a web-based discovery interface19 based on the Apache Solr system, provided robust searching and persistent links from the database to populate the repository, helped the team build a sophisticated search interface to their archival records, and provided the best productive search experience for CEST users.20
4 RESULTS
During the first year that the repository has been in operation (March 2020-March 2021), 114 clinical and policy questions have been submitted resulting in 135 ESRs and 108 RRRs. Questions had a variety of themes and topics with epidemiology of virus and masks being a common theme at the beginning of the pandemic and School reopening and COVID-19 vaccination a prominent theme in recent months. The majority of questions (n = 50; 41.3%) were posed by the EOC as the primary user of CEST. Table 1 displays the distribution of questions answered by research teams as of 22 April 2021.
| Research team | Number of questions | % of Total questions |
|---|---|---|
| EOC | 50 | 41.3% |
| Public health | 14 | 11.5% |
| Long-term care | 13 | 10.7% |
| Laboratory | 10 | 8.2% |
| Infectious disease | 8 | 6.6% |
| Epidemiology and modeling | 8 | 6.6% |
| Personal protective equipment | 7 | 5.7% |
| Critical care | 5 | 4.1% |
| Clinical/acute care | 2 | 1.6% |
| Vulnerable population | 2 | 1.6% |
| Indigenous health | 2 | 1.6% |
| Total number of questions | 121 | |
- Abbreviation: EOC, Emergency Operational Committee.
Figure 5 presents the number of ESRs and RRRs completed monthly during March 2020 and March 2021, with April 2020 having the highest number of reviews. Of a total of 108 Rapid Review Reports, 44 (40.7%) were completed between 2 to 10 days with 20 (18.5%) reviews completed in one or less than 1 day. The average time from question to Rapid Review completion was 21 days. Similarly, average search time per ESR was 10 days with 62 out of 135 (45.9%) ESRs completed in one or less than 1 day. Timeliness was an important element in the CEST process considering the urgency and uncertainty of COVID-19 inquiries, especially in the early weeks. When urgent questions arose, more resources were concentrated to meet the request in the shortest period of time. Some questions required more time depending on their complexity, prioritization, and availability of relevant literature. The CEST intake committee reviewed the questions for further refinement, decided which teams would work on which questions, and determined the timeline.
In the early stages of the work, the CEST saw its role as limited to the creation of evidence reviews as described above. How those reviews were used by the EOC and/or working group such as the provincial modeling group to make policy decisions was outside of our mandate. Nevertheless, CEST results led to policy-making and facilitated decision-making related to COVID-19. For example, evidence for mask-wearing was one of the early reviews that the CEST was asked to conduct, and that review contributed to the initial development of a mask use policy within the SHA. Another example was uncertainty around the minimum time required between procedures (settle time) that would protect healthcare workers from contracting COVID-19 at the workplace. This question resulted in an ESR and RRR (see Data S2) and a policy decision regarding settle times in various locations.
5 DISCUSSION
As part of the Saskatchewan COVID-19 response, key partners were engaged in a knowledge generation initiative that established the CEST. This partnership led to the production of high-quality, timely evidence reviews on COVID-19 that were used by system leaders as the starting point of a pandemic learning health system. Providing the best evidence possible for initial policy decisions that were later refined by local epidemiologic data, modeling that was informed by the CEST, and updated evidence reviews were essential to Saskatchewan's pandemic response. The systematic engagement of diverse stakeholders who share accountability for health problems and health outcomes is the foundation of an LHS.21 For the CEST project, stakeholders from policy, research, and practice domains harnessed their expertise, knowledge, and experience toward the common goal of keeping Saskatchewan residents and healthcare providers as safe as possible from the coronavirus.
An LHS bridges research, policy, and practice through rapid learning from the best evidence.21, 22 The key contribution of evidence-informed learning is to support ongoing improvement of health outcomes, innovation, and delivery of quality care.20 In the absence of a COVID-19 cure, the CEST initiative involved all partner institutions in the process of evidence syntheses. This collaborative strategy also aligned with the province's systems thinking approach in the development of the Learning Health System.12 The CEST developed a shared team vision, assessed information needs, established action plans, set priorities, mobilized stakeholders to commit resources, clarified roles in teamwork, and monitored and evaluated the process.23
The repository became the electronic center of CEST for knowledge circulation. This electronic infrastructure was designed around four characteristics that enhanced its effectiveness and functionality. First, the CEST repository was planned as a single electronic database for COVID-19 information in Saskatchewan. Responses and approaches to COVID-19 have been very inconsistent between countries, states, and even hospitals of the same region.24 One main reason for inconsistent health decision-making could be that COVID-19 information is scattered among multiple publications25 which could impede timely and effective application. In Saskatchewan, referring to a single repository contributed to more uniform and consistent decision-making related to COVID-19 across the province.
The second feature of the repository, and a common practice of an LHS,21, 22 was to develop scientific evidence to guide the process of decision-making, a vital need for COVID-19 response.26 Learning from the evidence available through the repository helped Saskatchewan's decision-makers to minimize mistakes, reduce acts of trial and error, and maximize informed decision-making in the rapidly changing global pandemic.
The third feature was the use of technology to promote the dynamics of the repository and accelerate the learning process which was crucial for the urgency of COVID-19 response. Emerging technologies augment the quality, scale, and speed of data sharing in their systems which is imperative for timely and better care delivery.21 Using technological tools such as the repository can also contribute to innovation and information safety in healthcare systems.21
A final feature was to design the repository in a way that it facilitated communication between the Evidence Support Team and the knowledge users. From an LHS lens, communication, in its many forms and various levels, shapes the social pillar of learning healthcare and ensures the participation of different stakeholders.21 COVID-19 requests were directly submitted via the dashboard for further assessment which initiated the cycle of communication between the CEST users and the Team. This ongoing exchange enabled the research team to understand knowledge users and conduct need-oriented searches.Other Canadian provinces have initiated similar Rapid Evidence Reviews related to COVID-19.27-29 COVID-19 datasets such as Public Health Ontario's Synopsis of Key Articles offer brief descriptions and summaries on COVID-19 articles rather than question-based Rapid Reviews.29 Compared to similar provincial projects such as Alberta's COVID-19 Scientific Advisory Group,28 CEST's advantage is its repository that provides a full picture of CEST activities and work-in-progress and enhances access to and sharing of COVID-19 evidence. The other advantage of CEST is that the questions originated from the Saskatchewan context yet synthesized data from multiple jurisdictions to offer the most relevant COVID-19 evidence.
Questions around COVID-19 are likely to persist without a cure for the virus. Thus, our short-term plan is to continue to inform decision-makers with the best available evidence. While the initial knowledge users were the EOC, the repository could also serve as a long-term knowledge translation tool for administrative, academic, and clinical purposes. To that end, our long-term goal for this Evidence Support Repository is to enable researchers to expand initial searches and translate findings on relevant COVID-19 questions to multiple audiences.
5.1 Challenges and opportunities
Thus far, CEST has been a combination of success and challenges. The team has worked to identify and address those challenges. This is characteristic of strong healthcare systems where weaknesses and limitations are acknowledged to create a sound and systematic intervention.23 Many of our early challenges in building the CEST were related to mobilizing diverse groups of partners. Although the various partners of CEST had pre-existing relationships, they had been working in silos that precluded the creation of a functional LHS in the province. These silos existed both within the SHA (ie, amongst the various portfolios of a relatively new single health authority) and between institutions (MoH, Ministry of Education, SHA, HQC, and USASK). While the urgency and scale of the pandemic was the incentive to bring these partners together, the CEST served as a catalyst to begin to eliminate these silos and move towards collaborative rapid learning. The CEST can provide a template for the future implementation of a provincial LHS.
The development of the project coincided with the province's COVID-19 health orders leading to the closure of many workplaces, facilities, and services. Most participants in the CEST began working remotely for the first time which required adjustments, technological challenges, learning, and planning. Despite these unavoidable challenges, the CEST ensured regular group meetings for remote teams to keep them updated and engaged.
Apart from remote work, the CEST process itself presented some difficulties early on. The teams of librarians, researchers, and clinical experts were not fixed all the time. Their assignment to a question was based on their availability and expertise in that subject. Although the team's diversity and expertise enriched CEST, it could also unsettle the team composition, affecting the consistent quality of the work. To tackle this challenge, each team was led by clinical expert, one librarian, and one researcher, who coordinated individual contributions and promoted consistency in team members' efforts.
The repository accessibility also created certain limitations. Up until November 16, 2020, the repository's username and password were only provided to CEST's members of the collaborating institutions. After that date, the Rapid Reviews became available to the public. Controlled accessibility may have acted as an obstacle to the widespread utilization of the COVID-19 database. We limited the initial scope of CEST for two main reasons. First, because we needed to manage the number of requests, especially at the beginning of the pandemic when we prioritized the EOC and PHICC questions. Second, the CEST initiative experienced financial restrictions which affected its accessibility and sustainability. While a provincial grant supported a team of five graduate students to write reviews for a limited time (June-September 2020), CEST continues to use existing resources.
Other challenges were related to the literature search and access to the COVID-19 research articles. First, searching involved many pre-prints, grey literature, and nonpeer-reviewed literature which increased time and effort spent searching, selecting, and assessing the literature. Thus, the librarians created searching protocols based on different deadlines to enable the completion of urgent requests and allow a more comprehensive search for less urgent requests. Second, Rapid Reviews by nature were typically English focused—which means COVID literature published in other languages was not considered. A third challenge was difficulties with sharing full-text articles with CEST users. Although at the beginning of the pandemic, many publishers began rapid dissemination by making their full-text publications available to the public, this privilege has been removed to a large extent. It was important to provide the full-text of studies to users so that they could assess the relevance and applicability of the study to the question if they wished. The challenge to the CEST is creating a better collaboration platform for different types of documents that the team needs to share. The opportunity going forward is creating a “generic link in the ESR” for users to access full-text evidence. This step will require superior IT protection, copyright considerations, and resources to be implemented.
The quality assessment of the produced evidence also posed challenges. In initial phases of CEST (late March 2020), the chief focus was on the rapid synthesis of the evidence. In the more recent period, weaknesses have been identified in the design and conduct of some of the published and pre-print literature.30-32 Especially in the initial phases of COVID-19, for example, many of the COVID-19 clinical trials lacked critical design features such as standard sample size, randomized allocation, the use of control groups, and double blinding.30, 32 Therefore, the CEST has recently undertaken actions to distinguish quality evidence. Essential to an LHS is the continuous evaluation of processes, quality, and outcomes to achieve learning and improvement.21 Building on this notion of an LHS, CEST has deployed a descriptive evaluative system to rate evidence before it is synthesized into a Rapid Review. The quality grading system involves four steps to describe the: (a) adequacy of the primary studies supporting the reviews, (b) methodological limitations from primary studies, (c) relevance of the findings in the primary studies to the review questions, and (d) generalizability of the findings in the primary studies and the overall coherence of the findings. This grading system will offer a better guarantee of the quality of the information for COVID-19 decision-making.The CEST is still grappling with the user evaluative component to their RRRs and ESRs. Initial outcome surveys of the uptake and application of the CEST products have had low response. Another targeted survey is planned for a sample of users to gather direct feedback about the utility of CEST products.
Moving forward, the CEST requires institutional commitment, adequate funding, and integrated structure which makes the work part of the job description for team members. Not all pandemic policy decision-making was informed by CEST reviews due in part to capacity issues and the time constraints of such reviews, even when performed in a rapid manner as described. Some policymakers perceived that a comprehensive review of the evidence was not necessary which poses another challenge for the CEST. These challenges could impact the maintenance of the system in the long run.
6 CONCLUSION
CEST was established to support Saskatchewan decision-makers to make timely evidence-based decisions during the COVID-19 pandemic. Despite challenges associated with mobilizing diverse stakeholders, remote work, the searching and quality of the literature, and the further need for evaluation, the CEST provided the Saskatchewan LHS with a template for successful collaboration. Various partners at all levels of health and academia came together to create learning cycles and facilitate evidence-based decision-making. We believe that this collaboration can remain as the legacy for the future of a province wide LHS.
ACKNOWLEDGMENTS
The CEST team gratefully acknowledges support from the College of Medicine (University of Saskatchewan), Health Quality Council, Saskatchewan Health Authority, and Saskatchewan Health Research Foundation for their support.
The CEST comprises Amir Reza Azizian, Jenny Basran, Catherine Boden, Vicky Duncan, Courtney Ellsworth, Fiona Fick, Lance Fox, Brianna Howell-Spooner, Stephen Lee, Michelle McCarron, Lukas Miller, Mark Mueller, Nazeem Muhajarine, Cory Neudorf, Geoffrey Shumilak, James Stempien, Molly Trecker, Susan Tupper, Sabira Valiani, Jason Vanstone, Heather Ward, Hazel Williams-Roberts, Brandy Winquist, Catherine Young.
CONFLICT OF INTEREST
The authors have no relevant conflicts of interests to declare.
