Appraising Qualitative Research for Evidence Syntheses: A Compendium of Quality Appraisal Tools
Abstract
As the movement toward evidence-based health policy continues to emphasize the importance of including patient and public perspectives, syntheses of qualitative health research are becoming more common. In response to the focus on independent assessments of rigor in these knowledge products, over 100 appraisal tools for assessing the quality of qualitative research have been developed. The variety of appraisal tools exhibit diverse methods and purposes, reflecting the lack of consensus as to what constitutes appropriate quality criteria for qualitative research. It is a daunting task for those without deep familiarity of the field to choose the best appraisal tool for their purpose. This article provides a description of the structure, content, and objectives of existing appraisal tools for those wanting to evaluate primary qualitative research for a qualitative evidence synthesis. We then discuss common features of appraisal tools and examine their implications for evidence synthesis.
Get full access to this article
View all access and purchase options for this article.
References
Barbour R. S. (2001). Checklists for improving rigour in qualitative research: A case of the tail wagging the dog? British Medical Journal, 322(7294), Article 1115.
Barbour R. S., Barbour M. (2003). Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice, 9, 179–186.
Barnett-Page E., Thomas J. (2009). Methods for the synthesis of qualitative research: a critical review. BMC medical research methodology, 9(1), 59. https://doi.org/10.1186/1471-2288-9-59
Belgrave L. L., Zablotsky D., Guadagno M. A. (2002). How do we talk to each other? Writing qualitative research for quantitative readers. Qualitative Health Research, 12, 1427–1439.
Booth A., Hannes K., Harden A., Noyes J., Harris J., Tong A. (2014). COREQ (Consolidated Criteria for Reporting Qualitative Studies). In Moher D., Altman D. G., Schulz K. F., Simera I., Wager E. (Eds.), Guidelines for reporting health research: A user’s manual (pp. 214–226). Oxford, UK: John Wiley.
Brundisini F., Vanstone M., Hulan D., DeJean D., Giacomini M. (2015). Type 2 diabetes patients’ and providers’ differing perspectives on medication nonadherence: A qualitative meta-synthesis. BMC Health Services Research, 15(1), Article 516.
Campbell R., Pound P., Pope C., Britten N., Pill R., Morgan M., Donovan J. (2003). Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine, 56, 671–684.
Carroll C. (2017). Qualitative evidence synthesis to improve implementation of clinical guidelines. British Medical Journal, 356, Article j80.
Carroll C., Booth A. (2015). Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? Research Synthesis Methods, 6, 149–154.
Carroll C., Booth A., Lloyd-Jones M. (2012). Should we exclude inadequately reported studies from qualitative systematic reviews? An evaluation of sensitivity analyses in two case study reviews. Qualitative Health Research, 22, 1425–1434.
Critical Appraisal Skills Programme. (2016). CASP qualitative research checklist [online]. Retrieved from http://www.casp-uk.net/casp-tools-checklists
Denzin N. K. (2009). The elephant in the living room: Or extending the conversation about the politics of evidence. Qualitative Research, 9, 139–160.
Dixon-Woods M., Booth A., Sutton A. J. (2007). Synthesizing qualitative research: A review of published reports. Qualitative Research, 7, 375–422.
Dixon-Woods M., Shaw R. L., Agarwal S., Smith J. A. (2004). The problem of appraising qualitative research. Quality and Safety in Health Care, 13, 223–225.
Dixon-Woods M., Sutton A., Shaw R., Miller T., Smith J., Young B., . . . Jones D. (2007). Appraising qualitative research for inclusion in systematic reviews: A quantitative and qualitative comparison of three methods. Journal of Health Services Research & Policy, 12, 42–47.
Downe S. (2008). Metasynthesis: A guide to knitting smoke. Evidence Based Midwifery, 6(1), 4–9.
Estabrooks C. A., Field P. A., Morse J. M. (1994). Aggregating qualitative findings: An approach to theory development. Qualitative Health Research, 4, 503–511.
Finlay L. (2002). Negotiating the swamp: The opportunity and challenge of reflexivity in research practice. Qualitative Research, 2, 209–230.
Finfgeld D. L. (2003). Metasynthesis: The state of the art—so far. Qualitative health research, 13(7), 893-904. https://doi.org/10.1177/1049732303253462
First Nations Governance Centre. (2014). Ownership, Control, Access and Possession (OCAP TM): The path to first nations information governance. Ottawa: First Nations Information Governance Centre. Retrieved from http://fnigc.ca/sites/default/files/docs/ocap_path_to_fn_information_governance_en_final.pdf
Garside R. (2014). Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how? Innovation: The European Journal of Social Science Research, 27, 67–79.
Gentles S. J., Jack S. M., Nicholas D. B., McKibbon K. (2014). Critical approach to reflexivity in grounded theory. The Qualitative Report, 19(44), 1–14. Retrieved from https://nsuworks.nova.edu/tqr/vol19/iss44/3
Giacomini M. K., Cook D. J. (2000). Users’ guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Journal of the American Medical Association, 284, 357–362.
Goldsmith M. R., Bankhead C. R., Austoker J. (2007). Synthesising quantitative and qualitative research in evidence-based patient information. Journal of Epidemiology and Community Health, 61, 262–270.
Hannes K., Lockwood C., Pearson A. (2010). A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qualitative Health Research, 20, 1736–1743.
Hannes K., Macaitis K. (2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: Update on a review of published papers. Qualitative Research, 12, 402–442.
Jensen L. A., Allen M. N. (1996). Meta-synthesis of qualitative findings. Qualitative Health Research, 6, 553–560.
Joanna Briggs Institute. (2016). Checklist for qualitative research [online]. Retrieved from http://joannabriggs.org/research/critical-appraisal-tools.html
Kandasamy S., Khalid A. F., Majid U., Vanstone M. (2017). Prostate cancer patient perspectives on the use of information in treatment decision-making: A systematic review and qualitative meta-synthesis. Ontario Health Technology Assessment Series, 17, 1–32.
Koch T., Harrington A. (1998). Reconceptualizing rigour: The case for reflexivity. Journal of Advanced Nursing, 28, 882–890.
Lewin S., Booth A., Glenton C., Munthe-Kaas H., Rashidian A., Wainwright M., . . . Noyes J. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings: Introduction to the series. Implementation Science, 13(Suppl. 1), Article 2.
Lewin S., Glenton C., Munthe-Kaas H., Carlsen B., Colvin C. J., Gülmezoglu M., . . . Rashidian A. (2015). Using qualitative evidence in decision making for health and social interventions: An approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Medicine, 12(10), e1001895.
Lockwood C., Munn Z., Porritt K. (2015). Qualitative research synthesis: Methodological guidance for systematic reviewers utilizing meta-aggregation. International Journal of Evidence-Based Healthcare, 13, 179–187.
Long A. F., Godfrey M. (2004). An evaluation tool to assess the quality of qualitative research studies. International Journal of Social Research Methodology, 7, 181–196.
Majid U., Kandasamy S., Arora N., Vanstone M. (in press). Patient perspectives and experiences. In HPV testing for primary cervical cancer screening: A health technology assessment. Ottawa: Canadian Agency for Drugs and Technologies in Health.
Masood M., Thaliath E. T., Bower E. J., Newton J. T. (2011). An appraisal of the quality of published qualitative dental research. Community Dentistry and Oral Epidemiology, 39, 193–203.
McInnes E., Wimpenny P. (2008). Using qualitative assessment and review instrument software to synthesise studies on older people’s views and experiences of falls prevention. International Journal of Evidence-Based Healthcare, 6, 337–344.
Melia K. M. (2010). Recognizing quality in qualitative research. In SAGE handbook of qualitative health research (pp. 559–574). Thousand Oaks: Sage.
Moher D., Liberati A., Tetzlaff J., Altman D. G. & Prisma Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097.
Noblit G. W., Hare R. D., Hare R. D. (1988). Meta-ethnography: Synthesizing qualitative studies Particularities: Collected Essay on Ethnography and Education 44, 93-123. USA: Sage Publications.
Noyes J., Popay J. (2007). Directly observed therapy and tuberculosis: How can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. Journal of Advanced Nursing, 57, 227–243.
O’Brien B. C., Harris I. B., Beckman T. J., Reed D. A., Cook D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89, 1245–1251.
Popay J., Rogers A., Williams G. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research, 8, 341–351.
Sandelowski M., Barroso J. (2002). Finding the findings in qualitative studies. Journal of nursing scholarship, 34(3), 213-219. https://doi.org/10.1111/j.1547-5069.2002.00213.x
Sandelowski M., Barroso J. (2003). Creating metasummaries of qualitative findings. Nursing research, 52(4), 226-233.
Sandelowski M., Barroso J., Voils C. I. (2007). Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Research in nursing & health, 30(1), 99-111. https://doi.org/10.1002/nur.20176
Saini M., Shlonsky A. (2012). Systematic synthesis of qualitative research. USA: Oxford University Press.
Sandelowski M. (2000). Focus on research methods whatever happened to qualitative description? Research in Nursing & Health, 23, 334–340.
Sandelowski M., Barroso J. (2006). Handbook for synthesizing qualitative research. New York, NY, USA. Springer.
Santiago-Delefosse M., Gavin A., Bruchez C., Roux P., Stephen S. L. (2016). Quality of qualitative research in the health sciences: Analysis of the common criteria present in 58 assessment guidelines by expert users. Social Science & Medicine, 148, 142–151.
Spencer L., Ritchie J., Lewis J., Dillon L. (2003). Quality in qualitative evaluation: A framework for assessing research evidence. London: Cabinet Office, Government Chief Social Researcher’s Office.
Stige B., Malterud K., Midtgarden T. (2009). Toward an agenda for evaluation of qualitative research. Qualitative Health Research, 19, 1504–1516.
Thomas J., Harden A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8(1), Article 45.
Thorne S. (2017). Metasynthetic madness: What kind of monster have we created? Qualitative Health Research, 27, 3–12.
Tong A., Sainsbury P., Craig J. (2007). Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19, 349–357.
Vanstone M., Kandasamy S., Giacomini M., DeJean D., McDonald S. D. (2016). Pregnant women’s perceptions of gestational weight gain: A systematic review and meta-synthesis of qualitative research. Maternal & Child Nutrition, 13, e12374.
Walsh D., Downe S. (2006). Appraising the quality of qualitative research. Midwifery, 22, 108–119.
Whiting P., Wolff R., Mallett S., Simera I., Savović J. (2017). A proposed framework for developing quality assessment tools. Systematic Reviews, 6(1), Article 204.
Yardley L. (2008). Demonstrating validity in qualitative psychology. In Qualitative psychology: A practical guide to research methods . 2, 235–251.
Biographies
Umair Majid was a MSc Candidate in the Health Research Methodology Program at McMaster University at the time this work was conducted. He is currently a PhD student at the Institute of Health Policy, Management, and Evaluation, University of Toronto, Ontario, Canada
Meredith Vanstone is an Assistant Professor in the Department of Family Medicine, member of the Centre for Health Economics and Policy Analysis, and Scientist in the McMaster Faculty of Health Sciences program for Education Research, Innovation & Theory at McMaster University, Hamilton, Ontario, Canada
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
- online_appendix.pdf
- 218.58 KB
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: July 26, 2018
Issue published: November 2018
Keywords
PubMed: 30047306
Authors
Metrics and citations
Metrics
Article usage*
Total views and downloads: 15218
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 103 view articles Opens in new tab
Crossref: 128
-
Health and medical experience of migrant workers: qualitative meta-syn...
-
Paramedicine research with family and bystanders: A methodological rev...
-
Exploration of Dementia Help Seeking Behaviours Amongst UK South Asian...
-
Self-reported explanations for self-injury by people with intellectual...
-
Barriers and enablers in doping, anti-doping, and clean sport: A quali...
-
Maternity care professionals’ preparedness for and experiences of scre...
-
National continuous professional development processes and systems for...
-
Therapists’ and patients’ experiences of using patients’ self-reported...
-
Transgender Youth, Challenges, Responses, and the Juvenile Justice Sys...
-
The prevalence of alcohol references in music and their effect on peop...
-
Parents’ experience of children with acquired brain injury undergoing ...
-
Impact of information credibility on social media information adoption...
-
Care giving and receiving for people with complex emotional needs with...
-
Critically appraising and utilising qualitative health research eviden...
-
Understanding return‐to‐employment experiences after burns: Qualitativ...
-
What is the impact of seasonal ambient changes on the incidence of fal...
-
Qualitative research in dental traumatology—A narrative review
-
Educator-Perceived Barriers and Facilitators to Structured-Physical Ac...
-
The psychosocial experiences of adults diagnosed with coeliac disease:...
-
Professional Identity Formation in Allied Health: A Systematic Review ...
-
Exploring the involvement of tax audit experts in the early stages of ...
-
Balancing on life's ladder: A meta‐ethnography of the existential expe...
-
Laboratory services in the context of prevention of mother-to-child tr...
-
Finding what works—Patients' long‐term experiences of weight maintenan...
-
Inclusive critical appraisal of qualitative and quantitative findings ...
-
Decision‐making in nursing research and practice—Application of the Co...
-
Context matters: A meta-ethnographic study on teachers’ workplace lear...
-
Parental engagement with complementary feeding information in the Unit...
-
Experiences of peer work in drug use service settings: A systematic re...
-
Healthcare staff's experiences of using manual physical restraint: A m...
-
A synthesis of qualitative studies on patient and caregiver experience...
-
A Systematic Literature Review of the Teaching Considerations and Prac...
-
Patient experiences of weight loss and eating after bariatric surgery:...
-
Are physiotherapy student-led services a suitable model of clinical ed...
-
Barriers and facilitators for obtaining support services among underse...
-
Development of a critical appraisal tool for assessing the reporting q...
-
Parents’ experience of children with acquired brain injury undergoing ...
-
Women’s experiences of care and support following perinatal death in h...
-
Hospital clinicians' perspectives of using compression therapy on veno...
-
A Meta-Synthesis of Disability Research in Western Africa
-
A Systematic Literature Review of the Career Choice of Helping Profess...
-
Methods and Ethics in Qualitative Research Exploring Young Children’s ...
-
Exploring internship nursing students' experiences regarding the effec...
-
Evidence Produced While Using Qualitative Methodologies Including Rese...
-
Fear of falling from the perspective of affected persons—A systematic ...
-
USE OF GROUNDED THEORY IN THE EXTRACTION, CODING AND ANALYSIS OF DATA ...
-
UTILIZAÇÃO DA GROUNDED THEORY NA EXTRAÇÃO, CODIFICAÇÃO E ANÁLISE DE DA...
-
Critical appraisal
-
Why do men who have sex with men practice condomless sex? A systematic...
-
“All the fun stuff, the teachers say, ‘that’s dangerous!’” Hearing fro...
-
“Children are precious cargo; we don’t let them take any risks!”: Hear...
-
Theory utilization in applied qualitative nursing research
-
Exploring opportunities and constraints of a certification scheme for ...
-
Indigenous women’s experiences of cervical cancer screening: Incorpora...
-
Integrating local and scientific knowledge in disaster risk reduction:...
-
Communication challenges in social isolation, subjective cognitive dec...
-
A systematic scoping review of communication skills training in medica...
-
Lived experiences of pregnancy and prison through a reproductive justi...
-
Rapid Evidence Assessment Protocol for the Meta-Analysis of Initiative...
-
Women's views and experiences of occasional alcohol consumption during...
-
A meta-synthesis of the perspectives and experiences of healthcare pro...
-
Creation of an online inventory for choosing critical appraisal tools
-
A Systematic Literature Review of the Contribution Accumulation Makes ...
-
Childhood perspectives of parental young onset dementia: A qualitative...
-
Student nurses’ experiences participating in a peer mentoring program ...
-
Leveraging breast cancer screening to promote timely detection, diagno...
-
Pregnant People's Perspectives On Cannabis Use During Pregnancy: A Sys...
-
Beyond harm-producing versus harm-reducing: A qualitative meta-synthes...
-
Barriers and Facilitators to Healthy Eating and Physical Activity Post...
-
Participant perspectives of automated short messaging service interven...
-
Evidence Produced While Using Qualitative Methodologies Including Rese...
-
Methods for Qualitative Analysis and Synthesis
-
Setting Inclusion and Exclusion Criteria
-
The Role and Potential of Information Communication Technology (ICT) i...
-
Experiences of living with long COVID and of accessing healthcare serv...
-
Exploring the Reasons for Low Cataract Surgery Uptake Among Patients D...
-
Exploring the Determinants of Food Choice in Chinese Mainlanders and C...
-
A systematic scoping review of published qualitative research pertaini...
-
Barriers and enablers to screening and diagnosing diabetes distress an...
-
Meta-ethnography in healthcare research: a guide to using a meta-ethno...
-
Gendered experiences of providing informal care for older people: a sy...
-
Behaviours that prompt primary school teachers to adopt and implement ...
-
Motherhood and decision-making among women living with HIV in develope...
-
“Walking into the unknown…” key challenges of pregnancy and early pare...
-
A mega-aggregation framework synthesis of the barriers and facilitator...
-
Value of Qualitative Research in Colorectal Surgery
-
A Scoping Review of Professional Identity Formation in Undergraduate M...
-
The experiences of those affected by parental young onset dementia: A ...
-
The Nature and Extent of Qualitative Research Conducted With Children ...
-
Coping and growing in dilemma: Clinical work experience of front‐line ...
-
Barriers and facilitators of engagement in psychological therapies amo...
-
Personal and social factors that influence physical activity levels in...
-
Understanding Shoulder Pain: A Qualitative Evidence Synthesis Explorin...
-
Decision-making for the infant sleep environment among families with c...
-
Clinician responses to legal requests for hastened death: a systematic...
-
What are the holistic experiences of adults living long-term with the ...
-
Nurses’ Perceived Causes of Medication Administration Errors: A Qualit...
-
Rapid qualitative evidence syntheses (rQES) in health technology asses...
-
The “quality” of JBI qualitative research synthesis: a methodological ...
-
Women’s views and experiences of augmentation of labour with synthetic...
-
Women’s views and experiences of augmentation of labour with synthetic...
-
Rapid reviews as an emerging approach to evidence synthesis in educati...
-
Supporting the use of research evidence in decision-making in crisis z...
-
Practices for evaluating qualitative research in information science
-
Meta-synthesis in Library & Information Science Research
-
The Integration of Resource Patients in Collaborative Research: A Mixe...
-
Book review
-
Lessons learned from applying Sandelowski and Barroso’s approach for s...
-
The Handbook of Behavior Change
-
Critical and Qualitative Approaches to Behavior Change
-
The use of digital storytelling as an approach to translating research...
-
Innovations in framework synthesis as a systematic review method
-
Living with diabetes: literature review and secondary analysis of qual...
-
Critical appraisal of qualitative research: necessity, partialities an...
-
Synthesising Qualitative Research
-
Sporting Activities for Individuals Who Experienced Trauma During Thei...
-
Technical and contextual barriers to oral care: New insights from inte...
-
Questions and answers about conducting systematic reviews in sport and...
-
Humanitarian migrant women's experiences of maternity care in Nordic c...
-
Sínteses de Evidências Qualitativas: guia introdutório
-
Facilitating informed choice about non-invasive prenatal testing (NIPT...
-
A typology of systematic reviews for synthesising evidence on health c...
-
A scoping review found increasing examples of rapid qualitative eviden...
-
Doing Critical Health Communication: Negotiating the Terrain of Transd...
-
Improving the content validity of the mixed methods appraisal tool: a ...
-
Qualitative Research in Nursing and Health Professions Regulation
-
Introduction to Qualitative Research for Novice Investigators
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
