Summary
Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.
This article is available free of charge.
Simply log in to access the full article, or register for free if you do not yet have a username and password.
Already registered?
One-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Not yet registered?
Register for free
References
- 1.
Population sciences, translational research, and the opportunities and challenges for genomics to reduce the burden of cancer in the 21st century.Cancer Epidemiol Biomarkers Prev. 2011; 20: 2105-2114
- 2.
Incongruence between test statistics and P values in medical papers.BMC Med Res Methodol. 2004; 4: 13
- 3.
Believe it or not: how much can we rely on published data on potential drug targets?.Nat Rev Drug Discov. 2011; 10: 712-713
- 4.
Drug development: raise standards for preclinical cancer research.Nature. 2012; 483: 531-533
- 5.
Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects.J Clin Epidemiol. 2011; 64: 1060-1069
- 6.
Comparison of effect sizes associated with biomarkers reported in highly cited individual articles and in subsequent meta-analyses.JAMA. 2011; 305: 2200-2210
- 7.
Why are so many epidemiology associations inflated or wrong? Does poorly conducted animal research suggest implausible hypotheses?.Ann Epidemiol. 2009; 19: 220-224
- 8.
Bias analysis. Chapter 19.in: Rothman KJ Greenland S Lash TL Modern epidemiology. 3rd edn. Lippincott–Williams–Wilkins, Philadelphia2008: 345-380
- 9.
Multiple comparisons and association selection in general epidemiology.Int J Epidemiol. 2008; 37: 430-434
- 10.
Why most discovered true associations are inflated.Epidemiology. 2008; 19: 640-648
- 11.
Potential etiologic and functional implications of genome-wide association loci for human diseases and traits.Proc Natl Acad Sci USA. 2009; 106: 9362-9367
- 12.
Expectations, validity, and reality in omics.J Clin Epidemiol. 2010; 63: 945-949
- 13.
Science mapping analysis characterizes 235 biases in biomedical research.J Clin Epidemiol. 2010; 63: 1205-1215
- 14. Statistical issues in drug development. 2nd edn. Wiley, New York2007
- 15.
Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials.Ann Intern Med. 2012; 157: 429-438
- 16.
Empirical evidence of design-related bias in studies of diagnostic tests.JAMA. 1999; 282: 1061-1066
- 17.
Evidence of bias and variation in diagnostic accuracy studies.CMAJ. 2006; 174: 469-476
- 18.
Prognostic effect size of cardiovascular biomarkers in datasets from observational studies versus randomised trials: meta-epidemiology study.BMJ. 2011; 343: d6829
- 19.
The false-positive to false-negative ratio in epidemiologic studies.Epidemiology. 2011; 22: 450-456
- 20.
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
- 21.
Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association.Circulation. 2009; 119: 2408-2416
- 22.
Assessment of cumulative evidence on genetic associations: interim guidelines.Int J Epidemiol. 2008; 37: 120-132
- 23.
Recommendations and proposed guidelines for assessing the cumulative evidence on joint effects of genes and environments on cancer occurrence in humans.Int J Epidemiol. 2012; 41: 686-704
- 24.
Increasing value and reducing waste: addressing inaccessible research.Lancet. 2014; (published online Jan 8.)
- 25.
Minimum information about a microarray experiment (MIAME)–toward standards for microarray data.Nat Genet. 2001; 29: 365-371
- 26.
A catalogue of reporting guidelines for health research.Eur J Clin Invest. 2010; 40: 35-53
- 27.
Describing reporting guidelines for health research: a systematic review.J Clin Epidemiol. 2011; 64: 718-742
- 28.
A gold standard publication checklist to improve the quality of animal studies, to fully integrate the three Rs, and to make systematic reviews more feasible.Altern Lab Anim. 2010; 38: 167-182
- 29.
Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research.PLoS Biol. 2010; 8: e1000412
- 30.
Comparison of protocols and registry entries to published reports for randomised controlled trials.Cochrane Database Syst Rev. 2011; 1: MR000031
- 31.
The continuing unethical conduct of underpowered clinical trials.JAMA. 2002; 288: 358-362
- 32.
The prevalence of underpowered randomized clinical trials in rheumatology.J Rheumatol. 2005; 32: 2083-2088
- 33.
The persistence of underpowered studies in psychological research: causes, consequences, and remedies.Psychol Methods. 2004; 9: 147-163
- 34.
Statistical power, sample size, and their reporting in randomized controlled trials.JAMA. 1994; 272: 122-124
- 35.
An evaluation of the feasibility, cost and value of information of a multicentre randomised controlled trial of intravenous immunoglobulin for sepsis (severe sepsis and septic shock): incorporating a systematic review, meta-analysis and value of information analysis.Health Technol Assess. 2012; 16: 1-186
- 36.
Nonsignificance plus high power does not imply support for the null over the alternative.Ann Epidemiol. 2012; 22: 364-368
- 37.
When are “positive” clinical trials in oncology truly positive?.J Natl Cancer Inst. 2011; 103: 16-20
- 38.
Current pharmacologic treatment of dementia: a clinical practice guideline from the American College of Physicians and the American Academy of Family Physicians.Ann Intern Med. 2008; 148: 370-378
- 39.
Repeat revascularization is a faulty end point for clinical trials.Circ Cardiovasc Qual Outcomes. 2012; 5: 249-250
- 40.
Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials.BMJ. 2007; 334: 786
- 41.
A systematic examination of the citation of prior research in reports of randomized, controlled trials.Ann Intern Med. 2011; 154: 50-55
- 42.
How to increase value and reduce waste when research priorities are set.Lancet. 2014; (published online Jan 8.)
- 43.
Perfect study, poor evidence: interpretation of biases preceding study design.Semin Hematol. 2008; 45: 160-166
- 44.
The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence.BMJ. 2010; 341: c4875
- 45.
What are the implications of optimism bias in clinical research?.Lancet. 2006; 367: 449-450
- 46.
Comparator bias: why comparisons must address genuine uncertainties. JLL Bulletin: commentaries on the history of treatment evaluation.(accessed Nov 29, 2013).
- 47.
Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review.BMJ. 2006; 333: 782
- 48.
Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study.BMJ. 2007; 335: 1202-1205
- 49.
Commentary: on ‘Quality in epidemiological research: hould we be submitting papers before we have the results and submitting more hypothesis generating research?’.Int J Epidemiol. 2007; 36: 944-945
- 50.
Review of clinical protocols at The Lancet.Lancet. 2001; 357: 1819-1820
- 51.
Open Medicine endorses PROSPERO.Open Med. 2011; 5: e65-e66
- 52.
Why prospective registration of systematic reviews makes sense.Syst Rev. 2012; 1: 7
- 53.
The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.Syst Rev. 2012; 1: 2
- 54.
Public availability of published research data in high-impact journals.PLoS One. 2011; 6: e24357
- 55.
Selective reporting in clinical trials: analysis of trial protocols accepted by The Lancet.Lancet. 2008; 372: 201
- 56.
Adjusting for nonignorable drop-out using semiparametric nonresponse models.J Am Stat Assoc. 1999; 94: 1096-1120
- 57. Statistical analysis with missing data. 2nd edn. Wiley, New York2002
- 58.
What are pragmatic trials?.BMJ. 1998; 316: 285
- 59.
Pragmatic trials—guides to better patient care?.N Engl J Med. 2011; 364: 1685-1687
- 60.
Patient-important outcomes in registered diabetes trials.JAMA. 2008; 299: 2543-2549
- 61.
Patient-important outcomes in diabetes—time for consensus.Lancet. 2007; 370: 1104-1106
- 62.
Size matters: just how big is BIG? Quantifying realistic sample size requirements for human genome epidemiology.Int J Epidemiol. 2009; 38: 263-273
- 63.
A formal risk-benefit framework for genomic tests: facilitating the appropriate translation of genomics into clinical practice.Genet Med. 2010; 12: 686-693
- 64.
Effect of population heterogenization on the reproducibility of mouse behavior: a multi-laboratory study.PLoS One. 2011; 6: e16461
- 65.
Systematic variation improves reproducibility of animal experiments.Nat Methods. 2010; 7: 167-168
- 66.
Emulating multicentre clinical stroke trials: a new paradigm for studying novel interventions in experimental models of stroke.Int J Stroke. 2009; 4: 471-479
- 67.
International, multicenter randomized preclinical trials in translational stroke research: it's time to act.J Cereb Blood Flow Metab. 2012; 32: 933-935
- 68.
Exploring the geometry of treatment networks.Ann Intern Med. 2008; 148: 544-553
- 69.
Genetic epidemiology with a capital E, ten years after.Genet Epidemiol. 2011; 35: 845-852
- 70.
The emergence of networks in human genome epidemiology: challenges and opportunities.Epidemiology. 2007; 18: 1-8
- 71.
Quality, quantity and harmony: the DataSHaPER approach to integrating data across bioclinical studies.Int J Epidemiol. 2010; 39: 1383-1393
- 72.
The importance of experimental design in proteomic mass spectrometry experiments: some cautionary takes.Briefings Functional Genomics. 2005; 3: 322-331
- 73.
The inexact use of Fisher's Exact Test in six major medical journals.JAMA. 1989; 261: 3430-3433
- 74.
Poor-quality medical research: what can journals do?.JAMA. 2002; 287: 2765-2767
- 75.
Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective.JAMA. 2012; 307: 1636-1640
- 76.
Stakeholder assessment of the evidence for cancer genomic tests: insights from three case studies.Genet Med. 2012; 14: 656-662
- 77.
Transparency and disclosure, neutrality and balance: shared values or just shared words?.J Epidemiol Community Health. 2012; 66: 967-970
- 78.
An empirical assessment of validation practices for molecular classifiers.Brief Bioinform. 2011; 12: 189-202
- 79.
Case studies in reproducibility.Brief Bioinform. 2011; 12: 288-300
- 80.
Repeatability of published microarray gene expression analyses.Nat Genet. 2009; 41: 149-155
- 81.
Reference bias in reports of drug trials.Br Med J (Clin Res Ed). 1987; 295: 654-656
- 82.
How citation distortions create unfounded authority: analysis of a citation network.BMJ. 2009; 339: b2680
- 83.
The scandal of poor medical research.BMJ. 1994; 308: 283-284
- 84.
Research needs grants, funding and money—missing something?.Eur J Clin Invest. 2012; 42: 349-351
- 85.
Persistence of contradicted claims in the literature.JAMA. 2007; 298: 2517-2526
- 86.
Embracing the concept of reproducible research.Biostatistics. 2010; 11: 375
- 87.
Reproducible research and the substantive context.Biostatistics. 2010; 11: 376-378
- 88.
Reproducible research: moving toward research the public can really trust.Ann Intern Med. 2007; 146: 450-453
- 89.
Reproducible research and biostatistics.Biostatistics. 2009; 10: 405-408
- 90.
Reproducible research in computational science.Science. 2011; 334: 1226-1227
- 91.
Improving validation practices in “omics” research.Science. 2011; 334: 1230-1232
- 92.
Reproducibility concerns.Nat Med. 2012; 18: 1736-1737
- 93. Evolution of translational omics: lessons learned and the path forward. Institute of Medicine National Academies of Sciences Press, Washington, DC2012
- 94.
Scientific research in the age of omics: the good, the bad, and the sloppy.J Am Med Inform Assoc. 2013; 20: 125-127
- 95. The future of reputation. Yale University Press, New Haven2007
- 96.
Testing treatments: better research for better healthcare, 2nd edn.(accessed Nov 29, 2013).
- 97.
Association of trial registration with the results and conclusions of published trials of new oncology drugs.Trials. 2009; 10: 116
- 98.
Many randomized trials of physical therapy interventions are not adequately registered: a survey of 200 published trials.Phys Ther. 2013; 93: 299-309
- 99.
Disclosure of conflicts of interest by authors of clinical trials and editorials in oncology.J Clin Oncol. 2007; 25: 4642-4647
- 100.
Frequency, nature, effects, and correlates of conflicts of interest in published clinical cancer research.Cancer. 2009; 115: 2783-2791
- 101.
Financial, nonfinancial and editors' conflicts of interest in high-impact biomedical journals.Eur J Clin Invest. 2013; 43: 660-667
Article info
Publication history
Published: January 08, 2014
Identification
Copyright
© 2014 Elsevier Ltd. All rights reserved.
ScienceDirect
Access this article on ScienceDirectLinked Articles
- How should medical science change?
-
In December, 2013, Randy Schekman received a Nobel Prize in Physiology or Medicine for his codiscovery (with James Rothman and Thomas Südhof) of the cellular machinery regulating vesicle traffic. He used the occasion to launch a ferocious attack against what he called “luxury journals”—Nature, Science, and Cell. Although he didn't mention The Lancet, JAMA, or The New England Journal of Medicine, it probably isn't unreasonable to think he would include us in his definition of “luxury journal”. This is what he wrote in The Guardian: “These luxury journals are supposed to be the epitome of quality, publishing only the best research.
- Full-Text
-
- Biomedical research: increasing value, reducing waste
-
Of 1575 reports about cancer prognostic markers published in 2005, 1509 (96%) detailed at least one significant prognostic variable.1 However, few identified biomarkers have been confirmed by subsequent research and few have entered routine clinical practice.2 This pattern—initially promising findings not leading to improvements in health care—has been recorded across biomedical research. So why is research that might transform health care and reduce health problems not being successfully produced?
- Full-Text
-