Skip to main content
SpringerLink
Log in

Peer Review of Impact: Could It Work?

  • Chapter
  • First Online:
The Evaluators’ Eye

Abstract

This chapter will describe the main theoretical approaches and literature necessary for the interpretation of the remaining chapters. It explores peer review as a group-led process, and examines what this means for the evaluation of ambiguous objects such as Impact. I also use this chapter to introduce the concept of a dominant definition, providing much of the necessary discourse for the development of this book’s theories and interpretations.

This is a human endeavour, so I think a lot of judgements came into it as part of the issue, I think, but human judgements are, including this one, are, you know, imprecise, coloured by many different things.

P0OutImp1(Post-evaluation)

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A Likert survey was used at the end of each round of interviews to test the validity of the qualitative analysis performed. This Likert survey used statements of the dominant themes emerging from the interviews to test the extent they reflected the opinions of the group. Only dominant themes that emerged from the analysis, and confirmed by the Likert surveys were used.

  2. 2.

    This term is taken directly from Chubb, J., Watermeyer, R. (2016) Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Studies in Higher Education 1–3. Doi:10.1080/03075079.2016.1144182.

  3. 3.

    Many applications these days offer applicants the courtesy of nominating reviewers. This reduces the burden on funding agencies and journals to search for a reviewer, and seemingly gives people a sense of control of the applications’ destiny.

  4. 4.

    Research investigating how concepts of Interdisciplinarity are evaluated by peer review panels, are commonly professed to be a model that also forms the basis of how we consider how Impact is evaluated. Katri Huutoniemi, Evaluating Interdisciplinary Research (10: Oxford: Oxford University Press, 2010).

References

  • Abramo, G., C.A. D’Angelo, and F. Rosati. 2015. Selection committees for academic recruitment: Does gender matter? Research Evaluation 24 (4): 392–404.

    Article  Google Scholar 

  • Aldag, R.J., and S.R. Fuller. 1993. Beyond fiasco: A reappraisal of the groupthink phenomenon and a new model of group decision processes. Psychological Bulletin 113 (3): 533.

    Article  Google Scholar 

  • Andersen, L.B., and T. Pallesen. 2008. “Not just for the money?” How financial incentives affect the number of publications at Danish research institutions. International Public Management Journal 11 (1): 28–47.

    Article  Google Scholar 

  • Arribas-Ayllon, M., and V. Walkerdine. 2008. Foucauldian discourse analysis. In The Sage Handbook of qualitative research in psychology, ed. Carla Willig and Wendy Stainton-Rogers, 91–108. London: Sage.

    Chapter  Google Scholar 

  • Auranen, O., and M. Nieminen. 2010. University research funding and publication performance—An international comparison. Research Policy 39 (6): 822–834.

    Article  Google Scholar 

  • Bornmann, L. 2012. Measuring the societal impact of research. EMBO Reports 13 (8): 673–676.

    Article  Google Scholar 

  • ———. 2013. What is the societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology 64 (2): 217–233.

    Article  Google Scholar 

  • Bornmann, L., G. Wallon, and A. Ledin. 2008. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes. PLoSOne 3 (10): e3480.

    Article  Google Scholar 

  • Bourdieu, P. 1975. The specificity of the scientific field and the social conditions of the progress of reason. Information (International Social Science Council) 14 (6): 19–47.

    Article  Google Scholar 

  • ———. 1984. Distinction: A social critique of the judgement of taste. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Bourdieu, Pierre. 1997. Capital cultural, escuela y espacio social. Siglo xxi.

    Google Scholar 

  • Buxton, M., and S. Hanney. 1996. How can payback from health services research be assessed? Journal of Health Services Research 1 (1): 35–43.

    Google Scholar 

  • Cetina, K.K., J. Clark, C. Modgil, S. Modgil, I.B. Cohen, K. Duffin, S. Strickland, R. Feldhay, Y. Elkana, and R.K. Merton. 1991. Merton’s sociology of science: The first and the last sociology of science? JSTOR 20 (4): 522–526.

    Google Scholar 

  • Chubb, J., and R. Watermeyer. 2016. Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Studies in Higher Education 1–13. https://doi.org/10.1080/03075079.2016.1144182.

  • Chubin, D.E., and E.J. Hackett. 1990. Peerless science: Peer review and US science policy. Albany: State University of New York Press.

    Google Scholar 

  • Cole, S., J.R. Cole, and G.A. Simon. 1981. Chance and consensus in peer review. Science 214 (4523): 881–886.

    Article  Google Scholar 

  • Collins, H.M., and R. Evans. 2002. The third wave of science studies: Studies of expertise and experience. Social Studies of Science 32 (2): 235–296.

    Article  Google Scholar 

  • Comer, D.R. 1995. A model of social loafing in real work groups. Human Relations 48 (6): 647–667.

    Article  Google Scholar 

  • Dahler-Larsen, P. 2007. Evaluation and public management. In The Oxford Handbook of public management, ed. E. Ferlie, L.E. Lynn Jr., and C. Pollitt. Oxford: Oxford University Press.

    Google Scholar 

  • ———. 2014. Constitutive effects of performance indicators: Getting beyond unintended consequences. Public Management Review 16 (7): 969–986.

    Article  Google Scholar 

  • de Jong, S.P., J. Smit, and L. van Drooge. 2015. Scientists’ response to societal impact policies: A policy paradox. Science and Public Policy 43 (1): 102–114.

    Article  Google Scholar 

  • De Rijcke, S., P.F. Wouters, A.D. Rushforth, T.P. Franssen, and B. Hammarfelt. 2016. Evaluation practices and effects of indicator use—A literature review. Research Evaluation 25 (2): 161–169.

    Article  Google Scholar 

  • Deem, R., S. Hillyard, and M. Reed. 2007. Knowledge, higher education, and the new managerialism: The changing management of UK universities. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Derrick, G., I. Meijer, and E. van Wijk. 2014. Unwrapping “impact” for evaluation: A co-word analysis of the UK REF2014 policy documents using VOSviewer. Proceedings of the Science and Technology Indicators Conference.

    Google Scholar 

  • Derrick, G.E., A.S. Haynes, S. Chapman, and W.D. Hall. 2011. The association between four citation metrics and peer rankings of research influence of Australia researchers in six fields of public health. PLoSOne 6: e18521.

    Article  Google Scholar 

  • Derrick, G.E., and V. Pavone. 2013. Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review. Science and Public Policy 40 (5): 563–575.

    Article  Google Scholar 

  • Derrick, G.E., and G.N. Samuel. 2014. The impact evaluation scale: Group panel processes and outcomes in societal impact evaluation. Social Science and Medicine, in press.

    Google Scholar 

  • ———. 2016. The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva 54 (1): 75–97.

    Article  Google Scholar 

  • Donovan, C., L. Butler, A.J. Butt, T.H. Jones, and S.R. Hanney. 2014. Evaluation of the impact of National Breast Cancer Foundation-funded research. The Medical Journal of Australia 200 (4): 214–218.

    Article  Google Scholar 

  • Fernández-Zubieta, A., A. Geuna, and C. Lawson. 2015. Mobility and productivity of research scientists1. Global Mobility of Research Scientists: The Economics of Who Goes Where and Why 105.

    Google Scholar 

  • Finkel, A. 2015. Research Engagement for Australia (REA): Measuring research engagement between universities and end users. Presentation to Universities Australia.

    Google Scholar 

  • Fogelholm, M., S. Leppinen, A. Auvinen, J. Raitanen, A. Nuutinen, and K. Väänänen. 2012. Panel discussion does not improve reliability of peer review for medical research grant proposals. Journal of Clinical Epidemiology 65 (1): 47–52.

    Article  Google Scholar 

  • Frodeman, R., and J. Parker. 2009. Intellectual merit and broader impact: The National Science Foundation’s broader impacts criterion and the question of peer review. Social Epistemology 23 (3–4): 337–345.

    Article  Google Scholar 

  • Geuna, A., and M. Piolatto. 2016. Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while). Research Policy 45 (1): 260–271.

    Article  Google Scholar 

  • Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, and P. Scott. 1994. The new production of knowledge: The dynamics of science and research in contemporary societies. London: SAGE.

    Google Scholar 

  • Giraudeau, B., C. Leyrat, A. Le Gouge, J. Leger, and A. Caille. 2011. Peer review of grant applications: A simple method to identify proposals with discordant reviews. PLoSOne 6 (11): e27557.

    Article  Google Scholar 

  • Grant, J., P.-B. Brutscher, S. Kirk, L. Butler, and S. Wooding. 2010. Capturing research impacts: A review of International practice. Documented briefing. RAND Corporation.

    Google Scholar 

  • Greene, J.C. 1997. Evaluation as advocacy. Evaluation Practice 18: 25–36.

    Article  Google Scholar 

  • ———. 2000. Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation 2000 (85): 13–26.

    Article  Google Scholar 

  • Hackett, E.J., and D. E. Chubin. 2003. Peer review for the 21st century: Applications to education research. Ed. National Research Council. Washington, DC.

    Google Scholar 

  • Hall, D., and S. Buzwell. 2013. The problem of free-riding in group projects: Looking beyond social loafing as reason for non-contribution. Active Learning in Higher Education 14 (1): 37–49.

    Article  Google Scholar 

  • Harnad, S. 1985. Rational disagreement in peer review. Science, Technology, & Human Values 10 (3): 55–62.

    Article  Google Scholar 

  • Haynes, A.S., G.E. Derrick, S. Chapman, S. Redman, W.D. Hall, J. Gillespie, and H. Sturk. 2011. From “our world” to the “real world”: Exploring the views and behaviour of policy-influential Australian public health researchers. Social Science & Medicine 72: 1047–1055.

    Article  Google Scholar 

  • HEFCE. 2010. REF2014: Panel criteria and working methods. http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf. Accessed 1 Mar 2016.

  • Hemlin, S., and S.B. Rasmussen. 2006. The shift in academic quality control. Science, Technology, & Human Values 31 (2): 173–198.

    Article  Google Scholar 

  • Herbst, M. 2007. Financing public universities. New York: Springer.

    Google Scholar 

  • Hicks, D. 2012. Performance-based university research funding systems. Research policy 41 (2): 251–261.

    Article  Google Scholar 

  • Hicks, D., P.F. Wouters, L. Waltman, S. De Rijcke, and I. Rafols. 2015. The Leiden Manifesto for research metrics. Nature 520 (7548): 429.

    Article  Google Scholar 

  • Huutoniemi, K. 2010. Evaluating interdisciplinary research. Oxford: Oxford University Press.

    Google Scholar 

  • ———. 2012. Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 42 (6): 897–921.

    Article  Google Scholar 

  • Ingwersen, P., and B. Larsen. 2014. Influence of a performance indicator on Danish research production and citation impact 2000–12. Scientometrics 101 (2): 1325–1344.

    Article  Google Scholar 

  • Janis, I.L. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Boston, MA: Houghton Mifflin Company.

    Google Scholar 

  • Johnston, S.C., J.D. Rootenberg, S. Katrak, W.S. Smith, and J.S. Elkins. 2006. Effect of a US National Institutes of Health programme of clinical trials on public health and costs. Lancet 367 (9519): 1319–1327.

    Article  Google Scholar 

  • Knorr-Cetina, K.D. 1983. The ethnographic study of scientific work: Towards a constructivist interpretation of science. London: Sage.

    Google Scholar 

  • ———. 1991. Epistemic cultures: Forms of reason in science. History of Political Economy 23 (1): 105–122.

    Article  Google Scholar 

  • Knott, M. 2015. Academic publications to become less important when funding university research. Sydney: Sydney Morning Herald.

    Google Scholar 

  • Kuruvilla, S., N. Mays, and G. Walt. 2007. Describing the impact of health services and policy research. Journal of Health Services Research & Policy 12 (suppl 1): 23–31.

    Article  Google Scholar 

  • Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.

    Book  Google Scholar 

  • Lamont, M., and K. Huutoniemi. 2011. Opening the black box of evaluation: How quality is recognized by peer review panels. Bulletin SAGW 2: 47–49.

    Google Scholar 

  • Langfeldt, L. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31 (6): 820–841.

    Article  Google Scholar 

  • ———. 2004. Expert panels evaluating research: Decision-making and sources of bias. Research Evaluation 13 (1): 51–62.

    Article  Google Scholar 

  • ———. 2006. The policy challenges of peer review: Managing bias, conflict of interests and multidisciplinary assessments. Research Evaluation 15 (1): 31–41.

    Article  Google Scholar 

  • Latane, B., K. Williams, and S. Harkins. 1979. Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology 37 (6): 822–832.

    Article  Google Scholar 

  • Laudel, G., and G. Origgi. 2006. Introduction to a special issue on the assessment of interdisciplinary research. Oxford: Oxford University Press.

    Google Scholar 

  • Leisyte, L., and J.R. Dee. 2012. Understanding academic work in a changing institutional environment. In Higher education: Handbook of theory and research, ed. J.D. Smart, 123–206. New York: Springer.

    Chapter  Google Scholar 

  • Levi, D. 2015. Group dynamics for teams. London: Sage Publications.

    Google Scholar 

  • Luukkonen, T. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation 21: 48–60.

    Article  Google Scholar 

  • Mark, M.M., and R.L. Shotland. 1985. Stakeholder-based evaluation and value judgments. Evaluation Review 9 (5): 605–626.

    Article  Google Scholar 

  • Martin, B.R. 2011. The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation 20 (3): 247–254.

    Article  Google Scholar 

  • Mayo, N.E., J. Brophy, M.S. Goldberg, M.B. Klein, S. Miller, R.W. Platt, and J. Ritchie. 2006. Peering at peer review revealed high degree of chance associated with funding of grant applications. Journal of Clinical Epidemiology 59 (8): 842–848.

    Article  Google Scholar 

  • Merton, R.K. 1973. The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago press.

    Google Scholar 

  • Molas-Gallart, J., and P. Tang. 2011. Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation 20 (3): 219–226.

    Article  Google Scholar 

  • Mollick, E., and R. Nanda. 2015. Wisdom or madness? Comparing crowds with expert evaluation in funding the arts. Management Science 62 (6): 1533–1553.

    Article  Google Scholar 

  • Nowotny, H., P. Scott, and M. Gibbons. 2001. Re-thinking science: Knowledge and the public in an age of uncertainty. Argentina: SciELO.

    Google Scholar 

  • Olbrecht, M., and L. Bornmann. 2010. Panel peer review of grant applications: What do we know from research in social psychology on judgement and decision making in groups? Research Evaluation 19 (4): 293–304.

    Article  Google Scholar 

  • Oortwijn, W.J., S.R. Hanney, A. Ligtvoet, S. Hoorens, S. Wooding, J. Grant, M.J. Buxton, and L.M. Bouter. 2008. Assessing the impact of health technology assessment in the Netherlands. International Journal of Technology Assessment in Health Care 24 (03): 259–269.

    Article  Google Scholar 

  • Pielke, R.A., and R. Byerly. 1998. Beyond basic and applied. Physics Today 51 (2): 42–46.

    Article  Google Scholar 

  • Pier, E.L., J. Raclaw, A. Kaatz, M. Brauer, M. Carnes, M.J. Nathan, and C.E. Ford. 2017. ‘Your comments are meaner than your score’: Score calibration talk influence intra- and inter-panel variability during scientific grant peer review. Research Evaluation 26 (1): 1–14.

    Article  Google Scholar 

  • Porter, A.L., and F.A. Rossini. 1985. Peer review of interdisciplinary research proposals. Science, Technology, & Human Values 10 (3): 33–38.

    Article  Google Scholar 

  • Porter, R. 2005. What do grant reviewers really want, anyway? Journal of Research Administration 36 (2): 5–13,13.

    Google Scholar 

  • Roumbanis, L. 2016. Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups. Social Studies of Science 47: 1–22.

    Google Scholar 

  • Roy, R. 1985. Funding science: The real defects of peer review and an alternative to it. Science, Technology, & Human Values 10 (3): 73–81.

    Article  Google Scholar 

  • Salter, A.J., and B.R. Martin. 2001. The economic benefits of publicly funded basic research: A critical review. Research policy 30 (3): 509–532.

    Article  Google Scholar 

  • Sandstrom, U., U. Heyman, and P. Van den Besselaar. 2014. The complex relationahip between competitive funding and performance. In Context counts: Pathways to master big and little data – STI, ed. E. Noyons, 523–533. Leiden: CWTS. https://doi.org/10.13140/2.1.5036.6728.

    Google Scholar 

  • Simms, A., and T. Nichols. 2014. Social loafing: A review of the literature. Journal of Management Policy and Practice 15 (1): 58.

    Google Scholar 

  • Sivertsen, G., and J. Schneider. 2012. Evaluering av den bibliometriske forskningsindikator.

    Google Scholar 

  • Smith, R. 2001. Measuring the social impact of research—Difficult but necessary. British Medical Journal 323: 528. https://doi.org/10.1136/bmj.323.7312.528

  • Smith, S., V. Ward, and A. House. 2011. ‘Impact’ in the proposals for the UK’s Research Excellence Framework: Shifting the boundaries of academic autonomy. Research Policy 40 (10): 1369–1379.

    Article  Google Scholar 

  • Solomon, M. 2006. Groupthink versus the wisdom of crowds: The social epistemology of deliberation and dissent. The Southern Journal of Philosophy 44 (S1): 28–42.

    Article  Google Scholar 

  • Spaapen, J., and L. Van Drooge. 2011. Introducing ‘productive interactions’ in social impact assessment. Research Evaluation 20 (3): 211–218.

    Article  Google Scholar 

  • Thaler Richard, H., and R. Sunstein Cass. 2008. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.

    Google Scholar 

  • Travis, G.D.L., and H.M. Collins. 1991. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values 16 (3): 322–341.

    Article  Google Scholar 

  • van Arensbergen, P. 2014. Talent proof. Selection processes in research funding and careers. Den Haag: Rathenau Instituut.

    Google Scholar 

  • van Arensbergen, P., I. van der Weijden, and P. van den Besselaar. 2014. The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels. Research Evaluation 23 (4): 298–311.

    Article  Google Scholar 

  • Van der Meulen, Barend, and Arie Rip. 2000. Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation 9 (1): 11–25.

    Article  Google Scholar 

  • Watermeyer, R. 2015. Lost in the ‘third space’: The impact of public engagement in higher education on academic identity, research practice and career progression. European Journal of Higher Education 5 (3): 331–347.

    Article  Google Scholar 

  • ———. 2016. Impact in the REF: Issues and obstacles. Studies in Higher Education 41 (2): 199–214.

    Article  Google Scholar 

  • Watermeyer, R., and A. Hedgecoe. 2016. Selling ‘impact’: Peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. Journal of Education Policy 31 (5): 651–665.

    Article  Google Scholar 

  • Willmott, H. 2011. Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization 18 (4): 429–442.

    Article  Google Scholar 

  • Wooding, S., S. Hanney, M. Buxton, and J. Grant. 2005. Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology 44 (9): 1145–1156.

    Article  Google Scholar 

  • Wooding, S., S. Hanney, A. Pollitt, M. Buxton, and J. Grant. 2011. Project retrosight: Understanding the returns from cardiovascular and stroke research: the policy report. Rand Health Quarterly 1 (1): 16.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Lancaster University, Lancaster, UK

    Gemma Derrick

Authors
  1. Gemma Derrick
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2018 The Author(s)

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Derrick, G. (2018). Peer Review of Impact: Could It Work?. In: The Evaluators’ Eye. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-63627-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-63627-6_2

  • Published:

  • Publisher Name: Palgrave Macmillan, Cham

  • Print ISBN: 978-3-319-63626-9

  • Online ISBN: 978-3-319-63627-6

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation