Skip to main content
Intended for healthcare professionals
Restricted access
Research article
First published October 2005

Difficulties in evaluating public engagement initiatives: reflections on an evaluation of the UK GM Nation? public debate about transgenic crops

Abstract

In the realm of risk management, and policy-making more generally, “public engagement” is often advocated as an antidote to pathologies associated with traditional methods of policy-making, and associated deficit-model-driven communication strategies. The actual benefits of public engagement are, however, difficult to establish without thorough evaluation of specific engagement processes. Unfortunately, rigorous evaluation is difficult, and, perhaps for this reason, it has rarely been undertaken. In this paper we highlight a number of these difficulties in the light of our experiences in evaluating a major engagement initiative, namely the GM Nation? publice debate on the possible commercialization of transgenic crops, which took place in Britain in 2003. The difficulties we identify seem likely to be relevant to many, if not most, engagement evaluations. They are concerned with both theoretical/normative (how one should evaluate) and practical (how one does evaluate) issues. We suggest a number of possible solutions to these evaluation difficulties.

Get full access to this article

View all access and purchase options for this article.

References

AEBC (2001) Crops on Trial. London: Agriculture and Environment Biotechnology Commission .
Bloor, M. (1978) “On the Analysis of Observational Data; a Discussion of the Worth and Uses of Inductive Techniques and Respondent Validation,” Sociology 12(3): 545–557 .
Bryman, A. and Cramer, D. (1997) Quantitative Data Analysis. London: Routledge .
Cabinet Office (2002) Risk: Improving Government's Ability to Handle Risk and Uncertainty. London: Cabinet Office Strategy Unit .
Chess, C. (2000) “Evaluating Environmental Public Participation: Methodological Questions,” Journal of Environmental Planning and Management 43(6): 769–784 .
Chess, C. and Purcell, K. (1999) “Public Participation and the Environment: Do We Know What Works?,” Environmental Science and Technology 33(16): 2685–2692 .
Clarke, A. (1999) Evaluation Research: an Introduction to Principles, Methods and Practice. London: SAGE .
Fiorino, D. J. (1990) “Citizen Participation and Environmental Risk: a Survey of Institutional Mechanisms,” Science, Technology, & Human Values 15(2): 226–243 .
Frewer, L. J. (1999) “Risk Perception, Social Trust, and Public Participation into Strategic Decision-making: Implications for Emerging Technologies,” Ambio 28: 569–574 .
Funtowicz, S. and Ravetz, J. (1992) “Risk Management as a Post-normal Science,” Risk Analysis 12(1): 95–97 .
Gaskell, G. and Bauer, M.W. (2001) Biotechnology 1996–2000: The Years of Controversy. London: The Science Museum .
Glaser, B. and Strauss, A. (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. London: Weidenfeld & Nicholson .
Halford, N. G. (2004) “Prospects for Genetically Modified Crops,” Annals of Applied Biology 145(1): 17–24 .
Horlick-Jones, T. (1998) “Meaning and Contextualisation in Risk Assessment,” Reliability Engineering and System Safety 59: 79–89 .
Horlick-Jones, T. (2004) “Experts in Risk?... Do They Exist?,” Health, Risk & Society 6(2): 107–114 .
Horlick-Jones, T. and Rosenhead, J. (2002) “Investigating Risk, Organisations and Decision Support through Action Research,” Risk Management: an International Journal 4(4): 45–63 .
Horlick Jones, T., Pidgeon, N., Walls, J. and Rowe, G. (2002) Proposal for the Evaluation of the UK Public Debate on the Possible Commercialisation of Genetically Modified Crops . Paper submitted to the GM Public Debate Steering Board, October 2002.
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W. and O'Riordan, T. (2004) A Deliberative Future? An Independent Evaluation of the GM Nation? Public Debate about the Possible Commercialisation of Transgenic Crops in Britain, 2003. University of East Anglia, Programme on Understanding Risk, Working Paper 04-02.
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W. and O'Riordan, T. (submitted) “On Evaluating the GM Nation? Public Debate about the Commercialisation of Transgenic Crops in Britain.”
Horlick-Jones, T., Walls, J., Rowe, G., Pidgeon, N., Poortinga, W., O'Riordan, T., Murdock, G., Tait, J. and Bruce, A. (2003) “Memorandum Submitted by the Understanding Risk Team and Collaborators,” in House of Commons Environment, Food and Rural Affairs Committee Conduct of the GM Public Debate. Eighteenth Report of the Session 2002–03 HC 1220, Ev50–Ev56. London: The Stationery Office.
House of Commons Environment, Food and Rural Affairs Committee (2003) Conduct of the GM Public Debate. Eighteenth Report of the Session 2002–03 HC 1220. London: The Stationery Office .
House of Lords Select Committee on Science and Technology (2000) Science and Society Third Report. HMSO, HL Paper 38.
Irwin, A. and Wynne, B. (eds) (1996) Misunderstanding Science? The Public Reconstruction of Science and Technology. Cambridge: Cambridge University Press .
Jasanoff, S. (1990) The Fifth Branch: Scientific Advisors as Policy Makers. Cambridge, MA: Harvard University Press .
Joss, S. (1995) “Evaluating Consensus Conferences: Necessity or Luxury?,” in S. Joss and J. Durant (eds) Public Participation in Science: The Role of Consensus Conferences in Europe, pp. 89–108. London: The Science Museum .
Lord Phillips of Worth Matravers, Bridgeman, J. and Ferguson-Smith, M. (2000) The BSE Inquiry [The Phillips Report]. London: The Stationery Office .
Myhr, A.I. and Traavik, T. (2003) “Genetically Modified (GM) Crops: Precautionary Science and Conflicts of Interests,” Journal of Agricultural and Environmental Ethics 16(3): 227–247 .
National Research Council (1996) Understanding Risk: Informing Decisions in a Democratic Society. Washington DC: National Academy Press .
Nielsen, C.P., Thierfelder K. and Robinson, S. (2003) “Consumer Preferences and Trade in Genetically Modified Foods,” Journal of Policy Modeling 25(8): 777–794 .
Oppenheim, A.N. (1992) Questionnaire Design, Interviewing and Attitude Measurement. London: Pinter .
Patton, M. (1990) Qualitative Evaluation and Research Methods, 2nd edn. London: Sage .
Pidgeon, N.F., Poortinga, W., Rowe, G., Horlick-Jones, T., Walls, J. and O'Riordan, T. (2005) “Using Surveys in Public Participation Processes for Risk Decision-making: the Case of the 2003 British GM Nation? Public Debate,” Risk Analysis 25(2): 467–479 .
POST (2001) Open Channels: Public Dialogue in Science and Technology. Parliamentary Office of Science and Technology, Report 152.
Public Debate Steering Board (PDSB) (2003) GM Nation? The Findings of the Public Debate. Department of Trade and Industry . URL: www.gmnation.org.uk.
Robson, C. (2002) Real World Research, 2nd edn. Oxford: Blackwell .
Rossi, P.H., Freeman, H.E. and Lipsey, M.W. (1999) Evaluation: a Systematic Approach, 6th edn. London: Sage .
Rowe, G. and Frewer, L.J. (2000) “Public Participation Methods: a Framework for Evaluation,” Science, Technology, & Human Values 25(1): 3–29 .
Rowe, G. and Frewer, L.J. (2004) “Evaluating Public Participation Exercises: a Research Agenda,” Science, Technology, & Human Values 29(4): 512–556 .
Rowe, G. and Frewer, L.J. (2005) “A Typology of Public Engagement Mechanisms,” Science, Technology, & Human Values 30(2): 251–290 .
Rowe, G., Horlick-Jones, T., Walls, J., Poortinga, W. and Pidgeon, N. (submitted) “Analysis of a Normative Framework for Evaluating Public Engagement Exercises: Reliability, Validity and Limitations.”
Rowe, G., Marsh, R. and Frewer, L.J. (2004) “Evaluation of a Deliberative Conference,” Science, Technology, & Human Values 29(1): 88–121 .
Shaw, I. (1999) Qualitative Evaluation. London: Sage .
Silverman, D. (1998) Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction. London: SAGE .
Thorpe, A. and Robinson, C. (2004) “When Goliaths Clash: US and EU Differences over the Labelling of Food Products Derived from Genetically Modified Organisms,” Agriculture and Human Values 21(4): 287–298 .
UK Government (2002) UK Government Response to AEBC Advice Submitted in April 2002. Press release, URL: http://www.aebc.gov.uk/aebc/reports/public_debate_advice.shtml.
Walls, J., Pidgeon, N., Weyman, A. and Horlick-Jones, T. (2004) “Critical Trust: Understanding Lay Perceptions of Health and Safety Risk Regulation,” Health, Risk & Society 6(2): 133–150 .
Wynne, B. (1991) “Knowledges in Context,” Science, Technology, & Human Values 16(1): 111–121 .

Cite article

Cite article

Cite article

OR

Download to reference manager

If you have citation software installed, you can download article citation data to the citation manager of your choice

Share options

Share

Share this article

Share with email
EMAIL ARTICLE LINK
Share on social media

Share access to this article

Sharing links are not relevant where the article is open access and not available if you do not have a subscription.

For more information view the Sage Journals article sharing page.

Information, rights and permissions

Information

Published In

Article first published: October 2005
Issue published: October 2005

Rights and permissions

Request permissions for this article.

Authors

Affiliations

Gene Rowe
Consumer Science Group at the Institute of Food Research, Norwich (UK)
Tom Horlick-Jones
School of Social Sciences at Cardiff University (Wales, UK)
John Walls
School of Environmental Sciences at the University of East Anglia (UK)
Nick Pidgeon
University of East Anglia (UK)

Metrics and citations

Metrics

Journals metrics

This article was published in Public Understanding of Science.

VIEW ALL JOURNAL METRICS

Article usage*

Total views and downloads: 779

*Article usage tracking started in December 2016


Altmetric

See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores



Articles citing this one

Receive email alerts when this article is cited

Web of Science: 130 view articles Opens in new tab

Crossref: 134

  1. Closing the gap: establishing a ‘feedback loop’ for effective parliame...
    Go to citation Crossref Google Scholar
  2. Delphi-based future scenarios: A bibliometric analysis of climate chan...
    Go to citation Crossref Google Scholar
  3. Igniting public engagement with biodiversity conservation: exploring t...
    Go to citation Crossref Google Scholar
  4. How Do Multiple Actors Conduct Science Communication About Omicron on ...
    Go to citation Crossref Google Scholar
  5. Mapping the knowledge frontiers of public risk communication in disast...
    Go to citation Crossref Google Scholar
  6. Scientists need professional development to practice meaningful public...
    Go to citation Crossref Google Scholar
  7. Exploring the challenges and opportunities of public engagement with f...
    Go to citation Crossref Google Scholar
  8. Media Framing of Climate Change Mitigation and Adaptation
    Go to citation Crossref Google Scholar
  9. Social Acceptance of Molecular Robots
    Go to citation Crossref Google Scholar
  10. Safety and global regulations for application of nanomaterials
    Go to citation Crossref Google Scholar
  11. US-China trade negotiation discourses in the press
    Go to citation Crossref Google Scholar
  12. Public engagement with science—Origins, motives and impact in academic...
    Go to citation Crossref Google Scholar
  13. Apropiación social de la ciencia y la tecnología en Medellín: contribu...
    Go to citation Crossref Google Scholar
  14. Media Framing of Climate Change Mitigation and Adaptation
    Go to citation Crossref Google Scholar
  15. Public Engagement in Transport Planning
    Go to citation Crossref Google Scholar
  16. Fracking bad language – hydraulic fracturing and earthquake risks
    Go to citation Crossref Google Scholar
  17. United Kingdom: The developing relationship between science and societ...
    Go to citation Crossref Google Scholar
  18. Visualizing Energy Participation: A Method for Practitioners and Resea...
    Go to citation Crossref Google Scholar
  19. Ethics of Biomarkers
    Go to citation Crossref Google Scholar
  20. The Practice of Public Engagement on Projects: From Managing External ...
    Go to citation Crossref Google Scholar
  21. Anatomy Education to the Public
    Go to citation Crossref Google Scholar
  22. New impulses from international development for more comprehensive and...
    Go to citation Crossref Google Scholar
  23. Improving environmental sanitation in the catchment area of Benya Lago...
    Go to citation Crossref Google Scholar
  24. Genetic Engineering and Society
    Go to citation Crossref Google Scholar
  25. Scientists vs laypeople: How genetically modified food is discussed on...
    Go to citation Crossref Google ScholarPub Med
  26. Appreciative inquiry for community engagement in Indonesia rural commu...
    Go to citation Crossref Google Scholar
  27. ‘Engagement’ of patients and healthcare professionals in regulatory ph...
    Go to citation Crossref Google Scholar
  28. The Impact of Perceptual and Situational Factors on Environmental Comm...
    Go to citation Crossref Google Scholar
  29. Not only speed matters – Crisis response in the hypothetical case of a...
    Go to citation Crossref Google Scholar
  30. Design Theory for Generating Alternatives in Public Decision Making Pr...
    Go to citation Crossref Google Scholar
  31. Evaluating Ocean Learning—The Principles and Practicalities of Evaluat...
    Go to citation Crossref Google Scholar
  32. A literature review on the relationship between risk governance and pu...
    Go to citation Crossref Google Scholar
  33. Shale development in the US and Canada: A review of engagement practic...
    Go to citation Crossref Google Scholar
  34. The GM-regulation game – the case of Hungary
    Go to citation Crossref Google Scholar
  35. A case study demonstrates a democratic methodology for making risk-bas...
    Go to citation Crossref Google Scholar
  36. Science to the rescue or contingent progress? Comparing 10 years of pu...
    Go to citation Crossref Google ScholarPub Med
  37. A theory of participation: what makes stakeholder and public engagemen...
    Go to citation Crossref Google Scholar
  38. Dilemmas of public participation in science policy
    Go to citation Crossref Google Scholar
  39. A Comparative Analysis of Attitudes on Communication Toward Stem Cell ...
    Go to citation Crossref Google Scholar
  40. Building Organizational Political Capacity Through Policy Learning: Co...
    Go to citation Crossref Google Scholar
  41. Outcomes of Group Model Building
    Go to citation Crossref Google Scholar
  42. Science communication training: what are we trying to teach?
    Go to citation Crossref Google Scholar
  43. Mass Media Roles in Climate Change Mitigation
    Go to citation Crossref Google Scholar
  44. Getting Out of Their Way: Do-It-Yourselfers, Sensing, and Self-Relianc...
    Go to citation Crossref Google Scholar
  45. Japan’s Nuclear Imaginaries Before and After Fukushima: Visions of Sci...
    Go to citation Crossref Google Scholar
  46. Hybrid Focus Groups as a Means to Investigate Practical Reasoning, Lea...
    Go to citation Crossref Google Scholar
  47. Health and medical research funding agencies’ promotion of public enga...
    Go to citation Crossref Google Scholar
  48. Opening up animal research and science–society relations? A thematic a...
    Go to citation Crossref Google ScholarPub Med
  49. The situational public engagement model in a municipal watershed prote...
    Go to citation Crossref Google Scholar
  50. Recent evidence on the effectiveness of group model building
    Go to citation Crossref Google Scholar
  51. Inviting Everyone to the Table: Strategies for More Effective and Legi...
    Go to citation Crossref Google Scholar
  52. Assessing the quality of a deliberative democracy mini-public event ab...
    Go to citation Crossref Google ScholarPub Med
  53. Risk communication and social media during food safety crises: a study...
    Go to citation Crossref Google Scholar
  54. Assessing Deliberative Design of Public Input on British Columbia Biob...
    Go to citation Crossref Google Scholar
  55. A Cross Disciplinary Embodiment
    Go to citation Crossref Google Scholar
  56. Organizational political capacity as learning
    Go to citation Crossref Google Scholar
  57. Towards a cross-paradigmatic framework of the social acceptance of ene...
    Go to citation Crossref Google Scholar
  58. Mass Media Roles in Climate Change Mitigation
    Go to citation Crossref Google Scholar
  59. Social Disputes over GMOs: An Overview
    Go to citation Crossref Google Scholar
  60. Upstream public engagement, downstream policy-making? The Brain Imagin...
    Go to citation Crossref Google Scholar
  61. Deliberative research as a tool to make value judgements
    Go to citation Crossref Google Scholar
  62. Expert involvement in policy development: A systematic review of curre...
    Go to citation Crossref Google Scholar
  63. From ‘trust us’ to participatory governance: Deliberative publics and ...
    Go to citation Crossref Google ScholarPub Med
  64. Reflective Ethical Mapping
    Go to citation Crossref Google Scholar
  65. Assessing participation in a community-based health planning and servi...
    Go to citation Crossref Google Scholar
  66. Valuing public engagement with energy system transitions: the importan...
    Go to citation Crossref Google Scholar
  67. Messing with nature? Exploring public perceptions of geoengineering in...
    Go to citation Crossref Google Scholar
  68. Bringing personalized medicine to the community through public engagem...
    Go to citation Crossref Google Scholar
  69. Towards a new framework for evaluating systemic problem structuring me...
    Go to citation Crossref Google Scholar
  70. Genetically modified animals from life-science, socio-economic and eth...
    Go to citation Crossref Google Scholar
  71. Evaluation of a Mixed Participatory Method to Improve Mutual Understan...
    Go to citation Crossref Google Scholar
  72. A public private people partnerships (P4) process framework for infras...
    Go to citation Crossref Google Scholar
  73. A Scientific Approach to Monitoring Public Perceptions of Scientific I...
    Go to citation Crossref Google Scholar
  74. Dialogue and science: Innovation in policy-making and the discourse of...
    Go to citation Crossref Google Scholar
  75. The tragedy of citizen deliberation – two cases of participatory techn...
    Go to citation Crossref Google Scholar
  76. Representing the Public in Public Engagement: The Case of the 2008 UK ...
    Go to citation Crossref Google Scholar
  77. Responsible healthcare innovation: anticipatory governance of nanodiag...
    Go to citation Crossref Google Scholar
  78. Democracy and risk-based decision-making: the next step in public invo...
    Go to citation Crossref Google Scholar
  79. Stakeholder engagement in comparative effectiveness research: how will...
    Go to citation Crossref Google Scholar
  80. Governing the moral economy: Animal engineering, ethics and the libera...
    Go to citation Crossref Google Scholar
  81. Social Learning Through Environmental Risk Analysis of Biodiversity an...
    Go to citation Crossref Google Scholar
  82. What Is Good Public Deliberation?
    Go to citation Crossref Google Scholar
  83. Calling controversy: assessing synthetic biology’s conflict potential
    Go to citation Crossref Google ScholarPub Med
  84. References
    Go to citation Crossref Google Scholar
  85. Mass Media Roles in Climate Change Mitigation
    Go to citation Crossref Google Scholar
  86. Community Engagement in Engineering Education
    Go to citation Crossref Google Scholar
  87. Public engagement coming of age: From theory to practice in STS encoun...
    Go to citation Crossref Google Scholar
  88. Deficits and dialogues: science communication and the public understan...
    Go to citation Crossref Google Scholar
  89. Technology assessment in Australia: the case for a formal agency to im...
    Go to citation Crossref Google Scholar
  90. Stakeholder engagement in food risk management...
    Go to citation Crossref Google Scholar
  91. Decisions, dilemmas and deliberation: exploring the legitimacy of the ...
    Go to citation Crossref Google Scholar
  92. Public participation: comparing approaches
    Go to citation Crossref Google Scholar
  93. Which indicators for the new public engagement activities? An explorat...
    Go to citation Crossref Google Scholar
  94. Mediating Science and Society in the EU and UK: From Information-Trans...
    Go to citation Crossref Google Scholar
  95. Public perception of risk
    Go to citation Crossref Google Scholar
  96. Perceived efficacy and attitudes towards genetic science and science g...
    Go to citation Crossref Google ScholarPub Med
  97. Early Exposures to Ecogenomics: Effects of Priming and Web Site Intera...
    Go to citation Crossref Google Scholar
  98. Science communication in Sub‐Saharan AFrica: The case of GMOs
    Go to citation Crossref Google Scholar
  99. Ethics of Biomarkers: the Borders of Investigative Research, Informed ...
    Go to citation Crossref Google Scholar
  100. Recruiting for representation in public deliberation on the ethics of ...
    Go to citation Crossref Google ScholarPub Med
  101. Food, publics, science
    Go to citation Crossref Google ScholarPub Med
  102. Prospects for the open treatment of uncertainty in environmental resea...
    Go to citation Crossref Google Scholar
  103. Genetically modified food in the news: media representations of the GM...
    Go to citation Crossref Google ScholarPub Med
  104. e-Democracy: A Group Decision and Negotiation-Oriented Overview
    Go to citation Crossref Google Scholar
  105. The public consultation to the UK Climate Change Act 2008: a critical ...
    Go to citation Crossref Google Scholar
  106. Public engagement with science and technology (PEST): good principle, ...
    Go to citation Crossref Google Scholar
  107. Using Latent Class Models to Explore Cross-national Typologies of Publ...
    Go to citation Crossref Google Scholar
  108. Discussing dialogue: perspectives on the value of science dialogue eve...
    Go to citation Crossref Google Scholar
  109. A Critical Assessment of Public Consultations on GMOs in the European ...
    Go to citation Crossref Google Scholar
  110. Consultations of stakeholders on the roles of research in relation to ...
    Go to citation Crossref Google ScholarPub Med
  111. You & Your Body...
    Go to citation Crossref Google Scholar
  112. Interpersonal Discussion Following Citizen Engagement About Nanotechno...
    Go to citation Crossref Google Scholar
  113. Analysis of a normative framework for evaluating public engagement exe...
    Go to citation Crossref Google Scholar
  114. The Evolution of ‘Public Understanding of Science’: Public Engagement ...
    Go to citation Crossref Google Scholar
  115. Canadians' Representation of Chemical, Biological, Radiological, Nucle...
    Go to citation Crossref Google Scholar
  116. Deliberating Competence...
    Go to citation Crossref Google Scholar
  117. Designing a Participatory Process for Stakeholder Involvement in a Soc...
    Go to citation Crossref Google Scholar
  118. Developments in nanotechnology public engagement in the UK: ‘upstream’...
    Go to citation Crossref Google Scholar
  119. Deliberating Competence...
    Go to citation Crossref Google Scholar
  120. The quality of food risk management in Europe: Perspectives and priori...
    Go to citation Crossref Google Scholar
  121. Interactive Technology Assessment in the Real World...
    Go to citation Crossref Google Scholar
  122. e -Participation and Decision Analysis
    Go to citation Crossref Google Scholar
  123. The Value of “Dialogue Events” as Sites of Learning: An exploration of...
    Go to citation Crossref Google Scholar
  124. Belief in Public Efficacy, Trust, and Attitudes Toward Modern Genetic ...
    Go to citation Crossref Google Scholar
  125. Citizen engagement processes as information systems: the role of knowl...
    Go to citation Crossref Google Scholar
  126. Societal concerns and risk decisions
    Go to citation Crossref Google Scholar
  127. Stimulating authentic community involvement in biotechnology policy in...
    Go to citation Crossref Google Scholar
  128. Towards Analytic‐deliberative Forms of Risk Governance in the UK? Refl...
    Go to citation Crossref Google Scholar
  129. What can we learn from 25 years of PUS survey research? Liberating and...
    Go to citation Crossref Google Scholar
  130. Risk and the Public Acceptance of New Technologies
    Go to citation Crossref Google Scholar
  131. Public engagement in food policy
    Go to citation Crossref Google Scholar
  132. On evaluating the GM Nation? Public debate...
    Go to citation Crossref Google Scholar
  133. Inside Deliberative Experiments: Dynamics of Subjectivity in Science P...
    Go to citation Crossref Google Scholar
  134. Participatory Decision-Making for Sustainable Consumption
    Go to citation Crossref Google Scholar

Figures and tables

Figures & Media

Tables

View Options

Get access

Access options

If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:


Alternatively, view purchase options below:

Purchase 24 hour online access to view and download content.

Access journal content via a DeepDyve subscription or find out more about this option.

View options

PDF/ePub

View PDF/ePub