Evaluating collaboration: The creation of an online tool employing Q methodology
Abstract
The continued prevalence of different forms of collaborative working within public policy requires adaption in evaluation practices. In recent years evaluation toolkits, audits and guides have migrated online, but with varying success. At their worst, such tools can offer a disengaging user experience, limited coverage of issues or normative bias. This article outlines POETQ, designed to be engaging, comprehensive and methodologically robust. An overview of this approach is set out alongside an analysis of its merits. The article concludes by reflecting on the kinds of evidence that policy makers actually want to generate in relation to the topic of collaboration.
Get full access to this article
View all access and purchase options for this article.
References
Abbott D, Townsley R, Watson D (2005) Multi-agency working in services for disabled children: What impact does it have on professionals? Health and Social Care in the Community 13: 155–163.
Abendstern M, Reilly S, Hughes J, et al. (2006) Levels of integration and specialisation within professional community teams for people with dementia. International Journal of Geriatric Psychiatry 21: 77–85.
Appleton-Dyer S, Clinton J, Carswell P, et al. (2012) Understanding evaluation influence within public sector partnerships: A conceptual model. American Journal of Evaluation 33: 532–546.
Ball R, Forbes T, Parris M, et al. (2010) The evaluation of partnership working in the delivery of health and social care. Public Policy and Administration 25: 387–407.
Barnes M, Bauld L, Benezeval M, et al. (2005) Health Action Zones: Partnerships for Health Equity. London: Routledge.
Brown L, Tucker C, Domokos T (2003) Evaluating the impact of integrated health and social care teams on older people living in the community. Health and Social Care in the Community 11: 85–94.
Brown LD, Feinberg ME, Greenberg MT (2012) Measuring coalition functioning: Refining constructs through factor analysis. Health Education and Behavior 39: 486–497.
Brown S (1980a) Political Subjectivity. New Haven, CT: Yale University Press.
Brown S (1980b) Political Subjectivity: Applications of Q Methodology in Political Science. New Haven, CT: Yale University Press.
Carnwell R, Buchanan J (2005) Learning from partnerships: Themes and issues. In: Carnwell R, Buchanan J (eds) Effective Practice in Health and Social Care: A Partnership Approach. Maidenhead: Open University Press, pp. 281–287.
Dickinson H (2006) The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future. Health and Social Care in the Community 14(5): 375–383
Dickinson H (2008) Evaluating Outcomes in Health and Social Care. Bristol: Policy Press.
Dickinson H (2010) The importance of being efficacious: English health and social care partnerships and service user outcomes. International Journal of Integrated Care 10: 346.
Dickinson H (2014) Performing Governance: Partnerships, Culture and New Labour. Basingstoke: Palgrave Macmillan
Dickinson H, Glasby J, Nicholds A, et al. (2013a) Joint commissioning in health and social care: An exploration of definitons, processes, services and outcomes. NIHR Service Delivery and Organisation programme, September. Retrieved from: http://www.netscc.ac.uk/hsdr/projdetails.php?ref=08-1806-260
Dickinson H, Jeffares S, Nicholds A, et al. (2013b) Beyond the Berlin Wall? Public Management Review. Epub ahead of print.
Dowling B, Powell M, Glendinning C (2004) Conceptualising successful partnerships. Health and Social Care in the Community 12: 309–317.
Eden S, Donaldson A, Walker G (2005) Structuring subjectivities? Using Q methodology in human geography. Area 37: 413–422.
Edwards A, Barnes M, Plewis I (2006) Working to prevent the social exclusion of children and young people: Final lessons from the National Evaluation of the Children’s Fund. London: Department of Education. Retrieved from: http://webarchive.nationalarchives.gov.uk/20130401151715/http://www.education.gov.uk/publications/standard/publicationdetail/page1/RR734.
El Ansari W, Weiss ES (2006) Quality of research on community partnerships: Developing the evidence base. Health Education Research 21: 175–180.
Fisher RA (1971) The Design of Experiments. New York: Hafner Publishing Company.
Gajda R (2004) Utilizing collaboration theory to evaluate strategic alliances. American Journal of Evaluation 25: 65–77.
Galesic M, Bosnjak M (2009) Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly 73: 349–360.
Greig R, Poxton R (2001) From joint commissioning to partnership working – will the new policy framework make a difference? Managing Community Care 9: 32–38.
Hardy B, Hudson B, Waddington E (2003) Assessing Strategic Partnership: The Partnership Assessment Tool. London: Office of the Deputy Prime Minister.
Health Development Agency (2003) The Working Partnership. London Retrieved from: http://www.webarchive.org.uk/wayback/archive/20140616160456/http://nice.org.uk/aboutnice/whoweare/aboutthehda/hdapublications/hda_publications.jsp?o=487.
Hoggett P (2006) Conflict, ambivalence, and the contested purpose of public organizations. Human Relations 59: 175–194.
Hughes J, Weiss J (2007) Simple rules for making alliances work. Harvard Business Review 14: 1–10.
Huxham C, Vangen S (2005) Managing to Collaborate: The Theory and Practice of Collaborative Advantage. Abingdon: Routledge.
Jackson M (1989) Assumptional analysis. Systems Practice 14: 11–28.
Jeffares S, Sullivan H, Bovaird T (2012) Beyond the Contract: The Challenge of Evaluating the Performance(s) of Public-Private Partnerships. In: Greve C, Hodge G (eds) Rethinking Public-Private Partnerships: Strategies for Turbulent Times. London: Routledge, pp. 166–187.
Jeffares S, Bovaird T (2010) Partnership beyond the contract: unpublished final report of the evaluation of the Service Birmingham Partnership. Birmingham: University of Birmingham.
Jeffares S, Skelcher C (2011) Democratic subjectivities in network governance: a Q methodology study of English and Dutch public managers. Public Administration 89: 1253–1273.
Kelman S (2007) The transformation of government in the decade ahead. In: Kettl DF, Kelman S (eds) Reflections on 21st Century Government Management. Washington, DC: IBM Center for the Business of Government, pp. 33–63.
Leutz W (1999) Five laws for integrating Medical and Social Services: Lessons from the United Stated and the United Kingdom. The Milbank Quarterly 77: 77–110.
Marcus B, Bosnjak M, Lindner S, et al. (2007) Compensating for low topic interest and long surveys: A field experiment in nonresponse in web surveys. Social Science Computer Review 25: 372–383.
Marek LI, Brock DP, Savla J (2014) Evaluating collaboration for effectiveness: Conceptualization and measurement. American Journal of Evaluation 1–19.
Markwell S (2003) Partnership Working: A Consumer Guide to Resources. London: Health Development Agency. Retrieved from: http://www.webarchive.org.uk/wayback/archive/20140616160513/http://nice.org.uk/aboutnice/whoweare/aboutthehda/hdapublications/hda_publications.jsp?o=503
Melhuish E, Belsky J, Leyland A, et al. (2008) Effects of fully-established Sure Start Local programmes on children and their families living in England: A quasi-experimental observational study. Lancet 372: 1641–1647.
Newman J (2001) Modernising Governance: New Labour, Policy and Society. London: SAGE.
Office of the Deputy Prime Minister (2005) A process evaluation of the negotiation of pilot Local Area Agreements. Available at: http://webarchive.nationalarchives.gov.uk/20130401151715/http://www.education.gov.uk/publications/standard/publicationdetail/page1/05%20lgfg%2003175.
Office of the Deputy Prime Minister and Department of Transport (2006) National evaluation of Local Strategic Partnerships: Formative evaluation and action research programme, 2002–2005. London. Retrieved from: https://www.gov.uk/government/publications.
O’Flynn J (2014) Crossing boundaries: The fundamental questions in public management and policy. In: O’Flynn J, Blackman D, Halligan J (eds) Crossing Boundaries in Public Management and Policy: The International Experience. London: Routledge, pp. 11–44.
Provan K, Milward H (2001) Do networks really work? A framework for evaluating public-sector organizational networks. Public Administration Review 61: 414–423.
Provan KG, Sydow J (2008) Evaluating inter-organizational relationships. In: Cropper S, Ebers M, Huxham C, et al. (eds) The Oxford Handbook of Inter-organizational Relations. Oxford: Oxford University Press, pp. 691–718.
Reber B, Kaufman S, Cropp F (2000) Assessing Q-assessor: A validation study of Q-assessor computer-based Q-sorts versus paper sorts. Operant Subjectivity 23: 192–209.
Schmolck P, Atkinson J (2012) PQMethod version 2.33. [Computer Software]. Retrieved from http://schmolck.userweb.mwn.de/qmethod/pqmanual.htm.
Sullivan H (2003) New forms of local accountability: Coming to terms with ‘many hands’. Policy and Politics 31: 353–370.
Sullivan H (2011) ‘Truth’ junkies: Using evaluation in UK public policy. Policy and Politics 39: 499–512.
Sullivan H, Skelcher C (2002) Working across Boundaries: Collaboration in Public Services. Basingstoke: Palgrave Macmillan.
Sullivan H, Barnes M, Matka E (2006) Collaborative capacity and strategies in area-based initiatives. Public Administration 84: 289–310.
Sullivan H, Williams P, Jeffares S (2012) Leadership for collaboration. Public Management Review 14: 41–66.
Van Excel J, Baker R, Mason H, et al. (2015) Public views on principles for health care priority setting: Findings of a European cross-country study using Q methodology. Social Science and Medicine 126: 128–137.
Watts S, Stenner P (2005) Doing Q-methodology: Theory, method and interpretation. Qualitative Research in Psychology 2: 67–91.
Williams P, Sullivan H (2009) Faces of integration. International Journal of Integrated Care 9: 1–13.
Woodland RH, Hutton MS (2012) Evaluating organizational collaborations: Suggested entry points and strategies. American Journal of Evaluation 33: 366–383.
Biographies
Stephen Jeffares is Lecturer in Public Policy Analysis at the School of Government and Society, University of Birmingham, UK.
Helen Dickinson is Associate Professor Public Governance at the Melbourne School of Government, University of Melbourne, Australia.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: January 7, 2016
Issue published: January 2016
Keywords
Authors
Metrics and citations
Metrics
Article usage*
Total views and downloads: 971
*Article usage tracking started in December 2016
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 12 view articles Opens in new tab
Crossref: 0
-
Stakeholders’ Perspectives on the Application of New Diagnostic Device...
-
Short Take: Sorting at a Distance: Q Methodology Online
-
Q Methodology in the COVID-19 Era
-
Introducing Q Methodology to Program Evaluators
-
AI, Public Service and Research Methodology
-
Governance of climate adaptation, which mode? An exploration of stakeh...
-
Holistic Governance for Sustainable Public Services: Reshaping Governm...
-
How do professionals perceive the governance of public–private partner...
-
Student and teacher attitudes towards overtraining and recovery in voc...
-
Control or influence? Conflict or solidarity? Understanding diversity ...
-
Why decentralize decision making? English local actors' viewpoints
-
Why do pregnant women participate in research? A patient participation...
Figures and tables
Figures & Media
Tables
View Options
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
loading institutional access options
AES and EES members can access this journal content using society membership credentials.
AES and EES members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
