Fernando Delgado
ABSTRACT
Despite the growing consensus that stakeholders affected by AI systems should participate in their design, enormous variation and implicit disagreements exist among current approaches. For researchers and practitioners who are interested in taking a participatory approach to AI design and development, it remains challenging to assess the extent to which any participatory approach grants substantive agency to stakeholders. This article thus aims to ground what we dub the “participatory turn” in AI design by synthesizing existing theoretical literature on participation and through empirical investigation and critique of its current practices. Specifically, we derive a conceptual framework through synthesis of literature across technology design, political theory, and the social sciences that researchers and practitioners can leverage to evaluate approaches to participation in AI design. Additionally, we articulate empirical findings concerning the current state of participatory practice in AI design based on an analysis of recently published research and semi-structured interviews with 12 AI researchers and practitioners. We use these empirical findings to understand the current state of participatory practice and subsequently provide guidance to better align participatory goals and methods in a way that accounts for practical constraints.
- Rediet Abebe and Kira Goldner. 2018. Mechanism design for social good. AI Matters 4, 3 (2018), 27–34.Google Scholar
Digital Library
- Julia Abelson, Pierre-Gerlier Forest, John Eyles, Patricia Smith, Elisabeth Martin, and Francois-Pierre Gauvin. 2003. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Social science & medicine 57, 2 (2003), 239–251.Google Scholar
- Open Government Partnership Ada Lovelace Institute, AI Now Institute. 2021. Algorithmic accountability for the public sector report. (2021). https://www.opengovpartnership.org/documents/ algorithmic-accountability-public-sector/Google Scholar
- Kars Alfrink, Thijs Turel, AI Keller, Neelke Doorn, and Gerd Kortuem. 2020. Contestable City Algorithms. (2020).Google Scholar
- Ali Alkhatib and Michael Bernstein. 2019. Street-level algorithms: A theory at the gaps between policy and decisions. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Oscar Alvarado, Elias Storms, David Geerts, and Katrien Verbert. 2020. Foregrounding Algorithms: Preparing Users for Co-design with Sensitizing Activities. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–7.Google ScholarDigital Library
- Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.Google ScholarDigital Library
- Asbjørn Ammitzbøll Flügge, Thomas Hildebrandt, and Naja Holten Møller. 2019. Street-Level Algorithms and AI in Bureaucratic Decision-Making: A Caseworker Perspective. 5 (2019), 40:1–40:23. Issue CSCW1. https://doi.org/10.1145/3449114Google ScholarDigital Library
- Mike Ananny and Kate Crawford. 2018. Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. new media & society 20, 3 (2018), 973–989.Google Scholar
- Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31, 3 (2023), 337–351. https://doi.org/10.1017/pan.2023.2Google ScholarCross Ref
- Hilary Arksey and Lisa O’Malley. 2005. Scoping studies: towards a methodological framework. International journal of social research methodology 8, 1 (2005), 19–32.Google ScholarCross Ref
- Sherry R Arnstein. 1969. A ladder of citizen participation. Journal of the American Institute of planners 35, 4 (1969), 216–224.Google ScholarCross Ref
- Kenneth J Arrow. 2012. Social choice and individual values. Yale university press.Google Scholar
- Shiri Azenkot, Catherine Feng, and Maya Cakmak. 2016. Enabling building service robots to guide blind people a participatory design approach. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 3–10.Google ScholarCross Ref
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022).Google Scholar
- Ellen Balka. 2010. Broadening discussion about participatory design. Scandinavian Journal of Information Systems 22, 1 (2010), 77–84.Google Scholar
- Julia Barnett and Nicholas Diakopoulos. 2022. Crowdsourcing impacts: exploring the utility of crowds for anticipating societal impacts of algorithmic decision making. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. 56–67.Google ScholarDigital Library
- Alex Bäuerle, Ángel Alexander Cabrera, Fred Hohman, Megan Maher, David Koski, Xavier Suau, Titus Barik, and Dominik Moritz. 2022. Symphony: Composing Interactive Interfaces for Machine Learning. In CHI Conference on Human Factors in Computing Systems. 1–14.Google Scholar
- Eric PS Baumer. 2017. Toward human-centered algorithm design. Big Data & Society 4, 2 (2017), 2053951717718854.Google ScholarCross Ref
- Eevi Beck. 2002. P for Political - Participation is Not Enough.Scandinavian J. Inf. Systems 14 (01 2002).Google Scholar
- Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623.Google ScholarDigital Library
- Jesse Josua Benjamin, Christoph Kinkeldey, Claudia Müller-Birn, Tim Korjakow, and Eva-Maria Herbst. [n. d.]. Explanation Strategies as an Empirical-Analytical Lens for Socio-Technical Contextualization of Machine Learning Interpretability. 6 ([n. d.]), 1–25. Issue GROUP. https://doi.org/10.1145/3492858Google ScholarDigital Library
- Cynthia L Bennett and Daniela K Rosner. 2019. The Promise of Empathy: Design, Disability, and Knowing the" Other". In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.Google ScholarDigital Library
- Aleks Berditchevskaia, Eirini Malliaraki, and Kathy Peach. 2021. Participatory AI for humanitarian innovation. (2021). https://media.nesta.org.uk/documents/Nesta_Participatory_AI_for_humanitarian_innovation_Final.pdfGoogle Scholar
- Rahul Bhargava, Anna Chung, Neil S Gaikwad, Alexis Hope, Dennis Jen, Jasmin Rubinovitz, Belén Saldías-Fuentes, and Ethan Zuckerman. 2019. Gobo: A system for exploring user control of invisible algorithms in social media. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing. 151–155.Google ScholarDigital Library
- Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Diaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022. Power to the People? Opportunities and Challenges for Participatory AI. In Equity and Access in Algorithms, Mechanisms, and Optimization (Arlington, VA, USA) (EAAMO ’22). Association for Computing Machinery, New York, NY, USA, Article 6, 8 pages. https://doi.org/10.1145/3551624.3555290Google ScholarDigital Library
- Gro Bjerknes and Tone Bratteteig. 1995. User participation and democracy: A discussion of Scandinavian research on system development. Scandinavian Journal of information systems 7, 1 (1995), 1.Google ScholarDigital Library
- Emily Black, Joshua Williams, Michael A Madaio, and Priya L Donti. 2020. A call for universities to develop requirements for community engagement in AI research. In The Fair and Responsible AI Workshop at the 2020 CHI Conference on Human Factors in Computing Systems.Google Scholar
- Design Jeanette Blomberg and Helena Karasti. 2012. Ethnography: Positioning ethnography within participatory design. In Routledge international handbook of participatory design. Routledge, 106–136.Google Scholar
- Susanne BØdker, Christian Dindler, and Ole Sejer Iversen. 2017. Tying Knots: Participatory Infrastructuring at Work. Comput. Supported Coop. Work 26, 1–2 (apr 2017), 245–273. https://doi.org/10.1007/s10606-017-9268-yGoogle ScholarDigital Library
- Susanne Bødker and Morten Kyng. 2018. Participatory Design That Matters—Facing the Big Issues. ACM Trans. Comput.-Hum. Interact. 25, 1, Article 4 (feb 2018), 31 pages. https://doi.org/10.1145/3152421Google ScholarDigital Library
- Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, 2021. On the Opportunities and Risks of Foundation Models. arXiv preprint arXiv:2108.07258 (2021).Google Scholar
- Elizabeth Bondi, Lily Xu, Diana Acosta-Navas, and Jackson A. Killian. 2021. Envisioning Communities: A Participatory Approach Towards AI for Social Good. Association for Computing Machinery, New York, NY, USA, 425–436. https://doi.org/10.1145/3461702.3462612Google ScholarDigital Library
- Alan Borning, Batya Friedman, and P Kahn. 2004. Designing for human values in an urban simulation system: Value sensitive design and participatory design. In Proceedings From the Eighth Biennial Participatory Design Conference. Citeseer.Google Scholar
- Nina Bozic Yams and Álvaro Aranda Muñoz. 2021. Poetics of Future Work: Blending Speculative Design with Artistic Methodology. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–8.Google Scholar
- Tone Bratteteig and Guri Verne. 2018. Does AI Make PD Obsolete? Exploring Challenges from Artificial Intelligence to Participatory Design. In Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial - Volume 2 (Hasselt and Genk, Belgium) (PDC ’18). Association for Computing Machinery, New York, NY, USA, Article 8, 5 pages. https://doi.org/10.1145/3210604.3210646Google ScholarDigital Library
- Tone Bratteteig and Ina Wagner. 2012. Disentangling power and decision-making in participatory design. In Proceedings of the 12th Participatory Design Conference: Research Papers-Volume 1. 41–50.Google ScholarDigital Library
- Tone Bratteteig and Ina Wagner. 2016. Unpacking the Notion of Participation in Participatory Design. Comput. Supported Coop. Work 25, 6 (dec 2016), 425–475. https://doi.org/10.1007/s10606-016-9259-4Google ScholarDigital Library
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.Google Scholar
- Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597.Google ScholarCross Ref
- Anna Brown, Alexandra Chouldechova, Emily Putnam-Hornstein, Andrew Tobin, and Rhema Vaithianathan. 2019. Toward Algorithmic Accountability in Public Services: A Qualitative Study of Affected Community Perspectives on Algorithmic Decision-Making in Child Welfare Services. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi-org.proxy.library.cornell.edu/10.1145/3290605.3300271Google ScholarDigital Library
- Jenna Burrell. 2016. How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big data & society 3, 1 (2016), 2053951715622512.Google Scholar
- Dan Calacci and Alex Pentland. [n. d.]. Bargaining with the Black-Box: Designing and Deploying Worker-Centric Tools to Audit Algorithmic Management. 6 ([n. d.]), 1–24. Issue CSCW2. https://doi.org/10.1145/3570601Google ScholarDigital Library
- Heloisa Candello, Mauro Pichiliani, and Claudio Pinhanez. 2021. Making Conversational Robots Learn from Children in a Collaborative Museum Exhibition. Association for Computing Machinery, New York, NY, USA, 208–212. https://doi-org.proxy.library.cornell.edu/10.1145/3462204.3482888Google Scholar
- Heloisa Candello, Mauro Pichiliani, Claudio Pinhanez, Sara Vidon, and Mairieli Wessel. 2020. Co-designing a conversational interactive exhibit for children. In Proceedings of the 2020 ACM Interaction Design and Children Conference: Extended Abstracts. 326–331.Google ScholarDigital Library
- Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, 2023. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2307.15217 (2023).Google Scholar
- Eshwar Chandrasekharan, Chaitrali Gandhi, Matthew Wortley Mustelier, and Eric Gilbert. 2019. Crossmod: A cross-community learning-based system to assist reddit moderators. Proceedings of the ACM on human-computer interaction 3, CSCW (2019), 1–30.Google ScholarDigital Library
- Kyla Chasalow and Karen Levy. 2021. Representativeness in Statistics, Politics, and Machine Learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 77–89.Google ScholarDigital Library
- Zhilong Chen, Jinghua Piao, Xiaochong Lan, Hancheng Cao, Chen Gao, Zhicong Lu, and Yong Li. [n. d.]. Practitioners Versus Users: A Value-Sensitive Evaluation of Current Industrial Recommender System Design. 6 ([n. d.]), 533:1–533:32. Issue CSCW2. https://doi.org/10.1145/3555646Google ScholarDigital Library
- Hao-Fei Cheng, Logan Stapleton, Ruiqi Wang, Paige Bullock, Alexandra Chouldechova, Zhiwei Steven Steven Wu, and Haiyi Zhu. 2021. Soliciting Stakeholders’ Fairness Notions in Child Maltreatment Predictive Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–17.Google ScholarDigital Library
- Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019. Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–12.Google ScholarDigital Library
- Parmit K Chilana, Andrew J Ko, and Jacob Wobbrock. 2015. From user-centered to adoption-centered design: a case study of an HCI research innovation becoming a product. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1749–1758.Google ScholarDigital Library
- Chhaya Chouhan, Christy M LaPerriere, Zaina Aljallad, Jess Kropczynski, Heather Lipford, and Pamela J Wisniewski. 2019. Co-designing for community oversight: Helping people make privacy and security decisions together. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–31.Google ScholarDigital Library
- Bill Cooke and Uma Kothari. 2001. Participation: The new tyranny?Zed books.Google Scholar
- A Feder Cooper, Emanuel Moss, Benjamin Laufer, and Helen Nissenbaum. 2022. Accountability in an algorithmic society: relationality, responsibility, and robustness in machine learning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 864–876.Google ScholarDigital Library
- Sasha Costanza-Chock. 2020. Design justice: Community-led practices to build the worlds we need. The MIT Press.Google Scholar
- Design Council. 2005. The ‘double diamond’ design process model. Design Council (2005).Google Scholar
- Andy Crabtree. 1998. Ethnography in participatory design. In Proceedings of the 1998 Participatory design Conference. Seattle, WA, 93–105.Google Scholar
- Kenneth Cukier and Viktor Mayer-Schoenberger. 2014. The rise of big data: How it’s changing the way we think about the world. Princeton University Press.Google Scholar
- Andy Dearden, Ann Light, Theodore Zamenopoulos, Paula Graham, Emma Plouviez, and Sophia de Sousa. 2014. Scaling up co-design: research projects as design things. In Proceedings of the 13th Participatory Design Conference: Short Papers, Industry Cases, Workshop Descriptions, Doctoral Consortium papers, and Keynote abstracts-Volume 2. 67–70.Google ScholarDigital Library
- Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, and Morgan Klaus Scheuerman. 2020. Bringing the People Back In: Contesting Benchmark Machine Learning Datasets. (2020). arxiv:2007.07399 [cs.CY]Google Scholar
- John Dewey. 1954. {1927} The Public and Its Problems. Athens, OH: Ohio University Press.Google Scholar
- Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758 (2021).Google Scholar
- Pedro Domingos. 2012. A few useful things to know about machine learning. Commun. ACM 55, 10 (2012), 78–87.Google ScholarDigital Library
- Paul Dourish, Christopher Lawrence, Tuck Wah Leong, and Greg Wadley. 2020. On Being Iterated: The Affective Demands of Design Participation. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376545Google ScholarDigital Library
- Carlos Duarte, Luis Carriço, David Costa, Daniel Costa, André Falcão, and Luís Tavares. 2014. Welcoming gesture recognition into autism therapy. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. 1267–1272.Google Scholar
- Upol Ehsan and Mark O Riedl. 2020. Human-centered explainable ai: Towards a reflective sociotechnical approach. In HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings 22. Springer, 449–466.Google ScholarDigital Library
- John Elliot. 1991. Action research for educational change. McGraw-Hill Education (UK).Google Scholar
- Jenny Fan and Amy X Zhang. 2020. Digital juries: A civics-oriented approach to platform governance. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–14.Google ScholarDigital Library
- Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods 5, 1 (2006), 80–92.Google ScholarCross Ref
- Jessie Finocchiaro, Roland Maio, Faidra Monachou, Gourab K Patro, Manish Raghavan, Ana-Andreea Stoica, and Stratis Tsirtsis. 2021. Bridging Machine Learning and Mechanism Design towards Algorithmic Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 489–503.Google ScholarDigital Library
- James S Fishkin and Robert C Luskin. 2005. Experimenting with a democratic ideal: Deliberative polling and public opinion. Acta politica 40, 3 (2005), 284–298.Google Scholar
- James S Fishkin and Jane Mansbridge. 2017. The prospects and limits of deliberative democracy: Introduction. Daedalus 146, 3 (2017), 6–13.Google Scholar
- Jodi Forlizzi. 2018. Moving beyond user-centered design. interactions 25, 5 (2018), 22–23.Google Scholar
- Jodi Forlizzi and John Zimmerman. 2013. Promoting service design as a core practice in interaction design. In Proceedings of the 5th International Congress of International Association of Societies of Design Research-IASDR, Vol. 13.Google Scholar
- Christopher Frauenberger, Judith Good, G. Fitzpatrick, and Ole Iversen. 2015. In pursuit of rigour and accountability in participatory design. International Journal of Human-Computer Studies 74 (2015), 93 – 106.Google ScholarDigital Library
- Christopher Frauenberger, Judith Good, Wendy Keay-Bright, and Helen Pain. 2012. Interpreting input from children: a designerly approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2377–2386. http://doi.org/10.1145/2207676.2208399Google ScholarDigital Library
- Batya Friedman. 1996. Value-sensitive design. interactions 3, 6 (1996), 16–23.Google Scholar
- Batya Friedman, Peter Kahn, and Alan Borning. 2002. Value sensitive design: Theory and methods. University of Washington technical report2-12 (2002).Google Scholar
- Archon Fung. 2006. Varieties of participation in complex governance. Public administration review 66 (2006), 66–75.Google Scholar
- Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022).Google Scholar
- Theodoros Georgiou, Lynne Baillie, Martin K Ross, and Frank Broz. 2020. Applying the Participatory Design Workshop Method to Explore how Socially Assistive Robots Could Assist Stroke Survivors. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. 203–205.Google Scholar
- Tarleton Gillespie. 2020. Content moderation, AI, and the question of scale. Big Data & Society 7, 2 (2020), 2053951720943234.Google ScholarCross Ref
- Janne Gleerup, Lars Hulgaard, and Simon Teasdale. 2019. Action research and participatory democracy in social enterprise. Social Enterprise Journal (2019).Google Scholar
- Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems. 1–19.Google ScholarDigital Library
- Judith Gregory. 2003. Scandinavian approaches to participatory design. International Journal of Engineering Education 19, 1 (2003), 62–74.Google Scholar
- Erik Grönvall and Stefan Lundberg. 2019. PYCIPEDIA: Supporting Local and Remote Collaboration between Social Workers.. In CSCW Companion. 195–200.Google ScholarDigital Library
- Lara Groves, Aidan Peppin, Andrew Strait, and Jenny Brennan. 2023. Going public: the role of public participation approaches in commercial AI labs. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1162–1173.Google ScholarDigital Library
- Hongyan Gu, Jingbin Huang, Lauren Hung, and Xiang ’Anthony’ Chen. [n. d.]. Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists. 5 ([n. d.]), 10:1–10:25. Issue CSCW1. https://doi.org/10.1145/3449084Google ScholarDigital Library
- Aaron Halfaker and R Stuart Geiger. 2020. Ores: Lowering barriers with participatory machine learning in wikipedia. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–37.Google ScholarDigital Library
- Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–19.Google ScholarDigital Library
- Alex Hanna and Tina M Park. 2020. Against Scale: Provocations and Resistances to Scale Thinking. arXiv preprint arXiv:2010.08850 (2020).Google Scholar
- Christina Harrington, Sheena Erete, and Anne Marie Piper. 2019. Deconstructing community-based collaborative design: Towards more equitable participatory design engagements. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–25.Google ScholarDigital Library
- Gillian R Hayes. 2014. Knowing by doing: action research as an approach to HCI. In Ways of Knowing in HCI. Springer, 49–68.Google Scholar
- Sara Heitlinger, Lara Houston, Alex Taylor, and Ruth Catlow. 2021. Algorithmic Food Justice: Co-Designing More-than-Human Blockchain Futures for the Food Commons. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Number 305. Association for Computing Machinery, 1–17. http://doi.org/10.1145/3411764.3445655Google ScholarDigital Library
- Nico Herbig, Santanu Pal, Josef van Genabith, and Antonio Krüger. 2019. Multi-Modal Approaches for Post-Editing Machine Translation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300461Google ScholarDigital Library
- Tom Hitron, Yoav Orlev, Iddo Wald, Ariel Shamir, Hadas Erel, and Oren Zuckerman. 2019. Can children understand machine learning concepts? The effect of uncovering black boxes. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–11.Google ScholarDigital Library
- Zoë Hitzig. 2020. The normative gap: mechanism design and ideal theories of justice. Economics & Philosophy 36, 3 (2020), 407–434.Google ScholarCross Ref
- Anna Lauren Hoffmann. 2020. Terms of inclusion: Data, discourse, violence. new media & society (2020), 1461444820958725.Google Scholar
- Stefan Holmlid. 2012. Participative; co-operative; emancipatory: From participatory design to service design. In Conference Proceedings ServDes. 2009; DeThinking Service; ReThinking Design; Oslo Norway 24-26 November 2009. Linköping University Electronic Press, 105–118.Google Scholar
- Naja Holten Møller, Irina Shklovski, and Thomas T Hildebrandt. 2020. Shifting concepts of value: Designing algorithmic decision-support systems for public services. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–12.Google ScholarDigital Library
- Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defining customer-centered systems. Elsevier.Google Scholar
- Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1120–1130.Google Scholar
- Ada Lovelace Institute. 2021. Participatory data stewardship. (2021). https://www.adalovelaceinstitute.org/report/participatory-data-stewardship/Google Scholar
- Modassir Iqbal, Katie Shilton, Mahmoud F. Sayed, Douglas Oard, Jonah Lynn Rivera, and William Cox. [n. d.]. Search with Discretion: Value Sensitive Design of Training Data for Information Retrieval. 5 ([n. d.]), 1–20. Issue CSCW1. https://doi.org/10.1145/3449207Google ScholarDigital Library
- Maia Jacobs, Jeffrey He, Melanie F. Pradier, Barbara Lam, Andrew C. Ahn, Thomas H. McCoy, Roy H. Perlis, Finale Doshi-Velez, and Krzysztof Z. Gajos. 2021. Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Number 659. Association for Computing Machinery, 1–14. http://doi.org/10.1145/3411764.3445385Google ScholarDigital Library
- Liu Jiang, Shixia Liu, and Changjian Chen. 2019. Recent research advances on interactive machine learning. Journal of Visualization 22, 2 (2019), 401–417.Google ScholarDigital Library
- Anson Kahng, Min Kyung Lee, Ritesh Noothigattu, Ariel Procaccia, and Christos-Alexandros Psomas. 2019. Statistical foundations of virtual democracy. In International Conference on Machine Learning. PMLR, 3173–3182.Google Scholar
- Simon Katan, Mick Grierson, and Rebecca Fiebrink. 2015. Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 251–254. https://doi.org/10.1145/2702123.2702474Google ScholarDigital Library
- Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P. M. Krafft. 2020. Toward Situated Interventions for Algorithmic Equity: Lessons from the Field. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 45–55. https://doi.org/10.1145/3351095.3372874Google ScholarDigital Library
- Harmanpreet Kaur, Eytan Adar, Eric Gilbert, and Cliff Lampe. 2022. Sensible AI: Re-imagining interpretability and explainability using sensemaking theory. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 702–714.Google ScholarDigital Library
- Jasmine Kaur, Rishabh Devgon, and Dhairya Chaudhary. [n. d.]. Work of Fiction: Using Speculative Design to Deliberate on the Future of Hiring. In Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing (New York, NY, USA, 2022-11-08) (CSCW’22 Companion). Association for Computing Machinery, 23–26. https://doi.org/10.1145/3500868.3559442Google ScholarDigital Library
- Claire Kayacik, Sherol Chen, Signe Noerly, Jess Holbrook, Adam Roberts, and Douglas Eck. 2019. Identifying the intersections: User experience+ research scientist collaboration in a generative machine learning interface. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- William R. Kearns, Neha Kaura, Myra Divina, Cuong Vo, Dong Si, Teresa Ward, and Weichao Yuwen. 2020. A Wizard-of-Oz Interface and Persona-Based Methodology for Collecting Health Counseling Dialog. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–9. https://doi.org/10.1145/3334480.3382902Google ScholarDigital Library
- Christopher Kelty. 2016. Too Much Democracy in All the Wrong Places: Toward a Grammar of Participation. Current Anthropology 58 (12 2016), S000–S000. https://doi.org/10.1086/688705Google ScholarCross Ref
- Maaike Kleinsmann and Rianne Valkenburg. 2008. Barriers and enablers for creating shared understanding in co-design projects. Design Studies 29, 4 (2008), 369–386. https://doi.org/10.1016/j.destud.2008.03.003Google ScholarCross Ref
- Tomoko Komatsu, Marisela Gutierrez Lopez, Stephann Makri, Colin Porlezza, Glenda Cooper, Andrew MacFarlane, and Sondess Missaoui. [n. d.]. AI should embody our values: Investigating journalistic values to inform AI technology design. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (New York, NY, USA, 2020-10-26) (NordiCHI ’20). Association for Computing Machinery, 1–13. https://doi.org/10.1145/3419249.3420105Google ScholarDigital Library
- P. M. Krafft, Meg Young, Michael Katell, Jennifer E. Lee, Shankar Narayan, Micah Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, Corinne Bintz, Daniella Raz, Pa Ousman Jobe, Franziska Putz, Brian Robick, and Bissan Barghouti. 2021. An Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 772–781. https://doi.org/10.1145/3442188.3445938Google ScholarDigital Library
- Kavita Krishnaswamy. 2017. Participatory design: repositioning, transferring, and personal care robots. In Proceedings of the companion of the 2017 acm/ieee international conference on human-robot interaction. 351–352.Google ScholarDigital Library
- Kavita Krishnaswamy, Srinivas Moorthy, and Tim Oates. 2017. Preliminary Survey Analysis in Participatory Design: Repositioning, Transferring, and Personal Care Robots. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. 171–172.Google ScholarDigital Library
- Bogdan Kulynych, David Madras, Smitha Milli, Inioluwa Deborah Raji, Angela Zhou, and Richard Zemel. 2020. Participatory Approaches to Machine Learning. International Conference on Machine Learning Workshop.Google Scholar
- Christopher A. Le Dantec and Sarah Fox. 2015. Strangers at the Gate: Gaining Access, Building Rapport, and Co-Constructing Community-Based Research. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 1348–1358. https://doi.org/10.1145/2675133.2675147Google ScholarDigital Library
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444.Google Scholar
- Hee Rin Lee, Selma Šabanović, Wan-Ling Chang, Shinichi Nagata, Jennifer Piatt, Casey Bennett, and David Hakken. 2017. Steps toward participatory design of social robots: mutual learning with older adults with depression. In Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction. 244–253.Google ScholarDigital Library
- Min Hun Lee, Daniel P. Siewiorek, Asim Smailagic, Alexandre Bernardino, and Sergi Bermúdez i Badia. [n. d.]. Co-Design and Evaluation of an Intelligent Decision Support System for Stroke Rehabilitation Assessment. 4 ([n. d.]), 156:1–156:27. Issue CSCW2. https://doi.org/10.1145/3415227Google ScholarDigital Library
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2019), 1–35.Google Scholar
- Min Kyung Lee, Ishan Nigam, Angie Zhang, Joel Afriyie, Zhizhen Qin, and Sicun Gao. 2021. Participatory Algorithmic Management: Elicitation Methods for Worker Well-Being Models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA, 2021-07-21) (AIES ’21). Association for Computing Machinery, 715–726. https://doi.org/10.1145/3461702.3462628Google ScholarDigital Library
- Michael Lempert and E Summerson Carr. 2016. Scale: Discourse and dimensions of social life. University of California Press.Google Scholar
- Fei-Fei Li. 2021. America’s global leadership in human-centered AI can’t come from industry alone. https://thehill.com/opinion/technology/561638-americas-global-leadership-in-human-centered-ai-cant-come-from-industryGoogle Scholar
- Xingjun Li, Yuanxin Wang, Hong Wang, Yang Wang, and Jian Zhao. 2021. NBSearch: Semantic Search and Visual Exploration of Computational Notebooks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Q Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–15.Google ScholarDigital Library
- Ann Light and Rosemary Luckin. 2008. Designing for social justice: people, technology, learning. (2008).Google Scholar
- Phoebe Lin and Jessica Van Brummelen. 2021. Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Number 239. Association for Computing Machinery, 1–12. http://doi.org/10.1145/3411764.3445377Google ScholarDigital Library
- Walter Lippmann. 1993. The phantom public. Transaction Publishers.Google Scholar
- Gabrielle Kaili-May Liu. 2023. Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback. arXiv preprint arXiv:2303.02891 (2023).Google Scholar
- Daria Loi, Thomas Lodato, Christine T Wolf, Raphael Arar, and Jeanette Blomberg. 2018. PD manifesto for AI futures. In Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial-Volume 2. 1–4.Google ScholarDigital Library
- Daria Loi, Christine T Wolf, Jeanette L Blomberg, Raphael Arar, and Margot Brereton. 2019. Co-designing AI futures: Integrating AI ethics, social computing, and design. In Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion. 381–384.Google ScholarDigital Library
- Matti Luhtala, Santeri Saarinen, Laura Virkki, Johanna Lappi-Ramula, Hans Hokka, Helen Tran, Matti Savolainen, Ilkka Heinonen, Sami Kanerva, Juha Rajala, 2018. Proactive Rescuework by Enhancing Situational Awareness: Modeling Resources, Services and People. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- C. B Macpherson. 1973. Democratic theory: essays in retrieval. Clarendon Press. OCLC: 606090.Google Scholar
- Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. 2022. Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–26.Google ScholarDigital Library
- Peter Mambrey, Gloria Mark, and Uta Pankoke-Babatz. 1998. User advocacy in participatory design: Designers’ experiences with a new communication channel. Computer Supported Cooperative Work (CSCW) 7, 3 (1998), 291–313.Google ScholarDigital Library
- Ji-Ye Mao, Karel Vredenburg, Paul W Smith, and Tom Carey. 2005. The state of user-centered design practice. Commun. ACM 48, 3 (2005), 105–109.Google ScholarDigital Library
- Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. 2020. Participatory problem formulation for fairer machine learning through community based system dynamics. arXiv preprint arXiv:2005.07572 (2020).Google Scholar
- Alexandra Mateescu and Madeleine Clare Elish. 2019. AI in context: the labor of integrating new technologies. Data & Society 30 (2019).Google Scholar
- Oussama Metatla, Alison Oldfield, Taimur Ahmed, Antonis Vafeas, and Sunny Miglani. 2019. Voice user interfaces in schools: Co-designing for inclusion with visually-impaired and sighted pupils. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–15.Google ScholarDigital Library
- Milagros Miceli and Julian Posada. 2022. The Data-Production Dispositif. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–37.Google ScholarDigital Library
- Milagros Miceli, Martin Schuessler, and Tianling Yang. 2020. Between subjectivity and imposition: Power dynamics in data annotation for computer vision. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–25.Google ScholarDigital Library
- Milagros Miceli, Tianling Yang, Adriana Alvarado Garcia, Julian Posada, Sonja Mei Wang, Marc Pohl, and Alex Hanna. [n. d.]. Documenting Data Production Processes: A Participatory Approach for Data Work. 6 ([n. d.]), 510:1–510:34. Issue CSCW2. https://doi.org/10.1145/3555623Google ScholarDigital Library
- Henrik Mucha, Dennis Mevißen, Sebastian Robert, Ricarda Jacobi, Kirsten Meyer, Winfried Heusler, and Daniel Arztmann. 2020. Co-Design Futures for AI and Space: A Workbook Sprint. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375203Google ScholarDigital Library
- Michael J. Muller. 2002. Participatory Design: The Third Space in HCI. L. Erlbaum Associates Inc., USA, 1051–1068.Google Scholar
- Michael J Muller. 2002. Participatory design: the third space in HCI. The humancomputer interaction handbook: fundamentals, evolving technologies and emerging applications.Google Scholar
- Michael J Muller and Allison Druin. 2012. Participatory design: the third space in human–computer interaction. CRC Press, 1125–1153.Google Scholar
- Michael J Muller and Sarah Kuhn. 1993. Participatory design. Commun. ACM 36, 6 (1993), 24–28.Google ScholarDigital Library
- Dylan Mulvin. 2021. Proxies: The cultural work of standing in. MIT Press.Google ScholarCross Ref
- Yuri Nakao and Yusuke Sugano. [n. d.]. Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (Tallinn Estonia, 2020-10-25). ACM, 1–12. https://doi.org/10.1145/3419249.3420157Google ScholarDigital Library
- Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo, Salomey Osei, 2020. Participatory research for low-resourced machine translation: A case study in african languages. arXiv preprint arXiv:2010.02353 (2020).Google Scholar
- Isabel Neto, Hugo Nicolau, and Ana Paiva. 2021. Community Based Robot Design for Classrooms with Mixed Visual Abilities Children. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Number 31. Association for Computing Machinery, 1–12. http://doi.org/10.1145/3411764.3445135Google ScholarDigital Library
- Laura J Neumann and Susan Leigh Star. 1996. Making infrastructure: The dream of a common language. In Participatory Design Conference, Vol. 4. 231–240.Google Scholar
- Ritesh Noothigattu, Snehalkumar Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel Procaccia. 2018. A voting-based system for ethical decision making. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.Google ScholarCross Ref
- PerOla Öberg. 2016. Deliberation. In Handbook on theories of governance. Edward Elgar Publishing.Google Scholar
- Judith S Olson and Wendy A Kellogg. 2014. Ways of Knowing in HCI. Vol. 2. Springer.Google Scholar
- [162] OpenAI. 2023. https://openai.com/blog/how-should-ai-systems-behaveGoogle Scholar
- David Owen and Graham Smith. 2015. Survey article: Deliberation, democracy, and the systemic turn. Journal of Political Philosophy 23, 2 (2015), 213–234.Google ScholarCross Ref
- Victoria Palacin, Matti Nelimarkka, Pedro Reynolds-Cuéllar, and Christoph Becker. 2020. The design of pseudo-participation. In Proceedings of the 16th Participatory Design Conference 2020-Participation (s) Otherwise-Volume 2. 40–44.Google ScholarDigital Library
- Lawrence A Palinkas, Sarah M Horwitz, Carla A Green, Jennifer P Wisdom, Naihua Duan, and Kimberly Hoagwood. 2015. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and policy in mental health and mental health services research 42 (2015), 533–544.Google Scholar
- Hyanghee Park, Daehwan Ahn, Kartik Hosanagar, and Joonhwan Lee. 2022. Designing Fair AI in Human Resource Management: Understanding Tensions Surrounding Algorithmic Evaluation and Envisioning Stakeholder-Centered Solutions. In CHI Conference on Human Factors in Computing Systems. 1–22.Google Scholar
- Joon Sung Park, Karrie Karahalios, Niloufar Salehi, and Motahhare Eslami. [n. d.]. Power Dynamics and Value Conflicts in Designing and Maintaining Socio-Technical Algorithmic Processes. 6 ([n. d.]), 110:1–110:21. Issue CSCW1. https://doi.org/10.1145/3512957Google ScholarDigital Library
- Sunjeong Park and Youn-kyung Lim. 2020. Investigating User Expectations on the Roles of Family-shared AI Speakers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1–13. https://doi.org/10.1145/3313831.3376450Google ScholarDigital Library
- Desmond U Patton, William R Frey, Kyle A McGregor, Fei-Tzin Lee, Kathleen McKeown, and Emanuel Moss. 2020. Contextual analysis of social media: The promise and challenge of eliciting context in social media posts with natural language processing. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 337–342.Google ScholarDigital Library
- Jens Pedersen. 2016. War and peace in codesign. CoDesign 12 (2016), 171 – 184.Google ScholarCross Ref
- Ismael Peña-López 2020. Innovative citizen participation and new democratic institutions: Catching the deliberative wave. (2020).Google Scholar
- Scott Pobiner and Timothy Murphy. 2018. Participatory approaches to machine learning. https://www2.deloitte.com/us/en/insights/focus/cognitive-technologies/participatory-design-artificial-intelligence.htmlGoogle Scholar
- Francesca Polletta. 2012. Freedom is an endless meeting. In Freedom Is an Endless Meeting. University of Chicago Press.Google Scholar
- Aare Puussaar, Kyle Montague, Sean Peacock, Thomas Nappey, Robert Anderson, Jennine Jonczyk, Peter Wright, and Philip James. [n. d.]. SenseMyStreet: Sensor Commissioning Toolkit for Communities. 6 ([n. d.]), 324:1–324:26. Issue CSCW2. https://doi.org/10.1145/3555215Google ScholarDigital Library
- Organizers Of Queerinai, Anaelia Ovalle, Arjun Subramonian, Ashwin Singh, Claas Voelcker, Danica J Sutherland, Davide Locatelli, Eva Breznik, Filip Klubicka, Hang Yuan, 2023. Queer In AI: A Case Study in Community-Led Participatory AI. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1882–1895.Google ScholarDigital Library
- Lauge Baungaard Rasmussen. 2004. Action research—Scandinavian experiences. AI & SOCIETY 18, 1 (2004), 21–43.Google ScholarDigital Library
- Daniella Raz, Corinne Bintz, Vivian Guetler, Aaron Tam, Michael Katell, Dharma Dailey, Bernease Herman, PM Krafft, and Meg Young. 2021. Face Mis-ID: An Interactive Pedagogical Tool Demonstrating Disparate Accuracy Rates in Facial Recognition. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 895–904.Google ScholarDigital Library
- People + AI Research. 2020. Boundary objects for participatory machine learning: PAIR Symposium 2020 Recap. https://medium.com/people-ai-research/pair-symposium-2020-recap-b7f206583e42Google Scholar
- People + AI Research. 2021. PAIR @ Chi 2021. https://medium.com/people-ai-research/pair-chi-2021-410323929162Google Scholar
- Samantha Robertson and Niloufar Salehi. 2020. What If I Don’t Like Any Of The Choices? The Limits of Preference Elicitation for Participatory Algorithm Design. arXiv preprint arXiv:2007.06718 (2020).Google Scholar
- Emma J Rose and Elin A Björling. 2017. Designing for engagement: using participatory design to develop a social robot to measure teen stress. In Proceedings of the 35th ACM International Conference on the Design of Communication. 1–10.Google ScholarDigital Library
- Joanna Saad-Sulonen, Amalia De Götzen, Nicola Morelli, and Luca Simeone. 2020. Service design and participatory design: time to join forces?. In Proceedings of the 16th Participatory Design Conference 2020-Participation (s) Otherwise-Volume 2. 76–81.Google ScholarDigital Library
- Selma Šabanović, Wan-Ling Chang, Casey C Bennett, Jennifer A Piatt, and David Hakken. 2015. A robot of my own: participatory design of socially assistive robots for independently living older adults diagnosed with depression. In International conference on human aspects of it for the aged population. Springer, 104–114.Google ScholarDigital Library
- Debjani Saha, Candice Schumann, Duncan Mcelfresh, John Dickerson, Michelle Mazurek, and Michael Tschantz. 2020. Measuring non-expert comprehension of machine learning fairness metrics. In International Conference on Machine Learning. PMLR, 8377–8387.Google Scholar
- Elizabeth B.-N. Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. CoDesign 4, 1 (2008), 5–18. https://doi.org/10.1080/15710880701875068Google ScholarCross Ref
- Devansh Saxena, Karla Badillo-Urquiola, Pamela Wisniewski, and Shion Guha. 2021. A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare. arXiv preprint arXiv:2107.03487 (2021).Google ScholarDigital Library
- Devansh Saxena and Shion Guha. 2020. Conducting Participatory Design to Improve Algorithms in Public Services: Lessons and Challenges. Association for Computing Machinery, New York, NY, USA, 383–388. https://doi.org/10.1145/3406865.3418331Google ScholarDigital Library
- Laura Schelenz, Ivano Bison, Matteo Busso, Amalia de Götzen, Daniel Gatica-Perez, Fausto Giunchiglia, Lakmal Meegahapola, and Salvador Ruiz-Correa. 2021. The Theory, Practice, and Ethical Challenges of Designing a Diversity-Aware Platform for Social Relations. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA, 2021-07-21) (AIES ’21). Association for Computing Machinery, 905–915. https://doi.org/10.1145/3461702.3462595Google ScholarDigital Library
- Daniel Schiff, Bogdana Rakova, Aladdin Ayesh, Anat Fanti, and Michael Lennon. 2020. Principles to practices for responsible AI: closing the gap. arXiv preprint arXiv:2006.04707 (2020).Google Scholar
- Kristen M Scott, Sonja Mei Wang, Milagros Miceli, Pieter Delobelle, Karolina Sztandar-Sztanderska, and Bettina Berendt. 2022. Algorithmic Tools in Public Employment Services: Towards a Jobseeker-Centric Perspective. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 2138–2148.Google ScholarDigital Library
- Hugo Scurto, Baptiste Caramiaux, and Frederic Bevilacqua. 2021. Prototyping Machine Learning Through Diffractive Art Practice. In Designing Interactive Systems Conference 2021 (New York, NY, USA, 2021-06-28) (DIS ’21). Association for Computing Machinery, 2013–2025. https://doi.org/10.1145/3461778.3462163Google ScholarDigital Library
- Nick Seaver. 2021. Care and scale: decorrelative ethics in algorithmic recommendation. Cultural Anthropology 36, 3 (2021), 509–537.Google ScholarCross Ref
- Elizabeth Seger, Aviv Ovadya, Ben Garfinkel, Divya Siddarth, and Allan Dafoe. 2023. Democratising AI: Multiple Meanings, Goals, and Methods. arXiv preprint arXiv:2303.12642 (2023).Google Scholar
- Cathrine Seidelin, Yvonne Dittrich, and Erik Grönvall. 2020. Co-designing data experiments: Domain experts’ exploration and experimentation with self-selected data sources. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. 1–11.Google ScholarDigital Library
- Amartya Sen. 1977. Social choice theory: A re-examination. Econometrica: journal of the Econometric Society (1977), 53–89.Google Scholar
- Amartya Sen. 1986. Social choice theory. Handbook of mathematical economics 3 (1986), 1073–1181.Google Scholar
- Mark P Sendak, William Ratliff, Dina Sarro, Elizabeth Alderton, Joseph Futoma, Michael Gao, Marshall Nichols, Mike Revoir, Faraz Yashar, Corinne Miller, 2020. Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR medical informatics 8, 7 (2020), e15182.Google ScholarCross Ref
- Hong Shen, Wesley H Deng, Aditi Chattopadhyay, Zhiwei Steven Wu, Xu Wang, and Haiyi Zhu. 2021. Value Cards: An Educational Toolkit for Teaching Social Impacts of Machine Learning through Deliberation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 850–861.Google ScholarDigital Library
- Hong Shen, Haojian Jin, Ángel Alexander Cabrera, Adam Perer, Haiyi Zhu, and Jason I Hong. 2020. Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–22.Google ScholarDigital Library
- Ben Shneiderman. 2021. Human-Centered AI. Issues in Science and Technology 37, 2 (2021), 56–61.Google Scholar
- Jesper Simonsen and Toni Robertson. 2012. Routledge international handbook of participatory design. Routledge.Google Scholar
- Zoe Skinner, Stacey Brown, and Greg Walsh. 2020. Children of Color’s Perceptions of Fairness in AI: An Exploration of Equitable and Inclusive Co-Design. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.Google Scholar
- Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2020. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423 (2020).Google Scholar
- C Estelle Smith, Bowen Yu, Anjali Srivastava, Aaron Halfaker, Loren Terveen, and Haiyi Zhu. 2020. Keeping community in the loop: Understanding wikipedia stakeholder values for machine learning-based systems. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Logan Stapleton, Min Hun Lee, Diana Qing, Marya Wright, Alexandra Chouldechova, Ken Holstein, Zhiwei Steven Wu, and Haiyi Zhu. 2022. Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 1162–1177.Google ScholarDigital Library
- Marc Steen. 2013. Co-Design as a Process of Joint Inquiry and Imagination. Design Issues 29, 2 (2013), 16–28. http://www.jstor.org/stable/24266991Google ScholarCross Ref
- Elias Storms, Oscar Alvarado, and Luciana Monteiro-Krebs. [n. d.]. ’Transparency is Meant for Control’ and Vice Versa: Learning from Co-designing and Evaluating Algorithmic News Recommenders. 6 ([n. d.]), 405:1–405:24. Issue CSCW2. https://doi.org/10.1145/3555130Google ScholarDigital Library
- Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Towards A Process Model for Co-Creating AI Experiences. In Designing Interactive Systems Conference 2021. Association for Computing Machinery, 1529–1543. http://doi.org/10.1145/3461778.3462012Google ScholarDigital Library
- Harini Suresh, Rajiv Movva, Amelia Lee Dogan, Rahul Bhargava, Isadora Cruxen, Angeles Martinez Cuba, Guilia Taurino, Wonyoung So, and Catherine D’Ignazio. 2022. Towards Intersectional Feminist and Participatory ML: A Case Study in Supporting Feminicide Counterdata Collection. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 667–678.Google Scholar
- Giovanni Maria Troiano, Matthew Wood, and Casper Harteveld. 2020. "And This, Kids, Is How I Met Your Mother": Consumerist, Mundane, and Uncanny Futures with Sex Robots. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 1–17. http://doi.org/10.1145/3313831.3376598Google ScholarDigital Library
- Anna Lowenhaupt Tsing. 2012. On NonscalabilityThe Living World Is Not Amenable to Precision-Nested Scales. Common Knowledge 18, 3 (2012), 505–524.Google ScholarCross Ref
- Kim M Unertl, Chris L Schaefbauer, Terrance R Campbell, Charles Senteio, Katie A Siek, Suzanne Bakken, and Tiffany C Veinot. 2016. Integrating community-based participatory research and informatics approaches to improve the engagement and health of underserved populations. Journal of the American Medical Informatics Association 23, 1 (2016), 60–73.Google ScholarCross Ref
- Kush Varshney, Tina Park, Inioluwa Deborah Raji, Gaurush Hiranandani, Narasimhan Harikrishna, Oluwasanmi Koyejo, Brianna Richardson, and Min Kyung Lee. 2021. Participatory Specification of Trustworthy Machine Learning. https://www.abstractsonline.com/pp8/#!/10390/session/446Google Scholar
- Daryoush Vaziri, David Golchinfar, Gunnar Stevens, and Dirk Schreiber. [n. d.]. Exploring Future Work - Co-Designing a Human-robot Collaboration Environment for Service Domains. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. Association for Computing Machinery, 153–164. http://doi.org/10.1145/3357236.3395483Google ScholarDigital Library
- Salomé Viljoen, Jake Goldenfein, and Lee McGuigan. 2021. Design choices: Mechanism design and platform capitalism. Big Data & Society 8, 2 (2021), 20539517211034312.Google ScholarCross Ref
- John Vines, Rachel Clarke, Peter Wright, John McCarthy, and Patrick Olivier. 2013. Configuring Participation: On How We Involve People in Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 429–438. https://doi.org/10.1145/2470654.2470716Google ScholarDigital Library
- John Vines, Peter C. Wright, David Silver, Maggie Winchcombe, and Patrick Olivier. 2015. Authenticity, Relatability and Collaborative Approaches to Sharing Knowledge about Assistive Living Technology. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 82–94. https://doi.org/10.1145/2675133.2675222Google ScholarDigital Library
- Nina B Wallerstein and Bonnie Duran. 2006. Using community-based participatory research to address health disparities. Health promotion practice 7, 3 (2006), 312–323.Google Scholar
- Greg Walsh and Eric Wronsky. 2019. AI + Co-Design: Developing a Novel Computer-Supported Approach to Inclusive Design. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (Austin, TX, USA) (CSCW ’19). Association for Computing Machinery, New York, NY, USA, 408–412. https://doi.org/10.1145/3311957.3359456Google ScholarDigital Library
- Christine Wolf and Jeanette Blomberg. [n. d.]. Evaluating the Promise of Human-Algorithm Collaborations in Everyday Work Practices. 3 ([n. d.]), 143:1–143:23. Issue CSCW. https://doi.org/10.1145/3359245Google ScholarDigital Library
- Christine T. Wolf, Haiyi Zhu, Julia Bullard, Min Kyung Lee, and Jed R. Brubaker. 2018. The Changing Contours of "Participation" in Data-Driven, Algorithmic Ecosystems: Challenges, Tactics, and an Agenda. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing (Jersey City, NJ, USA) (CSCW ’18). Association for Computing Machinery, New York, NY, USA, 377–384. https://doi.org/10.1145/3272973.3273005Google ScholarDigital Library
- Allison Woodruff, Sarah E Fox, Steven Rousso-Schindler, and Jeffrey Warshaw. 2018. A qualitative exploration of perceptions of algorithmic fairness. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–14.Google ScholarDigital Library
- Julia Woodward, Zari McFadden, Nicole Shiver, Amir Ben-hayon, Jason C Yip, and Lisa Anthony. 2018. Using co-design to examine how children conceptualize intelligent interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Deepika Yadav, Prerna Malik, Kirti Dabas, and Pushpendra Singh. 2019. Feedpal: Understanding Opportunities for Chatbots in Breastfeeding Education of Women in India. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 170 (nov 2019), 30 pages. https://doi.org/10.1145/3359272Google ScholarDigital Library
- Meg Young, Lassana Magassa, and Batya Friedman. 2019. Toward inclusive tech policy design: a method for underrepresented voices to strengthen tech policy documents. Ethics and Information Technology 21, 2 (2019), 89–103.Google ScholarDigital Library
- Bowen Yu, Ye Yuan, Loren Terveen, Zhiwei Steven Wu, Jodi Forlizzi, and Haiyi Zhu. 2020. Keeping designers in the loop: Communicating inherent algorithmic trade-offs across multiple objectives. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1245–1257.Google ScholarDigital Library
- Angie Zhang, Alexander Boltz, Chun Wei Wang, and Min Kyung Lee. 2022. Algorithmic management reimagined for workers and by workers: Centering worker well-being in gig work. In CHI Conference on Human Factors in Computing Systems. 1–20.Google ScholarDigital Library
- Yunfeng Zhang, Rachel Bellamy, and Kush Varshney. 2020. Joint optimization of AI fairness and utility: A human-centered approach. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 400–406.Google ScholarDigital Library
- Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022. StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. In CHI Conference on Human Factors in Computing Systems. 1–21.Google ScholarDigital Library
Index Terms
-
The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
-
Recommendations
-
The diversity of participatory design research practice at PDC 2002-2012
We investigate the diversity of participatory design research practice, based on a review of ten years of participatory design research published as full research papers at the Participatory Design Conferences (PDC) 2002-2012, and relate this body of ...
-
The role of design fiction in participatory design processes
NordiCHI '18: Proceedings of the 10th Nordic Conference on Human-Computer InteractionParticipatory design is in essence very malleable as any design technique could lend itself to it, as long as users and stakeholders are involved. Design fictions however, have more often been used as either a vehicle for critical designs, or as a sheer ...
-
Unpacking the Notion of Participation in Participatory Design
The paper explores what exactly it is that users participate in when being involved in participatory design (PD), relating this discussion to the CSCW perspective on collaborative design work. We argue that a focus on decision-making in design is ...
Comments