Anne Cleven
ABSTRACT
Within a consideration of cost effectiveness the evaluation of design science research artifacts is of major importance. In the past, a plenitude of approaches has been developed for this purpose -- partly artifact-specific, partly artifact-neutral. Nonetheless, there is a lack of a comprehensive overview over existing methods as well as a systemization of those with regard to fundamental structuring criteria. The paper at hand surveys existing methods and introduces a framework that equally supports the designer and the user of artifact evaluation approaches. Subsequent to the embedding of the framework into the design science research process two exemplary application scenarios are described.
- Avison, D. E., Lau, F., Myers, M. D. and Nielsen, P. A. 1999. Action Research. Communications of the ACM, 42, 2, 94--97. Google Scholar
Digital Library
- Becker, J. and Niehaves, B. 2007. Epistemological perspectives on IS research: a framework for analysing and systematizing epistemological assumptions. Information Systems Journal, 17, 2, 197--214.Google ScholarCross Ref
- Bhola, H. S. 1990. Evaluating "Literacy for Development" Projects, Programs and Campaigns. Evaluation Planning, Design and Implementation, and Utilization of Evaluation Results. Unesco Institute for Education, Hamburg, Germany.Google Scholar
- Brinkkemper, S., Saeki, M. and Harmsen, F. 1999. Metamodelling based assembly techniques for situational method engineering. Information Systems, 24, 3, 209--228.Google ScholarCross Ref
- Bucher, T., Riege, C. and Saat, J. 2008. Evaluation in der gestaltungsorientierten Wirtschaftsinformatik -- Systematisierung nach Erkenntnisziel und Gestaltungsziel. In Wissenschaftstheorie und gestaltungsorientierte Wirtschaftsinformatik, J. Becker, H. Krcmar, B. Niehaves, Eds. Physica, Heidelberg, Germany, 69--86.Google Scholar
- Burrell, G. and Morgan, G. 1979. Sociological paradigms and organisational analysis: Elements of the sociology of corporate life. Heinemann, London, UK.Google Scholar
- Carlsson, S. A. 2006. Towards an Information Systems Design Research Framework: A Critical Realist Perspective. In Proceedings of the 1st International Conference on Design Science Research in Information Systems and Technology (DESRIST), (Claremont, California).Google Scholar
- Chen, W. and Hirschheim, R. 2004. A paradigmatic and methodological examination of information systems research from 1991 to 2001. Information Systems Journal, 14, 197--235.Google ScholarCross Ref
- Cross, N. 2001. Designerly Ways of Knowing: Design Discipline Versus Design Science. Design Issues, 17, 3, 49--55.Google ScholarCross Ref
- Fettke, P. and Loos, P. 2003. Multiperspective Evaluation of Reference Models -- Towards a Freamwork. In ER 2003 Workshops, LNCS 2814, M. A. Jeusfeld, P. Ó., Eds., 80--91.Google Scholar
- Frank, U. 2007. Evaluation of Reference Models. In Reference Modeling for Business Systems Analysis, P. Fettke, P. Loos, Eds. IDEA Group, 118--139.Google Scholar
- Frank, U. 2000. Evaluation von Artefakten in der Wirtschaftsinformatik. In Evaluation und Evaluationsforschung in der Wirtschaftsinformatik, I. Häntschel, L. J. Heinrich, Eds. Oldenbourg, München et al., Germany, 35--48.Google Scholar
- Gregor, S. 2006. The Nature of Theory in Information Systems. MIS Quarterly, 30, 3, 611--642. Google ScholarDigital Library
- Hevner, A. R., March, S. T., Park, J. and Ram, S. 2004. Design Science in Information Systems Research. MIS Quarterly, 28, 1, 75--105. Google ScholarCross Ref
- Kuechler, W. and Vaishnavi, V. K. 2008. Theory Development in Design Science Research: Anatomy of a Research Project. In Proceedings of the 3rd International Conference on Design Science Research in Information Systems and Technology (DESRIST), (Atlanta, Georgia).Google Scholar
- Lee, J., Wyner, G. M. and Pentland, B. T. 2008. Process Grammar as a Tool for Business Process Design. MIS Quarterly, 32, 4, 757--778. Google ScholarDigital Library
- March, S. T. and Smith, G. F. 1995. Design and natural science research on information technology. Decision Support Systems, 15, 4, 251--266. Google ScholarDigital Library
- March, S. T. and Storey, V. C. 2008. Design Science in the Information Systems Discipline: An Introduction to the Special Issue on Design Science Research. MIS Quarterly, 32, 4, 725--730. Google ScholarDigital Library
- Österle, H. and Winter, R. 2003. Business Engineering. In Business Engineering -- Auf dem Weg zum Unternehmen des Informationszeitalters, H. Österle, R. Winter, Eds. Springer, Berlin, Germany, 4--20.Google Scholar
- Parsons, J. and Wand, Y. 2008. Using Cognitive Principles to Guide Classification in Information Systems Modelling. MIS Quarterly, 32, 4, 839--868. Google ScholarDigital Library
- Peffers, K., Tuunanen, T., Rothenberger, M. A. and Chatterjee, S. 2007. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24, 3, 45--77. Google ScholarDigital Library
- Pfeiffer, D. and Niehaves, B. 2005. Evaluation of conceptual models -- a structuralist approach. In Proceedings of the 13th European Conference on Information Systems (ECIS), (Regensburg, Germany), 459--470.Google Scholar
- Pries-Heje, J., Baskerville, R. and Venable, J. 2008. Strategies for design science research evaluation. In Proceedings of the 16th European Conference on Information Systems (ECIS), (Galway, Ireland), 255--266.Google Scholar
- Ritchey, T. 2006. Problem structuring using computer-aided morphological analysis. The Journal of the Operational Research Society, 57, 7, 792--801.Google ScholarCross Ref
- Scriven, M. 1998. The new science of evaluation. International Journal of Social Welfare, 7, 2, S. 79--86.Google Scholar
- Siau, K. and Rossi, M. 2007. Evaluation techniques for systems analysis and design modelling methods -- a review and comparative analysis. Information Systems Journal, 1--20.Google Scholar
- Simon, H. A. 1969. The Sciences of the Artificial. The MIT Press, Cambridge, Massachusetts. Google ScholarDigital Library
- Stockmann, R. 2006. Evaluation und Qualitätsentwicklung: Eine Grundlage für wirkungsorientiertes Qualitätsmanagement. Waxmann, Münster et al., Germany.Google Scholar
- Takeda, H., Veerkamp, P., Tomiyama, T. and Yoshikawa, H. 1990. Modeling design processes. AI Magazine, 11, 4, 37--48. Google ScholarDigital Library
- Venable, J. 2006. The Role of Theory and Theorising in Design Science Research. In Proceedings of the 1st International Conference on Design Science in Information Systems and Technology (DESRIST), (Claremont, California), 1--18.Google Scholar
- Walls, J. G., Widmeyer, G. R. and El Sawy, O. A. 2004. Assessing information system design theory in perspective: how useful was our 1992 initial rendition. Journal of Information Technology Theory and Application, 6, 3, 43--58.Google Scholar
- Wand, Y. and Weber, R. 1990. An Ontological Model of an Information System. IEEE Transactions on Software Engineering, 16, 11, 1282--1292. Google ScholarDigital Library
- Winter, R. 2008. Design science research in Europe. European Journal of Information Systems, 17, 470--475.Google ScholarCross Ref
- Wottawa, H. and Thierau, H. 1990. Lehrbuch Evaluation. Verlag Hans Huber, Bern et al., Switzerland.Google Scholar
- Yin, R. K. 2003. Case study Research: Design and Methods. SAGE Publications.Google Scholar
Index Terms
-
Design alternatives for the evaluation of design science research artifacts
-
Recommendations
-
Evaluating an Artifact in Design Science Research
SAICSIT '15: Proceedings of the 2015 Annual Research Conference on South African Institute of Computer Scientists and Information TechnologistsIn this paper, we describe the iterative evaluation of an artifact developed through the application of Design Science Research (DSR) methodology in a resource constrained environment. In the DSR process the aspect of evaluation is often done ...
-
Design science research evaluation
DESRIST'12: Proceedings of the 7th international conference on Design Science Research in Information Systems: advances in theory and practiceThe consensus view is that the rigorous evaluation of design science (DS) artifacts is essential. There are many types of DS artifacts and many forms of evaluation; what is missing is guidance for how to perform the evaluation, more specifically, what ...
-
Design science research post hevner et al.: criteria, standards, guidelines, and expectations
DESRIST'10: Proceedings of the 5th international conference on Global Perspectives on Design Science ResearchThere is ongoing debate about how the quality (rigour and relevance) of Design Science Research (DSR) should be judged This research investigates the state of the debate by surveying the opinions of IS scholars who write, review, edit, and publish DSR ...
Comments