Skip to main content
SpringerLink
Log in

A framework for empirical evaluation of conceptual modeling techniques

  • Original Article
  • Published:
Requirements Engineering Aims and scope Submit manuscript

Abstract

The paper presents a framework for the empirical evaluation of conceptual modeling techniques used in requirements engineering. The framework is based on the notion that modeling techniques should be compared via their underlying grammars. The framework identifies two types of dimensions in empirical comparisons—affecting and affected dimensions. The affecting dimensions provide guidance for task definition, independent variables and controls, while the affected dimensions define the possible mediating variables and dependent variables. In particular, the framework addresses the dependence between the modeling task—model creation and model interpretation—and the performance measures of the modeling grammar. The utility of the framework is demonstrated by using it to categorize existing work on evaluating modeling techniques. The paper also discusses theoretical foundations that can guide hypothesis generation and measurement of variables. Finally, the paper addresses possible levels for categorical variables and ways to measure interval variables, especially the grammar performance measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Avison DE, Fitzgerald G (1995) Information systems development: methodologies, techniques, and tools, 2nd edn. McGraw-Hill, London

    Google Scholar 

  2. Agarwal R, Sinha A, Tanniru M (1996) Cognitive fit in requirements engineering: a study of object and process models. J Manag Inf Syst 13(2):137–162

    Google Scholar 

  3. Agarwal R, De P, Sinha AP (1999) Comprehending object and process models: an empirical study. IEEE Trans Softw Eng 25(4):541–556

    Article  Google Scholar 

  4. Batra D, J Hoffer, R Bostrom (1990) Comparing representations with relational and EER models. Commun ACM 33(2):126–139

    Article  Google Scholar 

  5. Bodart F, Sim M, Patel A, Weber R (2001) Should optional properties be used in conceptual modelling? A theory and three empirical tests. Inf Syst Res 12(4):384–405

    Article  Google Scholar 

  6. Brooks FP (1998) The mythical man-month: essays of software engineering, Anniversary edition. Addison-Wesley

    Google Scholar 

  7. Brosey M, Schniederman B (1978) Two experimental comparisons of relational and hierarchical database models. Int J Man Machine Stud 10:625–637

    Article  CAS  Google Scholar 

  8. Bubenko JA Jr (1986) Information system methodologies: a research review. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Information system design methodologies: improving the practice. Elsevier, North Holland, pp 289–317

    Google Scholar 

  9. Burton-Jones A, Meso P (2002) How good are these UML diagrams? An empirical test of the Wand and Weber good decomposition model. In: Applegate L, Galliers R, DeGross JI (eds) Proceedings of the international conference on information systems 2002, December, 2002

  10. Chaos (1995) Standish group report on information system development. http://www.standishgroup.com/chaos.html. Accessed 12 Jul 1996]

  11. Collins AM, Quillan MR (1969) Retrieval time from semantic memory. J Verbal Learn Behav 8:240–247

    Google Scholar 

  12. Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technologies. MIS Q 13(3):319–340

    Google Scholar 

  13. Floyd C (1986) A comparative evaluation of systems development methods. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Proceedings of the IFIP WG 8.1 working conference on comparative review of information systems design methodologies: improving the practice. North-Holland, Amsterdam, pp 19–54

    Google Scholar 

  14. Gemino A (1999) Empirical comparison of system analysis techniques. PhD Thesis, University of British Columbia

  15. Gemino A (2004) Empirical comparisons of animation and narration in requirements validation. Req Eng J 9(3):153–168. DOI 10.1007/s00766-003-0182-0

    Google Scholar 

  16. Gemino A, Wand Y (2003) Evaluation of modeling techniques based on models of learning. Commun ACM 46(10):79–84

    Article  Google Scholar 

  17. Jarvenpaa S, Machesky J (1989) Data analysis and learning: an experimental study of data modeling tools. Int J Man Mach Stud 31:367–391

    Article  Google Scholar 

  18. Johnson J, Boucher KD, Connors K, Robinson J (2001) The criteria for success. Softw Mag 21(1):s3–s8

    Google Scholar 

  19. Kim YG, March S (1995) Comparing data modeling formalisms. Commun ACM 38(4):103–115

    Article  Google Scholar 

  20. Kim J, Hahn J, Hahn H (2000) How do we understand a system with (so) many diagrams? Cognitive integration processes in diagrammatic reasoning. Info Syst Res 11(3):284–303

    Article  Google Scholar 

  21. Kung CH, Solvberg A (1986) Activity modelling and behaviour modelling. In: Olle TW, Sol HG, Verrjin-Stuart AA (eds) Proceedings of the IFIP WG 8.1 working conference on comparative review of information systems design methodologies: improving the practice. North-Holland, Amsterdam, pp 145–171

    Google Scholar 

  22. Larkin J, Simon HA (1987) When a diagram is (sometimes) worth ten thousand words. Cogn Sci 11(1):65–99

    Google Scholar 

  23. Mayer RE (1989) Models for understanding. Rev Educ Res 59(1):43–64

    Google Scholar 

  24. Mayer RE (2001) Multimedia learning. Cambridge University Press, New York

    Google Scholar 

  25. Moore GC, I (1991) Development of an instrument to measure perceptions of adopting an information technology innovation. Inf Syst Res 2(3):192–222

    MATH  Google Scholar 

  26. Morris MG, Spier C, Hoffer JA (1999) An examination of procedural and object-oriented system analysis methods: does prior experience help or hinder performance. Decis Sci 30(1):107–136

    Google Scholar 

  27. Mylopoulos J (1992) Conceptual modeling and telos. In: Loucopoulos P, Zicari R (eds) Conceptual modeling, databases, and case: an integrated view of information systems development, chap 2. Wiley, New York, pp 49–68

  28. Norman D (1986) Cognitive engineering. In: Norman D, Draper S (eds) User centered design: new perspectives on human computer interaction. Lawrence Erlbaum Associates, Hillsdale, pp 31–61

    Google Scholar 

  29. Nosek J, Ahrens J (1986) An experiment to test user validation of requirements: data flow diagrams vs. task oriented menus. Int J Man Mach Stud 25:675–684

    Google Scholar 

  30. Oei JLH, van Hemmen LJ, Falkenberg ED, Brinkkemper S (1992) The meta model hierarchy: a framework for information systems concepts and techniques. Technical Report No. 92-17, Department of Informatics, Faculty of Mathematics and Informatics, Katholieke Universiteir, Nijmegen, pp 1–30

    Google Scholar 

  31. Ramsey R, Atwood M, Van Doren J (1993) Flowcharts versus program design languages: an experimental comparison. Commun ACM 26(6):445–449

    Article  Google Scholar 

  32. Siau KL (1996) Empirical studies in information modeling. PhD Thesis, University of British Columbia

  33. Vessey I, Conger S (1994) Requirements specification: learning object, process, and data methodologies. Commun ACM 37(5):102–113

    Article  Google Scholar 

  34. Wand Y, Weber R (1993) On the ontological expressiveness of information systems analysis and design grammars. J Inf Syst 3:217–237

    Google Scholar 

  35. Wand Y, Weber R (1995) On the deep structure of information systems. Inf Syst J 5:203–223

    Google Scholar 

  36. Wand Y, Weber R (2002) Information systems and conceptual modeling a research agenda. Inf Syst Res 13(4):203–223

    Google Scholar 

  37. Wang S (1996) Two MIS analysis methods: an experimental comparison. J Educ Bus Jan/Feb:136–141

    Google Scholar 

  38. Yadav S, Bravoco R, Chatfield A, Rajkumar T (1988) Comparison of analysis techniques for information requirements determination. Commun ACM 31(9):1090–1097

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by grants from the Social Sciences and Humanities and Natural Sciences and Engineering Research Councils of Canada.

Author information

Authors and Affiliations

  1. Faculty of Business Administration, Simon Fraser University, 8888 University Drive, Burnaby, BC, Canada, V5A 1S6

    Andrew Gemino

  2. Sauder School of Business, University of British Columbia, Vancouver, BC, Canada, V6T 1Z2

    Yair Wand

Authors
  1. Andrew Gemino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yair Wand
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew Gemino.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gemino, A., Wand, Y. A framework for empirical evaluation of conceptual modeling techniques. Requirements Eng 9, 248–260 (2004). https://doi.org/10.1007/s00766-004-0204-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-004-0204-6

Keywords

Search

Navigation