Empirical Research Methods in Requirements Engineering

The talk:

This is a full day tutorial I gave at the 15th IEEE International Conference on Requirements Engineering (RE'07) , in New Delhi, India, October 2007.

The slides for the talk.

Book chapter: Easterbrook, S. M., Singer, J., Storey, M, and Damian, D. Selecting Empirical Methods for Software Engineering Research. To appear in F. Shull and J. Singer (eds) "Guide to Advanced Empirical Software Engineering", Springer, 2007.

Abstract

This aim of this tutorial is to introduce the range of empirical methods used in RE research, to explain how to choose appropriate methods for particular research projects, and how to ensure the methods are used with a high level of scientific rigour. Using a blend of lecture, discussion, and practical exercises, the tutorial will provide RE researchers and practitioners with a foundation for planning, conducting and reviewing and empirical studies. The tutorial will cover all of the principal methods applicable to RE: controlled experiments, quasi-experiments, case studies, survey research, ethnographies, and action research and will relate these methods to relevant meta-theories in the philosophy and sociology of science. Special emphasis will be given to the challenges of investigating the social and cognitive processes that occur when people analyze human activities in order to understand and manage complex, evolving requirements. The tutorial will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing. The tutorial is relevant for researchers, who will be able to conduct and write more credible empirical studies; for reviewers who will be able to provide qualified judgments of papers; and for practitioners, who will be able to more effectively interpret published results within the context of their own organizations.

Introduction

Despite the widespread interest in empirical research in software engineering over the past decade or so, software engineering still lacks a coherent methodological foundation. There is little guidance available on which research methods are suitable, and how to choose between them. Many researchers select inappropriate methods due to a lack of understanding of the goals underlying the method employed, or due to a lack of knowledge about alternative methods. The problem is particularly acute in requirements engineering, where research crosses social and technological boundaries. Many important research questions in RE concern the social and cognitive processes that occur when people analyze human activities in order to understand and manage complex, evolving requirements.
The aim of this tutorial is to cover the key questions to consider in selecting empirical methods, from the philosophical issues about the nature of scientific knowledge, to the practical considerations for applying empirical methods. We identify and characterize the key empirical methods applicable to RE research, and explain the strengths and weaknesses of each. Our goal is to show how methods can be selected and combined to produce a valid research strategy in response to a well-defined research question.

Outline Program

The tutorial starts with basic issues in empirical studies, moves on a detailed look at each of the principal methods, and concludes with an examination of publication. The curriculum includes the following topics.

• Research Methodology
• Approaches for Empirical Studies
• Strategies for RE Research
• Tour of the major empirical methods
• Planning Empirical Research
• Different type of research question
• The role of empirical theories
• Validation strategies
• Replication and empirical induction
• Research ethics
• Philosophical Foundations
• What counts as empirical truth?
• 4 key philosophical stances: post-positivism, constructivism, critical theory, pragmatism
• Experiments
• Basic Experimental Design
• Quasi-Experiments
• Statistical Generalization
• Strengths and limitations
• Case Studies
• Uses of Case Study Research
• Designing the study
• Analytical Generalization
• Strengths and limitations
• Survey Research
• When to use survey research
• Sampling Strategies
• Dealing with bias
• Strengths and limitations
• Ethnographies
• The idea of members’ categories
• Participant Observation
• Analyzing qualitative data (incl. ideas from grounded theory)
• Strengths and limitations
• Action Research
• Engaging with Practitioners
• Finding authentic problems and authentic interventions
• Identifying learning outcomes
• Strengths and limitations
• Mixed methods research
• Strategies for a research programme
• Data collection and Analysis
• Sources of data
• Measurement theory
• Selecting statistical models
• Coding and analyzing qualitative data
• Presenting the Results
• Explicating the design
• Dealing with threats to validity
• Making datasets available

Intended Audience

The intended audience for this tutorial includes students, researchers, and practicing requirements engineers, that is, both the producers and consumers of empirical studies. This tutorial will help requirements engineers identify and avoid common mistakes by giving them a solid grounding in the fundamentals of empirical research methods. The tutorial will be structured in such a way as to emphasize the principles and design of empirical studies.

• For researchers, this tutorial will provide a starting point for learning how to conduct empirical studies. They will learn how to select among the available methods, plan a study, and assess the practical challenges.
• For reviewers, the tutorial will provide guidance on how to judge the quality and validity of reported empirical studies. They will be able to use the criteria presented in this tutorial to assess whether research papers presenting empirical results are suitable for publication, allowing them to raise the quality of publications and give appropriate feedback to authors.
• For practitioners, the tutorial will provide a better awareness of how to interpret the claims made by researchers about new requirements engineering methods and tools. Practitioners will also gain deeper insights into the roles they can play in designing and conducting empirical studies in collaborative research projects. As well, they will read published studies more effectively and be better able to identify results suitable for use in their workplace.

Further Reading

An extensive list of further readings can be found on the website for my graduate course, Empirical Research Methods in Software Engineering.