Empirical Studies of Software Engineering: A Roadmap

From Maisqual Wiki

Jump to: navigation, search

Dewayne E. Perry, Electrical and Computer Engineering, University of Texas at Austin.

Adam A. Porter, Computer Science, University of Maryland.

Lawrence G. Votta, Motorola



In this article we summarize the strengths and weaknesses of empirical research in software engineering. We argue that in order to improve the current situation we must create better studies and draw more credible interpretations from them. We finally present a roadmap for this improvement, which includes a general structure for software empirical studies and concrete steps for achieving these goals: designing better studies, collecting data more effectively, and involving others in our empirical enterprises.

Key research Pointers

  • Empirical study play a fundamental role in modern science, helping us understand how and why things work, and allowing us to use this understanding to materially alter our world.
  • Defining and executing studies that change how software development is done is the greatest challenge facing empirical researchers.
  • The key to meeting this challenge lies in understanding what empirical studies really are and how they can be most effectively used - not in new techniques or more intricate statistics.
  • If we want empirical studies to improve software engineering research and practice, then we need to create better studies and we need to draw more credible conclusions from them.
  • Concrete steps we can take today include: designing better studies, collecting data more effectively, and involving others in our empirical enterprises.


 author = {Perry, Dewayne E. and Porter, Adam A. and Votta, Lawrence G.},
 title = {Empirical studies of software engineering: a roadmap},
 booktitle = {Proceedings of the Conference on The Future of Software Engineering},
 series = {ICSE '00},
 year = {2000},
 isbn = {1-58113-253-0},
 location = {Limerick, Ireland},
 pages = {345--355},
 numpages = {11},
 url = {http://doi.acm.org/10.1145/336512.336586},
 doi = {http://doi.acm.org/10.1145/336512.336586},
 acmid = {336586},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {empirical studies, software engineering},


  1. Introduction
  2. Why empirical studies?
  3. The state of empirical research
    1. Current strengths
    2. Systemic problems
  4. Future Challenges to empirical studies
    1. Creating Better Empirical Studies
    2. Credible interpretations
    3. Designing an Empirical Study
  5. The structure of an empirical study
    1. Research Context
    2. Hypothesis
    3. Study Design
    4. Threats to Validity
    5. Data Analysis and Presentation
    6. Results and Conclusions
  6. Concrete steps
    1. Designing the studies
    2. Getting the data
    3. Involving others
  7. Summary



An empirical study is a test that compares what we believe to what we observe. They are a strong fundamentals of many sciences, but are not really used in Software Engineering.

Using studies to conduct changes in a development group faces 2 difficulties: designing and conducting individual studies and - even worse - conceptualising and organising a body of work that could serve as a basis for the move[1][2][3].

Why empirical studies?

Development processes widely vary from one project to another. Because we don't know the fundamental mechanisms that drive costs and benefits, tools and methods can't be precise, and any decision is essentially a random act.

Empirical studies address this problem and give better confidence in the knowledge we have of situations.

Empirical studies learn from the difference between theory and reality. The steps are the following:

  • formulate an hypothesis,
  • observe a situation,
  • abstract observation into data,
  • analyse data,
  • draw conclusions with respect to the tested question.

Benefits of empirical studies, if the previous procedure is respected, are:

  • knowledge is encoded more rapidly,
  • low-payoff or erroneous research ideas are discarded quickly,
  • high-payoff areas are recognized and correctly valued, and,
  • important practical issues are considered.

The state of empirical research

Current Strengths

  • Growing interest and confidence in empirical studies,
  • empirical validation is commonly accepted for testing,
  • more and more conferences and articles are published[4],
  • researchers are more aware of validating their theories within practical contexts[5][6].

Systemic problems

There are difficulties and misconceptions in the implementation of studies, which often leads to decreased credibility.

Data is not enough. Interpreting them is fine, but understanding why they are as they are is better.

Studies should answer questions, not only present observations.

Future challenges to Empirical Studies

Creating better empirical studies

  • Define clear goals.
  • Design empirical studies more effectively.
  • Maximise the information we get out of the empirical study.

Following principles should be followed:

  • Causal: if A then B.
  • Actionable: I can control A.
  • General: the property is true for a wide range of parameters.

If the topic is too large, do several, complementary studies. Use each study to refine our hypothesis.

Share and recheck your work with others. Make it easy for them to reproduce the results.

Credible Interpretation

Ensure validity. There are two types of variables: dependent variable (outputs we want to be modified) and independent variables (that define the study's setting), and three types of validity: internal, external, and construct validity:

  • Construct validity means that the independent and dependent variables accurately model the abstract hypotheses.
  • Internal validity means that changes in the dependent variables can be safely attributed to changes in the independent variables.
  • External validity means that the study's results generalise to settings outside the study.

Good management of costs of empirical studies allows to better conduct them, or do more of them. Choose the good precision, accuracy, measurement means for the intended purpose.

Structure of an empirical studies

An empirical study should have the following components:

  • research context,
  • hypotheses,
  • experimental design,
  • threats to validity,
  • data analysis and presentation, and
  • results and conclusions.

Concrete steps

Designing the study

  • Ultimately, the most important thing researchers can do is to ask insightful questions.
  • If the issue addressed is too big or too complex, try to conduct a family of studies instead, and address complementary, more simple parts.
  • Build partnerships: discuss the question, ask different views..
  • In vivo, long-running experiments are meaningful. These can't always be ran, but bring meaningful constraints. Respect the rights of the subjects[7].

Getting the data

  • Retrospective artefacts analysis: VCS could be used to help understand work patterns.
  • Simulation and Mathematical Modeling: e.g. [8]

Involving Others

No single study gives unequivocal results. It is necessary that the research community integrates and compares studies that address common hypotheses. There are three approaches to this:

  • Conduct a literature review, qualitatively summarise results, and manually synthesise them.
  • Meta-analysis: first try to reconcile the primary experiments, and then re-analyse the studies with the agreed upon definitions[9].
  • Some ad hoc approaches mix both.

Researchers should be trained to conduct good empirical studies. Students should follow a course that:

  • Show experiments can be used to evaluate hypotheses concerning open research issues,
  • Teach students to design and conduct experiments to evaluate their own research, and
  • Teach basic statistical procedures for collecting and analysing data from their own experiments.


This article discussed the important issues that must be addressed in creating a rigorous and credible empirical discipline for software engineering.

The three goals proposed to improve empirical studies are:

  • designing better studies,
  • getting the data and
  • involving others in our empirical enterprises.


  1. N. Fenton, S.L. Pfleeger, and R. Glass, Science and Substance: a Challenge for Software Engineers. IEEE Software, 1994. 11(4), p. 86-95.
  2. V. Basili, Editorial. Empirical Software Engineering Journal, 1996. 1(2).
  3. P.M. Johnson, Project LEAP: Lightweight, Empirical, Anti-measurement dysfunction, and Portable Software Developer Improvement, in Department of Information and Computer Sciences. 1997, University of Hawaii, Honolulu.
  4. D. Pregibon, et al., Statistical Software Engineering, 1996, National Academy of Sciences: Washington, D.C.
  5. W.F. Tichy, P. Lukowicz, L. Prechelt, and E.A. Heinz, Experimental Evaluation in Computer Science: A Quantitative study. Journal of Systems and Softwares, 1995. 28(1): p. 9-18.
  6. M.V. Zelkowitz and D. Wallace, Experimental validation in Software technology. Information and Software Technology, 1997. 39(11): p. 735-744.
  7. C.M. Judd, E.R. Smith, and L.H. Kidder, Research Methods in Social Relations. 1991, Fort Worth, TX: Holt, Rinehart and Winston, Inc.
  8. J.A. Solheim and J.H. Rowland, An Empirical Study of Testing and Integration Strategies Using Artificial Software Systems. IEEE Transactions on Software Engineering, 1993. 19(10):p. 941-949.
  9. G.V. Glass, B. McGaw, and M.L. Smith, Meta-analysis in social research. 1981, Beverly Hills, CA: Sage.


Online version on the author's website:

Access in ACM library:

Personal tools