12 Steps to Useful Software Metrics

From Maisqual Wiki

Jump to: navigation, search

Linda Westfall, The Westfall Team.



12 Steps to Useful Software Metrics introduces the reader to a practical process for establishing and tailoring a software metrics program that focuses on goals and information needs. The process provides a practical, systematic, start-to-finish method of selecting, designing and implementing software metrics. It outlines a cookbook method that the reader can use to simplify the journey from software metrics in concept to delivered information.


    title={12 Steps to Useful Software Metrics}, 
    volume={57 Suppl 1}, 
    number={May 2006}, 
    journal={Proceedings of the Seventeenth Annual Pacific Northwest Software Quality Conference}, 
    publisher={The Westfall Team}, 
    author={Westfall, Linda and Road, Custer}, 


12 Steps to useful Software Metrics

What are software metrics?

Companies are using metrics to better understand, track, control and predict software projects, processes and products.

Software metrics are [Goodman-93]: "The continuous application of measurement-based techniques to the software development process and its products to supply meaningful and timely management information, together with the use of the techniques to improve that process and its products".

Software metrics provide:

  • Information for technical decisions
  • Information required by management

Some basic measurement theory

Fenton [1]: "Measurement is the process by which numbers or symbols are assigned to attributes of entities in the real world in such a way as to describe them according to clearly defined rules."

There are very few metrics that have a standardised definition or mapping system. Counter-examples include McCabe Cyclomatic Complexity or Function Point Counting standard from the Function Point User Group.

However their selection and definition and the consistent use of a mapping system within the organisation are critical.

Introduction to the twelve steps

Steps 1-4: define customers needs, select metrics that match them. Steps 5-10: designing, tailoring metrics: definition, models, counting criteria, benchmarks and objectives, reporting mechanisms. Steps 11-12: implementation issues: collection, minimising impact of human factors.

There has been a shift in software metrics so that the question is not "What should I measure?" has become "Why should I measure?" [2]

Step 1: Identify Metrics Customers

Person/people that will use the metric. Customers may include: functional management, project management, software engineers/programmers, tests managers/testers, specialists (marketing, software quality assurance, support, etc.), customers/users.

Step 2: Target goals

As Basili[3] stated in the goal/question/metric approach.

This step selects one or more measurable goals (e.g. on-time delivery, delivering the software with required level of quality/performance, etc.).

Step 3: Ask Questions

Define questions that need to be answered in order to ensure goals are obtained.

Step 4: Select metrics

Select the metrics that provide the information needed to answer these questions, answers that give objectives to the selected metrics.

An individual metrics performs one of four functions:

  • Understand software process, product, services.
  • Evaluate against established standards and goals.
  • Control resources and processes.
  • Predict attributes of software[4].

Step 5: Standardized definitions

Either use standardized definitions or create your own (but make them explicit and clear).

Step 6: Choose a Measurement Function

i.e. how you are going to calculate the metrics.

"If we try to include of all the elements that affect the attribute or characterise the entity, our model can become so complicated that it's useless. Being pragmatic means not trying to create the most comprehensive model."

Step 7: Establish a Measurement Method

Define Base Measures, Units, and how they are to be calculated.

Example: SLOC is well-known and widely accepted, but there is no industry-accepted standard on how to count lines of code.

Once selected, communicate it so others won't misunderstand or misuse it.

Step 8: Define Decision Criteria

According to the ISO 15939 standard, decision criteria are the "thresholds, targets, or patterns used to determine the need for action or further investigation, ot to describe the level of confidence in a given result.

In other words, you need decision criteria to obtain guidance that will help you interpret results.

  • Decision criteria for control type metrics usually take the form of thresholds, variances or control limits.
  • Evaluate type metrics (i.e. what good?) may be: "no more than x% failures, with 2/3 minor and 1/3 major".
  • For predict and evaluate metrics, it is the "level of confidence in a given result" part of the standard that applies.

Step 9: Define report mechanisms

This includes defining the report format (table, charts, etc.), data extraction and reporting cycle (how often data are extracted and the report generated), reporting mechanisms (the way the report is delivered (hard copy, email, published, etc.), distribution (who receives the report), and availability (restrictions on metrics access).

Step 10: Determine additional Qualifiers

A good metric is a generic metric; try to find other views on the metric.

Step 11: Collect Data

The owner (person who has direct access to the source of data) of the data is probably the best person for generating the data, because:

  • data is collected as it is being generated,
  • anomalies are more likely to be detected,
  • no duplicates in data recording.

Owners must agree to do the work, convinced of the importance and usefulness of collecting data. Management has to support the program by allowing resources and time.

A procedure has to be established and documented, people have to be trained. Data collection must be objective, unambiguous, convenient (a part of the process) and accessible.

Data collection has to be automated, for repeatability, for reducing human factor errors and for price.

Step 12: The people side of the metrics equation


  1. Norman E. Fenton, 1991, Software Metrics, A Rigorous Approach, Chapman & Hall, London.
  2. Paul Goodman, 1993, Practical Implementation of Software Metrics, McGraw Hill, London.
  3. V.R. Basili, H.D. Rombach, 1988, The TAME Project: Towards Improvement-Oriented Software Environments. In IEEE Transactions in Software Engineering 14(6).
  4. Watts Humphrey, 1989, Managing the Software Process, Addison-Wesley, Reading.
Personal tools