ISO/IEC 9126-3 Internal Quality Measures: Are They Still Useful

From Maisqual Wiki

Jump to: navigation, search

Witold Suryn, Department of Software and IT Engineering, Ecole de Technologie Supérieure, Montréal, Québec.

Blanca Gil, Department of Software and IT Engineering, Ecole de Technologie Supérieure, Montréal, Québec.


Contents


Abstract

This paper analyzes the applicability and relevance of internal quality measures presented and published in ISO/IEC 9126-3:2003 standard for software quality measurement and evaluation. As a result, the recommendations for modifications to the current set of ISO/IEC 9126-3 set of measures as well as to the existing quality model have been proposed.


Contents

  1. Introduction
  2. Methodology
  3. Analysis
    1. Definition of axioms
    2. Identification of basic hypotheses
  4. Improvement options
  5. Conclusions


Bibtex

@article{suryn:2005,
 title={ISO/IEC 9126-3 Internal Quality Measures: Are They Still Useful}, 
 url={http://profs.etsmtl.ca/wsuryn/research/SQE-Publ/iso9126-3%20are%20they%20useful-review%20%28submitted%20HCII2005%29.pdf}, 
 journal={Proceedings of HCI International},  
 author={Suryn, W. and Gil, B.}, 
 year={2005}, 
}


Notes

This paper investigates the definition, identification, choice, documentation, evaluation, traceability, predictive capacity and practical uses of internal quality measures.

Methodology

The following four phases have been decided for the standard analysis:

  • Phase 1: Identification and definition of the axioms.
  • Phase 2: Construction of the set of basic hypotheses.
  • Phase 3: Construction of the analysis engine.
  • Phase 4: Analysis and recommendation for improvement of measures.

Analysis

Definition of axioms

Axioms are fundamentals analysis criteria shared by all measures. All axioms have been grouped into three main categories:

  • Category I: Scope, lifecycle and tracking (estimation and prediction).
  • Category II: Applicability and relevance.
  • Category III: Measurement - quality and nature of measurement.


Category I: Scope, lifecycle and tracking (estimation and prediction)

Axiom 1
The metrics listed in this International Technical Report are not intended to be an exhaustive set (P1.vi).
Axiom 2
Th e views of internal quality, external quality and quality in use change during the software lifecycle (P1.3).
Axiom 3
The fundamental nature of the software product quality represented by internal quality remains unchanged unless redesigned (P1.5).
Axiom 4
The current state of the art does not provide all the support necessary for the purposes of prediction (P1.5).
Axiom 5
[Internal metrics] can be used to predict values of the external metrics (P1.6).
Axiom 6
The correlation between internal attributes and external measures is never perfect and the effect [...] will be determined by experience and will depend on the particular context [...] (P1.14).
Axiom 7
It is often difficult to design a rigorous theoretical model that provides a strong relationship between internal metrics and external metrics (P3.3) – Axiom also in ISO/IEC 9126-1 (P1.15).
Axiom 8
Internal metrics should also have predictive validity, which means that they should correlate with some desired external measures (P1.17)
Axiom 9
The internal metrics standard shows the relationship between external and internal metrics (P5.8).
Axiom 10
Internal metrics are of most interest during the development process (P5.12).
Axiom 11
Internal metrics are of little value unless there is evidence that they are related to external quality


Category II: Applicability and relevance

Axiom 21
This report is applicable to any kind of software product, although each of the metrics is not always applicable to every kind of software product (P3.vi).
Axiom 22
Quality requirements cannot be completely defined before the beginning of design (P1.4).
Axiom 23
The hierarchy [characteristics and sub-characteristics] is not perfect, as some attributes may contribute to more than one sub-characteristic (P1.14).
Axiom 24
The internal metrics may be applied to a non-executable software product during its development stages (such as request for proposal, requirements, definition, design specification or source code) (3) – Axiom also in ISO/IEC 9126-1 (P3.15).
Axiom 25
[...] internal metrics [...] measure intrinsic properties including those, which can be derived from simulated behaviour (P1.15).
Axiom 26
The measurements of internal metrics use number or frequencies of software composition elements which appear for example on source code statements, the control graph, data flow and state transition representations (P1.15).
Axiom 27
Documentation can also be evaluated using internal metrics (P1.15).
Axiom 28
The metrics listed in this [metric table] are not intended to be an exhaustive set and may not have been validated (P3.4).
Axiom 29
Additional specific metrics for particular purposes are provided in other related documents, such as functional size measurements or precise time efficiency measurement (P3.4).
Axiom 30
Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself (P5.4).
Axiom 31
Modularity and traceability are examples of internal attributes, which can be measured (P5.12).
Axiom 32
ISO/IEC 12207 SLCP Reference: identifies software lifecycle process(es) where the metric is applicable (P3.4).
Axiom 33
Target audience: Identifies the user(s) of the measurement results (P3.4).


Category III: Measurement

Axiom 40
Measurements should be objective, empirical using a valid scale, and reproducible (P1.16).
Axiom 41
Internal measure [...] is not derived from measures of the behaviour of the system of which it is a part (P3.4).
Axiom 42
Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself (P3.4).

Identification of basic hypotheses

The following hypotheses are proposed:

  • Hypothesis 1: Reducing the applicability of measures to software lifecycle supporting processes severely minimizes the possibilities of developing a quality software product
  • Hypothesis 2: The selection of artefacts to be measured could affect the applicability of measures
  • Hypothesis 3: Current classification (relationship to quality model) of measures could affect their applicability
  • Hypothesis 4: Properties of artefacts to be measured could affect the applicability of measures
  • Hypothesis 5: Current inconsistencies in prediction capacity could affect the applicability of measures
  • Hypothesis 6: The actual set of internal quality measures may require modifications in order to improve the applicability of the standard.


Hypothesis 1: Reducing the applicability of measures to software lifecycle supporting processes severely minimizes the possibilities of developing a quality software product

ISO/IEC 9126-3 uses the ISO/IEC 12207 generic model as a referenced model of software lifecycle. ISO/IEC 12207 proposes the following generic lifecycle process model:

  • Supporting lifecycle process
    • 6.1 Documentation process
    • 6.2. Configuration management process
    • 6.4. Verification process
    • 6.5. Validation process
    • 6.6. Joint review process
  • Development process
    • 5.3.4. Software requirements analysis
    • 5.3.5. Software architectural design
    • 5.3.6. Software detailed design
    • 5.3.7. Software coding and testing

ISO/IEC 9126-3 does not provide full lifecycle coverage, although information should be available at all levels of the development, as stated in ISO/IEC 9126-1 Annex A.1.2 page 15: Internal metrics can be applied to a non-executable software product (such as a specification or source code) during designing and coding. When developing a software product the intermediate products should be evaluated using internal metrics which measure intrinsic properties, including those which can be derived from simulated behaviour.

Thus measures from development process (ISO/IEC 12207) should be considered along with the supporting lifecyce development process.


Hypothesis 2: The selection of artefacts to be measured could affect the applicability of measures

According to ISO/IEC 9126-3, input to measurements are Review Report, Requirements Specification, Design and Source Code. But:

  • These lack preciseness and completeness, if existing.
  • Usability pure internal metrics strongly depends on the precise indication of required artefacts but their indication or a reference to software lifecycle process is missing.

As recommendation, a new list of possible artefacts is proposed:

  • If nothing else is indicated, the generic input to measurement could be chosen from:
    • Joint Review Report,
    • Validated Requirement Specification Documents,
    • Standards and Regulations,
    • Verification Report.
  • If nothing else is indicated or none specific product is pointed, products to be measured could be chosen from:
    • Functional Model,
    • Data Model,
    • Event Model,
    • Hardware Model,
    • Source code,
    • User Documentation,
    • Technical Documentation,
    • Use Case Diagram,
    • Activity Diagram,
    • State Diagram,
    • Sequence Diagram,
    • Class/Object Diagram,
    • Component Diagram,
    • Deployement Diagram.


Hypothesis 3: Current classification (relationship to quality model) of measures could affect their applicability and Hypothesis 4: Properties of artefacts to be measured could affect the applicability of measures

Pure internal metrics are unquestionnably important to software quality, but are mentionned as being "informative" in Annex E and do not appear in the quality model. An enhanced model with added characteristics named "Internal technical measures" is presented below:

ISO/IEC 9126-3 Internal Quality:

  • Functionnality
  • Usability
  • Maintainability
  • Reliability
  • Efficiency
  • Portability
  • Internal Quality Measures:
    • Development Standard Measures
      • Self-descriptiveness
      • Self-containedness
      • Pure reusability
      • Pure maintainability
    • Size
    • Complexity


Hypothesis 5: Current inconsistencies in prediction capacity could affect the applicability of measures

The prediction capacity of internal measurements is a fundamental applicability feature in ISO/IEC 9126. The following conclusions were drawn:

  • The nature of prediction is non-orthogonal, i.e. there are internal quality measures that are related to more than one external quality measure.
  • The number of internal quality measures that remain non-related (or, in other words, do not serve the predictability purpose) is relatively small (7 out of 70, or 10%).
  • The number of external quality measures that have no visible relation to internal quality measures if much greater (28 out of 112, or 25%).

All the three above observations indicate that the prediction mechanism between internal and external quality is not complete, leaving too many external and internal quality measures/attributes unrelated to each other.


Hypothesis 6: The actual set of internal quality measures may require modifications in order to improve the applicability of the standard.

The authors propose the following adds/modifications/deletions to the existing set of internal quality measures:


Candidates for inclusion:

  • Functionnality
    • Computational completeness
    • Precision accuracy
    • Data exchangeability completeness (data format based)
  • Efficiency
    • Estimated Software platform response time
    • Estimated Data management response time
    • Estimated Transmission response time
    • Estimated I/O Utilization size
  • Usability
    • Demonstration consistency
    • User Interface consistency
    • User Documentation consistency
    • Output Messages consistency
    • Help consistency
    • Help understandability
    • Completeness of training material
    • Default value availability
    • Operation procedure reduction
    • Consistency with others known systems
  • Reliability
    • Fault density
  • Portability
    • Internationalization
    • Ease of installation
  • Maintainability
    • Activity recording legibility
    • Modification complexity
    • Parameterized availability
    • Reuse utilization
    • Programing Style consistency
    • Frameworks uilization
    • Patterns utilization
    • Programs libraries utilization
    • Data stores / procedures utilization
    • Technical Documentation consistency
    • Technical Documentation understandability
    • Completeness of Technical Documentation


Candidates for deletion

  • Reliability
    • Incorrect operation avoidance
  • Efficiency
    • Turnaround time
    • Transmission Utilization


Candidates for modifications

  • Functionality
    • Functional adequacy
    • Functional implementation completeness
    • Computational accuracy
    • Access auditability
    • Data corruption prevention
    • Intersystem standard compliance
    • Precision
    • Data exchangeability
    • Interface consistency
  • Usability
    • Demonstration capability
  • Efficiency
    • Response time
    • Throughput time
    • I/O utilisation
    • I/O utilisation message density
    • Memory utilisation message density
  • Reliability
    • Fault detection
    • Test adequacy
    • Failure avoidance
    • Restoration effectiveness

Improvement Options

The improvement options proposed in this article can be divided in two main categories:

  • Changes to the ISO/IEC 9126 quality model -- these don't change the fundamental core of the model,
  • Adds/Modifications/Deletions in internal quality measures.

Conclusions

The changes proposed should be seen as maintenance of the norm to follow the quickly evolving world of software engineering.


See also

Papers:

Standards:

Personal tools