Free/Libre Open Source Quality Models

From Maisqual Wiki

Jump to: navigation, search

Free/Libre Open Source Quality Models - a comparison between two approaches

Kersten Haaland - UNU-MERIT, Maastricht, the Netherland.

Arne-Kristian Groven, Department of applied research in ifnormation technology, Norsk Regnesentral, Solo, Norway.

Ruediger Glott, UNU-MERIT, Maastricht, The Netherland.

Annat Tannenberg, Freecode AS, Oslo, Norway.



This paper presents a comparison of a first generation software quality model (OpenBRR) and a second generation software quality model (QualOSS) by applying them to the case of Asterisk, a FLOSS implentation of a telephone private branch exchange (PBX, VoIP). Examining the trends and evolution of software quality models and identifying differences in the approaches and assessment outcomes, the results indicate significant progress in the development of open source quality models. However, it appears that tool support, which characterizes the second generation quality model, does not achieve to fully support the need for human interference. Therefore, future FLOSS quality models might call either for an even stronger reliance on tools and the abandonment of the human factor, or for an effective integration of both; the human factor and tools support. Effective, in this regard, means that the subjectivity aligned with human interference becomes marginal. At any rate, it appears that there is still a way to go.


  1. Introduction
  2. Background and Context
    1. Traditional software quality models
    2. First generation FLOSS quality models
    3. Second generation FLOSS quality models
  3. The measurement methodologies applied on Asterisk VoIP software
    1. The OpenBRR methodology
    2. The QualOSS methodology
  4. Presenting and comparing the measurement ressults
    1. OpenBRR measurement results
    2. QualOSS measurement results
    3. Discussion of results
  5. Comparing the assessment methods
  6. Conclusions


 author = {Ruediger Glott and Arne-Kristian Groven and Kirsten Haaland and Anna Tannenberg},
 title = {Quality Models for Free/Libre Open Source Software???Towards the ???Silver Bullet????},
 journal ={Software Engineering and Advanced Applications, Euromicro Conference},
 volume = {0},
 isbn = {978-0-7695-4170-9},
 year = {2010},
 pages = {439-446},
 doi = {},
 publisher = {IEEE Computer Society},
 address = {Los Alamitos, CA, USA},



Questions addressed in this study are:

  • What trends can be observed in the eovlution of FLOSS quality models?
  • How do Asterisk can be assessed regarding both models?
  • Conclusions between quality and future of FLOSS quality models.

Background and Context

Traditional software quality models

There are different views on quality, as investigated in [1] and [2]:

  • User view on quality: focusing in users' needs. e.g. reliability, performance, efficiency, maintainability, usability, etc.
  • Manufacturing view on quality: conformance to specifications and capability to produce the software according to the software process. e.g. defect count and staff effort rework cost.
  • Product view on quality: the characteristics of the produt are defined by the characteristics of its subparts. e.g. complexity measures, design and code measures.

Over the last 4 decades, 2 main directions can be distinguished:

  • Quality management [3], [4], [5], [6], which provides a more qualitative approach.
  • Quality models, which provide a more quantitative view on quality. Two subdirections can be identified:
    • Adherence to a process or capability level, e.g. CMM, CMMi, ISO/IEC 15504, ISO/IEC 9000.
    • Assessment through a set of metrics and attributes to quantify quality, e.g. [7], [8], [9] and ISO/IEC 9126.

McCall identified the following characteristics:

  • Product revision
    • Maintainability
    • Flexibility
    • Testability
  • Product transition
    • Portability
    • Reusability
    • Interoperability
  • Product operation
    • Correctness
    • Reliability
    • Efficiency
    • Integrity
    • Usability

First generation of FLOSS quality models

Traditional quality models lack some dimensions when comping to FLOSS projects, most notably the community.

Some FLOSS quality models:

  • Open Source Maturity Model - OSMM Capgemini[10]
  • Open Source Maturity Model - OSMM Navica
  • Qualification and Selection of Open Source Software - QSOS Atos Origin[11]
  • Open Business Readiness Rating - OpenBRR[12].

For all models data gathering and evaluation is manual work.

Second generation of FLOSS quality models

Draw on previous experience (traditional quality models and first generation quality models), and have a more extensive tool support.

  • QualOSS
  • QualiPSo - OpenSource Maturity Model, OMM, still in development.
  • SQO-OSS:

Measurement methodology applied on Asterisk VoIP software

OpenBRR and QualOSS have been applied to Asterisk.

OpenBRR methodology

There are 7 categories (characteristics), to be weighted manually, representing 27 metrics. There are weights for categories and for metrics inside each category.

The QualOSS methodology

QualOSS[13] assesses the quality of FLOSS endeavour from a business point of view.

Qualoss assessment.png

QualOSS relies on tools (cvsanaly[14] and bicho[15]).Measurement results are documented in a spreadsheet that is filled both automatically and manually.

Like with OpenBRR, basis for thresholds is more experience-based than statistical.

Metrics are the following:

  • Maintainability (15 indicators total)
    • percentage of unassigned issues,
    • percentage of comments,
    • evolution of the number of lines of code between successive releases,
    • average number of patchs per issue,
    • measures of cyclomatic complexity and efferent coupling, etc.
  • Reliability
    • stability evolution
    • importance of corrections
    • violations of code conventions
  • Security metrics are all based on the National Vulnerabilities Database
  • Documentation (6 indicators)
    • types of available documentation (manuals) and the information that is provided in these documents
    • knowledge generally available (forums, mailing lists, etc.)
  • Tests (8 indicators)
    • test report availability
    • unit and systems test coverage adequacy
    • ease of testing
    • likelihood of future test reports
    • availability of automated tests

Comparing the assessment methods

Quality model methods provide more objective measurability and are simpler to use than qualitative approaches. But they have a reduced notion of quality to a few relatively simple and static attributes, and represent a leaner and narrower perspective on quality.


Key trends between both models (1st and 2nd generation models) are:

  • from manual and descriptive to more automated and analytical models,
  • a few metrics to hundreds of them.

Whenever the tool support is working as intended, the QualOSS is a source of more insight compared to a method like OpenBRR, as illustrated in some of the results presented in this article. On the risky side of QualOSS is the pre-determined definition of what is a good trend or score constitutes.

Overall, it appears that human expertise, especially knowledge of context conditions and development trends with a FLOSS endeavour, is decisive for the usability of both quality models. OpenBRR relies on this input by design. QualOSS tried to largely eliminate such direct input on the measurement process but, occasionally, seems to rely on it when tools are not available or when the results of the assessment must be interpreted.


  1. Garvin, D.A. "What does 'Product Quality' really means?", Sloan Management Review, no. 1, pp. 25-43, 1984.
  2. Kitchenham, B. and Pfleeger, S.L., "Software Quality: the elusive target", IEEE Software, no. 1, pp. 12-21, 1996.
  3. Crosby, P.B., "Quality is free: the art of making quality certain", new York, McGraw-Hill, 1979.
  4. Deming, W.E. Out of the crisis: quality, productivity and competitive position, Cambridge Univ. Press, 1988.
  5. Huggins, L.P., Total quality management and the contributions of A.V. Feigenbaum, Journal of Management History, Vol 4, 1, pp. 60-67, 1998.
  6. Weinberg, G.M., "Quality Software Management, Vol. 3: Congruent Action" Dorset House Publishing Company, Inc., September 1994.
  7. McCall, J.A., Richards, P.K., and Walters, G.F., 'Factors in Software Quality", Nat'I Tech. Information Service, Vol. 1, 2, and 3, 1977.
  8. Boehm, Barry W., Brown, J.R., and Lipow, M.: "Quantitative evaluation of software quality", International Conference on Software Engineering, Proceedings of the 2nd International Conference on Software Engineering, 1976.
  9. Boehm, B.W., Brown, J.R., Kaspar, H., Lipow, M., McLeod, G., Merritt, M.: "Characteristics of Software Quality", North Holland, 1978.
  10. Capgemini's Open Source Maturity Model (OSMM) assessment available at: .
  13. Deprez J.-C., Haaland K., and Kamseu F., QualOSS Methodology & QualOSS assessment methods. QualOSS deliverable D4.1. , 2008.
Personal tools