SQCM v2.1: a coverage number needs a measurement story

A coverage percentage looks simple. Ninety percent appears better than eighty. But the number leaves several questions unanswered: which tests ran, which files were instrumented, which revision was checked, and whether the entire suite completed successfully.

Those questions are the center of SQCM v2.1, the new edition of the ScaleQuality Code Maturity Model. The five domains remain: Security, Reliability, Maintainability, AI Code Durability, and Supply Chain. The revision explains more precisely how evidence becomes a score, with particular attention to measured test coverage and to comparisons over time.

Improvements above the old boundary now count

The current scoring calibration uses valid measured coverage as a continuous input to Reliability and Maintainability. It no longer substitutes the same high-coverage category for distinct measured percentages above a bucket boundary.

In a controlled regression fixture, holding the other inputs fixed, coverage of 70%, 80%, 90.5%, and 100% produced Reliability scores of 84, 86, 89, and 91. That is the intended behavior: the instrument retains a distinction that the older categorical path could lose.

These are synthetic examples, not a conversion table for every repository. Findings and engineering practices still matter. Rounding can hide a small change. A critical-exposure cap can hold the overall score steady even while the coverage-dependent domains improve. More coverage also does not prove that assertions are effective or that important behavior is tested.

The publication is paper v2.1. The calculation described here is score v2.2. Those numbers identify different things: a document and a calibration. An older run keeps the version that produced it.

Four paths can lead to test evidence

A connected diagnosis can check out a pinned revision, run supported tests, and read a fresh report. A CI client can instead supply a report generated in its own environment. An agent comparing a proposed change can use a stricter pair of before-and-after receipts. When runtime coverage is unavailable, the diagnosis can still examine test structure and practices.

These paths should not be confused. A report supplied by a pipeline is not independent attestation of that pipeline. Structural test evidence is not a measured percentage of executed lines. And the general coverage runner can accept a report from passing tests when part of a suite failed, provided that partial status accompanies the result. A complete successful suite and a partial run do not establish the same thing.

The paper describes each path and its limits, including a practical detail for projects with several repositories: the connected runtime path examined in this revision targets the primary successfully analyzed repository. A project-level coverage percentage should not automatically be read as a line-weighted measurement of every connected repository.

The same score can describe different scopes

Our previous paper emphasized measuring an artifact where it actually exists. A container recipe is not the built image; a repository-only scan cannot establish everything about what gets deployed.

That principle still holds. One claim accompanying it was too broad, however. We wrote that supplying image evidence could only lower a score and that withholding evidence could not improve it. The current composition does not support that general guarantee.

If Supply Chain was previously unavailable, clean image evidence can make it measurable and change the set of domains in the overall calculation. In a controlled example, the overall score moved from 95 to 96 with that new evidence. Conversely, leaving adverse evidence out can make a result look better. The revised paper explicitly withdraws the earlier guarantee.

A useful comparison therefore needs more than two numbers. It needs the scoring version, repository and artifact scope, coverage origin, exclusions, and analysis depth. A different number can mean better code, different evidence, or a different measuring instrument.

What we verified, and what remains open

The revision is grounded in an implementation audit and 194 passing tests across 19 focused internal suites. They check contracts such as report handling, revision consistency, composition, persistence, and controlled examples. They do not prove that SQCM predicts incident rates or that a scanner finds every vulnerability.

We also kept the earlier performance observations in their proper context. The three fast CI measurements published in v2.0 were historical runs on one commit under stated hardware conditions. They were not repeated for this edition, and they are not a promise that a connected diagnosis can install dependencies and run an arbitrary test suite in five seconds.

Some comparison restrictions are enforced in dedicated evolution reports but not in every summary chart. The paper says so. Publishing the method should make those boundaries easier to see, not conceal them behind a maturity label.

Read the revised methodology

The research page contains the consolidated paper, an English PDF, its Portuguese translation, the version DOI, and a supplementary validation note. Earlier editions remain available under their own permanent records.

The purpose of the revision is practical: when a team sees a coverage percentage or a maturity improvement, it should be able to understand what was measured and what would make the comparison meaningful. A number becomes useful when the evidence behind it can be examined.

Cite this edition: SQCM v2.1 · DOI 10.5281/zenodo.22781322.