SQCM v2.0: a repository is not the whole artifact

A month ago we published the ScaleQuality Code Maturity Model (SQCM) v1.0: a model that scores a repository from observable evidence, with no questionnaires, in minutes.

Today we are publishing version 2.0. It keeps the construct unchanged and corrects the instrument in two places, and both corrections came from measuring rather than from designing.

A repository is not the whole artifact

A substantial part of what an application ships was written by someone else. It arrives through a package manifest, a transitive dependency of a transitive dependency, a base image, or an infrastructure template. A model that reads only first-party code evaluates only part of what gets delivered.

So v2.0 adds a fifth domain: Supply Chain. It is a domain and not a subcategory of Security, for two reasons that matter in practice.

It fails differently. A first-party weakness is authored: somebody wrote the vulnerable pattern. Supply-chain exposure is inherited: the code was correct when it was adopted and became vulnerable while nobody touched it. A repository can degrade in this domain with zero commits, which is exactly why a point-in-time assessment is not enough.

It is fixed differently. A first-party weakness is fixed by changing code. Supply-chain exposure is usually fixed by changing a version constraint, which is a different decision, taken by different people, carrying different risk.

One thing the domain deliberately does not claim: presence of an advisory is not proof of exploitability. We report presence, labelled as presence. Reachability is declared roadmap, not an implied capability.

Measure where the artifact exists

This is the principle we think matters most in v2.0, and it sounds obvious until you notice how often it is violated:

Measure each dimension where its artifact exists. Where the artifact is out of reach, report the dimension as unmeasured. Never substitute a proxy and present it as the thing.

The case that forced us to write it down is container images. An image's vulnerabilities live in the image: its operating-system packages, its runtime, the layers the build produced. A repository contains a recipe for that image, not the image.

Three approaches exist and two of them are dishonest. You can scan the base image named in the Dockerfile, which is not what the customer deploys. You can infer staleness from the image tag, which is a heuristic on a string presented as a measurement. Or you can measure the image where it exists: in the pipeline that just built it.

We take the third path, and it gives the model a property most instruments do not have. A dimension can be measured or not depending on where the assessment runs, and the verdict says which. A repository assessment reports image contents as unmeasured, never as zero vulnerabilities. Withholding evidence cannot improve a verdict, so nobody games the score by staying quiet.

Following the data across files

Pattern-based analysis matches shapes inside one file. It sees eval(userInput) when both appear together, and it misses the case that actually occurs in production, where the untrusted value enters one function, passes through two more, and reaches the dangerous operation in a fourth file.

v2.0 adds interprocedural taint analysis for that path, and the finding carries the path itself: where the data entered, which files it crossed, where it landed. It is validated against public applications whose folder structure encodes the expected vulnerability class, which checks classification and not only detection, and against a reference application in ordinary health where the correct answer is to report nothing at all.

Because it needs its own runtime and gigabytes of memory, it does not run inside the pull-request gate. The fast assessment runs on every change; the deep one runs out of band. Any assessment that asks for depth where the deep engine is unavailable is served by the fast engine and records that it was, so the provenance never claims a depth it did not deliver.

We published the instrument's own failures

A model that asks organizations to trust measurement owes the same discipline to the measurement of itself. So the paper publishes four occasions where a number was an artefact of how or where we were measuring rather than a property of what we measured.

Two of them are about speed. On a two-core machine, every stage's measured duration includes the other stages, so a per-stage table reads like an attribution when it is partly a division of scarcity. And a client polling every five seconds for an analysis that finishes in five seconds produced stable, reproducible, entirely artefactual numbers.

The other two are about environment, and they are worse because they are silent. A query engine reading its own rule file with the wrong default character encoding aborts, the parse still succeeds, and a failed query is correctly recorded as not measured: an analysis that never happened, reported as an absence rather than an error. And a rule whose precision depends on type resolution can find nothing in one environment and everything in another, with no error on either.

We report them because the general lesson is the useful part: an instrument's results are a property of the instrument and its environment together, and a validation run on the author's machine measures a system nobody else will run.

Read the paper

The full methodology, including the limitations we declare and the validation agenda we have not completed, is at scalequality.io/research/sqcm, with the PDF and a permanent DOI.

The bibliography is now self-contained at 34 references, so the paper resolves its own citations without sending you to the previous version. v1.0 remains available under its own DOI.

What has not changed is the commitment underneath both versions. An instrument earns the right to be believed by being explicit about what it saw, what it did not, and how confident it is in the difference.