The chain, end to end
A control only makes sense once you can see where it sits. The chain runs: a developer’s commit, the dependency resolution that turns a manifest into actual bytes, the build that combines them, the metadata the build emits about itself, the registry that stores the result, and the admission decision that lets it run. Each hop is a place where what you intended and what you got can diverge.
Trace one release through it. A reviewed commit lands on a protected branch. CI resolves dependencies through a private proxy, checking each against the hash in the lockfile. The build runs in a fresh, network-isolated container over that verified input set, so the input list is complete by construction. The builder emits a signed provenance attestation naming the source, the commit, the builder identity and the resolved inputs, plus an SBOM generated from the finished artifact. The artifact is signed by digest and pushed. At deploy time an admission controller resolves the tag to a digest, checks the signature and the signer’s identity, evaluates the provenance against policy, and only then admits the workload. Nothing in that flow asks anyone to take a human’s word.
Four ways trust breaks
You do not need an exhaustive attack catalogue to design the defenses. Four failure shapes cover almost everything, and each maps to one control. (For enumerating a system’s own threats, see threat modeling; this section exists only to motivate what follows.)
Dependency compromise. A package you depend on — directly or four levels down the transitive graph — ships code nobody reviewed. The cause may be a maintainer account takeover, an abandoned package that changed hands, or a resolver that silently preferred a public name over your internal one. Control: pin by content hash, and control which registry you resolve from.
Build-system compromise. The source is clean but the environment is not: a reused runner carrying state from a previous job, a step that fetches an unpinned script at build time, a workflow file changed without review. Control: hermetic builds on ephemeral runners, plus provenance recording what built what.
Artifact tampering. The bytes change between build and deploy — in a registry, a mirror, a cache, or because a mutable tag was repointed. Control: sign by digest and verify at every consumption point.
Publishing-credential theft. Nothing in your build breaks; someone simply publishes under your identity with a stolen token. Control: have no long-lived publish credential to steal.
Pinning, lockfiles, and integrity hashes
A version constraint is not a pin. A range such as ^4.17.0 resolves to whatever is newest at install time, so two builds of the same commit can contain different code. Three levels exist and only the third is a real control. A range guarantees nothing. An exact version gives reproducible resolution, but a version number is a label, not an authenticator; it says nothing about the bytes behind it. Version plus integrity hash makes the resolver refuse anything whose content does not match, so republishing under an existing version fails loudly instead of silently. Every mature ecosystem has this: package-lock.json integrity fields, pip’s --require-hashes, go.sum backed by a checksum database, Cargo.lock, and container references by @sha256: digest rather than by tag.
# pip: with --require-hashes, an unhashed requirement is a hard error
requests==2.32.3 \
--hash=sha256:63ac... # wheel
--hash=sha256:55365... # sdist
# containers: pin the digest, because tags are mutable pointers
FROM python:3.12-slim@sha256:2a1f...
# CI steps are dependencies too - pin by commit SHA, not by tag
uses: actions/checkout@8f4b7f8... # 40-hex commit, not @v4Two things pinning does not do. It does not cover what the lockfile omits: install-time scripts, build plugins and CI actions routinely sit outside it, and an action pinned to a mutable tag is an unpinned dependency running with your job’s privileges. And pinning freezes your vulnerability exposure along with your builds. A pin without an update pipeline is how a service ends up years behind on a library with a known remote-code-execution advisory. Pair every pin with automated bump pull requests, batch low-risk ones weekly, and require human review for major versions and for any package whose maintainer set has changed.
Narrowing the intake: proxies, allowlists, vendoring
Pinning verifies what you fetched. Narrowing the intake reduces what you can fetch at all, and gives you one place to observe and enforce.
A pull-through proxy in front of every public index is the highest-leverage piece. It caches every artifact you have ever resolved, so an upstream deletion or republish cannot change your build; it produces one audit log of what entered the organisation; and it is the natural place to enforce an allowlist or a quarantine window on brand-new releases.
Scoped namespaces matter more than teams expect. Register internal package names or scopes in the private registry and configure resolvers never to fall back to a public index for them. A resolver that treats public and private sources as one flat namespace, preferring the higher version number, is the most common way an internal name gets satisfied by something you did not publish.
Vendoring — committing dependency source into your own repository — is the strongest intake control and the worst ergonomically: every upgrade becomes a reviewable diff in your own pull requests, which is exactly the point and exactly the cost. Reasonable for a small, slow-moving dependency set; unreasonable at twelve hundred transitive packages. Base images deserve the same thinking — mirror them, rebuild on a schedule, pin downstream to your mirror’s digests.
SBOM: inventory you can query, not a control
A software bill of materials is a machine-readable list of the components inside an artifact, in SPDX or CycloneDX. Be blunt about what it does: an SBOM prevents nothing. It is not a gate and not a signature. What it buys is answer latency at incident time. When an advisory lands on a widely used library at 2am, the question is which of your four hundred deployed artifacts contain it, at which versions, in which clusters — and an SBOM inventory answers in seconds instead of a day of grepping build logs.
Two details decide whether you get that payoff. Generate the SBOM from the built artifact, not the manifest: the manifest describes intent, the artifact contains reality, including base-image packages, statically linked libraries and vendored copies no manifest mentions. And key the inventory by artifact digest, joined to deployment data, so a query returns running workloads rather than file names. An SBOM attached as an attestation and mirrored into a queryable store is useful; one written to a build log is decoration.
Expect imperfection: detection quality varies sharply by ecosystem and version strings are often ambiguous for vendored code. Treat matches as a triage list, not a verdict.
Provenance attestations: in-toto and SLSA as a maturity model
Provenance answers a different question from the SBOM: not what is inside this artifact but what produced it. An in-toto attestation is a signed statement binding a subject (an artifact digest) to a typed predicate (the claims). SLSA provenance is one such predicate, carrying the source repository and commit, the builder identity, the build entry point, the parameters and the resolved inputs. The envelope is signed by the builder, not by a developer — the value is that the claim comes from the system that observed the build, not from someone asserting it afterwards.
Read SLSA’s build levels as a maturity ladder, not a certification. Level 1 means provenance exists and is available; alone it stops nothing, but it forces the build to be scripted and consistent. Level 2 adds signed provenance from a hosted build service, so it can no longer be fabricated on a laptop. Level 3 requires hardened, isolated infrastructure where the provenance signing key is unreachable from the build steps — the first level that meaningfully resists a compromised build step.
Sequence it instead of attempting a fleet-wide jump: Level 1 everywhere because it is nearly free, Level 2 on anything shipping outside the team, Level 3 on the few artifacts with the widest blast radius. The step teams skip is verification. Provenance nobody checks is a receipt in a drawer; policy must assert concrete fields — builder identity, expected source repository, protected ref.
Hermetic and reproducible builds
A hermetic build declares all its inputs and does not reach the network for anything undeclared. In practice: split the build into an explicit fetch phase that resolves and hash-verifies dependencies into a content-addressed cache, and a compile phase that runs with networking disabled against only that cache. Two properties follow — the build cannot silently acquire an input, and the input list in the provenance is complete rather than best-effort.
Reproducible goes further: the same declared inputs produce a bit-identical output on a different machine, at a different time, run by a different party. This is the strongest verification available anywhere in the stack, because it removes the builder from the trusted set. An independent rebuilder can rebuild your artifact and compare digests; if they match, no signature is needed to believe the artifact corresponds to the source, and if they diverge you know something between source and artifact is wrong without needing any insider access.
Getting there is real engineering. The recurring sources of nondeterminism, roughly in the order they bite: embedded build timestamps (fix with a set SOURCE_DATE_EPOCH); filesystem iteration order and metadata inside archives (sort entries, normalise uid, gid, mtime); absolute paths, hostnames and build-user names baked into debug information (build under a fixed root); locale- and timezone-sensitive formatting; parallel steps racing on a shared output; and randomised hash seeds in code generators. A useful intermediate goal is diff-reproducible: rebuild twice, diff, and drive the difference down to a short documented list of benign fields.