Cloud secrets architecture — manager, rotation, injection, versioning, audit

Why architecture matters here

Secrets fail on leaks and lifecycle. A committed key gets scanned in seconds. A stale credential surfaces during pentest. Rotation without downstream update breaks apps.

The architecture matters because the failure modes are structural, not careless. A credential has to reach a running process somehow, and every path that delivers it also creates a way for it to escape and a dependency that can stop the process from starting. What follows works through those paths and the tradeoffs each one forces.

What counts as a secret, and why config cannot hold it

A secret is any value whose disclosure hands an attacker a capability they did not already have. That definition is narrower than "sensitive" and wider than "password". A database hostname is not a secret even though you would rather not publish it; a webhook signing key is, because anyone holding it can forge traffic you will then trust. The test is capability transfer, not embarrassment.

Configuration and secrets look identical at the point of use - both arrive as key/value pairs a process reads at startup - and that surface similarity is why teams keep putting them in the same file. They diverge on every other axis. Configuration is meant to be read by humans during review, diffed in pull requests, pasted into support tickets and copied between environments. A secret must survive none of that. Configuration has no revocation story because there is nothing to revoke; a secret needs an owner, an expiry, an audit trail, and an answer to "who else has seen this value". Changing config is a roll-forward you can undo at leisure; changing a secret is a distributed state transition that has to be coordinated with the resource on the other end.

Once you accept that the two need different machinery, the shape of the rest follows. Config belongs in the repository. Secrets belong in a store that can answer three questions the repository cannot: who is permitted to read this, who actually read it, and what is the live value right now.

Secret zero - the credential you cannot store anywhere

The moment you centralise secrets you manufacture a new problem. The store authenticates its callers, so every workload needs a credential in order to talk to the store. That credential is itself a secret. Putting it in the store is circular; writing it to disk next to the application recreates exactly the exposure the store was built to remove. This is the bootstrapping problem, usually called secret zero.

The naive answers fail in instructive ways. Baking a store token into the container image means every registry reader holds it, and it survives in the layer history forever even after you "remove" it. Passing it in at deploy time relocates it into the CI system, which then becomes the highest-value target in the estate while typically having weaker access review than production. Encrypting it at rest just moves the question to the decryption key. A single shared token per cluster scales badly in the way that matters most: it cannot be attributed to a workload in the audit trail, and revoking it after one pod is compromised takes down every other pod that was using it.

The structural observation is that secret zero cannot be solved by storing it better. Any bearer credential you place on a host is a credential that anyone reaching the host also holds. It is only solvable by removing the need to place one there at all.

Workload identity dissolves the bootstrap problem

The way out is to stop authenticating a workload by something it holds and start authenticating it by something it is. The platform the workload runs on already knows facts the workload itself cannot forge: which node the scheduler placed it on, which service account it runs under, which image digest it was launched from, which repository and branch a pipeline job originated in. If the platform will attest to those facts, the secret store can trust the attestation instead of a stored password.

Mechanically the attestation is a signed, short-lived document. The runtime - a cluster API server, a hypervisor metadata endpoint, a CI provider's OIDC issuer - mints a token whose claims describe the workload and whose signature chains back to a key the store already trusts. The workload presents it, the store validates the signature, checks the claims against a policy, and issues a session carrying whatever permissions that policy allows. Nothing durable is ever written to the workload's filesystem, and the attestation usually expires in minutes, so intercepting one buys an attacker very little.

Notice what moved. The trust root is no longer a secret; it is a property of the infrastructure. Forging it now requires compromising the platform's signing key or the scheduler itself, not reading a file. How those trust relationships are expressed, scoped and evaluated - policy conditions, role assumption, federation between providers - belongs to the identity layer and is covered in cloud IAM architecture. What matters here is the consequence: with attestation in place there is no first secret to protect, and the entire class of "how did that credential end up on the box" incidents stops existing.

The architecture: every piece explained

The top strip is storage. Secret creator generates or provides. Secret manager holds. Versioning preserves history + rollback. Access policy gates who reads.

The middle row is delivery. Rotation is scheduled or triggered. Injection can be env vars, files, or SDK. Short-lived leases issue dynamic credentials with TTL. Audit log records every access.

The lower rows are ops. Break-glass is emergency access with heavy audit. Compliance gates. Ops runs rotation drills + workload identity migration.

Static credentials versus credentials generated on demand

A static secret is a value someone created once that both sides now remember: a database password typed into a form, a vendor-issued API key, a signing key generated at project setup. A dynamic secret does not exist until a workload asks for it. The store holds a privileged connection into the backing system and, on request, creates a fresh principal there - a new database role, a new session, a short-dated certificate - returns the credential, and schedules its own revocation.

The difference is not convenience, it is a different threat model. Exposure of a static credential is unbounded in both directions: you cannot tell when it leaked, and it keeps working until a human notices. The blast radius is every system that trusts it and every copy anyone ever made. A generated credential scoped to one workload and valid for thirty minutes turns a leak into thirty minutes of one workload's permissions. Detection stops being the thing that limits damage, because expiry limits it whether or not anybody is watching.

Generation also makes attribution real. Each lease maps to a single requester, so a slow query on the database traces back to the workload that asked for the role rather than to a shared account named app_user that forty services share. In an incident that one property is worth more than most detective controls.

The catch is that the backing system has to support principal creation and cleanup, and short TTLs across a large fleet mean constant churn in that system's own catalogue. Where the backend cannot do it - most third-party SaaS keys, most partner integrations - you are back to static values, and rotation is the only lever available.

The retrieval path, and what each topology couples you to

There are four common shapes for moving a value from the store into a running process. The choice is mostly a decision about which failure you would rather own.

Direct SDK call

The application imports the store's client and fetches what it needs, usually at startup and again as a lease approaches expiry. This is the most honest option: the dependency is visible in the code, and the application gets to decide how it degrades when the fetch fails. It is also the one where retries, backoff and cache invalidation are your problem, and every language runtime in the fleet needs its own correct implementation of them.

Init-time injection

A separate step - an init container, an entrypoint wrapper, a configuration management run - fetches secrets before the application starts and hands them over as files or environment variables. The application stays entirely ignorant of the store, which makes this the easiest retrofit for software you did not write. The coupling is brutal in one specific way: the store must be reachable at every process start. A store outage does not disturb pods that are already running, but it silently blocks every restart, scale-out and rollout, so it presents as an inability to recover rather than as an immediate error.

Sidecar agent

A co-located process authenticates once, fetches on the application's behalf, writes to a shared volume or serves on loopback, and renews leases in the background. Renewal becomes invisible to the application and the store sees one client per pod instead of one per connection. You pay in resident memory and CPU multiplied by pod count, and you inherit a startup ordering problem: if the agent is not ready before the application reads, the application reads nothing and usually does not say so clearly.

Driver-mounted files

The platform mounts secrets into the pod as files, sourced from the store by a node-level driver. Applications read files, which they already know how to do, and a rotation can be reflected by rewriting the file in place. The subtlety is that rewriting a file is not the same as the application noticing: unless the process watches for changes or is restarted, it happily keeps using whatever it read at boot. Node-level drivers also concentrate risk, since a fault there affects every pod scheduled on that node.

Whichever you pick, be explicit that the store is now on the critical path for either request serving or process startup, and give it an availability budget at least as good as whatever depends on it. Its own dependency on the key hierarchy underneath is a separate concern - see cloud KMS architecture.

Caching in memory, and the TTL you can actually justify

Fetching on every use is the safest thing you could do, and nobody does it, because it puts the store in the path of every request and multiplies its load by your request rate. So values get cached in process memory, and the cache lifetime becomes the knob that trades availability against revocation latency.

Read that tradeoff precisely. A long lifetime means a process rides out a store outage for that long, which is exactly the resilience you wanted. It also means that after you revoke a credential, workloads keep presenting the old one until their caches age out. Revocation latency is bounded below by the cache lifetime, not by how quickly you clicked the button. Five minutes and twelve hours are the same design with radically different incident behaviour: at twelve hours, containing a compromised credential means restarting the fleet.

Two refinements earn their complexity. First, split the clocks: use a short refresh interval so the cached copy is normally fresh, but allow a stale copy to be served past the refresh point when the store is unreachable, up to a longer hard ceiling. That gives fast revocation in steady state and survives brief outages. Second, push revocation down to the source wherever the backend allows it - dropping a database role invalidates it immediately no matter what any cache believes. Expiry is the fallback for backends where you cannot invalidate at source.

One absolute: do not cache to disk. A memory cache dies with the process. A disk cache outlives it, gets captured in volume snapshots, and turns up in a forensic image months later.

Versioning and staged rollout

Store each secret as an immutable series of versions with a mutable pointer that consumers resolve at read time. Consumers should ask for the pointer, not for a version identifier, so that promoting a new value is one write and reverting is another. Requesting a pinned version is for debugging and for the narrow window inside a coordinated change; it is not how applications should normally read.

Immutability buys more than rollback. It gives the audit trail something precise to record - this identity read version seven at this timestamp, rather than "read the secret" - which is what lets you reconstruct, after an incident, exactly which value was live during the window in question. It also makes a rollout staged rather than atomic: because superseded versions still exist and still work, a fleet halfway through picking up a change is in a legal state rather than a broken one.

Staging is the practical payoff. Promote the pointer, then watch two signals: the authentication error rate at the resource, and a counter reporting which version each instance currently holds. If the new value is wrong, moving the pointer back restores service without anyone hunting for the old password. Without versioning, a bad secret update is an outage whose recovery time is however long it takes a human to find the previous value - which, if it was overwritten in place, may be never.

Rotation is a two-phase change, not a swap

Nearly every rotation-induced outage has the same root cause: somebody treated it as a single event. Replacing a credential touches two independent systems - the store, and the resource that validates the credential - and they cannot be updated in the same instant. If the resource stops honouring the old value before every consumer has picked up the new one, you have an outage, and its length is your slowest cache lifetime.

The invariant to design around is overlap. The new credential must be accepted by the resource strictly before the old one stops being accepted, and the interval between those two moments must exceed the worst-case propagation delay across the fleet. Concretely that is three phases: make the new value valid while the old one still is; promote the pointer and let consumers drift onto it; revoke the old value only afterwards, once you have evidence nothing is still using it.

That evidence is the step most often skipped. Before revoking, check the audit trail for reads of the superseded version and, where the backend exposes it, sessions currently authenticated with it. Revoking on a timer rather than on evidence is how a batch job that runs weekly gets broken by a daily rotation.

Two design consequences follow. The resource must tolerate two valid credentials at once - two passwords, two enabled API keys, an overlapping set of accepted signing keys. Where it genuinely cannot, rotation is inherently a brief outage and has to be scheduled as one rather than pretended away. And the revocation phase must actually run: a rotation that issues new values and never retires old ones has increased the number of live credentials rather than reduced exposure. Provider-specific mechanics for driving this cycle automatically are covered in AWS Secrets Manager rotation; the shape above is what any implementation has to satisfy.

Environment variables, files, and process memory

Environment variables are the default because they are the easiest thing to inject, and they are the worst of the three destinations. Being process-wide and inherited, every child gets them, including the shell spawned to run a health check. On many systems they are readable through the process table. Frameworks print them in startup banners; crash handlers serialise them into reports; error trackers and APM agents collect the environment block by default; a container inspect command shows them to anyone with access to the runtime socket. None of these require an attacker - they are ordinary operational behaviour writing secrets into logs you will retain for a year.

Files are better in the ways that matter. Permissions are per-file, so you can restrict to the owning user rather than the whole process tree. A file can be replaced in place, so a rotation need not force a restart. And nothing in the normal diagnostic path reads them speculatively. Mount them from a memory-backed filesystem so the bytes never reach persistent storage, set the mode explicitly instead of inheriting a umask, and keep them out of any directory that gets archived into a support bundle.

Best of all is neither: fetch the value into a variable, use it to establish whatever connection or signature needs it, and drop the reference. Where the language permits, hold it in a mutable buffer you can overwrite rather than an interned string that lingers until collection. This is not a defence against an attacker with arbitrary memory read - it is a defence against the mundane path where a heap dump gets attached to a bug report.

One rule spans all three: whatever you choose, make sure the logging framework cannot serialise it. Redaction filters keyed on field names catch most cases; structured logging that dumps whole objects will walk straight around them.

Sprawl, detection, and what to do when one leaks

Secrets spread because copying one is the fastest way to unblock somebody. The store holds the canonical value while copies accumulate in a config file made "just for local testing", a pipeline variable, a chat message, a screenshot attached to a ticket, an infrastructure state file, a laptop's shell history. Sprawl is less a discipline failure than the predictable result of a workflow in which obtaining the real value is harder than pasting it. Fix the workflow and the copies stop appearing; lecture people and they will not.

Detection has to run at several points because each layer catches what the others miss. Pre-commit hooks give the fastest feedback and are the only layer that prevents rather than reports, but they run on a developer's machine and can be bypassed. Server-side scanning at push time cannot be bypassed and catches the skipped hooks. Full history scanning across every branch catches what predates the hooks - necessary because rewriting history does not remove a value from forks, clones or the hosting platform's cached views of dangling commits. All three generate false positives, so budget for triage: a scanner people have learned to ignore is worse than no scanner, because it manufactures the belief that something is watching.

When one does leak, the order of operations matters more than the speed. Revoke first. The instinct is to rotate - mint a replacement and update consumers - but until the old value stops being accepted the attacker still has access, and a clean rotation takes time you do not have. Invalidate the exposed credential at the resource, accept whatever outage that causes, then issue the replacement, then reconnect the consumers. Only afterwards pull the audit trail for the exposure window and look for use you cannot attribute to your own workloads.

Then remove the value from wherever it surfaced, while understanding that removal is cleanup rather than remediation. A key pushed to a public repository should be treated as permanently public no matter how quickly the commit was deleted.

Audit logging and break-glass

Audit for secrets answers a narrower question than general access logging: which identity read which version of which secret, when, and from where. Record all four. The version is the field teams usually omit and the one that makes post-incident reconstruction possible, because it tells you whether the credential an attacker used was the value that was live at the time.

Interpret the log knowing its blind spot. Caching means reads are far rarer than uses - a process that fetched once at boot and has been authenticating for a week produces exactly one line. Absence of reads is therefore not absence of use, and concluding from a quiet log that a secret is unused before deleting it is a reliable way to break something obscure. Correlate with activity at the resource instead. What the log is genuinely excellent for is the inverse signal: reads that should not exist at all. A workload identity reading a secret it has never read before, a human principal reading something only services should touch, one identity fanning out across many secrets in a short window - these make cheap alerts with low false-positive rates precisely because normal read patterns are so boring.

Break-glass is the deliberate exception: a path that grants access when the normal path is unavailable or too slow, for the incident where the identity provider itself is the thing that is down. Design it as a real control rather than an oversight. It should be a distinct credential or role that is never used in ordinary operation; invoking it should require an out-of-band step so it cannot happen silently; the grant should expire on its own rather than depending on someone remembering to remove it; and its use should page somebody other than the person using it. Then exercise it on a schedule. An emergency path nobody has tested in a year is not an emergency path, it is a standing credential nobody is watching.

End-to-end flow

End-to-end: DB password stored in secret manager with versioning. Rotation weekly: new version created; DB users updated; apps rotate via SDK. Access via role-based policy. Audit log records access. Compliance report generated monthly. Short-lived leases used where possible; the long-lived password is now the exception.

That summary hides the interesting part, so trace one workload concretely. A payments service runs as a pod whose service account is bound, by policy, to a role that may request the reporting_ro Postgres role and nothing else. At startup the node-level driver presents the pod's attestation token to the store; the store checks the signature and the claims, matches the policy, asks Postgres to create a fresh role with a two-hour lease, and writes the username and password into a memory-backed file the pod already has mounted. No credential was ever built into the image, passed through the pipeline, or written to a disk that survives the pod.

Ninety minutes in, the agent renews. Postgres gets a new role; the file is rewritten; the connection pool notices the file changed and opens replacements, draining the old connections rather than killing them. The previous role is dropped only after the store sees no sessions holding it. That is the overlap rule applied to a lease rather than to a rotation, and it is the same invariant either way.

Meanwhile the one genuinely static credential in the system - a partner API key the vendor will not let you generate programmatically - is stored as versioned material with a quarterly rotation. Its rotation is manual and follows the three phases: register the second key with the partner, promote the pointer, watch the version counter until every instance reports the new one, then retire the first key. The audit trail is what tells you the fleet has converged, and it is the only reason the retire step is safe to run.