Cloud KMS architecture — envelope encryption, hierarchy, HSM, audit, rotation

Why architecture matters here

KMS mistakes are expensive. A key deleted without recovery loses data forever. A KMS API rate limit under-provisioned throttles the entire application. A cross-account grant leaks encryption to unintended tenants.

The architecture matters because envelope encryption + hierarchy protects against both leaks and cost blowouts. Audit + rotation keep the system honest over time.

With the pieces mapped, you can implement encryption + key management that meets both product needs and compliance.

The architecture: every piece explained

The top strip is the hierarchy. Application generates a DEK (Data Encryption Key) for each object or short session. KEK (Key Encryption Key) wraps the DEK before storage. Root key lives in an HSM and never leaves.

The middle row is the controls. Envelope encryption uses the KEK to wrap the DEK; only the DEK ever encrypts bulk data (cheap and fast). Access policy defines who can use which key for encrypt/decrypt. Rotation is scheduled or on-demand; envelope makes it fast (rewrap DEKs, not data). Regional isolation stores keys in specific regions for data-residency.

The lower rows are governance. Audit log records every key use. Cross-account grant lets one account decrypt data owned by another under specific conditions. Ops handles key lifecycle, break-glass procedures, and BYOK for regulated customers.

Envelope encryption, in one paragraph, and where the details live

Every managed KMS is built on the same trick: the service never encrypts your bulk data. It holds one long-lived key inside a hardware boundary and uses it only to wrap short, per-object symmetric keys that your application uses locally. The reason is not cryptographic elegance, it is arithmetic - an HSM does thousands of operations per second over a network, while a local AES-GCM loop does gigabytes per second in the data path, and no amount of provisioning closes that gap. Wrapping a 32-byte key is a constant-cost operation no matter whether the payload is 4KB or 40GB, which is what makes both the latency and the per-request bill tolerable.

That mechanism - the GenerateDataKey call, the plaintext/ciphertext key pair it returns, encryption context binding, the on-disk envelope layout, and scrubbing the plaintext key from memory - is covered in depth in KMS envelope encryption architecture, and this article does not re-derive it. What follows is everything around that mechanism: how keys are named and versioned, how they are born and destroyed, what the hardware boundary actually promises, and why a key management service quietly becomes the availability floor of every system that touches it.

The hierarchy is a naming and trust structure, not only a crypto one

The DEK-under-KEK picture is the cryptographic hierarchy. Sitting on top of it is an equally important resource hierarchy, which is what policies, quotas, residency and billing actually attach to, and the three major providers name it differently while modelling nearly the same thing.

In AWS, the unit is a KMS key (historically "customer master key"), identified by a UUID key ID and a full ARN that embeds account and region. It lives directly in an account and region with no intermediate container, and it carries an alias - a mutable human-readable pointer such as alias/prod-payments. In GCP, the path is explicit: project, then location, then key ring, then crypto key, then crypto key version. The key ring is a permission and locality grouping; it cannot be deleted or moved, which surprises people who treat it as a folder. In Azure, the container is a key vault (or a Managed HSM pool), the object is a key, and each key has versions addressed by URL.

The alias or path indirection matters more than it looks. Ciphertext, snapshots, and resource configurations that reference a key by its immutable ID are pinned to that exact key forever; references made through an alias can be repointed. If you ever intend to migrate to new key material - after an incident, an acquisition, or a move from software to HSM protection - the applications must be reading the alias, not the raw ID, or the migration turns into a code change across every consumer.

Key versions, and why rotation does not re-encrypt anything

A key is not a single blob of bytes. It is a named container holding an ordered set of key versions, each with its own material and its own state. Exactly one version is the primary (AWS calls the equivalent the current backing key); it is the one used for every new wrap operation. Every other version stays around in a decrypt-only capacity.

This is what makes rotation cheap. When a wrapped DEK is produced, the resulting ciphertext blob records which key version produced it. On unwrap, the service reads that identifier and reaches for the matching version - the caller does not choose, and usually cannot. So rotating a key means: generate a new version, mark it primary, and stop. Nothing that already exists is touched. Ciphertext written last year is still unwrapped by last year's version, indefinitely, and the storage cost of that is a few dozen bytes of key material per version.

Automatic versus manual

Automatic rotation is a flag on the key with a fixed or configurable period; the service mints a new version on schedule with no involvement from you. Manual rotation is an API call that does the same thing on demand - useful after a suspected compromise or a staff departure, and the only option for key material you imported yourself, because the service has no way to generate a successor for material it did not create.

There is a second, heavier meaning of "manual rotation" that people conflate with the first: creating an entirely new key, repointing the alias, and re-wrapping or re-writing existing data under it. That is the only operation that genuinely retires old material, because as long as old versions remain in the key, the service can still decrypt everything ever written under them. Automatic rotation limits how much data any single piece of key material protects going forward; it does not shrink the exposure of what is already written.

Two follow-on consequences are worth internalising. First, rotation does not rotate DEKs. If a data key leaked, rotating the KEK changes nothing about that object - you must re-encrypt it. Second, rotation is not revocation. Disabling or destroying the key is revocation; rotation is hygiene.

The lifecycle: create, enable, disable, schedule destruction

A key moves through a small state machine, and the interesting transitions are the ones you cannot undo.

Enabled is the working state. Disabled keeps the material intact but rejects every cryptographic operation, so all data protected by it becomes unreadable immediately and reversibly. This is the single most useful operational primitive in the whole service: it is a dry run for deletion, an instant containment action during an incident, and a way to prove which systems actually depend on a key by turning it off in a staging account and watching what breaks.

Pending deletion is where the deliberate friction lives. You do not get to delete a key immediately; you schedule its destruction after a mandatory waiting period measured in days, during which the key is unusable but recoverable by cancelling. The waiting period exists because key destruction is the only truly irreversible action in a cloud account. Deleting a VM loses a VM; deleting a key loses everything that key ever protected, including backups, snapshots, and the archive nobody remembered was encrypted under it. The window is there so that the alarms fire, the systems fail loudly, and a human has time to intervene.

Destroyed means the key material is gone. The key's identifier usually survives as a tombstone, so ciphertext referencing it produces a clear "this key has been destroyed" error rather than a confusing not-found - but the bytes are unrecoverable, by design and by contract. Cloud support cannot restore them; there is no escrow copy. When this is what you want, it is called crypto-shredding: erasing a tenant's data by erasing one key instead of chasing petabytes of replicas, backups, and analytics copies. When it is not what you want, it is a permanent outage.

The lifecycle rule that follows: destruction must be gated on evidence of non-use, and the evidence is the audit log. Disable, wait through a full business cycle including monthly and quarterly batch jobs, confirm zero decrypt attempts, then schedule.

Where key material comes from, and the trust argument for each

There are three provenances, and they trade convenience against how much of the provider you have to believe.

Generated inside the service

The default. Material is created inside the hardware boundary and never exists in plaintext outside it. You get automatic rotation, replication, and durability handled for you. The trust argument is that the provider generated a key you have never seen and asserts it cannot extract it - you are trusting the implementation, the operational controls, and the attestation of the certification body that examined them.

Imported (BYOK)

You generate material in your own HSM, fetch a public wrapping key and a one-time import token from the service, wrap your material under it, and upload. The service can now use the key, but you retain the only other copy. The trust story improves in one narrow but real way: because you hold the original, you can prove the material's origin and you can re-import it after deleting it, which makes deletion a reversible control on your side rather than the provider's. The costs are concrete - no automatic rotation, an import token that expires, material that may carry its own expiry after which the key stops working, and the absolute requirement that you never lose your copy, because a re-import is the only recovery path.

External key stores

The strongest separation: the key never enters the provider at all. Their service holds only a reference and makes an outbound call to your HSM or key manager for every wrap and unwrap. This is what regulators mean by hold-your-own-key, and it genuinely means the provider cannot decrypt your data without a network round trip you can observe and refuse. The price is severe: your key manager and the link to it become a hard synchronous dependency of every read, its latency is added to every operation, and its outage is a total data outage in the cloud. Almost nobody should choose this without a compliance mandate that names it.

HSM backing, protection levels, and what the boundary promises

Managed KMS keys are backed by hardware security modules, but "HSM-backed" is a spectrum rather than a boolean. Providers expose it as a protection level or tier: software-protected keys, where material is held and used by the service's own software with the key encrypted at rest; shared multi-tenant HSM keys, where operations execute inside certified hardware that also serves other customers under cryptographic separation; and single-tenant HSM offerings such as dedicated HSM clusters or managed HSM pools, where the hardware partition is yours alone and you hold the security-officer credentials.

FIPS 140 validation is the shorthand used to compare these, and its levels describe escalating physical and logical protections - roughly: basic algorithm correctness, then tamper-evidence and role-based authentication, then tamper-response that actively zeroises material when the enclosure is breached, then extreme environmental attack resistance. The number is not a quality score; it describes what the module does when someone attacks the box. Which specific level a given tier holds changes over time and per region, so read the provider's current validation certificate rather than trusting a blog - including this one.

What the boundary actually promises is narrow and worth stating plainly: key material cannot be exported in plaintext, cryptographic operations happen inside the module, and administrative changes can be gated on quorum approval by multiple officers. What it does not promise is anything about authorised use. An attacker who obtains credentials that the key policy permits will get clean, successful, fully-logged decryptions. The HSM defends the key, not the data; access control and audit defend the data.

The request path - why KMS is an availability dependency of everything

Follow a read that touches encrypted data. The service fetches ciphertext and a wrapped key, then makes a network call to a regional KMS endpoint, which authenticates the caller, evaluates policy, performs the unwrap inside a hardware module, and returns the plaintext key. Only then does the actual decryption happen. That call sits synchronously in the request path, and it means the KMS in that region is now a hard dependency of your storage layer, your database, your queue, your secrets fetch at process start, and your disk attach at instance boot.

The mitigation is caching. Applications cache unwrapped data keys for a bounded TTL, or reuse one DEK across many objects, so a burst of reads costs one KMS call rather than thousands. Every cache decision is an explicit trade against blast radius: a longer TTL and broader key reuse mean fewer calls and lower cost, but a leaked plaintext DEK in memory now unlocks more objects for longer. Pick the TTL from the incident you are willing to have, not from the bill.

Quota exhaustion is a real outage cause

KMS request limits are typically enforced per account, per region, shared across every service that calls it on your behalf. That sharing is the trap. A one-off analytics job that reads a few hundred million small encrypted objects issues one unwrap per object, saturates the account's request budget, and the throttling lands on completely unrelated production systems - instance launches stall, function invocations fail to decrypt their configuration, database connections cannot fetch credentials. The symptom presents as widespread 5xx in services that have nothing in common except the account they run in, and the throttled call is often buried several layers below the code you are looking at. Defences are structural: batch and reuse data keys, enable per-bucket or per-container key reuse features (see S3 encryption options for the object-storage version of this), split noisy batch workloads into their own account, and raise quotas before the migration rather than during it.

Regionality and the multi-region key problem

Keys are regional. This is not an implementation detail, it is the core of the data-residency guarantee - material generated in a region stays in that region's hardware, which is exactly what a residency requirement asks for. It is also the single most common way disaster recovery plans fail on the day they are needed.

The failure is mechanical. Ciphertext, snapshots, and backups embed a reference to the key that wrapped them, and that reference names a region. Copy an encrypted snapshot to your DR region and the copy operation must re-encrypt it under a key that exists there; if you did not create one, or did not grant the copy operation permission to use it, the copy fails or silently lands unencrypted. Restore from an object-storage backup into a fresh account in another region and every unwrap fails, because the key ARN in the metadata points at a region and account that the restore target cannot reach.

Multi-region keys exist to solve this: a primary key and replicas in other regions that share the same key material and a related key identifier, so ciphertext produced in one region can be decrypted in another with no re-encryption step. They are not a global key. Each replica is an independent resource with its own policy, its own grants, and its own enablement state, and coordinating rotation and deletion across replicas is your job. GCP's equivalent is placing the key ring in a multi-region or dual-region location; Azure replicates vault contents within a geography with a paired-region failover model.

The DR test that matters is not "can I restore the snapshot" but "can I restore it into a different region and a different account with production credentials revoked". Run that once and you will find the key reference problem before it finds you.

Key policies and grants are not the same thing as IAM

Access to a key is governed by mechanisms that sit alongside, and sometimes above, the identity system. The identity half - principals, roles, bindings, and how a request is evaluated - is covered in Cloud IAM architecture and is not repeated here. What is specific to KMS is the extra gates.

The first is the key policy: a resource policy attached to the key itself, listing which principals may use or administer it. In AWS this gate is mandatory - the key policy must allow the caller, and an IAM policy alone is never sufficient unless the key policy has explicitly delegated to IAM. That default delegation clause, granting the account root the ability to manage the key, is what stops you from bricking a key, and removing it in a well-meaning tightening pass is the classic way to create a key nobody on earth can administer any more. GCP takes the other route: keys are ordinary resources governed by IAM on the key or key ring, so there is one gate, not two. Azure has both models - legacy vault access policies and Azure RBAC - and running a vault with a mix of them is a reliable source of confusing denials.

The second mechanism is the grant: a narrow, programmatic, independently revocable delegation, typically issued so that a service can use a key on your behalf for the lifetime of one resource - a volume, a queue, a database instance. Grants are created and retired by services automatically, they can be constrained to specific operations and to a specific encryption context, and they do not appear in the key policy document, which is why a permissions audit that reads only policies will miss them. List grants explicitly when you are answering "who can decrypt this".

Cryptographic isolation between tenants

Once keys are cheap to create, per-tenant keys become an architectural option rather than a luxury, and they buy four distinct things. Independent revocation: disabling one tenant's key stops access to exactly that tenant's data. Deletion that actually completes: crypto-shredding one key satisfies an erasure request without hunting every replica, backup and analytics extract. Per-tenant audit: the key ID in the log line answers whose data was read. Residency: a tenant that must stay in one jurisdiction gets a key that cannot leave it.

The costs are equally concrete. Keys carry a monthly charge each, so a per-tenant key at a hundred thousand tenants is a line item, not a rounding error. Key sprawl makes policy review and rotation tracking harder. And request quotas do not scale with key count - they are usually per account, so splitting into many keys does not buy throughput.

The cheaper middle ground is one key with a per-tenant encryption context bound into every wrap, so a wrapped key stolen from one tenant cannot be unwrapped under another tenant's context and the context appears in every audit record. Understand what that does not give you: it is one key, so it revokes as one key and shreds as one key. If your compliance story includes per-tenant erasure or per-tenant revocation, context is not a substitute - you need separate keys for the tenants that require it, and a shared key for the long tail that does not.

Audit logging is the primary detective control

When data at rest is opaque, the only record of who read it is the key-use log. Every wrap, unwrap, key generation, policy change, grant issuance and administrative action is recorded with the caller identity, the key, the operation, the source address, and the encryption context. That log, not the storage layer's access log, is the authoritative answer to "was this data accessed".

Two practical warnings. First, data-plane key usage is high volume - a busy account can produce more KMS log records than all other services combined, and the log retention bill can exceed the KMS bill. Sample or route it deliberately rather than discovering the cost later. Second, the control-plane events are the ones that need real alerting, and they are rare enough to alert on individually: ScheduleKeyDeletion and its cancellation, key disable and enable, any change to a key policy, any new grant on a sensitive key, and any change to rotation configuration. A destruction schedule that nobody noticed for the whole waiting period is a permanent data loss that the system politely warned you about for days.

On the data plane, alert on shape rather than volume: a principal decrypting under a key it has never touched before, a spike in access-denied results (either a misconfiguration or someone probing), and decrypt activity from a region or network path that no legitimate workload uses.

The cost model, and why envelope encryption is what keeps it viable

Managed KMS bills on two axes: a recurring charge per key per month, and a charge per cryptographic request. Dedicated single-tenant HSM offerings replace the per-key charge with a per-hour charge for the hardware pool, which is a different order of magnitude and only rational at scale or under mandate.

The per-request axis is where designs get expensive, and it explains the entire architecture. If the service encrypted your data directly, every read of every object would be one billable, rate-limited, network-latency-bearing request - your KMS bill and your KMS latency would scale linearly with your traffic. With envelope encryption, the request count scales with the number of keys you unwrap, not the number of bytes or reads you serve, and caching or reusing data keys drives it down further. This is why the cost model, the quota model, and the latency model all improve together with the same change, and why "reduce KMS calls" is nearly always the correct first optimisation.

The other axis, per-key charges, is what makes per-tenant key strategies a real decision. Model it before committing: number of keys times monthly rate, plus request volume after caching. Check current published pricing rather than assuming - the rates differ per provider, per protection level, and per region.

Failure modes that actually take systems down

The key is disabled or destroyed while data still depends on it. The data is not corrupted; it is simply inert. Disable is recoverable, destroy is not. This is the reason for the mandatory waiting period, and the reason destruction should be preceded by weeks of audit-log evidence that nothing is calling the key.

A cross-account key reference breaks a restore. Data encrypted under a key owned by another account remains readable only while that account's key policy permits you. When the sharing account revokes the grant, deletes the key, or is closed after a project ends, every backup taken under it becomes unrecoverable - and this is typically discovered during a restore drill years later. Anything you must be able to restore should be encrypted under a key your own account owns.

Region mismatch on DR. Covered above, and it belongs on this list because it is the most common one: the restore fails not because the data is missing but because the key it names does not exist where you are restoring.

Rotation was never actually enabled. Automatic rotation is off by default on most key types and is not enabled by creating the key through a console wizard. Nobody notices, because nothing breaks - the key simply keeps working with the same material for years until an auditor asks. Detect it with a scheduled inventory query over every key's rotation state, not with a policy document that says rotation is required.

Key policy lockout. A tightening pass removes the last principal with administrative rights on the key. The key still encrypts and decrypts, so nothing alarms, but nobody can change its policy, enable rotation, or schedule its deletion again. Recovery usually requires provider support, if it is possible at all.

Imported key material expired. BYOK material can be uploaded with an expiry. When it passes, the key stops working with no deployment, no change, and no obvious cause - the classic three-in-the-morning page with an empty deployment log.

End-to-end flow

End-to-end: an app stores a file. It generates a DEK (256-bit AES), encrypts the file locally, then calls KMS to encrypt the DEK with the KEK; stores the encrypted DEK alongside the ciphertext. On read, calls KMS to decrypt the DEK (audited); decrypts the file with the DEK. Rotation day: rewrap DEKs under a new KEK version; original ciphertext unchanged; total rotation minutes for millions of files. Every KMS call is audited in CloudTrail / Cloud Audit Logs. Cross-account grant allows an analytics account to decrypt using a specific KEK for a specific role.