AWS KMS architecture — CMKs, envelope, grants, multi-region

Three kinds of KMS key — and only one of them is yours

Every encrypted resource in an AWS account is protected by a KMS key, but the three flavours differ so much in what you can control that treating them as one thing is the first architectural mistake.

Customer managed keys are the ones you create with CreateKey. You own the key policy, the alias, the tags, the rotation setting, the grants, and the decision to schedule deletion. They are the only keys you can audit end to end and the only ones you can revoke. They carry a per-key monthly charge, so key count is a design decision rather than free.

AWS managed keys are created for you the first time you tick "encryption" on a service, and they wear an alias in the reserved aws/ namespace — alias/aws/s3, alias/aws/ebs, alias/aws/rds. You can read their key policy and see their usage in CloudTrail, but you cannot edit the policy, cannot create grants on them, cannot disable them, and cannot delete them. They rotate on AWS's schedule, roughly annually, with no knob. They do not carry the per-key monthly charge; their requests still bill.

AWS owned keys are worse for visibility and better for cost: they live in an AWS-owned account, not yours, they do not appear in ListKeys, their use does not appear in your CloudTrail, and they are free. Several services default to them.

The practical rule: if a compliance answer requires the words "we can revoke it" or "we can prove who decrypted it", the resource must be on a customer managed key. Defaults are not.

KeySpec and KeyUsage — symmetric, asymmetric, and HMAC

A KMS key is defined by two immutable properties chosen at CreateKey time and never changeable afterwards: KeySpec (what the material is) and KeyUsage (what it may be asked to do). Get either wrong and the fix is a new key plus a data migration.

SYMMETRIC_DEFAULT is a 256-bit AES key used in GCM mode, and it is the right answer for approximately everything. It is the only spec that supports envelope encryption via GenerateDataKey, the only one that accepts an encryption context, and the only one eligible for automatic rotation.

Asymmetric specs come in RSA sizes (RSA_2048, RSA_3072, RSA_4096) and elliptic curves (ECC_NIST_P256, ECC_NIST_P384, ECC_NIST_P521, ECC_SECG_P256K1). The private half never leaves KMS; GetPublicKey hands you the public half so that encryption or signature verification can happen offline, with no KMS call and no quota consumption. KeyUsage is ENCRYPT_DECRYPT or SIGN_VERIFY, never both — RSA keys can technically do either, but KMS forces you to declare one, which is the correct cryptographic hygiene.

HMAC_224 through HMAC_512 give you keyed MACs with GenerateMac and VerifyMac under KeyUsage: GENERATE_VERIFY_MAC — useful for tamper-evident tokens where you want the secret to be unextractable rather than sitting in a config store. NIST curves can also be created for KEY_AGREEMENT and used with DeriveSharedSecret.

The trap: asymmetric and HMAC keys do not support automatic rotation, do not accept an encryption context, and their cryptographic operations are billed and rate-limited far more aggressively than symmetric ones. Reach for them only when a counterparty genuinely needs a public key or a signature.

Encrypt versus GenerateDataKey — the 4KB ceiling and what it forces

KMS gives you two entirely different ways to protect bytes, and the boundary between them is a hard size limit that shapes every design above it.

Encrypt sends your plaintext to KMS and gets ciphertext back. For a symmetric key the plaintext is capped at 4096 bytes; for RSA keys the ceiling is much lower still, bounded by the modulus and OAEP padding. That is enough for a password, a token, a config value, or another key — and nothing else. It is a direct-use API, one billable request per operation, with your data crossing the wire.

GenerateDataKey is the API the whole service is built around. It returns a fresh AES key twice: once in plaintext for you to use locally, once wrapped under the KMS key for you to store beside the ciphertext. KeySpec is AES_256 or AES_128, or you can ask for NumberOfBytes of raw material. Your bulk data never touches KMS.

aws kms generate-data-key \
  --key-id alias/prod-payments \
  --key-spec AES_256 \
  --encryption-context tenant=acme,purpose=ledger

Two siblings matter operationally. GenerateDataKeyWithoutPlaintext returns only the wrapped copy — the right call for a write-ahead or delegation path where the process creating the envelope is not the process that will fill it, because it never holds plaintext material it does not need. ReEncrypt rewraps an existing wrapped data key from one KMS key to another entirely server-side, so a key migration never exposes the data key to your application. GenerateDataKeyPair does the same job for asymmetric data keys.

The design consequence is blunt: if you find yourself calling Encrypt in a loop over records, you have built a per-row dependency on a rate-limited network service. Envelope, cache, and move on.

The key policy is the root of trust — IAM alone is never enough

This is the single most AWS-specific fact about KMS, and it inverts the mental model that AWS IAM trains you into. For almost every other resource, an identity policy granting an action is sufficient. For a KMS key it is not. Every KMS key carries a mandatory resource policy — the key policy — and a request is denied unless the key policy allows it, whether or not IAM says yes.

What makes IAM appear to work is one statement in the default key policy:

{
  "Sid": "Enable IAM User Permissions",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
  "Action": "kms:*",
  "Resource": "*"
}

That statement does not grant anyone anything directly. It delegates authorization for this key to the account's IAM, so identity policies in account 111122223333 become capable of granting access. "Resource": "*" inside a key policy means this key, not all keys.

Which is why removing it produces a key that still works and can no longer be administered. KMS tries to stop you: PutKeyPolicy and CreateKey run a lockout safety check that rejects a policy leaving nobody able to manage the key, and overriding it requires actively passing BypassPolicyLockoutSafetyCheck: true. Treat that parameter as a loaded weapon, and deny it outright in an SCP if you can.

Cross-account access is two-sided and people forget the second side. The key policy must name the external account or one of its principals, and the external account must grant its own principal the KMS action in an identity policy. Either alone is a denial. Cross-account callers must also use the full key ARN — an alias in another account will not resolve.

kms:ViaService, kms:CallerAccount, and encryption context as a condition

A key policy that says "this role may Decrypt" is usually far too broad, because the role can then decrypt anything under that key from anywhere. The KMS-specific condition keys are how you narrow it. The general mechanics of the Condition block — operators, IfExists, set operators — belong to AWS IAM conditions; what follows is only what is peculiar to KMS.

kms:ViaService restricts a permission to requests that an integrated AWS service makes to KMS on the principal's behalf, with values shaped like s3.us-east-1.amazonaws.com or ebs.eu-west-1.amazonaws.com. This is the workhorse: it lets a role read encrypted objects through S3 while being unable to call Decrypt directly with a stolen ciphertext blob. Note the corollary — a policy gated only on ViaService denies your own direct API calls, which is usually what you want and always what confuses people the first time.

kms:CallerAccount evaluates to the account of the calling principal, which is how you keep a service-mediated grant from being used by an identity in another account.

Encryption context is the second lever. It is additional authenticated data bound into the AES-GCM operation: not secret, echoed into CloudTrail, and required to match exactly on decrypt. Because it appears as kms:EncryptionContext:<key> and kms:EncryptionContextKeys, it is also a policy condition.

"Condition": {
  "StringEquals": {
    "kms:ViaService": "s3.us-east-1.amazonaws.com",
    "kms:CallerAccount": "111122223333",
    "kms:EncryptionContext:tenant": "acme"
  }
}

Because condition values in string comparisons accept IAM policy variables, that last literal can become ${aws:PrincipalTag/tenant}, and one statement then isolates every tenant on a single shared key: a ciphertext written under one tenant's context is undecryptable by a principal tagged for another, and the mismatch lands in CloudTrail with both values attached. What it buys you is per-request separation, not per-tenant lifecycle — disabling or destroying the key still takes every tenant with it, so anything promising per-customer erasure or revocation needs its own key.

Grants, grant tokens, and two different ways to take one back

Grants are the AWS-specific authorization mechanism that never appears in a policy document, which is why a permissions audit that reads only key policies and IAM is incomplete. Run ListGrants before you claim to know who can decrypt something.

A grant is a narrow, programmatic delegation created with CreateGrant. It names a GranteePrincipal, an optional RetiringPrincipal, a list of Operations (Decrypt, GenerateDataKey, ReEncryptFrom, CreateGrant itself, and so on), and optional Constraints — either EncryptionContextEquals for an exact match or EncryptionContextSubset for a required subset. Grants are additive: they can only widen access, never deny.

Most grants in a real account were not created by you. When you attach an encrypted EBS volume, create an encrypted RDS instance, or set encrypted Lambda environment variables, the service creates a grant on the key so it can keep decrypting for the life of that resource, and retires the grant when the resource is deleted. The condition key kms:GrantIsForAWSResource lets you allow exactly this pattern — services may create grants on your behalf, humans may not.

CreateGrant returns a GrantId and a GrantToken. The token exists because grants are eventually consistent: a freshly created grant may not be visible to the very next Decrypt. Pass the token in that call's GrantTokens parameter and the operation succeeds immediately instead of failing on a race that only shows up under load.

Removal has two verbs with different owners. RevokeGrant is the key administrator taking access away by GrantId. RetireGrant is the grantee or the retiring principal voluntarily giving it up, usable with either the grant token or the grant ID. Both remove the grant; only one of them is a security action.

Aliases, key ARNs, and what rotation actually changes

A KMS key has three identifiers and applications should almost always use the third. The key ID is a UUID. The key ARN — arn:aws:kms:us-east-1:111122223333:key/1234abcd-... — embeds the account and Region and is what ciphertext metadata, snapshots and resource configurations pin themselves to, permanently. The alias is a mutable pointer, alias/prod-payments, that you can repoint with UpdateAlias to a different key without touching a line of application code. Aliases are per-Region and per-account, and kms:RequestAlias and kms:ResourceAliases let you write policy against them.

Now the part that gets misremembered. Enabling automatic rotation with EnableKeyRotation adds a new backing key and marks it current. It does not change the key ID, the key ARN, the alias, the key policy, the grants, or anything already written. Old backing keys are retained forever; each ciphertext records which one produced it, and KMS reaches for the right one on decrypt. Rotation re-encrypts nothing. You can also set a custom RotationPeriodInDays with a floor of ninety days rather than the default year, force an immediate rotation with RotateKeyOnDemand, and inspect what has happened with ListKeyRotations.

Automatic rotation applies only to symmetric encryption keys with origin AWS_KMS. Imported material, asymmetric keys, HMAC keys and keys in custom key stores are all excluded — for those, "rotation" means creating a new key and repointing the alias.

That alias-repoint path is also the only one that retires material rather than merely adding some, and on AWS it has a concrete shape: CreateKey, UpdateAlias, then a ReEncrypt sweep over the stored wrapped data keys, which rewraps them server-side without your application ever holding plaintext. Budget for it, because it is a real migration and not a checkbox.

Multi-Region keys — one key material, several independent resources

KMS keys are regional, and the classic disaster-recovery failure is not missing data but a ciphertext that names a key ARN in a Region the restore target cannot reach. Multi-Region keys exist for exactly this.

Create one with CreateKey and MultiRegion: true; its key ID is prefixed mrk-. ReplicateKey then creates a replica in another Region that shares the same key material and the same key ID — the ARNs differ only in the Region field. Ciphertext produced in one Region decrypts in another with no re-encryption step, which is what makes cross-Region client-side encryption, DynamoDB global tables and cross-Region backup copies work cleanly.

What people over-read is the word "same". Only the material and the key ID are shared; policy, grants, aliases, tags and enabled state are per-replica, so a role granted Decrypt in us-east-1 has nothing in eu-west-1 and disabling the primary leaves the replicas happily decrypting. Rotation is the exception, coordinated from the primary so the material stays in step. UpdatePrimaryRegion promotes a replica when you need to move the primary.

You also cannot change your mind: a single-Region key can never become multi-Region, and vice versa. Decide at creation. Deletion runs in dependency order — you must schedule the replicas away before the primary can go, and the primary's own deletion will not complete while a replica exists.

The condition keys kms:MultiRegion and kms:MultiRegionKeyType let a service control policy or SCP forbid multi-Region keys entirely where data residency rules require it, or require them where DR does.

Deletion, the waiting window, and disable as the reversible test

Deleting a KMS key is the only action in an AWS account that can permanently destroy data you still hold. Every snapshot, backup, archive and analytics extract encrypted under it becomes inert at once, and there is no escrow copy — AWS Support cannot recover it, by contract.

So ScheduleKeyDeletion does not delete anything. It moves the key to PendingDeletion for a PendingWindowInDays you choose between 7 and 30, defaulting to 30. During the window the key is unusable, so everything that depends on it fails loudly, which is the entire point: the window exists so alarms fire while CancelKeyDeletion is still an option.

Do not use it as the test, though. DisableKey produces the same breakage signal instantly and takes it back instantly with EnableKey, so the safe order on AWS is DisableKey first, then a CloudTrail query proving no Decrypt, GenerateDataKey or DescribeKey event against that key ARN over a window long enough to include your slowest periodic job, and only then ScheduleKeyDeletion.

Two AWS-specific wrinkles. For imported key material, DeleteImportedKeyMaterial removes the material immediately with no waiting window — a genuinely fast crypto-shred, and a genuinely fast outage. And deleting an alias with DeleteAlias deletes only the pointer; the key and everything it protects are untouched, which surprises people in both directions.

The AWS-specific safety net is EventBridge: KMS emits an event when a key enters pending deletion, and a rule on it plus a CloudTrail alarm on ScheduleKeyDeletion and PutKeyPolicy is a few minutes of setup against a class of loss that has no recovery path.

Custom key stores — CloudHSM-backed and external

By default KMS holds your key material in its own multi-tenant HSM fleet. A custom key store moves it somewhere you control, and you inherit the availability of wherever you moved it.

An AWS CloudHSM key store (CustomKeyStoreType: AWS_CLOUDHSM) backs KMS keys with material in your own CloudHSM cluster. The prerequisites are specific: an active cluster in the same account and Region, at least two HSMs in different Availability Zones, the cluster's trust anchor certificate, and a dedicated kmsuser crypto user whose password you hand to KMS. Keys created there report Origin: AWS_CLOUDHSM. The API surface your applications call is unchanged — same GenerateDataKey, same key ARN shape, same CloudTrail — which is the whole appeal.

An external key store (EXTERNAL_KEY_STORE) goes further: the material never enters AWS at all. KMS forwards every cryptographic operation through an XKS proxy to your own key manager. This is what a regulator means by hold-your-own-key, and it is real — AWS cannot decrypt without a round trip you can observe and refuse.

The costs are not subtle. Both types support symmetric encryption keys only — no asymmetric, no HMAC, no automatic rotation. Both make an external system a hard synchronous dependency of every decrypt in the Region: if the cluster loses quorum, if the kmsuser password is rotated out from under KMS, or if the XKS proxy is unreachable, the key store disconnects and every operation under those keys fails. And CloudHSM bills per HSM-hour, which is a different order of magnitude from a per-key monthly charge.

Choose a custom key store when a written control requires it. Choosing it for a general feeling of extra safety trades a well-run AWS dependency for one you now have to run yourself.

Quotas, throttling, and the cost drivers you can actually move

KMS enforces a request-rate quota on cryptographic operations that is shared per account, per Region, across every caller — your code, and every AWS service calling on your behalf. That sharing is the trap. Symmetric operations get a generous shared budget; RSA asymmetric operations get a budget lower by orders of magnitude; management APIs such as CreateKey, CreateGrant and ScheduleKeyDeletion have their own, much smaller, limits. Look up the current numbers for your Region rather than assuming — they differ, and the largest Regions are higher than the rest.

Throttling also rarely presents as a KMS error. It presents as EC2 instances failing to launch, Lambda functions failing to decrypt their environment variables, S3 returning 5xx and RDS connections failing — a spray of unrelated symptoms whose only common factor is the account, with the ThrottlingException buried several layers below the code you are reading.

The AWS levers, in order of effect: turn on S3 Bucket Keys, which collapse per-object GenerateDataKey calls into a bucket-level data key (see S3 encryption options); adopt the AWS Encryption SDK's caching CMM so a bounded-TTL data-key cache absorbs the read burst; and only then open a Service Quotas request, which is the slowest lever and the one that does nothing for the next hour.

Cost tracks the same shape. You pay a monthly charge per customer managed key — each multi-Region replica counts as its own key — plus a per-request charge that is higher for asymmetric and HMAC operations, plus CloudHSM hours if you run a custom key store. There is a fourth line nobody budgets for: KMS data-plane calls are logged as CloudTrail management events, and on a busy account that log volume can cost more than the KMS usage itself. You can exclude kms.amazonaws.com from a trail if you have decided you do not need it — decide deliberately, because it is the only record of who read your data.