Why it matters
Encryption at rest is table stakes for enterprise HDFS. Regulators expect it, security teams require it, and customer contracts often mandate it. Without HDFS Encryption Zones, the only way to encrypt data was full-disk encryption at the DataNode level, which is coarse-grained, does not survive backup workflows, and does not protect against attackers who compromise the OS but not the disks.
Zone-level encryption is finer grained: different sensitivity levels can live in different zones with different key rotation schedules, different KMS backends, and different access policies. This gives security teams the flexibility to match encryption to actual risk instead of applying a blanket policy.
The architecture
An encryption zone is an HDFS directory declared as encrypted. Every file created under an encryption zone gets its own Data Encryption Key generated at file creation time. The DEK encrypts the file's bytes on the client using AES-CTR. The DEK itself is then wrapped by the Encryption Zone Key, and the wrapped DEK is stored as an extended attribute on the file's inode.
The EZ Key never leaves the KMS. When a client wants to decrypt a file, it fetches the file's wrapped DEK from HDFS, sends it to KMS along with proof of authorization, and receives the unwrapped DEK back. All decryption happens client-side using the unwrapped DEK.
How it works end to end
Creating a new file in an encryption zone works like this: the client asks the NameNode to create the file, the NameNode calls KMS to generate a new DEK wrapped with the current EZ key, the wrapped DEK is stored as an xattr on the inode, and the raw DEK is returned to the client. The client uses the raw DEK to encrypt bytes locally before streaming them to DataNodes, so the DataNodes only ever see ciphertext.
Reading is symmetric: the client fetches file metadata including the wrapped DEK, sends the wrapped DEK to KMS asking for it to be unwrapped, and receives the plaintext DEK back. The client then reads ciphertext from DataNodes and decrypts locally.
Key rotation is straightforward for the EZ Key: a new version is generated in KMS. Existing files continue to work because their wrapped DEKs reference the old EZ Key version. New files are wrapped with the new EZ Key version. Optional re-encryption walks the zone and updates every wrapped DEK to the newest EZ Key version.