Why it matters
Skipping upgrades to avoid the operational effort is how clusters end up years behind on patches. Old versions accumulate CVEs, become unsupportable, and eventually force a painful greenfield migration. Rolling upgrade turns each version bump into a routine operation instead of an event.
Rolling upgrade also enables faster experimentation with new features. Teams that can upgrade in a day can try new versions in staging weekly; teams that need a maintenance window upgrade twice a year.
The architecture
The upgrade sequence starts with HDFS. In an HA cluster, the standby NameNode is upgraded first. Because the standby is not serving clients, upgrading it is safe: it comes back up on the new version, catches up on edit log, and rejoins as standby. Then an admin-initiated failover moves the active role to the new-version NameNode. The old active is upgraded next.
DataNodes upgrade in small batches. Take one or a few DataNodes offline, upgrade them, restart, and confirm they rejoin healthy. Move to the next batch. Replication protects data during this: any block temporarily missing a replica gets re-replicated automatically.
How it works end to end
YARN upgrades next. The standby ResourceManager comes up on the new version, failover happens, old active upgrades. NodeManagers upgrade in batches. Running applications survive the NM restart if the AM is checkpoint-enabled and can reconnect; if not, they may need to restart their containers.
HBase, Hive, and other services follow their own rolling upgrade playbooks. HBase uses region migration to keep the cluster available: regions on the upgrading RegionServer move to peers, the RegionServer upgrades, then regions rebalance back.
The finalize step commits the upgrade. Until you finalize, HDFS keeps a copy of pre-upgrade metadata that lets you roll back. Finalize deletes this copy and reclaims space. Only finalize after enough runtime on the new version that you are confident about rollback readiness — typically several days to weeks.