Why it matters
Production HBase issues are stressful precisely because the failure modes are non-obvious. A slow query can be caused by GC, HDFS, cache miss, or a hot region — all indistinguishable at the query level. Systematic troubleshooting saves both time and stress.
Documenting a troubleshooting playbook also enables knowledge transfer. Every team member can respond to common issues without deep expertise.
The architecture
Slow read symptoms decompose into: block cache miss (fix by growing cache); high HFile count per region (fix by triggering compaction); slow HDFS (fix at HDFS layer); GC pauses on the RegionServer (fix by tuning JVM). Check each in order and identify which is dominant.
Slow write symptoms decompose into: slow WAL sync (usually HDFS or disk issue), memstore full blocking (increase memory or add regions), hot row contention (redesign row keys). Metrics differentiate them: WAL sync latency, memstore-full-block counter, per-row RPC latency.
How it works end to end
Failed operations usually mean regions are in transition or a RegionServer is unhealthy. Check the HMaster UI for regions-in-transition count. If nonzero, investigate why they are stuck: RegionServer death, split failure, corrupt HFile.
Stuck regions from RegionServer failure need HMaster reassignment. Usually automatic; if not, hbase shell 'assign' can force it.
Cluster-wide slowness with no clear per-component signal often means ZooKeeper. Check ZK request latency and quorum health.
Individual RegionServer death: check logs for OutOfMemoryError, RegionServerAbortedException, or explicit shutdown. Root cause is usually GC or manual intervention.