Latency
Read and write latency metrics are foundational to understanding cluster health. Cassandra exposes percentile-based histograms—p50 (median), p95 (95th percentile), and p99 (tail latency)—for both read and write operations. A rising trend across any percentile signals investigation is needed. P50 increases indicate baseline slowdown, often from node saturation or GC pauses. P99 increases are more concerning because they reflect the worst-case user experience—even if 95% of requests are fast, tail latencies translate directly to slow page loads or timeouts for affected users.
Compare latencies across nodes to identify outliers. If one node consistently shows higher latency than peers, it may have a faulty disk, higher GC pressure, or be running slower hardware. Local vs cross-DC latencies also matter—cross-DC requests are inherently slower due to network distance and replication overhead. Set baseline targets for your workload (e.g., p99 read < 50ms, p99 write < 20ms) and alert when they're exceeded for sustained periods. Track latency with and without read repair enabled, as read repair adds blocking overhead during inconsistency detection.
Pending Compactions
Compaction is Cassandra's background process for merging SSTables, reclaiming disk space, and improving read performance. Monitor pending compactions using nodetool compactionstats. This shows the current number of tasks in the compaction queue waiting to execute. Rising compaction queue depth indicates the node is falling behind—new SSTables are accumulating faster than compaction can merge them. A queue of 10–50 is normal during write-heavy loads. Over 100 tasks indicates serious backlog and demands investigation.
High compaction backlogs delay garbage collection of deleted data (tombstones) and can eventually block writes if the node runs out of disk space. Root causes include: insufficient compaction throughput (tune compaction_throughput_mb_per_sec in cassandra.yaml), suboptimal compaction strategy (STCS vs LCS vs TWCS), or inadequate hardware resources. Monitor compaction duration and output to ensure tasks complete in reasonable time. Long-running compactions can spike read latency during the merge phase. Consider reducing write throughput, adding nodes, or upgrading disk to faster hardware if compaction chronically lags.
Dropped Mutations
Dropped mutations represent writes that were accepted by the coordinator but then rejected before being sent to replicas. This happens when the coordinator node is overloaded and cannot keep up with incoming write traffic. A non-zero dropped mutation count signals the cluster is under severe stress. Unlike timeouts (which the client sees immediately), drops are silent from the coordinator's perspective—the write is lost unless the client detects the problem through read-your-writes verification or a later timeout.
Monitor dropped mutation counters for each node. A single spike during peak load may be acceptable, but persistent or growing counts indicate capacity is insufficient. Mitigation strategies include: reducing write throughput from clients (implement backpressure), adding nodes to distribute the load, tuning heap size and thread pools to handle more concurrent writes, or implementing write-rate limiting in the application. Also check for network issues—if the coordinator cannot reach replicas, it may drop writes to avoid queue explosion. Review write_request_timeout_in_ms (default 2 seconds); timeout values too short can trigger unnecessary drops.
JMX vs Prometheus → Metrics Collection Architecture
Cassandra exposes operational metrics through two primary mechanisms: Java Management Extensions (JMX) and direct Prometheus scraping. JMX is the traditional approach—it provides real-time access to MBeans (managed beans) representing internal Cassandra state. Tools like Nodetool connect via JMX to query metrics, trigger operations (like compaction or repair), and inspect cluster health. The data is fetched on-demand and reflects the current instant, making JMX ideal for ad-hoc investigation and immediate operational control.
Prometheus, by contrast, scrapes HTTP endpoints at regular intervals (typically every 15–60 seconds), collecting time-series data for historical analysis, trend detection, and alerting. Modern deployments often use both: JMX for interactive CLI operations and debugging, and Prometheus for long-term monitoring, correlation, and trend analysis. Prometheus enables retroactive investigation (e.g., "Was there a latency spike at 3am?"), alerting on threshold breaches or anomalies, and dashboard-based visualization. The trade-off is data freshness—Prometheus data lags behind real-time by at least the scrape interval, and retention is limited by storage capacity. Configure Prometheus scrape intervals based on your sensitivity; tighter intervals increase storage overhead but improve detection responsiveness.
Memory and Garbage Collection Metrics
Cassandra's JVM memory usage directly impacts throughput and latency. Key metrics include heap size (total and used), memtable consumption, and garbage collection pause times. A steadily rising heap trend signals load increase or a potential memory leak—investigate before it causes cascading GC failures. Track old-generation (heap) memory separately from young-generation; young GC collections are fast (typically 10–50ms) and acceptable, but full GC collections pause all threads and can spike read/write latency by hundreds of milliseconds.
Monitor GC metrics using Prometheus exporters or directly via JMX: G1GC collection counts, collection durations, and pause times. Aim for young GC pauses under 50ms and rare or absent full GC events. During off-peak windows, trigger nodetool drain followed by nodetool flush to reduce in-flight memtable data and minimize surprise GC freezes during high traffic. Tune heap size via -Xmx flag in cassandra-env.sh; oversizing wastes memory and increases GC duration, while undersizing causes frequent GC and reduced performance. A typical rule is 4–8 GB for small-to-medium clusters; large clusters may need 16–32 GB depending on working set size.
Disk I/O and Storage Metrics
Disk I/O metrics reveal whether writes are backlogging or reads are hitting expensive disk seeks. Track SSTable read rates, flush rates (memtable → disk), and compaction throughput. Rising disk latency (p99 read times) often precedes visible query slowdowns. Monitor free disk space aggressively—Cassandra requires headroom for compaction to proceed safely. When disk space drops below 25% capacity, compaction may stall, causing SSTables to accumulate and making the cluster unavailable or triggering write timeouts.
Watch disk utilization percentage and set alerts: warning at 70%, critical at 85%. For write-heavy workloads, ensure the commit log disk is separate from SSTables and has dedicated throughput; commit log stalls block all writes cluster-wide. Use tools like iostat to profile disk performance; identify whether I/O is limited by throughput (MB/s) or IOPS (operations/sec). SSDs are strongly recommended for production to achieve consistent low latency; spinning disks introduce unpredictable seek delays that spike tail latencies. Monitor compaction metrics alongside disk I/O to correlate stalls and plan capacity upgrades proactively.
Network and Replication Metrics
Multi-node clusters depend on stable inter-node communication. Monitor bytes sent and received between nodes, tracking replication stream throughput. Sudden drops in inter-node traffic may indicate network issues, silent node failures, or connectivity problems. For multi-data-center setups, track cross-DC replication lag—if replication from the primary DC to a remote DC lags significantly, consistency guarantees weaken and repair becomes necessary to reconcile divergent state.
Watch for dropped messages due to backlog; non-zero dropped message counts indicate the remote node cannot keep up with inbound traffic and should be investigated or scaled. Network bandwidth between DCs is often a bottleneck; prioritize compressing replication traffic and tuning internode_compression in cassandra.yaml. Monitor connection pool exhaustion—if inter-node connection pools are exhausted, new inter-node requests queue or fail. For multi-DC deployments, consider asymmetric replication (heavy traffic toward one DC) and use nodetool status to verify rack distribution and replication balance across DCs.
Cache Hit Ratios — Row and Key Caches
Cassandra caches frequently accessed data to reduce disk I/O. Row cache holds full rows in memory, while key cache tracks SSTable block locations and partition key offsets. High row cache hit rates (> 80%) indicate the working set fits in cache and many reads avoid disk, keeping latency low. Low hit rates suggest either the dataset is larger than cache capacity or access patterns are random (cache unfriendly). Key cache miss rates above 20% indicate the node is frequently searching SSTable indexes, which adds latency.
Tune cache sizes based on workload. For workloads with temporal locality (e.g., time-series data where recent time ranges are queried), row cache provides value. For workloads with uniform or random access, key cache is more cost-effective. Monitor cache eviction rates—high eviction counts mean the cache is too small and thrashing, incurring overhead without benefit. Use Prometheus metrics like cache_hits and cache_misses to compute hit ratios. Adjust row_cache_size_in_mb and key_cache_size_in_mb in cassandra.yaml, then benchmark to find the optimal point between memory usage and hit rate improvement.
Thread Pool Metrics — Throughput and Queue Depth
Cassandra uses dedicated thread pools for reads, writes, internal tasks, and materialized view updates. Monitor pool size, active thread count, and queue depth for each pool. A rising read pool queue depth indicates read throughput is constrained—the pool cannot process requests fast enough, and clients experience increased latency. Similarly, a growing write pool backlog means write latency will climb and mutations may eventually be dropped if queues overflow.
Queue depth trending above 50% of the pool size suggests capacity headroom is shrinking and you're approaching a wall. Adjust pool sizes via cassandra.yaml, but be aware that oversized pools waste memory and increase context switching overhead. The sweet spot typically requires experimentation; small pools maximize cache efficiency but risk throughput loss, while large pools improve throughput but increase latency variance. Monitor also for thread pool starvation—if all threads in a pool are blocked waiting for I/O or locks, new requests queue indefinitely. For CPU-bound workloads, set pool sizes to 2x the number of CPU cores; for I/O-bound workloads, larger pools are acceptable.
Hinted Handoff and Anti-Entropy Metrics
Hinted handoff (hints queued and delivered) reveals how many writes were accepted for temporarily unavailable nodes. When a write is accepted at a quorum but a target replica is down, Cassandra stores a "hint" on another node and replays it once the replica recovers. A high hint count indicates frequent node downtime or network flakiness. Once hints accumulate beyond a threshold (typically 3 hours of storage), they expire and are discarded, risking data loss for that write.
Anti-entropy repair metrics show how many token ranges were repaired, how long repair took, and whether mismatches were found. A cluster with few repair mismatches suggests good health; many mismatches indicate either recent node failures or insufficient repair frequency. Schedule repairs regularly (weekly or bi-weekly depending on failure rates and data TTLs) to keep replicas consistent. Monitor repair_jitter_ms to stagger repair operations across the cluster and avoid thundering-herd effects. For large clusters, use subrange repairs (nodetool repair -pr for local ranges) or incremental repair to minimize impact on production traffic.
Read Repair and Reconciliation Metrics
Read repair tracks how often reads trigger consistency checks across replicas and how often discrepancies are discovered and fixed. A read repair rate above 5% of total reads suggests replicas are diverging frequently—investigate hinted handoff status, repair frequency, or recent node failures. Each read repair blocks the client briefly (a few milliseconds) to apply fixes in the background, so high rates increase read tail latency. Disable read repair if replication factor is 1 or if you rely exclusively on scheduled repair; enable it at QUORUM or ALL consistency levels for data-critical operations where consistency is paramount.
Monitor the reconciliation overhead using JMX metrics. Track read repair on a per-keyspace basis to identify which tables generate the most repairs. If certain tables show high repair rates, consider improving data model design (e.g., reducing partition size) or increasing repair frequency for those tables. Adjusting read_repair_chance in cassandra.yaml controls the probability of read repair being triggered; set it to 0 if you repair regularly, or increase it temporarily if investigating consistency issues.
Tombstone and Deletion Metrics
Tombstones mark deleted data but consume disk space and slow reads until compaction removes them. Monitor tombstone ratios (tombstones per partition) and count them during compaction operations. A high ratio indicates frequent deletes or TTL expiration accumulating faster than compaction clears them. Compaction with high tombstone density can trigger read timeouts during the merge phase, particularly if the partition is large and compaction requires examining millions of tombstones.
To reduce tombstone overhead, ensure TTL values are appropriate for your data model—avoid setting TTL to far-future dates if deletions are frequent. Consider using time-window compaction strategy (TWCS) for time-series data where entire partitions expire together and can be dropped en masse. For explicitly deleted data, monitor deletion patterns; if many deletes happen shortly after inserts, consider redesigning the schema to avoid explicit deletes (e.g., use flags or soft-delete patterns). Use nodetool garbagecollect -z to estimate tombstone removal potential before manual compaction.
Request Timeout and Failure Metrics
Timeouts occur when Cassandra nodes cannot respond within configured thresholds: read_request_timeout_in_ms (default 5 seconds) or write_request_timeout_in_ms (default 2 seconds). Rising timeout rates indicate the cluster is under stress—latency is exceeding thresholds and clients are experiencing errors. Distinguish between read timeouts (slow disk, high GC, or slow network), write timeouts (overloaded coordinators, full commit logs, or replica unavailability), and unavailable exceptions (insufficient replicas online to meet quorum).
Unavailable exceptions are most critical—they indicate the cluster cannot satisfy consistency requirements and demand immediate investigation. Read timeouts often signal slow disk or high GC pause times; profile GC and I/O patterns. Write timeouts typically result from overloaded coordinators or the commit log filling up. Increase timeout values only as a last resort; they mask underlying problems and delay problem discovery. Instead, investigate root causes: add capacity, tune compaction, optimize GC, or reduce write throughput from clients. Correlate timeout spikes with compaction queue depth, dropped mutations, and latency percentiles to diagnose the cause.
Monitoring Tools and Integration Patterns
Common tools for Cassandra monitoring include Prometheus + Grafana for time-series visualization, Datadog for managed observability, New Relic, and Cassandra's built-in Nodetool for CLI inspection. Configure Prometheus scrape intervals (typically 15–60 seconds) to balance data freshness and storage overhead. A 15-second interval generates 5,760 data points per metric per day; for a 100-metric cluster, that's 576,000 points daily. Use 60-second intervals for non-critical metrics to reduce storage. Set up Prometheus recording rules to compute 5-minute and 15-minute rolling averages, enabling faster alerting on sustained trends without reacting to momentary spikes.
Integrate with alerting systems (Alertmanager for Prometheus, PagerDuty, Slack webhooks) to notify on-call engineers when thresholds are breached. Create dashboards showing cluster-wide aggregates (total throughput, average latency across nodes) and per-node breakdowns (disk usage, memory, GC pause times). Segment metrics by read/write operations and internal vs client-facing traffic. Build separate dashboards for operators (infrastructure health) and application teams (throughput and latency trends). Export alerting rules as code (Terraform, Helm) to ensure consistency across environments and version control alert definitions.
Alert Thresholds and Best Practices
Define alerts based on operational impact, not arbitrary thresholds. Critical alerts should trigger immediate paging: compaction queue > 100 tasks (cluster falling behind), pending mutations > 50 or growing (writes being dropped), any node unavailable (quorum at risk), disk > 90% (risk of cluster becoming unavailable), or GC pause > 500ms (widespread latency impact). Warning-level alerts notify the team but don't page: compaction queue > 50, disk > 70%, p99 latency > 100ms, cache hit ratio < 50%, or cross-DC replication lag > 1 minute. Implement gradual escalation—initial alerts notify the team, persistent violations trigger page-outs to on-call.
Set alert windows to avoid noise; require conditions to hold for 5 minutes before firing to avoid reacting to transient spikes. Document runbooks for each alert, specifying investigation steps and mitigation actions. Regularly review alert history and tune thresholds based on false positives and missed issues. Combine metrics with application-level observability (request tracing, slow-query logs, error rates) to correlate database issues with user impact. Set up synthetic tests (e.g., periodic read/write operations) to detect cluster issues before they affect real traffic. Establish a post-mortem process for outages to refine alert sensitivity and prevent recurrence.