Documentation for Tutela v3.9.0.
Cache dependency latency
Use this runbook when cache operations are slow or their P95 latency alert fires. A slow caller pool, network path and cache server can produce similar symptoms; increasing server capacity only helps some of them.
Prerequisites
- An affected request time and access to Service health.
- An operator with caller/cache monitoring and the installed cache deployment details.
Investigate
-
Check whether customer request latency rose at the same time as cache latency. Identify the affected caller rather than averaging unrelated services together.
-
In Grafana, inspect P95 cache operation time in milliseconds:
histogram_quantile(0.95,sum(rate(tutela_redis_operation_duration_ms_bucket[5m])) by (le, service))The shipped warning is above 50 ms for five minutes. Compare P50, operation rate and error rate over the same window. A high P95 with few operations needs careful interpretation.
-
Compare caller CPU/memory, connection-pool pressure where that caller exposes it, and cache server connections, memory and CPU. Missing pool metrics for a caller are not proof that its pool is healthy.
-
For AWS ElastiCache, use the managed cache's monitoring and network configuration. For Helm, distinguish a chart-provisioned cache from an externally managed cache before inspecting resources.
Correct the cause
- Only one caller is slow: inspect its resource limits, request concurrency and pool configuration before resizing the shared cache.
- All callers are slow with server pressure: reduce avoidable load and have the cache operator assess supported capacity changes.
- Network delay or timeouts: check the caller-to-cache path and recent endpoint/network changes.
- Latency accompanied by errors: follow Cache dependency errors and fix the error cause first.
Record prior settings. Reverse an unsuccessful pool or capacity configuration change through the deployment's supported mechanism. Do not flush data as a generic latency remedy.
Expected outcome and recovery
The affected caller's P95 falls below its configured threshold under comparable load, customer request latency improves, and error rates do not increase. Restore deferred work gradually while watching both caller and server measurements.
Support handoff
Escalate persistent latency that interrupts work or affects multiple callers. Include P50/P95, sample volume, caller/resource observations, cache deployment type and the time of any recent changes.