Documentation for Tutela v3.9.0.
Cache dependency errors
Use this runbook when cache operations fail or the Redis error-rate alert fires. Errors may affect performance, rate limits or other controls; identify the caller and operation before assessing customer impact.
Prerequisites
- Access to Service health and the affected workflow's time and error.
- An operator with cache/client monitoring and authority to inspect the configured cache service.
Investigate
-
Check Service health and whether users see errors, delay or a warning that inspection could not complete.
-
In Grafana, compare each caller's error fraction:
sum(rate(tutela_redis_operations_total{status="error"}[5m])) by (service)/ sum(rate(tutela_redis_operations_total[5m])) by (service)The shipped warning is above 5% for five minutes. Compare operation counts as well as the ratio; a handful of errors at low traffic can produce a high percentage.
-
Group the error counter by service and operation to identify which caller is affected. Inspect its redacted error: authentication, connection timeout and server capacity errors require different fixes.
-
For AWS deployments using ElastiCache, have the operator inspect that cache's status, network access and connection/memory metrics. Do not look for a local Redis container when the deployment uses a managed service.
-
For Helm, inspect the installed Redis workload only if the release provisions one. For an external cache, use its operator's monitoring and access controls instead.
Correct the cause
| Finding | Action |
|---|---|
| Authentication or TLS mismatch after a change | Restore the approved credential/certificate configuration through the deployment's secret/configuration process. |
| Connection timeout | Check caller-to-cache network reachability and the configured endpoint. |
| Exhausted connections or client pool | Compare the caller's concurrency with pool/server limits; reduce avoidable load while sizing the affected limit. |
| Cache memory or availability problem | Repair or resize the cache through its supported deployment procedure. |
Record the previous configuration before changing it and restore it if the change worsens the incident. Do not flush the cache or disable a control to conceal a dependency error.
Expected outcome and recovery
The affected operation succeeds, its error fraction stays below threshold at representative load, and the original customer workflow succeeds. If outage handling permitted traffic to continue, verify the control itself is working again using Inspection dependency unavailable.
Support handoff
Escalate persistent dependency failures or any uncertain enforcement outcome. Include caller, operation, error category, rate/volume, cache deployment type and recent endpoint/credential changes without sharing secrets.