Skip to main content

Documentation for Tutela v3.9.0.

Low cache hit rate

Use this runbook when cache misses increase or the cache hit-rate alert fires. A cold cache after a restart can be normal; sustained misses can add latency and load to downstream services.

Prerequisites​

  • An administrator who can describe the affected workflow and when it became slower.
  • A deployment operator with Grafana access and the time of any recent restart or configuration change.

Investigate​

  1. Check whether users are seeing slow requests or errors. If only the metric changed, compare it with traffic volume before making a configuration change.

  2. In Grafana, compare hits and misses for each service and key prefix:

    sum(rate(tutela_cache_hits_total[5m])) by (service, key_prefix)
    /
    (
    sum(rate(tutela_cache_hits_total[5m])) by (service, key_prefix)
    + sum(rate(tutela_cache_misses_total[5m])) by (service, key_prefix)
    )

    The shipped warning requires a ratio below 0.8 for ten minutes and more than 0.1 cache lookups per second. Inspect your installed rule if it has been customized. A quiet cache or absent metrics do not establish a failure.

  3. Compare the onset with a restart, an intentional invalidation, a new workload, or a cache configuration change. Separate a temporary cold cache from misses that persist after repeated requests.

  4. Compare cache dependency latency, error rate and memory pressure over the same interval. Determine whether the affected service can reach its configured cache.

Correct the cause​

FindingAction
Cold cache after a restartWatch the hit rate under normal repeated traffic. Avoid restarting again or flushing the cache, which starts warming over.
New requests reuse few keysCompare the workload with the earlier baseline. Low reuse alone does not justify increasing cache capacity.
Cache dependency errors or delayFollow the corresponding cache error or latency runbook before changing expiration settings.
Misses began after an expiration or key configuration changeReview the changed setting with its owner. Restore the previous supported value if it caused the regression.
Evictions or exhausted capacityHave the operator assess memory and the installed eviction policy before resizing.

Record previous settings and change one variable at a time. Cache flushing removes useful entries and can increase downstream load; it is not a general recovery step.

Expected outcome and recovery​

Repeated requests reuse cached results where expected, latency returns to its prior range at comparable load, and dependency errors do not increase. For a workload with little reuse, record the new baseline and have the monitoring owner review the alert rather than manufacturing cache hits.

Support handoff​

Escalate persistent low reuse accompanied by customer delay or dependency pressure. Include the affected service/key prefix, hit and miss rates, traffic change, restart time and cache error/latency observations. Do not include cached customer values.

Next steps​