Skip to main content

Documentation for Tutela v3.9.0.

Inspection dependency unavailable

Use this runbook when a cache-dependent control cannot operate normally or the Redis fail-open alert fires. Some controls may allow work to continue; others use a limited local fallback. Continued requests do not prove that the original control was applied.

Prerequisites​

  • The affected client, service, request time and any visible outage message.
  • An administrator who can identify the intended control and its outage behavior.
  • An operator with dependency and control metrics.

Investigate​

  1. Check Service health, then identify which control and caller are affected. Ask the administrator to confirm the effective outage setting for this connection.

  2. In Grafana, inspect the rate of fail-open events:

    rate(tutela_redis_fail_open_total[5m])

    Shipped thresholds are above 0.1 per second for two minutes (warning) and above 1 per second for one minute (critical).

  3. Identify the operation and fallback mode:

    sum(rate(tutela_redis_degraded_control_total[5m])) by (service, operation, mode)

    Distinguish allowing an operation without its usual dependency check from applying a bounded local alternative. The fallback is not equivalent to normal distributed enforcement.

  4. Compare cache errors and latency with the incident window. Inspect representative request evidence without assuming an allowed request was fully checked.

Correct the cause​

  • Repair authentication, connectivity or cache capacity using Cache dependency errors or Cache dependency latency.
  • Identify the period and workflows that used fallback handling. Have the control owner assess whether work should be paused while the dependency is repaired.
  • Keep the configured policy posture unless its owner explicitly authorizes a scoped temporary change. Record its previous value, expiration and restoration step.
  • Do not change every client to fail-open as a dependency repair, and do not describe fallback decisions as equivalent to normal inspection.

Expected outcome and recovery​

The dependency is healthy, new fallback events stop at representative load, and a synthetic request demonstrates the intended control again. Check both a permitted case and the control's expected rejection case using approved synthetic inputs. Restore any temporary exception and repeat the checks.

Support handoff​

Escalate immediately if required controls are bypassed or their effect is uncertain. Include the operation, fallback mode, affected interval, rate, client versions and dependency error. Preserve the incident's evidence limitations even after service recovers.

Next steps​