Skip to main content

Documentation for Tutela v3.9.0.

High error rate

Use this runbook when requests repeatedly fail or an HTTP error-rate alert fires. The shipped alert measures the fraction of HTTP 5xx responses per service; a policy block or authentication failure needs a different diagnosis.

Prerequisites​

  • Access to Service health and the affected request's visible status or request ID.
  • A deployment operator with Grafana access and the incident time range.

Investigate​

  1. Record the response status, task, client version and time. Check whether failures affect one client, one provider or multiple workflows.

  2. Inspect Settings > Support & diagnostics > Service health. For an unhealthy service, start with Service unavailable.

  3. In Grafana, select the incident interval and compare the per-service server-error rate:

    sum(rate(tutela_api_request_total{status=~"5.."}[5m])) by (job)
    /
    sum(rate(tutela_api_request_total[5m])) by (job)

    The shipped warning is above 1% and critical above 5%, sustained for five minutes. Compare the numerator and request volume: a small denominator can produce a large ratio from a few errors. No traffic does not establish a healthy error ratio.

  4. Identify the affected path/status in the dashboard and compare onset with a configuration or release change. Inspect a representative request rather than retrying the whole workload.

Correct the cause​

FailureAction
401/403 authentication or permission errorCheck credential expiry, principal and scope with its owner. Do not grant broad administrative access to silence the error.
Policy or budget rejectionReview the specific decision and scope. A deliberate rejection is not a server outage.
429 or overload responseReduce concurrent requests and respect returned retry guidance; stop tight retry loops.
Provider-specific failureCheck the saved provider/model connection and its status.
Repeated 5xx from one serviceHave the operator inspect that service and its dependency failure, correlated by time and request ID.

Restore the previous configuration if an authorized change introduced the failure, using the supported recovery procedure. Record both the original and corrected values.

Expected outcome and recovery​

The failed synthetic request now produces its intended outcome, the service's error ratio falls below its alert threshold for the evaluation window, and the corresponding Activity evidence is present. Do not count a policy block as an unexpected server error.

Support handoff​

Escalate persistent 5xx or failures across multiple users. Include status, path, service, error fraction and request volume, first failure time, request ID and recent changes. Exclude authorization headers and raw payloads.

Next steps​