Documentation for Tutela v3.9.0.
High error rate
Use this runbook when requests repeatedly fail or an HTTP error-rate alert fires. The shipped alert measures the fraction of HTTP 5xx responses per service; a policy block or authentication failure needs a different diagnosis.
Prerequisites
- Access to Service health and the affected request's visible status or request ID.
- A deployment operator with Grafana access and the incident time range.
Investigate
-
Record the response status, task, client version and time. Check whether failures affect one client, one provider or multiple workflows.
-
Inspect Settings > Support & diagnostics > Service health. For an unhealthy service, start with Service unavailable.
-
In Grafana, select the incident interval and compare the per-service server-error rate:
sum(rate(tutela_api_request_total{status=~"5.."}[5m])) by (job)/sum(rate(tutela_api_request_total[5m])) by (job)The shipped warning is above 1% and critical above 5%, sustained for five minutes. Compare the numerator and request volume: a small denominator can produce a large ratio from a few errors. No traffic does not establish a healthy error ratio.
-
Identify the affected path/status in the dashboard and compare onset with a configuration or release change. Inspect a representative request rather than retrying the whole workload.
Correct the cause
| Failure | Action |
|---|---|
| 401/403 authentication or permission error | Check credential expiry, principal and scope with its owner. Do not grant broad administrative access to silence the error. |
| Policy or budget rejection | Review the specific decision and scope. A deliberate rejection is not a server outage. |
| 429 or overload response | Reduce concurrent requests and respect returned retry guidance; stop tight retry loops. |
| Provider-specific failure | Check the saved provider/model connection and its status. |
| Repeated 5xx from one service | Have the operator inspect that service and its dependency failure, correlated by time and request ID. |
Restore the previous configuration if an authorized change introduced the failure, using the supported recovery procedure. Record both the original and corrected values.
Expected outcome and recovery
The failed synthetic request now produces its intended outcome, the service's error ratio falls below its alert threshold for the evaluation window, and the corresponding Activity evidence is present. Do not count a policy block as an unexpected server error.
Support handoff
Escalate persistent 5xx or failures across multiple users. Include status, path, service, error fraction and request volume, first failure time, request ID and recent changes. Exclude authorization headers and raw payloads.