Documentation for Tutela v3.9.0.
Review service reliability
Use this runbook when the error-budget alert fires or repeated service failures require a reliability review. The shipped calculation measures HTTP non-5xx availability against a 99.9% objective over 30 days. It does not measure latency, policy correctness or completeness of Activity evidence.
Prerequisites
- An administrator who can describe customer impact and an operator with the service's metrics history.
- The service name, installed alert rule and recent incident/change history.
Investigate
-
Check whether a failure is still happening. For a current outage, follow Service unavailable or High error rate first.
-
In Grafana, inspect the 30-day error-budget consumption by service:
(1 - (sum(rate(tutela_api_request_total{status!~"5.."}[30d])) by (job)/ sum(rate(tutela_api_request_total[30d])) by (job))) / (1 - 0.999)A value of 0.8 represents 80% of the allowed server-error fraction. The shipped warning is above 0.80 and critical above 0.95 for five minutes. Confirm your installed rules before applying these numbers.
-
Check metric retention, scrape gaps and request volume. A recently deployed service may not have 30 days of evidence; absent or sparse data cannot establish that the reliability objective was met.
-
Compare periods of elevated 5xx with incident records and release/configuration changes. Identify whether a few outages or a sustained error rate consumed the budget.
Correct the cause
- Current failures: repair the failing service or dependency using its incident runbook.
- Repeated failures from one change: review the change and its supported recovery options with the release owner.
- Historical consumption after service recovery: prioritize reliability work and review discretionary changes with the service owner. A restart does not reset the 30-day history.
- Incomplete measurement: restore monitoring and document the gap. Do not fill missing intervals with assumed successful requests.
Record any temporary change restrictions and the condition for lifting them. Changing the objective or alert threshold is a service-owner decision with a recorded rationale, not a repair for poor availability.
Expected outcome and recovery
Current errors are controlled and the service's customer tasks succeed. The historical budget may remain above threshold until older failures leave the window; report that separately from current service health. Document the cause, impact and corrective work for each contributing incident.
Support handoff
Escalate critical budget consumption, recurrent outages or incomplete measurement that prevents a reliable assessment. Include the service, objective/window, measured error fraction, request volume, data gaps and linked incident timeline.