Documentation for Tutela v3.9.0.
Service unavailable
Use this runbook when Tutela is unreachable, Service health reports an unhealthy dependency, or the ServiceDown alert fires. That alert means monitoring could not collect a service's metrics; it can indicate a monitoring-access failure as well as an actual outage.
Prerequisites
- An administrator who can inspect Settings > Support & diagnostics, when the application is reachable.
- A deployment operator with monitoring access and, for host checks, access to the customer-owned runtime host or Kubernetes namespace.
- The first failure time, affected application address and any request ID.
Investigate
-
Check whether one workflow or the entire application is unavailable. From a client network that normally reaches Tutela, compare the gateway health and readiness endpoints.
-
If the application opens, inspect Service health and identify the failed dependency. A healthy gateway can coexist with unavailable inspection.
-
Compare the incident time with authorized maintenance and CloudFormation stack events or a Helm rollout. Do not assume a prolonged failure is normal maintenance.
-
In Grafana, compare the affected target with other targets using:
up == 0If every target is missing, investigate monitoring and its connectivity before assuming every application service failed. A missing series is different from a recorded zero.
Operator: AWS runtime host
On an authorized shell on the runtime host, list the installed containers:
docker ps -a --filter name=tutela- --format 'table {{.Names}}\t{{.Status}}'
A running container with a failed metrics scrape needs its scrape error checked. An exited or restarting container needs its health and startup failure investigated. Use the actual container name below, copied from the list:
TUTELA_CONTAINER='tutela-gateway'
docker inspect --format '{{json .State.Health}}' "$TUTELA_CONTAINER"
Read the latest probe's exit code and output. Inspect the affected service's logs locally for the same time window; redact credentials and customer content before sharing excerpts.
Operator: Helm deployment
Use your installed release's namespace and pod name, not the AWS container names:
TUTELA_NAMESPACE='your-namespace'
kubectl -n "$TUTELA_NAMESPACE" get pods
kubectl -n "$TUTELA_NAMESPACE" describe pod YOUR_POD_NAME
Check readiness, restart events and the reported reason. An image-pull failure, failed readiness probe and exhausted resource limit require different corrections.
Correct the cause
| Finding | Action |
|---|---|
| Service works but monitoring gets an authentication error | Repair the monitoring credential/configuration through the deployment's supported configuration process. Restarting healthy services will not correct a wrong credential. |
| Container or pod repeatedly restarts | Capture the failing probe and startup error, then correct the identified configuration or dependency. |
| Runtime is ready but the application address fails | Inspect DNS, certificate and load balancer target health. |
| Readiness names a failed dependency | Repair that dependency, then repeat readiness and the affected workflow. |
Record previous settings before changing them. If a deployment change caused the outage, use the approved release recovery procedure and its compatibility checks rather than an arbitrary image rollback.
Expected outcome and recovery
The affected service reports ready, monitoring collects it again, and the original customer task succeeds. If inspection was affected, repeat a known synthetic request and check its policy result and Activity record. A cleared monitoring alert alone is insufficient.
Support handoff
Escalate immediately when users cannot access required workflows or inspection remains unavailable. Include the service, failure time, deployment/client versions, readiness result, container or pod state, recent change and redacted probe error.