Documentation for Tutela v3.9.0.
High CPU utilization
Use this runbook when a service approaches its configured CPU limit or users see delays during high load. Tutela's container CPU alert is relative to that container's configured limit, not a single CPU core or the entire host.
Prerequisites
- The slow or failing workflow and its incident time.
- An operator with container resource metrics and access to the runtime host or Kubernetes namespace.
Investigate
-
Check whether requests are slow, failing, or processing late. Identify the affected service before changing capacity.
-
In Grafana, inspect:
tutela_container_cpu_limit_percentThe shipped warning is above 80% for five minutes. Compare request volume, latency and recent deployments. A missing or invalid limit can raise
ContainerResourceLimitUnavailable; do not substitute host CPU percentage as if it were the same measurement. -
For an AWS runtime host, use an authorized host shell:
docker stats --no-streamCompare the affected container with others. Docker's CPU display uses a different denominator from Tutela's limit percentage, so do not compare the two numbers directly.
-
For Helm, use the installed namespace and pod name:
TUTELA_NAMESPACE='your-namespace'kubectl -n "$TUTELA_NAMESPACE" describe pod YOUR_POD_NAMEInspect requests, limits, restart events and scheduling constraints. Use container monitoring for actual CPU consumption and throttling.
Correct the cause
- Avoidable burst or retry loop: pause the bulk task or reduce its concurrency, then check whether normal traffic recovers.
- Sustained legitimate load: assess both the service limit and available host/cluster capacity. Apply supported sizing changes through the deployment configuration.
- High CPU with little traffic: correlate the service's logs and recent changes; a hot loop or repeated dependency failure needs repair rather than repeated restarts.
- Missing resource limit: have the operator restore the release's supported resource configuration before trusting the percentage alert.
Record the previous limit and workload setting. If a capacity change causes scheduling or host contention, restore the previous setting through the same deployment mechanism. Do not raise every service's limit beyond available capacity.
Expected outcome and recovery
The affected service handles representative traffic within its configured limit, latency improves, and error rate or processing lag does not increase. Observe at least the alert's five-minute evaluation window before restoring deferred bulk work gradually.
Support handoff
Escalate sustained saturation that blocks work, unexplained CPU use at low traffic, or repeated throttling after sizing changes. Include service, CPU limit, utilization, request rate, latency and change time.