Documentation for Tutela v3.9.0.
High memory utilization
Use this runbook when a container approaches its memory limit or repeatedly restarts after an out-of-memory event. A restart can temporarily lower memory while leaving the cause unresolved.
Prerequisites
- The affected service and incident interval.
- An operator with resource monitoring and authorized runtime-host or Kubernetes access.
Investigate
-
Check Service health and whether customer requests fail when the service restarts.
-
In Grafana, inspect:
tutela_container_memory_limit_percentThe shipped warning is above 80% for five minutes, relative to a finite container limit.
ContainerResourceLimitUnavailablerequires checking the configured limit; host memory is a different measurement. -
Compare memory growth with request rate, file sizes, imports and restart times. Distinguish stable high use from growth that continues under steady traffic.
Operator: AWS runtime host
Run from an authorized host shell. Replace tutela-SERVICE with the actual affected container name from docker ps -a:
docker stats --no-stream
docker inspect tutela-SERVICE --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'
A true OOM flag supports a memory-limit diagnosis; exit code and restart count provide context. Inspect host capacity as well before increasing a container limit.
Operator: Helm deployment
Use the installed namespace and pod name:
TUTELA_NAMESPACE='your-namespace'
kubectl -n "$TUTELA_NAMESPACE" describe pod YOUR_POD_NAME
Look for OOMKilled, the previous termination state, configured requests/limits and node pressure. A pod eviction and a container exceeding its own limit need different capacity corrections.
Correct the cause
- Oversized or highly concurrent work: reduce the relevant workload's concurrency or size and check whether memory stabilizes.
- Insufficient configured capacity: assess available host/cluster memory, then change the affected component through its supported deployment configuration.
- Continuing growth at steady load: preserve graphs and redacted error details for support. A restart may restore service temporarily but is not a demonstrated fix.
- Node or host pressure: coordinate capacity across services; raising one limit may worsen other services' failures.
Record original settings. Reverse a problematic sizing change through the same configuration mechanism, and restore deferred work gradually after observing stable memory.
Expected outcome and recovery
Memory stabilizes under comparable traffic, restart counts stop increasing, and the affected customer workflow succeeds with its expected Activity record. Check a full alert evaluation window and a representative workload cycle.
Support handoff
Escalate repeated OOM kills, data-processing interruptions or unexplained steady memory growth. Include service, limit, peak use, restart history, workload type and the time of recent changes; exclude raw payloads or memory dumps containing customer data.