Skip to main content

Documentation for Tutela v3.9.0.

High memory utilization

Use this runbook when a container approaches its memory limit or repeatedly restarts after an out-of-memory event. A restart can temporarily lower memory while leaving the cause unresolved.

Prerequisites​

  • The affected service and incident interval.
  • An operator with resource monitoring and authorized runtime-host or Kubernetes access.

Investigate​

  1. Check Service health and whether customer requests fail when the service restarts.

  2. In Grafana, inspect:

    tutela_container_memory_limit_percent

    The shipped warning is above 80% for five minutes, relative to a finite container limit. ContainerResourceLimitUnavailable requires checking the configured limit; host memory is a different measurement.

  3. Compare memory growth with request rate, file sizes, imports and restart times. Distinguish stable high use from growth that continues under steady traffic.

Operator: AWS runtime host​

Run from an authorized host shell. Replace tutela-SERVICE with the actual affected container name from docker ps -a:

docker stats --no-stream
docker inspect tutela-SERVICE --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.RestartCount}}'

A true OOM flag supports a memory-limit diagnosis; exit code and restart count provide context. Inspect host capacity as well before increasing a container limit.

Operator: Helm deployment​

Use the installed namespace and pod name:

TUTELA_NAMESPACE='your-namespace'
kubectl -n "$TUTELA_NAMESPACE" describe pod YOUR_POD_NAME

Look for OOMKilled, the previous termination state, configured requests/limits and node pressure. A pod eviction and a container exceeding its own limit need different capacity corrections.

Correct the cause​

  • Oversized or highly concurrent work: reduce the relevant workload's concurrency or size and check whether memory stabilizes.
  • Insufficient configured capacity: assess available host/cluster memory, then change the affected component through its supported deployment configuration.
  • Continuing growth at steady load: preserve graphs and redacted error details for support. A restart may restore service temporarily but is not a demonstrated fix.
  • Node or host pressure: coordinate capacity across services; raising one limit may worsen other services' failures.

Record original settings. Reverse a problematic sizing change through the same configuration mechanism, and restore deferred work gradually after observing stable memory.

Expected outcome and recovery​

Memory stabilizes under comparable traffic, restart counts stop increasing, and the affected customer workflow succeeds with its expected Activity record. Check a full alert evaluation window and a representative workload cycle.

Support handoff​

Escalate repeated OOM kills, data-processing interruptions or unexplained steady memory growth. Include service, limit, peak use, restart history, workload type and the time of recent changes; exclude raw payloads or memory dumps containing customer data.

Next steps​