Skip to main content

Documentation for Tutela v3.9.0.

Low disk space

Use this runbook when a filesystem is nearly full or a disk-space alert fires. Exhausted storage can stop writes, prevent containers from starting and interrupt event processing or backups.

Prerequisites​

  • Access to Service health and an operator authorized to inspect the customer-owned runtime host or Kubernetes storage.
  • The affected filesystem, alert start time and recent backup or workload changes.

Investigate​

  1. Check Service health for services that cannot write or restart. Defer optional imports and backup jobs that would add data to the affected volume.

  2. In Grafana, inspect available root filesystem space:

    node_filesystem_avail_bytes{mountpoint="/"}
    / node_filesystem_size_bytes{mountpoint="/"} * 100

    Shipped root-volume alerts warn below 20% and become critical below 10%, sustained for five minutes. Check other data volumes separately. DiskMetricsUnavailable or MonitoringCollectorUnavailable means the measurement is missing, not that free space is adequate.

Operator: AWS runtime host​

Run these read-only commands from an authorized host shell:

df -h /
df -h /var/lib/docker
docker system df -v

The first two commands show filesystem capacity, used space and available space. The last identifies Docker storage usage; it does not prove any listed object is safe to delete. If Docker uses a different data directory, inspect that directory's filesystem as well.

Operator: Helm deployment​

Use the installed namespace and affected claim name:

TUTELA_NAMESPACE='your-namespace'
kubectl -n "$TUTELA_NAMESPACE" get pvc
kubectl -n "$TUTELA_NAMESPACE" describe pvc YOUR_CLAIM_NAME

These show claim capacity and provisioning events, not bytes remaining inside the filesystem. Use your storage monitoring for actual free space and involve the cluster operator for node-level pressure.

Correct the cause​

  • Unexpected log growth: identify the noisy service and repair the underlying repeated error. Apply the deployment's supported log retention settings.
  • Backup accumulation: confirm retention and successful off-host storage before removing any expired backup through its supported management process.
  • Legitimate data growth: have the storage operator expand the correct volume and filesystem using the deployment's storage procedure. Verify both layers grew.
  • Unknown usage: preserve the measurements and escalate. Do not delete database files, broker segments, Docker volumes or live logs by guesswork.

Do not use a broad Docker prune command on the production runtime. Storage expansion is generally not reversed by shrinking a live volume; treat it as a capacity change with its own recovery plan. Restore temporary workload limits after capacity is stable.

Expected outcome and recovery​

Available space remains above the configured threshold under normal writes, affected services become ready, and a new synthetic request produces Activity evidence. Confirm backups or imports deferred during the incident are rescheduled deliberately.

Support handoff​

Escalate critical or rapidly falling free space, write failures, or suspected data loss. Include filesystem/claim, capacity and free space, growth rate, affected services and the largest identified storage category.

Next steps​