Documentation for Tutela v3.9.0.
Low disk space
Use this runbook when a filesystem is nearly full or a disk-space alert fires. Exhausted storage can stop writes, prevent containers from starting and interrupt event processing or backups.
Prerequisites
- Access to Service health and an operator authorized to inspect the customer-owned runtime host or Kubernetes storage.
- The affected filesystem, alert start time and recent backup or workload changes.
Investigate
-
Check Service health for services that cannot write or restart. Defer optional imports and backup jobs that would add data to the affected volume.
-
In Grafana, inspect available root filesystem space:
node_filesystem_avail_bytes{mountpoint="/"}/ node_filesystem_size_bytes{mountpoint="/"} * 100Shipped root-volume alerts warn below 20% and become critical below 10%, sustained for five minutes. Check other data volumes separately.
DiskMetricsUnavailableorMonitoringCollectorUnavailablemeans the measurement is missing, not that free space is adequate.
Operator: AWS runtime host
Run these read-only commands from an authorized host shell:
df -h /
df -h /var/lib/docker
docker system df -v
The first two commands show filesystem capacity, used space and available space. The last identifies Docker storage usage; it does not prove any listed object is safe to delete. If Docker uses a different data directory, inspect that directory's filesystem as well.
Operator: Helm deployment
Use the installed namespace and affected claim name:
TUTELA_NAMESPACE='your-namespace'
kubectl -n "$TUTELA_NAMESPACE" get pvc
kubectl -n "$TUTELA_NAMESPACE" describe pvc YOUR_CLAIM_NAME
These show claim capacity and provisioning events, not bytes remaining inside the filesystem. Use your storage monitoring for actual free space and involve the cluster operator for node-level pressure.
Correct the cause
- Unexpected log growth: identify the noisy service and repair the underlying repeated error. Apply the deployment's supported log retention settings.
- Backup accumulation: confirm retention and successful off-host storage before removing any expired backup through its supported management process.
- Legitimate data growth: have the storage operator expand the correct volume and filesystem using the deployment's storage procedure. Verify both layers grew.
- Unknown usage: preserve the measurements and escalate. Do not delete database files, broker segments, Docker volumes or live logs by guesswork.
Do not use a broad Docker prune command on the production runtime. Storage expansion is generally not reversed by shrinking a live volume; treat it as a capacity change with its own recovery plan. Restore temporary workload limits after capacity is stable.
Expected outcome and recovery
Available space remains above the configured threshold under normal writes, affected services become ready, and a new synthetic request produces Activity evidence. Confirm backups or imports deferred during the incident are rescheduled deliberately.
Support handoff
Escalate critical or rapidly falling free space, write failures, or suspected data loss. Include filesystem/claim, capacity and free space, growth rate, affected services and the largest identified storage category.