Documentation for Tutela v3.9.0.
Delayed activity processing
Use this runbook when Activity or usage records arrive late, or a consumer-lag alert fires. Lag measures how far a consumer's committed position is behind the broker; it is not directly a number of seconds or proof that records were lost.
Prerequisites
- A known request time and its eventual appearance time, if available.
- Access to Service health and an operator with broker/consumer monitoring access.
Investigate
-
Confirm workspace and time filters before classifying a record as delayed. Compare one synthetic request's completion time with when its event becomes visible.
-
Check the health of analytics and its storage dependencies. A healthy broker cannot make a consumer commit records when its destination is failing.
-
In Grafana, identify the affected group and topic:
tutela_kafka_broker_consumer_lagShipped thresholds are above 1,000 for warning and 10,000 for critical, sustained for five minutes. Confirm the installed rule when interpreting an alert.
-
Watch whether lag is growing, flat or falling while comparing incoming traffic and consumer progress. Missing lag measurements do not establish an empty queue.
-
For the packaged AWS runtime with a
tutela-redpandacontainer, an authorized host operator can inspect the analytics group:docker exec tutela-redpanda rpk group describe analytics-consumerCompare committed positions and lag by partition. For Helm or externally managed brokers, use that broker's authorized consumer-group tooling instead of this container command.
Correct the cause
- Consumer stopped or restarting: inspect its startup error and repair the failed dependency before restarting through the deployment procedure.
- Storage writes failing: restore the destination's health and capacity, then confirm consumer progress resumes.
- Consumer healthy but incoming load exceeds processing: defer optional bulk work and have the operator assess supported capacity changes.
- One partition remains stuck: preserve its group/topic/partition context and error for support; do not reset offsets or replay business actions as a diagnostic shortcut.
Record any temporary load limit and the previous configuration. Offset resets can skip or duplicate data and require a separate recovery decision.
Expected outcome and recovery
Committed positions advance, backlog falls under normal traffic, and the known synthetic event becomes visible. Compare the original missing interval separately; a cleared lag alert does not prove that every historical record was recovered.
Support handoff
Escalate growing critical lag or a consumer that cannot make progress. Include group, topic, partition where relevant, lag trend, first affected time, storage health and the synthetic request's appearance delay.