Skip to main content

Documentation for Tutela v3.9.0.

Delayed activity processing

Use this runbook when Activity or usage records arrive late, or a consumer-lag alert fires. Lag measures how far a consumer's committed position is behind the broker; it is not directly a number of seconds or proof that records were lost.

Prerequisites​

  • A known request time and its eventual appearance time, if available.
  • Access to Service health and an operator with broker/consumer monitoring access.

Investigate​

  1. Confirm workspace and time filters before classifying a record as delayed. Compare one synthetic request's completion time with when its event becomes visible.

  2. Check the health of analytics and its storage dependencies. A healthy broker cannot make a consumer commit records when its destination is failing.

  3. In Grafana, identify the affected group and topic:

    tutela_kafka_broker_consumer_lag

    Shipped thresholds are above 1,000 for warning and 10,000 for critical, sustained for five minutes. Confirm the installed rule when interpreting an alert.

  4. Watch whether lag is growing, flat or falling while comparing incoming traffic and consumer progress. Missing lag measurements do not establish an empty queue.

  5. For the packaged AWS runtime with a tutela-redpanda container, an authorized host operator can inspect the analytics group:

    docker exec tutela-redpanda rpk group describe analytics-consumer

    Compare committed positions and lag by partition. For Helm or externally managed brokers, use that broker's authorized consumer-group tooling instead of this container command.

Correct the cause​

  • Consumer stopped or restarting: inspect its startup error and repair the failed dependency before restarting through the deployment procedure.
  • Storage writes failing: restore the destination's health and capacity, then confirm consumer progress resumes.
  • Consumer healthy but incoming load exceeds processing: defer optional bulk work and have the operator assess supported capacity changes.
  • One partition remains stuck: preserve its group/topic/partition context and error for support; do not reset offsets or replay business actions as a diagnostic shortcut.

Record any temporary load limit and the previous configuration. Offset resets can skip or duplicate data and require a separate recovery decision.

Expected outcome and recovery​

Committed positions advance, backlog falls under normal traffic, and the known synthetic event becomes visible. Compare the original missing interval separately; a cleared lag alert does not prove that every historical record was recovered.

Support handoff​

Escalate growing critical lag or a consumer that cannot make progress. Include group, topic, partition where relevant, lag trend, first affected time, storage health and the synthetic request's appearance delay.

Next steps​