Skip to main content

Documentation for Tutela v3.9.0.

Missing activity events

Use this runbook when a completed request cannot be found in Activity or the InspectorEventDropped alert fires. Missing evidence does not prove that a policy did nothing or that no external action occurred.

Prerequisites​

  • Permission to view Activity in the affected workspace.
  • The application, request or session ID, and time/timezone.
  • An operator with event-processing and monitoring access.

Investigate​

  1. Open Activity, select the correct workspace and time range, and clear unrelated filters. Confirm that your role can view the relevant records.

  2. Check whether the client actually sent the request through Tutela rather than directly to the provider or tool server. A configured connection that was not used creates no evidence for that request.

  3. Send one harmless request through the intended connection if safe to do so. Note its time and ID; do not replay the missing business action just to produce evidence.

  4. Inspect Service health for inspection and analytics. For late rather than missing events, follow Delayed activity processing.

  5. In Grafana, inspect the dropped-event counter:

    increase(tutela_inspector_event_dropped_total[5m])

    Any positive value means events were discarded at the inspector's publish queue. This is different from a backlog that can still be processed. Inspect a longer incident window to understand the extent.

Correct the cause​

  • Filters or permissions: correct the view or involve an authorized reviewer, then find the original record.
  • Client bypassed Tutela: correct its gateway connection and test a new synthetic request. Do not claim the original request was inspected.
  • Processing backlog: recover the consumer or its dependency and watch the backlog drain.
  • Dropped events: have the operator correlate queue pressure and downstream availability, reduce avoidable load, and repair the failed delivery path. Restarting a service does not reconstruct a discarded event.

Preserve the incident window and missing IDs. Record unrecoverable evidence gaps and their effect on investigations; do not fabricate records to make counts match.

Expected outcome and recovery​

A new synthetic request appears with the expected application and decision, no new dropped events occur at representative load, and any backlog is draining. The original missing interval remains documented unless its records were actually recovered.

Support handoff​

Escalate any confirmed dropped-event alert or persistent missing evidence. Include the event time, affected client, request ID, counter increase, source health and whether the original action may already have executed.

Next steps​