Documentation for Tutela v3.9.0.
Slow requests
Use this runbook when requests complete slowly or a latency alert fires. First separate provider response time from Tutela inspection and infrastructure delay so the corrective action targets the actual bottleneck.
Prerequisites
- One representative request ID, its approximate duration, client and time.
- Access to Service health and an operator with Grafana access.
Investigate
-
Compare one small synthetic request with the affected workflow. Note whether delay occurs before any response, during streaming, or during file processing. Avoid repeatedly resubmitting a slow business action.
-
Check Service health and the configured provider. Determine whether the problem affects one provider, client or all traffic.
-
In Grafana, compare inspection volume with P95 duration for the same interval:
sum(rate(tutela_api_request_duration_ms_count{job="inspector"}[5m]))histogram_quantile(0.95, sum(rate(tutela_api_request_duration_ms_bucket{job="inspector"}[5m])) by (le))The shipped inspector warning is above 200 ms for five minutes. P95 means 95% of recorded samples were at or below that duration. With very few requests it can fluctuate; an absent series is not zero latency.
-
Compare the AI gateway's phases to locate the delay:
histogram_quantile(0.95, sum(rate(tutela_ai_gateway_phase_duration_ms_bucket[5m])) by (le, phase)) -
Compare CPU, memory and cache measurements for the same service and time. Check whether the change began with a deployment, large file workload or provider incident.
If the alert names the classifier
Inspect overall classifier latency using the seconds-based histogram, converted to milliseconds:
histogram_quantile(0.95, sum(rate(tutela_classification_latency_seconds_bucket{job="classifier"}[5m])) by (le)) * 1000
The shipped critical alert fires above 1,000 ms for two minutes. A separate warning covers the fast classification stage (tier="T1") above 10 ms for five minutes. Confirm the installed rule, identify the stage in the alert, and compare its memory pressure and request load. Seconds and milliseconds are different units; do not compare the raw seconds histogram to a millisecond threshold.
Correct the cause
- Provider delay: confirm provider status and model selection. Apply the client's supported timeout/retry settings; do not make overlapping retries for an action that may already run.
- Resource saturation: defer optional bulk work and reduce avoidable concurrency. Have the operator size the affected component using its actual limit and host capacity.
- Cache delay: follow Cache dependency latency to separate a caller pool problem from server latency.
- Inspection failure or timeout: follow Inspection unavailable or blocked; permissive handling is not a repair.
Record any changed concurrency or capacity setting and restore temporary throttles only after measurements stabilize.
Expected outcome and recovery
The same small request completes within the deployment's agreed target, the affected latency measurement recovers at comparable load, and errors do not increase. Confirm policy and Activity evidence for requests affected by inspection delay.
Support handoff
Escalate sustained delay that interrupts work, or latency accompanied by errors. Include the request duration, selected window, sample rate, slow phase, resource observations and any recent change.