basicagent.org

BasicAgent

AI Agent Monitoring: Latency, Retries, Eval Signals, Sandbox/Promote

Monitoring for agent pipelines—track latency, retries, eval pass rates, and sandbox/promote tags with evidence-linked alerts for fast incident response.

Monitoring is the live pulse of your agent fleet. Wire metrics to governance evidence so alerts point to spans, prompts, and validation outcomes.

What to track

  • Latency, timeouts, retries (per stage/agent).
  • Fallback rate and reason codes.
  • Token and cost drift per model.
  • Eval pass rate (Layered-CoT verdicts).
  • Sandbox vs promoted outputs (separate SLOs).

Code: emit metrics with sandbox/promote

def record(span, verdict):
    metrics.emit("latency_ms", span.latency_ms, tags={"stage": span.name})
    metrics.emit("retry_count", span.retries, tags={"stage": span.name})
    metrics.emit("sandbox", int(span.is_sandbox), tags={"stage": span.name})
    metrics.emit("eval_pass", int(verdict.ok), tags={"stage": span.name})

Alerts that matter

  • Timeout rate or latency p95 spikes.
  • Retry storms or fallback rate > threshold.
  • Eval pass rate drops below target.
  • Cost per successful completion climbs.
  • Sandbox outputs leaking into promoted path (should be zero).

Governance tie-in

  • Link alerts to audit spans (run_id, trace_id, prompt version).
  • Store decisions and verdicts in append-only logs (/llm-audit-trail-agent-pipelines/).
  • Track model/prompt/tool versions to explain regressions.

Create account

Create account