Agent observability: what to log, what to alert on, and what to keep

Traditional observability asks “is the service up and fast?” Agent observability asks a harder question: “did the system decide well?” — and answering it requires recording decisions, not just durations. This piece is the practical spec: what to log, what to watch, what to keep, and where the standards are.

The unit of observation is the trajectory

Logs-per-request thinking fails for agents because one user request fans into many model calls and tool invocations whose sequence is the story. The unit that matters is the trajectory: one goal, every decision and action taken toward it, and the outcome — captured as ordered events under one run ID. Our trajectory-log template is the concrete JSONL shape; the non-negotiable fields:

• Identity: run ID, agent name, model ID, prompt/config version — the field everyone forgets and every incident review needs first.
• Decisions: the plan, each tool call with arguments, each gate decision (approved/denied, by whom).
• Data hygiene flags: whether untrusted content entered the context — your injection forensics depend on this bit.
• Economics: tokens in/out and cost per call, rolled up per trajectory.
• Outcome: success/failure/abandoned, turns used, wall time.

Digest large tool results (hash + summary) rather than storing raw payloads — you keep replayability without building a shadow copy of your PII estate inside the logging system.

Standards: adopt the schema, pin the version

The OpenTelemetry GenAI semantic conventions are the vendor-neutral naming for model-call spans (tokens, model, operation), now extending to agent and MCP tool spans — emit them and any OTel-compatible backend can render your traces. One honest caveat: the conventions are still maturing, so pin the schema version you adopt and treat upgrades as a change, not a background event. Tracing products (Langfuse, Phoenix, Weave — see resources) speak OTel increasingly natively; the conventions are the portability layer that keeps you free to switch.

The four alerts that earn their pager

Dashboards inform; alerts interrupt. Only four things are worth the interruption:

1. Trajectory cost outliers — a run past p99 cost is a loop, a retry spiral, or an attack. Cap in the harness; alert at the cap.
2. Behavioral outliers — a tool called in a sequence never seen before, especially after untrusted content entered context. This is your agent-era intrusion signal.
3. Eval regression in production — sampled trajectories re-scored against your golden-task criteria; a falling pass rate is drift announcing itself.
4. Gate anomalies — approval overrides spiking (agent misbehaving) or flatlining at zero for months (reviewers rubber-stamping; both are signals).

Retention: three tiers, one answer to two masters

• Hot (≈30 days): full trajectories, queryable — debugging and incident response.
• Warm (≈1 year): digested trajectories (decisions, gates, outcomes, costs; payload digests only) — drift analysis, quarterly reviews, cost attribution.
• Archive (per obligation): for regulated processes, whatever your record-keeping rules already say for the equivalent human-performed process. The agent doesn’t change the retention rule; it inherits it.

Deletion is part of the spec: trajectories referencing personal data need the same erasure path as agent memory — digests make this tractable.

The two-hour starting point

Wrap your model client so every call appends one JSONL event with run ID, model, tokens, and latency; wrap your tool dispatcher to do the same with arguments and a result digest; write both to one file per trajectory. That’s it — you now out-observe most production agent deployments, and every later upgrade (OTel emission, a tracing backend, alerting) is a sink change, not a re-instrumentation.