Agent observability: what to log, what to alert on, and what to keep

Traditional observability asks “is the service up and fast?” Agent observability asks a harder question: “did the system decide well?” — and answering it requires recording decisions, not just durations. This piece is the practical spec: what to log, what to watch, what to keep, and where the standards are.

The unit of observation is the trajectory

Logs-per-request thinking fails for agents because one user request fans into many model calls and tool invocations whose sequence is the story. The unit that matters is the trajectory: one goal, every decision and action taken toward it, and the outcome — captured as ordered events under one run ID. Our trajectory-log template is the concrete JSONL shape; the non-negotiable fields:

• Identity: run ID, agent name, model ID, prompt/config version — the field everyone forgets and every incident review needs first.
• Decisions: the plan, each tool call with arguments, each gate decision (approved/denied, by whom).
• Data hygiene flags: whether untrusted content entered the context — your injection forensics depend on this bit.
• Economics: tokens in/out and cost per call, rolled up per trajectory.
• Outcome: success/failure/abandoned, turns used, wall time.

Digest large tool results (hash + summary) rather than storing raw payloads — you keep replayability without building a shadow copy of your PII estate inside the logging system.

Standards: adopt the schema, pin the version

The OpenTelemetry GenAI semantic conventions are the vendor-neutral naming for model-call spans (tokens, model, operation), and they now cover agent and MCP spans too — emit them and any OTel-compatible backend can render your traces. The OpenTelemetry tracing tutorial builds this step by step. What you need to know about their state, as of 2026-10-08:

• They moved. The conventions now live in their own repository, open-telemetry/semantic-conventions-genai; core semconv v1.42.0 (June 2026) deprecated the copies it held. The new repository has no tagged release yet.
• Everything is still Development status. There is no stable GenAI or agent convention. The agent operations (invoke_agent, invoke_workflow, execute_tool) and the MCP client and server spans (named after mcp.method.name) are defined, but can still change.
• Renames are queued. The gen_ai.client.token.usage histogram and its gen_ai.token.type attribute are being replaced by per-direction counters such as gen_ai.client.inference.usage.input_tokens, and gen_ai.usage.cache_creation.input_tokens becomes gen_ai.usage.cache_write.input_tokens. If you build alerts on metric names, expect to update them.
• The MCP spec carries trace context. The 2026-07-28 MCP revision standardizes passing traceparent in a request’s _meta, so client and server spans can join one trace across the protocol boundary.

So record which convention version each service emits (commit or date of the conventions you followed, if there’s no release tag) and treat upgrades as a change, not a background event. Tracing products (Langfuse, Phoenix, Weave — see resources) speak OTel increasingly natively; the conventions are the portability layer that keeps you free to switch.

If you run Langfuse, plan for Langfuse v4: its data model now treats observations (LLM calls, tool runs, agent steps) as the primary queryable unit, and Langfuse Cloud becomes v4-only on 16 November 2026, so older SDKs must be upgraded before then (v4 changelog). Self-hosted v3 receives security patches through January 2027.

The four alerts that earn their pager

Dashboards inform; alerts interrupt. Only four things are worth the interruption:

1. Trajectory cost outliers — a run past p99 cost is a loop, a retry spiral, or an attack. Cap in the harness; alert at the cap.
2. Behavioral outliers — a tool called in a sequence never seen before, especially after untrusted content entered context. This is your agent-era intrusion signal.
3. Eval regression in production — sampled trajectories re-scored against your golden-task criteria; a falling pass rate is drift announcing itself.
4. Gate anomalies — approval overrides spiking (agent misbehaving) or flatlining at zero for months (reviewers rubber-stamping; both are signals).

Retention: three tiers, one answer to two masters

• Hot (≈30 days): full trajectories, queryable — debugging and incident response.
• Warm (≈1 year): digested trajectories (decisions, gates, outcomes, costs; payload digests only) — drift analysis, quarterly reviews, cost attribution.
• Archive (per obligation): for regulated processes, whatever your record-keeping rules already say for the equivalent human-performed process. The agent doesn’t change the retention rule; it inherits it.

Deletion is part of the spec: trajectories referencing personal data need the same erasure path as agent memory — digests make this tractable.

The two-hour starting point

Wrap your model client so every call appends one JSONL event with run ID, model, tokens, and latency; wrap your tool dispatcher to do the same with arguments and a result digest; write both to one file per trajectory. That’s it — you now out-observe most production agent deployments, and every later upgrade (OTel emission, a tracing backend, alerting) is a sink change, not a re-instrumentation.

One caution for the libraries you instrument with: observability and gateway packages sit in the process that holds your secrets, which makes them supply-chain targets. The March 2026 compromise of two LiteLLM releases is the reference case; pin and hash-lock them like anything else (see the supply-chain guide). Logs are also evidence: under the EU AI Act, high-risk systems need automatic record-keeping (Art. 12) from 2 December 2027 for Annex III use cases (timeline), and the governance framework makes the trajectory log its third control layer.