FinOps for AI agents: metering token spend before it meters you

Agent costs don’t grow like software costs. They grow like usage × verbosity × retries × context length — four multipliers, each of which an engineer can change with one innocent commit. If you’re serious about agents in production, cost telemetry is part of the architecture, not a finance afterthought. Here is the reference shape.

The three-layer architecture

%% caption: Cost governance architecture — enforce at the gateway, explain from the traces
flowchart TB
    A["Agents and apps"]:::agent --> G["LLM gateway<br/>routing · budgets · keys"]:::workflow
    G --> P["Model providers<br/>hosted APIs · local serving"]:::system
    A -. "OTel GenAI spans" .-> O["Observability store<br/>traces · dashboards · alerts"]:::system
    G -. "usage and cost records" .-> O
    O --> F(["Finance and owner reports"]):::outcome

Layer 1 — Instrumentation (standards, not vendors). Emit every model call as a span following the OpenTelemetry GenAI semantic conventions — model name, token counts in/out, and the operation it served. Standardizing on OTel keeps you portable across observability vendors and lets one dashboard cover ten teams’ agents. The OpenTelemetry tracing tutorial wires this up end to end.

Know the state of the standard before you build dashboards on it (checked 2026-10-08):

• The GenAI conventions moved out of core OpenTelemetry semantic conventions into their own repository, open-telemetry/semantic-conventions-genai. Core semconv v1.42.0 (June 2026) deprecated the old gen_ai.* definitions there. The new repository has no tagged release yet, and every GenAI document is still at Development status, so names can change.
• Metric names are changing. The last released definition used one histogram, gen_ai.client.token.usage, split by a gen_ai.token.type attribute. Unreleased changes in the new repository replace it with per-direction counters such as gen_ai.client.inference.usage.input_tokens, gen_ai.client.inference.usage.output_tokens, gen_ai.client.inference.usage.cache_read.input_tokens and gen_ai.client.inference.usage.cache_write.input_tokens, plus per-operation histograms. The span attribute gen_ai.usage.cache_creation.input_tokens is being renamed gen_ai.usage.cache_write.input_tokens.
• What to build on: the span attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (plus gen_ai.request.model and gen_ai.provider.name) appear under the same names in both versions, so compute cost from those and treat metric names as a mapping you’ll update. Record which convention version each service emits, and treat moving to a new one as a change, not a background event.

Cost metrics only mean anything if the token counts are right, and cache tokens are where they usually go wrong. Cached reads and cache writes are priced differently from ordinary input on most providers, so attribute them separately or your cost-per-outcome will be off by a wide margin on prompt-cache-heavy agents.

Layer 2 — Gateway (the enforcement point). Route all model traffic through one gateway (LiteLLM is the common open-source choice; several commercial equivalents exist, and the gateway config template sketches tiers and budgets). The gateway is where cost controls live, because it’s the only place that sees every call: per-team API keys, budgets with hard caps, model allowlists, and automatic fallback routing. Instrumentation without an enforcement point is a dashboard of regrets.

A gateway is also the most valuable target in your stack, because it holds every provider key. In March 2026 two malicious LiteLLM releases (1.82.7 and 1.82.8) were published to PyPI and stole credentials from any environment that installed them. Run a pinned, hash-locked gateway version, ideally from an image you build and scan yourself; the supply-chain guide walks through the controls.

Layer 3 — Attribution (the FinOps part). Tag every call with the dimensions finance will ask for — team, application, environment, and task type. Cost-per-call is trivia; cost-per-outcome (per resolved ticket, per document processed, per PR merged) is the number that decides whether the agent survives budget season.

Where traces land matters too. If you use Langfuse, note that Langfuse v4 changed its data model so observations (LLM calls, tool runs, agent steps) are the primary unit, and Langfuse Cloud becomes v4-only on 16 November 2026. Older SDKs need upgrading before then (Python SDK 4.7.0+ or JS/TS SDK 5.4.0+ per the v4 changelog); self-hosted v3 gets security patches through January 2027.

The four agent-specific cost patterns

1. The context ratchet. Conversations and memories grow; input tokens dominate spend. Watch input:output ratio per task type — a rising ratio means context needs pruning, not a bigger budget.
2. The retry spiral. A failing tool or a strict validator can put an agent in a paid loop. Cap attempts per goal in the harness and alert on outlier trajectories (cost per trajectory > p99).
3. The model-mismatch tax. Frontier models doing classification is the agent era’s “dev database on production hardware.” Route by task tier at the gateway (choosing an LLM for agents covers the planner/worker/router split); re-evaluate quarterly as prices move.
4. The silent multiplier. Multi-agent systems re-send overlapping context on every hop. Meter per-trajectory, not per-call, or orchestration overhead hides inside averages.

Minimum viable setup, in order

1. Gateway in front of all model traffic (an afternoon), with per-team keys.
2. OTel GenAI spans from the gateway into the observability stack you already run.
3. One dashboard: spend by team × model × task type, plus trajectory-cost outliers.
4. Hard budget caps per environment — enforced at the gateway, alerting at 80%, blocking at 100% in non-production.
5. Cost-per-outcome metrics, defined with the business owner, reported monthly.

The cost data comes from the same trajectory log described in agent observability, so build one pipeline, not two. And if you need an external taxonomy for runaway spend, the OWASP LLM Top 10 2026 lists it as LLM06:2026 Unbounded Consumption.