How to trace AI agents with OpenTelemetry (agent, LLM, and tool spans)

Sooner or later someone will ask “why did that agent run cost four times more than yesterday’s?” or “why did it take 40 seconds?” Logs can’t answer that well. A trace can: a tree of timed spans that shows each model call with its token counts and each tool call with its duration and errors, all nested under the run that caused them.

OpenTelemetry (OTel) is the vendor-neutral standard for traces, and its GenAI semantic conventions say what to call those spans and attributes. If you follow them, Jaeger, Grafana Tempo, Langfuse, Phoenix and most APM vendors can show your agent traces without custom parsing. In this guide we build the instrumentation by hand, so you can see exactly what auto-instrumentation libraries do for you.

Step 1: The span tree we’re building

Our agent follows the agent loop: call the model, run any tools it asks for, and repeat until it answers. The GenAI conventions map that loop onto three kinds of span:

%% caption: The span tree for one agent run (children in time order)
flowchart LR
  A["invoke_agent trip-helper<br/>INTERNAL · total tokens"]:::agent --> C1["① chat mock-model-1<br/>CLIENT · tokens · finish reason"]:::system
  A --> T1["② execute_tool get_weather<br/>INTERNAL"]:::workflow
  A --> T2["③ execute_tool search_hotels<br/>INTERNAL · error.type if it fails"]:::workflow
  A --> C2["④ chat mock-model-1<br/>CLIENT · tokens · finish reason"]:::system

On any span that fails, set error.type and an ERROR status.

Important caveat: the GenAI conventions now live in their own repository, open-telemetry/semantic-conventions-genai, and everything in it is still at “Development” status. As of October 2026 the repository has no tagged release. We pinned this tutorial to commit 06ec68e (2026-10-07), and Step 10 lists the changes already queued. Expect a rename or two before these become stable.

Step 2: Set up the project

mkdir otel-agent && cd otel-agent
python3 -m venv .venv && source .venv/bin/activate
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

You’ll create four files: tracing.py (SDK setup), model.py (the model), agent.py (the instrumented loop) and report.py (the cost and latency analysis).

Step 3: Configure the SDK (tracing.py)

"""Wire up the OpenTelemetry SDK: console by default, OTLP when an endpoint is set."""
import os

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor, ConsoleSpanExporter, SimpleSpanProcessor)


def setup_tracing() -> TracerProvider:
    provider = TracerProvider(resource=Resource.create({"service.name": "trip-helper"}))
    if os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"):
        from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
        provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    else:
        provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)
    return provider

Without configuration, spans print to stdout as JSON. Set the standard OTEL_EXPORTER_OTLP_ENDPOINT variable and they go to a collector instead. The OTLP exporter reads that variable itself and adds /v1/traces.

Step 4: A model you can run offline (model.py)

The mock returns scripted replies and estimated token counts (about four characters per token), so you can run the whole tutorial without an API key. The second class handles any server that speaks the Chat Completions API.

"""Two interchangeable models: an offline mock, and any OpenAI-compatible endpoint."""
import json
import os
import time


def rough_tokens(obj) -> int:
    return max(1, len(json.dumps(obj)) // 4)  # ~4 characters per token


class MockModel:
    """Scripted, deterministic stand-in so the tutorial runs offline."""
    name = "mock-model-1"
    provider = "mock"

    def chat(self, messages, tools):
        time.sleep(0.2)  # pretend network + generation time
        tool_results = [m for m in messages if m["role"] == "tool"]
        hotel_failed = any("error" in m["content"] for m in tool_results[-1:])
        if not tool_results:
            calls = [
                {"id": "call_1", "name": "get_weather", "arguments": {"city": "Lisbon"}},
                {"id": "call_2", "name": "search_hotels",
                 "arguments": {"city": "Lisbon", "max_price": 150}},
            ]
        elif hotel_failed and len(tool_results) < 4:
            calls = [{"id": f"call_{len(tool_results) + 1}", "name": "search_hotels",
                      "arguments": {"city": "Lisbon", "max_price": 150}}]
        else:
            calls = []
        if calls:
            content = None
        elif hotel_failed:
            content = "Lisbon looks sunny (24°C), but the hotel search kept timing out."
        else:
            content = "Lisbon looks sunny (24°C). Hotel Central is available at 130 EUR/night."
        message = {"role": "assistant", "content": content,
                   "tool_calls": [{"id": c["id"], "type": "function", "function": {
                       "name": c["name"], "arguments": json.dumps(c["arguments"])}}
                       for c in calls] or None}
        return {
            "model": self.name,
            "message": message,
            "tool_calls": calls,
            "finish_reason": "tool_calls" if calls else "stop",
            "input_tokens": rough_tokens(messages) + rough_tokens(tools),
            "output_tokens": rough_tokens(message),
        }


class OpenAICompatModel:
    """Any Chat Completions-compatible server: OpenAI, Ollama, llama.cpp, vLLM, ..."""

    def __init__(self):
        from openai import OpenAI  # pip install openai
        self.client = OpenAI()  # reads OPENAI_API_KEY and OPENAI_BASE_URL
        self.name = os.environ["MODEL"]
        self.provider = os.getenv("GEN_AI_PROVIDER", "openai")

    def chat(self, messages, tools):
        resp = self.client.chat.completions.create(
            model=self.name, messages=messages, tools=tools)
        choice = resp.choices[0]
        msg = choice.message
        calls = [{"id": tc.id, "name": tc.function.name,
                  "arguments": json.loads(tc.function.arguments or "{}")}
                 for tc in (msg.tool_calls or [])]
        return {
            "model": resp.model,
            "message": msg.model_dump(exclude_none=True),
            "tool_calls": calls,
            "finish_reason": choice.finish_reason,
            "input_tokens": resp.usage.prompt_tokens,
            "output_tokens": resp.usage.completion_tokens,
        }

Step 5: Instrument the agent loop (agent.py)

Here’s where the conventions come in. Read the three span helpers closely, because they’re the pattern you’ll reuse in your own agent:

"""A tiny tool-calling agent loop, traced with OpenTelemetry GenAI conventions."""
import json
import os
import sys
import time

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

from model import MockModel, OpenAICompatModel
from tracing import setup_tracing

AGENT_NAME = "trip-helper"

# --- Tools: plain Python functions ------------------------------------------

def get_weather(city: str) -> dict:
    time.sleep(0.05)
    return {"city": city, "forecast": "sunny", "high_c": 24}


HOTEL_FAILS = int(os.getenv("HOTEL_FAILS", "0"))  # simulate N timeouts


def search_hotels(city: str, max_price: int) -> dict:
    global HOTEL_FAILS
    time.sleep(0.6)  # a slow upstream API: watch it in the trace
    if HOTEL_FAILS > 0:
        HOTEL_FAILS -= 1
        raise TimeoutError("hotel API did not answer in time")
    return {"city": city, "hotels": [{"name": "Hotel Central", "price": max_price - 20}]}


TOOLS = {"get_weather": get_weather, "search_hotels": search_hotels}

TOOL_SCHEMAS = [
    {"type": "function", "function": {
        "name": "get_weather", "description": "Weather forecast for a city",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
                       "required": ["city"]}}},
    {"type": "function", "function": {
        "name": "search_hotels", "description": "Find hotels in a city under a nightly price",
        "parameters": {"type": "object", "properties": {
            "city": {"type": "string"}, "max_price": {"type": "integer"}},
            "required": ["city", "max_price"]}}},
]

# --- The traced agent loop ---------------------------------------------------

tracer = trace.get_tracer("trip-helper-agent", "0.1.0")


def call_model(model, messages):
    """One inference call = one `chat {model}` CLIENT span."""
    with tracer.start_as_current_span(
        f"chat {model.name}",
        kind=SpanKind.CLIENT,
        attributes={  # sampling-relevant attributes go in at creation time
            "gen_ai.operation.name": "chat",
            "gen_ai.provider.name": model.provider,
            "gen_ai.request.model": model.name,
        },
    ) as span:
        try:
            reply = model.chat(messages, TOOL_SCHEMAS)
        except Exception as exc:  # the SDK marks the span ERROR and records
            span.set_attribute("error.type", type(exc).__qualname__)  # the exception
            raise
        span.set_attribute("gen_ai.response.model", reply["model"])
        span.set_attribute("gen_ai.response.finish_reasons", [reply["finish_reason"]])
        span.set_attribute("gen_ai.usage.input_tokens", reply["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", reply["output_tokens"])
        return reply


def run_tool(call):
    """One tool execution = one `execute_tool {name}` INTERNAL span."""
    with tracer.start_as_current_span(
        f"execute_tool {call['name']}",
        kind=SpanKind.INTERNAL,
        attributes={
            "gen_ai.operation.name": "execute_tool",
            "gen_ai.tool.name": call["name"],
            "gen_ai.tool.call.id": call["id"],
            "gen_ai.tool.type": "function",
            "gen_ai.agent.name": AGENT_NAME,
        },
        record_exception=False,  # we record it ourselves below
        set_status_on_exception=False,
    ) as span:
        try:
            return json.dumps(TOOLS[call["name"]](**call["arguments"]))
        except Exception as exc:  # report the failure, let the model react
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            span.set_attribute("error.type", type(exc).__qualname__)
            return json.dumps({"error": str(exc)})


def agent_loop(model, task, max_turns):
    messages = [{"role": "user", "content": task}]
    usage = {"input": 0, "output": 0}
    for _ in range(max_turns):
        reply = call_model(model, messages)
        usage["input"] += reply["input_tokens"]
        usage["output"] += reply["output_tokens"]
        messages.append(reply["message"])
        if not reply["tool_calls"]:
            return reply["message"].get("content") or "", usage
        for call in reply["tool_calls"]:
            result = run_tool(call)
            messages.append({"role": "tool", "tool_call_id": call["id"], "content": result})
    raise RuntimeError(f"no final answer after {max_turns} turns")


def run_agent(model, task, max_turns=6):
    """One agent run = one `invoke_agent {name}` INTERNAL span around everything."""
    with tracer.start_as_current_span(
        f"invoke_agent {AGENT_NAME}",
        kind=SpanKind.INTERNAL,
        attributes={
            "gen_ai.operation.name": "invoke_agent",
            "gen_ai.agent.name": AGENT_NAME,
            "gen_ai.request.model": model.name,
        },
    ) as span:
        try:
            answer, usage = agent_loop(model, task, max_turns)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__qualname__)
            raise
        span.set_attribute("gen_ai.usage.input_tokens", usage["input"])
        span.set_attribute("gen_ai.usage.output_tokens", usage["output"])
        return answer


if __name__ == "__main__":
    provider = setup_tracing()
    model = OpenAICompatModel() if os.getenv("MODEL") else MockModel()  # MODEL set = real model
    task = sys.argv[1] if len(sys.argv) > 1 else "Plan a weekend in Lisbon: weather, and a hotel under 150 EUR."
    print("ANSWER:", run_agent(model, task), file=sys.stderr)
    provider.shutdown()  # flush spans before exit

Three design choices are worth a closer look:

• Nesting comes for free. start_as_current_span makes each span the parent of any span opened inside it. The chat and tool spans therefore become children of invoke_agent without any IDs being passed around.
• A tool failure isn’t an agent failure. run_tool marks its own span as ERROR with error.type, records the exception as a span event, and then returns the error to the model as data, so the model can retry. The run still succeeds, and the trace shows the failure that was recovered from.
• Prompts and completions are not recorded. The conventions define gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions and gen_ai.tool.call.arguments/.result, but they’re opt-in because they often contain personal data. Add them deliberately, with a retention policy, never by default.

Step 6: Run it and read the console spans

python agent.py > spans.json

The answer goes to stderr and the spans to spans.json. Here’s one chat span exactly as the console exporter printed it (resource block trimmed):

{
    "name": "chat mock-model-1",
    "kind": "SpanKind.CLIENT",
    "parent_id": "0xd4f5af5c1be6edba",
    "start_time": "2026-10-09T00:15:25.687414Z",
    "end_time": "2026-10-09T00:15:25.887686Z",
    "status": { "status_code": "UNSET" },
    "attributes": {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "mock",
        "gen_ai.request.model": "mock-model-1",
        "gen_ai.response.model": "mock-model-1",
        "gen_ai.response.finish_reasons": ["tool_calls"],
        "gen_ai.usage.input_tokens": 141,
        "gen_ai.usage.output_tokens": 76
    }
}

The spans are printed in the order they end, so children come first and invoke_agent comes last. Now simulate one timeout from the hotel API:

HOTEL_FAILS=1 python agent.py > spans-timeout.json

The failed tool span carries everything an on-call engineer needs:

{
  "name": "execute_tool search_hotels",
  "status": {
    "status_code": "ERROR",
    "description": "hotel API did not answer in time"
  },
  "attributes": {
    "gen_ai.operation.name": "execute_tool",
    "gen_ai.tool.name": "search_hotels",
    "gen_ai.tool.call.id": "call_2",
    "gen_ai.tool.type": "function",
    "gen_ai.agent.name": "trip-helper",
    "error.type": "TimeoutError"
  },
  "events": [
    {
      "name": "exception",
      "attributes": {
        "exception.type": "TimeoutError",
        "exception.message": "hotel API did not answer in time"
      }
    }
  ]
}

Step 7: Answer “why did this run cost X and take Y?”

Raw spans hold the answer, but you have to add them up. report.py rebuilds the tree from the console output, prices each chat span, and splits the wall time between model and tools:

"""Turn console-exported spans into a cost-and-latency breakdown."""
import json
import sys
from datetime import datetime

# Example prices in USD per million tokens. Replace with your provider's real rates.
PRICE_PER_M = {"input": 3.00, "output": 15.00}


def load_spans(path):
    text, spans, pos = open(path).read(), [], 0
    decoder = json.JSONDecoder()
    while pos < len(text):
        while pos < len(text) and text[pos].isspace():
            pos += 1
        if pos < len(text):
            span, pos = decoder.raw_decode(text, pos)
            spans.append(span)
    return spans


def ms(span):
    t = lambda s: datetime.fromisoformat(s.replace("Z", "+00:00"))
    return (t(span["end_time"]) - t(span["start_time"])).total_seconds() * 1000


def main(path):
    spans = load_spans(path)
    by_parent = {}
    for s in spans:
        by_parent.setdefault(s["parent_id"], []).append(s)

    def show(span, depth=0):
        a = span["attributes"]
        tin, tout = a.get("gen_ai.usage.input_tokens", 0), a.get("gen_ai.usage.output_tokens", 0)
        cost = ""
        if a.get("gen_ai.operation.name") == "chat":
            usd = (tin * PRICE_PER_M["input"] + tout * PRICE_PER_M["output"]) / 1e6
            cost = f"  in={tin:>5} out={tout:>4}  ${usd:.5f}"
        err = "  ERROR " + a["error.type"] if "error.type" in a else ""
        print(f"{'  ' * depth}{span['name']:<34}{ms(span):>7.0f} ms{cost}{err}")
        for child in sorted(by_parent.get(span["context"]["span_id"], []),
                            key=lambda s: s["start_time"]):
            show(child, depth + 1)

    for root in by_parent.get(None, []):
        show(root)

    chats = [s for s in spans if s["attributes"].get("gen_ai.operation.name") == "chat"]
    tools = [s for s in spans if s["attributes"].get("gen_ai.operation.name") == "execute_tool"]
    root = by_parent[None][0]
    tin = sum(s["attributes"].get("gen_ai.usage.input_tokens", 0) for s in chats)
    tout = sum(s["attributes"].get("gen_ai.usage.output_tokens", 0) for s in chats)
    usd = (tin * PRICE_PER_M["input"] + tout * PRICE_PER_M["output"]) / 1e6
    print(f"\n{len(chats)} model calls, {len(tools)} tool calls, "
          f"{tin} input + {tout} output tokens = ${usd:.5f}")
    print(f"wall time {ms(root):.0f} ms: model {sum(map(ms, chats)):.0f} ms, "
          f"tools {sum(map(ms, tools)):.0f} ms")


if __name__ == "__main__":
    main(sys.argv[1])

Here are both runs side by side. The prices are examples, and the token counts come from the mock’s estimate:

$ python report.py spans.json
invoke_agent trip-helper             1054 ms
  chat mock-model-1                     201 ms  in=  141 out=  76  $0.00156
  execute_tool get_weather               50 ms
  execute_tool search_hotels            600 ms
  chat mock-model-1                     200 ms  in=  284 out=  33  $0.00135

2 model calls, 2 tool calls, 425 input + 109 output tokens = $0.00291
wall time 1054 ms: model 401 ms, tools 651 ms

$ python report.py spans-timeout.json
invoke_agent trip-helper             1857 ms
  chat mock-model-1                     200 ms  in=  141 out=  76  $0.00156
  execute_tool get_weather               51 ms
  execute_tool search_hotels            601 ms  ERROR TimeoutError
  chat mock-model-1                     200 ms  in=  275 out=  47  $0.00153
  execute_tool search_hotels            600 ms
  chat mock-model-1                     200 ms  in=  359 out=  33  $0.00157

3 model calls, 3 tool calls, 775 input + 156 output tokens = $0.00466
wall time 1857 ms: model 601 ms, tools 1252 ms

The trace answers both questions:

• Why did it cost about 60% more? Not because one call got pricier. One tool timeout added a whole extra model turn. On top of that, input tokens grow every turn (141, then 275, then 359) because the agent resends the whole conversation each time. Agent cost depends mostly on the number of turns times the context size, so watch both.
• Why did it take 1.9 s instead of 1.1 s? Tools took 1.25 s of it, and search_hotels ran twice at 600 ms each. Model time grew only by the one extra call. The fix belongs in the hotel client (a timeout budget or a cache), not in the prompt.

You’ll do the same analysis in Jaeger or your APM tool. The difference is that it then works across thousands of runs, grouped by gen_ai.agent.name or gen_ai.request.model.

Step 8: Send traces to Jaeger (Docker)

Jaeger v2 accepts OTLP directly and has a trace UI. The official image is multi-arch, and we ran it on ARM64:

docker run -d --rm --name jaeger \
  -p 16686:16686 -p 4317:4317 -p 4318:4318 \
  jaegertracing/jaeger:2.21.0

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 HOTEL_FAILS=1 python agent.py

Open http://localhost:16686, choose the service trip-helper, and open the newest trace. You’ll see the same tree as report.py printed: the invoke_agent span at 1.85 s, three chat children, and the failed execute_tool search_hotels marked as an error with its TimeoutError event. Click a chat span to see its gen_ai.usage.* attributes. When you’re done, run docker stop jaeger.

Port 4318 is OTLP over HTTP, which is what this exporter uses. Port 4317 is OTLP over gRPC, for the opentelemetry-exporter-otlp-proto-grpc package. The Troubleshooting section shows what happens if you mix them up.

Without Docker, stay with the console exporter and report.py. The spans are identical, and only the destination changes.

Step 9: Optional: trace a real model

Set MODEL and the agent switches to OpenAICompatModel. The OpenAI SDK reads OPENAI_BASE_URL and OPENAI_API_KEY, so the same code works with a hosted API or a local server:

pip install openai

# Local, no key needed. Ollama shown here; llama.cpp and vLLM also serve /v1.
OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_API_KEY=ollama \
MODEL=qwen3.6:35b GEN_AI_PROVIDER=ollama \
python agent.py > real.json && python report.py real.json

We tested this against a local Ollama server (on a non-default port, with OLLAMA_HOST) with the tools working end to end. We did not test a hosted provider for this guide. For one, set your API key and MODEL, and leave OPENAI_BASE_URL unset or point it at the provider’s OpenAI-compatible endpoint. Set GEN_AI_PROVIDER to the convention’s well-known value (openai, anthropic, gcp.gemini, mistral_ai and so on). Local servers such as Ollama aren’t on that list, and a custom value is allowed.

Our first run against the local model showed something new:

invoke_agent trip-helper            78071 ms
  chat qwen3.6:35b                    69422 ms  in=  360 out= 454  $0.00789
  execute_tool get_weather               50 ms
  execute_tool search_hotels            600 ms
  chat qwen3.6:35b                     7997 ms  in=  885 out= 171  $0.00522

The first chat span took 69 s. On the next run it took 7 to 16 s. The trace makes the cause obvious: it’s a cold start (the model loading into memory), not slow generation. (A local model has no per-token bill, so here the dollar column only shows what the same token counts would cost at the example prices.) The run agents locally tutorial covers keeping models warm.

Step 10: What’s already changing in the conventions

The conventions are still in Development, so pin a version and expect churn. These changes are already merged on main of semantic-conventions-genai (in changelog.d/, not yet released as of 2026-10-08):

• Token metrics are being split. The gen_ai.client.token.usage histogram and its gen_ai.token.type attribute are replaced by per-direction instruments: gen_ai.client.inference.usage.input_tokens, .output_tokens, .cache_read.input_tokens, .cache_write.input_tokens and .reasoning.output_tokens. Dashboards built on the old histogram will need updating.
• Cache attribute rename. gen_ai.usage.cache_creation.input_tokens becomes gen_ai.usage.cache_write.input_tokens. Cache-token attributes are also removed from the invoke_agent span, because adding them up across models is misleading. Read them from the individual inference spans instead.
• Inference metrics are consolidated. A new gen_ai.client.inference.duration metric is added, and the time-to-first-chunk metrics move to gen_ai.client.inference.*.
• invoke_agent (internal) is trimmed. gen_ai.provider.name and gen_ai.agent.version are no longer on it, and gen_ai.agent.id was removed from internal agent spans.
• Finish reasons. gen_ai.output.messages[].finish_reason is deprecated in favour of gen_ai.response.finish_reasons, which this tutorial already uses.

The same repository also defines spans for MCP calls (mcp.method.name) and for Agent Skills loading (gen_ai.skill.name), which are worth reading once your agent uses those. In practice: keep attribute names in one module (as agent.py does), so a rename means changing one file and not twenty.

Where to go next

Traces tell you what happened. Evals tell you whether it was right: pair this with evaluating your agent and use trace data as the input to your golden tasks. For multi-agent systems, the same three span types nest one level deeper. See multi-agent orchestration.