Structured outputs for agents: make models emit JSON you can actually trust

Every step where one component consumes another’s output is a place your agent can silently rot: the planner hands subtasks to the dispatcher, a tool result feeds a decision, a triage agent files a note. If those handoffs are prose, you’re parsing vibes. Structured outputs make each handoff a contract — and this tutorial builds the enforcement machinery, because the contract is worthless without it.

Step 1 — Know the three failure modes

Models break JSON in exactly three ways, and your pipeline needs an answer for each:

Wrapping — valid JSON buried in prose or markdown fences (“Sure! Here’s your JSON: …”). Answer: extract before parsing.
Malformed syntax — single quotes, trailing commas, truncation. Answer: parse, and feed the error back.
Schema violations — valid JSON, wrong shape: missing keys, invented enum values, a string where a number belongs. Answer: validate, and feed the violation back.

The pattern for all three is the same: detect precisely, tell the model exactly what was wrong, retry with a bounded budget.

Step 2 — Build the pipeline

Create structured.py. The mock model fails like real ones do — first a syntax error, then a schema violation, then success — so you can watch every recovery path fire:

import json

REQUIRED = {"category": str, "summary": str}
ALLOWED_CATEGORIES = {"task", "reference", "archive"}

def extract_json(text: str) -> str:
    """Failure mode 1: strip prose/fences around the first {...} block."""
    start, end = text.find("{"), text.rfind("}")
    return text[start:end + 1] if start != -1 and end > start else text

def validate(raw: str):
    """Failure modes 2 and 3: returns (data, None) or (None, precise_error)."""
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as e:
        return None, f"invalid JSON: {e}"
    for key, typ in REQUIRED.items():
        if key not in data:
            return None, f"missing required key: {key!r}"
        if not isinstance(data[key], typ):
            return None, f"{key!r} must be a {typ.__name__}"
    if data["category"] not in ALLOWED_CATEGORIES:
        return None, (f"category must be one of {sorted(ALLOWED_CATEGORIES)}, "
                      f"got {data['category']!r}")
    return data, None

class FlakyMockModel:
    """Fails the way real models fail, deterministically, for testing."""
    def __init__(self):
        self.calls = 0
    def complete(self, prompt: str) -> str:
        self.calls += 1
        if self.calls == 1:   # single quotes -> JSONDecodeError
            return "{'category': 'task', 'summary': 'Renew passport'}"
        if self.calls == 2:   # invented enum value -> schema violation
            return '{"category": "todo", "summary": "Renew passport"}'
        return '{"category": "task", "summary": "Renew passport"}'

def structured_call(model, prompt: str, max_attempts: int = 3) -> dict:
    error = None
    for attempt in range(1, max_attempts + 1):
        ask = prompt if error is None else (
            f"{prompt}\n\nYour previous reply failed validation: {error}\n"
            f"Reply with corrected JSON only — no prose, no fences.")
        data, error = validate(extract_json(model.complete(ask)))
        if data is not None:
            print(f"valid on attempt {attempt}")
            return data
        print(f"attempt {attempt} rejected: {error}")
    raise ValueError(f"no valid output after {max_attempts} attempts: {error}")

if __name__ == "__main__":
    result = structured_call(
        FlakyMockModel(),
        'Classify this note as JSON {"category": ..., "summary": ...}: '
        "'Renew passport by Friday'")
    print("result:", result)

Run it:

python structured.py

You’ll watch attempt 1 die on syntax, attempt 2 die on the invented "todo" category, and attempt 3 succeed — because each retry told the model precisely what to fix. Vague retries (“that was wrong, try again”) barely help; precise errors fix >95% of failures within two attempts with real models.

Step 3 — Use native structured outputs where you can

Modern APIs enforce schemas server-side — JSON mode and schema-constrained generation in Claude, GPT, and Gemini class APIs, and grammar-based sampling (GBNF) in llama.cpp locally. Use them: constrained decoding can’t produce invalid syntax at all. Keep the validation loop anyway. Native modes guarantee shape, not sense — a schema-valid refund of €0.00 for order “0000” still needs your semantic checks, and every system has handoffs (sub-agent to sub-agent, cached outputs, older providers) where native enforcement isn’t available.

Step 4 — Design contracts that models can hit

Reliability is half enforcement, half contract design:

Flat beats nested. Every nesting level multiplies failure modes.
Enums beat free text — "category": "task" is checkable; "category": "a to-do item" is a judgment call.
Forbid extras (additionalProperties: false in JSON Schema) so hallucinated fields fail loudly instead of flowing downstream.
One schema, one place — share the contract between your prompt, your validator, and your golden tasks, or they will drift apart. The tool-schema template is the reusable starting point.

Troubleshooting

The model wraps JSON in markdown fences no matter what I say

Stop fighting it in the prompt — handle it in code. extract_json exists precisely because “JSON only, no fences” is a suggestion the model will eventually ignore. Extraction + validation is cheaper than prompt whack-a-mole.

Validation passes but downstream code still breaks

Your schema is looser than your consumer. Every assumption the consuming code makes (non-empty strings? positive numbers? known keys only?) must exist in the validator — additionalProperties: false and enums close most of the gap.

Retries succeed but latency tripled

Count retries per call in your metrics. A rising retry rate usually means the schema grew too complex for the model tier — flatten the schema or upgrade the model for that step, don’t just raise max_attempts.

Next in this learning path How to evaluate an AI agent: build a golden-task eval harness from scratch