How to choose an LLM for your agent: Claude, GPT, Gemini, or open-weights

The wrong question is “which model is best?” — the leaderboard answer changes quarterly and mostly measures things agents don’t do. The right question is “which model is best at each job inside my system?” That one has a stable method, even as the names shift.

Why general benchmarks mislead for agents

An agent asks a model to do three unusual things: emit precisely formatted tool calls over long trajectories, notice and recover when a tool fails, and know when to stop. General chat benchmarks measure none of these. Two models with near-identical benchmark scores can differ wildly in multi-step tool reliability — and that difference, compounded over a 15-step trajectory, is the whole ballgame: 98% per-step reliability is ~74% per-trajectory; 90% is ~21%.

The seven criteria that actually matter

Tool-calling reliability — valid calls, correct arguments, sustained over many turns. The dominant factor.
Error recovery — shown a failure, does it adapt or repeat itself?
Instruction persistence — does the system prompt still bind on turn 30, or has it drifted?
Long-context behavior — not the advertised window; the quality of attention across a full, tool-result-stuffed context.
Latency & cost shape — agents multiply per-call latency by every step; a slow frontier model can lose to a fast good-enough one.
Deployment constraints — data residency, privacy, offline needs. If data can’t leave, open-weights served locally is your bracket, and the comparison happens inside it.
Structured-output support — native JSON Schema enforcement removes a whole class of failures. All three hosted families and llama.cpp have it, with different parameter names and schema limits; see structured outputs for agents.

The strategy: tier, don’t marry

Mature agentic systems are polyglot, and yours should plan to be:

Tier	Job	What to use
Planner	Decompose goals, recover from surprises, synthesize	A frontier model (Claude, GPT, Gemini class)
Worker	Routine bounded steps: extract, classify, summarize, draft	A mid-tier or small hosted model, or a strong open-weights model
Router/guard	Pick a category, validate a format, gate an input	A small fast model — increasingly, a local one

Current examples per tier (as of October 2026)

Names change every few months; the tiers don’t. This table is a starting shortlist for your own evals, not a ranking. Model IDs are the ones each provider’s model list showed on 2026-10-08.

Family	Planner candidates	Worker candidates	Router/guard candidates
Claude (Anthropic)	`claude-opus-5-5`; `claude-fable-5-1` for the hardest long-horizon work	`claude-sonnet-5-5`	`claude-haiku-5-5`
GPT (OpenAI)	`gpt-6-astra`	`gpt-6.1-sol`	`gpt-6-luna`
Gemini (Google)	`gemini-3.1-pro-preview` (the Pro tier is preview-only)	`gemini-3.8-flash`	`gemini-3.5-flash-lite`
Open weights	gpt-oss-120b; large Qwen, GLM and Kimi models	gpt-oss-20b; Qwen 3.8, Gemma 4	the smallest model that passes your router evals

Two things changed how this tier choice works. First, most hosted frontier models now have an effort or reasoning-level setting, so one model can cover two tiers: high effort for planning, low for routine steps. Measure both settings, because the cost difference can be larger than between two model names. Second, open-weight models are now credible workers and routers for tool calling. Check a tool-use leaderboard such as the Berkeley Function-Calling Leaderboard (BFCL) for a shortlist, then run your own golden tasks, because leaderboard rank and your workload don’t always agree.

Two mechanics make this cheap: a gateway so routing is config rather than code, and per-tier golden tasks so promotion and demotion between tiers is a measurement, not a debate.

The eval-driven decision, in five steps

Write ~20 golden tasks from your workload — including tool-failure and should-not-act cases.
Run each candidate N times per task (nondeterminism is real); record pass rate, tokens, latency, and cost per completed trajectory — cost per token flatters verbose models.
Kill candidates that fail your constraint bracket (residency, latency ceiling) before admiring their scores.
Pick per tier, not overall. The best planner is rarely the best router.
Re-run quarterly and on every major model release. The method is stable; the winners aren’t — that’s the point of owning evals instead of opinions.

The traps

Benchmark chasing: switching planners for a two-point leaderboard delta while your tools lack validation is optimizing the wrong layer.
Cost-per-token myopia: a cheaper model that takes three extra recovery turns is a more expensive model.
Single-vendor identity: “we’re a <vendor> shop” is a procurement statement, not an architecture. Keep the harness provider-agnostic and the identity costs you nothing when it changes.

The tier routing itself lives in your gateway layer; see where that sits in the agentic AI tech stack.