The agentic AI tech stack, layer by layer: what you actually need to build agents

Ask five vendors “what do I need for agents?” and you’ll get five diagrams where their box is the biggest. Here’s the vendor-free version: seven layers, what each does, the main options, and — most usefully — how much each choice actually matters. Product names in each layer are current as of October 2026; the layers themselves have been stable for longer.

%% caption: The agentic stack, layer by layer — and the two cross-cutting concerns
flowchart TB
    U["Users and apps"]:::human --> H
    subgraph stack ["The agentic stack"]
        H["Harness / orchestration"]:::workflow --> P["Protocols: MCP · A2A · AG-UI"]:::system
        P --> T["Tools and knowledge<br/>your APIs, data, memory"]:::system
        H --> G["Gateway and serving"]:::workflow
        G --> M["Models<br/>hosted and local"]:::agent
    end
    H -. "traces" .-> O["Evals and observability"]:::workflow
    T -. "enforced by" .-> C["Security and governance controls"]:::human

Layer 1 — Models (the engine)

The reasoning engine: Claude, GPT, and Gemini at the frontier; open-weights families such as gpt-oss, Qwen, Gemma, GLM, Kimi, DeepSeek, Mistral, and Llama where you need local control or lower cost. The strategic point: agents are the layer where model choice is most swappable — a good harness talks to any of them, and most mature systems route different task tiers to different models. Full decision framework: choosing an LLM for agents.

Layer 2 — Serving & gateway (the fuel line)

Hosted APIs need a gateway (LiteLLM, OpenRouter, or a cloud provider’s equivalent) for routing, budgets, keys, and fallback. Local needs a serving engine: llama.cpp for single-box and edge, vLLM or SGLang for GPU throughput, Ollama for developer convenience. Nearly every serving option speaks the OpenAI-compatible protocol now (llama.cpp’s server also offers an Anthropic-compatible endpoint), which is what makes Layer 1 swappable. Gateways are also a supply-chain risk: pin versions and verify packages (agent supply-chain security).

Layer 3 — Harness / orchestration (the chassis)

The loop that turns a model into an agent. Options in ascending machinery: your own ~40-line loop, provider SDKs (Claude Agent SDK, OpenAI Agents SDK, Google ADK), and full frameworks (LangGraph, CrewAI, Pydantic AI, AWS Strands, and Microsoft Agent Framework, which reached 1.0 in April 2026 and replaces AutoGen and Semantic Kernel for new work) when you need durable state and graph orchestration. TypeScript shops have Mastra and the Vercel AI SDK. The build-vs-buy call gets its own verdict.

Layer 4 — Protocols (the connectors)

MCP to connect agents to tools, A2A for agent-to-agent delegation across ownership boundaries, AG-UI for streaming agent state into frontends. MCP and A2A are both now hosted by the Agentic AI Foundation under the Linux Foundation, and AG-UI reached 1.0 in September 2026. One rule: protocols are seams, so keep adapters thin and your system survives the ecosystem’s churn.

Layer 5 — Knowledge & memory (what it knows)

Retrieval over your corpus (pgvector if you run Postgres; Qdrant, Weaviate, Chroma otherwise), agent memory for what the system learns in use, and plain files for profile knowledge. The RAG vs. memory vs. fine-tuning diagnostic sorts out which knowledge problem you have.

Layer 6 — Evals & observability (whether it works)

Golden-task evals gating changes, OpenTelemetry traces using the GenAI semantic conventions (still marked “Development” status, so pin the version you use), and a trace store (Langfuse, Arize Phoenix, W&B Weave, or your existing observability stack). This layer is chronically under-invested and is where reliability actually comes from.

Layer 7 — Security & governance (whether it’s safe)

Tool-layer allowlists and validation, approval tiers, injection defenses, and cost controls. Enforced in Layers 2–4 — never in the prompt. In regulated industries this layer also carries the governance framework.

Where the choices actually matter

The two layers you can’t buy: your tools (they encode your business) and your evals (they encode your definition of “works”). Everything else is increasingly commodity — which is exactly where your innovation budget shouldn’t go.