Run agents locally: LLM serving plus MCP on your own hardware

Local serving matters for three audiences: laptops on planes, workloads whose data can’t leave the building, and anyone who wants to understand the stack one layer deeper than an API key. This guide is for the second and third.

Every command below was run on an NVIDIA DGX Spark, an ARM64 (aarch64) Ubuntu box with a GB10 GPU. The throughput numbers in Step 6 come from that machine. Expect different numbers on yours, but the same steps.

Step 1 — Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j
./build/bin/llama-server --version

The last line should print a build number and commit:

version: 0.6.0-dev (build 11514, commit de7fa0a3c)
built with GNU 13.3.0 for Linux aarch64

Drop -DGGML_CUDA=ON if you don’t have an NVIDIA GPU and CUDA toolkit (nvcc --version tells you). On a Mac, Metal is enabled by default, so no flag is needed. llama.cpp is first-class on ARM64. It was born on Apple Silicon, and it picks up NEON/SVE automatically. On the Spark, CMake detected the GB10 itself (CMAKE_CUDA_ARCHITECTURES=121a-real), and the full CUDA build took 2 minutes 47 seconds on 20 cores.

Step 2 — Pick a model in GGUF format

You need an open-weights instruct model that was trained for tool calling, in GGUF format, at a quantization your memory allows. Q4_K_M is the pragmatic default: roughly a quarter the size of the full-precision weights, with little quality loss.

This guide uses Qwen3-8B at Q4_K_M (4.7 GB, Apache 2.0) from Qwen’s own GGUF repository. It’s small enough for a 16 GB laptop and reliable at tool calls. Gemma, Llama, Mistral, and gpt-oss models work the same way. Swap the repo name and keep everything else.

You don’t have to download anything by hand. llama-server -hf fetches the file from Hugging Face on first run and caches it (Step 3). For your records, here is the file’s checksum once it lands:

sha256sum ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B-GGUF/snapshots/*/Qwen3-8B-Q4_K_M.gguf

d98cdcbd03e17ce47681435b5150e34c1417f50b5c0019dd560e4882c5745785  Qwen3-8B-Q4_K_M.gguf

If you already have a .gguf on disk, use -m /path/to/model.gguf instead of -hf in the next step.

Step 3 — Serve an OpenAI-compatible endpoint

./build/bin/llama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \
  --port 8080 --ctx-size 16384 -ngl 99 --reasoning off

What each flag does:

• -hf Qwen/Qwen3-8B-GGUF:Q4_K_M downloads the model (first run only) and loads it.
• --port 8080 pins the port. The default is changing in a future release, so don’t rely on it.
• --ctx-size 16384 sets the context window. Tool schemas and tool results eat context fast, so give agents room.
• -ngl 99 puts every layer on the GPU.
• --reasoning off turns off Qwen3’s “thinking” mode. Leave it on and the model may write over a thousand hidden tokens before a one-line answer. See Troubleshooting.

You don’t need --jinja any more. Chat-template support, which tool calling depends on, is on by default in current builds. Wait for listening on http://127.0.0.1:8080, then try it from a second terminal:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Say hello in five words."}]}'

The reply comes back in the standard OpenAI shape, with something like "content": "Hello! How can I help?". It also has an extra timings block with real tokens per second, so you can measure speed without extra tooling. llama-server speaks the OpenAI chat-completions protocol, so every SDK and agent framework that accepts a base_url now works against your own hardware.

Step 4 — Point an agent loop at it, with MCP tools

Here is the core of this guide: a real agent loop with a local model, using tools from a real MCP server. Set up a project next to tutorial #1’s server:

mkdir local-agent && cd local-agent
python3 -m venv .venv && source .venv/bin/activate
pip install openai "mcp[cli]"

Copy server.py from tutorial #1 into this folder. It’s the order_status server built on the mcp 2.x MCPServer class. Then create agent.py:

import asyncio, json, sys
from openai import OpenAI
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

# Any OpenAI-compatible server works here: just change base_url.
llm = OpenAI(base_url="http://localhost:8080/v1", api_key="local")
SERVER = StdioServerParameters(command=sys.executable, args=["server.py"])


def to_openai_tool(t):
    """MCP tool schema -> OpenAI function-calling schema."""
    return {"type": "function", "function": {
        "name": t.name, "description": t.description or "",
        "parameters": t.input_schema}}


async def run(question: str, max_turns: int = 5):
    async with stdio_client(SERVER) as (read, write):
        async with ClientSession(read, write) as mcp:
            await mcp.initialize()
            tools = [to_openai_tool(t) for t in (await mcp.list_tools()).tools]
            print("tools:", [t["function"]["name"] for t in tools])

            messages = [{"role": "user", "content": question}]
            for _ in range(max_turns):
                resp = llm.chat.completions.create(
                    model="local", messages=messages, tools=tools)
                msg = resp.choices[0].message
                if not msg.tool_calls:  # model answered: we're done
                    return msg.content
                messages.append(msg.model_dump(exclude_none=True))
                for call in msg.tool_calls:
                    args = json.loads(call.function.arguments)
                    print(f"-> {call.function.name}({args})")
                    result = await mcp.call_tool(call.function.name, args)
                    text = "".join(c.text for c in result.content if c.type == "text")
                    print(f"<- {text}")
                    messages.append({"role": "tool", "tool_call_id": call.id,
                                     "content": text})
            return "Stopped: hit max_turns without a final answer."


if __name__ == "__main__":
    q = sys.argv[1] if len(sys.argv) > 1 else \
        "What's the status of orders 1001 and 1002?"
    print(asyncio.run(run(q)))

Run it, with llama-server from Step 3 still up:

python agent.py

tools: ['order_status']
-> order_status({'order_id': '1001'})
<- shipped
-> order_status({'order_id': '1002'})
<- processing
The status of order 1001 is **shipped**, and the status of order 1002 is **processing**.

Three things happened there, and none of them needed the cloud:

1. The agent launched the MCP server as a subprocess and discovered its tools. The tool schema in agent.py is never written by hand.
2. The model put two tool calls in one turn, one per order. The loop ran both and sent the results back.
3. The model wrote the final answer from those results. The whole run took about 3 seconds.

Try python agent.py "Has order 9999 shipped yet?". The tool returns not found, and the model reports that back without inventing a status. The max_turns cap is the guardrail every agent loop needs. A small model that gets stuck calling the same tool will stop after five turns, not spin forever.

The only local-specific line in that file is base_url. Point it at a hosted OpenAI-compatible endpoint, or at Ollama or vLLM, and the loop and the MCP wiring stay exactly the same.

Step 5 — Optional: let llama-server host the MCP server itself

Current llama-server builds can also launch stdio MCP servers themselves (experimental). Declare them in a mcp.json file that uses the same format as Cursor. Use absolute paths, because the server spawns the process:

{
  "mcpServers": {
    "hello": {
      "command": "/home/you/local-agent/.venv/bin/python",
      "args": ["/home/you/local-agent/server.py"]
    }
  }
}

./build/bin/llama-server -hf Qwen/Qwen3-8B-GGUF:Q4_K_M \
  --port 8080 --ctx-size 16384 -ngl 99 --reasoning off \
  --mcp-servers-config /home/you/local-agent/mcp.json

The startup log confirms the server found your tool (MCP warmup: 'hello' discovered 1 tools). The tool is exposed as <server>_<tool>. You can list and call it over HTTP:

curl -s localhost:8080/tools
curl -s localhost:8080/tools -H "Content-Type: application/json" \
  -d '{"tool": "hello_order_status", "params": {"order_id": "1002"}}'

{"plain_text_response":"processing"}

This route is meant for llama-server’s built-in Web UI, which runs its own agent loop over these tools. It does not add the tools to plain /v1/chat/completions requests. Ask the API about order 1001 without passing tools and the model replies that it has no access to order details. For your own agents, use the Step 4 loop: it’s portable to any backend, and you control the loop and its limits. Also note that this flag switches CORS to localhost-only, and the server runs MCP commands with your user’s privileges. Only declare servers you trust.

Step 6 — Measure throughput on your hardware

Never trust someone else’s tokens-per-second figure, including this one. Stop the server first so it doesn’t compete for the GPU, then run the bundled benchmark:

./build/bin/llama-bench \
  -m ~/.cache/huggingface/hub/models--Qwen--Qwen3-8B-GGUF/snapshots/*/Qwen3-8B-Q4_K_M.gguf \
  -p 512,4096 -n 128

Measured on the DGX Spark (GB10, llama.cpp b11514, Qwen3-8B Q4_K_M, all layers on GPU). These are ranges across three benchmark runs:

Test	What it measures	Tokens/s
pp512	Reading a 512-token prompt	2,800–3,100
pp4096	Reading a 4,096-token prompt	2,775–3,010
tg128	Generating 128 tokens	36–43

The server’s own timings agreed. Generation ran at 34–38 tokens/s in the Step 3 requests. Run-to-run spread is normal, especially if anything else is using the GPU, so benchmark more than once.

How to read those numbers for agents: prompt processing is fast, so a 2,000-token tool schema and conversation history costs under a second per turn. Generation is the bottleneck. A 300-token answer takes about 8 seconds. That’s why short tool-call turns feel snappy and long essays don’t, and it’s why thinking mode (Step 3) hurts so much locally. The same benchmark with -ngl 0, which keeps the weights on the CPU, generated only 3.7 tokens/s. That’s about ten times slower, which is usable for a smoke test and not for an agent.

Step 7 — Know what you gave up

Local 7–14B models are noticeably weaker than frontier hosted models at multi-step planning and at recovering from tool errors. The honest pattern is to use local models for extraction, summarization, classification, and agents that make only a few tool calls, and hosted frontier models for long-horizon orchestration. Qwen3-8B handled the two-tool, one-turn task above cleanly. Measure on your tasks (evaluate your agent shows how) before committing either way.

Alternatives to the stack used here: Ollama wraps llama.cpp with model management and is the fastest way to start on a laptop. vLLM is the better fit for a GPU server handling many concurrent users. Both expose OpenAI-compatible endpoints, so the Step 4 agent runs against them unchanged. Only base_url changes.

Troubleshooting

A one-line answer takes 40 seconds (and reasoning_content is huge)

Qwen3 and other hybrid “thinking” models reason before answering by default. Our first “Say hello in five words” request produced 1,571 hidden reasoning tokens, about 41 seconds at 38 tokens/s, to output “Hi there! How are you?”. Start the server with --reasoning off for tool-calling agents. Turn it back on only for tasks that measurably benefit.

ModuleNotFoundError: No module named 'mcp.server.fastmcp'

You’re running older MCP server code on the mcp 2.x SDK. In 2.x, FastMCP was renamed: use from mcp.server import MCPServer. Tool schemas on the client side moved from t.inputSchema to t.input_schema. If you have to keep the old code for now, pin "mcp[cli]<2". See migrating to the 2026 spec.

The model says it “doesn’t have access” even though --mcp-servers-config loaded the tool

The tools that --mcp-servers-config loads are used by the built-in Web UI and the /tools endpoint. They are not added to your /v1/chat/completions requests. An API client has to send tools itself and run the calls, which is what the Step 4 loop does.

Build succeeds but generation is very slow

Check that the build actually used your accelerator. llama-bench prints a backend column, and nvidia-smi should list llama-server holding several GB (about 7 GB for this model at a 16k context). Make sure you passed -DGGML_CUDA=ON at configure time (delete build/ and re-run CMake if not) and -ngl 99 at serve time. On the Spark, generation with the weights on the CPU dropped from about 40 to 3.7 tokens/s.

Tool calls come back as prose instead of structured tool_calls

Make sure the model was actually trained for tool calling. Many otherwise-good models weren’t. Use the instruct variant, not a base model. Check that you haven’t disabled chat templates with --no-jinja. For a GGUF with a broken embedded template, you can pass a known-good one with --chat-template-file.

Where to go next

Local or hosted, an agent you haven’t measured is a guess. Evaluate your agent builds the test harness that tells you whether a local model is good enough for your task. Structured outputs for agents covers the JSON failures small models are most prone to.