# Best Local Models for Hermes Agent in 2026

> Hermes Agent rejects any model under 64,000 tokens of context. Which open-weight models clear that bar on 8GB, 16GB, 24GB and 32GB of VRAM.

Source: https://capitalandcompute.net/blog/best-local-models-for-hermes-agent/
Published: 2026-09-04
Updated: 2026-10-13

Most guides to running [Hermes Agent](/blog/hermes-agent-explained/) on your own hardware start with the model. That is the wrong end. Hermes publishes a hard requirement that disqualifies candidates before capability enters the argument, and it is the one thing no other guide to this question mentions.

Nous states it plainly: [Hermes Agent requires at least 64,000 tokens of context](https://nousresearch.github.io/hermes-agent/docs/integrations/providers) for agent use with tools, and smaller windows are rejected at startup. Not degraded. Rejected.

64,000

Minimum context tokens

Rejected at startup below this, whatever the model scores

4,096

What Ollama gives you by default

On any GPU under 24GB, which is sixteen times too small

16GB

Cache cost at 64K for a 27B model

At full precision, on top of the weights, from the published formula

Key takeaways

* Hermes rejects any model offering under 64,000 tokens of context at startup. Almost every local-model recommendation you will read ignores this, and it is the actual reason most local Hermes setups fail.
* Two official sources contradict each other. Ollama tells you to leave context blank and auto-detect; Ollama auto-detects 4,096 tokens on a card under 24GB; Hermes then refuses to start.
* The context window is not free memory. For a 27B-class model a 64,000-token cache costs about 16GB at full precision, which is what turns a model whose weights fit a 24GB card into one that will not run.
* Nous names gemma4:31b as its best local pick for full agentic work, but the surrounding table in that same guide is stale and still lists models that cannot call tools at all.
* A tool-capable model still makes zero tool calls if the server is started wrong. llama.cpp needs --jinja and vLLM needs --enable-auto-tool-choice plus a matching parser.

## The rule that disqualifies most local models

The wording in the Nous documentation is worth quoting, because the strictness is the point: “Hermes Agent requires at least 64,000 tokens of context for agent use with tools. Smaller windows are rejected at startup because the system prompt, tool schemas, and working conversation state need enough room for reliable multi-step workflows.”

An agent turn is not a chat turn. Before your question arrives, the window already holds a system prompt, the JSON schemas for forty-odd tools, the running conversation, and the output of whatever the last tool returned. A 8K window is consumed before the model does any work.

Three ways to set it, all documented:

| Setup             | Command                                     |
| ----------------- | ------------------------------------------- |
| Ollama, one-off   | OLLAMA\_CONTEXT\_LENGTH=64000 ollama serve  |
| Ollama, systemd   | Environment="OLLAMA\_CONTEXT\_LENGTH=64000" |
| Ollama, Modelfile | PARAMETER num\_ctx 64000                    |

And the trap underneath all three: you cannot set context length through the OpenAI-compatible `/v1/chat/completions` API. It has to be configured server-side, before the server starts. Nous calls this the single biggest source of confusion when connecting Ollama to tools like Hermes.

## Two official sources that contradict each other

This is the part worth knowing before you spend an evening debugging.

Ollama ships a first-party Hermes integration. Its [documentation page](https://docs.ollama.com/integrations/hermes) gives you `ollama launch hermes` as a one-command setup, and for manual setup it says to leave context length blank and let it auto-detect.

Ollama’s auto-detected default, per the Nous provider documentation, is 4,096 tokens on any GPU with less than 24GB of VRAM. It only reaches 32,768 in the 24GB to 48GB band, and only clears the Hermes floor at 48GB and above.

So on a 12GB or 16GB card, following Ollama’s own advice produces a 4,096-token window, sixteen times below what Hermes will accept, and Hermes refuses to start. Both pages are official. Both were checked on September 4, 2026\. Set the variable by hand and ignore the auto-detect advice unless your card has 48GB.

One more way this breaks quietly

Parallel slots divide the window rather than duplicating it. Running llama.cpp with `-c 64000 -np 4` gives each slot 16,000 tokens, below the per-session minimum, even though the total looks correct. The Nous documentation calls this out explicitly. If you serve Hermes alongside other clients, size `-c` as slots multiplied by 64,000.

## What actually survives: the VRAM funnel

Here is the filter applied to eleven current builds from the Ollama library, every one of which carries the tool-calling capability tag and a native context window at or above 64K.

This first pass counts weights only, against raw card capacity. That is the generous reading, and it is deliberate: the next section adds the context cache and the picture gets considerably worse.

__Which local models survive the Hermes requirements__
| Stage                                                   | Tasks remaining | Percent of candidate pool kept |
| ------------------------------------------------------- | --------------- | ------------------------------ |
| Tool-capable builds (Carry the Ollama tools tag)        | 11              | 100%                           |
| Clear 64K context (Native window at or above the floor) | 11              | 100%                           |
| Weights fit 32GB (RTX 5090 class)                       | 11              | 100%                           |
| Weights fit 24GB (RTX 3090 or 4090)                     | 10              | 91%                            |
| Weights fit 16GB (Mainstream laptop or GPU)             | 5               | 45%                            |
| Weights fit 8GB (Entry gaming GPU)                      | 4               | 36%                            |

Eleven tool-capable local builds filtered by weights alone against card capacity. Not one is removed by the 64,000-token context requirement itself. The card removes them, and this count is before any context cache is allocated.Source: [Build sizes and context windows from Ollama library tag pages, read September 4, 2026\. Weights-only capacity comparison; cache cost is added in the next section.](https://ollama.com/library)

The shape is the finding. The context requirement removes nothing from this set, because modern open-weight models ship large windows as standard. What removes them is having to hold that window in memory at the same time as the weights.

## The context window is not free, and this is where it falls apart

This is the step almost every local-model guide skips, and once you do the arithmetic most of the recommendations you will read stop working.

Every token in the conversation sits in the KV cache, and its size follows directly from the model architecture: `2 x layers x KV heads x head dimension x tokens x bytes per element`. Those three architecture numbers come out of the model config, so this is checkable rather than a rule of thumb. The [RAM sizing guide](/blog/how-much-ram-to-run-a-local-llm/) works the formula through in full.

Run it at the 64,000 tokens Hermes demands:

| Model         | Per token | Cache at 64K, FP16 | Cache at 64K, 8-bit | Q4 weights |
| ------------- | --------- | ------------------ | ------------------- | ---------- |
| Llama 3.1 8B  | 0.125 MB  | 8.0 GB             | 4.0 GB              | 5.0 GB     |
| Qwen3.8-27B   | 0.25 MB   | **16.0 GB**        | 8.0 GB              | 17.1 GB    |
| Llama 3.3 70B | 0.3125 MB | 20.0 GB            | 10.0 GB             | 42.0 GB    |

Read the Qwen3.8-27B row against a 24GB card. The weights are 17.1GB and fit comfortably. Add the cache Hermes requires and you need 33.1GB at full precision, or 25.1GB with an 8-bit cache. **A 27B model does not run Hermes on a 24GB card**, even though every guide that stops at the download size says it does.

The official recommendation runs into the official requirement

Nous names `gemma4:31b` as its best local pick and lists it at “24+ GB RAM”. That figure is a reasonable reading of a 20GB download. It is not a reading of a 20GB download plus a 64,000-token cache, which is what Hermes itself mandates. On the arithmetic above, a model in that class needs a 32GB card to hold both, or an aggressively quantised cache on 24GB. The recommendation and the requirement come from the same documentation set and do not quite meet.

Two levers if a model almost fits. Quantise the cache, with `--cache-type-k q4_0 --cache-type-v q4_0` on llama.cpp, which roughly quarters it for a quality drop most people do not notice at agent workloads. Or drop a size class, which is usually the better trade: a 12B model running at a full 64K beats a 27B model that swaps to system RAM every turn.

Note also that a bigger model does not automatically mean a bigger cache. Llama 3.3 70B is two and a half times Qwen3.8-27B on parameters but only 25 percent worse per token, because grouped-query attention uses four times the KV heads at half the head dimension. Check the architecture, not the parameter count.

## Picks by card

Sized for weights plus a 64,000-token cache, not weights alone. The cache column assumes 8-bit quantisation, which is the setting that makes most of these viable.

| Your hardware | Run this                       | Weights | Realistic at 64K                                              |
| ------------- | ------------------------------ | ------- | ------------------------------------------------------------- |
| 8GB card      | granite4.1:3b                  | 2.1GB   | Yes, but a 3B model is weak at multi-step tool sequences      |
| 12GB card     | granite4.1:8b                  | 5.3GB   | Yes, with room for the cache. Apache 2.0                      |
| 16GB card     | gemma4:12b-it-q4\_K\_M         | 7.6GB   | The value pick. 256K native window, vision and tools          |
| 24GB card     | gemma4:26b-a4b-it-qat          | 16GB    | Marginal. A 26B MoE with 4B active, needs a quantised cache   |
| 32GB card     | qwen3.6:27b-q4\_K\_M           | 17GB    | Comfortable. The build Ollama names for Hermes                |
| 32GB, best    | gemma4:31b-it-q4\_K\_M         | 20GB    | The pick Nous names, and the class that genuinely needs 32GB  |
| Always-on     | nemotron-3.5-lightning:30b-a3b | 25GB    | 3B active and a 1M native window. Built for agent duty cycles |

The honest headline for anyone shopping: **12GB runs a real Hermes agent, 16GB runs a good one, and 24GB is the point where a 27B model stops being the obvious answer rather than the point where it starts.**

## The official guide is out of date

Worth flagging, because it is the first table most people find. The Nous [local Ollama guide](https://nousresearch.github.io/hermes-agent/docs/guides/local-ollama-setup) names gemma4:31b as “currently the best local option with tool-call support”, which is sound. The comparison table below it is not: it still lists gemma2:27b, gemma2:9b and llama3.2:3b, and the guide correctly marks all three as having no tool calling at all.

Three of the four rows in the official table are models that cannot run Hermes. They are there as context for a hardware ladder, but read quickly it looks like a shortlist. The current Ollama library has tool-capable builds at every one of those sizes, which is what the table above uses.

## A tool tag is not tool calling

The last failure mode, and the most frustrating, because the model looks fine and simply never calls anything.

A model advertised as supporting tools still produces no tool calls if the server is started without the right flags. The Nous provider documentation is specific about each backend:

| Backend   | Required flags                                                            |
| --------- | ------------------------------------------------------------------------- |
| llama.cpp | \--jinja, without which llama-server ignores the tools parameter entirely |
| vLLM      | \--enable-auto-tool-choice **and** \--tool-call-parser <name>             |
| SGLang    | \--context-length 65536 --tool-call-parser qwen                           |
| LM Studio | lms load <model> --context-length 64000                                   |

vLLM ships parsers named `hermes`, `llama3_json`, `mistral`, `deepseek_v3`, `deepseek_v31`, `xlam` and `pythonic`. Pick the one matching your model family. Without both flags, the documentation notes, the model emits its tool calls as plain text and the agent does nothing with them. On llama.cpp you can confirm the server took the flag by checking `http://localhost:8080/props`.

## The honest ceiling

Local models are further behind on agentic work than on chat, and the gap is not mainly about raw intelligence. It is about holding a tool chain together over many steps without drifting, which is a [documented weak point for local models in agentic coding](/blog/why-local-llms-fail-agentic-coding/).

Set expectations accordingly. A 12B model on a 12GB card will handle summarising, tagging, file wrangling and routine edits inside Hermes. It will not hold a long refactor together the way a frontier API model does. If the privacy requirement is real, that is a fair trade. If it is a preference, the [hosted picks](/blog/best-models-for-hermes-agent/) start at roughly five dollars a month in tokens, which is less than the electricity for a card running all day.

For hardware sizing beyond Hermes, the [RAM sizing guide](/blog/how-much-ram-to-run-a-local-llm/) has the general arithmetic, and [Qwen3.8-27B](/blog/qwen-3-8-27b-local-economics/) covers what free open weights do to the economics.

## Frequently asked questions

## Frequently asked questions

What is the best local model for Hermes Agent?

On 16GB, gemma4:12b is the value pick. On 32GB, gemma4:31b, which Nous names as the best local option with tool-call support. The usual advice to run a 27B model on a 24GB card does not survive the arithmetic: a 27B model needs about 16GB for a 64,000-token cache at full precision on top of 17GB of weights, so 24GB is not enough at the context Hermes requires.

Why does Hermes Agent refuse to start with my local model?

Almost always the context window. Hermes rejects any model offering under 64,000 tokens at startup, and Ollama defaults to 4,096 tokens on a GPU with less than 24GB of VRAM. Set OLLAMA\_CONTEXT\_LENGTH=64000 before starting the server. It cannot be set through the OpenAI-compatible API.

Can I run Hermes Agent on 8GB of VRAM?

Barely. Four of eleven current tool-capable builds fit 8GB on weights alone, but once a 64,000-token cache is added only the 3B class really works, and a 3B model is weak at the multi-step tool sequences Hermes is built around. 12GB is a far more realistic floor and 16GB is comfortable.

Why does my local model never call any tools?

The server flags, not the model. llama.cpp ignores the tools parameter entirely without --jinja. vLLM needs --enable-auto-tool-choice together with a --tool-call-parser matching the model family. Without them the model writes its tool calls out as ordinary text and the agent never executes them.

How much VRAM does a 64,000-token context actually need?

It follows from the architecture: 2 x layers x KV heads x head dimension x tokens x bytes per element. At 64,000 tokens that is about 8GB for Llama 3.1 8B, 16GB for Qwen3.8-27B and 20GB for Llama 3.3 70B at full precision, halving with an 8-bit cache. Quantising with --cache-type-k q4\_0 and --cache-type-v q4\_0 is the usual fix when a model almost fits.

Should I use ollama launch hermes?

It is a genuine first-party shortcut and it works, but do not follow its advice to leave context length blank. Ollama auto-detects 4,096 tokens below 24GB of VRAM, which Hermes rejects. Set the context variable explicitly first, then launch.

## Sources

* Nous Research (2026). _Hermes Agent documentation: LLM and model providers_ (the 64,000-token minimum, Ollama context defaults, tool-calling flags per backend, parallel-slot division). Verified 2026-09-04\. <https://nousresearch.github.io/hermes-agent/docs/integrations/providers>
* Nous Research (2026). _Hermes Agent documentation: run Hermes locally with Ollama_ (the gemma4:31b recommendation and the hardware table). Verified 2026-09-04\. <https://nousresearch.github.io/hermes-agent/docs/guides/local-ollama-setup>
* Nous Research (2026). _Hermes Agent documentation: run local LLMs on Mac_ (KV cache sizes at f16, q8\_0 and q4\_0). Verified 2026-09-04\. <https://nousresearch.github.io/hermes-agent/docs/guides/local-llm-on-mac>
* Ollama (2026). _Hermes Agent integration_ (the `ollama launch hermes` command and the auto-detect guidance). Verified 2026-09-04\. <https://docs.ollama.com/integrations/hermes>
* Ollama (2026). _Model library_ (build sizes, quantisations, native context windows and tool capability tags for gemma4, qwen3.6, qwen3.8, nemotron-3.5-lightning and granite4.1). Read 2026-09-04\. <https://ollama.com/library>
* Capital & Compute. _Local LLM hardware dataset_ (card memory and the 1GB driver reserve used in the funnel arithmetic), asOf 2026-09-03\. <https://capitalandcompute.net/ai-models/>

_Build sizes, context windows and documentation quotes were verified against the sources above on September 4, 2026\. The funnel counts weights against card capacity only; cache cost is computed separately from the published KV formula and the per-model architecture figures it takes, and both bases are stated where they appear. See the [editorial standards](/editorial-standards/)._