Open-source LLMs

Why run a local model

• Privacy. Soul, Skills, and tool I/O never leave your network. Critical when an employee touches customer data, a Postgres replica, or finance records.
• Cost. No per-token bill. Routines that run on a tight cron can become uneconomical on the big closed models; locally hosted Llama / Qwen / Mistral don't care how often you call them.
• Model choice. The open ecosystem ships specialized weights — coding-tuned, retrieval-tuned, JSON-tuned — that the big labs don't. You can also stay on a stable model forever; nobody deprecates a checkpoint you downloaded.

The shape of the integration

Genosyn's in-process agent loop talks to your model over an OpenAI-compatible HTTP API — no CLI to install, nothing to spawn. The runtime path is always:

Genosyn agent loop (in-process, runner + chat)
   └─ HTTP to an OpenAI-compatible /v1/chat/completions endpoint
        └─ your local server (Ollama / vLLM / llama.cpp / LM Studio)
             └─ the model weights on your GPU or Mac

So you need two pieces wired up:

1. A local server that exposes an OpenAI-compatible API. Most popular runtimes do this out of the box.
2. An AI Model on the employee, configured from Settings → AI Model with the provider kind set to Custom. Paste the base URL + model id (+ an optional key). Done.

Step 1 — Run a local server

Pick one of these. They all expose /v1/chat/completions and /v1/models so any OpenAI-compatible client just works.

Ollama (easiest)

Best for a Mac, a single GPU, or just kicking the tires. Ships a model registry and serves on http://localhost:11434.

# install
curl -fsSL https://ollama.com/install.sh | sh

# pull a tool-capable model
ollama pull qwen2.5-coder:32b

# serve (auto-starts on macOS; this is for Linux/manual)
ollama serve

Ollama exposes the OpenAI-compatible API at http://localhost:11434/v1.

vLLM (best throughput)

For a real GPU box. Highest tokens/sec, batches concurrent requests across multiple employees. Native OpenAI-compatible server.

Prefer to run it by hand instead? The bare server is one pip install:

pip install vllm

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Endpoint: http://<host>:8000/v1. The two tool-call flags are required for MCP tool use to work — without them, vLLM will return tool calls as raw text and the agent will treat them as a normal message.

llama.cpp (most portable)

Smallest dependency surface. CPU works; with a GPU it's fast too. Ships llama-server as its OpenAI-compatible endpoint.

# from a release binary, or build from source
llama-server \
  -m ./qwen2.5-coder-32b-instruct-q5_k_m.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --jinja \
  --chat-template-file qwen2.5-coder.jinja

Endpoint: http://<host>:8080/v1. --jinja enables the chat template required for tool calls.

LM Studio (GUI-friendly)

Native macOS / Windows / Linux app. Browse models in a UI, flip the local server on in one click. Endpoint: http://localhost:1234/v1. Good for Mac users who don't want to live in the terminal.

Step 2 — Wire it into Genosyn

From the app, either during the hire wizard (Step 2: Model) or afterwards at Settings → AI Model:

1. Pick the Custom provider kind — any OpenAI-compatible endpoint.
2. Base URL: http://host.docker.internal:11434/v1 for Ollama on the same Mac/Windows host, or the full LAN URL of your GPU box.
3. Model id: the raw model name your server exposes — e.g. qwen2.5-coder:32b.
4. API key: leave blank for Ollama / vLLM / llama.cpp. Most local servers ignore the key entirely.
5. Click Continue. That's it — the next chat or routine run hits your endpoint.
6. Check the context window on the model card. Genosyn asks your server for it on save; if it reads Unknown, set it by hand — see below.

Tell Genosyn your context window

A run budgets its history against the model's window, dropping the oldest tool results when the next prompt wouldn't fit. It can only do that if it knows the number, and self-hosted servers disagree about whether to publish one on /v1/models:

vLLM

Reports max_model_len — whatever you passed to --max-model-len. Detected automatically.

LM Studio

Reports max_context_length. Detected automatically.

llama.cpp

Reports n_ctx — what you passed to -c. Detected automatically.

Ollama

Reports nothing. Set it by hand: it defaults to a num_ctx of 4096 unless your Modelfile or OLLAMA_CONTEXT_LENGTH says otherwise — far smaller than the weights allow, and a common surprise.

Use Set manually on the model card for anything not detected. A number you type always wins over the probe, so it survives key rotations and re-saves; Clear hands the field back.

Docker networking

If you installed Genosyn through genosyn install, the app runs inside a Docker container. localhost inside the container is not the same as on your host — the LLM server on the host won't be reachable as http://localhost:11434 from the employee.

macOS / Win

Use http://host.docker.internal:11434. Docker Desktop wires this magic hostname to the host automatically.

Linux

Add --add-host=host.docker.internal:host-gateway to your docker run, or run the LLM server bound to 0.0.0.0 and use the host's LAN IP.

Same machine, second container

Easiest is a shared user-defined network (docker network create genosyn-net) and reference the LLM container by name.

Hardware sizing

Rough rules of thumb for picking weights against your GPU memory. Quantized GGUFs at q4_k_m or q5_k_m are the practical baseline — the quality loss vs the original is small and the memory savings are large.

8–12 GB VRAM

7B–8B models at q4. Good for chat. Light tool use; expect occasional plan errors on multi-step routines.

16–24 GB VRAM

13B–14B at q5, or 32B at q4. The sweet spot for a single solid employee.

32–48 GB VRAM

32B at q6/q8, or 70B at q4. Comparable to mid-tier closed models on most code/ops tasks.

80+ GB VRAM (or M-series Mac with 64+ GB unified)

70B at q5+ or DeepSeek-V3 with offloading. State of the art for open weights.

Troubleshooting

• Model replies but never calls a tool. Almost always the chat template / tool-call parser. Check that your server has tool-call support enabled (--jinja for llama.cpp, --enable-auto-tool-choice for vLLM, and a chat template that emits a function-call block).
• Runs hang on the first message. Network — the agent inside the container can't reach your host. See the Docker networking section above.
• Model loops or hallucinates tool names. Context window. Genosyn injects the Soul + every Skill + the MCP tool catalog at each turn; that's often 8k–16k tokens before the first user message. Run models with at least 32k context for serious work.
• "This model's maximum context length is N tokens." The prompt outgrew the window. Genosyn drops old tool results and retries once, so this shouldn't fail a run — but seeing [compact] with reason overflow in the log means it was caught late. Set the model's context window on its card and the next run budgets ahead instead of reacting.
• Employee forgets what a tool told it earlier. Look for [compact] in the run log: history was dropped to fit the window. Give the model a longer context, or trim the Skills and tools that ride along on every turn.
• Slow. Quantize down (q8 → q5), enable batching on vLLM, or pin the layers to GPU (--n-gpu-layers in llama.cpp). If your GPU is saturated, the answer is more hardware, not more tuning.