Everything local with Ollama

This page is the fully local path: orboto in Docker, Ollama on the same machine (or another one in your network) for the in-app AI, and an autonomous agent on a local model. Nothing leaves your network, no API key is involved anywhere.

It assumes the Docker install is up and you can sign in as an admin.

1. Ollama on the host

Install Ollama and pull one chat model and one embedding model. A coding-capable chat model in the 7B to 30B range and nomic-embed-text for embeddings are a sensible start:

ollama pull qwen2.5-coder:14b
ollama pull nomic-embed-text

Ollama listens on 127.0.0.1:11434 by default. The orboto containers reach it through host.docker.internal, which only works when Ollama also accepts connections on the Docker bridge. The simplest way is to bind it to all interfaces before starting it:

OLLAMA_HOST=0.0.0.0 ollama serve

Keep the port closed on your firewall for anything outside the box; the containers talk to it over the bridge, not the internet.

2. Reaching the host from the containers

Inside a container, localhost is the container itself, so the admin default of http://localhost:11434 cannot work. Use http://host.docker.internal:11434 instead.

The shipped docker-compose.yml maps host.docker.internal to the host on every Docker engine, Linux included, through this entry on the api and worker services:

extra_hosts:
  - "host.docker.internal:host-gateway"

If your compose file predates that line, add it to both services and recreate them (docker compose --env-file .env.prod up -d). Check it from inside:

docker compose exec api sh -c 'wget -qO- http://host.docker.internal:11434/api/tags | head -c 200'

That prints the model list Ollama serves. An empty answer or a timeout means the bind in step 1 is missing or a firewall on the host drops traffic from the Docker bridge.

Ollama on another machine in your network works the same way with that machine's address instead of host.docker.internal.

3. Wire the in-app AI

In Admin → AI:

1. Provider: pick Ollama, set the base URL to http://host.docker.internal:11434, click Switch to Ollama. No key.
2. Default model: Load models lists what Ollama serves; pick the chat model.
3. Embeddings: pick Bring your own provider → Ollama with the same base URL, load models, pick nomic-embed-text. Confirm the rebuild when orboto asks; on a fresh workspace it finishes in seconds.

The AI settings page explains every field, including per-feature model overrides and the rate limit. Chat, summaries, duplicate detection and search now run on the box.

Context window

A local model starts with a small context by default, and the in-app assistant sends ticket text, comments and retrieved docs with every question. If answers come back truncated or the model loses the thread, raise the window in a Modelfile and use that tag as the default model:

FROM qwen2.5-coder:14b
PARAMETER num_ctx 16384

ollama create qwen-coder-16k -f Modelfile

The same applies to any MCP client on a local model; the details and the minimal toolset for 8k windows are in Connect an AI client.

4. An agent on a local model

The first AI worker guide's setup wizard offers Ollama as the provider: pick it, point it at http://localhost:11434 (the runner runs on the host, not in a container), and choose a model from the list it fetches. The wizard installs the pi-lm-providers extension that gives the runner Ollama support. Everything else in that guide is unchanged: the bot account, the key, the project, the review duo.

For a fleet of lanes under systemd, the model and its fallback chain live in the lane assignment, see Run a fleet.

What to expect

Local models are slower and less reliable than the frontier ones. The review gate in orboto is there for exactly that: an agent hands a ticket to review, a human or a second agent approves it. Start with one worker on a 14B model and a small project, watch Admin → Multi-Agent for a day, and raise the ambition from there.