Run an agent fleet

New to agents? Start with Your first AI worker - a guided, one-command setup that gets a single agent (or an implement + review duo) running in ten minutes. This page is the scale-up path.

Work routing and fleets explains how orboto lets agents pull work safely - leases, routing labels, review lanes, pause switches. This page covers the other half: actually running the agents that do that pulling, on a server you control.

What pi-runner is

orboto pi-runner is a small supervisor process, built into the orboto CLI, that keeps one coding-agent session alive and feeds it work. Concretely, it:

• starts and supervises a pi coding-agent session (the integrated coding-agent runtime orboto's fleet tooling drives),
• delivers messages from that identity's orboto agent inbox to the session as it works (steering it if it's mid-turn, prompting it if it's idle),
• self-tasks the session: once it has sat idle, pi-runner injects a prompt that tells it to pull its next ticket with work-next - the same pull Work routing and fleets describes, just triggered by a timer instead of a human,
• automatically switches to a fallback model when the current one hits a provider rate limit or quota error, so a fleet running overnight doesn't just stall on a 429,
• clones and refreshes the working copies the agent needs, and resumes its session on restart instead of starting over.

One running pi-runner process is one lane: a single agent identity, working in a single role (implementer, reviewer, and so on). A fleet is simply several lanes running side by side.

The lane registry: configuration lives in orboto, not on the box

Added in v0.176.0 A lane no longer carries its own configuration. It starts with nothing but an identity - your orboto URL, its API key, its lane name - registers itself, and asks orboto what to do. Which projects it serves, in what order, in which role, with which model, how urgent a ticket it may take, how many tickets a day, which repositories it works in, what its idle prompt says, whether it is paused - all of that lives in the workspace, on the Admin → Agent Lanes page or behind orboto lanes assign.

That inversion is what makes a fleet manageable. Changing what a lane does is a two-click act in the UI that reaches the running process within one heartbeat (30 seconds at worst). Nobody logs into the server, nobody edits a file, nothing restarts:

• A lane that is idle switches immediately.
• A lane that is mid-ticket is told to finish the ticket it holds cleanly - post its summary, close its work session - and only then picks up the new assignment. Work in flight is never abandoned to a config change.

The older model, where every knob sat in the lane's environment file and every change meant an ssh session and a restart, is still fully supported as pinned mode - useful when a box must not depend on the registry at all. Everything else on this page assumes the default.

Prerequisites

• A Linux server you control - any VM, dedicated box, or container host. No specific provider is required; this page is written to work anywhere.
• The orboto CLI installed on that server.
• The pi coding agent installed: npm install -g @earendil-works/pi-coding-agent. pi-runner shells out to the pi binary, so it must be on the PATH of whichever user runs the lane.
• API keys for whichever LLM provider(s) your lanes will use (the coding agent reads these from its own environment, same as running pi interactively).
• An orboto account with the agent-lane admin permissions (admin:agent_lanes:read to see the fleet page, :write to assign, :delete to remove a lane). Super-admins have them already.

Step 1: create one bot identity per lane

Every lane authenticates to orboto as its own bot identity - a user account created for automation rather than a person. A lane will not register under a human account at all; the registration is refused. Give each lane its own identity rather than sharing one across lanes:

• Activity, comments, and commits stay attributable to the specific lane that did the work.
• You can pause or revoke one lane without touching any other.
• Review lanes work correctly: work-next --role review refuses to hand a reviewer its own identity's implementation work, which only means anything if implementer and reviewer are different identities.
• A lane name binds to the identity that first registered it, so one lane's key can never quietly take over another lane's slot.

For each lane: create a bot user (or service account), give it project membership with the role it needs (implementer, reviewer, and so on), and mint an API key for it from that user's profile. Keep the key somewhere you can paste it into the lane's environment file in Step 4.

Step 2: smoke-test one lane by hand

Before wiring up a systemd fleet, run one lane in a terminal to confirm the pieces fit together:

export ORBOTO_BASE_URL=https://your-orboto-host.example.com/api
export ORBOTO_TOKEN=orb_xxxxxxxxxxxxxxxxxxxxx   # the bot's API key

orboto pi-runner --remote --lane worker-1

You should see timestamped status lines on stderr: the runner registers the lane, reports the assignment version it applied, and - because a brand-new lane has no assignment yet - settles into

[pi-runner] lane worker-1 v0 idle (lane has no project assignment) - heartbeating, waiting for an assignment

That is the correct, healthy starting state, not an error. Leave it running and give it something to do in Step 3; the same process will pick it up.

Step 3: assign the lane

Open Admin → Agent Lanes. A lane that has registered itself is already listed; you can also create one in advance with New lane and start the runner afterwards. Press Reassign and fill in:

The same assignment from the command line:

orboto lanes assign worker-1 --role implementation \
  --model your-provider/your-model \
  --fallback-models your-provider/your-model,your-other-provider/their-model \
  --project ACME:high --repo https://git.example.com/acme/acme.git --ref main

--project is repeatable and replaces the lane's whole project list, in the order you pass it - a partial list unassigns the rest. Omit it entirely to leave the projects untouched (so orboto lanes assign worker-1 --model X changes only the model); pass --no-projects to clear them. --repo and --ref are positional companions: the n-th --repo belongs to the n-th --project. Add --paused or --resume to flip the lane switch.

orboto lanes list prints every lane with its state, current ticket, projects and applied assignment version; orboto lanes show worker-1 prints one lane's full assignment.

Back in the terminal from Step 2, the running process applies the new assignment on its own:

[pi-runner] assignment changed (v0 -> v1) - applying
[pi-runner] lane worker-1 v1 applied: role implementation, 1 project(s) [ACME], model your-provider/your-model
[pi-runner] lane worker-1 up - session "orboto-lane-worker-1", heartbeat every 30s

Step 4: worker vs. review lanes

The role a lane pulls with decides what kind of ticket it sees:

• implementation (the default) - pulls open, unclaimed tickets and does the actual work: it's the role that produces the commit.
• review - pulls tickets waiting for review instead of open ones, and never a ticket its own identity implemented (see Review lanes). Always run review lanes under a separate bot identity from the implementer lanes they check, and ideally on a different model - the cross-check is the point.
• preflight and integration - attach to a ticket without reassigning it or moving its status, for lanes that validate or integrate work alongside the implementer rather than owning the ticket themselves.

A fleet commonly runs more implementer lanes than review lanes, since review work is faster per ticket. Use routing labels (agent:<tag>) when you want finer steering than role alone provides.

Step 5: deploy under systemd

orboto ships a generic systemd template unit plus an example environment file. Because the configuration lives in orboto, the unit and the env file are now nearly identical for every lane - only the lane name and the key differ.

1. Create the service user the lanes run as:

   sudo useradd -r -m -d /var/lib/orboto-agents -s /usr/sbin/nologin orboto-agent

2. Install the binary at /usr/local/bin/orboto (however you normally install the orboto CLI on this host).

3. Create the template unit at /etc/systemd/system/orboto-agent@.service. It's a systemd template unit: the @ lets you start any number of instances (orboto-agent@worker-1, orboto-agent@review-1, ...), each reading its own environment file at /etc/orboto/agents/<lane>.env:

   # orboto pi-agent lane - one lane = one bot identity = one env file.
   # The env file holds the lane's IDENTITY only; everything the lane does
   # comes from orboto. Logs: journalctl -u orboto-agent@<lane> -f
   
   [Unit]
   Description=orboto pi agent lane %i
   After=network-online.target
   Wants=network-online.target
   
   [Service]
   Type=simple
   User=orboto-agent
   EnvironmentFile=/etc/orboto/agents/%i.env
   # Remote mode manages its own checkouts under the lane root
   # ($HOME/.orboto-lanes/<lane>), so this is only the starting cwd.
   WorkingDirectory=/var/lib/orboto-agents
   ExecStart=/usr/local/bin/orboto pi-runner --remote $ORBOTO_AGENT_EXTRA_ARGS
   Restart=on-failure
   RestartSec=10
   # A wedged provider call must not hold the lane forever on stop.
   TimeoutStopSec=30
   
   # Hygiene - the lane needs exactly: its workdir, the network, and /tmp.
   NoNewPrivileges=true
   PrivateTmp=true
   ProtectSystem=full
   ReadWritePaths=/var/lib/orboto-agents
   
   [Install]
   WantedBy=multi-user.target

4. Write one environment file per lane at /etc/orboto/agents/<lane>.env. Three identity variables, plus the credentials the box needs:

   # --- lane identity - the whole contract ---
   ORBOTO_BASE_URL=https://your-orboto-host.example.com/api
   ORBOTO_TOKEN=orb_xxxxxxxxxxxxxxxxxxxxx   # the lane's bot key
   ORBOTO_AGENT_LANE=worker-1               # matches the systemd instance name
   
   # --- provider key(s) for the model this lane is assigned ---
   YOUR_PROVIDER_API_KEY=...
   
   # --- git credentials for private repositories (optional) ---
   ORBOTO_AGENT_GIT_USER=x-access-token
   ORBOTO_AGENT_GIT_TOKEN=...

   Lock the file down - it holds an API key and, if you use one, a repo token: sudo chmod 600 /etc/orboto/agents/<lane>.env and confirm it's owned by root.

   The lane name must match the part after the @ in the unit instance, so journalctl, the file and the lane registry all read the same way.

5. Reload systemd and start each lane:

   sudo systemctl daemon-reload
   sudo systemctl enable --now orboto-agent@worker-1
   sudo systemctl enable --now orboto-agent@review-1

From here on, that box is done. Adding a project to a lane, swapping its model, tightening its priority ceiling, moving it to a different repository - all of it happens in orboto.

Retargeting a running lane

Added in v0.176.0 This is the day-to-day operation of a fleet, so it is worth stating plainly what happens when you press Apply assignment:

1. orboto stores the new assignment and bumps the lane's version.
2. The runner notices - either from the version echoed in its next heartbeat (30 seconds at worst) or, usually sooner, from the notification orboto sends the lane.
3. If the lane is idle, it tears down its session and applies the new assignment straight away. If it is working a ticket, it is told to finish that ticket cleanly first and only then switches; a session that wedges is given 30 minutes before the switch is forced and the abandoned lease is reclaimed server-side.
4. Working copies follow: a repository the new assignment no longer names is parked, not deleted, and the trees the new assignment does name are cloned or re-synced before the agent gets going.

The lane list shows the applied version, so you can see at a glance whether every lane has caught up with what you assigned.

Checkout lifecycle

Added in v0.176.0 A lane owns its working trees; you only name a repository and a branch in the assignment. Under the lane's root (~/.orboto-lanes/<lane> by default) the runner keeps one directory per assigned repository, plus its own state directory.

• Every wake verifies the base: fetch, check out the assigned branch, hard-reset to the remote. The agent never has to remember to pull.
• Nothing is reused blindly, and nothing is deleted in anger. A tree that cannot safely be reused - uncommitted changes, commits that exist nowhere else, not a git checkout at all, or a repository the assignment no longer names - is renamed to a timestamped sibling and replaced with a fresh clone. The log line says why; the parked copy keeps the evidence.
• Parked trees are cleaned up on a schedule: they are dropped once they pass a grace period (7 days by default) or when the lane's checkout directory exceeds its disk budget (10 GB by default), oldest first. Active trees are never pruned - if the budget cannot be met without deleting live work, that is reported rather than done.
• Credentials never travel in the assignment. Set ORBOTO_AGENT_GIT_TOKEN (and ORBOTO_AGENT_GIT_USER, default x-access-token) in the lane's environment file and the runner supplies it through a git credential helper - it reaches neither the process list nor the repository's config. Without it, whatever the host already has (a credential helper, an ssh agent) stays in charge. Every log line is scrubbed of embedded URL credentials, so a token cannot leak through git's own error text either.
• Each lane keeps its own state, so several lanes on one box never cross-report which ticket they are working on.

Watching lanes

• The fleet page: Admin → Agent Lanes shows every lane - online or offline, idle, working or paused, the ticket it currently holds, its assigned projects with their caps, its model, and the assignment version it has applied. A lane counts as online while its last heartbeat is recent.
• Logs: journalctl -u orboto-agent@<lane> -f - the runner prefixes every status line with a timestamp, and echoes a condensed view of what the agent said and which tools it called, so you can follow a lane's progress without attaching to the session directly.
• Active leases: orboto work-sessions --mine (run as, or with the token of, the lane's own identity) shows exactly what that lane currently holds; orboto work-sessions --ticket ACME-42 shows who - if anyone - holds a specific ticket.
• The workspace UI: tickets a lane is actively working show its identity as assignee, same as a human's would.

Pausing

Three switches, from narrowest to widest:

• The lane switch Added in v0.176.0 - Pause on the lane's assignment (or orboto lanes assign <lane> --paused). The runner keeps heartbeating, so the lane stays visible in the fleet page, but it pulls nothing. Resume with --resume; the lane picks work up again within one heartbeat.
• The per-bot toggle and the workspace-wide switch in Admin → Multi-Agent, described under pausing. These are identity-level, so they work the same whether the identity is connected through pi-runner, the MCP server, or anything else.

You never need to stop the systemd service to pause a lane, and stopping it is the worse option: a stopped lane also disappears from the fleet page, so you lose the one place that would have told you it was off.

Pinned mode: configuration on the box

The original model - every knob in the lane's environment file, a restart for every change - is still supported, unchanged. Use it when a box must run without depending on the registry, or when a lane's behaviour is deliberately frozen.

Override the unit's start command with a systemd drop-in (sudo systemctl edit orboto-agent@worker-1):

[Service]
ExecStart=
ExecStart=/usr/local/bin/orboto pi-runner --project ${ORBOTO_AGENT_PROJECT} --session-name %i $ORBOTO_AGENT_EXTRA_ARGS

and put the lane's configuration in its environment file instead of in orboto:

The same variables double as single-knob overrides on a remote lane: an ORBOTO_AGENT_* value set on the box still beats what orboto says - but never silently. Each override writes a warning to the log naming the server value it is shadowing, so a forgotten leftover is visible in journalctl instead of quietly steering the lane. Pin one knob to debug something, then delete the line.

Troubleshooting