Monitoring and health checks

This page covers what your own monitoring stack needs to watch a self-hosted orboto instance: the two health endpoints, the metrics endpoint, and log behaviour. It applies to self-hosted installs - on orboto Cloud, monitoring is the platform's job.

Health endpoints

Endpoint	Checks	Use it for
`GET /health`	The process is up. Nothing else.	Container liveness - restart the container if this stops responding.
`GET /ready`	Postgres and your S3-compatible storage are reachable. Returns `503` if either is down.	Container readiness - don't route traffic here until this is `200`. Also a good target for an external uptime monitor.

Point your orchestrator's liveness probe at /health and its readiness probe at /ready. If you don't run an orchestrator, an external uptime monitor hitting /health on an interval is enough to know the process is alive; hit /ready as well if you want to be alerted specifically on a database or storage outage rather than a process crash.

Sanity-check both from the host right after you deploy:

curl -i https://your-domain.example.com/health   # expect: 200, body {"status":"ok"}
curl -i https://your-domain.example.com/ready    # expect: 200 once Postgres + storage are reachable

No monitoring stack yet? You don't need Prometheus to get basic coverage. Any free or paid external uptime-check service (the kind that just polls a URL on an interval and emails/pages you on failure) works fine pointed at /health - that alone catches "the whole instance is down", which is most of what actually goes wrong on a small deployment. Add /ready as a second check once you want to distinguish "the process crashed" from "the database is unreachable but the process is still up".

Metrics

GET /metrics exposes Prometheus-format metrics: process and HTTP-request metrics plus a handful of orboto-specific counters. It is gated in production so it never leaks route topology or request volume publicly - one of these must be set, or the endpoint returns 503:

METRICS_ALLOWED_CIDRS - comma-separated CIDR allowlist (IPv4 and IPv6), e.g. 10.0.0.0/8,127.0.0.1/32, so your Prometheus host can scrape it.
METRICS_BASIC_AUTH - user:pass; the scraper sends HTTP Basic auth.

In development, with both unset, /metrics is open on localhost.

# prometheus.yml
scrape_configs:
  - job_name: orboto-api
    metrics_path: /metrics
    # If you use METRICS_BASIC_AUTH instead of a CIDR allowlist:
    # basic_auth: { username: prom, password: <pass> }
    static_configs:
      - targets: ["orboto-api:3000"]

Metrics worth alerting on:

Metric	Meaning
`http_request_duration_seconds_bucket`	HTTP latency histogram (per route/method/status) - drives p50/p95/p99.
`http_request_duration_seconds_count`	Request rate; filter `status_code=~"5.."` for the error rate.
`orboto_job_queue_size`	Background job count by state - queue depth.
`nodejs_eventloop_lag_p99_seconds`	Event-loop lag - the canary for a saturated process.
`nodejs_heap_size_used_bytes`, `process_resident_memory_bytes`	Memory.
`process_cpu_seconds_total`	CPU (rate it for the number of cores in use).

Any Prometheus-compatible dashboard tool can visualise these once scraping works - point it at the same Prometheus data source you configured above.

Capacity, embedding and restore metrics

Added in v0.212.0 These counters and gauges are on the same endpoint:

Metric	Meaning
`orboto_capacity_queue_depth{resource}`	Claims queued on a build capacity resource right now; idle resources report 0.
`orboto_capacity_wait_seconds`, `orboto_capacity_hold_seconds`	Histograms of the time from filing to grant and from grant to release or expiry.
`orboto_capacity_claim_transitions_total{resource,transition}`	Claim state changes, including `expired` and `yielded`.
`orboto_capacity_claims_filed_total{resource,client}`	Claims filed, by `ci`, `cli`, `mcp`, `web` or `api`.
`orboto_capacity_reservations_total{resource,event}`	Reservation events: `booked`, `granted`, `used`, `released`, `expired`, `expired_unused`.
`orboto_capacity_overrun_alerts_total{resource}`	Alerts for a claim held for twice its expected time.
`orboto_embedding_calls_total{outcome,reason}`	Calls to the embedding provider. `outcome` is `ok`, `retried` (one in-call retry) or `failed` (one call that used up its retries); `reason` is `status`, `timeout`, `network` or `none`. A steady `retried` rate with `timeout` points at a slow provider.
`orboto_restore_drill_total{status}`	Restore drills on a scratch instance by `passed`, `failed` or `rejected`.
`orboto_scratch_suppressed_total{path}`	Outbound side effects a scratch instance suppressed, by gate.

Logs

The API logs structured JSON (via Pino). Authorization headers and passwords are redacted automatically before a line is written, so logs are safe to ship to a third-party aggregator. Control verbosity with LOG_LEVEL (info in production, debug in development).

docker compose logs -f api

Error tracking (optional)

Set SENTRY_DSN on the API service to start shipping unhandled API exceptions and frontend crashes to a Sentry-compatible error tracker. The web bundle picks up the same DSN automatically. Leaving it unset simply turns this integration off - orboto's own in-app error log for administrators keeps working either way.

Added in v0.229.0 orboto keeps personal and infrastructure data out of what it sends, whatever the SDK would collect by default:

no user fields, client IP, cookies or request and response bodies
no database query data and no database user name on query spans
no identity headers such as forwarded-for
token-bearing URLs (reset, invitation, magic-link and setup links) and sensitive query values (code, state, token, email, search terms) arrive as [REDACTED], from the API and from the browser

Traces are sent as transactions rather than streamed spans, so GlitchTip accepts them (it discards streamed span items). The Sentry CLI is not part of the images.

Troubleshooting

/metrics always returns 503. Cause: in production, neither METRICS_ALLOWED_CIDRS nor METRICS_BASIC_AUTH is set - the endpoint refuses to serve anyone. Fix: set one of the two.

My Prometheus host gets 401 or 403 from /metrics. Cause: its IP isn't inside METRICS_ALLOWED_CIDRS, or the Basic Auth credentials don't match METRICS_BASIC_AUTH. Fix: widen the CIDR to include the scraper's real source IP, or fix the credentials on either side.

/ready flaps between 200 and 503. Cause: Postgres or the S3-compatible storage is intermittently unreachable from the API container - a network issue, not an application bug. Fix: check the health of those two services and the network path between them and the API container.

Nothing shows up in my error tracker. Cause: SENTRY_DSN isn't set. Fix: set it on the API service; it takes effect on the next restart.