Import and sync limits

Added in v0.230.0 orboto checks the size of every archive, file and answer before it keeps it in memory: backup archives you restore or import, calendar files, CalDAV servers, another orboto instance you move from, and mail it fetches over IMAP. Input above a limit is refused with a message that names the limit. A refused restore or import changes nothing.

Where to set the limits

Added in v0.230.0 Admin -> Backup -> Import and sync limits lists every limit you can change, in four groups: Full restore, Project import, Import from another instance and Calendar (.ics import and CalDAV). Each row shows the unit, the allowed range, the default and the setting key. Save stores one row. You need the admin:system:write permission to see and change the section.

A change applies to the next restore, import or sync, without a restart. A value outside the range is refused with errors.system.invalid_config_number, and a stored value that is somehow outside its range reads as the default. Every limit is also a system setting you can change through the API:

orboto patch /system/admin/config '{"key":"restore_max_entries","value":"200000"}'

The sections below list each limit with its key, default and range.

Restoring a backup

Added in v0.230.0 An archive you upload is streamed to a temporary file on the server and never held in memory as a whole. A sealed (encrypted) archive is decrypted into a second temporary file. Both files are removed when the request ends, also when it fails. This holds for the setup wizard restore, the read-only inspection, Admin -> Backup -> Import, the full restore and the download of a restore drill.

A restore reads the archive in two passes and never holds a table in memory:

1. Check. orboto reads every table line by line and keeps only what the checks need: row counts, the keys that identify each row, the keys that other rows point at, the users in the archive and the files the rows refer to.
2. Restore. Inside the restore transaction orboto reads each table again and inserts it in batches of at most 500 rows or 1 MiB. A batch the database refuses is retried row by row, so the error names the row. Stored files stream straight into your storage, and a file may not be longer than its entry in the archive says.

There is no cap on the number of rows or on the size of a table. A workspace with millions of rows restores like a small one, as long as the archive stays within the limits below.

The entry count is read from the end of the ZIP file before the archive's file list is parsed, so an archive that claims millions of entries is refused at once. The read-only inspection on Admin -> Backup -> Import applies the same limits, so an archive over a limit is refused there, before anything is wiped.

To judge an archive before you upload it, count the files you expect inside: a few hundred tables plus one entry per stored file must stay below Most entries in one archive, and the database plus the files, unpacked, below Largest archive, decompressed. orboto counts the bytes it really unpacks, not only the sizes the archive declares, because an archive can understate its sizes (a zip bomb).

What a refusal looks like

Added in v0.230.0 A refused restore names the limit it hit:

None of these leaves anything behind: the restore is not committed and the temporary files are removed. If a real backup of yours hits a limit, raise the limit and upload again.

An archive comment is refused

Added in v0.230.0 A ZIP file can carry a free-text comment after its end record. orboto archives never do, and orboto reads the end record from the very end of the file, so an archive with a comment is refused with "Invalid backup data in archive" and the reason the end record does not end the archive. If another tool re-saved your archive and added a comment, download the original from the run history again.

One archive at a time

Added in v0.230.0 The setup wizard restore, the inspection, the import, the full restore, the restore drill and the import from another instance share one lock per instance, so only one archive is staged at a time. While one of them runs, a second answers 409 with errors.system.import_already_running: "A workspace import is already running on this instance. Wait until it has finished, then check the result before you start another one."

• Added in v0.230.4 A run frees the instance before it answers, so a request you send after the answer is never refused.
• Added in v0.230.4 A request sent while another run is still going is refused at once.
• The lock is held for the whole run and freed when the run ends. A run stuck in a request that never ends keeps the lock until its database connection ends. Restart the API to free it.

Importing a project

Added in v0.230.0 These limits apply when you import a project archive as a new project, replace a project from an archive, or import a project archive on Admin -> Backup -> Import. On that page a project archive is first accepted up to Largest archive a restore accepts and then refused when it is larger than Largest project archive.

A project export holds about fifteen files. An archive with more than 64 entries is refused, and orboto unpacks only the files a project export holds. It drops each file's text once it has been read. A file that unpacks to more than 1 MB and to more than 200 times its compressed size is refused too; for such a file the message names a smaller limit than your setting.

The refusals are the same messages as for a restore: 413 errors.backups.archive_upload_too_large for an upload over the archive limit, errors.backups.archive_too_large and errors.backups.archive_entry_too_large for the unpacked sizes.

Moving a workspace from another instance

Added in v0.230.0 The import from another orboto instance reads every answer of the source under a limit, and nothing is written when one is exceeded:

• The source's version answer may be at most 64 KiB and its list of stored files at most 64 MiB. Both are fixed.
• The source's archive streams to a temporary file and may not be larger than Largest archive a restore accepts. The restore that follows runs under the restore limits above, and orboto checks the archive once, not twice.
• Every file streams straight into your storage. It may not be longer than the size the source announced, and a file listed with a size above Largest single entry, decompressed makes the list invalid.
• A source that sends nothing at all for the idle timeout ends the import.

A breach answers 400 with errors.system.import_source_too_large or errors.system.import_source_timeout, naming the part that failed and the setting. A file that is longer than announced ends the import with errors.system.import_object_transfer_failed. Every file this run wrote is removed again; start the import again.

Calendars

Added in v0.230.0 These limits cover the .ics import (personal absences and company closures) and CalDAV sync.

The .ics import

An upload over Largest .ics file is refused with 413 and errors.absences.ics_file_too_large. A file with more events than Most events in one .ics import is not imported at all, so you never end up with half of it; the message asks you to split the file or to raise the limit (errors.absences.ics_too_many_events).

Before orboto parses the file it also refuses a file that nests its entries deeper than 8 levels (errors.absences.ics_too_deep) or holds more than 20000 entries (errors.absences.ics_too_many_components). Those two bounds are fixed. Event titles are shortened to 500 characters and notes to 5000. The import result lists at most 100 warnings and then one line that counts the rest.

CalDAV

Every request to your CalDAV server runs under Seconds to wait for a CalDAV server, the time to read the answer included. An answer is read only up to Largest CalDAV server answer, and an answer whose XML nests deeper than 32 levels is refused. One sync stores at most Most busy events per CalDAV sync busy windows.

• Connecting answers 400 with errors.calendar.caldav_response_too_large or errors.calendar.caldav_timeout. The Connect CalDAV form shows a message that names the limit and asks the person to have an administrator raise it; see Messages when connecting.
• A sync that hits a limit keeps the busy times it had before and records the reason, caldav_response_too_large, caldav_timeout or caldav_too_many_events, as the account's lastError. In Profile -> Connected calendars the connection shows a message that names the limit and tells the person to ask an administrator to raise it (see CalDAV sync messages); the reason code is in GET /calendar/accounts.

Inbound mail over IMAP

Added in v0.230.0 orboto never downloads a message larger than the maximum message size of inbound mail (the system setting inbound_max_size_mb, default 10 MB). It asks the mail server for unread messages up to that size, checks the size the server reports again, and caps the bytes it reads, so a server that reports a wrong size cannot push more into memory.

A message above the limit stays unread in the mailbox for a person to handle. orboto marks it with the IMAP keyword $OrbotoTooLarge, so later polls skip it, and reports it once in the result of Poll now as oversized. The account's row under Admin -> Inbound IMAP counts them ("3 messages skipped as too large"). One poll fetches at most 200 messages; the rest follow with the next poll. See IMAP polling.

Connected services

Added in v0.230.0 The answers of services orboto talks to are capped with fixed values:

The error answer of an import source or a git provider is read up to 64 KB and used only for the message. An answer above its cap fails that call, and the import, the poll or the delivery reports the error.

For operators: where temporary files go

Added in v0.230.0 Archives are staged in the system's temporary directory unless you set IMPORT_STAGING_DIR. Point it at a volume with enough free space for the largest archive you allow, and twice that for a sealed archive while it is decrypted. Each request writes into its own private folder (orboto-import- followed by random characters, permission 0700), the file inside is readable by the API only (0600), and the folder is removed when the request ends.

At start-up the API:

1. creates IMPORT_STAGING_DIR when it is missing;
2. refuses to start when it cannot write the directory, or when everybody may write it without the sticky bit (on such a directory any local user could replace or delete a staged archive);
3. removes the folders (orboto-import-* and orboto-restore-drill-*) that a process killed in the middle of a restore left behind.

The same sweep runs again every time an archive operation finishes and releases the lock. It touches only folders older than the running process and older than 15 minutes, and only while no restore or import holds the lock, so a running upload is never swept.

Troubleshooting

Added in v0.230.0 Each entry below concerns a limit or a refusal described on this page.

• "A workspace import is already running on this instance" - another restore, inspection, import, restore drill or import from another instance holds the lock. Wait for it and check its result. When nothing is running, a request got stuck: restart the API to free the lock.
• The API does not start after you set IMPORT_STAGING_DIR - the log names the directory and the reason: it cannot be created, it is not writable by the API, or it is writable by everybody without the sticky bit. Fix the permissions (for example chmod 700, or chmod 1777 for a shared directory) and start again.
• A restore stops with "The archive holds N entries" - your attachment history has many files. Raise Most entries in one archive; the ZIP reader holds about 1 to 2 KB per entry while the file list is open, so 100000 entries cost 100 to 200 MB of memory.
• A restore is refused as "Invalid backup data in archive" - see An archive comment is refused and the table of refusals above. When the archive came straight from a backup run and nothing touched it, download it again and compare its size.
• An .ics import yields nothing and says it holds too many events - split the file, or raise Most events in one .ics import. Repeating events count once per occurrence.
• A CalDAV connection shows a limit message and the busy times do not move - the message on the connection (Profile -> Connected calendars) names the setting: Largest CalDAV server answer (caldav_response_too_large), Seconds to wait for a CalDAV server (caldav_timeout) or Most busy events per CalDAV sync (caldav_too_many_events). The same reason is the account's lastError in GET /calendar/accounts. Raise the matching limit or reduce what the calendar holds; CalDAV sync messages lists every message.
• The IMAP card counts skipped messages - messages larger than the maximum message size wait unread in the mailbox. Open them in the mailbox, or raise inbound_max_size_mb.