Backups and restore

An installation of Labtagon AI OS keeps its state in three places: Postgres, the object store with the files’ bytes, and the keys. A restore needs all three, from times that fit together.

Postgres

Postgres holds everything except the bytes of files: chats, tool servers with their versions and code bundles, approvals, settings, memories, the audit log, and the stored sign-ins and keys of tool servers as age ciphertext.

The chart creates a CloudNativePG cluster (postgres.create) but sets up no backup for it. Set up CloudNativePG’s backups yourself, with continuous WAL archiving so that you can restore to a point in time, to storage outside the cluster.

• Keep the backups at least as long as storage.graceDays (90 days by default), or lower that value to your retention: an object no file names any more stays in the store for that long, so a restore of Postgres within it still finds every file.
• The database is called ai-os and is owned by ai-os; the chart’s Database resource keeps it when the release goes (databaseReclaimPolicy: retain).
• With postgres.create: false, your own Postgres and its backups are yours.

The files

Files’ bytes are in Garage or your S3, encrypted before they leave AI OS (ADR 0093): the bucket, its volumes and any copy hold only ciphertext.

• Garage takes a snapshot of its metadata every 6 hours (storage.garage.metadataSnapshotInterval) into its metadata volume. Take volume snapshots of its volumes after one.
• An off-site copy: storage.offsite.enabled: true runs rclone on a schedule (17 3 * * * by default) and copies new objects to another S3. It deletes nothing there, so a write-only, object-locked bucket suits it.
• External S3: use your provider’s versioning or replication.
• With storage.mode: postgres, files are in Postgres and its backup covers them.

The keys you must never lose

Key	Where	Without it
age-identity (keys Secret)	keys.existingSecret, ideally from OpenBao	No stored sign-in or key of a tool server can be decrypted; people have to enter them again.
approval-signing-key, approval-verify-key	keys Secret	Every signed approval is invalid. The keys of LLM providers are encrypted with a key derived from the signing key: they can no longer be read either.
agent-token-key	keys Secret	The tokens AI OS issued to clients no longer verify.
egress-ca.crt, egress-ca.key, session-key	keys Secret	Keep them with the others; the chart’s notes list how each is made.
Storage key storage-key (and previous-keys)	<release>-storage-key, or storage.encryption.existingSecret	No file can be read.

Keep these apart from the database backup: in OpenBao (keys.externalSecret, storage.encryption.externalSecret) or another safe place. When the chart made the storage key itself, copy the Secret <release>-storage-key out of the cluster right after the install. It keeps that Secret when the release goes.

Restore

1. Postgres to the chosen time, with CloudNativePG’s point-in-time recovery.
2. The objects: with Garage, its volumes as they are, or on each node the volume snapshot taken after a metadata snapshot (stop the node, put the snapshot in place of the metadata database, start it, run garage repair -a --yes tables); with an off-site copy, copy it back into the bucket (rclone copy, nothing deleted).
3. The same keys, the storage key and its previous keys included, in their Secrets.
4. Check that every file names an object the store has, and that files open. Objects the restored database does not know stay for the grace period and are then removed.

A one-node Garage cannot simply be restored into an HA one: start the HA Garage, copy the bucket over, then switch.

Tool servers’ own volumes

A tool server with persistent work keeps it on a volume of its own (work-<name>) in its owner’s namespace. For a backup tool that selects by label, instances.workVolumeLabels puts labels on these volumes and on the pods that mount them, for example { velero/backup: enabled }.