Disaster Recovery

KubeManta stores its state — settings, alert rules, audit log, diagnosis cache, runbooks, Helm configs, AI guardrails — in a SQLite database inside a Kubernetes PVC at /data/kubemanta/kubemanta.db. The terminal home directory (/data/kubemanta/home) is also PVC-backed.

Your cluster's workload data is never stored by KubeManta — it is fetched live from the K8s API on demand.

---

What to back up

Data	Path on PVC	Notes
SQLite database	/data/kubemanta/kubemanta.db	Settings, alert rules, audit log, runbooks, diagnosis cache, Helm configs, guardrails
Terminal home	/data/kubemanta/home/	~/.claude, kubeconfigs, shell history, saved tool state

Your license key is stored in the kubemanta-license Kubernetes Secret — back this up separately or record it from the original install.

---

Option A+ — continuous SQLite → S3 replication (Litestream)

For a near-zero-RPO posture that survives node and PVC loss, KubeManta can continuously replicate the SQLite database to object storage using Litestream. This is off by default — a default install is unchanged.

When persistence.replication.enabled=true together with persistence.enabled=false:

• the DB directory becomes an emptyDir (you can drop the ReadWriteOnce PVC entirely),
• a restore init-container rebuilds kubemanta.db from the bucket on every boot, and
• a replicate sidecar ships the WAL continuously (sub-second RPO).

persistence:
  enabled: false          # DB dir becomes emptyDir; object storage is the source of truth
  replication:
    enabled: true
    bucket: my-kubemanta-db-backups     # REQUIRED — your own bucket, NOT in this cluster
    path: kubemanta-db
    region: us-east-1
    # endpoint: https://minio.example.com   # for MinIO / S3-compatible
    # forcePathStyle: true                   # for MinIO / path-style
    # credentialsSecret: kubemanta-s3-creds  # empty = use IRSA / workload identity
    age:
      recipients:                          # WITHOUT THIS THE REPLICA IS CLEARTEXT
        - age1qz...                        # your age public key(s)

Two things that decide whether your restore is usable

1. Set age.recipients, or the replica is cleartext. Only a handful of settings are field-encrypted inside the database; the rest of the file is not. Without age, everything below leaves the cluster in the clear and the bucket's own controls are your only boundary:

• audit_log — 365 days of retention
• terminal_audit / terminal_commands — 180 days of the command lines your operators typed, with account and timestamp
• users — SSO identity mapping and local password hashes
• workspace_files — imported pod logs

If you cannot use age, treat the bucket as holding that material: SSE-KMS, block public access, and a region your data-residency commitments allow.

2. Export the <release>-enc Secret and store it with your break-glass material — it is NOT in the bucket. It lives in the namespace, not the database. A restore without it brings everything back except a handful of settings that stay permanently unreadable: cloud credentials, SSO config, saved Helm values and the LLM key. The restore reports success and the gap surfaces mid-incident, which is the worst shape a backup failure can take.

kubectl get secret <release>-enc -n kubemanta-system -o yaml > kubemanta-enc.backup.yaml

Keep that file wherever you keep your other break-glass secrets — not in the replication bucket, which would defeat the separation.

Prove it before you rely on it

Run an attended kill-pod failover test against your real object store once, before you depend on this. No amount of configuration review substitutes for watching a restore actually complete. Never host the bucket in the cluster you are observing (a correlated failure defeats the whole point).

The manual PVC-copy / snapshot procedures below remain valid and are the simplest path when replication is off.

---

Backup procedure

Option 1 — Copy from a running pod

# Backup SQLite database
kubectl cp kubemanta-system/$(kubectl get pod -n kubemanta-system -l app=kubemanta-agent -o jsonpath='{.items[0].metadata.name}'):/data/kubemanta/kubemanta.db ./kubemanta.db

# Backup terminal home (optional)
kubectl cp kubemanta-system/$(kubectl get pod -n kubemanta-system -l app=kubemanta-agent -o jsonpath='{.items[0].metadata.name}'):/data/kubemanta/home ./kubemanta-home-backup

Option 2 — Snapshot the PVC

Use your cluster's volume snapshot capability (requires VolumeSnapshot CRDs and a CSI driver that supports snapshots):

kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: kubemanta-backup-$(date +%Y%m%d)
  namespace: kubemanta-system
spec:
  volumeSnapshotClassName: your-snapshot-class
  source:
    persistentVolumeClaimName: kubemanta-data
EOF

---

Restore procedure

Step 1 — Reinstall or scale down

Scale down the agent to avoid concurrent writes:

kubectl scale deploy kubemanta-agent -n kubemanta-system --replicas=0

Step 2 — Copy the backup into the PVC

# Start a temporary pod to access the PVC
kubectl run restore-helper --image=busybox --restart=Never \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"kubemanta-data"}}],"containers":[{"name":"restore-helper","image":"busybox","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data/kubemanta"}]}]}}' \
  -n kubemanta-system

# Copy the backup file in
kubectl cp ./kubemanta.db kubemanta-system/restore-helper:/data/kubemanta/kubemanta.db

# Clean up the helper pod
kubectl delete pod restore-helper -n kubemanta-system

Step 3 — Scale back up

kubectl scale deploy kubemanta-agent -n kubemanta-system --replicas=1
kubectl rollout status deploy/kubemanta-agent -n kubemanta-system --timeout=120s

Step 4 — Verify

Open the KubeManta UI and confirm settings, alert rules, and runbooks are present.

---

Recovery time objective

Step	Estimated time
Scale down	< 1 min
Copy backup into PVC	1–2 min
Scale up + health check	2–3 min
Verify in UI	1 min
Total	~5–7 min

RPO depends on your backup frequency. The SQLite file is small (typically < 10 MB) and can be backed up frequently.

---

License key recovery

If you lose your license key:

1. Check the kubemanta-license Secret — this works offline and is the fastest path:

   kubectl get secret kubemanta-license -n kubemanta-system -o jsonpath='{.data.key}' | base64 -d
   

2. Otherwise use the license portal at api.kubemanta.com/portal. Enter the email you bought with and it emails you a sign-in link; your keys are listed once you follow it. There is no password and no account to create — the link is the authentication.

3. For the free tier you can also re-run the signup call with the same email; the existing key is returned rather than a new one being minted ("existing": true).

---

Reinstall on a new cluster

A new cluster has a different kube-system namespace UID, so it counts as a new cluster activation. With a single-cluster Pro license, the old activation is released automatically once the licensing service's heartbeat window lapses after uninstall. The agent pings every 6 hours by default, so allow hours rather than minutes; if you need the new cluster live immediately, contact us to release the activation.

Activations are not admin seats

A cluster activation tracks which cluster a license runs on. An admin seat is what you pay for, and tracks a person who can change the cluster. Moving to a new cluster does not consume or free an admin seat.

For faster migration, contact support to manually deactivate the old cluster activation.