Disaster Recovery

KubeManta stores its state — settings, alert rules, audit log, diagnosis cache, runbooks, Helm configs, AI guardrails — in a SQLite database inside a Kubernetes PVC at /data/kubemanta/kubemanta.db. The terminal home directory (/data/kubemanta/home) is also PVC-backed.

Your cluster's workload data is never stored by KubeManta — it is fetched live from the K8s API on demand.

---

What to back up

Data	Path on PVC	Notes
SQLite database	/data/kubemanta/kubemanta.db	Settings, alert rules, audit log, runbooks, diagnosis cache, Helm configs, guardrails
Terminal home	/data/kubemanta/home/	~/.claude, kubeconfigs, shell history, saved tool state

Your license key is stored in the kubemanta-license Kubernetes Secret — back this up separately or record it from the original install.

---

Option A+ — continuous SQLite → S3 replication (Litestream)

For a near-zero-RPO posture that survives node and PVC loss, KubeManta can continuously replicate the SQLite database to object storage using Litestream. This is off by default — a default install is unchanged.

When persistence.replication.enabled=true together with persistence.enabled=false:

• the DB directory becomes an emptyDir (you can drop the ReadWriteOnce PVC entirely),
• a restore init-container rebuilds kubemanta.db from the bucket on every boot, and
• a replicate sidecar ships the WAL continuously (sub-second RPO).

persistence:
  enabled: false          # DB dir becomes emptyDir; object storage is the source of truth
  replication:
    enabled: true
    bucket: my-kubemanta-db-backups     # REQUIRED — your own bucket, NOT in this cluster
    path: kubemanta-db
    region: us-east-1
    # endpoint: https://minio.example.com   # for MinIO / S3-compatible
    # forcePathStyle: true                   # for MinIO / path-style
    # credentialsSecret: kubemanta-s3-creds  # empty = use IRSA / workload identity

Prove it before you rely on it

Enabling replication renders correctly, but zero-data-loss restore is not proven until you run an attended kill-pod failover test against your real object store — a Helm render cannot demonstrate it. Do that once before you depend on it. Never host the bucket in the cluster you are observing (a correlated failure defeats the whole point).

The manual PVC-copy / snapshot procedures below remain valid and are the simplest path when replication is off.

---

Backup procedure

Option 1 — Copy from a running pod

# Backup SQLite database
kubectl cp kubemanta-system/$(kubectl get pod -n kubemanta-system -l app=kubemanta-agent -o jsonpath='{.items[0].metadata.name}'):/data/kubemanta/kubemanta.db ./kubemanta.db

# Backup terminal home (optional)
kubectl cp kubemanta-system/$(kubectl get pod -n kubemanta-system -l app=kubemanta-agent -o jsonpath='{.items[0].metadata.name}'):/data/kubemanta/home ./kubemanta-home-backup

Option 2 — Snapshot the PVC

Use your cluster's volume snapshot capability (requires VolumeSnapshot CRDs and a CSI driver that supports snapshots):

kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: kubemanta-backup-$(date +%Y%m%d)
  namespace: kubemanta-system
spec:
  volumeSnapshotClassName: your-snapshot-class
  source:
    persistentVolumeClaimName: kubemanta-data
EOF

---

Restore procedure

Step 1 — Reinstall or scale down

Scale down the agent to avoid concurrent writes:

kubectl scale deploy kubemanta-agent -n kubemanta-system --replicas=0

Step 2 — Copy the backup into the PVC

# Start a temporary pod to access the PVC
kubectl run restore-helper --image=busybox --restart=Never \
  --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"kubemanta-data"}}],"containers":[{"name":"restore-helper","image":"busybox","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data/kubemanta"}]}]}}' \
  -n kubemanta-system

# Copy the backup file in
kubectl cp ./kubemanta.db kubemanta-system/restore-helper:/data/kubemanta/kubemanta.db

# Clean up the helper pod
kubectl delete pod restore-helper -n kubemanta-system

Step 3 — Scale back up

kubectl scale deploy kubemanta-agent -n kubemanta-system --replicas=1
kubectl rollout status deploy/kubemanta-agent -n kubemanta-system --timeout=120s

Step 4 — Verify

Open the KubeManta UI and confirm settings, alert rules, and runbooks are present.

---

Recovery time objective

Step	Estimated time
Scale down	< 1 min
Copy backup into PVC	1–2 min
Scale up + health check	2–3 min
Verify in UI	1 min
Total	~5–7 min

RPO depends on your backup frequency. The SQLite file is small (typically < 10 MB) and can be backed up frequently.

---

License key recovery

If you lose your license key:

1. Check the kubemanta-license Secret — this works offline and is the fastest path:

   kubectl get secret kubemanta-license -n kubemanta-system -o jsonpath='{.data.key}' | base64 -d
   

2. Otherwise use the license portal at api.kubemanta.com/portal. Enter the email you bought with and it emails you a sign-in link; your keys are listed once you follow it. There is no password and no account to create — the link is the authentication.

3. For the free tier you can also re-run the signup call with the same email; the existing key is returned rather than a new one being minted ("existing": true).

---

Reinstall on a new cluster

A new cluster has a different kube-system namespace UID, so it counts as a new cluster activation. With a single-cluster Pro license, the old activation is automatically freed 30 minutes after the old cluster stops sending heartbeats (after uninstall). The new cluster then activates within one heartbeat cycle.

Activations are not admin seats

A cluster activation tracks which cluster a license runs on. An admin seat is what you pay for, and tracks a person who can change the cluster. Moving to a new cluster does not consume or free an admin seat.

For faster migration, contact support to manually deactivate the old cluster activation.