Troubleshooting¶
ImagePullBackOff on kubemanta pods¶
Symptom: kubectl get pods -n kubemanta-system shows ImagePullBackOff or ErrImagePull.
Causes and fixes:
No license key / wrong key¶
# Check if the pull secret exists
kubectl get secret kubemanta-pull-secret -n kubemanta-system
# Recreate with the correct key
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version 1.0.0 \
-n kubemanta-system --reset-then-reuse-values \
--set license.key=YOUR_CORRECT_KEY
The chart creates the kubemanta-pull-secret from license.key. If the key is wrong or expired, the pull secret contains an invalid password and image pulls fail.
Pull secret not in the right namespace¶
If missing, re-run the Helm upgrade with --set license.key=....
Registry connectivity¶
Verify the agent pod can reach registry.kubemanta.com:
kubectl run net-test --image=busybox --restart=Never -n kubemanta-system -- \
wget -qO- https://registry.kubemanta.com/v2/
kubectl delete pod net-test -n kubemanta-system
Expert mode returning 403 on write operations¶
Symptom: AI write tools (delete pod, restart deployment, scale) or terminal Helm commands fail with 403 / permission denied.
Cause: expertMode.enabled is false (the default). Without it, the kubemanta-expert cluster-admin ServiceAccount is not created.
Fix:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version 1.0.0 \
-n kubemanta-system --reset-then-reuse-values \
--set expertMode.enabled=true
kubectl rollout restart deploy/kubemanta-agent -n kubemanta-system
Note: rbac.allowWrites=true is also required for the reader SA to perform pod/deployment write ops. These are separate controls:
rbac.allowWrites=true— grants the main agent SA write verbs (pod restart/delete, deploy scale)expertMode.enabled=true— creates the cluster-admin SA used by the terminal and Helm write ops
AI diagnosis not working / "AI is disabled"¶
Check the master AI switch¶
In Admin → AI Guardrails, verify AI Enabled is on.
No API key provided¶
The AI diagnosis panel requires an LLM API key entered in AI Settings (the key icon in the top bar). The key is passed per-request and never stored.
Data residency restriction¶
If Local Only is enabled in AI Guardrails, only providers in the allowlist are accepted. Check that your selected provider is in the allowlist.
Rate limit¶
The default AI rate limit is 10 requests per minute per session (security.aiRateLimitPerMin). If you are hitting it, increase via Helm:
Metrics: bandwidth charts are empty¶
Symptom: Metrics → Built-in shows CPU/memory fine, but Network Bandwidth is empty.
Cause 1 — kubelet scrape not enabled (the default). Bandwidth needs the opt-in nodes/proxy RBAC grant:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version 1.0.0 \
-n kubemanta-system --reset-then-reuse-values \
--set metrics.kubeletScrape.enabled=true
Cause 2 — metrics source set to prometheus. When the runtime metrics source is prometheus, KubeManta performs zero kubelet scrapes by design. Check (and see live scrape health — last error, nodes scraped/failed) with:
kubectl exec -n kubemanta-system deploy/kubemanta-agent -- \
python3 -c "import urllib.request, json; r=urllib.request.urlopen('http://localhost:8080/metrics/source'); print(json.loads(r.read()))"
Also allow a couple of minutes after enabling — rates need at least two consecutive samples (the sampler runs every 60s by default).
Metrics: Prometheus sub-tab is blank despite Prometheus being reachable¶
If the Prometheus sub-tab reports it is reachable but has no namespace/pod-labelled series, your Prometheus is scraping a standalone/Docker cadvisor (series carry only id/interface labels) instead of the cluster's kubelet/cadvisor. Point it at the cluster — kube-prometheus-stack does this out of the box, or use Admin → Observability → Generate scrape config to get the exact read-only RBAC + prometheus.yml job. See Observability.
GKE Ingress / TLS issues¶
Ingress has a wrong (ephemeral) IP instead of the static IP¶
The global.staticIpName annotation only takes effect at LB creation. If the LB already exists with an ephemeral IP, delete the ingress and let GKE recreate it:
kubectl delete ingress kubemanta -n kubemanta-system
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version 1.0.0 \
-n kubemanta-system --reset-then-reuse-values --set global.staticIpName=YOUR_STATIC_IP_NAME
ManagedCertificate stuck in Provisioning¶
DNS must resolve to the ingress IP before GCP will issue the certificate. Check:
kubectl get ingress kubemanta -n kubemanta-system # ADDRESS should be the static IP
kubectl get managedcertificate -n kubemanta-system # STATUS: Active when cert is ready
If stuck for more than 1 hour:
kubectl delete managedcertificate kubemanta-staging-cert -n kubemanta-system
helm upgrade kubemanta ... --reset-then-reuse-values # Chart recreates it; GCP retries immediately
spec.tls conflict with GCE Ingress¶
For ingressClassName=gce with global.tlsProvider=gcp-managed, the chart omits spec.tls from the Ingress manifest. If you see spec.tls in the rendered ingress (e.g. from a manual override), remove it — GCE LB provisioning stalls silently when spec.tls references a missing secret.
ALB health check failing (EKS)¶
Symptom: Pods are Running but the ALB target group shows UNHEALTHY.
Cause: The default health check path / redirects to /login (HTTP 307 → HTTP 200), but the ALB health check follows redirects and may see an unexpected response.
Fix: Ensure the ALB target group health check uses /healthz (returns 200 with no auth required), or use success codes 200-401:
# This is set automatically by the chart; verify the annotation is present:
kubectl get ingress kubemanta -n kubemanta-system -o yaml | grep healthcheck
If missing, upgrade to the latest chart version:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version 1.0.0 \
-n kubemanta-system --reset-then-reuse-values
Terminal / PTY not connecting¶
Symptom: Opening the terminal tab shows a spinner or "WebSocket connection failed".
Cause: WebSocket connections to /api/agent/terminal/ws/* must be routed directly from the Ingress to the agent service — they cannot pass through Next.js (App Router does not support HTTP→WebSocket upgrade).
Fix: Verify the Ingress has a dedicated rule for the terminal path:
If the terminal path rule is missing, upgrade to the latest chart version.
Checking the license status¶
# From inside the cluster
kubectl exec -n kubemanta-system deploy/kubemanta-agent -- \
python3 -c "import urllib.request, json; r=urllib.request.urlopen('http://localhost:8080/license'); print(json.loads(r.read()))"
Returns {"state": "active"|"unlicensed"|"grace", "plan": "free"|"pro", "expiry": "...", "banner": "..."}.