Troubleshooting¶
ImagePullBackOff on kubemanta pods¶
Symptom: kubectl get pods -n kubemanta-system shows ImagePullBackOff or ErrImagePull.
Causes and fixes:
No license key / wrong key¶
# Check if the pull secret exists
kubectl get secret kubemanta-pull-secret -n kubemanta-system
# Recreate with the correct key
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version CHART_VERSION \
-n kubemanta-system --reset-then-reuse-values \
--set license.key=YOUR_CORRECT_KEY
Which version?
Replace CHART_VERSION with the release you are installing. The newest is at
the top of the Changelog, or read it from the
registry once you are logged in:
Always pin it. An unpinned install resolves to whatever stable points at
today, which is not necessarily the release you tested against.
The chart creates the kubemanta-pull-secret from license.key. If the key is wrong or expired, the pull secret contains an invalid password and image pulls fail.
Pull secret not in the right namespace¶
If missing, re-run the Helm upgrade with --set license.key=....
Registry connectivity¶
Verify the agent pod can reach registry.kubemanta.com:
kubectl run net-test --image=busybox --restart=Never -n kubemanta-system -- \
wget -qO- https://registry.kubemanta.com/v2/
kubectl delete pod net-test -n kubemanta-system
Expert mode returning 403 on write operations¶
Symptom: AI write tools (delete pod, restart deployment, scale) or Helm operations from the UI fail with 403 / permission denied.
Cause: expertMode.enabled is false (the default). Without it, the <release>-expert cluster-admin ServiceAccount (kubemanta-expert for the default release name) is not created.
Fix:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version CHART_VERSION \
-n kubemanta-system --reset-then-reuse-values \
--set expertMode.enabled=true
kubectl rollout restart deploy/kubemanta-agent -n kubemanta-system
Note: rbac.allowWrites=true is also required for the reader SA to perform pod/deployment write ops. These are separate controls:
rbac.allowWrites=true— grants the main agent SA write verbs (pod restart/delete, deploy scale)expertMode.enabled=true— creates the cluster-admin SA used for Helm write ops
The browser terminal does not use Expert Mode: kubectl there runs as the terminal's own ServiceAccount (<release>-terminal), with KubeManta's read permissions and, only with rbac.allowWrites, its writes. So a forbidden in the terminal usually means rbac.allowWrites is off (or terminal.clusterCredentials.enabled=false withheld the credential). kubectl get secret is refused there unless the namespace is listed in terminal.secretsNamespaces (for one more release, also rbac.secretsNamespaces, or rbac.tenantNamespaces with rbac.clusterWide=false), and always in KubeManta's own namespace; the product's own Secret views are not affected.
With rbac.allowWrites=true, a write in KubeManta's own namespace fails with The KubeManta terminal cannot change anything in KubeManta's own namespace. That is by design: the namespace holds the admin and encryption credentials. If writes fail in every namespace and the install notes said the terminal is read-only on this cluster, the cluster does not serve ValidatingAdmissionPolicy (Kubernetes 1.30+), or a GitOps tool rendered the chart with helm template without the cluster's API versions; pass --api-versions admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy.
If the terminal says it is not configured on this install, the agent image is newer than the chart that deployed it: upgrade the chart to the matching version. If kubectl in the terminal has no credential after an upgrade, the reverse is true — the chart is newer than the agent image.
AI diagnosis not working / "AI is disabled"¶
Check the master AI switch¶
In Admin → AI → Guardrails, verify AI Enabled is on.
No API key provided¶
AI needs a provider configured by an admin in AI Settings. The key is stored encrypted on the server and never sent back to a browser. Ask an admin to check that the provider and key are set.
Data residency restriction¶
A provider that is not ticked under Allowed Providers in AI Guardrails is refused, whether or not Local Only is on, and the error names the provider and what is allowed. Tick your provider (a list with nothing ticked restricts nothing). With Local Only on, only Ollama and a Custom endpoint are accepted. See AI Guardrails → Data residency.
"The AI guardrail configuration could not be read"¶
AI is refused with HTTP 503 when KubeManta cannot read its AI policy, so the policy is never silently skipped. Check the agent log and Admin → System; AI resumes as soon as the configuration can be read again.
Rate limit¶
The default AI rate limit is 10 requests per minute per session (security.aiRateLimitPerMin). If you are hitting it, increase via Helm:
Network: the bandwidth chart is empty¶
Symptom: Metrics shows CPU/memory fine, but the Bandwidth panel on the Network page is empty.
Cause 1 — the kubelet scrape was turned off, or cannot reach your kubelets. The scrape is on by default, so first confirm it is still enabled:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version CHART_VERSION \
-n kubemanta-system --reset-then-reuse-values \
--set metrics.kubeletScrape.enabled=true
If it is already on, the agent most likely cannot reach port 10250 on your nodes — some clusters firewall it from pods. Route the scrape through the API server instead:
That is the only setting that grants nodes/proxy, a much broader verb than the nodes/metrics the direct scrape uses, so prefer the direct path where it works.
Cause 2 — metrics source set to prometheus. When the runtime metrics source is prometheus, KubeManta performs zero kubelet scrapes by design. Check the source, and the live scrape health (last error, nodes scraped and failed), in Admin → Observability.
Also allow a couple of minutes after enabling — rates need at least two consecutive samples (the sampler runs every 60s by default).
Metrics: Prometheus sub-tab is blank despite Prometheus being reachable¶
If the Prometheus sub-tab reports it is reachable but has no namespace/pod-labelled series, your Prometheus is scraping a standalone/Docker cadvisor (series carry only id/interface labels) instead of the cluster's kubelet/cadvisor. Point it at the cluster — kube-prometheus-stack does this out of the box, or use Admin → Observability → Generate scrape config to get the exact read-only RBAC + prometheus.yml job. See Observability.
GKE Ingress / TLS issues¶
Ingress has a wrong (ephemeral) IP instead of the static IP¶
The global.staticIpName annotation only takes effect at LB creation. If the LB already exists with an ephemeral IP, delete the ingress and let GKE recreate it:
kubectl delete ingress kubemanta -n kubemanta-system
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version CHART_VERSION \
-n kubemanta-system --reset-then-reuse-values --set global.staticIpName=YOUR_STATIC_IP_NAME
ManagedCertificate stuck in Provisioning¶
DNS must resolve to the ingress IP before GCP will issue the certificate. Check:
kubectl get ingress kubemanta -n kubemanta-system # ADDRESS should be the static IP
kubectl get managedcertificate -n kubemanta-system # STATUS: Active when cert is ready
If stuck for more than 1 hour:
kubectl delete managedcertificate kubemanta-cert -n kubemanta-system # the name from the get above; the chart renders <release>-cert
helm upgrade kubemanta ... --reset-then-reuse-values # Chart recreates it; GCP retries immediately
spec.tls conflict with GCE Ingress¶
For ingressClassName=gce with global.tlsProvider=gcp-managed, the chart omits spec.tls from the Ingress manifest. If you see spec.tls in the rendered ingress (e.g. from a manual override), remove it — GCE LB provisioning stalls silently when spec.tls references a missing secret.
ALB health check failing (EKS)¶
Symptom: Pods are Running but the ALB target group shows UNHEALTHY.
Cause: The default health check path / redirects to /login (HTTP 307 → HTTP 200), but the ALB health check follows redirects and may see an unexpected response.
Fix: Ensure the ALB target group health check uses /healthz (returns 200 with no auth required), or use success codes 200-401:
# This is set automatically by the chart; verify the annotation is present:
kubectl get ingress kubemanta -n kubemanta-system -o yaml | grep healthcheck
If missing, upgrade to the latest chart version:
helm upgrade kubemanta oci://registry.kubemanta.com/kubemanta/charts/stable/kubemanta \
--version CHART_VERSION \
-n kubemanta-system --reset-then-reuse-values
Terminal / PTY not connecting¶
Symptom: Opening the terminal tab stays on "Connecting to your terminal…", then shows "The terminal could not be opened." with a Retry button. A terminal that was already working shows "connection lost — use Retry to reconnect" instead. (Before 1.4.0 the same failure appeared as "connection failed after retries" in red inside the terminal.)
Cause: WebSocket connections to /api/agent/terminal/ws/* must be routed directly from the Ingress to the agent service — they cannot pass through Next.js (App Router does not support HTTP→WebSocket upgrade).
Fix: Verify the Ingress has a dedicated rule for the terminal path:
If the terminal path rule is missing, upgrade to the latest chart version.
The agent never becomes ready on Cilium¶
Symptom: on a cluster running Cilium, the agent pod never becomes ready: it cannot resolve names or reach the Kubernetes API server.
Cause: Cilium matches neither cluster pods nor the API server through the 0.0.0.0/0 rule in KubeManta's egress policies (security.blockImds, on by default; air-gap mode; the Expert Mode egress allowlist). Up to 1.4.0 that cut the agent off on Cilium. From 1.5.0 the policies name in-cluster pods with a selector, and the chart adds a CiliumNetworkPolicy, <release>-apiserver, that allows the API server on its ports only.
Fix: upgrade to 1.5.0 or later, then check the policy was rendered:
- It is missing and you render the chart with
helm templateor a GitOps tool: the renderer cannot see that the cluster serves the CiliumNetworkPolicy API. Setsecurity.cilium.apiServerPolicy=true. - It is there and the agent still cannot reach the API server: your control plane listens on a port other than 443 or 6443. Add it to
security.cilium.apiServerPorts.
The agent restarts after failed liveness probes¶
Symptom: the agent pod shows restarts, and kubectl describe names a failed liveness probe rather than OOMKilled.
The agent's liveness probe calls /healthz every 15 seconds with a 5-second timeout, and the kubelet restarts the container after 6 consecutive failures — about 90 seconds of not answering. /healthz does no work of its own, so a timeout means the agent was busy with something else for that long. The agent logs a warning well before that, whenever its event loop falls more than half a second behind:
kubectl -n kubemanta-system logs deploy/kubemanta-agent -c agent --previous | grep "event loop stalled"
blocked by:names the step that held the loop and how long it ran — up to three per stall, longest first. Include this line when you contact support; it is usually all that is needed. It names code, never your data.blocked by: unknown — no single callback ran longer than …means the time went to many short steps rather than one long one. Several such lines close together point at sustained load rather than one slow operation.blocked by: unknown — asyncio debug mode is offmeans the diagnostics were switched off withdiagnostics.loopDebug=false. They are on by default, at a cost of a few percent of CPU; set the value back totrueto get the attribution.
The agent stays at 0/1 after an upgrade¶
Symptom: right after helm upgrade, the agent pod shows 0/1 for a few minutes, and kubectl describe shows Startup probe failed events.
This is expected. The agent updates its database before it starts listening, and on a large scan history that can take minutes. A startup probe gives the agent up to 15 minutes to answer for the first time, and the liveness probe starts only after that. Watch the update finish:
If the pod is still 0/1 after 15 minutes, the kubelet restarts it and the update starts again. Collect the log above and contact support.
Checking the license status¶
Open Admin → License. It shows the licence state, tier and expiry, and any warning about validation or the offline grace period. The licence endpoint needs a signed-in session; no API key reaches it.
Locked out of the admin account¶
Wrong password, a lost second factor, a deactivated account, an IP ban and disabled password login all look identical at the login form. This clears every one of them:
kubectl exec -it -n kubemanta-system deploy/kubemanta-agent -- \
python3 -m app.recover_admin --username <you> --password-stdin