Skip to content

Prometheus

Connect your existing Prometheus so KubeManta can surface long-retention metrics, let operators write ad-hoc PromQL queries, and give the AI copilot access to your application metrics during investigations.

What it gives you

  • Long-retention metrics — trend charts covering 1h through 30d / 90d windows, far past KubeManta's built-in 7-day sampler
  • Ad-hoc PromQL — run any range query directly from the Metrics → Prometheus sub-tab (first 8 series are charted)
  • Application metrics for the AI — when the AI copilot investigates an issue, it can query your latency histograms, custom counters, and business KPIs
  • CPU, memory, and network — all sourced from your Prometheus if you prefer not to use KubeManta's built-in sampler

Tier: Free

Prerequisites

  1. Prometheus running — already deployed and scraping your cluster. We'll point KubeManta at it.
  2. Network path — KubeManta must reach your Prometheus URL (typically http://prometheus.monitoring:9090 for in-cluster, or a public URL for managed Prometheus).
  3. Correct scrape config — Prometheus must scrape the cluster's kubelet cadvisor for pod-scoped metrics to work. If Prometheus only scrapes standalone cadvisor or a Docker-run cadvisor, pod-scoped charts will be blank (we'll detect this and tell you). See Observability for details.

Generate the scrape config

If your Prometheus isn't yet scraping the cluster's kubelet/cadvisor, KubeManta can generate the exact RBAC + prometheus.yml job snippet for you (Admin → Observability → Generate scrape config).

Configure in KubeManta

  1. In the sidebar, click Admin (gear icon).
  2. Open the Observability tab.
  3. Under Configure sources, select Prometheus from the dropdown.
  4. Paste your Prometheus URL (e.g., http://prometheus.monitoring:9090).
  5. Click Test to verify KubeManta can reach it.
  6. Click Save.

The URL is stored encrypted at rest. You can edit it anytime without restarting the agent.

Verify

  1. Click Metrics in the sidebar (OBSERVE group).
  2. Open the Prometheus sub-tab.
  3. You should see three empty-state cards: CPU, Memory, and Network. These populate with data as you select a window (1h / 6h / 24h / 30d / 90d).
  4. Try the Ad-hoc PromQL box — paste a simple query like up and click Query. If your Prometheus is reachable and has series, you'll see results.

Expected states

State Meaning Next step
Setup not configured You haven't entered a Prometheus URL yet Follow "Configure in KubeManta" above
Configured, can't reach The URL is saved, but KubeManta can't connect (firewall, DNS, service down) Verify the URL is correct and KubeManta has network reach
Configured, reachable, but every chart is blank Prometheus is reachable but has no pod-scoped series (common gotcha: it's scraping standalone cadvisor) See "Prometheus reachable, but every panel is blank" tip below
Working — charts populate Success! Proceed to test the AI copilot's metric queries (if you have Pro + AI enabled)

Prometheus reachable, but every chart is blank

This almost always means your Prometheus is scraping a standalone or Docker-run cadvisor instead of the cluster's kubelet. Those exports have id/interface labels but no namespace/pod labels — so KubeManta's pod-scoped queries return nothing.

Fix: Reconfigure Prometheus to scrape the cluster's kubelet/cadvisor (kube-prometheus-stack does this out of the box). Or generate the exact scrape config via Admin → Observability → Generate scrape config for your existing Prometheus.

Troubleshooting

"Configured, can't reach" — but Prometheus is definitely up

Check the URL format: - ✅ Correct: http://prometheus.monitoring:9090 (in-cluster service DNS) - ✅ Correct: https://prometheus.example.com (managed Prometheus, public) - ❌ Wrong: prometheus.monitoring:9090 (missing protocol) - ❌ Wrong: http://prometheus.monitoring:9090/ (trailing slash sometimes causes issues)

Check the network path: - If KubeManta is in kubemanta-system and Prometheus is in monitoring, the agent pod must have egress to that namespace (no restrictive NetworkPolicy). - For managed Prometheus (e.g., in a different VPC or cloud provider), ensure the firewall allows outbound HTTPS to that host.

Check the logs: 

kubectl logs -n kubemanta-system deploy/kubemanta-agent | grep observability

"Ad-hoc PromQL" returns "no data"

Did you select a time window? The Prometheus sub-tab has buttons for 1h / 6h / 24h / 30d / 90d. Make sure you picked one before running a query.

Does the series exist? Try a simple test query like node_uname_info (exported by node-exporter if installed). If that returns nothing, your Prometheus may not have the metrics you expect.

After a Prometheus failover, metrics are stale

KubeManta caches Prometheus responses briefly. If you swapped the backend Prometheus instance (e.g., HA failover), update the URL (even if it's the same hostname) and click Save to bust the cache.