Observability

Two things live here: how to connect KubeManta to your existing Prometheus / Grafana / Alertmanager stack (the observability connector), and how to read KubeManta's own logs.

---

Observability connector

KubeManta can consume your existing monitoring stack instead of asking you to stand up a parallel one. Sources are configured in Admin → Observability, stored encrypted at rest, and admin-editable at runtime — the deploy-time agent.prometheusUrl (env PROMETHEUS_URL) is only a seed/back-compat default.

Source	What it does
Prometheus	Guarded PromQL query proxy — powers the Metrics → Prometheus sub-tab (long-retention windows + ad-hoc PromQL), Network → Bandwidth, and the AI's metric queries.
Grafana	Import the operator's own dashboards, embed them in the Metrics → Grafana sub-tab, and optionally embed one as the Overview. The Grafana API key is used server-side only; the browser embeds by URL + UID and authenticates to Grafana with Grafana's own session.
Alertmanager	KubeManta acts as an alert consumer — a receiver webhook ingests firing alerts into the Activity feed and alert history.

Do you need Prometheus at all?

Not for basic container metrics. KubeManta's built-in source covers live CPU/memory, ~7-day trends, and — with the opt-in metrics.kubeletScrape.enabled Helm flag — pod network bandwidth, all with zero Prometheus (see Metrics). What Prometheus uniquely adds:

• Long-term retention — months/years of history vs the built-in sampler's ~7-day window
• Your application metrics — latency histograms, custom counters: KubeManta never collects these
• kube-state-metrics / node-exporter series
• PromQL for the AI copilot — the AI can query your app metrics during an investigation

BYO Prometheus must scrape the cluster's kubelet/cadvisor

KubeManta's pod-scoped panels query series by namespace/pod labels. Those labels only exist when Prometheus scrapes the cluster's kubelet/cadvisor — kube-prometheus-stack does this out of the box, as does a kubernetes_sd_configs role: node job relabelled to the API-server proxy path (/api/v1/nodes/${1}/proxy/metrics/cadvisor).

Common gotcha: standalone/Docker cadvisor

A Prometheus that scrapes a standalone or Docker-run cadvisor produces series with id/interface labels and zero namespace/pod labels — so every pod-scoped KubeManta chart returns nothing and the panels look blank even though Prometheus is reachable and answering queries. The Metrics → Prometheus sub-tab detects and calls out this exact state.

Generate scrape config (Admin → Observability, backed by GET /observability/scrape-config) emits a copy-pasteable setup for your existing Prometheus, with the cluster's real API-server address auto-detected from in-cluster config:

• Read-only RBAC: a kubemanta-metrics-scraper ServiceAccount + ClusterRole + ClusterRoleBinding (no write verb anywhere), plus a token Secret for an out-of-cluster Prometheus
• A prometheus.yml scrape_configs block: kubernetes_sd_configs role: node, relabelled so every node's kubelet cadvisor is scraped through the API-server proxy — Prometheus never needs direct kubelet network reach

It is pure string generation — nothing is applied to the cluster.

Tiering

Configuring sources and the Alertmanager receiver webhook (ingest + view your own alerts) are Free. The historical Prometheus query proxy and auto-investigation of inbound critical alerts require a license (observability feature, Pro).

Endpoints

Endpoint	Auth	Purpose
GET/PUT /admin/observability/sources	Admin	Read / write source config (secrets masked as •••• on read, preserved on write).
POST /admin/observability/sources/test	Admin	Reachability test for a chosen source.
GET /observability/grafana/dashboards	Admin	List the operator's Grafana dashboards.
GET /observability/grafana/overview	Authenticated	Whether/what to embed as the Overview (URL + UID, no token).
GET /observability/status	Authenticated	Lightweight configured-or-not signal (backs the Metrics sub-tab shell).
GET /observability/scrape-config	Admin	Generate the BYO-Prometheus RBAC + prometheus.yml (read-only string generation, ?namespace= for the SA namespace, default monitoring).
POST /observability/prom/query · /query_range	Authenticated + observability	Guarded PromQL proxy.
POST /integrations/alertmanager/webhook	Receiver secret	Ingest Alertmanager firings (authenticated by the stored receiver secret, constant-time compared).

SSRF posture

Outbound calls to a configured source resolve the host once and pin the connection to that address (closing DNS-rebind), disable redirects, cap response size, and use a short timeout. Because Prometheus/Grafana/Alertmanager are legitimately in-cluster private services, the policy allows private addresses but hard-blocks cloud metadata (169.254.169.254, IPv6 fd00:ec2::254), link-local, loopback, and the in-cluster Kubernetes API — so the connector can never be turned into a confused deputy against the cluster API. The reachable host is structurally bound to the admin-configured source URL, never raw user input.

Alertmanager receiver setup

Point an Alertmanager receiver at KubeManta and authenticate it with the receiver secret you set in Admin → Observability:

# Alertmanager config (excerpt)
receivers:
  - name: kubemanta
    webhook_configs:
      - url: https://kubemanta.example.com/api/agent/integrations/alertmanager/webhook?token=YOUR_RECEIVER_SECRET
        send_resolved: false

Critical firings additionally trigger KubeManta's read-only auto-investigation when the observability feature is licensed and an agent API key is configured; an unlicensed install still ingests and displays the alert.

---

KubeManta's own logs

The agent and UI log to stdout — read them with kubectl logs:

kubectl logs -n kubemanta-system deploy/kubemanta-agent -f
kubectl logs -n kubemanta-system deploy/kubemanta-ui -f

Component-tagged loggers

Agent log lines are emitted under component-tagged logger names (kubemanta.<component>), so you can grep for one subsystem:

Logger	Subsystem
kubemanta.auth	Login / sessions
kubemanta.access	IP access control / bans
kubemanta.alerts	Alert rules engine
kubemanta.license	Licensing / Keygen
kubemanta.scanning	Container scanning
kubemanta.observability	Observability connector
kubemanta.terminal	Terminal sidecar
kubemanta.startup	Boot / migrations
kubemanta.error	Handled errors

# Only the licensing subsystem
kubectl logs -n kubemanta-system deploy/kubemanta-agent | grep kubemanta.license

# Handled errors across the agent
kubectl logs -n kubemanta-system deploy/kubemanta-agent | grep kubemanta.error

For an audited timeline of actions (not just logs) — HTTP requests, writes, AI decisions, terminal sessions — see Activity & Audit, which also documents the tamper-evident hash-chain and CSV/JSON export for SIEM ingestion.

---

Related

• Activity & Audit — the audited action timeline + export
• Metrics — built-in CPU/memory/bandwidth trends, the Prometheus sub-tab, and the Grafana embed
• Alerts — KubeManta's own rules engine and integrations