Observability¶

Three things live here: how to connect KubeManta to your existing Prometheus / Grafana / Alertmanager stack (the observability connector), how to scrape KubeManta's own metrics into your Prometheus, and how to read KubeManta's own logs.

Observability connector¶

KubeManta can consume your existing monitoring stack instead of asking you to stand up a parallel one. Sources are configured in Admin → Observability, stored encrypted at rest, and admin-editable at runtime — the deploy-time agent.prometheusUrl (env PROMETHEUS_URL) is only a seed/back-compat default.

Source	What it does
Prometheus	Guarded PromQL query proxy — powers the Metrics page when connected (long-retention windows + ad-hoc PromQL), Network → Bandwidth, and the AI's metric queries.
Grafana	Import the operator's own dashboards, embed them in the Grafana tab beside Metrics, and optionally embed one as the Overview. The Grafana API key is used server-side only; the browser embeds by URL + UID and authenticates to Grafana with Grafana's own session. KubeManta's Content-Security-Policy frames only the configured Grafana origin, read at page load: reload after changing the URL, and expect a frame that Grafana redirects to an SSO provider on another origin to be refused (details).
Alertmanager	KubeManta acts as an alert consumer — a receiver webhook ingests firing alerts into the alert history on the Alerts page and the Activity feed.

Do you need Prometheus at all?¶

Not for basic container metrics. KubeManta's built-in source covers live CPU/memory, ~7-day trends, and — through the kubelet scrape, on by default (metrics.kubeletScrape.enabled) — pod network bandwidth, all with zero Prometheus (see Metrics). What Prometheus uniquely adds:

Long-term retention — months/years of history vs the built-in sampler's ~7-day window
Your application metrics — latency histograms, custom counters: KubeManta never collects these
kube-state-metrics / node-exporter series
PromQL for the AI copilot — the AI can query your app metrics during an investigation

BYO Prometheus must scrape the cluster's kubelet/cadvisor¶

KubeManta's pod-scoped panels query series by namespace/pod labels. Those labels only exist when Prometheus scrapes the cluster's kubelet/cadvisor — kube-prometheus-stack does this out of the box, as does a kubernetes_sd_configs role: node job relabelled to the API-server proxy path (/api/v1/nodes/${1}/proxy/metrics/cadvisor).

Common gotcha: standalone/Docker cadvisor

A Prometheus that scrapes a standalone or Docker-run cadvisor produces series with id/interface labels and zero namespace/pod labels — so every pod-scoped KubeManta chart returns nothing and the panels look blank even though Prometheus is reachable and answering queries. The Metrics → Prometheus sub-tab detects and calls out this exact state.

Generate scrape config (Admin → Observability, backed by GET /observability/scrape-config) emits a copy-pasteable setup for your existing Prometheus, with the cluster's real API-server address auto-detected from in-cluster config:

Read-only RBAC: a kubemanta-metrics-scraper ServiceAccount + ClusterRole + ClusterRoleBinding (no write verb anywhere), plus a token Secret for an out-of-cluster Prometheus
A prometheus.yml scrape_configs block: kubernetes_sd_configs role: node, relabelled so every node's kubelet cadvisor is scraped through the API-server proxy — Prometheus never needs direct kubelet network reach

It is pure string generation — nothing is applied to the cluster.

Tiering¶

Configuring sources and the Alertmanager receiver webhook (ingest + view your own alerts) are Free. The historical Prometheus query proxy and auto-investigation of inbound critical alerts require a license (observability feature, Pro).

Endpoints¶

Endpoint	Auth	Purpose
`GET/PUT /admin/observability/sources`	Admin	Read / write source config (secrets masked as `••••` on read, preserved on write).
`POST /admin/observability/sources/test`	Admin	Reachability test for a chosen source.
`GET /observability/grafana/dashboards`	Admin	List the operator's Grafana dashboards.
`GET /observability/grafana/overview`	Authenticated	Whether/what to embed as the Overview (URL + UID, no token).
`GET /observability/status`	Authenticated	Lightweight configured-or-not signal (backs the Metrics sub-tab shell).
`GET /observability/scrape-config`	Admin	Generate the BYO-Prometheus RBAC + `prometheus.yml` (read-only string generation, `?namespace=` for the SA namespace, default `monitoring`).
`POST /observability/prom/query` · `/query_range`	Authenticated + `observability`	Guarded PromQL proxy.
`POST /integrations/alertmanager/webhook`	Receiver secret	Ingest Alertmanager firings (authenticated by the stored receiver secret, constant-time compared).

SSRF posture¶

Outbound calls to a configured source resolve the host once and pin the connection to that address (closing DNS-rebind), disable redirects, cap response size, and use a short timeout. Because Prometheus/Grafana/Alertmanager are legitimately in-cluster private services, the policy allows private addresses but hard-blocks cloud metadata (169.254.169.254, IPv6 fd00:ec2::254), link-local, loopback, and the in-cluster Kubernetes API — so the connector can never be turned into a confused deputy against the cluster API. The reachable host is structurally bound to the admin-configured source URL, never raw user input.

Alertmanager receiver setup¶

Point an Alertmanager receiver at KubeManta and authenticate it with the receiver secret you set in Admin → Observability:

# Alertmanager config (excerpt)
receivers:
  - name: kubemanta
    webhook_configs:
      - url: https://kubemanta.example.com/api/agent/integrations/alertmanager/webhook?token=YOUR_RECEIVER_SECRET
        send_resolved: false

Critical firings additionally trigger KubeManta's read-only auto-investigation when the observability feature is licensed and an agent API key is configured; an unlicensed install still ingests and displays the alert.

Scraping KubeManta's own metrics¶

The agent serves Prometheus text-format metrics about itself at GET /metrics: kubemanta_build_info, kubemanta_uptime_seconds, kubemanta_http_requests_total (by status class), kubemanta_sqlite_up, and AI token and request totals by provider (kubemanta_ai_tokens_total, kubemanta_ai_requests_total).

The endpoint is authenticated. Prometheus authenticates with an API key that holds only the scrape:read scope, sent as a Bearer token. That scope opens GET /metrics and nothing else, so the key sitting in your Prometheus configuration cannot read cluster data, query your Prometheus through KubeManta, or change anything. A key without scrape:read — including one with the broader metrics:read — is refused at /metrics.

Upgrading from 1.3.1 with a metrics:read scrape key

Up to 1.3.1, /metrics accepted a metrics:read key in the X-API-Key header. It no longer does: create a scrape:read key as below and use it instead.

1. Create the scrape key¶

Use a dedicated viewer account, so the key outlives any one person:

In Admin → Users & Groups → Users, create a local account such as prometheus-scraper with the role viewer.
Sign in as that account (you will be asked to set its password), open the account menu → API keys, and create a key with only scrape:read ticked. Copy the secret — it is shown once.

Store it as a Secret in the namespace your Prometheus reads it from:

read -rs KM_SCRAPE_KEY        # paste the key; keeps it out of shell history
kubectl -n kubemanta-system create secret generic kubemanta-scrape \
  --from-literal=token="$KM_SCRAPE_KEY"
unset KM_SCRAPE_KEY

Check it before wiring Prometheus:

kubectl -n kubemanta-system port-forward svc/kubemanta-agent 8080:8080 &
curl -s -H "Authorization: Bearer <key>" http://localhost:8080/metrics | head

The Service is named <release>-kubemanta-agent (kubemanta-agent for a release named kubemanta); its port is named http (8080 unless you changed service.port).

2a. Prometheus Operator — ServiceMonitor¶

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubemanta-agent
  namespace: kubemanta-system
  labels:
    release: kube-prometheus-stack   # whatever your Prometheus' serviceMonitorSelector matches
spec:
  namespaceSelector:
    matchNames: [kubemanta-system]
  selector:
    matchLabels:
      app.kubernetes.io/instance: kubemanta   # your Helm release name
      app.kubernetes.io/component: agent
  endpoints:
    - port: http
      path: /metrics
      interval: 60s
      authorization:
        type: Bearer
        credentials:
          name: kubemanta-scrape   # the Secret from step 1, same namespace
          key: token

2b. Plain Prometheus — scrape config¶

Mount the Secret into the Prometheus pod, then:

scrape_configs:
  - job_name: kubemanta-agent
    metrics_path: /metrics
    scrape_interval: 60s
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/kubemanta-scrape/token
    static_configs:
      - targets: ["kubemanta-agent.kubemanta-system.svc:8080"]

Notes:

Each scrape is recorded in the access log like any other API call; an interval of 30–60 seconds is plenty for these series.
If a NetworkPolicy of your own restricts ingress to the agent, allow your Prometheus pods to reach its port.
Revoking the key in API keys (or deactivating the account) stops the scrape on the next request.
401 at /metrics: the key is missing, mistyped, revoked, or its account is deactivated; a ServiceMonitor must reference a Secret in its own namespace. 403 api_key_insufficient_scope: the key lacks scrape:read (a metrics:read key is not a scrape key).

Readiness and degraded controls¶

The agent's /readyz reports whether it is ready and which of its own safety controls are degraded. It needs no credential and answers 200 unless the agent is shedding load (503), so read its body rather than its status code:

kubectl exec -n kubemanta-system deploy/kubemanta-agent -- \
  python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8080/readyz').read().decode())"

Field	When it appears	Meaning
`status`	always	`ready`, `degraded` or `saturated`
`degraded_controls`	while a control is degraded	Each control that is running in a degraded state, for example the AI guardrail configuration when it cannot be read (AI is then refused).
`audit_writer`	when any audit row was lost or shed since the agent started	The audit writer's counters: `rows_lost` (audit-trail rows that never reached the tamper-evident trail), `access_rows_shed` (access-log rows dropped while the write queue was full), `write_retries`, `queue_depth` and `queue_max`.

The same audit_writer counters are always included in /kubemanta/status. The writer retries a batch that meets a locked or busy database before it writes anything after it, so a row counted in rows_lost is a real gap in the audit trail, not a transient error: alert on it.

KubeManta's own logs¶

The agent and UI log to stdout — read them with kubectl logs:

kubectl logs -n kubemanta-system deploy/kubemanta-agent -f
kubectl logs -n kubemanta-system deploy/kubemanta-ui -f

Component-tagged loggers¶

Agent log lines are emitted under component-tagged logger names (kubemanta.<component>), so you can grep for one subsystem:

Logger	Subsystem
`kubemanta.auth`	Login / sessions
`kubemanta.access`	IP access control / bans
`kubemanta.alerts`	Alert rules engine
`kubemanta.license`	Licensing / Keygen
`kubemanta.scanning`	Container scanning
`kubemanta.observability`	Observability connector
`kubemanta.terminal`	Terminal sidecar
`kubemanta.startup`	Boot / migrations
`kubemanta.error`	Handled errors

# Only the licensing subsystem
kubectl logs -n kubemanta-system deploy/kubemanta-agent | grep kubemanta.license

# Handled errors across the agent
kubectl logs -n kubemanta-system deploy/kubemanta-agent | grep kubemanta.error

For an audited timeline of actions (not just logs) — HTTP requests, writes, AI decisions, terminal sessions — see Activity & Audit, which also documents the tamper-evident hash-chain and CSV/JSON export for SIEM ingestion.

Activity & Audit — the audited action timeline + export
API reference → Scopes — what scrape:read and the other key scopes open
Metrics — built-in CPU/memory trends, Prometheus history on the same page, and the Grafana embed
Alerts — KubeManta's own rules engine and integrations