Alerts¶
Reactive Kubernetes event watching plus a proactive rules engine with thirteen condition types, delivered through seven outbound integrations and able to consume Alertmanager firings inbound.
Tier: Free — some features Pro
What it does¶
Alerts covers both reactive event watching and a proactive rules engine, plus outbound integrations. KubeManta watches Kubernetes Warning events across tenant namespaces — CrashLoopBackOff, OOMKilled, BackOff, and similar — classifying them by severity and deduplicating for five minutes so a flapping resource doesn't flood the feed with near-identical entries.
The rules engine supports thirteen condition types: pod restart threshold, pod phase match, pod not ready, deployment degraded, node condition, PVC unbound, event pattern, job failed, metric threshold, scan severity threshold, system event, application health, and log pattern. Rules are evaluated every 30 seconds, with a per-rule cooldown so a persistently failing condition fires once and then waits rather than re-alerting every evaluation cycle.
Alerting on an application's health¶
An application health rule watches one saved Application and fires when its
rollup verdict goes bad and stays bad. Pick the application, choose whether it
fires on degraded (which means degraded or down — nobody wants an alert that
stays silent through a total outage because they picked the milder trigger) or
only on down, and set how long the bad state must persist before it fires.
It deliberately fires once for the whole application, not once per member: a
three-pod app with one cause should page you once. An unknown verdict never
fires — that means the selector matches nothing, which is a configuration
mistake rather than an outage, and paging for it trains people to ignore the
alert. A cluster read failure does not fire either: the API server blipping is
not your application going down.
Alerting on a log pattern¶
A log pattern rule matches text in your pods' logs and fires on N matches within a window — never per line. A crashlooping pod reprints its error on every restart, so per-line firing is a pager that never stops.
Configure it with the text to match, a pod selector, optionally a container
name, and the threshold and window. Matching is substring by default and
regular expressions are opt-in. That is a deliberate safety choice: the match
runs inside the evaluation loop, and a catastrophically backtracking regex would
be a denial of service against alerting itself — the subsystem whose failure is
least visible, because a dead alerter looks exactly like a quiet cluster.
Substring matching covers the common case (OutOfMemoryError, panic:) and is
linear by construction.
If you do use a regex, the risk is bounded rather than eliminated, and it is worth knowing how: every line is truncated before matching, and the scan stops once it has spent its time budget for that tick — so one bad rule costs a slice of one evaluation instead of your alerting.
Log scanning has its own interval (60s by default) rather than running on every
30-second tick, and is bounded per tick in pods scanned and bytes read, so
turning it on does not make the agent read your cluster continuously. Those
bounds are environment-tunable (ALERT_LOG_SCAN_INTERVAL_SECS,
ALERT_LOG_MAX_PODS, ALERT_LOG_MAX_BYTES_PER_POD,
ALERT_LOG_MAX_BYTES_PER_TICK) if your cluster needs different limits.
Integrations cover Slack, Telegram, Email/SMTP, PagerDuty, CloudWatch, Datadog, and a generic Webhook. Each enabled destination is attempted independently, with its own 10-second timeout, so one destination's outage never silently suppresses the others. They are attempted in sequence, so several slow destinations add up.
Telegram alerts are sent as plain text, on purpose. Telegram's Markdown
modes require escaping a long list of characters, and alert text is not ours to
control — it carries pod names, container names and raw Kubernetes event
messages. A single unescaped _ or *, routine in a name like
api_gateway-7d9f, makes Telegram reject the whole message, so the alert about
the strangest-looking workload would be the one that never arrived. Setting up
the destination takes two steps that are easy to do in the wrong order: a bot
cannot open a conversation, so message it (or add it to your group) before
asking it to message you, or the first Test returns "chat not found". Alertmanager
ingest lets KubeManta act as an alert consumer as well as a source — point an
Alertmanager receiver at its webhook and inbound firings land in the same
feed as everything KubeManta detects itself.
The receiver's reply says what it actually did, so a misconfigured sender is
distinguishable from a working one. It reports how many alerts were ingested,
how many failed to store, and how many were resolved_ignored — Alertmanager
sends resolved notifications by default and KubeManta has no auto-resolve, so
those are counted and dropped rather than passed over in silence. A shared
secret is required: with none configured the webhook refuses everything, since
it writes to your alert history. A body that is not a JSON object is rejected
with a 400 rather than a server error, because Alertmanager retries a 5xx
forever and a malformed sender would otherwise loop against it.
Every delivery attempt is recorded. For each alert, each configured channel gets its own receipt — sent, failed, or skipped — with the destination, the HTTP status or error, and how long the attempt took. This exists because "the alert fired" and "the alert arrived" are different claims, and only the second one matters at 03:00: an integration that has been quietly rejecting every message since a token expired looks identical, from the alert feed alone, to one that is working. Receipts also make delivery auditable after the fact — you can answer "was anyone told, and when" from the record rather than from memory. URLs and tokens in destinations are redacted before storage, so a receipt can be read by anyone who can read alerts without handing them the webhook.
Starting from a template¶
Four templates sit above the rule list: crash-looping pods, OOMKilled, image cannot be pulled, and stuck Pending. All four were always expressible by hand; the gap was having to know that "my images are broken" lives under Pod → Phase Match → an ImagePullBackOff chip, which is knowledge about KubeManta's data model rather than about your cluster.
A template fills the form in and stops there. It never saves a rule, because the namespace and the channels are the parts only you can know.
Stuck Pending carries a delay, and the others do not. CrashLoopBackOff and ImagePullBackOff are backoff states — the kubelet has already retried and given up, so seeing one is the alert — and OOMKilled is terminal. Pending is different: every healthy pod is Pending for a moment while it schedules and pulls an image, so a rule on Pending with no floor fires on every normal rollout. The template therefore asks for ten minutes, and the field ("Only if it has been that way for") appears whenever you select Pending on any rule. Leave it empty and the rule fires immediately, which is the old behaviour and is what rules created before this field existed still do.
The namespace is a list, not a text box¶
A rule's namespace is chosen from the namespaces the cluster actually has, rather than typed. A namespace that does not exist would save a rule matching nothing: no error, a healthy-looking row, and an alert that never arrives.
A saved rule keeps its own namespace as an option even if that namespace is gone or hidden, so opening an old rule to change its cooldown cannot silently re-scope it to somewhere else.
Setup¶
Reach Alerts Manager from the SIGNALS group in the sidebar. Three sub-tabs cover the surface: Alerts, the live feed; Rules, CRUD plus firing history; and Integrations, destination configuration. To create a rule, go to the Rules tab, choose New Rule — or one of the four templates above the list, which opens the same form already filled in — then pick a condition type, set thresholds and a cooldown, and use Test to dry-run it before enabling. Test evaluates the rule against the live cluster and reports what it would fire on; it never delivers anything, so it is safe to run against a configured Slack or PagerDuty. To configure a destination, go to Integrations, pick a provider card, enter credentials, Save, then Test. Alert history clears from the Alerts tab; firing history clears from Rules → Firings.
Asking KubeManta things from Slack¶
Slack outbound — alerts arriving in a channel — is the destination described above and works on every tier. Slack inbound, asking KubeManta questions from a channel, is separate and off by default, because it means cluster state leaves for a third party.
Turn it on with a Slack app in Socket Mode:
--set slack.inbound.enabled=true \
--set slack.inbound.appToken=xapp-… # Socket Mode app token (connections:write)
--set slack.inbound.botToken=xoxb-… # bot token, used to post replies
Both tokens are stored in KubeManta's Kubernetes Secret, never as plain environment on the Deployment.
Socket Mode means one OUTBOUND WebSocket. There is no ingress to open, no certificate to obtain and no unauthenticated route added, so it works on a cluster with no public entry point. It is useless in a true air-gap — slack.com is unreachable by definition — and KubeManta says so instead of retrying: with air-gap mode on it does not dial at all.
Who can do what is not configured here. A Slack user is linked to a KubeManta account by the email their workspace has verified, and their role is read live on every command — so a viewer's Slack account can only read, and an admin who is demoted loses admin on their next message. An unlinked Slack user is nobody and is told to ask an admin.
Commands are read-only by construction. KubeManta never changes your cluster
in response to a chat message; actions arrive as an approval you confirm. Ask it
for Cluster Health, /pods <namespace>, /nodes, /warnings or /help.
What Free includes, and where Pro starts¶
Alerting is part of the Free tier, and the number of rules is unlimited. Write as many as your cluster needs, delivered to Slack, Telegram or email.
Pro adds the operational integrations (PagerDuty, Datadog, CloudWatch and the generic Webhook), retained firing history, and AI auto-investigation of a firing.
If a licence lapses, every rule you have keeps evaluating and keeps delivering — being downgraded must not quietly stop the alert you depend on. Configuration for a Pro destination can also be saved while it is switched off; only turning it on is refused, so an unrelated edit never fails for a reason the form cannot explain.
Troubleshooting¶
Because detection and delivery both run inside the same KubeManta process, a
dead agent means no alerts fire and nothing announces that fact from inside
the product — the honest failure mode of a single-process alerting system.
Enabling the alert heartbeat (off by default) emits a low-severity alert
every hour (ALERT_HEARTBEAT_INTERVAL_SECS, default 3600s) through your
configured integrations; because you alert on its absence rather than its
presence, it's the one signal that can survive the
process it's reporting on.
The heartbeat reports what it can actually check. It carries the age of the last completed rule evaluation, and if that goes stale — or has never happened — it escalates to a critical "alerting stalled" message instead of arriving as usual. That case is not hypothetical: rule evaluation can stop while the process stays up and healthy, and a heartbeat that kept saying "running and evaluating rules" through it would be worse than no heartbeat, because you would trust it. If a rule appears configured correctly but never fires, use the Test action on the rule to dry-run the condition directly against current cluster state rather than waiting for the 30-second evaluator, which narrows down whether the condition itself is wrong or the delivery integration is. If one integration is silent while others deliver correctly, check that integration's own Test button — each destination is attempted independently, so a single misconfigured one degrades on its own without affecting the rest.
When an alert should have reached you and did not, read its delivery receipts
before changing any configuration. They separate the three cases that look
identical from the outside: the alert never fired (no receipts at all), it
fired and the channel refused it (a failed receipt carrying the status and
the provider's own message), or it fired and the channel was never tried (a
skipped receipt, which names the reason — either the integration is disabled,
or the rule's own channel selection did not include it). Only the second case is
a credentials problem, and only the first is a rules problem.
Generated from KubeManta 1.2.0 — this page is rendered from the product's own documentation, so it cannot drift from what ships. Manifest built 2026-09-12T02:14:45Z.