Core Concepts
Insights Engine
24 built-in rules with a lifecycle: durable episodes with typed evidence, flap damping, an expiry watchdog, first-class mutes, and a per-rule policy layer.
On this page
The insights engine evaluates 24 built-in rules against every cluster KubeBolt
holds a live connection to: the one you are viewing and the ones kept connected
in the background. It runs every 60 seconds (--insight-interval on the
server) over the informer cache and Metrics Server readings, so it needs no
agent and no AI. A metrics-only cluster (no live API connection) is not
evaluated. Each finding carries the affected resource, a message that names
what the rule read, and a suggestion with remediation steps or kubectl
commands.
Since 2.1.0 a finding is no longer a light that is either on or off. It has a history, an owner and a policy: episodes record what happened, mutes record what you decided to tolerate, and the rule matrix records how loud each rule is allowed to be.
The rules surface problems; they never fix anything. A remediation happens only when someone runs an action: you, Kobi on your approval, or Autopilot on KubeBolt Cloud.
Rule definitions
Each card shows the rule id, its default severity, what makes it fire, its class (see the rule matrix) and, for the six rules that have one, the tunable bar with its shipped default.
- crash-loop critical
A container is waiting in CrashLoopBackOff with more than 3 restarts. Says when the kubelet is killing it for a failing liveness probe, or when each run is OOMKilled.
- oom-killed critical
A container's last termination was OOMKilled. Clears once the new run holds Ready for 2 minutes, or longer than the gap between its last two kills (at most 24 h).
- zero-replicas critical
A Deployment that wants replicas has 0 available. A first rollout still in progress does not count.
- node-not-ready critical
A node's Ready condition is not True.
- image-pull-backoff critical
A container is waiting in ImagePullBackOff or ErrImagePull.
- progress-deadline-exceeded critical
A Deployment's Progressing condition is False with ProgressDeadlineExceeded. The suggestion says whether a rollback can help.
- missing-config-dependency critical
A container is stuck in CreateContainerConfigError: a missing ConfigMap, Secret or key, a runAsNonRoot conflict, or a kubelet read timeout.
- helm-release-failed critical
A Helm release's latest revision is in the failed state.
- cpu-throttle-risk warning
A pod's CPU use stays above 80% of its CPU limit for 2 minutes. Only pods whose containers all set a limit.
- memory-pressure warning
A pod's memory use stays above 85% of its own limit for 2 minutes (titled Memory Near Limit: this is not node memory pressure).
- hpa-maxed-out warning
An HPA is at maxReplicas and wants more (ScalingLimited: TooManyReplicas, or CPU above its target). A pinned min = max HPA does not fire.
- pvc-pending warning
A PVC is Pending with no volume bound or being provisioned. A WaitForFirstConsumer claim that no pod uses yet does not fire.
- frequent-restarts warning
A container has restarted more than 5 times and its current run has not yet held Ready for 2 minutes.
- evicted-pods warning
A pod was evicted in the last 30 minutes, for node pressure or for going over its own ephemeral-storage limit.
- readiness-probe-failing warning
A Running pod with a readiness probe (or readiness gates) has not been Ready for over 2 minutes.
- liveness-probe-failing warning
The kubelet reports the liveness probe failing more than once on a pod that still exists.
- service-no-endpoints warning
A Service with a selector (not headless, not ExternalName) has no ready endpoints.
- policy-no-match warning
A NetworkPolicy's podSelector matches no pod, and no workload in its namespace has pods with those labels.
- pdb-no-match warning
A PodDisruptionBudget's selector matches no pod, and no workload in its namespace has pods with those labels.
- helm-release-hook-pending warning
A Helm release has sat in a pending-* state for over 5 minutes, usually a hook that never completed.
- cert-expiring warning
A cert-manager Certificate expires within 14 days. One that has already expired fires as critical.
- argocd-out-of-sync warning
An Argo CD Application is OutOfSync or Degraded.
- resource-underrequest info
A pod's CPU or memory request stays below 40% of what it actually uses for 2 minutes.
- policy-orphan info
A namespace has running pods and no NetworkPolicy (nor CiliumNetworkPolicy). System namespaces are skipped.
Rules that read usage (cpu-throttle-risk, memory-pressure,
resource-underrequest) need Metrics Server and only fire after the condition
has held for 2 minutes, so a brief spike does not open an episode. Rules that
read CRDs (cert-expiring, argocd-out-of-sync) stay silent on a cluster
without cert-manager or Argo CD, and the Helm rules read the release Secrets
described in Helm Applications.
When the pod is a KubeBolt agent’s, oom-killed and memory-pressure
(Memory Near Limit) suggest the helm upgrade that raises its memory limit:
twice the current limit, at least 256Mi, pinned to the chart version the
agent runs, on the promread pod’s own values key when it is that pod. See
The Agent.
An insight clears when the rule observes recovery, not when a timer runs out:
a crash-looping or OOM-killed container clears once its new run has held Ready
long enough, a pending PVC once it binds. Clocks appear only where there is no
live state to read: evicted-pods reports evictions from the last 30 minutes,
and a liveness probe that recovered without a restart clears after a silence
sized from the probe’s own period.
Where insights show up
- Insights (pinned in the cluster sidebar) has three tabs: Active, History and Bursts, with a Critical / Warning / Info filter. Each card carries Ask Kobi (when Kobi is configured) and View episode →, plus Silence for editors and Adjust rule for admins.
- The Overview’s Insights and Cluster health cards, and the per-cluster verdicts on Home and Fleet.
- Notifications, when a genuinely new episode opens or one resolves.
- Kobi’s
get_insights,get_insight_episodesandget_insight_episodetools, also served by the read-only MCP server.
Episodes
Every firing insight opens an episode: a durable row keyed on the
fingerprint (cluster, rule and resource), with an append-only timeline of
transitions: opened, flapped, severity moved, muted, unmuted, resolved,
expired. Each transition records who caused it (system, rule:<id>,
watchdog, user:<id>) and why.
Typed evidence. A rule records what it actually saw when it fired: the
numbers and objects behind the verdict, not a re-read of the cluster minutes
later. The episode keeps it, so the recommendation can be checked against the
state that produced it even after the cluster has moved on. Kobi reads the same
evidence through get_insight_episode.
Flap damping. A fingerprint that re-fires within 10 minutes of resolving is the same episode flapping: the flap counter goes up, no new episode is created, and no second notification is sent. Outside that window it is a genuinely new episode.
The expiry watchdog. A firing episode whose cluster stopped reporting is no
longer verifiable: the agent left, the cluster was deleted. After
15 minutes without a signal (KUBEBOLT_INSIGHT_EXPIRE_TTL) the watchdog,
which sweeps once a minute, flips it to expired and it drops out of the
Active list. expired is not resolved: nobody observed recovery, and the
distinction is preserved in the timeline. If the same condition comes back
later, the new episode links to the expired one.
Resolutions are typed, so history reads as a narrative instead of a count:
auto_recovered, remediated (an executed action cleared it, with the action
id attached), manual, and rule_changed (a policy edit cleared it).
Active, History and Bursts. Active lists what is firing right now; a resolved or expired episode moves to History. History reads 24 hours, 7 days or 30 days back, filters by state (firing, resolved, expired) and severity, and switches between This cluster and All clusters. Each episode opens a detail page with its evidence, its full timeline and its recurrence: how often this exact fingerprint has come back.
Bursts groups what broke together by shared cause, so 48 evictions from one
node rotation read as one event instead of 48. It answers “what happened at 3am
last Wednesday?”: pick 24h, 7d or 30d, or a single day with On a day,
and the window stays in the URL so the view can be linked. The tab shows the
cluster you have open; This cluster / All clusters widens it to the fleet.
See the N episodes on a burst opens exactly that burst’s episodes, so the
count on the link matches the list. How bursts are formed is under
the shift report.
History answers for dead clusters. An episode keeps the cluster’s persisted display name, so a post-mortem on a cluster you already deleted still reads in names rather than UUIDs, which is precisely when the question arrives.
Retention rides the hourly maintenance pass. In the open-source edition,
resolved and expired episodes and their timelines are kept for
KUBEBOLT_FINDINGS_RETENTION_HORIZON (30 days by default), and each
fingerprint’s insight record for KUBEBOLT_INSIGHTS_RETENTION_HORIZON (7
days). Firing episodes are never pruned. On KubeBolt Cloud the
window follows your plan.
Mutes
A mute silences one rule on one resource in one cluster. It is a display layer, not a suppression: the engine keeps evaluating and episodes keep recording; only your attention is spared.
- Silence on an insight card offers For 7 days, For 30 days, Until it resolves and Permanently…. A permanent mute demands a written reason, so whoever finds it in six months knows why it exists. An “until it resolves” mute is consumed when the episode resolves; an expired one is deleted a day after its date.
- Mutes subtract everywhere insights are counted: the Overview cards, the
cluster health score, the Home and Fleet verdicts, the shift report and the
API. The Active tab counts them (N muted) and can show them in place; the
API returns them with
includeMuted=true. - Critical severity pierces a mute. An escalation gets through, which is why the engine never consults the mute list: piercing needs the underlying signal intact.
- Every mute and unmute lands in the episode timeline and the audit trail with the actor’s name and the reason.
- Reading the list needs no particular role; creating or lifting a mute needs Editor.
- Administration → Insights → Silenced lists the org’s mutes with cluster names, not UUIDs, next to the rules switched off by policy.
Re-muting the same (cluster, rule, resource) key updates the terms instead of
stacking duplicates.
The shift report
Home’s greeting tells you what happened while you were away (see Home and Fleet for how it reads). The window hangs from your own presence anchor: the last time the Home page actually rendered for you, not your last login, which token refresh would freeze. Your first shift defaults to a 24-hour window; an anchor older than 30 days is clamped, and the report says so.
It reports events, not standing state: what opened, what auto-recovered, what was remediated, what expired, how many of the newly opened episodes are still burning, and how many were critical. A condition that merely spans your absence belongs to the Active list, not here, so the report goes quiet when nothing actually happened.
Bursts are clustered deterministically, with no AI involved: the same window in produces the same grouping out. Only malfunction rules take part (an expectation is standing state, not an event). Episodes whose start times fall within 5 minutes of each other chain into one burst, and a burst needs at least 5 episodes; fewer stay as loose findings. Each burst is then classified in this order:
| Kind | Shown as | Seeded by |
|---|---|---|
node_rotation | Node rotation | node-not-ready |
node_pressure | Node pressure | evicted-pods (not an eviction for a pod’s own storage limit) |
mass_rollout | Mass rollout | Rollout rules (progress-deadline-exceeded, image-pull-backoff, readiness-probe-failing, crash-loop) across at least 5 distinct workloads |
unknown_burst | Unattributed | Nothing recognizable: an honest “we don’t know” |
The blast radius is plain arithmetic over the members: resources affected, recovered on their own, remediated, still firing, expired, and the longest episode. The longest episode of the window is named and linked, and the suppression counters (findings silenced while you were away, rules off by policy) are part of the story.
The rule matrix
Rules split into two classes, and the class decides which knob you get:
| Class | Count | What you can change |
|---|---|---|
| Malfunction | 15 | Only the numeric bar, and only on rules that have one. Severity is locked and the rule can never be switched off: broken is broken anywhere. The only full silence for one resource is a mute. |
| Expectation | 9 | Severity is the dial, off included. These are violated on purpose in a development cluster. |
Six rules carry a numeric threshold: crash-loop and frequent-restarts
(restarts), resource-underrequest (request/usage ratio), cpu-throttle-risk
and memory-pressure (usage/limit ratio), and cert-expiring (days before
expiry). Moving the bar keeps the rule meaning what its name says; the other
eighteen are boolean and have no dial.
Administration → Insights → Rules (Admin) lists all 24 grouped by class,
with a one-sentence description, their knobs, and the honesty column
Ignored 30d: episodes of that rule closed in the last 30 days without an
acknowledgement, an action, or a mute. A rule you have turned off is still
evaluated and counted: the Active tab shows a banner with how many findings
your policy is hiding, and Silenced lists the rule under Off by policy
with a Turn back on button. Turning a rule off resolves its open episodes as
rule_changed, never as a recovery. A change applies on the next evaluation.
The global override layer (one setting per rule, across every cluster) ships in the open-source distribution. Layering a second set of overrides per environment category (production, staging, testing, development) needs the cluster classification and is a KubeBolt Cloud feature, in the Environment tuning tab; there, the effective value resolves knob by knob as environment override → global override → shipped default.
Endpoints
All under /api/v1. Only GET /insights needs a connected cluster; the rest
answer from stored state, because history has to answer for clusters that are
gone.
| Method | Endpoint | Description |
|---|---|---|
| GET | /insights | Active insights for the current cluster (severity, includeMuted), with hiddenByProfile and profile |
| GET | /insights/summary | Per-cluster state for the fleet, served without a live connector |
| GET | /insights/episodes | Episode history: status, severity, rule, cluster, since, until, limit, page |
| GET | /insights/episodes/{id} | One episode with its evidence, transitions and recurrence |
| GET | /insights/operational-episodes | The deterministic bursts over a window (since, until; 24 hours by default, 31 days at most) |
| GET | /insights/operational-episodes/{id}/episodes | The episodes of one burst |
| GET | /insights/shift-report | ”While you were away”, scoped to your presence anchor |
| POST | /account/dashboard-seen | The presence beacon Home fires after rendering |
| GET · POST | /insights/mutes | List (any role) and create (Editor, audited) |
| DELETE | /insights/mutes/{id} | Lift a mute (Editor, audited) |
| GET · PUT · DELETE | /admin/insight-policies[/{rule}] | The rule policy layer (Admin, audited) |
Configuration
| Variable | Default | Purpose |
|---|---|---|
KUBEBOLT_INSIGHT_EXPIRE_TTL | 15m | A firing episode with no signal beyond this flips to expired |
KUBEBOLT_INSIGHTS_RETENTION_HORIZON | 168h (7 days) | How long each fingerprint’s resolved or expired insight record is kept |
KUBEBOLT_FINDINGS_RETENTION_HORIZON | 720h (30 days) | Also bounds resolved and expired episodes and their timelines (open-source edition) |
The evaluation interval is the server flag --insight-interval (seconds,
default 60). The full list of variables is in
Environment Variables. Insight notifications are
configured separately: see Notifications.