KubeBolt docs
ENES GitHub

Core Concepts

Insights Engine

24 built-in rules with a lifecycle: durable episodes with typed evidence, flap damping, an expiry watchdog, first-class mutes, and a per-rule policy layer.

On this page
  1. Rule definitions
  2. Where insights show up
  3. Episodes
  4. Mutes
  5. The shift report
  6. The rule matrix
  7. Endpoints
  8. Configuration

The insights engine evaluates 24 built-in rules against every cluster KubeBolt holds a live connection to: the one you are viewing and the ones kept connected in the background. It runs every 60 seconds (--insight-interval on the server) over the informer cache and Metrics Server readings, so it needs no agent and no AI. A metrics-only cluster (no live API connection) is not evaluated. Each finding carries the affected resource, a message that names what the rule read, and a suggestion with remediation steps or kubectl commands.

Since 2.1.0 a finding is no longer a light that is either on or off. It has a history, an owner and a policy: episodes record what happened, mutes record what you decided to tolerate, and the rule matrix records how loud each rule is allowed to be.

The rules surface problems; they never fix anything. A remediation happens only when someone runs an action: you, Kobi on your approval, or Autopilot on KubeBolt Cloud.

Rule definitions

Each card shows the rule id, its default severity, what makes it fire, its class (see the rule matrix) and, for the six rules that have one, the tunable bar with its shipped default.

8 Critical 14 Warning 2 Info
  • crash-loop critical

    A container is waiting in CrashLoopBackOff with more than 3 restarts. Says when the kubelet is killing it for a failing liveness probe, or when each run is OOMKilled.

    malfunction · restarts > 3

  • oom-killed critical

    A container's last termination was OOMKilled. Clears once the new run holds Ready for 2 minutes, or longer than the gap between its last two kills (at most 24 h).

    malfunction

  • zero-replicas critical

    A Deployment that wants replicas has 0 available. A first rollout still in progress does not count.

    malfunction

  • node-not-ready critical

    A node's Ready condition is not True.

    malfunction

  • image-pull-backoff critical

    A container is waiting in ImagePullBackOff or ErrImagePull.

    malfunction

  • progress-deadline-exceeded critical

    A Deployment's Progressing condition is False with ProgressDeadlineExceeded. The suggestion says whether a rollback can help.

    malfunction

  • missing-config-dependency critical

    A container is stuck in CreateContainerConfigError: a missing ConfigMap, Secret or key, a runAsNonRoot conflict, or a kubelet read timeout.

    malfunction

  • helm-release-failed critical

    A Helm release's latest revision is in the failed state.

    malfunction

  • cpu-throttle-risk warning

    A pod's CPU use stays above 80% of its CPU limit for 2 minutes. Only pods whose containers all set a limit.

    expectation · usage/limit > 0.80

  • memory-pressure warning

    A pod's memory use stays above 85% of its own limit for 2 minutes (titled Memory Near Limit: this is not node memory pressure).

    expectation · usage/limit > 0.85

  • hpa-maxed-out warning

    An HPA is at maxReplicas and wants more (ScalingLimited: TooManyReplicas, or CPU above its target). A pinned min = max HPA does not fire.

    expectation

  • pvc-pending warning

    A PVC is Pending with no volume bound or being provisioned. A WaitForFirstConsumer claim that no pod uses yet does not fire.

    malfunction

  • frequent-restarts warning

    A container has restarted more than 5 times and its current run has not yet held Ready for 2 minutes.

    malfunction · restarts > 5

  • evicted-pods warning

    A pod was evicted in the last 30 minutes, for node pressure or for going over its own ephemeral-storage limit.

    malfunction

  • readiness-probe-failing warning

    A Running pod with a readiness probe (or readiness gates) has not been Ready for over 2 minutes.

    malfunction

  • liveness-probe-failing warning

    The kubelet reports the liveness probe failing more than once on a pod that still exists.

    malfunction

  • service-no-endpoints warning

    A Service with a selector (not headless, not ExternalName) has no ready endpoints.

    expectation

  • policy-no-match warning

    A NetworkPolicy's podSelector matches no pod, and no workload in its namespace has pods with those labels.

    expectation

  • pdb-no-match warning

    A PodDisruptionBudget's selector matches no pod, and no workload in its namespace has pods with those labels.

    expectation

  • helm-release-hook-pending warning

    A Helm release has sat in a pending-* state for over 5 minutes, usually a hook that never completed.

    malfunction

  • cert-expiring warning

    A cert-manager Certificate expires within 14 days. One that has already expired fires as critical.

    expectation · days < 14

  • argocd-out-of-sync warning

    An Argo CD Application is OutOfSync or Degraded.

    malfunction

  • resource-underrequest info

    A pod's CPU or memory request stays below 40% of what it actually uses for 2 minutes.

    expectation · request/usage < 0.40

  • policy-orphan info

    A namespace has running pods and no NetworkPolicy (nor CiliumNetworkPolicy). System namespaces are skipped.

    expectation

Rules that read usage (cpu-throttle-risk, memory-pressure, resource-underrequest) need Metrics Server and only fire after the condition has held for 2 minutes, so a brief spike does not open an episode. Rules that read CRDs (cert-expiring, argocd-out-of-sync) stay silent on a cluster without cert-manager or Argo CD, and the Helm rules read the release Secrets described in Helm Applications.

When the pod is a KubeBolt agent’s, oom-killed and memory-pressure (Memory Near Limit) suggest the helm upgrade that raises its memory limit: twice the current limit, at least 256Mi, pinned to the chart version the agent runs, on the promread pod’s own values key when it is that pod. See The Agent.

An insight clears when the rule observes recovery, not when a timer runs out: a crash-looping or OOM-killed container clears once its new run has held Ready long enough, a pending PVC once it binds. Clocks appear only where there is no live state to read: evicted-pods reports evictions from the last 30 minutes, and a liveness probe that recovered without a restart clears after a silence sized from the probe’s own period.

Where insights show up

Episodes

Every firing insight opens an episode: a durable row keyed on the fingerprint (cluster, rule and resource), with an append-only timeline of transitions: opened, flapped, severity moved, muted, unmuted, resolved, expired. Each transition records who caused it (system, rule:<id>, watchdog, user:<id>) and why.

An insight episode: the timeline, the recurrences with the same fingerprint, and the recommendation.
An insight episode: the timeline, the recurrences with the same fingerprint, and the recommendation. KubeBolt 2.1.1

Typed evidence. A rule records what it actually saw when it fired: the numbers and objects behind the verdict, not a re-read of the cluster minutes later. The episode keeps it, so the recommendation can be checked against the state that produced it even after the cluster has moved on. Kobi reads the same evidence through get_insight_episode.

Flap damping. A fingerprint that re-fires within 10 minutes of resolving is the same episode flapping: the flap counter goes up, no new episode is created, and no second notification is sent. Outside that window it is a genuinely new episode.

The expiry watchdog. A firing episode whose cluster stopped reporting is no longer verifiable: the agent left, the cluster was deleted. After 15 minutes without a signal (KUBEBOLT_INSIGHT_EXPIRE_TTL) the watchdog, which sweeps once a minute, flips it to expired and it drops out of the Active list. expired is not resolved: nobody observed recovery, and the distinction is preserved in the timeline. If the same condition comes back later, the new episode links to the expired one.

Resolutions are typed, so history reads as a narrative instead of a count: auto_recovered, remediated (an executed action cleared it, with the action id attached), manual, and rule_changed (a policy edit cleared it).

Active, History and Bursts. Active lists what is firing right now; a resolved or expired episode moves to History. History reads 24 hours, 7 days or 30 days back, filters by state (firing, resolved, expired) and severity, and switches between This cluster and All clusters. Each episode opens a detail page with its evidence, its full timeline and its recurrence: how often this exact fingerprint has come back.

Bursts groups what broke together by shared cause, so 48 evictions from one node rotation read as one event instead of 48. It answers “what happened at 3am last Wednesday?”: pick 24h, 7d or 30d, or a single day with On a day, and the window stays in the URL so the view can be linked. The tab shows the cluster you have open; This cluster / All clusters widens it to the fleet. See the N episodes on a burst opens exactly that burst’s episodes, so the count on the link matches the list. How bursts are formed is under the shift report.

History answers for dead clusters. An episode keeps the cluster’s persisted display name, so a post-mortem on a cluster you already deleted still reads in names rather than UUIDs, which is precisely when the question arrives.

Retention rides the hourly maintenance pass. In the open-source edition, resolved and expired episodes and their timelines are kept for KUBEBOLT_FINDINGS_RETENTION_HORIZON (30 days by default), and each fingerprint’s insight record for KUBEBOLT_INSIGHTS_RETENTION_HORIZON (7 days). Firing episodes are never pruned. On KubeBolt Cloud the window follows your plan.

Mutes

A mute silences one rule on one resource in one cluster. It is a display layer, not a suppression: the engine keeps evaluating and episodes keep recording; only your attention is spared.

Re-muting the same (cluster, rule, resource) key updates the terms instead of stacking duplicates.

The shift report

Home’s greeting tells you what happened while you were away (see Home and Fleet for how it reads). The window hangs from your own presence anchor: the last time the Home page actually rendered for you, not your last login, which token refresh would freeze. Your first shift defaults to a 24-hour window; an anchor older than 30 days is clamped, and the report says so.

It reports events, not standing state: what opened, what auto-recovered, what was remediated, what expired, how many of the newly opened episodes are still burning, and how many were critical. A condition that merely spans your absence belongs to the Active list, not here, so the report goes quiet when nothing actually happened.

Bursts are clustered deterministically, with no AI involved: the same window in produces the same grouping out. Only malfunction rules take part (an expectation is standing state, not an event). Episodes whose start times fall within 5 minutes of each other chain into one burst, and a burst needs at least 5 episodes; fewer stay as loose findings. Each burst is then classified in this order:

KindShown asSeeded by
node_rotationNode rotationnode-not-ready
node_pressureNode pressureevicted-pods (not an eviction for a pod’s own storage limit)
mass_rolloutMass rolloutRollout rules (progress-deadline-exceeded, image-pull-backoff, readiness-probe-failing, crash-loop) across at least 5 distinct workloads
unknown_burstUnattributedNothing recognizable: an honest “we don’t know”

The blast radius is plain arithmetic over the members: resources affected, recovered on their own, remediated, still firing, expired, and the longest episode. The longest episode of the window is named and linked, and the suppression counters (findings silenced while you were away, rules off by policy) are part of the story.

The rule matrix

Rules split into two classes, and the class decides which knob you get:

ClassCountWhat you can change
Malfunction15Only the numeric bar, and only on rules that have one. Severity is locked and the rule can never be switched off: broken is broken anywhere. The only full silence for one resource is a mute.
Expectation9Severity is the dial, off included. These are violated on purpose in a development cluster.

Six rules carry a numeric threshold: crash-loop and frequent-restarts (restarts), resource-underrequest (request/usage ratio), cpu-throttle-risk and memory-pressure (usage/limit ratio), and cert-expiring (days before expiry). Moving the bar keeps the rule meaning what its name says; the other eighteen are boolean and have no dial.

Administration → Insights → Rules (Admin) lists all 24 grouped by class, with a one-sentence description, their knobs, and the honesty column Ignored 30d: episodes of that rule closed in the last 30 days without an acknowledgement, an action, or a mute. A rule you have turned off is still evaluated and counted: the Active tab shows a banner with how many findings your policy is hiding, and Silenced lists the rule under Off by policy with a Turn back on button. Turning a rule off resolves its open episodes as rule_changed, never as a recovery. A change applies on the next evaluation.

The global override layer (one setting per rule, across every cluster) ships in the open-source distribution. Layering a second set of overrides per environment category (production, staging, testing, development) needs the cluster classification and is a KubeBolt Cloud feature, in the Environment tuning tab; there, the effective value resolves knob by knob as environment override → global override → shipped default.

Endpoints

All under /api/v1. Only GET /insights needs a connected cluster; the rest answer from stored state, because history has to answer for clusters that are gone.

MethodEndpointDescription
GET/insightsActive insights for the current cluster (severity, includeMuted), with hiddenByProfile and profile
GET/insights/summaryPer-cluster state for the fleet, served without a live connector
GET/insights/episodesEpisode history: status, severity, rule, cluster, since, until, limit, page
GET/insights/episodes/{id}One episode with its evidence, transitions and recurrence
GET/insights/operational-episodesThe deterministic bursts over a window (since, until; 24 hours by default, 31 days at most)
GET/insights/operational-episodes/{id}/episodesThe episodes of one burst
GET/insights/shift-report”While you were away”, scoped to your presence anchor
POST/account/dashboard-seenThe presence beacon Home fires after rendering
GET · POST/insights/mutesList (any role) and create (Editor, audited)
DELETE/insights/mutes/{id}Lift a mute (Editor, audited)
GET · PUT · DELETE/admin/insight-policies[/{rule}]The rule policy layer (Admin, audited)

Configuration

VariableDefaultPurpose
KUBEBOLT_INSIGHT_EXPIRE_TTL15mA firing episode with no signal beyond this flips to expired
KUBEBOLT_INSIGHTS_RETENTION_HORIZON168h (7 days)How long each fingerprint’s resolved or expired insight record is kept
KUBEBOLT_FINDINGS_RETENTION_HORIZON720h (30 days)Also bounds resolved and expired episodes and their timelines (open-source edition)

The evaluation interval is the server flag --insight-interval (seconds, default 60). The full list of variables is in Environment Variables. Insight notifications are configured separately: see Notifications.