Insights Engine

The insights engine evaluates 24 built-in rules against every registered cluster. Each finding carries the affected resource, a human-readable message, and a specific suggestion with remediation steps or kubectl commands.

Since 2.1.0 a finding is no longer a light that is either on or off. It has a history, an owner and a policy: episodes record what happened, mutes record what you decided to tolerate, and the rule matrix records how loud each rule is allowed to be.

Rule definitions

Episodes

Every firing insight opens an episode — a durable row keyed on the fingerprint, with an append-only timeline of transitions: opened, flapped, severity moved, muted, unmuted, resolved, expired. Each transition records who caused it (system, rule:<id>, watchdog, user:<id>) and why.

Typed evidence. A rule records what it actually saw when it fired — the numbers and objects behind the verdict, not a re-read of the cluster minutes later. The episode keeps it, so the recommendation can be checked against the state that produced it even after the cluster has moved on. Kobi reads the same evidence through get_insight_episode.

Flap damping. A fingerprint that re-fires within 10 minutes of resolving is the same episode flapping — the flap counter goes up, no new episode is created, and no second notification is sent. Outside that window it is a genuinely new episode.

The expiry watchdog. A firing episode whose cluster stopped reporting is no longer verifiable — the agent left, the cluster was deleted. After 15 minutes without a signal (KUBEBOLT_INSIGHT_EXPIRE_TTL) the watchdog flips it to expired and it drops out of the Active list. expired is not resolved: nobody observed recovery, and the distinction is preserved in the timeline.

Resolutions are typed, so history reads as a narrative instead of a count: auto_recovered, remediated (an executed action cleared it, with the action id attached), manual, and rule_changed (a policy edit cleared it).

Active and History. The Insights page splits into two tabs. History has server-side filters (status, severity, rule, cluster, time window) and pagination, and each episode opens a detail page with its full timeline and its recurrence — how often this exact fingerprint has come back.

Retention rides the hourly maintenance pass with the insights horizon (KUBEBOLT_INSIGHTS_RETENTION_HORIZON, 7 days by default). Firing episodes are never pruned.

Mutes

A mute silences one rule on one resource in one cluster. It is a display layer, not a suppression: the engine keeps evaluating and episodes keep recording — only your attention is spared.

• Always bounded: an expiry date, or until resolved. A permanent mute is possible but demands a written reason, so whoever finds it in six months knows why it exists.
• Mutes subtract everywhere insights are counted, the Overview KPI included. The Insights page is the only surface that can show them back, behind includeMuted=true.
• Critical severity pierces a mute. An escalation gets through, which is why the engine never consults the mute list — piercing needs the underlying signal intact.
• Every mute and unmute lands in the episode timeline and the audit trail with the actor’s name and the reason.
• Reading the list needs no particular role; creating or lifting a mute needs Editor.
• Administration → Insights → Silenced lists them with cluster names, not UUIDs.

Re-muting the same (cluster, rule, resource) key updates the terms instead of stacking duplicates.

The shift report

Home’s greeting tells you what happened while you were away. The window hangs from your own presence anchor — the last time the Home page actually rendered for you — not from your last login, which token refresh would freeze. Your first shift defaults to a 24-hour window; an anchor older than 30 days is clamped, and the report says so.

It reports events, not standing state: what opened, what auto-recovered, what was remediated, what expired, how many of the newly opened episodes are still burning, and how many were critical. A condition that merely spans your absence belongs to the Active list, not here — so the report goes quiet when nothing actually happened.

Bursts are clustered deterministically — the same window in produces the same grouping out — into node_rotation, node_pressure, mass_rollout, or an honest unknown_burst when nothing recognizable fits. The longest episode of the window is named and linked, and the mute counters (“silenced while you were away”, “silences standing now”) are part of the story.

The rule matrix

Rules split into two classes, and the class decides which knob you get:

Six rules carry a numeric threshold: crash-loop (restarts), frequent-restarts (restarts), resource-underrequest (request/usage ratio), cpu-throttle-risk and memory-pressure (usage/limit ratio), and cert-expiring (days before expiry). Moving the bar keeps the rule meaning what its name says; the other eighteen are boolean and have no dial.

Administration → Insights → Rules lists all 24 with their class, a one-sentence description, their knobs, and the honesty column Ignored 30d — episodes of that rule closed in the last 30 days without an acknowledgement, an action, or a mute. A rule you have turned off is still evaluated and counted: the Insights page reports how many findings your policy is hiding rather than pretending they did not happen.

Endpoints

None of these require a connected cluster — history has to answer for clusters that are gone.

Configuration

Insight notifications are configured separately — see Notifications.