How KubeBolt works

It is 3am and something breaks. Here is what happens next.

We follow a Kubernetes incident from start to finish: what detects it, what the model reads, what it may touch and what is written down by morning.

01 · See, understand, act

See Understand Act Verify

Cluster Map

prod-eu-west-1 Kubernetes · Hubble
payments3
payments-apiService
payments-api-7f8d4Pod · CrashLoopBackOff
payments-api-7f8d4-p2kqPod
checkout2
checkout-webService
checkout-web-6494Pod
external1
stripe-api443/TCP

1.70 req/s · 2xx 0.85/s · 5xx 0.85/s · avg latency 68.4 ms

Insights

ActiveHistory 24 rules
!

Liveness probe failingWARNING

Pod payments/payments-api-7f8d4-xk2m liveness probe has failed 6 times.

Suggestion: Check the liveness probe config against what the app actually does at startup.

!

Zero Available ReplicasCRITICAL

Deployment payments/payments-api has 0 available replicas.

!

Pod not passing readinessWARNING

Pod payments/payments-api-7f8d4-p2kq is Running but has not been Ready for over 2 minutes.

Conditions firing right now · 24 rules

Incident context

payments-api

Deploymentrevision 43 · 8m ago

Pods7 restarts

Metricserror rate +18.7%

Networkpayments path degraded

Eventsliveness probe failed

Logs3 relevant errors

One operational context

Six sources correlated on the same workload. Without leaving this screen.

KobiINVESTIGATING

Why has payments-api been restarting since 03:06?

querying your cluster

get_resource_detail payments-api

Seven restarts in eight minutes, all with the same exit code. The pod starts and something kills it.

get_pod_logs payments-api-7f8d4-xk2m

There is no application error in the logs. The last message is “listening on :8080”, and the stop signal arrives right after it.

get_workload_history payments-api

Commit 4127 added a migration at startup. Since then the container takes 41 s to become ready.

Root cause. The liveness probe is still at initialDelaySeconds: 8, so kubelet kills the container 33 s before it can be ready. It isn't the app — it's the probe.

You operatewith your RBAC
ScaleSet imageSet resourcesSet envRestartRollbackCordonDrain
Audit trail

03:02 · scale payments-api 24 → 36you

01:12 · set image checkout-webyou

yesterday · drain ip-10-0-3-042ops

Every operation is recorded, with who and when.

Kobi proposeslow risk

Relax the liveness probe on payments-api

deployments · payments/payments-api

The container has taken 41 s to start since commit 4127 and the probe is still at 8 s, so kubelet kills it too early.

patch_probeinitialDelaySeconds: 8 → 45

What will happen

3 pod(s) will be terminated

1 Service(s) will be left without endpoints

Would apply · patch_probe

Execute · patch probeDismiss

If the check doesn't pass, it undoes its own change.

Set image · payments-api

ROLLOUT COMPLETE 12s elapsed
READY26/26
UPDATED26/26

Pods (26) · rolling update · reversible

Liveness probe failing

RESOLVED · AUTO_RECOVERED
VerificationVERIFIED

rollout complete · 26/26 ready

liveness probe passing · 90 s straight

error rate back to baseline

FIRST SEEN03:14:07
RESOLVED03:15:34
DURATION87 s
Recurrence · same fingerprint

today · 87 sthis one

6 days ago · 13 mauto_recovered

20 days ago · 4 mauto_recovered

If any of the three fails, it reverts and escalates the incident to you.

See. Understand. Act.

From signal to resolution, with nothing to reconstruct along the way.

01 · See, understand, act

See

See your entire cluster.

Health, workloads, dependencies, network and cost in one place.

All green. Until it isn't.

See

What's happening, right now.

Deterministic rules surface what is broken and what is about to break, each with a concrete recommendation. Not an alert — a finding with a name.

24 rules in production. The one that fires most: the liveness probe.

Understand

The context, before the answer.

KubeBolt correlates the deployment, restarts, metrics, traffic, events and relevant logs into one operational context. Kobi reasons over that context to find the cause.

KubeBolt builds the operational context. Kobi reasons over it.

Understand

The cause, not the symptom.

Kobi reasons over that context, already built: it doesn't go looking, it already has it. That's why it gets there fast — and why you can follow every step.

The intelligence doesn't come from the model. It comes from the context in front of it.

Act

You operate. Or Kobi prepares it.

The operations you already know, one click away, with your RBAC. Or Kobi proposes the action, validates it against the cluster and shows you what will happen before touching anything.

KubeBolt is still an operations platform even if you never enable autonomous mode.

Act

Execute with control.

The action comes from a closed catalog and the rollout is visible pod by pod, with live ready and updated counters.

This is production changing in front of you.

Verify

Verified, not just applied.

KubeBolt doesn't call an action good because it finished: it checks that the probe passes, that the error rate is back to baseline and that the workload holds. If it doesn't, it reverts.

And it keeps the episode, with the previous ones. Three with the same fingerprint is no longer bad luck.

See. Understand. Act.

From signal to resolution, with nothing to reconstruct along the way.

02 · The six layers

Six layers, from alert to report.
Code detects and limits. The model reasons.

  1. L1

    Detector

    24 rules watch your Kubernetes cluster state: crash loops, OOM, images that won't pull, expiring certificates. Nothing to configure and no credits spent.

    100% deterministic
  2. L2

    Router

    Classifies what was detected and decides whether it needs investigating. This is where credits start to count.

    AI
  3. L3

    Investigator

    Cross-checks events, logs, metrics and recent changes until it finds the cause.

    AI
  4. L4

    Planner

    Writes the plan. If confidence doesn't reach the rule's threshold, it stays a suggestion.

    AI
  5. L5

    Executor

    Applies only what passes the guardrail. The plan is validated against code, not against a prompt.

    AI + deterministic
  6. L6

    Postmortem

    Leaves the timeline, the evidence and a draft analysis.

    AI

Each rule is a Skill: it carries its confidence threshold and says which catalog actions are valid for that incident. If it isn't in the catalog, the finding still arrives, with its recommendation, and you resolve it.

03 · How far it goes

The model proposes.
It never executes.

Tools run in the backend and the model only sees results. Before touching Kubernetes production, every action goes through code the model can't change.

If a step isn't on the list, the plan is rejected and escalates to a person.

A new action takes a release

Changing the prompt doesn't widen the catalog.

A log can't execute anything

Logs are text a third party may have written. That's why the model proposes.

With the agent's RBAC

It runs with a Kubernetes ServiceAccount and is audited: who authorized it, when and on which resources.

Unattended window

Days, hours and timezone. Outside it, the incident is logged and nothing acts.

BYOK on OSS and EE Self-Hosted: your key or endpoint, and AI traffic goes from your install to your provider. On SaaS, AI is managed. Autonomy levels

Try it

Break something
and see how far Kobi goes.

A closed action catalog and your cluster's RBAC. The only thing that changes is who approves.

1Pick the control level

2Break something

three get resolved · one reverts · one stays out

Kobi Copilot · approve and run

Kobi diagnoses, shows you the exact command and runs it on your click. Destructive actions always carry your signature.

Who actsKobi + you
Who approvesYou press Run
SpeedSeconds to the fix
Kobi payments-api · CrashLoopBackOff Approve and run
03:14:07 Alert CRITICAL · payments-api · CrashLoopBackOff · 7 restarts
03:14:19 Diagnosis commit #4127 raised startup from 8 to 41 s and the liveness probe kills the pod before it finishes · confidence 0.91
03:14:48 Proposal patch_probe payments-api · 90 s startupProbe
patch_probe payments-api · 90 s startupProbe
$kubectl patch deployment payments-api -n payments -p '{"spec":{"template":{"spec":{"containers":[{"name":"api","startupProbe":{"httpGet":{"path":"/healthz","port":8080},"periodSeconds":10,"failureThreshold":9}}]}}}}'
closed catalogreversiblenon-destructive
Approve and run
Kobi shows you the exact command and runs it on your click, with your RBAC and an audit log.
signed · you waited 00:04
03:15:06 Execution startupProbe applied · 3/3 pods Ready in 34 s · run on your click
03:15:38 Verification pods Ready · no restarts in 60 s · p99 back to 180 ms
03:15:40 Postmortem postmortem written · links commit #4127 and who merged it
Resolved and verified in 91 s. With your signature where it was needed. 1 click from you · waited 00:04

Illustrative script using actions from the real catalog.

04 · Verify and revert

If the fix does not hold,
it undoes it and calls you.

Applying isn't resolving. Autopilot saves the previous values before touching anything, then checks the result. It works today.

Captures the previous state

The values it will change are captured first, so the revert restores exactly what was there.

Checks the result

Pods Ready and no restarts. It isn't enough for the action to apply without an error.

passes

Resolved

The episode is saved with the previous ones, in case it comes back.

fails

Reverts if it fails

In autonomous mode it undoes itself. A rollback that needs approval waits for you.

Escalates to a person

With what was tried and why it didn't work. That's the honest ending when the catalog falls short.

Postmortem · L6

By morning
it is already written.

The last step fixes nothing: it tells the story. Ready to paste into the channel or the ticket.

Sample, using the incident on this page.

Postmortem · payments-api

RESOLVED · AUTO_RECOVERED

The timelineFrom the first signal to the last Ready pod.

  1. Deploy of commit #4127
  2. Alert · CrashLoopBackOff · 7 restarts
  3. Cause · startup went from 8 to 41 s and the liveness kills the pod
  4. Action · patch_probe · 90 s startupProbe
  5. Verified · pods Ready · p99 back to 180 ms

The action logIn autonomous mode it's signed by Autopilot.

patch_probepayments/payments-apiappliedAutopilot

The five whysThe model writes it, you sign it.

Why was the pod restarting?
The liveness probe killed it before it finished starting.
Why now?
Commit #4127 took startup from 8 to 41 s.
Why wasn't it caught earlier?
Proposal: review probes whenever startup time changes.

05 · What you install and what you don't

Outbound, never inbound.
What you install and what you don't.

The four answers a platform team asks for before installing anything in production.

An agent that dials out, never in

Outbound gRPC channel from inside the cluster. Your API server is never exposed: no VPN, no bastion, no open ports.

ServiceAccount and three RBAC levels

No kubeconfig on the operator's laptop. Start with metrics or read-only and move up to operator whenever you want.

Already running Prometheus? Keep it

If you have it, it's joined to resources, events and changes. If not, the agent uses kubelet metrics.

Which data leaves and where it's processed

Only what's needed to investigate leaves the cluster. What is sent, how long it's kept and where, in the trust model.

The channel

What crosses the channel,
and what never leaves.

Leaves

  • Kubelet metricsCPU, memory and network for nodes and pods.
  • L3/L4/L7 network flowsSource, destination, verdict, code and latency.
  • Kubernetes objectsCluster state, without Secret values.
  • Logs (on demand)Only when a diagnosis needs them.

Never crosses

  • Secret valuesRedacted before leaving the API layer.
  • Network payloadsMetadata travels, never the request body.
  • Your kubeconfigCredentials stay in the agent.
  • Another org's dataRow-level isolation per organization in the database.

On self-hosted nothing reaches KubeBolt: no product telemetry, no usage ping. Trust model

Why the other way round

Traditional modelLens · Kite · K9s · direct kubectl KubeBoltThe agent dials out, nothing comes in
API server Reachable from the client Stays private, never exposed
Credentials On every user's machine Only in the agent, with a ServiceAccount
Private clusters VPN or jump host Private GKE and EKS, isolated VPCs
Audit In the cluster, not centralized Centralized in the control plane

Where the metrics come from

DaemonSet mode
One pod per node reads kubelet metrics and, if you run Cilium, Hubble events. No prior dependencies.
Prom-read mode
For AMP, Azure Monitor or GMP: KubeBolt queries your Prometheus and forwards the series, without touching your config.
Integrations
OpenCost, Trivy, Kyverno, Falco and ArgoCD are detected on their own. If they're in the cluster, they're used.

How much is kept

OSS · self-hosted ∞
Free 15 days
Team 30 days
Business 90 days
Enterprise 365 days

Around 1 GB per 100 pods at 30 days. Cardinality decides, not days. On self-hosted it's a value in your Helm chart. Detection rules don't depend on retention. See the plans

06 · Also, the panel

Everything you'd do with kubectl.
From the browser.

  • kubectl exec -it

    A terminal in any pod

    Multi-pod and multi-cluster, with no kubeconfig on the client and no VPN.

  • kubectl exec -- ls

    Runtime filesystem

    What's in the pod now, not what shipped in the image.

  • kubectl logs

    Multi-pod logs

    At workload level and on demand, without waiting for your stack.

  • kubectl port-forward

    Port-forward without exposure

    Grafana or ArgoCD with no public LoadBalancer or Ingress.

  • kubectl edit · scale · rollout

    Full resource CRUD

    With validation and CRDs. Deleting Namespaces, Nodes, PV, PVC or RBAC is vetoed by design.

  • kubectl config use-context

    Multi-cluster by default

    One agent per cluster, all in the same control plane.

Roadmap · in design

If you use GitOps, the fix goes to the repo.

Autopilot already verifies every fix and reverts it if things get worse. What's coming is having it written in your repo, where the truth lives.

  1. KubeBoltopens the PR
  2. Your repoGitHub · GitLab
  3. ArgoCDsyncs
  4. Your clusterno drift

Before opening a PR we confirm that file produces that workload. Your checks and branch protection rule.

Most tools watch. KubeBolt operates.

“KubeBolt is a Kubernetes operations platform: one place to see, understand and operate your clusters, and Kobi works from that same context. The UI is excellent, but the product is two pieces: the Core (API), home to the correlation engine, and the agent that brings it the facts without exposing anything.”

Leafar Maina · Founder, CLM Cloud Solutions

Cloud

Connect your cluster
in under 2 minutes.

Create the account, install the agent, and on the first alert you'll see Kobi working with you. Start in suggest-only.

Free · 2 clusters · no time limit

Example · the app generates it with your token docs

helm install kubebolt-agent \
  oci://…/helm/kubebolt-agent \
  … --set cluster.name=<cluster-name>

Rather see it with a person? Book 20 minutes

Open source on your cluster

Install KubeBolt on the cluster you want to monitor. The agent is added from inside the product.

bash
$ helm install kubebolt \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  --namespace kubebolt --create-namespace --wait

Apache 2.0 · ★ on GitHub · the agent dials out, never in.

What's next

One note per release, no noise.