Ask most “AI Ops” products how they reached a conclusion and you get a shrug dressed up as a feature. A model looked at your incident, thought about it, and returned an answer. Maybe it’s right. Maybe it hallucinated a Deployment that doesn’t exist. You can’t tell, because the reasoning lives inside a single opaque call — one prompt in, one verdict out. That’s fine for a chatbot. It is a genuinely bad idea to hand that shape the ability to kubectl delete in production.
We built KubeBolt Autopilot on the opposite premise: an autonomous operations engine should be as auditable as the team that oversees it. Not “trust the model,” but “show your work at every step, and let code decide what can be touched.” That principle turned into a concrete architecture: six layers, each with one job, each observable on its own. This post is an honest walk through why it’s shaped that way, what it buys you, and where it costs more than the black-box approach it replaces.
The problem with the black-box LLM
The seductive version of AI Ops is a single agent with a huge context window and cluster credentials. Hand it the alert, let it reason, let it act. It demos beautifully. It also fails in three ways that matter more than the demo.
It’s unaccountable. When one call does detection, diagnosis, planning and execution at once, there’s no seam to inspect. You can’t answer “why did it scale that deployment?” with anything better than “the model decided.” The post-incident review turns into archaeology.
It’s expensive by default. Every incident, including the textbook ones (the pod that OOMKilled after a deploy, the image pull that failed on a typo’d tag), pays for a reasoning pass on a frontier model. You’re renting a senior engineer’s brain for questions a small model answers just as well.
It’s dangerous when it acts. An LLM that decides and executes has no natural brake on its own mistakes. If it hallucinates a resource name or misreads a risk level, the same faulty reasoning that produced the plan also carries it out. There’s no second opinion, because there’s no second component.
The fix isn’t a better prompt. It’s separation of concerns — the oldest idea in systems design, applied to the newest kind of component. Split the work into layers, give each a narrow job, and you get the two things black boxes can’t offer: an audit trail and a place to put guardrails.
The architecture: six layers, one incident
Every incident walks the same path. Each layer has a defined input and output, so each can be tested on its own and have its model swapped without touching the rest.
L1 Detector rules, no model "Is this an incident Autopilot can handle?"
L2 Router Claude Haiku 5.5 "Is it real, and how much does it matter?"
L3 Investigator Claude Sonnet 5.5 "What's actually wrong?"
L4 Planner Claude Sonnet 5.5 "What do we do, within the catalog?"
L5 Executor AI + deterministic "Do it, with typed actions only."
L6 Postmortem Claude Opus 5.5 "Write down what happened."
Each stage has its own model and a fallback in case it fails, and a platform admin can change both. It runs on Anthropic’s agent SDK or, on Enterprise, on a gateway engine that talks to OpenAI-compatible models such as GLM or Kimi. A circuit breaker sits in front: if the provider stops answering, Autopilot stops starting runs until it’s back, instead of burning one after another. But the interesting part isn’t which models we call. It’s what we let them do.
L1 · Detector (rules)
The first layer never talks to a model. It’s the same rules engine that ships as our open source Insights Engine: 24 deterministic checks that recognize the problems Kubernetes produces every day. Crash loops, OOMKills, images that won’t pull, failing probes, replicas at zero.
What those rules do is surface, not resolve. Every finding reaches you with its recommendation, and you resolve it. Autopilot only opens an incident when the finding belongs to one of the 8 types in its catalog that start a run; the rest it lets through. Each type in the catalog is a declarative entry:
{
ruleId: "image-pull-backoff",
titlePatterns: ["imagepull", "image pull"],
minConfidenceToAct: 0.85, // rollback is destructive: demand high confidence
productionNamespaceRequiresApproval: true,
typicalActionKinds: ["rollback_deployment", "verify_pods_ready"],
}
That entry says what to recognize, how much confidence it takes to act, and which actions usually fix it. Adding an incident type means adding an entry. Whatever isn’t in the catalog never enters the pipeline, and doesn’t spend a single token.
L2 · Router (Claude Haiku 5.5)
A cataloged finding still doesn’t deserve an investigation. This layer asks the cheapest, coarsest question: is this real, and how much does it matter? Clusters are noisy. Plenty of what looks like an incident is a restart that already recovered on its own.
So Layer 2 is a fast triage on Claude Haiku 5.5, with no extended reasoning and five turns at most. It classifies the finding against the catalog, decides its real severity (info, warning or critical), and decides whether it goes on. Severity takes evidence of impact: before calling anything critical, it checks how many available replicas the owning workload has left. If it’s noise, it stops here. Nobody gets paged, and no Sonnet call gets spent.
L3 · Investigator (Claude Sonnet 5.5)
Now the work gets hard, and now we bring out a model that reasons. The Investigator runs Claude Sonnet 5.5 in a multi-turn tool-use loop, up to fourteen turns. It doesn’t get a data dump: it asks for what it needs, with the same read tools Kobi Copilot gives people (logs, describe, events, metrics, recent deploys), under Autopilot’s own identity.
It iterates — form a hypothesis, gather evidence, refine — until it lands on a cause it’s confident in or runs out of turns. If it can’t get there, it doesn’t improvise: it flags that a person is needed and says why. When the cause is outside the cluster, it reads your AWS account read-only, and even then it doesn’t fix it: an outage outside the cluster is fixed outside it, and it names it for you. If the incident lands in the middle of a node rotation or a mass rollout, it says the burst is the cause instead of blaming the workload.
The output is structured: a cause summary, a confidence level and the evidence trail, so whoever reviews it later sees what the model saw, not just its conclusion. That trail is the antidote to the black box.
L4 · Planner (Claude Sonnet 5.5)
Diagnosing and remediating are different skills, so they’re different layers. The Planner takes the Investigator’s cause and writes a concrete plan: an ordered list of actions, their risk, the expected duration, and how to undo it if something goes wrong.
The actions come from a closed catalog. This is the whole list:
| Action | What it does | Risk |
|---|---|---|
restart_deployment | Restarts the rollout and recreates the pods | low |
scale_deployment | Changes replicas (never on an HPA-managed deployment) | medium |
patch_deployment_resources | Adjusts CPU and memory requests and limits | low |
rollback_deployment | Goes back to the previous revision | high — always needs approval |
patch_hpa | Adjusts the HPA’s minimum, maximum or CPU target | medium |
patch_probe | Relaxes a probe’s timings, never its path or port | medium |
verify_pods_ready | Waits for the pods to be ready | low |
inform_only | Recommends without touching the cluster | low |
Six change the cluster; the other two verify or inform. If the model asks for something that isn’t on the list, it doesn’t exist: every action is a typed tool, and an invented parameter is rejected at the tool. If no action in the catalog fixes the fault (an image with a code bug, a config missing from day one), the plan ends in inform_only, names the manual fix, and the incident goes to a person.
On top sits each incident type’s confidence threshold: 0.7 in general, and 0.85 where a wrong action does more damage, such as a rollback. If confidence doesn’t get there, the plan stays a suggestion. The LLM can suggest; it can’t invent capabilities.
L5 · Executor (AI + deterministic)
This is the layer that lets us sleep, and it’s worth being precise about why. By default, the Executor is an agent on Claude Sonnet 5.5, and all it can do is call the catalog’s typed actions, in the plan’s order. Every call goes through a gate written in code that the model can’t change: it waits for approval when one is needed, blocks protected namespaces, refuses to scale to zero what shouldn’t be, and honors a cancellation. If an action fails, it stops. It doesn’t retry or carry on with the rest.
On the Enterprise edition there’s a stricter option, autopilot.executor: code: the approved plan runs with no model, behind the same barriers. And before changing anything it checks two things a model never checked: that the incident’s condition is still present, and that each action still fits the workload as it is now — again after an approval that may have taken a while.
Then comes verification: verify_pods_ready waits for the pods to be ready, 180 seconds by default. If they aren’t, Autopilot rolls back on its own and restores the values from before the change. It’s the same protected, RBAC-bound, audited path a person uses when they approve an action from Copilot.
L6 · Postmortem (Claude Opus 5.5)
Once the incident closes, one last layer runs, off the critical path so it never slows a fix. Claude Opus 5.5 writes the postmortem: the timeline, the cause, the impact the evidence proves, and the action items, exportable to PDF. It’s the one place we happily spend the big model, because it runs once per incident and it’s the document people actually read once the dust settles.
Why layering earns its keep
Two axes justify the whole design. Collapse the layers and you lose both.
Cost separation. The layers follow a deliberate gradient. The Detector costs nothing. Per token, Sonnet 5.5 costs 20 times what Haiku 5.5 does, and Opus 5.5, 40 times. So the cheap model filters, the middle one does the real work, and the expensive one writes one thing per incident. A single monolithic agent would charge big-model prices for every stumble, noise included.
Risk separation. The model proposes; code disposes. Diagnosis and planning are creative work, and that’s where models shine. But acting on production infrastructure is exactly the kind of task you want rigid, predictable and bounded. With a closed catalog of typed actions (L4) and a code gate in front of each one (L5), we get creative diagnosis without creative destruction. The blast radius of a hallucination is an action that doesn’t exist, not a deleted namespace.
On top come the guardrails the operator controls. Three modes: suggest_only (investigates and proposes, never touches; it’s the default), approve_and_execute (you approve each action) and autonomous (in open beta, and anything destructive still needs your sign-off). kube-system is always protected and can’t be unprotected, and you can block more namespaces. Autonomous mode can be limited to a time window. An approval nobody answers expires. And credit spend can be spread across the month, so one bad day doesn’t eat everyone else’s.
The cheapest call is the one you don’t make
If you take one thing from this post, make it this: the cheapest LLM call is the one you never make. But it’s worth being exact about where the saving comes from.
It doesn’t come from the rules resolving incidents. They resolve none. It comes from the Detector deciding what gets in: of everything the 24 rules find, Autopilot only takes on the 8 types it knows how to fix with its catalog. The rest reaches you as a finding, with its recommendation, without spending a token. And of what does get in, the triage stops the noise with the cheapest model before it reaches a more expensive one.
That flips the usual AI Ops pitch, which starts with the frontier model and hands it the keys. We start by bounding things: what gets in, what can be done, and who approves. The model works inside that perimeter. It’s less flashy. It’s far easier to trust.
The honest tradeoffs
This architecture isn’t free, and pretending otherwise would be exactly the magic-answer dishonesty we’re arguing against. Here’s what it costs.
It’s more complex to build and operate. Six coordinated stages, per-incident state, a queue with a concurrency cap, fallback models, a circuit breaker, two engines — that’s a lot more moving parts than “call the model, run the output.” A monolithic agent is genuinely simpler to stand up. We take on the complexity because auditability and safety aren’t optional for production infrastructure, but it’s real complexity, and it’s ours to carry.
We spend more on engineering to spend less on inference. A lot of our effort goes to the unglamorous part: writing and maintaining rules, curating the action catalog, hardening the gate, measuring each model before making it the default. That’s human work a pure-LLM approach skips by pushing everything to the model. We think it’s the right trade, but the cost moves from your inference bill into our code.
Layer boundaries add latency and handoffs. Moving an incident through discrete stages, saving state at each one, is slower than a single uninterrupted reasoning stream. We think a few extra seconds on an incident are a fair price for knowing exactly what happened and being able to roll it back.
The catalog has a ceiling. Autopilot only fixes what its catalog covers: 8 incident types and six actions. Everything else ends with a person, with the investigation done. That’s the safety net working, but it means coverage depends on how many types we know how to handle well, and that’s ongoing investment, not a solved problem.
Real numbers, presented honestly
Before we change a stage’s default model, we measure it with a harness that tests the stage on its own, against cases recorded in our lab. A replay server answers the read tools, with no cluster involved. These are lab numbers, not an SLA, but they’re measured:
- Triage on Haiku 5.5: 195 runs over 39 cases, the same rule and severity decisions as its predecessor, $0.32 in total (under two tenths of a cent per run) and an 8-second median.
- Investigator on Sonnet 5.5: 31 cases, the right cause in 93 of 93 runs, the remedy named in 50 of 50, a person requested when it should be in 54 of 54, $0.021 per run and a 10-second median.
What we haven’t yet measured with the same care is the end-to-end cost of an incident, so we don’t quote one. We used to publish “under a dollar and under 90 seconds”; it was an estimate from when the product was an MVP, and we won’t stand behind it without measuring it.
What we will defend is the shape: a system where every incident has a paper trail, where code decides what can be touched, and where the LLM is a component you can reason about instead of a black box you have to trust.
If you want to see how it all fits together, it’s in How it works, and the autonomy levels are in the Kobi docs. The Layer 1 rules are open source, Apache 2.0, if you’d rather read the code; Autopilot is part of KubeBolt Cloud and the Enterprise edition. The most trustworthy way to run autonomy on infrastructure is to make it something you can inspect — so we did.