Ask most “AI Ops” products how they reached a conclusion and you get a shrug dressed up as a feature. A model looked at your incident, thought about it, and returned an answer. Maybe it’s right. Maybe it hallucinated a Deployment that doesn’t exist. You can’t tell, because the reasoning lives inside a single opaque call — one prompt in, one verdict out. That’s fine for a chatbot. It is a genuinely bad idea to hand that shape the ability to kubectl delete in production.

We built KubeBolt Autopilot on the opposite premise: an autonomous operations engine should be as auditable as the humans it stands in for. Not “trust the model,” but “show your work at every step, and let a boring, deterministic layer hold the keys.” That principle turned into a specific architecture — six layers, each with one job, each observable on its own. This post is the honest tour of why it’s shaped that way, what it buys you, and where it costs more than the black-box approach it replaces.

The problem with LLM-as-black-box

The seductive version of AI Ops is a single agent with a big context window and cluster credentials. Feed it the alert, let it reason, let it act. It demos beautifully. It also fails in three ways that matter more than the demo.

It’s unaccountable. When one call does detection, diagnosis, planning, and execution all at once, there’s no seam to inspect. You can’t answer “why did it scale that deployment?” with anything better than “the model decided to.” Post-incident review becomes archaeology.

It’s expensive by default. Every incident — including the 60-70% that are textbook, the pod that OOMKilled after a deploy, the image pull that failed on a typo’d tag — pays for a frontier-model reasoning pass. You’re renting a senior engineer’s brain to answer questions a case statement could answer.

It’s dangerous when it acts. An LLM that both decides and executes has no natural check on its own mistakes. If it hallucinates a resource name or misreads a risk level, the same faulty reasoning that produced the plan also carries it out. There’s no second opinion, because there’s no second component.

The fix isn’t a better prompt. It’s separation of concerns — the oldest idea in systems design, applied to the newest kind of component. Split the work into layers, give each a narrow responsibility, and you get the two things black boxes can’t offer: an audit trail and a place to put guardrails.

The architecture: six layers, one incident

Every incident flows through the same pipeline. It enters at the top and descends only as far as it needs to — most stop early. Each layer is a Go interface with a defined input and output, so we can test them in isolation and swap a model in one without touching the others.

L1  Detector      deterministic   "Do I already know this pattern?"
L2  Router        Claude Haiku    "Is this signal or noise?"
L3  Investigator  Claude Sonnet   "What is actually wrong?"
L4  Planner       Sonnet / Opus   "What should we do about it?"
L5  Executor      deterministic   "Do it — safely, under guardrails."
L6  Postmortem    Claude Opus     "Write down what happened."

The whole thing runs on the Claude Agent SDK, with model calls routed through an internal gateway that fails over across providers (Anthropic API → Google Vertex → AWS Bedrock) with a circuit breaker, so a single provider blip doesn’t stall an investigation. But the interesting part isn’t which models we call. It’s when we refuse to call them.

L1 — Detector (deterministic)

The first layer never talks to a model. It’s the same rules engine that ships today as our open-source Insights Engine: a set of deterministic checks that recognize the incidents Kubernetes produces by the thousand. BackOff with a restart count over five and an OOMKill signature? That’s a memory-limit problem, and we know the shape of the fix without asking anyone. Image pull failing on a missing tag? Known pattern, known response.

func (r *CrashLoopRule) Match(incident *Incident) (bool, float64) {
    return incident.Reason == "BackOff" && incident.RestartCount > 5, 0.95
}

Target latency here is under 50 milliseconds, because there’s no network round-trip to a model at all. If a rule matches with high confidence, we skip every LLM layer below and go straight to a plan. If it matches with middling confidence, we pass the hint down to the Investigator as a head start rather than a verdict. If nothing matches, we escalate.

This is the layer that makes the economics work, and I’ll come back to it — it’s the whole thesis.

L2 — Router (Claude Haiku)

If L1 doesn’t recognize the incident, the question isn’t yet “what’s the root cause?” It’s the cheaper, coarser question: is this even worth investigating? Clusters are noisy. A lot of what looks like an incident is a normal restart, a scheduled scale-down, or the fourth duplicate of an event we already have open.

So Layer 2 is a fast, cheap triage pass with Claude Haiku — a couple hundred output tokens, well under a fraction of a cent, capped around a second. It returns one of three verdicts:

noise — transient, expected, or duplicate. Stop here. Don’t wake anyone, don’t spend a Sonnet call.
investigate — real, but not on fire. Escalate.
critical — production impact, escalate with urgency.

The point of L2 is protective. It stands between the cheap deterministic world above it and the expensive reasoning world below it, and its entire job is to keep noise from ever reaching the models that cost real money.

L3 — Investigator (Claude Sonnet)

Now the work gets genuinely hard, and now — only now — we bring out a reasoning model. The Investigator runs Claude Sonnet in a multi-turn tool-use loop. It doesn’t get a data dump; it asks for what it needs, using the same tools our Copilot exposes to human operators — get_pod_logs, describe_deployment, get_recent_events, query_metric — except running under Autopilot’s own service account.

It iterates: form a hypothesis, pull evidence, refine, repeat, until it reaches a root cause it’s confident in (our bar is a confidence score above 0.7) or it runs out of turns. If it can’t get there, it doesn’t bluff — it flags requires_human and escalates to a person with everything it gathered attached. The output is structured: a root-cause summary, a confidence score, and the actual evidence trail, so a human reviewing later sees what the model saw, not just its conclusion.

That evidence trail is the antidote to the black box. The Investigator’s reasoning is a record, not a vibe.

L4 — Planner (Sonnet or Opus)

Diagnosis and remediation are different skills, so they’re different layers. The Planner takes the Investigator’s findings and produces a concrete RemediationPlan: an ordered list of actions, a risk level, an expected duration, and — critically — a rollback strategy for when things go sideways.

Model choice here is deliberate. A simple, single-step fix gets Sonnet. A complex, multi-step plan with real blast radius gets Claude Opus, where the extra reasoning quality earns its cost. We spend the expensive model only on the decisions that are actually expensive to get wrong.

Then comes the seam that matters most. The plan the LLM produces is not trusted. Before it goes anywhere near your cluster, a deterministic ActionRegistry validates every action against a closed whitelist. If the model proposes an action Kind that isn’t in the registry, the plan is rejected outright and a human is pulled in. The LLM can suggest; it cannot invent capabilities.

L5 — Executor (deterministic)

Here’s the layer that lets us sleep. The Executor does not use an LLM. It’s plain Go. It takes the validated plan and walks it action by action, and it is the only component in the system with the ability to mutate your cluster.

Everything the Executor can do lives in a closed catalog of roughly 25 actions — scale_deployment, rollout_undo, patch_deployment_resources, restart_pod, drain_node, and so on — each tagged with a risk level, whether it’s reversible, and whether it always requires human approval. Destructive actions (delete_deployment, patch_secret, drain_node) require sign-off no matter what mode you’re in. For every action, the Executor:

Checks the cluster’s mode allows this risk level; if not, it stops and requests approval.
Sends the command to the in-cluster agent over a persistent WebSocket.
Waits for the result with a timeout.
Persists the outcome immediately — audit in real time, not after the fact.
On failure, triggers the rollback strategy.

func (e *Executor) Execute(ctx context.Context, plan *RemediationPlan, mode AutopilotMode) (*Execution, error) {
    execution := e.createExecution(plan)
    for i, action := range plan.Actions {
        if !e.canExecute(action, mode) {
            return e.requestApproval(execution, plan, i) // guardrail, not suggestion
        }
        result := e.executeAction(ctx, action)
        execution.ActionResults = append(execution.ActionResults, result)
        e.saveExecution(execution)
        if result.Status == "failed" {
            return e.rollback(ctx, execution, plan, i, result.Error)
        }
    }
    return e.verify(ctx, execution, plan)
}

This is L5 as it ships today as our deterministic write-ops surface — the same guarded, RBAC-bound, fully audited execution path whether a human clicks “approve” in Copilot or Autopilot runs it autonomously. The rollback triggers are themselves deterministic: an action fails, or verification fails (pods don’t stabilize within 180 seconds), or a key metric degrades sharply post-change (latency doubles, error rate 5x’s). Any of those, and the Executor reverses course on its own.

L6 — Postmortem (Claude Opus)

Once an incident reaches resolved or failed, a final layer runs asynchronously — off the critical path, so it never slows a fix. Claude Opus writes the postmortem: a timeline, root-cause narrative, impact assessment, and structured action items, rendered to a PDF. This is the one place we happily spend the big model and take our time, because a postmortem’s whole value is quality, and it’s the artifact humans actually read after the dust settles.

Why the layering earns its keep

Two axes justify the whole design. Collapse the layers and you lose both.

Cost separation. The layers map onto a deliberate cost gradient: free (L1) → nearly free (Haiku) → moderate (Sonnet) → expensive-when-needed (Opus). An incident only pays for the intelligence it actually requires. The routine majority is resolved for effectively nothing. The genuinely hard cases get frontier-model reasoning — but they’re the minority, so the average cost stays low. In our MVP this keeps end-to-end resolution comfortably under $1 an incident, most of them in under 90 seconds. A single monolithic agent would charge frontier-model prices for every hiccup, noise included.

Risk separation. The LLM proposes; a deterministic component disposes. Diagnosis and planning are creative work, and that’s where models shine. But acting on production infrastructure is exactly the kind of task you want to be rigid, predictable, and bounded — the opposite of what a language model is. By putting a whitelist-validating registry (L4) and a non-LLM executor (L5) between the model’s ideas and your cluster, we get creative diagnosis without creative destruction. The blast radius of a hallucination is a rejected plan, not a deleted namespace.

Layered on top of that are the operator-facing guardrails: modes that run from suggest_only (investigate, never touch) through approve_and_execute up to autonomous; blocked namespaces (kube-system and friends are off-limits by default); cooldowns; freeze schedules; per-org concurrency limits; and a plan TTL of four hours so nobody approves a fix built for a cluster state that no longer exists. There’s even massive-outage detection: past 50 events a minute from one cluster, Autopilot stops investigating individually and pages a human, because 300 simultaneous failures aren’t 300 incidents — they’re one, and a person should see it.

The insight that makes it cheap: Layer 1 does the heavy lifting

If you take one thing from this post, take this: the cheapest LLM call is the one you never make.

The counterintuitive core of the design is that the first layer, the dumbest one, resolves the majority of incidents. Somewhere around 60-70% of what a Kubernetes cluster throws at you falls into well-worn patterns — CrashLoopBackOff from a bad config, OOMKills that want a higher memory limit, failed schedules, image pull errors, replicas that never go Ready. These don’t need reasoning. They need recognition. And recognition is a deterministic rules engine’s home turf: sub-50ms, no tokens, and — this matters — more reliable than an LLM, because a rule that matches Reason == "BackOff" matches it the same way every single time.

So the expensive layers exist to handle the long tail, not the common case. Most incidents are recognized and resolved by L1 before Haiku is even consulted. That’s not a fallback path; it’s the main path. The AI is the exception handler.

This inverts the usual AI Ops sales pitch, which leads with the frontier model. We lead with determinism and treat the LLM as a scalpel for the cases that genuinely need one. It’s less flashy. It’s a lot cheaper to run, and a lot easier to trust.

The honest tradeoffs

This architecture is not free, and pretending otherwise would be exactly the kind of magic-answer dishonesty we’re arguing against. Here’s what it costs.

It’s more complex to build and operate. Six coordinated layers, a state machine per incident, an orchestrator handling dedup and backpressure, provider failover, a persistent agent WebSocket — that’s substantially more moving parts than “call the model, run the output.” A monolithic agent is genuinely simpler to stand up. We took on the complexity because auditability and safety aren’t optional in production infrastructure, but it’s real complexity, and it’s ours to carry.

We spend more on engineering to spend less on inference. The determinism-first bet means a lot of our effort goes into the un-glamorous layer: writing and maintaining deterministic rules, curating the action whitelist, hardening the executor. That’s human work that a pure-LLM approach skips by pushing everything to the model. We think it’s the right trade — rules are cheaper, faster, and more reliable than tokens at runtime — but the cost moves from your inference bill to our codebase.

The layer boundaries add latency and handoffs. Passing an incident through discrete stages, persisting state at each, is slower per step than one uninterrupted reasoning stream would be. We claw it back by having most incidents exit early at L1, but for the hard cases that traverse all six layers, the structure has overhead. We think a few extra seconds on a genuinely complex incident is a fine price for knowing exactly what happened and being able to roll it back.

Determinism has a ceiling. A rules engine only catches what someone thought to encode. Novel failure modes fall through to the LLM layers by design — that’s the safety net working — but it does mean L1’s hit rate is a function of how good our rules are, and that’s an ongoing investment, not a solved problem.

Real numbers, framed honestly

Here’s where we are, stated plainly. These are MVP-and-testing figures, not an SLA, and we’d rather you hold us to the architecture than to a benchmark.

In testing, the deterministic first layer handles the majority of common incidents — the 60-70% we designed it to catch — without a single model call. End-to-end, the incidents Autopilot resolves come in under 90 seconds and under $1 each, because the cost gradient does its job: the cheap cases stay cheap and the expensive model only shows up when it’s earning its keep. The layers that make this real are shipping as part of the Autopilot preview, with L1 already open-source in the Insights Engine and L5 already live as the guarded write-ops surface behind Copilot.

We won’t dress those up as guarantees. What we’ll stand behind is the shape: a system where every incident has a paper trail, where a deterministic layer holds the keys, and where the LLM is a component you can reason about rather than a black box you have to trust.

If you want the full architecture — the layer contracts, the action whitelist, the guardrail model — it’s all in the technical docs, and the engine is open source, Apache 2.0 if you’d rather just read the code. That’s the whole point. The most trustworthy way to run infrastructure autonomy is to make it something you can inspect — so we did.