KubeBolt

Cloud Native Guayaquil · CNCF · Thursday, Oct 29, 2026

AI agents operating Kubernetes: how they work under the hood

An EKS cluster · namespace tienda

kubectl get pods -n tienda NAME READY STATUS RESTARTS AGE catalog-api-6f9c8d7b5-2xkpt 1/1 Running 0 3h12m catalog-api-7d4b9c6f8-qm7rn 0/1 ImagePullBackOff 0 4m orders-api-5b8f7c9d4-hz2lw 1/1 Running 0 3h12m shop-web-8c6d5f4b7-vt9np 1/1 Running 0 3h12m kubectl rollout status deploy/catalog-api -n tienda Waiting for deployment "catalog-api" rollout to finish: 1 old replicas are pending termination...

A deploy that won't finish.

And the image is the same one the working version uses.

We ask Kobi

Question for Kobi, KubeBolt's copilot

Open-source KubeBolt, connected to the cluster and its AWS account.
Kobi can read both, but it can't change anything in AWS.

Before the demo · the lab

A shop on EKS and what it uses from AWS

AWS · US-EAST-2 RESOURCE TYPE Network and DNS Compute and containers Databases Storage Security and identity Messaging and audit Clients web and load test ALB the entry point EKS · TIENDA-EKS shop-web the website orders-api orders catalog-api catalog checkout purchase 2 ARM EC2 nodes · Auto Scaling · vpc-cni Route 53 · private zone db.tienda.internal RDS PostgreSQL the shop's database Secrets Manager the password, via External Secrets Rotation Lambda changes the password S3 the catalog · with Pod Identity KMS customer-managed key SQS and its DLQ one message per order Lambda confirms DynamoDB the orders ECR tienda/python · the images IAM the node role and the Pod Identity roles CloudTrail every change, who made it and when KubeBolt with Kobi open source Reads the cluster and account. In AWS, read-only.

orders-api reaches its database, an RDS PostgreSQL, through a private Route 53 name. The password is in Secrets Manager and a Lambda rotates it.

catalog-api reads the catalog from an S3 bucket with Pod Identity. The bucket is encrypted with a customer-managed KMS key.

checkout stores each order in DynamoDB and publishes it to an SQS queue. A Lambda reads it from the queue and confirms it. Anything that fails five times ends up in the DLQ.

The images come from ECR, and CloudTrail records every change made in the account, IAM ones included.

KubeBolt reads the cluster and the account. In AWS it uses a role that can only read.

Before the talk I broke one of these pieces. Kobi doesn't know which.

● Live

Live demo

“We started a catalog-api deploy a few minutes ago and it won't finish. What's going on?”

KubeBolt Kobi

open source · EKS · read-only AWS connector

The October 22 test · one question

What Kobi read, and in what order

Kobi the model Cluster Kubernetes Gateway MCP · KubeBolt ECR tienda/python CloudTrail audit IAM roles WHAT IT FOUND pod events ImagePullBackOff and a 403 on ecr:BatchGetImage: “explicit deny in a resource-based policy” mcp__aws__call ecr GetRepositoryPolicy now on Sonnet Deny for all, except a list of ARNs. The node role is on that list. ecr DescribeImages The tag exists, same digest as revision 1. The image is fine. cloudtrail LookupEvents The policy changed at 11:40:18, via aws-cli. The deploy went out at 11:41:20. iam GetRole The role's real ARN has a path: …:role/tienda/tienda-eks-node answer Fix the ARN in the policy and in the code that generates it. Rollback only as a stopgap. 1 question · 7 tools, 4 of them in AWS · 32 credits ≈ $0.16, the cheapest test in the lab

What it found with GetRole

In IAM, drop the path and it's another ARN

In the repository policy's exception
arn:aws:iam::111122223333:role/tienda-eks-node
The node role, read with GetRole
arn:aws:iam::111122223333:role/tienda/tienda-eks-node

The exception lacks /tienda. To IAM it's another role, so the node role is left out.

Only the new pod fails today. Another pod on a node without the image cached will fail too.

Illustrative account number

More tests in the same lab

Ten incidents on EKS, tested live

On October 21 and 22. On AKS I tested eleven more.

01 · EC2 · RDS

A deleted security group rule

Found who deleted it and when. The exact rule came from CloudTrail.

02 · ELB · EC2

A NACL that locks out the ALB

The cluster was green. Found rule 50 and who added it.

03 · IAM · S3 · KMS

A role without its policy

Proposed the two missing permissions, with the key's ARN.

04 · KMS

A disabled KMS key

Set the previous test's error aside and found the second DisableKey.

05 · Secrets Manager

A rotated secret

Found the right RotateSecret and proposed a rollout restart. It never saw the value.

06 · RDS

A ten-second RDS reboot

I asked once everything was green again. It said who rebooted it and when.

07 · ECR · IAM

An image that won't pull

The ARN without the path. It's the one in the demo.

08 · Route 53

A private CNAME that changes

The record pointed to an instance that doesn't exist.

09 · EKS · ASG

A node that disappears

Found who terminated it, and that both CoreDNS replicas were on it.

10 · SQS · Lambda

A Lambda with concurrency 0

Counted 641 orders in the DLQ and proposed the fix in two steps.

Under the hood · 1

How an agent works

Question from the operator Model Haiku by default Sonnet if it leaves the cluster asks for a tool reads the result KubeBolt get_pod_logs, get_events, get_external_connections, call Cluster Kubernetes Cloud read-only Answer

Kobi decides what it wants to read, but KubeBolt controls what it can read and how it reaches Kobi.

Under the hood · 2 · the view from above

A question's path, from the pod to the account

CLUSTER · EKS KUBEBOLT AWS · THE CUSTOMER'S ACCOUNT orders-api pod readiness: cannot reach database shop-db…:5432 (timed out) Kobi starts on Haiku, moves to Sonnet logs Service shop-db type ExternalName → db.tienda.internal get_external_connections db.tienda.internal route53 · private zone connector available 1 Nodes spec.providerID → 2 instances clusterNetwork i-0…, i-0… · us-east-2 suggested first calls, without listing the account load_skill aws-investigation 2 3 KubeBolt MCP gateway mcp__aws__call a Is it on the list? 16 families, per consumer b Writes the script the model never sends code c Assumes customer role external ID · session policy STS AWS MCP server hosted by AWS · run_script signed with the assumed session Customer role read-only · external ID deny: secrets and data The account ec2 · rds · cloudtrail · route53 d Compacts the response CloudTrail, Lambda, notes 4

The logs name the piece: a timeout to shop-db, the database, through a name that isn't from RDS.

get_external_connections follows the ExternalName to db.tienda.internal. On EKS, a private name is usually a Route 53 private zone, and there is a connector.

The node instances come from spec.providerID. With them Kobi reaches their security groups without listing the account.

Kobi moves up to Sonnet, loads the AWS skill and calls mcp__aws__call with service, operation and parameters.

The gateway assumes the customer role with a session policy. The AWS MCP server runs the read with that session.

The response comes back compacted, with KubeBolt's notes. It's the only thing the model reads.

In amber, the four layers of what Kobi knows about AWS. In blue, the only thing that goes out to the account.

Under the hood · 3 · the amber circles

Where Kobi learns what it knows about AWS

1

The address classification

result of get_external_connections

The service, the resource, the region and the first calls.

2

The connector hint

session prompt

When to leave the cluster. At most five calls per question, each one scoped.

3

The aws-investigation skill

before the first AWS call

Where each symptom's cause lives and how to query CloudTrail.

4

The gateway notes

alongside each response

What KubeBolt cut and why, and how to read the rest.

Haiku sometimes doesn't load a skill, but it always reads a tool result.We saw it in the AKS tests · that's why two of the four layers go inside a result

Code · copilot/connector_skill.go

That's why the skill arrives with the first AWS response


Under the hood · 4 · circle 1

How Kobi knows it has to leave the cluster

Cluster

orders-api → shop-db → db.tienda.internal

The logs name the piece: a timeout to the database, through an ExternalName Service.

Classification

A private name in an EKS cluster

Usually a record in a Route 53 private zone. Connector available. First call: route53 ListHostedZonesByVPC.

And the nodes

EC2 instances read from spec.providerID

To get to their security groups without listing the whole account.

extclassify/aws.go

“Without this the name stayed unknown, with no connector, and Kobi was told to leave it to the operator.”

22AWS services by their address
15Azure resource types

The harness

The AWS MCP server has no read-only mode.

Its tool, run_script, runs whatever Python it's given, with the caller's identity.

That's why Kobi never passes it code.

Code · mcpcatalog/aws_binding.go · step b

Kobi asks for an operation, KubeBolt writes the script

What Kobi asks for: the call tool


    

What KubeBolt sends to run_scripta single line, split here


    

The harness · steps a and c of the diagram

All three filters come from the same table

awsFamilyTable

16 read families

The admin approves them, and everything else comes from them.

KubeBolt · step a

The gateway list

The typed call is rejected before it leaves for AWS. 37 services for Kobi.

STS · step c

The session policy

73 actions in 1,882 of the 2,048 characters STS allows. Even if the role had ReadOnlyAccess, the session can't go beyond that.

IAM · in the customer's account

The role policy

23 explicitly denied actions, and nothing outside the list if it comes through the MCP server (aws:ViaAWSMCPService).

“…are all generated from them, so no two of those can disagree.” · mcpcatalog/aws_policy.go

Code · mcpcatalog/aws_policy.go · step a

Why forbidden operations are checked first

awsDeniedOperations: GetSecretValue · Get*Secret* · GetParameter* · Decrypt · GetObject* · GetItem · Query · Scan · Get*Token* · AssumeRole* …


Code · mcpcatalog/aws_binding.go · step d

KubeBolt also checks the response


    

A different operation

If the script ran something else, the response is discarded.

Lambda variables

They arrive with their name and the value [REDACTED]. In CloudTrail too.

CloudTrail, one row per event

No reads, and credentials replaced.

The harness

What Kobi can't do in your account

KubeBolt, STS and IAM enforce these rules, even if the model asks for something else.

Send code

It only passes service, operation and parameters.

Skip a filter

List, session and role come from the same table.

Exceed 73 actions

It's the session's ceiling, even if the role allows more.

Read secrets or data

23 actions denied in the role and 35 patterns in the gateway.

See credentials in a response

Those from Lambda and CloudTrail arrive replaced.

Use the MCP server for anything else

The role denies anything not on the list if it comes that way.

Use the role from another organization

The external ID is 32 random bytes per organization.

Use the RDS Data API

Nor DynamoDB streams or SSO. They're out, even for reads.

When Kobi gets it wrong

Each failure could be patched with one more line in the prompt.

Do that for every failure and the prompt never stops growing.

What I did was change what the tools return to Kobi.

An example · a role without its policy

The AWS error didn't say it all

not authorized to perform: s3:GetObject on resource: …/featured.json
RoundWhat changedWhat Kobi proposed
1stNothingOnly s3:GetObject. The bucket uses a customer-managed KMS key: the read would still have failed.
2ndThe AWS skill, rewritten to say soOnly s3:GetObject. It loaded it and gave the same answer.
3rdawsDenials in get_pod_logsNames kms:Decrypt. In the follow-up, both permissions with the key's ARN.

The model paid more attention to the error in front of it than to a sentence inside a 9 KB skill.

Code · copilot/log_aws_denials.go

get_pod_logs adds what the error doesn't say


Fixes

Four more failures, fixed in the tools

AccessDenied on Route 53

Before
After an AccessDenied on Route 53: “I don't have access,” and it handed the command to the operator.
Now
A note says which other operations are still enabled. It moved on to ListHostedZonesByVPC and read the CNAME.

A rule deleted by its ID

Before
A rule revoked by its ID left only sgr-…
Now
The exact rule: tcp 5432 from 10.0.64.0/18, created by CloudFormation.

The DLQ that looked empty

Before
“No messages in the DLQ,” with about 404 orders inside.
Now
It reads it: 641 messages. And proposes the fix in two steps.

Cilium, which wasn't installed

Before
Cilium isn't in the cluster, the API answered “forbidden” and Kobi chased a false hypothesis.
Now
The response says it isn't installed.

Tokens

You pay for every line the model reads

1,623→5

log lines in one read, which ended up as 4.4 KB. Lines that only differ in the time show up once, with the number of copies.

185

copies of the same probe line: 35 of the 37 KB in one response, twice in the same question.

Tokens · Azure

A 93 KB Activity Log that didn't fit in a response

ACTIVITY LOG · 90 DAYS · ONE POSTGRESQL DATABASE Raw, top 50 93 KB One row per operation 16 KB Filtered by the parameter 1.7 KB 32 KB: what fits in a response

Raw, the write that set the previous value fell past the cutoff. Filtered, Kobi found the earlier value: “leave it at 0”.

Code · copilot/log_repeats.go

Grouping repeated lines


Tokens · which model

Starts on Haiku. Moves to Sonnet the first time it leaves the cluster.

97of99

Haiku failures came after it left the cluster. It left in 142 of 144 runs, after about 8 calls.

1/15

of Sonnet's cost inside a single cluster, answering just as well.


  

Now it only escalates if what get_external_connections finds points outside the cluster. On AKS, the side that didn't reach Azure went from about 49 credits to 3 or 4.

Tokens · the cache

The follow-up that cost double

1
Question

50 credits

Started on Haiku and ended on Sonnet: eight rounds, fifteen tools.

2
Pause

27 minutes

Restarting the API to load a fix.

3
Cache

Expired after 5 minutes

Sonnet wrote the conversation's 72,000 tokens again.

4
Follow-up

99 credits

Twice as much as the question, with two calls to AWS.

Within a few minutes

The follow-up reads almost everything from cache. Once each message stored its time, it fell from 22 to 8 credits.

Compacting

A 29,600-token conversation came down to 10,300.

At the start of the conversation

50,000 → 35,000

tokens of context before the first question, roughly.

Tokens · what it costs

Between $0.16 and $0.75 per incident

Image that won't pull Disabled KMS key Rotated secret Role without its policy RDS reboot Node that disappears Private CNAME that changes Lambda with concurrency 0 NACL in front of the ALB Security group rule 32 · $0.16 50 · $0.25 55 · $0.28 34 + 22 · $0.28 56 · $0.28 57 · $0.29 69 + 19 · $0.44 67 + 39 · $0.53 52 + 63 · $0.58 50 + 99 · $0.75 QUESTION FOLLOW-UP FOLLOW-UP WITH AN EXPIRED CACHE

Credits for each scenario's passing run, EKS lab, Oct 21 and 22 · 100 credits ≈ $0.50

Where it fails

Where Kobi got it wrong

A rollback with admin credentials in progress

AKS · Oct 12

It offered to go back to a revision that read credentials from another Secret: the database admin's.

Today it rules that out. The history still needs to warn when two revisions read credentials from different places.

A fix that turned off DNS mitigated

AKS · Oct 20

It proposed a CoreDNS block without the fallthrough keyword. Pasted as is, the front end stopped resolving anything.

Now, to change an object it hasn't read, it says what changes and asks to read it first.

It pointed at the wrong person in progress

AKS · Oct 13

A permission revoked through SQL leaves no record. It named someone who had nothing to do with it as the “most likely contact.”

Next fix: with no record of the change, don't offer the author of another.

Checking before proposing open

EKS · Oct 22

Sometimes it skips a read it's allowed to do: the bucket encryption, the AWSCURRENT version.

It says so and leaves it as an option. I don't have the fix yet.

In all of these cases, a person had to approve the change.

How I build it

How I test Kobi

Break Ask Evaluate Fix the data Recreate the prompt barely changes
11 + 10incidents reproduced on AKS and EKS
16revisions of the Azure note in ten days; 5 of the AWS one in a single day

I build it almost on my own, with AI agents that write code with me. Another agent checks each of Kobi's answers against what actually happened.

Takeaways

If you're building one

Put the limits outside the model

In the gateway, the session and the role. And generate all three from the same source.

When it fails, fix what it reads

Add the missing data to the tool result. Leave the prompt as it is.

Measure how much it reads

Bytes and tokens per tool, as well as whether the answer was right.

Test it with real failures

Break a test cluster and save each answer with what it cost.

Thank you

Questions.

KubeBolt is open source, under the Apache 2.0 license. Everything you saw today is in the repository.

github.com/clm-cloud-solutions/kubebolt linkedin.com/in/leafarmaina
KubeBolt
KubeBolt
← → navigate · I index · F full screen · ? help
Notes
00:00
Notes
Next