Cloud Native Guayaquil · CNCF · Thursday, Oct 29, 2026
An EKS cluster · namespace tienda
A deploy that won't finish.
And the image is the same one the working version uses.
We ask Kobi
Open-source KubeBolt, connected to the cluster and its AWS account.
Kobi can read both, but it can't change anything in AWS.
Before the demo · the lab
orders-api reaches its database, an RDS PostgreSQL, through a private Route 53 name. The password is in Secrets Manager and a Lambda rotates it.
catalog-api reads the catalog from an S3 bucket with Pod Identity. The bucket is encrypted with a customer-managed KMS key.
checkout stores each order in DynamoDB and publishes it to an SQS queue. A Lambda reads it from the queue and confirms it. Anything that fails five times ends up in the DLQ.
The images come from ECR, and CloudTrail records every change made in the account, IAM ones included.
KubeBolt reads the cluster and the account. In AWS it uses a role that can only read.
Before the talk I broke one of these pieces. Kobi doesn't know which.
● Live
Live demo
“We started a catalog-api deploy a few minutes ago and it won't finish. What's going on?”
open source · EKS · read-only AWS connector
The October 22 test · one question
What it found with GetRole
The exception lacks /tienda. To IAM it's another role, so the node role is left out.
Only the new pod fails today. Another pod on a node without the image cached will fail too.
Illustrative account number
More tests in the same lab
On October 21 and 22. On AKS I tested eleven more.
Found who deleted it and when. The exact rule came from CloudTrail.
The cluster was green. Found rule 50 and who added it.
Proposed the two missing permissions, with the key's ARN.
Set the previous test's error aside and found the second DisableKey.
Found the right RotateSecret and proposed a rollout restart. It never saw the value.
I asked once everything was green again. It said who rebooted it and when.
The ARN without the path. It's the one in the demo.
The record pointed to an instance that doesn't exist.
Found who terminated it, and that both CoreDNS replicas were on it.
Counted 641 orders in the DLQ and proposed the fix in two steps.
Under the hood · 1
Kobi decides what it wants to read, but KubeBolt controls what it can read and how it reaches Kobi.
Under the hood · 2 · the view from above
The logs name the piece: a timeout to shop-db, the database, through a name that isn't from RDS.
get_external_connections follows the ExternalName to db.tienda.internal. On EKS, a private name is usually a Route 53 private zone, and there is a connector.
The node instances come from spec.providerID. With them Kobi reaches their security groups without listing the account.
Kobi moves up to Sonnet, loads the AWS skill and calls mcp__aws__call with service, operation and parameters.
The gateway assumes the customer role with a session policy. The AWS MCP server runs the read with that session.
The response comes back compacted, with KubeBolt's notes. It's the only thing the model reads.
In amber, the four layers of what Kobi knows about AWS. In blue, the only thing that goes out to the account.
Under the hood · 3 · the amber circles
The service, the resource, the region and the first calls.
When to leave the cluster. At most five calls per question, each one scoped.
Where each symptom's cause lives and how to query CloudTrail.
What KubeBolt cut and why, and how to read the rest.
Haiku sometimes doesn't load a skill, but it always reads a tool result.We saw it in the AKS tests · that's why two of the four layers go inside a result
Code · copilot/connector_skill.go
Under the hood · 4 · circle 1
The logs name the piece: a timeout to the database, through an ExternalName Service.
Usually a record in a Route 53 private zone. Connector available. First call: route53 ListHostedZonesByVPC.
To get to their security groups without listing the whole account.
extclassify/aws.go
“Without this the name stayed unknown, with no connector, and Kobi was told to leave it to the operator.”
The harness
The AWS MCP server has no read-only mode.
Its tool, run_script, runs whatever Python it's given, with the caller's identity.
That's why Kobi never passes it code.
Code · mcpcatalog/aws_binding.go · step b
What Kobi asks for: the call tool
What KubeBolt sends to run_scripta single line, split here
The harness · steps a and c of the diagram
The admin approves them, and everything else comes from them.
KubeBolt · step a
The typed call is rejected before it leaves for AWS. 37 services for Kobi.
STS · step c
73 actions in 1,882 of the 2,048 characters STS allows. Even if the role had ReadOnlyAccess, the session can't go beyond that.
IAM · in the customer's account
23 explicitly denied actions, and nothing outside the list if it comes through the MCP server (aws:ViaAWSMCPService).
“…are all generated from them, so no two of those can disagree.” · mcpcatalog/aws_policy.go
Code · mcpcatalog/aws_policy.go · step a
awsDeniedOperations: GetSecretValue · Get*Secret* · GetParameter* · Decrypt · GetObject* · GetItem · Query · Scan · Get*Token* · AssumeRole* …
Code · mcpcatalog/aws_binding.go · step d
If the script ran something else, the response is discarded.
They arrive with their name and the value [REDACTED]. In CloudTrail too.
No reads, and credentials replaced.
The harness
KubeBolt, STS and IAM enforce these rules, even if the model asks for something else.
It only passes service, operation and parameters.
List, session and role come from the same table.
It's the session's ceiling, even if the role allows more.
23 actions denied in the role and 35 patterns in the gateway.
Those from Lambda and CloudTrail arrive replaced.
The role denies anything not on the list if it comes that way.
The external ID is 32 random bytes per organization.
Nor DynamoDB streams or SSO. They're out, even for reads.
When Kobi gets it wrong
Each failure could be patched with one more line in the prompt.
Do that for every failure and the prompt never stops growing.
What I did was change what the tools return to Kobi.
An example · a role without its policy
| Round | What changed | What Kobi proposed |
|---|---|---|
| 1st | Nothing | Only s3:GetObject. The bucket uses a customer-managed KMS key: the read would still have failed. |
| 2nd | The AWS skill, rewritten to say so | Only s3:GetObject. It loaded it and gave the same answer. |
| 3rd | awsDenials in get_pod_logs | Names kms:Decrypt. In the follow-up, both permissions with the key's ARN. |
The model paid more attention to the error in front of it than to a sentence inside a 9 KB skill.
Code · copilot/log_aws_denials.go
Fixes
Tokens
log lines in one read, which ended up as 4.4 KB. Lines that only differ in the time show up once, with the number of copies.
copies of the same probe line: 35 of the 37 KB in one response, twice in the same question.
Tokens · Azure
Raw, the write that set the previous value fell past the cutoff. Filtered, Kobi found the earlier value: “leave it at 0”.
Code · copilot/log_repeats.go
Tokens · which model
Haiku failures came after it left the cluster. It left in 142 of 144 runs, after about 8 calls.
of Sonnet's cost inside a single cluster, answering just as well.
Now it only escalates if what get_external_connections finds points outside the cluster. On AKS, the side that didn't reach Azure went from about 49 credits to 3 or 4.
Tokens · the cache
Started on Haiku and ended on Sonnet: eight rounds, fifteen tools.
Restarting the API to load a fix.
Sonnet wrote the conversation's 72,000 tokens again.
Twice as much as the question, with two calls to AWS.
The follow-up reads almost everything from cache. Once each message stored its time, it fell from 22 to 8 credits.
A 29,600-token conversation came down to 10,300.
50,000 → 35,000
tokens of context before the first question, roughly.
Tokens · what it costs
Credits for each scenario's passing run, EKS lab, Oct 21 and 22 · 100 credits ≈ $0.50
Where it fails
It offered to go back to a revision that read credentials from another Secret: the database admin's.
Today it rules that out. The history still needs to warn when two revisions read credentials from different places.
It proposed a CoreDNS block without the fallthrough keyword. Pasted as is, the front end stopped resolving anything.
Now, to change an object it hasn't read, it says what changes and asks to read it first.
A permission revoked through SQL leaves no record. It named someone who had nothing to do with it as the “most likely contact.”
Next fix: with no record of the change, don't offer the author of another.
Sometimes it skips a read it's allowed to do: the bucket encryption, the AWSCURRENT version.
It says so and leaves it as an option. I don't have the fix yet.
In all of these cases, a person had to approve the change.
How I build it
I build it almost on my own, with AI agents that write code with me. Another agent checks each of Kobi's answers against what actually happened.
Takeaways
In the gateway, the session and the role. And generate all three from the same source.
Add the missing data to the tool result. Leave the prompt as it is.
Bytes and tokens per tool, as well as whether the answer was right.
Break a test cluster and save each answer with what it cost.
Thank you
Questions.
KubeBolt is open source, under the Apache 2.0 license. Everything you saw today is in the repository.