The AWS connector shipped on October 27 in KubeBolt 2.5, the open-source edition. Before releasing it, we tested it with ten incidents in an EKS lab: a network rule removed, a KMS key disabled, a Lambda that stops consuming. In each one, Kobi got the question an operator would ask, with no hints.
The most useful thing those tests left is where the fix goes when Kobi gets it wrong: the data goes where the model looks. It was almost never one more sentence in the prompt. It was a piece of data placed next to what Kobi was already reading.
What’s underneath, in five lines:
- Kobi writes no code against your account. It picks an operation from a list, and KubeBolt makes the call with a session that can do no more than that list.
- Ten incidents, all tested live. Five landed on the first question, and the other five needed fixes between one run and the next.
- When Kobi got it wrong, rewriting the skill didn’t move it. Putting the data next to the error did.
- The cheapest test cost 32 credits, and most cost between 50 and 100.
- The connector is in KubeBolt 2.5 open source. Not in KubeBolt Cloud yet.
The route of a question
The lab’s case: orders-api can no longer reach its database, an RDS PostgreSQL the cluster reaches through a private name. This is how the question travels from the pod to the account:
- AThe address classification
- BThe connector hint and the dependencies guide
- CThe aws-investigation skill
- DThe gateway's notes
- Backend logs.
get_pod_logsgives the timeout toshop-db…:5432, with its start and end. - The address. The ExternalName Service leads to
db.tienda.internal, a private name. - The classification.
get_external_connectionsrecognizes it as a Route 53 private zone, with the connector available, and adds the nodes' instances. - The typed call. Kobi asks
mcp__aws__callfor a service, an operation and parameters. Never code. - The script. The gateway checks the operation against the list, writes the script and assumes the customer's role with its external ID and a session policy.
- The read. The AWS MCP Server runs the script in the account, with the assumed session.
- The answer. Compacted by the gateway, with no secret values: cause, who and when.
As in Azure, Kobi has no skill per service. What it knows about AWS arrives in four layers, and two of them travel inside a tool result, right when Kobi needs them.
Kobi writes no code against your account
The AWS MCP Server has no read-only mode: its tool runs Python with the caller’s identity. So the guards live in KubeBolt and in the role, and repeat on purpose:
- lines of model code that reach the account
- 0
- read families, approved by the admin
- 16
- AWS calls per turn at most; the prompt asks for five
- 12
- The model writes no codeKobi passes a service, an operation and parameters to a typed tool, and KubeBolt writes the script the AWS server runs.
- One table, three filtersThe gateway's allowlist, the session policy and the role's policy come from the same table of families, so they can't contradict each other.
- The session can do no more than the listEvery call assumes the role with a session policy. Even if the customer's role had ReadOnlyAccess, the session stops at the list.
- No secrets, no dataThe role denies Secrets Manager and Parameter Store values, KMS decryption, S3 objects, DynamoDB items and a function's code.
- What comes back, cleanA Lambda's environment variables arrive with their name and the value covered, in CloudTrail too, like the credentials in a response.
- Nothing else through the MCP serverThe role's policy denies any action off the list when the call comes through the AWS MCP Server (aws:ViaAWSMCPService).
- An external ID nobody picksKubeBolt generates one per organization, and the admin only copies it into the role's trust policy.
- Whole services outThe RDS Data API, DynamoDB Streams and SSO allow no operation at all, not even reads.
Ten incidents
Each incident was broken the way it would really happen, by the lab’s user with aws-cli, and asked about with no hints. A run is one question. When it took more than one, something in KubeBolt changed between runs.
- 01 1 + follow-up
The database's rule
The RDS security group rule that lets the pods in is removed.
Cause, author and time in one question; the exact rule, in the follow-up.
Reads ec2 · rds · cloudtrail50 + 99 credits
- 02 5 runs
Ingress from the internet
A network ACL rule shuts the ALB out, and inside the cluster everything stays green.
With the fixes, cause, author and time in one question and a follow-up, also on a freshly built lab.
Reads elbv2 · ec2 · cloudtrail97 and 115 credits
- 03 3 runs
The role without its policy
catalog-api loses the policy to read its bucket, encrypted with its own KMS key.
The key named in the answer and checked in the follow-up, with both permissions.
Reads iam · s3 · kms34 + 22 credits
- 04 2 runs
The disabled key
S3 returns KMS.DisabledException, with an earlier incident still in the logs.
The second of three changes to the key, kept apart from the earlier incident.
Reads kms · cloudtrail50 credits
- 05 1 question
The rotated password
Secrets Manager rotates the password and the pods keep the old one.
The right RotateSecret and the rollout restart in one question, never seeing the value.
Reads secretsmanager · cloudtrail55 credits
- 06 1 question
A ten-second restart
The RDS instance restarts and the shop fails for seconds. Asked about afterwards.
The restart, who and when, with the source of each fact.
Reads rds · cloudtrail56 credits
- 07 1 question
The image that won't pull
The ECR repository policy leaves the nodes' role out.
The policy, its author and the role's ARN written without its path.
Reads ecr · iam · cloudtrail32 credits
- 08 3 runs
The DNS record
The database's private CNAME points to an instance that doesn't exist.
The record in the first answer and the missing instance, in the follow-up.
Reads route53 · ec2 · rds88 credits
- 09 1 question
A node that disappears
A node group instance is terminated, with both CoreDNS replicas on it.
The author, the replacement and the DNS gap, with a proposal to spread CoreDNS.
Reads eks · autoscaling · cloudtrail57 credits
- 10 2 runs
Orders that never confirm
The Lambda that confirms orders is set to concurrency 0. The queue and its DLQ grow.
The DLQ read and the fix in two steps; the author, in the follow-up.
Reads sqs · lambda · cloudtrail67 + 39 credits
The lab has a fence a real account wouldn’t: it denies CloudTrail in us-east-1, where AWS logs IAM and Route 53 changes. That’s why in two tests Kobi couldn’t name the author, and it said so.
Up close: the rotated password
It’s the best example of why Kobi has to leave the cluster. The cause is in AWS, the failure shows in the cluster, and the fix isn’t the obvious one: the Secret already has the new password, but the pods read it at startup.
The data goes where the model looks
The role test shows it best. catalog-api reads from S3 with its own role, and the bucket encrypts with its own KMS key. Someone detaches the role’s policy. The app’s error names the action that failed first, s3:GetObject, and doesn’t say that reading that object also needs kms:Decrypt on the key.
The first try was the usual one: rewrite the AWS skill to say so. Kobi loaded it and gave the same answer. So the data went where the error is:
What Kobi had in front of it, in catalog-api's logs
… is not authorized to perform: s3:GetObject on resource: …/featured.jsonWhat get_pod_logs now puts next to that line
"awsDenials": [{
"action": "s3:GetObject",
"resource": "arn:aws:s3:::…/featured.json",
"alsoGuardedBy": "An S3 object is also guarded by its bucket's policy
(s3 GetBucketPolicy) and by the bucket's default encryption
(s3 GetBucketEncryption). With SSE-KMS under a customer-managed key,
reading an object also needs kms:Decrypt on that key, …"
}]- 1st run Proposes giving back s3:GetObject only. The read would have kept failing, now on the key.
- 2nd run, skill rewritten Loads the skill and gives the same answer. The new sentence changed nothing.
- 3rd run, with awsDenials Names kms:Decrypt in the answer. In the follow-up it reads the key and gives both permissions.
It wasn’t a one-off. Almost every fix in the lab followed the same rule:
| Where Kobi got it wrong | What KubeBolt now puts where it looks |
|---|---|
| One Route 53 denial made it give up on the service | After an AccessDenied, a note: AWS denied an operation, not the service, and these are the others allowed |
| The rule, removed by its id, didn’t show in CloudTrail | CloudTrail also keeps each change’s response, with the full rule |
| It looked up the KMS key by id and missed the change | A note: DisableKey names the key by its ARN |
| It called a DLQ with about 404 orders in it empty | GetQueueAttributes carries a note when the queue has a DLQ |
| It read 185 copies of the same log line | Lines identical but for the time show once, with how many and since when |
| It took an event that added up two episodes as the start of the outage | An event with more than 30 minutes between its first and last sighting carries that span and a note |
None of these is a new rule in the prompt. Every failure could have been patched with one more sentence, and that’s how prompts get bloated. A piece of data doesn’t compete with the error the model has in front of it, because it sits right next to it.
Up close: the disabled key
This test put two of those pieces of data to work at once. Within the same hour there were three error stretches in the logs, two of them from the same key, and three changes to it in CloudTrail. With the same five calls, the second run got further because the answers carried the notes: the one on searching by id, and the time of each denial.
- 1 DescribeKey KeyState: Disabled
- 2 CloudTrail by id only its creation
- 3 CloudTrail by source failed Decrypts
- 4 SimulatePrincipalPolicy false deny, no region
- 5 CloudTrail, writes DisableKey 16:31:43
Cause, author and time in one question, but three of the five calls answered misleadingly.
- 1 DescribeKey KeyState: Disabled
- 2 CloudTrail by id empty, with the note
- 3 CloudTrail by ARN the three changes
- 4 CloudTrail, writes repeats what it had
- 5 The role, in us-east-1 AccessDenied, the fence
The three changes and, as the cause, the second DisableKey. The last two calls were spare.
When an agent gets it wrong, telling it more rarely fixes it. Putting the missing data where it’s already looking does.
What still goes wrong
The list of gaps from the lab has more entries than I’m telling here. Two of the ones that worry me least:
- Checking before proposing. When the cause looks clear, Kobi sometimes skips a read it’s allowed and has calls to spare for, like confirming a secret’s current version before proposing a restart. It says so and offers it as an option, but doesn’t check.
- The time zone when asking CloudTrail. With times in two different zones, Kobi asked CloudTrail for a shifted window and missed the change. It happens mostly in development, with the API on local time.
What it isn’t yet
The connector is in KubeBolt 2.5, the open-source edition. Today KubeBolt assumes the customer’s role from the identity it runs with, which works for self-hosted installs. It isn’t in KubeBolt Cloud yet.
What’s left
Ten incidents left a short rule: the data goes where the model looks. That’s what makes a read-only connector worth having. Kobi reads your account without writing a line of code, and what it reads arrives with what it needs not to get it wrong.