The AI running in your cluster

Your teams already deploy agents and models on Kubernetes. Hardly anyone can say today which ones run, with what permissions, and why one is slow.

Designed · no date

Which AI agents run, and with what permissions

An inventory of your cluster's agents: their isolation posture, their runs and what they call. How it works

An inventory of the AI agents in your cluster: the declared ones (kagent, agents.x-k8s.io) and the ones nobody declared. For each, the model it uses and the MCP tools it calls, and an isolation posture from 0 to 10 (permissions, egress and human approval) with the evidence behind each control. It keeps their runs after the pod is gone and maps what each one calls. Nothing extra to install.

The agents running in your cluster, declared or not, with their isolation posture from 0 to 10.

Agents75 declared · 2 undeclared
Open agent insights11 1 critical · 5 high
Models in use3+2 direct calls to providers
MCP tool servers6 1 unavailable

AgentSourceModelRuntimePostureStatus

invoice-readerfinance Undeclared api.openai.comdirect runc 3/10 Running

helm-agentkagent kagent · Declarative claude-haikugateway runc 4/10 Not ready

code-reviewerplatform kagent · Sandbox claude-sonnetgateway gVisor 9/10 Ready

critical · helm-agent

ServiceAccount kagent/helm-agent is bound to cluster-admin.

support-rag · kagent · BYO · llama-3.1-8b in-cluster · tools: search-kb, get-order, create-ticket

7/10

Container

  • Isolated runtimeruntimeClassName not set → runc
  • User namespaceshostUsers not set
  • Pod Securitynamespace support · enforce=restricted
  • Hardened containerrunAsNonRoot · drop ALL · seccomp RuntimeDefault

Harness

  • ServiceAccount permissionsRole support/support-rag · get, list configmaps
  • Restricted egressno NetworkPolicy selects app=support-rag
  • Human approvalcreate-ticket requires approval

Coexistence

  • Bounded resourceslimits 500m / 1Gi · PriorityClass agents-low

RunStartedDurationResult

invoice-reader-29384511cronjob · finance 14:00 38 s Completed

code-reviewer-pr-1284sandbox · platform 13:52 2 min 10 s Completed

invoice-reader-29384451exit 1 · last log: 429 from api.openai.com 13:00 12 s Failed

incident-triage-7f2cjob · ops-agents 12:41 51 s Completed

14 runs kept in the last 24 h · their pods are gone

Agents

Models and tools

Backends

incident-triage agent · kagent
support-rag agent · kagent BYO
invoice-reader cronjob · undeclared
agentgateway gateway
vllm-llama inference
kb-mcp MCP server
api.anthropic.com external · 443
postgres-kb statefulset
api.openai.com external · 443
direct · no gateway

Agent → models and tools → backends

  • incident-triageagent · kagent

    • agentgatewaygateway

      • api.anthropic.comexternal · 443

  • support-ragagent · kagent BYO

    • vllm-llamainference

    • kb-mcpMCP server

      • postgres-kbstatefulset

  • invoice-readercronjob · undeclared

    • api.openai.comexternal · 443 · direct · no gateway

  • support-ragvllm-llama · kb-mcp → postgres-kb
  • incident-triageagentgateway → api.anthropic.com
  • invoice-readerapi.openai.com · direct, no gateway

observeddirect to a provider

Would it help you?

See it without instrumenting

Latency, errors and dependencies for every service without touching the code, and the cluster's history beyond the hour Kubernetes keeps.

Designed · no date

Latency and errors per service, no code changes

Every service with its rate, errors, latency and what it depends on, measured with eBPF. How it works

Rate, errors and latency (p50, p95 and p99) for every service and route, measured with eBPF from the node: no SDK, not a line changed. Each service with its dependencies, databases outside the cluster included, and rules that compare before and after every deploy. Kobi reads it when it investigates, and you switch it on from the setup wizard whenever you want. Later, the trace of a slow request, span by span.

Rate, errors and latency for every route, before and after the deploy.

orders-api · 3 replicas · rev 42 deployed 14:02

Requests42.1req/ssteady vs the window before
5xx5.8%was 0.2% before rev 42
p951.42swas 180 ms · p99 2.9 s
Dependencies41 slow · 0 failing
Routesserver side

Routereq/s5xxp95before

POST/api/checkout 6.2 14.1% 1.42 s 180 ms

GET/api/orders/:id 21.4 0.3% 64 ms 61 ms

POST/api/orders 4.7 0.9% 210 ms 190 ms

p95 by routelast hour

/api/checkout/api/orders/api/orders/:id

rev 42 · 14:02 500 ms

13:1013:4014:10

orders-api · outbound calls, measured in its pods

Dependenciesp95
  • orders-db postgresqlSELECT order_items · 18.6/s 1.18 swas 22 ms
  • payments-api http6.2 calls/s · 0.1% errors 140 msp95
  • catalog-api http11.9 calls/s · 0.0% errors 35 msp95
  • cart-cache redis40.3 calls/s · 0.0% errors 2 msp95
dependency slow · warning

orders-db explains 83% of the p95 of POST /api/checkout since 14:02.

Kobi · diagnosis

  1. p95 of POST /api/checkout went from 180 ms to 1.42 s right after rev 42.
  2. The time is in SELECT order_items on orders-db. Pods are healthy: no restarts, CPU at 31% of limit.
  3. Proposed: roll back to rev 41. Needs approval.

Review rollback planOpen in chat

prod-eu-1 · every service, measured from the node

5xx1.9%all routes · last hour
p95412ms2.3× the window before
Throughput186req/ssteady
Coverage34of 384 services without signals
Latency · all routesp50 · p95 · p99
deploy · orders-api rev 42

13:3014:0014:30

p50p95p99

Service5xxp95vs before

orders-api 5.8% 1.42 s ×7.9

shop-web 1.2% 1.61 s ×4.1

payments-api 0.1% 140 ms ×1.0

catalog-api 0.0% 35 ms ×1.0

POST /checkout

1.61 s8 spans · 4 services
  • shop-web POST /checkout
    1.61 s
  • orders-api POST /api/checkout
    1.52 s
  • orders-db SELECT orders
    12 ms
  • orders-db SELECT order_items
    1.18 s
  • payments-api POST /api/payments
    140 ms
  • catalog-api GET /api/catalog/:sku
    35 ms
orders-db · SELECT order_items73% of the trace

measured inorders-api pod, client side

inside orders-dbnot visible from the cluster

query textnot captured (off)

Kobi

The slow span started with rev 42. The previous revision ran this route at 180 ms p95.

Would it help you?

Kobi where you already work

In the incident channel, in your AI assistant and in a space of its own where you decide what it may do alone.

Designed · no date

Kobi in Slack: mention it and it investigates

Kobi investigates, alerts and asks for approval right in your Slack channel. How it works

Type @Kobi in the incident thread and Kobi investigates right there: you see its steps, and the answer arrives with the cause and the evidence. Alerts come with a button to investigate them, and each channel only sees the clusters you link to it. It starts read-only; proposed actions come after, reviewed and run with the role of whoever clicks. Cloud first, open source after.

You mention Kobi and it answers with the cause, the evidence and a proposed action.

# prod-incidentsThis channel is linked to prod-eu-west

MR

Marta Ruiz14:14

@Kobi why is checkout-api restarting?

KobiAPP14:15

5 steps, 38 s

Root cause. checkout-api is OOMKilled because revision 42, rolled out at 14:02, lowered the memory limit from 512Mi to 256Mi. Under normal traffic it peaks at 310Mi.

Evidence

  • 14 restarts since 14:03, all OOMKilled. Revision 41 ran 9 days at 512Mi with none.

Proposed actionMedium risk · reversible

Roll back checkout-api to revision 41

rollback_deployment · payments/checkout-api

Review and runOpen in KubeBolt

Runs with the KubeBolt role of whoever clicks. Editor or admin required.

KobiAPP14:03

Critical · insightfirst seen 14:03

Container OOMKilled

payments/checkout-api · prod-eu-west

Investigate with KobiSilence 1 hOpen in KubeBolt

MR

Marta Ruiz14:04

Investigate with Kobi · 5 steps, 38 s

The alert arrives as a message you can act on, not as a webhook.

Kobi AutopilotAPP14:18

Autopilot · approval neededMedium risk · reversible

Roll back checkout-api to revision 41

rollback_deployment · Deployment/payments/checkout-api

Pods return to revision 41 with the 512Mi memory limit. One rolling restart of 3 replicas.

ApproveRejectView incident

Editors and admins of this cluster's team can decide. Your click is checked against your KubeBolt role.

Kobi AutopilotAPP14:20

Approved by Diego Paredes, admin in KubeBolt. Rolling checkout-api back to revision 41.

  • checkout-api restartingtoday
  • What happened overnight?yesterday
  • Over-provisioned namespacesMonday

Kobi

Kubernetes SRE agent by KubeBolt. Investigates incidents, explains what changed and proposes fixes you approve.

  • What is unhealthy right now?
  • What happened overnight?
  • Where are we over-provisioned?
  • Any critical security findings?

In a direct message with Kobi, outside any channel.

Would it help you?

From diagnosis to change

Plans you can review, fixes that land in your repo, what's left over on the bill and the causes that live in your cloud.

Stage 03 · Business · Q4 2026 – Q1 2027

Rightsizing plans you review and export

From recommendation to change: Kobi builds the plan, you export it, and it checks afterwards that it held. How it works

Kobi turns rightsizing recommendations into a plan ordered by risk, with its diff, its rollback and the reason for whatever it leaves out. The model writes no figures: they come from the engine, on measured data. You export it as commands or as a YAML diff, and afterwards Kobi checks that the cluster ended up the way the plan said.

Steps ordered by risk, with their diff and what's left out. The figures come from the engine, not the model.

Estimated saving$312/ moof $418 detected
Steps31 prerequisite · 1 left out
Max riskMediumstep 3 needs approval
Sources2rightsizing · insights
  1. 1

    Add a PodDisruptionBudget to payments-apiLow risk

    Prerequisite: without it, step 3 could restart all 3 replicas at once.

  2. 2

    Lower requests of checkout-workerLow risk$146 / mo

    Stateless, 7 days of stable history.

    -     cpu: 1000m
    +     cpu: 300m       # p95 210m
    -     memory: 2Gi
    +     memory: 768Mi   # p95 512Mi
  3. 3

    Lower CPU of payments-apiMedium risk · needs approval$166 / mo

    Payments path: CPU only, memory stays.

How is the payments plan going?

Kobiprod-eu-west

Steps 1 and 2 hold in the cluster. Step 3 no longer applies as is: someone changed payments-api's image on Tuesday. I'll re-evaluate it once it has 7 days of metrics.

Plan · payments · v2verified 2 of 3
  • 1 · PodDisruptionBudget payments-apidisruptionsAllowed=1 · 3/3 Ready
  • 2 · checkout-worker requests0 OOMKilled · throttling 1.8% · p95 +2.1% · 48 h
  • !3 · payments-api CPU · expiredspec changed · new image · 2 days of metrics

Saving · estimated vs observed$146 → $139 / mo

Would it help you?

Design views with sample data: they may change before they ship.

Want to know when it lands?

What's next

One note per release, no noise.

See the roadmap by stage Something missing? Tell us