← Back to the blog
The list of 5,632 ReplicaSets: up to 2.6.3 it arrived as one 33.5 MiB response; since 2.6.4, as 12 pages of 500

KubeBolt 2.7: how we read a large cluster without loading its API server

KubeBolt keeps a live copy of every cluster so that moving around the portal asks nothing of your API server. 2.7 brings it in pages, stops re-reading it and shows you how much is left, on a cluster with 5,632 ReplicaSets too.

When you open a cluster in KubeBolt, the pod list, the map and the insights don’t ask your API server anything. They read from a copy of the cluster that KubeBolt keeps in memory and that keeps itself up to date. That’s why the portal answers instantly, and why spending an hour looking at a cluster adds almost nothing to its load.

That copy has a price: it has to be brought in whole on connect. On a small cluster that takes a few seconds. A few weeks ago, a customer connected a development cluster with 140 namespaces and 5,632 ReplicaSets, and the first connect couldn’t finish. The fixes shipped across four releases, from 2.6.4 to 2.7. Here’s how the load works and what changed. If you only want to know what shipped:

  • The initial load arrives in pages of 500 objects, through the agent too.
  • KubeBolt no longer downloads whole lists every few minutes: the copy check counts without downloading, and Helm’s history stays in the cluster until a view asks for it.
  • While a cluster loads, you see the steps, what has arrived and how much is left.
  • Agent 1.4.10 gets more memory by default, and when a response doesn’t fit it answers with an error instead of restarting.
  • When a cluster opens dozens of insights at once, you get one summary instead of one alert each.

Why KubeBolt loads the cluster on connect

There are two ways to show a cluster in an interface. One is to ask the API server every time someone opens a view, so that every list, filter and refresh is a request. The other is what Kubernetes’ own controllers do: ask for the full list once, then listen only for changes. KubeBolt does the second, with client-go’s informers.

The full load only happens again the first time, when KubeBolt restarts or when the connection was lost. What has to be live, such as logs, the terminal or a Helm release's detail, is asked of the cluster at that moment.
  1. On connect, KubeBolt checks what its role may read and asks for the full list of every resource type it’s allowed.
  2. From then on it keeps one watch per type, and the API server sends only what changes: a new pod, a scaled Deployment, an event.
  3. The lists, the cluster map and the insights, evaluated every 60 seconds, read from that copy. Moving around the lists and the map sends no requests to your cluster.

The full load only happens again in three cases: the first time, when KubeBolt restarts and when the connection was lost. The rest of the time, what travels is the changes.

The expensive part is bringing the copy in once. From then on, what travels is the changes.

What a large cluster’s load weighed

Our customer’s cluster connected through the agent, which opens an outbound channel to KubeBolt so nothing in the cluster has to be exposed. Every response from the API server crosses that channel, and the ReplicaSet list crossed it in one go: 33.5 MiB in a single response.

The cause is a Kubernetes detail. Informers ask for the list in pages of 500, but with resourceVersion=0, which lets the API server answer from its cache. From the cache, the API server ignores the page size and returns the whole collection. The agent had to hold that response in memory, twice while it relayed it, under the chart’s 128 Mi limit at the time. Kubernetes killed it for lack of memory halfway through the connect.

The customer measured the 33.5 MiB on their own cluster. The memory is from our lab, with a cluster of the same size and the chart's default limit at the time.
  1. Since 2.6.4, the initial load asks for its pages without resourceVersion=0, and the API server honours them. The 5,632 ReplicaSets arrive in 12 pages of 500, and each page is still served from the API server’s cache, without reading all of etcd.
  2. In our lab, with a cluster of the same size, the agent’s memory during the connect went from 113–125 MiB down to about 64 MiB.
  3. Watches don’t change: changes still arrive one by one.

What KubeBolt re-read with nobody looking

With the cluster connected, we measured everything that crossed the agent’s channel, minute by minute. Two periodic reads weighed far more than they should.

The first is the copy check. Every 5 minutes, KubeBolt counts Pods, Deployments and ReplicaSets in the cluster and in its copy, to catch them drifting apart. To count, it asked for a single object and read the total from the remainingItemCount field, but it carried the same resourceVersion=0, so the API server sent back the three lists in full. On a cluster that size, that was about 40 MB every five minutes.

The second is Helm’s history. Helm stores every revision of every release as a Secret holding the whole release: the chart, the values and every rendered manifest. KubeBolt downloaded it on connect and listed it again every minute, for two insights and for the Applications page, though only a release’s detail view needs that history.

ReadBeforeSince 2.6.4 and 2.6.5
Copy check, every 5 minutesThe full lists of Pods, Deployments and ReplicaSetsOne object per type: 2–3 KB
Helm Secrets with the Overview open7,225 KiB a minute11.7 KiB a minute
The agent’s whole channel, at rest7,342 KiB a minute227 KiB a minute
Applications list, once loaded136 ms0.5 ms

The Helm figures are from our lab, with 10 releases and 18 revisions. What the table doesn’t show is where that work landed: each of those reads also went through your cluster’s API server, and the copy check and the insights ran whether anyone had KubeBolt open or not.

Now the Secrets copy leaves Helm’s out, and their lists and counts read metadata only, once a minute. A release’s content is read when something needs it: the latest revision, once, for the Applications list or for an insight about a failed release, and the detail view reads only that release.

While it loads, you see how much is left

Up to 2.6.4, a loading cluster showed “Connecting to cluster” and nothing else. If the connect failed, there was no way to know why or when it would try again. Since 2.6.5, the loading screen shows the whole process.

A generated view with the product's own strings. The numbers are an example, sized like the cluster this post is about.
  1. Four steps: agents connected, access checked (with the kinds the role can’t read, if any), loading the cluster, and the first health check.
  2. A bar that moves with the bytes, because they are what takes time to cross the channel: a ReplicaSet weighs many times what an Event does. When the load is waiting on one large collection, it names it.
  3. What has arrived and how fast, the time elapsed, an estimate of what is left, and a log of what happens.
  4. The cluster’s CPU and memory, which come from the agents’ metrics and don’t wait for the load to finish.

The way a slow connect behaves changed with the screen:

  • The connect timeouts count the time with nothing arriving from the cluster. A connect that keeps receiving pages carries on for up to 10 minutes, and a stuck agent fails at the same point as before.
  • When a connect fails, KubeBolt waits 30 seconds before retrying, then 1, 2 and 4 minutes, and 5 at most, and the screen says when the next attempt is. Before, every request from the portal started a new connect: in the lab, that was 710 MB in 14 minutes without ever connecting.
  • If it can’t list the namespaces, it retries, and if it still can’t, it tells you which kinds are left without access. Before, it quietly narrowed itself to default, kube-system and kube-public.

Agent 1.4.10, with headroom

1.4.10 raises the agent’s default limits to 256 Mi of memory and 500m of CPU. Requests don’t change, so scheduling doesn’t either. A response that doesn’t fit in its memory no longer takes it down: the agent accepts responses up to a quarter of its memory limit, 63 MiB at most, and above that it answers with an error that names the cap, the limit and the value to raise, resources.limits.memory.

If an agent runs out of memory anyway, the OOMKilled and Memory Near Limit insights give you the helm upgrade that doubles it. And since 2.7, the Clusters page flags a cluster whose agent should be upgraded, with the command ready to copy.

One summary instead of hundreds of alerts

A large cluster’s first evaluation opens many insights at once. On that development cluster it was nearly 600, most of them Services with no ready endpoints, which is normal in such an environment, and each one went out as a notification.

Since 2.6.4, when a cluster opens 5 or more insights within 60 seconds, Slack, Discord and instant email get one summary once the cluster has been quiet for 30 seconds. It says how many there are of each severity, names the critical ones and lists every rule with its count and the namespaces most affected. If the burst doesn’t stop, a partial summary arrives every 5 minutes at most. In the lab, a reconnect with 502 insights left four single alerts and one summary per channel.

On screen, a cluster with many namespaces reads better too. Workloads by namespace, on the Overview, shows 12 namespaces, those that need attention first, and leaves the rest behind Show all, which says when a hidden one needs attention. The Namespaces page pages 50 at a time, counts them all and filters by name.

If you run KubeBolt yourself

Upgrade the backend to 2.7.0 and the agent to 1.4.10 in every cluster, with that cluster’s kubectl context:

helm upgrade kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  --version 2.7.0 -n kubebolt --reset-then-reuse-values

helm upgrade kubebolt-agent oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  --version 1.4.10 -n kubebolt-system --reset-then-reuse-values
  • --reset-then-reuse-values needs Helm 3.14 or later. Don’t use --reuse-values: it keeps the previous chart’s defaults, and the agent would stay on 128 Mi.
  • If you raised the connect timeouts by hand to get a large cluster connected, you can set them back to their defaults.
  • If the agent’s namespace has a ResourceQuota on limits, leave room for the new ones.

What changes with size

On a small cluster, you’ll notice almost none of this, apart from a loading screen that lasts a few seconds. On a large one, it decides whether the cluster connects at all. And on every cluster, KubeBolt asks your API server for a good deal less than before.