KubeBolt docs
ENES GitHub

Project

Compatibility

Which agent version pairs with which backend, what breaks when they disagree, and which Kubernetes versions and providers are supported.

On this page
  1. Backend and agent
  2. Upgrading
  3. What changed at agent 1.0
  4. Kubernetes versions
  5. Providers
  6. Release notes

KubeBolt and kubebolt-agent ship as independently versioned artifacts — the agent has its own chart, image and release tag. They are coupled through exactly one contract: the metric and label schema the agent emits and the backend’s queries consume.

Current: KubeBolt 2.7.0 (EE 2.7.1-e2), agent 1.4.10.

Backend and agent

KubeBoltAgentSchemaNotes
1.10.x – 2.7.x1.0.x – 1.4.xv1.0, Prometheus-canonicalCurrent. Any 1.x agent from 1.0 pairs with any backend from 1.10. Nothing the agent depends on changed from 2.0 through 2.7 — the same metric and label schema, and agents 1.4.1 through 1.4.10 change no flag or protocol
1.10.x and later0.2.xmismatchThe agent emits the legacy schema; the backend’s queries look for canonical names. Empty dashboards
1.9.x and earlier1.0.xmismatchThe reverse. Empty dashboards. Do not run this combination
1.9.x and earlier0.2.xlegacy v0.xBoth sides pre-canonical. Works, but the newer features (right-sizing P95, network drops, external endpoints with FQDN) only exist on the canonical pair
1.5.x0.1.xearlyPre-3-tier-RBAC. Stable, but no agent-proxy and no Hubble flows

So the rule is short: agent ≥ 1.0 with backend ≥ 1.10, and keep them on the same generation. Agent 1.1 through 1.4 are reliability, cardinality and OpenCost-sourcing releases on the same schema — 1.4.0 widens the default dropNetworkInterfaces list, 1.4.1 is a security patch, 1.4.2 makes the chart install on OpenShift out of the box, 1.4.3 reports its real version, 1.4.4 moves the chart’s vmagent sidecar to v1.154.0, 1.4.5 is a security rebuild, 1.4.6 moves vmagent to v1.155.0, 1.4.7 is a dependency update, 1.4.8 moves vmagent to v1.156.0 and stores each pod’s network once, and 1.4.9 forgets Hubble flow keys idle for 15 minutes (its series no longer only grow) and its node-exporter scrape finds kube-prometheus-stack’s node-exporter by default, and 1.4.10 ships only the OpenCost families KubeBolt reads and raises its default limits to 256Mi / 500m — so an agent upgrade inside the 1.x line is never gated on a backend upgrade. One nuance with 1.4.8: it sends no pod network for hostNetwork pods (their traffic is the node’s); backend 2.6.0 charts such a pod’s network from its node, an older backend shows it empty.

Run agent chart 1.4.10 with agent image 1.4.10, which is the default. From 1.4.2 the image declares a numeric user; on OpenShift, chart 1.4.2 or later with an older image pinned through image.tag fails the runAsNonRoot check.

Since 2.7.0 KubeBolt says when a cluster’s agent is out of date. Agent 1.4.9 reads as update available: 1.4.10 exists and nothing in 1.4.9 needs fixing. Agents below 1.4.9 read as update recommended: below 1.4.8 each pod’s network is stored twice, and below 1.4.9, with Hubble flows on, flow keys are never forgotten. KubeBolt never upgrades an agent itself; the Clusters page gives the reasons and the command (Multi-Cluster).

A mismatch does not crash anything. Both sides start, the agent registers, samples reach the metrics store. The only visible symptom is that the panels for that cluster render empty — which is why it gets misdiagnosed as a data problem. See Troubleshooting.

Detecting a mismatch

When a legacy agent connects to a current backend, the backend logs, once per registration:

{
  "level": "WARN",
  "msg": "agent below minimum version — legacy schema",
  "agent_version": "0.2.2",
  "min_agent_version": "1.0.0",
  "hint": "upgrade kubebolt-agent helm chart to >=1.0.0; v0.x emits the legacy schema and dashboards will render empty"
}

The check is fail-soft: the agent still connects and still ships samples. An empty or unparseable agent_version is silent, because not every client sets it.

Upgrading

Upgrade the backend first, then the agent in every cluster that runs one, each with that cluster’s kubectl context:

# 1. Where each release runs and which version it is on
helm list -A | grep kubebolt

# 2. Keep a copy of the values you set
helm get values kubebolt -n kubebolt -o yaml > kubebolt-values.yaml

# 3. The backend
helm upgrade kubebolt \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  -n kubebolt --reset-then-reuse-values

# 4. The agent, in each cluster that runs one
helm upgrade kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  -n kubebolt-system --reset-then-reuse-values

Each command takes the latest release; add --version X.Y.Z to pick one. Use the namespaces helm list shows: the Add cluster wizard installs the agent in kubebolt-system, the quick start puts it next to the backend in kubebolt. An agent installed from the KubeBolt UI (Administration → Agents & Ingest → Integrations → KubeBolt Agent) is not a Helm release: it runs the latest image unless a tag was chosen at install, so kubectl -n kubebolt-system rollout restart daemonset/kubebolt-agent picks up the new one.

Never --reuse-values across chart versions. It keeps the previous chart’s defaults, so the images the chart pins — VictoriaMetrics, vmagent — stay on the old tag, the one a security release replaced, and a value block added since can fail to render. --reset-then-reuse-values (Helm 3.14+) starts from the new chart’s defaults and lays your values over them; a values file (-f) does the same.

What to expect. The API pod is replaced, not rolled, so the UI is down while the new pod starts; agents reconnect on their own and VictoriaMetrics restarts over the same volume. Users, settings, registered clusters, insight history and Kobi’s conversations live on the chart’s persistent volume (on by default) and survive the upgrade. Once the new version runs, the Update chip leaves the top bar.

Skipping versions. Read the Upgrading section of every release between your version and the target (release notes). In 2.x, these ask something of you:

VersionWhat to check
2.0.0Upgrade the chart, not only the image: the ClusterRole gains the rules the security scanners’ CRDs need
2.2.0An MCP host set up with an Everything API token: replace it with an MCP only one
2.3.0api.resources pinned in your values: the chart’s default memory limit is now 1Gi (request 128Mi), because the API caches every Events object it watches
2.4.0A cluster whose agent stops stays listed, out of contact, until you delete it. Agent records the previous version left “connected” start to expire
2.5.0copilot.maxTokens: 4096 in your values keeps cutting long answers: remove it to use the per-model cap. Behind an egress firewall, Kobi’s cloud connectors need the endpoints the notes list
2.5.2OpenShift: the web runs under its own ServiceAccount (<release>-web), which needs no SCC
2.6.0A VictoriaMetrics you run yourself needs -maxLabelsPerTimeseries=60
2.6.1An AWS connector role created earlier needs ecr:GetLifecyclePolicy and ecr:List* for Kobi’s new ECR reads (the connector drawer prints the updated policy)
2.6.3RBAC you manage yourself (rbac.create=false): grant get, list and watch on externalsecrets, secretstores and clustersecretstores (external-secrets.io), or Kobi cannot read them. KubeBolt Cloud: in Approve & execute, every change Autopilot makes now waits for a person
2.6.6If you pinned the Azure MCP Server sidecar image yourself (mcpConnectors.azure.sidecar.image, or a mirror in a private registry), move it to 4.0.0-beta.6: the API speaks to 4.0.
2.6.7KUBEBOLT_AGENT_PROXY_CONNECT_TIMEOUT or KUBEBOLT_AGENT_PROXY_STUCK_TIMEOUT set below the request timeout + 5 s (35 s with the default 30 s), in your values or in Settings: the API raises it and logs a warning; raise it yourself to silence the warning. The connect deadline now defaults to 60s
2.7.0One source of cost: an agent release with both agent.promRead.cost.enabled and the OpenCost exporter (collectors.exporters.opencost, or opencost.enabled) fails helm upgrade to 1.4.10 with a message naming both; turn one off. The agent’s limits: the new 256Mi / 500m apply unless your values set resources (a full copy of the chart’s values file does); a ResourceQuota on limits in the agent’s namespace needs room for them. Your own scrape.metricRelabelConfigs replaces the chart’s keep regex: add the cost families the chart’s default lists. OpenCost series outside the 13 cost families and OpenCost’s kube_* (the GPU families, OpenCost’s own HTTP metrics, pod_info, …) are no longer stored once the backend is upgraded, whatever the agent’s version: a query of your own that reads them comes back empty

Rolling back. helm history kubebolt -n kubebolt lists the revisions; helm rollback kubebolt <REVISION> -n kubebolt restores the previous chart and images. What the newer version wrote stays on the volumes.

On a fleet, upgrade every agent before declaring the rollout done. A partial rollout is recoverable — finish it and the next refetch repopulates the dashboards — but until then the lagging clusters show empty panels and the backend logs one WARN each on every reconnect. After moving off a pre-1.0 agent, this prints nothing once none is left:

kubectl logs -n kubebolt deployment/kubebolt-api --tail=50 | grep "agent below minimum"

What changed at agent 1.0

Agent 1.0.0 renamed every metric and label to follow the Prometheus convention for Kubernetes — the de-facto schema of cAdvisor, kube-state-metrics, node-exporter and Hubble. There is no dual emission: the agent ships only the canonical names. The highlights:

Future major schema changes bump the agent major version and the backend major version in lockstep. Minor and patch agent releases stay backward-compatible with the current backend minor: additive metrics and labels only, never renames.

Kubernetes versions

VersionSupport
1.24 and newerFull
1.20 – 1.23Works with feature degradation. KubeBolt reads discovery.k8s.io/v1 EndpointSlices and batch/v1 CronJobs, which only exist from 1.21; on 1.20, which serves neither, the service-without-endpoints insight has nothing to read

Check yours:

kubectl version -o json | jq -r .serverVersion.gitVersion

Providers

ProviderMetrics ServerNotes
Amazon EKSNot installed by defaultInstall it (manifest or EKS add-on) — see the EKS guide. Managed Prometheus (AMP) supported, with SigV4 auth via IRSA or Pod Identity
Google GKEPre-installedManaged Prometheus (GMP) supported, with Workload Identity
Azure AKSPre-installedAzure Monitor managed Prometheus supported, with Workload Identity
OpenShiftProvided by the clusterInstalls under the default restricted-v2 SCC with no workarounds from KubeBolt 2.3.0 and agent 1.4.2. Older releases need the workarounds in the OpenShift guide, which also covers the one item still open
k3s / k0sPre-installedWorks out of the box
kubeadmManual installkubectl apply -f the metrics-server release manifest
Docker DesktopManual installSee the Docker Desktop guide
MinikubeAddonminikube addons enable metrics-server
kind / k3dManual installMay need --kubelet-insecure-tls. Note the Falco caveat: the “nodes” share one kernel

The 1.10 release was validated across GKE with Dataplane V2, EKS, AKS, and GKE with Calico OSS.

Metrics Server is recommended, not required. Without it the Overview’s live commitment bars read “no data”; everything else — including historical CPU and memory through the agent — keeps working.

Release notes

Per-version notes are published on GitHub releases, and the agent keeps its own changelog in the same repository. See Changelog.