Project
Compatibility
Which agent version pairs with which backend, what breaks when they disagree, and which Kubernetes versions and providers are supported.
On this page
KubeBolt and kubebolt-agent ship as independently versioned artifacts —
the agent has its own chart, image and release tag. They are coupled through
exactly one contract: the metric and label schema the agent emits and the
backend’s queries consume.
Current: KubeBolt 2.7.0 (EE 2.7.1-e2), agent 1.4.10.
Backend and agent
| KubeBolt | Agent | Schema | Notes |
|---|---|---|---|
| 1.10.x – 2.7.x | 1.0.x – 1.4.x | v1.0, Prometheus-canonical | Current. Any 1.x agent from 1.0 pairs with any backend from 1.10. Nothing the agent depends on changed from 2.0 through 2.7 — the same metric and label schema, and agents 1.4.1 through 1.4.10 change no flag or protocol |
| 1.10.x and later | 0.2.x | mismatch | The agent emits the legacy schema; the backend’s queries look for canonical names. Empty dashboards |
| 1.9.x and earlier | 1.0.x | mismatch | The reverse. Empty dashboards. Do not run this combination |
| 1.9.x and earlier | 0.2.x | legacy v0.x | Both sides pre-canonical. Works, but the newer features (right-sizing P95, network drops, external endpoints with FQDN) only exist on the canonical pair |
| 1.5.x | 0.1.x | early | Pre-3-tier-RBAC. Stable, but no agent-proxy and no Hubble flows |
So the rule is short: agent ≥ 1.0 with backend ≥ 1.10, and keep them on the
same generation. Agent 1.1 through 1.4 are reliability, cardinality and
OpenCost-sourcing releases on the same schema — 1.4.0 widens the default
dropNetworkInterfaces list, 1.4.1 is a security patch, 1.4.2 makes the
chart install on OpenShift out of the box, 1.4.3 reports its
real version, 1.4.4 moves the chart’s vmagent sidecar to v1.154.0, 1.4.5 is a
security rebuild, 1.4.6 moves vmagent to v1.155.0, 1.4.7 is a dependency
update, 1.4.8 moves vmagent to v1.156.0 and stores each pod’s network once, and
1.4.9 forgets Hubble flow keys idle for 15 minutes (its series no longer only
grow) and its node-exporter scrape finds kube-prometheus-stack’s node-exporter
by default, and 1.4.10 ships only the OpenCost families KubeBolt reads and
raises its default limits to 256Mi / 500m — so an agent upgrade inside the 1.x
line is never gated on a backend upgrade.
One nuance with 1.4.8: it sends no pod network for hostNetwork pods (their
traffic is the node’s); backend 2.6.0 charts such a pod’s network from its
node, an older backend shows it empty.
Run agent chart 1.4.10 with agent image 1.4.10, which is the default. From
1.4.2 the image declares a numeric user; on OpenShift, chart 1.4.2 or later with
an older image pinned through image.tag fails the runAsNonRoot check.
Since 2.7.0 KubeBolt says when a cluster’s agent is out of date. Agent 1.4.9 reads as update available: 1.4.10 exists and nothing in 1.4.9 needs fixing. Agents below 1.4.9 read as update recommended: below 1.4.8 each pod’s network is stored twice, and below 1.4.9, with Hubble flows on, flow keys are never forgotten. KubeBolt never upgrades an agent itself; the Clusters page gives the reasons and the command (Multi-Cluster).
A mismatch does not crash anything. Both sides start, the agent registers, samples reach the metrics store. The only visible symptom is that the panels for that cluster render empty — which is why it gets misdiagnosed as a data problem. See Troubleshooting.
Detecting a mismatch
When a legacy agent connects to a current backend, the backend logs, once per registration:
{
"level": "WARN",
"msg": "agent below minimum version — legacy schema",
"agent_version": "0.2.2",
"min_agent_version": "1.0.0",
"hint": "upgrade kubebolt-agent helm chart to >=1.0.0; v0.x emits the legacy schema and dashboards will render empty"
}
The check is fail-soft: the agent still connects and still ships samples.
An empty or unparseable agent_version is silent, because not every client
sets it.
Upgrading
Upgrade the backend first, then the agent in every cluster that runs one, each with that cluster’s kubectl context:
# 1. Where each release runs and which version it is on
helm list -A | grep kubebolt
# 2. Keep a copy of the values you set
helm get values kubebolt -n kubebolt -o yaml > kubebolt-values.yaml
# 3. The backend
helm upgrade kubebolt \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
-n kubebolt --reset-then-reuse-values
# 4. The agent, in each cluster that runs one
helm upgrade kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
-n kubebolt-system --reset-then-reuse-values
Each command takes the latest release; add --version X.Y.Z to pick one. Use
the namespaces helm list shows: the Add cluster wizard installs the agent in
kubebolt-system, the quick start puts it next to the backend in kubebolt.
An agent installed from the KubeBolt UI (Administration → Agents & Ingest →
Integrations → KubeBolt Agent) is not a Helm release: it runs the latest
image unless a tag was chosen at install, so
kubectl -n kubebolt-system rollout restart daemonset/kubebolt-agent picks up
the new one.
Never --reuse-values across chart versions. It keeps the previous
chart’s defaults, so the images the chart pins — VictoriaMetrics, vmagent —
stay on the old tag, the one a security release replaced, and a value block
added since can fail to render. --reset-then-reuse-values (Helm 3.14+)
starts from the new chart’s defaults and lays your values over them; a values
file (-f) does the same.
What to expect. The API pod is replaced, not rolled, so the UI is down while the new pod starts; agents reconnect on their own and VictoriaMetrics restarts over the same volume. Users, settings, registered clusters, insight history and Kobi’s conversations live on the chart’s persistent volume (on by default) and survive the upgrade. Once the new version runs, the Update chip leaves the top bar.
Skipping versions. Read the Upgrading section of every release between your version and the target (release notes). In 2.x, these ask something of you:
| Version | What to check |
|---|---|
| 2.0.0 | Upgrade the chart, not only the image: the ClusterRole gains the rules the security scanners’ CRDs need |
| 2.2.0 | An MCP host set up with an Everything API token: replace it with an MCP only one |
| 2.3.0 | api.resources pinned in your values: the chart’s default memory limit is now 1Gi (request 128Mi), because the API caches every Events object it watches |
| 2.4.0 | A cluster whose agent stops stays listed, out of contact, until you delete it. Agent records the previous version left “connected” start to expire |
| 2.5.0 | copilot.maxTokens: 4096 in your values keeps cutting long answers: remove it to use the per-model cap. Behind an egress firewall, Kobi’s cloud connectors need the endpoints the notes list |
| 2.5.2 | OpenShift: the web runs under its own ServiceAccount (<release>-web), which needs no SCC |
| 2.6.0 | A VictoriaMetrics you run yourself needs -maxLabelsPerTimeseries=60 |
| 2.6.1 | An AWS connector role created earlier needs ecr:GetLifecyclePolicy and ecr:List* for Kobi’s new ECR reads (the connector drawer prints the updated policy) |
| 2.6.3 | RBAC you manage yourself (rbac.create=false): grant get, list and watch on externalsecrets, secretstores and clustersecretstores (external-secrets.io), or Kobi cannot read them. KubeBolt Cloud: in Approve & execute, every change Autopilot makes now waits for a person |
| 2.6.6 | If you pinned the Azure MCP Server sidecar image yourself (mcpConnectors.azure.sidecar.image, or a mirror in a private registry), move it to 4.0.0-beta.6: the API speaks to 4.0. |
| 2.6.7 | KUBEBOLT_AGENT_PROXY_CONNECT_TIMEOUT or KUBEBOLT_AGENT_PROXY_STUCK_TIMEOUT set below the request timeout + 5 s (35 s with the default 30 s), in your values or in Settings: the API raises it and logs a warning; raise it yourself to silence the warning. The connect deadline now defaults to 60s |
| 2.7.0 | One source of cost: an agent release with both agent.promRead.cost.enabled and the OpenCost exporter (collectors.exporters.opencost, or opencost.enabled) fails helm upgrade to 1.4.10 with a message naming both; turn one off. The agent’s limits: the new 256Mi / 500m apply unless your values set resources (a full copy of the chart’s values file does); a ResourceQuota on limits in the agent’s namespace needs room for them. Your own scrape.metricRelabelConfigs replaces the chart’s keep regex: add the cost families the chart’s default lists. OpenCost series outside the 13 cost families and OpenCost’s kube_* (the GPU families, OpenCost’s own HTTP metrics, pod_info, …) are no longer stored once the backend is upgraded, whatever the agent’s version: a query of your own that reads them comes back empty |
Rolling back. helm history kubebolt -n kubebolt lists the revisions;
helm rollback kubebolt <REVISION> -n kubebolt restores the previous chart and
images. What the newer version wrote stays on the volumes.
On a fleet, upgrade every agent before declaring the rollout done. A partial rollout is recoverable — finish it and the next refetch repopulates the dashboards — but until then the lagging clusters show empty panels and the backend logs one WARN each on every reconnect. After moving off a pre-1.0 agent, this prints nothing once none is left:
kubectl logs -n kubebolt deployment/kubebolt-api --tail=50 | grep "agent below minimum"
What changed at agent 1.0
Agent 1.0.0 renamed every metric and label to follow the Prometheus convention for Kubernetes — the de-facto schema of cAdvisor, kube-state-metrics, node-exporter and Hubble. There is no dual emission: the agent ships only the canonical names. The highlights:
- Pod-scope labels:
pod_namespace/pod_name→namespace/pod - CPU: the derived
*_usage_coresgauges are gone; the UI usesrate(*_seconds_total[Xm]) - Network:
pod_network_*_bytes_total→container_network_*_bytes_total - Memory:
container_memory_rss_bytes→container_memory_rss; page faults collapse intocontainer_memory_failures_totalwith afailure_typelabel - Volumes: PVCs only, as
kubelet_volume_stats_* - Hubble flows:
src_*/dst_*→source_*/destination_*
Future major schema changes bump the agent major version and the backend major version in lockstep. Minor and patch agent releases stay backward-compatible with the current backend minor: additive metrics and labels only, never renames.
Kubernetes versions
| Version | Support |
|---|---|
| 1.24 and newer | Full |
| 1.20 – 1.23 | Works with feature degradation. KubeBolt reads discovery.k8s.io/v1 EndpointSlices and batch/v1 CronJobs, which only exist from 1.21; on 1.20, which serves neither, the service-without-endpoints insight has nothing to read |
Check yours:
kubectl version -o json | jq -r .serverVersion.gitVersion
Providers
| Provider | Metrics Server | Notes |
|---|---|---|
| Amazon EKS | Not installed by default | Install it (manifest or EKS add-on) — see the EKS guide. Managed Prometheus (AMP) supported, with SigV4 auth via IRSA or Pod Identity |
| Google GKE | Pre-installed | Managed Prometheus (GMP) supported, with Workload Identity |
| Azure AKS | Pre-installed | Azure Monitor managed Prometheus supported, with Workload Identity |
| OpenShift | Provided by the cluster | Installs under the default restricted-v2 SCC with no workarounds from KubeBolt 2.3.0 and agent 1.4.2. Older releases need the workarounds in the OpenShift guide, which also covers the one item still open |
| k3s / k0s | Pre-installed | Works out of the box |
| kubeadm | Manual install | kubectl apply -f the metrics-server release manifest |
| Docker Desktop | Manual install | See the Docker Desktop guide |
| Minikube | Addon | minikube addons enable metrics-server |
| kind / k3d | Manual install | May need --kubelet-insecure-tls. Note the Falco caveat: the “nodes” share one kernel |
The 1.10 release was validated across GKE with Dataplane V2, EKS, AKS, and GKE with Calico OSS.
Metrics Server is recommended, not required. Without it the Overview’s live commitment bars read “no data”; everything else — including historical CPU and memory through the agent — keeps working.
Release notes
Per-version notes are published on GitHub releases, and the agent keeps its own changelog in the same repository. See Changelog.