Configuration
The Agent
The optional, outbound-only agent that unlocks historical metrics, network flows, cost and remote-cluster access.
On this page
KubeBolt works with just a kubeconfig. The agent is the optional next step: a lightweight component you install in a cluster to unlock historical metrics, Hubble network flows, cost data and, for clusters KubeBolt can’t reach, the tunnel its API calls travel through. It connects outbound only over gRPC — the cluster’s API server is never exposed, which is what makes private, firewalled and egress-restricted clusters workable.
The chart runs up to two workloads, configured in the Add cluster wizard:
- DaemonSet (always) — collects kubelet and cAdvisor stats and node load
and pressure per node, and ships them to KubeBolt’s metrics store. One pod in
the cluster, elected through a Lease, also reads Hubble flows
(
hubble.enabled=true, off by default) and scrapes any extra exporters such as OpenCost. Withrbac.mode=readeroroperatorthe DaemonSet also carries the tunnel that lets KubeBolt reach the cluster’s API. promreadDeployment (whenagent.promRead.enabled=true) — a single replica, added next to the DaemonSet, that reads the metric families the DaemonSet doesn’t produce (kube-state-metrics, node load and pressure, disk and network errors) from a Prometheus you already run, including the managed offerings (AMP, GMP, Azure Monitor) with per-provider auth (basic, bearer, AWS SigV4, GCP IAM, Azure Workload Identity). See Metrics.
The chart
The agent ships as its own OCI chart, generated with the right values by the wizard:
helm install kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
--namespace kubebolt-system --create-namespace \
--set backendUrl=kubebolt-agent-ingest.kubebolt.svc.cluster.local:9090
Key values (see the chart’s values.yaml for the full, commented reference):
| Value | Default | Purpose |
|---|---|---|
backendUrl | — (required) | Where the agent dials home — a gRPC dial target as <host:port>, no scheme. In-cluster this is the KubeBolt chart’s <release>-agent-ingest Service on 9090, not the web Service |
cluster.name / cluster.id | — / the kube-system UID | How the cluster registers and displays in KubeBolt |
rbac.mode | reader | How much the agent’s ServiceAccount may do: metrics, reader or operator (details). reader and operator turn the tunnel on. Run operator only with auth.mode set; the chart doesn’t enforce it |
auth.mode | disabled | Backend authentication: disabled, tokenreview (same cluster as the backend), or ingest-token (issued in the admin UI; required for cross-cluster and Cloud), with the token in auth.ingestToken.existingSecret |
tls.* | off | Transport TLS and optional mTLS (CA bundle, client cert, SNI override) |
hubble.* | off | Network-flow collection from an existing Hubble relay, with its address and TLS |
collectors.* | — | Tune what gets collected: dropNetworkInterfaces, extra exporters |
opencost.* | off | Installs the OpenCost sub-chart and scrapes it. For an OpenCost you already run, list it under collectors.exporters instead (details). Either one together with agent.promRead.cost.enabled fails the render: one source of cost |
agent.promRead.* | off | promread mode — endpoint url, per-provider auth, optional cost.enabled, and deployment.* for that pod’s resources and scheduling |
agent.deferNodeStress / agent.deferNodeNetwork | false | Leave load/PSI, or node network, to a node-exporter you already scrape |
scrape.* | off | Optional vmagent scrape sidecar with relabeling and discovery controls |
serviceAccount.annotations | — | Cloud identity for promread: the IRSA role or the Workload Identity binding |
resources | 10m/64Mi requested, 500m/256Mi limit (100m/128Mi before 1.4.10) | Per DaemonSet pod. Requests stay small so the agent fits on busy nodes. The limits leave room for the API server responses the agent relays: see Memory on a large cluster |
tolerations / nodeSelector / affinity | tolerate every taint | Where the DaemonSet runs. Narrow tolerations to keep it off dedicated pools |
priority.* | off | A PriorityClass so the agent can preempt pods on full nodes |
gomemlimit | 90% of each container’s memory limit | Pin an explicit Go memory target instead |
logLevel | info | debug, info, warn, error |
image.tag | the chart’s version | Pin the agent image |
extraEnv | [] | Agent variables without a value of their own |
Versioning
The agent follows its own release cadence (currently the 1.4.x line) on a stable v1.0 metric/label schema. The schema is the only contract between the two sides, so any 1.x agent ≥ 1.0 pairs with any KubeBolt backend ≥ 1.10 — upgrade either side independently within those bounds. Agent 0.2.x emits the pre-canonical schema and renders empty dashboards against a modern backend; the backend logs a WARN on registration when one connects.
promRead is the one mode with a narrower window: it shipped in agent 1.1.0
alongside backend 1.13.0. Its samples still reach the metrics store on
backends 1.10–1.12, but the Prometheus (read) integration card doesn’t render
there. That is a floor for one mode, not for the agent. Full matrix in
Compatibility.
The agent’s own changelog lists every version on its line, without the backend releases in between: packages/agent/CHANGELOG.md.
Upgrade with --reset-then-reuse-values (or a values file) rather than
--reuse-values, which keeps the old chart’s defaults — the vmagent sidecar’s
image tag included. The steps, in Compatibility → Upgrading.
Since 2.7.0 KubeBolt says when a cluster’s agent is out of date, judged by its
oldest live agent: Clusters, Fleet and Home show it, and Kobi
answers with the same helm upgrade command. KubeBolt never runs that command:
the agent is your Helm release, often under GitOps, and upgrading it from
KubeBolt would take write access to its DaemonSet. See
Multi-Cluster.
Security posture: outbound-only connections (gRPC, plus HTTP
remote_write when the scrape sidecar is on), no inbound ports, runs as
non-root with a read-only root filesystem, and authenticates with a scoped
ingest token in cross-cluster setups. Metrics flow through the backend — the
agent never writes to the metrics store directly.
Memory on a large cluster
For a cluster connected through it (rbac.mode=reader or operator), the
agent relays the API server’s responses and holds each one whole while it
sends it. The backend asks for lists in pages of 500 objects, but a large
cluster’s first connect still moves tens of MB through the agent.
- Defaults since 1.4.10: a
256Mimemory limit and500mof CPU. At128Mithe agent was OOMKilled on a cluster with thousands of ReplicaSets, and at100ma large cluster’s first connect was throttled. ThepromreadDeployment keeps its own limits. - A response too large is an error, not an OOMKill. One relayed body is
capped at a quarter of the container’s memory limit, and never above the
channel’s 63 MiB. Past it, the agent answers with an error that names the
cap, the limit and
resources.limits.memory. - On a cluster with thousands of objects, raise the limit on the agent’s
release:
--set resources.limits.memory=512Mi. When an agent pod is OOMKilled or stays near its limit, the OOMKilled and Memory Near Limit insights give thehelm upgradethat raises it.