Configuration
Metrics
Three metric sources — Metrics Server for live data, the agent for history and flows, or your existing Prometheus (including managed offerings) — and the VictoriaMetrics store that keeps the history.
On this page
KubeBolt has three metric sources. They compose: every cluster starts with Metrics Server, and the agent adds history when you want it.
| Source | Setup | What you get |
|---|---|---|
| Metrics Server | None (pre-installed on most distros) | Live CPU/memory for pods and nodes |
| Agent — collectors / scrape | Install the agent | Historical metrics in KubeBolt’s VictoriaMetrics: kubelet and node signals, optional vmagent scrape sidecar for Prometheus targets, optional Hubble network flows |
| Agent — read your Prometheus | Agent with promRead | Same history, sourced from the Prometheus you already run — including Amazon Managed Prometheus, Google Managed Prometheus, and Azure Monitor |
Metrics Server (baseline)
The Metrics Collector polls metrics.k8s.io/v1beta1 (PodMetrics and
NodeMetrics) every 30 seconds (the API’s --metric-interval flag). Results
are kept in an in-memory cache, so this source has no history of its own.
Graceful degradation: if Metrics Server is not installed, KubeBolt shows all resource state and events normally; the usage bars read “Metrics Server not detected”. The Collector tells “not installed” apart from “403 Forbidden”.
Namespace-scoped fallback: when cluster-wide metrics access is denied, the Collector polls per accessible namespace instead, and node metrics are skipped.
Agent-shipped history
The agent ships samples over gRPC to the KubeBolt API, which writes them into the metrics store. That history powers the Capacity and Reliability tabs, workload metric history, and the Cost tab:
- Built-in collectors (always on) — the DaemonSet ships kubelet and cAdvisor stats plus node load and pressure (PSI) signals.
- Hubble flows (
hubble.enabled=true, off by default) — on Cilium clusters, network flows read from the Hubble relay. - Scrape sidecar (
scrape.enabled=true) — a bundledvmagentscrapes in-cluster Prometheus targets: pods annotatedprometheus.io/scrape: "true"by default, plus dedicated node-exporter and kube-state-metrics jobs you can turn on. It has defensive cardinality caps (maxSeriesPerTarget, a total-series cap) so one misbehaving exporter can’t flood the store. vmagent ships over Prometheus remote_write toscrape.remoteWriteUrl— empty means it scrapes but drops the samples — so it needs the backend’s receiver turned on. - promRead (
agent.promRead.enabled=true) — a single-replica Deployment queries your existing Prometheus’squery_rangeAPI for the families the collectors don’t produce (kube-state-metrics, node load/PSI, disk, network errors), plus the OpenCost families withagent.promRead.cost.enabled=true. Auth modes:none,basicAuth,bearer,awsSigV4(AMP via IRSA),gcpIam(GMP via Workload Identity),azureWorkloadIdentity. Defaults: poll every30s,step30s,lookback60s.
scrape and promRead are mutually exclusive — the chart fails at
helm template if both are enabled. If a node-exporter source already
ships load/PSI series, set agent.deferNodeStress=true to avoid duplicate
series.
Metrics store
History lives in VictoriaMetrics. The Helm chart bundles a single-node one as a StatefulSet; you can point KubeBolt at one you already run instead.
| Chart value | Default | What it does |
|---|---|---|
metrics.storage.embedded.enabled | true | Deploy the bundled VictoriaMetrics |
metrics.storage.embedded.retention | 30d | How long samples are kept. Raise persistence.size with it |
metrics.storage.embedded.persistence.size | 10Gi | Its volume. persistence.enabled and storageClass sit next to it |
metrics.storage.embedded.resources | 100m/256Mi requested, 1000m/2Gi limit | The VictoriaMetrics pod’s requests and limits |
metrics.storage.embedded.extraArgs | [] | Appended to the chart’s own flags (--dedup.minScrapeInterval=30s, --maxLabelsPerTimeseries=60); a flag you repeat wins |
metrics.storage.externalUrl | — | Your VictoriaMetrics, used when embedded.enabled=false |
metrics.storage.existingSecret | — | Secret with username and password and/or token for that store |
An external store has to speak VictoriaMetrics’ API: KubeBolt writes with
/api/v1/import/prometheus and /api/v1/write and queries it back, so a plain
Prometheus is not enough. Start it with -maxLabelsPerTimeseries=60.
A restart of the store leaves no gap in agent metrics. Since 2.6.6 the API holds a batch the store refused for a passing reason (connection error, timeout, 5xx, 429) and writes it again once the store answers, oldest first and with the samples’ own timestamps. Up to 64 MiB is held, about four minutes of a busy install; past that, the oldest batches are dropped. Any other 4xx is not retried. It works with any 1.x agent.
Outside Helm, KUBEBOLT_METRICS_STORAGE_URL (default http://localhost:8428)
and its _TOKEN / _USERNAME / _PASSWORD siblings point the API at the
store; Docker Compose bundles one, with retention set by
KUBEBOLT_METRICS_RETENTION (default 30d). With no store set and none
answering at the default URL, the API logs that history is off: live CPU and
memory still come from Metrics Server, everything historical stays empty. See
Environment variables.
Prometheus remote_write receiver
KubeBolt can also accept Prometheus remote_write at POST /api/v1/prom/write
— from the agent’s scrape sidecar or from a Prometheus you run. The receiver is
off by default: turn it on with KUBEBOLT_REMOTE_WRITE_ENABLED=true
(Helm: metrics.remoteWrite.enabled) or in
Administration → Agents & Ingest → Configuration. Its authentication
is KUBEBOLT_REMOTE_WRITE_AUTH_MODE: disabled (the default — no bearer token
checked), permissive (a token is checked and a bad or missing one is logged, but the
request still passes) or enforced (a valid ingest token is required; anything
else gets 401). Use enforced for anything reachable from outside
the cluster. With the agent on auth.mode=ingest-token, vmagent sends the
same token, so one token covers both paths.
What gets through:
- Only KubeBolt’s metric families by default —
kube_*,node_*,container_*,kubelet_*,pod_flow_*,pod_dns_*,hubble_*,kubebolt_*,process_*andup. Other series are dropped unlessKUBEBOLT_PROM_WRITE_ALLOW_CUSTOM_SERIES=true, which the Docker Compose file sets and the Helm chart does not. - Within per-tenant limits — 10,000 samples per second with a 100,000
burst, and 1,000,000 active series, each adjustable through the
KUBEBOLT_PROM_WRITE_DEFAULT_*variables.
Querying
Historical metrics are exposed through the API (/api/v1/metrics/query and
/api/v1/metrics/query_range) using PromQL, scoped to the selected cluster.