KubeBolt docs
ENES GitHub

Configuration

Metrics

Three metric sources — Metrics Server for live data, the agent for history and flows, or your existing Prometheus (including managed offerings) — and the VictoriaMetrics store that keeps the history.

On this page
  1. Metrics Server (baseline)
  2. Agent-shipped history
  3. Metrics store
  4. Prometheus remote_write receiver
  5. Querying

KubeBolt has three metric sources. They compose: every cluster starts with Metrics Server, and the agent adds history when you want it.

SourceSetupWhat you get
Metrics ServerNone (pre-installed on most distros)Live CPU/memory for pods and nodes
Agent — collectors / scrapeInstall the agentHistorical metrics in KubeBolt’s VictoriaMetrics: kubelet and node signals, optional vmagent scrape sidecar for Prometheus targets, optional Hubble network flows
Agent — read your PrometheusAgent with promReadSame history, sourced from the Prometheus you already run — including Amazon Managed Prometheus, Google Managed Prometheus, and Azure Monitor

Metrics Server (baseline)

The Metrics Collector polls metrics.k8s.io/v1beta1 (PodMetrics and NodeMetrics) every 30 seconds (the API’s --metric-interval flag). Results are kept in an in-memory cache, so this source has no history of its own.

Graceful degradation: if Metrics Server is not installed, KubeBolt shows all resource state and events normally; the usage bars read “Metrics Server not detected”. The Collector tells “not installed” apart from “403 Forbidden”.

Namespace-scoped fallback: when cluster-wide metrics access is denied, the Collector polls per accessible namespace instead, and node metrics are skipped.

Agent-shipped history

The agent ships samples over gRPC to the KubeBolt API, which writes them into the metrics store. That history powers the Capacity and Reliability tabs, workload metric history, and the Cost tab:

scrape and promRead are mutually exclusive — the chart fails at helm template if both are enabled. If a node-exporter source already ships load/PSI series, set agent.deferNodeStress=true to avoid duplicate series.

Metrics store

History lives in VictoriaMetrics. The Helm chart bundles a single-node one as a StatefulSet; you can point KubeBolt at one you already run instead.

Chart valueDefaultWhat it does
metrics.storage.embedded.enabledtrueDeploy the bundled VictoriaMetrics
metrics.storage.embedded.retention30dHow long samples are kept. Raise persistence.size with it
metrics.storage.embedded.persistence.size10GiIts volume. persistence.enabled and storageClass sit next to it
metrics.storage.embedded.resources100m/256Mi requested, 1000m/2Gi limitThe VictoriaMetrics pod’s requests and limits
metrics.storage.embedded.extraArgs[]Appended to the chart’s own flags (--dedup.minScrapeInterval=30s, --maxLabelsPerTimeseries=60); a flag you repeat wins
metrics.storage.externalUrl—Your VictoriaMetrics, used when embedded.enabled=false
metrics.storage.existingSecret—Secret with username and password and/or token for that store

An external store has to speak VictoriaMetrics’ API: KubeBolt writes with /api/v1/import/prometheus and /api/v1/write and queries it back, so a plain Prometheus is not enough. Start it with -maxLabelsPerTimeseries=60.

A restart of the store leaves no gap in agent metrics. Since 2.6.6 the API holds a batch the store refused for a passing reason (connection error, timeout, 5xx, 429) and writes it again once the store answers, oldest first and with the samples’ own timestamps. Up to 64 MiB is held, about four minutes of a busy install; past that, the oldest batches are dropped. Any other 4xx is not retried. It works with any 1.x agent.

Outside Helm, KUBEBOLT_METRICS_STORAGE_URL (default http://localhost:8428) and its _TOKEN / _USERNAME / _PASSWORD siblings point the API at the store; Docker Compose bundles one, with retention set by KUBEBOLT_METRICS_RETENTION (default 30d). With no store set and none answering at the default URL, the API logs that history is off: live CPU and memory still come from Metrics Server, everything historical stays empty. See Environment variables.

Prometheus remote_write receiver

KubeBolt can also accept Prometheus remote_write at POST /api/v1/prom/write — from the agent’s scrape sidecar or from a Prometheus you run. The receiver is off by default: turn it on with KUBEBOLT_REMOTE_WRITE_ENABLED=true (Helm: metrics.remoteWrite.enabled) or in Administration → Agents & Ingest → Configuration. Its authentication is KUBEBOLT_REMOTE_WRITE_AUTH_MODE: disabled (the default — no bearer token checked), permissive (a token is checked and a bad or missing one is logged, but the request still passes) or enforced (a valid ingest token is required; anything else gets 401). Use enforced for anything reachable from outside the cluster. With the agent on auth.mode=ingest-token, vmagent sends the same token, so one token covers both paths.

What gets through:

Querying

Historical metrics are exposed through the API (/api/v1/metrics/query and /api/v1/metrics/query_range) using PromQL, scoped to the selected cluster.