Core Concepts
Cost & Rightsizing (Beta)
Per-cluster cloud spend powered by OpenCost — run-rate, idle cost, and money-priced rightsizing recommendations. How each number is computed, the three ways to connect OpenCost, and how to verify it.
On this page
When OpenCost data flows from a cluster, a Cost sub-tab (marked Beta) appears on its dashboard, and the spend figures on Home and Fleet start counting it. There is no new backend service to operate: the tab is a lens over cost metric families the agent ships into KubeBolt’s metrics store, scoped per cluster like everything else. KubeBolt does not call OpenCost’s API — OpenCost is a Prometheus exporter, and its metrics travel the same path as the rest.
The tab shows only on a cluster whose cost samples have reached KubeBolt (it
checks for node_total_hourly_cost, once a minute). A cluster with no cost
feed gets no empty tab.
What the Cost tab shows
- Card row — four cards: the monthly run-rate (with the hourly rate and the cost per pod in its sentence, and the top three namespaces), idle cost (what you pay for capacity no workload requested), the monthly savings the rightsizing recommendations would recover, and efficiency (how much of the CPU you requested is in use). Idle cost lights amber when 60 % or more of the spend is idle, and names the node where most of it sits.
- Cost trends — over the selected range, in $/day: the spend run-rate split into compute, network and storage, and allocated against total cost (the gap between them is idle).
- Breakdown — cost by namespace (with each namespace’s CPU efficiency), by workload, and by node (spot nodes tagged), so “who is spending this?” is two clicks away.
- Rightsizing recommendations, in money — the over-provisioned recommendations from the same engine as the Capacity tab, each priced in $/month and sorted by what it saves. Near-limit and no-specs recommendations hand no money back, so they stay on Capacity. Each row links to the workload; nothing is applied from here.
Kobi reads the same engine through get_right_sizing, so asking it in the chat
and reading the Capacity or Cost screen give the same numbers. The
recommendations are not recomputed for the conversation.
How the numbers are computed
Every figure is a query over OpenCost’s metrics for the selected cluster.
| Figure | Computed as |
|---|---|
| Monthly run-rate | sum(node_total_hourly_cost) × 730 — the current hourly cost projected to a month, not month-to-date |
| Allocated | Each container’s CPU allocation (container_cpu_allocation) × its node’s node_cpu_hourly_cost, plus its memory allocation in GiB (container_memory_allocation_bytes) × the node’s node_ram_hourly_cost |
| Idle | Run-rate minus allocated. It includes node system overhead and the headroom you keep on purpose, so not all of it is reclaimable |
| Network | Load-balancer and egress cost families (kubecost_load_balancer_cost, kubecost_network_*_egress_cost), shown apart from compute |
| Storage | pv_hourly_cost; $0 where OpenCost has no disk price |
| Rightsizing savings | Reclaimable cores × the cluster’s average CPU hourly rate, plus reclaimable GiB × the average RAM rate, × 730 |
| Efficiency | CPU in use ÷ CPU requested, across all namespaces |
Prices are the list prices OpenCost resolves for your provider and region, not your invoice: committed-use discounts, reservations and negotiated rates are not in them. Expect a steady gap against billing; the value is the relative picture. On-prem, bare metal and kind there is no price list to detect, and everything reads $0 until you give OpenCost a custom one.
How rightsizing decides
Rightsizing is three fixed rules, not a model, so the same data always gives the same answer. They apply to CPU and memory separately, over the P95 of the last 7 days of each Deployment (its ReplicaSets rolled up), StatefulSet or DaemonSet, sampled at 5-minute resolution:
| State | When it fires | What it suggests | Severity |
|---|---|---|---|
| Near limit | The P95 reaches 80 % of the limit and sits within 50m CPU or 100Mi memory of it | A limit of P95 × 1.5 | critical |
| Over-provisioned | The P95 is under half the request, and the gap is over 50m CPU or 100Mi memory | A request of P95 × 1.2 | warning |
| No specs | No request and no limit, and the P95 is above 50m CPU or 100Mi memory | Setting them | info |
The absolute floors keep near-idle controllers quiet, where a percentage means nothing. Suggestions round to 10m and 10Mi. The result is recomputed at most every 5 minutes per cluster; every screen and Kobi read the same answer.
The P95 leaves out the busiest 5 % of the week, about 8 hours, so a short spike
doesn’t move it. Use it for requests. For a memory limit, check the max on the
workload’s Monitor tab too (Monitor charts on long ranges).
When recommendations appear
Rightsizing waits for 2 days of usage history by default. Below that the P95 has not seen two daily peaks and a quiet stretch reads as the steady state, so there are no recommendations at all:
- The Rightsizing opportunity card on Capacity and the savings card on Cost
show a dash,
collecting samples — needs 2 days, and how much history there is so far, e.g.14h of 2 days collected. - The recommendation lists read
Collecting samples — right-sizing needs 2 days of usage history. - Kobi’s
get_right_sizingsays it is still collecting and suggests no resize.
From then on the recommendations are computed over the history there is; the
Capacity card’s caption says how much (P95 over 2d). From 7 days the P95
covers the full week.
Self-hosted, an admin can change the minimum under Administration → System → General → Right-sizing → Minimum usage history: 12 to 168 hours, applied to every cluster and to Kobi with no restart. Lower it for a trial so a new cluster shows recommendations on its first day; below 48 hours they are marked as an early estimate. On Cloud we set the minimum; it is not an organization setting.
Three cases worth knowing:
- A workload that has never been sampled gets no recommendation, even once the rest of the cluster has history. A cluster without the metrics agent never gets any.
- A workload that really uses nothing still reads as over-provisioned: a measured zero is data.
- If the metrics store doesn’t answer, the cards read
couldn’t read usage history — tries again in 5 mininstead of claiming the cluster is well sized.
Requirements
- The KubeBolt agent, chart 1.3.0 or newer (OpenCost sourcing shipped in that line). The agent is what ships the cost samples, in all three modes, and rightsizing reads the usage samples it ships. Agent 1.4.10 ships every cost family on every path, with the right labels: see What each path ships.
- OpenCost metrics reaching KubeBolt via one of the three modes below.
Three ways to connect OpenCost
Pick one under Cost monitoring (OpenCost) in the Add cluster wizard —
it renders the final Helm values. The flags below go on the agent’s
helm upgrade; on an existing release, add --reset-then-reuse-values to keep
its backend URL, auth and cluster settings. See
Connect clusters.
1. Install OpenCost for me (bundled) — the agent chart installs the
official OpenCost sub-chart and scrapes it at
http://<release>-opencost.<namespace>.svc:9003/metrics with no further
setting. Simplest path if you don’t run OpenCost today. OpenCost itself reads
usage from a Prometheus to attribute cost to workloads, so point the sub-chart
at yours — the wizard asks for the URL and fills these in:
--set opencost.enabled=true \
--set opencost.opencost.prometheus.internal.serviceName=<prometheus-service> \
--set opencost.opencost.prometheus.internal.namespaceName=<prometheus-namespace> \
--set opencost.opencost.prometheus.internal.port=9090
For a Prometheus outside the cluster, use
opencost.opencost.prometheus.internal.enabled=false,
opencost.opencost.prometheus.external.enabled=true and
opencost.opencost.prometheus.external.url=<url> instead. If the sub-chart’s
Service name differs (a fullnameOverride), set opencost.exporterUrl.
Without a reachable Prometheus the sub-chart defaults to
prometheus-server.prometheus-system and OpenCost fails to start where that
Service doesn’t exist. If it reaches a Prometheus without the usage history it
needs, node pricing still flows but the per-namespace and per-workload
breakdown is partial: a total that looks right over a breakdown that does not
add up to it.
2. Scrape my OpenCost — you already run OpenCost; the agent scrapes its exporter. List it under the agent’s exporters:
--set-string collectors.exporters.opencost=http://opencost.opencost.svc.cluster.local:9003/metrics
The scrape is leader-elected: one agent pod in the cluster does it, so the DaemonSet does not multiply the same samples by the number of nodes.
3. Read from my Prometheus — your existing Prometheus already scrapes
OpenCost; the agent reads the cost metric families through the same
promRead connection used for metrics:
--set agent.promRead.enabled=true \
--set agent.promRead.cost.enabled=true \
--set agent.promRead.url=http://prometheus.monitoring:9090
promRead is mutually exclusive with the scrape sidecar, so the wizard switches
the cluster’s metrics source to it: one Prometheus feeds metrics and cost. In
this mode the agent asks for the cost families listed below, appended to the
core KubeBolt metrics and never replacing them. Before agent 1.4.10 it asked
for eight of them, so volume claims, spot nodes, load balancers and network
egress never arrived this way.
A Prometheus that scrapes OpenCost without honor_labels (a kube-prometheus
ServiceMonitor’s default) puts the OpenCost pod’s namespace, pod,
container and instance on every cost series and renames OpenCost’s own to
exported_*. Read as it came, every container’s cost landed in the namespace
OpenCost runs in. Since agent 1.4.10 promRead gives cost families their
exported_* labels back; the cluster and tenant labels it stamps are never
replaced.
One source of cost. With the exporter (modes 1 and 2) and
agent.promRead.cost.enabled both on, every cost family was stored twice and
every total read double. Since agent 1.4.10 the chart refuses that
combination: helm upgrade fails with a message naming both. Pick one:
promRead.cost reads OpenCost from your Prometheus, the exporter reads
OpenCost directly.
What each path ships
One list of 13 cost families is what every path ships, asks for and keeps:
| What it prices | Families |
|---|---|
| Nodes | node_total_hourly_cost, node_cpu_hourly_cost, node_ram_hourly_cost |
| Containers | container_cpu_allocation, container_memory_allocation_bytes |
| Volumes | pv_hourly_cost, pod_pvc_allocation |
| Spot nodes | kubecost_node_is_spot |
| Network | kubecost_load_balancer_cost, kubecost_network_internet_egress_cost, kubecost_network_nat_gateway_egress_cost, kubecost_network_region_egress_cost, kubecost_network_zone_egress_cost |
- The exporter (modes 1 and 2) forwards these and OpenCost’s
kube_*families, nothing else, since agent 1.4.10. From backend 2.7.0 the rest (the GPU families, OpenCost’s own HTTP metrics,pod_info, …) is dropped on arrival from older agents too, counted askubebolt_agent_grpc_dropped_series_total{reason="opencost_unread"}. On a test cluster OpenCost’s series went from 1,213 to about 500. - promRead (mode 3) asks for these 13.
- The scrape sidecar’s default keep rule keeps them since 1.4.10. Your own
scrape.metricRelabelConfigsreplaces that rule: add the cost families to it, or volume, spot, load-balancer and egress cost are dropped.
The GPU families stay out until a KubeBolt view reads them: on a node without a GPU they are a series per container that always reads zero. On KubeBolt Cloud the 13 families count as core series on every plan.
Verify
# 1. OpenCost is exporting
# bundled: the Service is <release>-opencost in the agent's namespace
kubectl port-forward -n kubebolt svc/kubebolt-agent-opencost 9003:9003
# your own: adjust namespace and Service, e.g. -n opencost svc/opencost
curl -s localhost:9003/metrics | grep node_total_hourly_cost
# 2. The agent is scraping it (the leader pod only)
kubectl logs -n kubebolt -l app.kubernetes.io/name=kubebolt-agent --tail=50 | grep -i exporter
Then the Cost tab appears on the cluster dashboard within a couple of minutes. The OpenCost card under Administration → Agents & Ingest → Integrations reports the same state, for the cluster selected in the top bar — it separates running from feeding by checking whether this cluster’s cost samples actually reached storage:
| Symptom | Cause |
|---|---|
| Card “installed”, with “no cost samples reaching KubeBolt yet” | OpenCost runs, but nothing ships its metrics — the exporter is not wired on the agent |
| Card “degraded” — “OpenCost pods not fully ready” | Often the bundled sub-chart crash-looping because it has no reachable Prometheus |
| Card “not installed” | No pods labelled app.kubernetes.io/name=opencost, and no pod named opencost* in the opencost, kubecost or monitoring namespaces |
| Total looks right, breakdown is empty or partial | OpenCost’s Prometheus lacks the usage history for allocation |
| Everything at $0 | OpenCost has no price list for the provider — on-prem and kind need a custom one |
| Network trend empty | Mode 3 on an agent before 1.4.10, which does not read the network families, or the cluster has no priced load balancers or egress |
| Every total reads about double | The exporter and promRead.cost both on, on an agent before 1.4.10: turn one off |
| Every container’s cost sits in OpenCost’s namespace | Mode 3 on an agent before 1.4.10, reading a Prometheus that scrapes OpenCost without honor_labels: upgrade the agent |
The agent’s helm upgrade fails naming agent.promRead.cost.enabled and the OpenCost exporter | Two sources of cost: turn one off. Until then the release stays on its previous version |
Cost samples are ordinary metrics: they follow the retention of KubeBolt’s
metrics store (30 days on the bundled VictoriaMetrics,
metrics.storage.embedded.retention in the kubebolt chart), so history goes
back as far as that.