KubeBolt docs
ENES GitHub

Getting Started

Troubleshooting

Symptom, cause, fix — for the failures that actually happen: pods stuck Pending, empty dashboards, the limited-access banner, 503s, Cluster unreachable, a lost admin password, and an agent that can't reach the backend.

On this page
  1. Pods stay Pending after helm install
  2. The agent is connected but the dashboards are empty
  3. Amber banner: “Limited access — N of M resource types restricted by RBAC”
  4. 503 “cluster not connected” right after a switch or a restart
  5. ”Cluster unreachable” on a cluster that is reachable
  6. ”No contact with this cluster” or “Waiting for the agent”
  7. The agent cannot reach the backend
  8. The agent is OOMKilled on a large cluster
  9. The admin password is lost
  10. The terminal will not open on an agent-connected cluster
  11. Kobi’s panel is missing
  12. Still stuck

Find your symptom. Each entry says what you see, why it happens, and what to do. The commands assume the Helm release kubebolt in namespace kubebolt and the agent in kubebolt-system; use yours (helm list -A | grep kubebolt). Scanner-specific failures live on their own pages — Trivy, Kyverno, Falco.

Pods stay Pending after helm install

Symptom. helm install --wait times out, and kubebolt-api-… and kubebolt-victoriametrics-0 sit in Pending.

Cause. The chart creates two PersistentVolumeClaims — 10 GiB for VictoriaMetrics and 1 GiB for KubeBolt’s own data — on the cluster’s default StorageClass. With no default (EKS without the EBS CSI driver, some bare-metal clusters), they never bind.

Fix.

kubectl -n kubebolt get pvc
kubectl get storageclass

Install a provisioner (on EKS, the EBS CSI driver add-on) and mark a StorageClass as default, or name one explicitly with --set metrics.storage.embedded.persistence.storageClass=<class> and --set auth.persistence.storageClass=<class>.

The agent is connected but the dashboards are empty

Symptom. The cluster shows as connected, the agent pod is Running — and the history panels render blank.

Cause and fix, in the order worth checking:

CheckWhat it looks likeFix
Agent older than 1.0The API log has agent below minimum version — legacy schema, with agent_version and min_agent_versionUpgrade the agent (below)
The metrics store is downEvery history panel is empty on every cluster; Administration → System → Health shows VictoriaMetrics not reportingkubectl -n kubebolt get pods,pvc — see the previous entry if its PVC is Pending
A cap is cutting ingestA banner on the Overview: Ingestion truncated by plan limits or Metric series cap engagedOn Cloud, the plan’s caps (Free: 10 nodes, 150 pods, 25,000 series); what’s past them is not recorded. Self-hosted, raise the series caps in Administration → Agents & Ingest → Configuration
Only a few minutes inLive values show, history doesn’t yetWait: history needs the agent to ship a few minutes of samples

The schema mismatch is the hard one to spot. The agent and the backend ship as independently versioned artifacts coupled by one contract: the metric and label names the agent emits and the backend queries. When those disagree, samples still flow into the metrics store and the queries simply match nothing. Nothing crashes. Check:

kubectl logs -n kubebolt deployment/kubebolt-api --tail=200 | grep "agent below minimum"

Upgrade the agent past the floor and the next refetch repopulates the panels:

helm upgrade kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  -n kubebolt-system --reset-then-reuse-values

No output from that grep and still-empty panels after the other checks means the mismatch may be the other way round — a current agent against a backend older than 1.10. See Compatibility for the matrix.

On a fleet, this is per cluster. A backend serving a mix of current and stale agents shows empty dashboards only for the clusters that lag, and logs one WARN per legacy agent on every reconnect.

Amber banner: “Limited access — N of M resource types restricted by RBAC”

Symptom. A warning strip under the topbar, and resource lists that are missing whole types. Restricted types are dimmed with a shield icon in the sidebar.

Cause. The permission probe ran and found that the credentials KubeBolt connects with cannot list/watch some resource types: a kubeconfig user with narrower RBAC, the kubebolt chart installed with rbac.create=false and a narrower role of your own, or an agent whose ClusterRole was edited by hand.

Fix. Check what was probed — GET /api/v1/cluster/permissions returns the full probe result — then widen the role behind those credentials. For an agent, re-applying the chart’s tier restores its ClusterRole:

helm upgrade kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  -n kubebolt-system --reset-then-reuse-values --set rbac.mode=reader

The agent’s tiers are described in Connecting Clusters and RBAC. An agent in rbac.mode=metrics does not raise this banner: its cluster is metrics-only, and resource views answer 503 cluster is monitored-only. A missing optional CRD — Cilium policies, Argo CD applications, cert-manager certificates — does not count toward this banner either and produces a separate, neutral notice instead. The banner is dismissible; dismissing it does not change any permission.

Two related cases. Types KubeBolt may only read namespace by namespace stay without access when listing the namespaces keeps failing, and the views say so instead of showing a partial list (RBAC). Without access to events, the Events page shows Access Restricted, GET /events answers 403, and Kobi is told that seeing no events does not mean there are none.

503 “cluster not connected” right after a switch or a restart

Symptom. Requests fail with 503 cluster not connected for a few seconds after switching clusters or restarting the backend. (Kobi still answers in that window — about KubeBolt and Kubernetes — but its cluster tools have nothing to read until the connector is up.)

Cause. The connector is still warming. A cold connect builds informer caches for every watched resource type and does not serve reads until they sync. The budget is the Cluster connect timeout, how long the sync may go without receiving anything: 45 seconds by default, in Administration → System → General (5–600 s, applies on the next connect), or KUBEBOLT_CACHE_SYNC_TIMEOUT_SECONDS at boot. A sync that keeps receiving pages carries on, up to 10 minutes. Meanwhile the page shows the steps and a bar that follows the data still to arrive (Multi-Cluster).

Fix. Wait. It is a startup state, not an error, and the warm connector pool makes later switches instant. Two other 503 bodies are not warm-up:

If it never resolves, read the next entry.

”Cluster unreachable” on a cluster that is reachable

Symptom. The page shows Cluster unreachable — “Could not connect to the selected cluster” — while kubectl against the same cluster works. After a failed attempt the API waits 30 seconds, then 1, 2 and 4 minutes, at most 5, before connecting again, and the loading page and the 503 say when the next attempt is; switching to the cluster tries at once. The API log says why, one cluster runtime failed to connect line per attempt:

kubectl -n kubebolt logs deployment/kubebolt-api | grep -iE "failed to connect|staying disconnected|stopped|timed out"

Cause. The connect received nothing from the cluster for longer than its budget, or was still going after 10 minutes. Which budget depends on how KubeBolt reaches the cluster:

Reached…ErrorBudget
Directly (in-cluster, kubeconfig)cache sync stopped: no response from the cluster for 45s — cluster may be unreachableCluster connect timeout — 45 s default
Through an agentconnect to context … stopped: no response from the cluster for 60s — agent may be stuckConnect deadline — 60 s default (25 s before 2.6.7)

Before 2.6.5 both were fixed deadlines (timed out waiting for cache sync, connect timed out after 25s): a large cluster still receiving pages failed at the same point on every attempt. Upgrading fixes that case without raising anything.

On a reachable cluster the usual culprit is one enormous collection — Events. KubeBolt waits for every informer before it serves any, so one slow informer holds back the rest. Reproduced on OpenShift 4.20 / Kubernetes 1.33.6: GET /api/v1/events returned ~138 MiB, while Nodes and Deployments were ready in ~1.8 s and still could not be served.

Fix. Confirm the size first:

kubectl get events -A --no-headers | wc -l

Then, in this order:

Detail is in OpenShift, where this was first reproduced. It is not OpenShift-specific — any churning cluster can hit it.

”No contact with this cluster” or “Waiting for the agent”

Symptom. An amber strip, “No contact with this cluster since …”, over pages that show the cluster’s last known state — or a spinner, “Waiting for the agent”, instead of the page.

Cause. KubeBolt reaches this cluster through its agent, and no agent is connected. Right after a KubeBolt restart the agents dial back in within a minute. If they don’t, the agent, its node or the network went down; KubeBolt can’t tell which from its side.

Fix. In that cluster, check the agent (kubectl -n kubebolt-system get pods and its logs — next entry). If you decommissioned the cluster, delete it from Clusters: a cluster whose agent stops stays listed until you do.

The agent cannot reach the backend

Symptom. The agent pod runs, or never starts, and the cluster never appears — or the Add cluster wizard keeps waiting.

Start here, with kubectl pointing at the agent’s cluster:

kubectl -n kubebolt-system get pods -l app.kubernetes.io/name=kubebolt-agent
kubectl -n kubebolt-system logs ds/kubebolt-agent --tail=100

A healthy agent logs dialing backend and then channel registered. A failed attempt logs shipper session ended, will reconnect with the error, and retries with backoff. Causes and fixes, in the order worth checking:

CheckWhat it looks likeFix
Wrong clusterNothing in kubebolt-systemkubectl pointed at another context. Switch and run the install again
Token Secret missingPods stuck in ContainerCreating; kubectl describe pod says secret "kubebolt-agent-token" not foundCreate the Secret in the agent’s namespace, with the token under the key token, named as in auth.ingestToken.existingSecret
Chart refuses to renderbackendUrl is required, or auth.mode=ingest-token requires auth.ingestToken.existingSecretSet the missing value
backendUrl has a schemeDial or address errors in the logIt is a gRPC dial target, host:port, no https://
Wrong address or portconnection refusedA self-hosted backend’s <release>-agent-ingest Service listens on 9090; behind the Gateway API listener or a TLS load balancer it is usually 443. KubeBolt Cloud’s is agent.kubebolt.io:443
Egress blockedTimeoutsThe agent only ever dials outbound. Allow egress to the backend host and port
TLS mismatchTLS handshake errors, or the connection closes before the first message--set tls.enabled=true for any endpoint that terminates TLS (typically 443), and off for a plain 9090
Token rejectedcode = Unauthenticated desc = agent auth: invalid credentialsMissing, mistyped or expired token, against a backend that enforces agent auth (Cloud always does). Issue a new token in Administration → Agents & Ingest → Agent Tokens, replace the Secret, then kubectl -n kubebolt-system rollout restart ds/kubebolt-agent
Token revokedcode = PermissionDenied desc = agent auth: credential rejectedSame fix: a new token
Plan cap (Cloud)code = ResourceExhausted desc = cluster cap reached (2/2) for plan "free" — remove a cluster or upgrade to add anotherDelete a cluster you no longer use under Clusters, or change plan in Administration → Billing

Then confirm the backend saw anything at all: Administration → Agents & Ingest → Activity shows samples per second, active series, connection events (an auth rejected count means the token) and a list of the agents currently connected. If your agent is absent from that list and nothing was rejected, the traffic never reached the backend — that is a network or address problem, not an auth one.

The agent is OOMKilled on a large cluster

Symptom. On a cluster connected through its agent, the agent pods restart with OOMKilled during the first connect, or a list fails with … exceeds N MiB, a quarter of this agent's memory limit (M MiB): raise the agent's resources.limits.memory.

Cause. The agent relays the API server’s responses and holds each one whole. On a cluster with thousands of ReplicaSets, a list is tens of MB. The agent’s default limit was 128Mi before 1.4.10; since 1.4.10 it is 256Mi, and a body past a quarter of the limit is refused with the error above instead of taking the pod down.

Fix. Upgrade the agent to 1.4.10, or raise its memory limit on its release:

helm upgrade kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  -n kubebolt-system --reset-then-reuse-values --set resources.limits.memory=512Mi

The OOMKilled and Memory Near Limit insights on an agent pod give this command already filled in: twice the current limit, at least 256Mi, pinned to the chart version the agent runs (for the promread pod, its own agent.promRead.deployment.resources.limits.memory). More in The Agent.

The admin password is lost

Symptom. You cannot log in to a self-hosted KubeBolt and nobody has the generated first-boot password. (On KubeBolt Cloud, use Forgot password? on the sign-in page.)

Cause. The password is printed to the API log exactly once, on the boot that seeds the admin user. A later restart does not reprint it.

Fix. First try the Secret: on an in-cluster install, the generated first-boot password is also stored in kubebolt-admin-password in the release namespace (it does not track later password changes, and it isn’t created when you set the password yourself):

kubectl -n kubebolt get secret kubebolt-admin-password \
  -o jsonpath='{.data.password}' | base64 -d ; echo

If the password was changed since, reset it. The helm upgrade path is the simple one — the new pod resets the password on boot and then starts normally:

helm upgrade kubebolt \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  -n kubebolt --reset-then-reuse-values --set auth.resetAdminPassword='NEW-PASSWORD'

Log in with it, change it from the account menu, then clear the value:

helm upgrade kubebolt \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  -n kubebolt --reset-then-reuse-values --set auth.resetAdminPassword=

Running --reset-admin-password with kubectl exec inside the live pod does not work: the running API holds the embedded database’s lock, so the reset can’t open it. Without Helm, scale deployment/kubebolt-api to 0 and run the reset as a one-shot Job on the same PVC (kubebolt-api --reset-admin-password=NEW-PASSWORD with KUBEBOLT_DATA_DIR=/data) — the exact Job is in the chart README. On the single binary or single container, stop the process and run kubebolt --reset-admin-password=NEW-PASSWORD against the same data directory. Minimum length is 8 characters.

Clear auth.resetAdminPassword on the next helm upgrade, as above. While it is set, every new pod resets the password on boot and the value sits in your release values.

On a fresh install, set the password instead of hunting for it: --set auth.adminPassword=... (see Authentication). The first-boot log line is only there on the boot that seeded the admin user:

kubectl -n kubebolt logs deployment/kubebolt-api | grep "Generated admin password"

The terminal will not open on an agent-connected cluster

Symptom. A pod’s terminal shows an error instead of a shell, and the pod file browser and port-forward fail too. Since 2.6.7 the terminal says why: the agent’s mode does not allow exec. Before, it printed agent-proxy SPDY: expected 101 Switching Protocols, got 403.

Cause. The agent runs in reader mode, the chart’s default. It reads, and the API server refuses pods/exec to it. On a cluster connected through a kubeconfig, it is that context’s credentials that may not exec.

Fix. Upgrade the agent to the operator tier, which adds create on pods/exec, pods/portforward and pods/eviction (the agent’s RBAC tiers):

helm upgrade kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  -n kubebolt-system --reset-then-reuse-values --set rbac.mode=operator

Since 2.6.7 a refused terminal is not recorded as an opened session: the audit trail keeps one open record with result: error and the cluster’s reason.

Kobi’s panel is missing

The Kobi button (bottom right) and ⌘J appear only when Kobi Copilot is configured — an API key saved in Administration → AI (Kobi), or set with KUBEBOLT_AI_API_KEY at boot. The API logs which at startup:

kubectl -n kubebolt logs deployment/kubebolt-api | grep -i "AI copilot"

AI copilot enabled or AI copilot disabled (KUBEBOLT_AI_API_KEY not set); a key saved in the UI later doesn’t change that line. See Enabling Kobi. On KubeBolt Cloud there is no key to set: Kobi needs a confirmed email, and pauses when the month’s AI credits run out (AI credits). If the panel opens and Kobi answers but can’t read the cluster, that is the connector, not the copilot: read the 503 entry above.

Still stuck

Open an issue with the backend version, the agent version, and the log line you saw, at github.com/clm-cloud-solutions/kubebolt, or tell us at kubebolt.io/feedback.