Getting Started
Troubleshooting
Symptom, cause, fix — for the failures that actually happen: pods stuck Pending, empty dashboards, the limited-access banner, 503s, Cluster unreachable, a lost admin password, and an agent that can't reach the backend.
On this page
- Pods stay Pending after helm install
- The agent is connected but the dashboards are empty
- Amber banner: “Limited access — N of M resource types restricted by RBAC”
- 503 “cluster not connected” right after a switch or a restart
- ”Cluster unreachable” on a cluster that is reachable
- ”No contact with this cluster” or “Waiting for the agent”
- The agent cannot reach the backend
- The agent is OOMKilled on a large cluster
- The admin password is lost
- The terminal will not open on an agent-connected cluster
- Kobi’s panel is missing
- Still stuck
Find your symptom. Each entry says what you see, why it happens, and what to
do. The commands assume the Helm release kubebolt in namespace kubebolt
and the agent in kubebolt-system; use yours (helm list -A | grep kubebolt).
Scanner-specific failures live on their own pages —
Trivy, Kyverno, Falco.
Pods stay Pending after helm install
Symptom. helm install --wait times out, and kubebolt-api-… and
kubebolt-victoriametrics-0 sit in Pending.
Cause. The chart creates two PersistentVolumeClaims — 10 GiB for VictoriaMetrics and 1 GiB for KubeBolt’s own data — on the cluster’s default StorageClass. With no default (EKS without the EBS CSI driver, some bare-metal clusters), they never bind.
Fix.
kubectl -n kubebolt get pvc
kubectl get storageclass
Install a provisioner (on EKS, the EBS CSI driver add-on) and mark a
StorageClass as default, or name one explicitly with
--set metrics.storage.embedded.persistence.storageClass=<class> and
--set auth.persistence.storageClass=<class>.
The agent is connected but the dashboards are empty
Symptom. The cluster shows as connected, the agent pod is Running — and
the history panels render blank.
Cause and fix, in the order worth checking:
| Check | What it looks like | Fix |
|---|---|---|
| Agent older than 1.0 | The API log has agent below minimum version — legacy schema, with agent_version and min_agent_version | Upgrade the agent (below) |
| The metrics store is down | Every history panel is empty on every cluster; Administration → System → Health shows VictoriaMetrics not reporting | kubectl -n kubebolt get pods,pvc — see the previous entry if its PVC is Pending |
| A cap is cutting ingest | A banner on the Overview: Ingestion truncated by plan limits or Metric series cap engaged | On Cloud, the plan’s caps (Free: 10 nodes, 150 pods, 25,000 series); what’s past them is not recorded. Self-hosted, raise the series caps in Administration → Agents & Ingest → Configuration |
| Only a few minutes in | Live values show, history doesn’t yet | Wait: history needs the agent to ship a few minutes of samples |
The schema mismatch is the hard one to spot. The agent and the backend ship as independently versioned artifacts coupled by one contract: the metric and label names the agent emits and the backend queries. When those disagree, samples still flow into the metrics store and the queries simply match nothing. Nothing crashes. Check:
kubectl logs -n kubebolt deployment/kubebolt-api --tail=200 | grep "agent below minimum"
Upgrade the agent past the floor and the next refetch repopulates the panels:
helm upgrade kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
-n kubebolt-system --reset-then-reuse-values
No output from that grep and still-empty panels after the other checks means
the mismatch may be the other way round — a current agent against a backend
older than 1.10. See Compatibility for the matrix.
On a fleet, this is per cluster. A backend serving a mix of current and stale agents shows empty dashboards only for the clusters that lag, and logs one WARN per legacy agent on every reconnect.
Amber banner: “Limited access — N of M resource types restricted by RBAC”
Symptom. A warning strip under the topbar, and resource lists that are missing whole types. Restricted types are dimmed with a shield icon in the sidebar.
Cause. The permission probe ran and found that the credentials KubeBolt
connects with cannot list/watch some resource types: a kubeconfig user
with narrower RBAC, the kubebolt chart installed with rbac.create=false
and a narrower role of your own, or an agent whose ClusterRole was edited by
hand.
Fix. Check what was probed — GET /api/v1/cluster/permissions returns the
full probe result — then widen the role behind those credentials. For an
agent, re-applying the chart’s tier restores its ClusterRole:
helm upgrade kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
-n kubebolt-system --reset-then-reuse-values --set rbac.mode=reader
The agent’s tiers are described in
Connecting Clusters and RBAC.
An agent in rbac.mode=metrics does not raise this banner: its cluster is
metrics-only, and resource views answer 503 cluster is monitored-only.
A missing optional CRD — Cilium policies, Argo CD applications,
cert-manager certificates — does not count toward this banner either and
produces a separate, neutral notice instead. The banner is dismissible;
dismissing it does not change any permission.
Two related cases. Types KubeBolt may only read namespace by namespace stay
without access when listing the namespaces keeps failing, and the views say so
instead of showing a partial list (RBAC).
Without access to events, the Events page shows Access Restricted,
GET /events answers 403, and Kobi is told that seeing no events does not
mean there are none.
503 “cluster not connected” right after a switch or a restart
Symptom. Requests fail with 503 cluster not connected for a few seconds
after switching clusters or restarting the backend. (Kobi still answers in
that window — about KubeBolt and Kubernetes — but its cluster tools have
nothing to read until the connector is up.)
Cause. The connector is still warming. A cold connect builds informer caches
for every watched resource type and does not serve reads until they sync. The
budget is the Cluster connect timeout, how long the sync may go without
receiving anything: 45 seconds by default, in
Administration → System → General (5–600 s, applies on the next connect),
or KUBEBOLT_CACHE_SYNC_TIMEOUT_SECONDS at boot. A sync that keeps receiving
pages carries on, up to 10 minutes. Meanwhile the page shows the steps and a
bar that follows the data still to arrive
(Multi-Cluster).
Fix. Wait. It is a startup state, not an error, and the warm connector pool makes later switches instant. Two other 503 bodies are not warm-up:
no cluster selected — choose a cluster to continue— pick one in the cluster switcher.cluster is monitored-only — enable the agent-proxy (rbac.mode=reader|operator) for resource views— the agent runs inmetricsmode; upgrade it with--set rbac.mode=reader.
If it never resolves, read the next entry.
”Cluster unreachable” on a cluster that is reachable
Symptom. The page shows Cluster unreachable — “Could not connect to
the selected cluster” — while kubectl against the same cluster works. After
a failed attempt the API waits 30 seconds, then 1, 2 and 4 minutes, at most 5,
before connecting again, and the loading page and the 503 say when the next
attempt is; switching to the cluster tries at once. The API log says why, one
cluster runtime failed to connect line per attempt:
kubectl -n kubebolt logs deployment/kubebolt-api | grep -iE "failed to connect|staying disconnected|stopped|timed out"
Cause. The connect received nothing from the cluster for longer than its budget, or was still going after 10 minutes. Which budget depends on how KubeBolt reaches the cluster:
| Reached… | Error | Budget |
|---|---|---|
| Directly (in-cluster, kubeconfig) | cache sync stopped: no response from the cluster for 45s — cluster may be unreachable | Cluster connect timeout — 45 s default |
| Through an agent | connect to context … stopped: no response from the cluster for 60s — agent may be stuck | Connect deadline — 60 s default (25 s before 2.6.7) |
Before 2.6.5 both were fixed deadlines (timed out waiting for cache sync,
connect timed out after 25s): a large cluster still receiving pages failed
at the same point on every attempt. Upgrading fixes that case without raising
anything.
On a reachable cluster the usual culprit is one enormous collection —
Events. KubeBolt waits for every informer before it serves any, so one slow
informer holds back the rest. Reproduced on OpenShift 4.20 / Kubernetes
1.33.6: GET /api/v1/events returned ~138 MiB, while Nodes and Deployments were
ready in ~1.8 s and still could not be served.
Fix. Confirm the size first:
kubectl get events -A --no-headers | wc -l
Then, in this order:
- Raise the budget. Direct connections: Cluster connect timeout in
Administration → System → General (no restart). Agent-connected clusters:
Connect deadline in Administration → Agents & Ingest → Configuration
→ Agent-proxy resilience, or
KUBEBOLT_AGENT_PROXY_CONNECT_TIMEOUT(a Go duration,60sby default since 2.6.7; it is never applied below the request timeout + 5 s). Loosening the agent one weakens a protection that exists for genuinely wedged agents. - Give the API room for its Events. The API caches them; the chart’s
default memory limit is 1 GiB. If the API pod restarts during its first
sync, raise it:
--set api.resources.limits.memory=2Gi. - Last resort: drop
eventsfrom theget/list/watchrule of KubeBolt’s ClusterRole. The cluster comes up immediately; the Events tab and any insight that reads Events go empty, and the permission probe honestly reports one fewer resource type.
Detail is in OpenShift, where this was first reproduced. It is not OpenShift-specific — any churning cluster can hit it.
”No contact with this cluster” or “Waiting for the agent”
Symptom. An amber strip, “No contact with this cluster since …”, over pages that show the cluster’s last known state — or a spinner, “Waiting for the agent”, instead of the page.
Cause. KubeBolt reaches this cluster through its agent, and no agent is connected. Right after a KubeBolt restart the agents dial back in within a minute. If they don’t, the agent, its node or the network went down; KubeBolt can’t tell which from its side.
Fix. In that cluster, check the agent (kubectl -n kubebolt-system get pods
and its logs — next entry). If you decommissioned the cluster, delete it from
Clusters: a cluster whose agent stops stays listed until you do.
The agent cannot reach the backend
Symptom. The agent pod runs, or never starts, and the cluster never appears — or the Add cluster wizard keeps waiting.
Start here, with kubectl pointing at the agent’s cluster:
kubectl -n kubebolt-system get pods -l app.kubernetes.io/name=kubebolt-agent
kubectl -n kubebolt-system logs ds/kubebolt-agent --tail=100
A healthy agent logs dialing backend and then channel registered. A failed
attempt logs shipper session ended, will reconnect with the error, and
retries with backoff. Causes and fixes, in the order worth checking:
| Check | What it looks like | Fix |
|---|---|---|
| Wrong cluster | Nothing in kubebolt-system | kubectl pointed at another context. Switch and run the install again |
| Token Secret missing | Pods stuck in ContainerCreating; kubectl describe pod says secret "kubebolt-agent-token" not found | Create the Secret in the agent’s namespace, with the token under the key token, named as in auth.ingestToken.existingSecret |
| Chart refuses to render | backendUrl is required, or auth.mode=ingest-token requires auth.ingestToken.existingSecret | Set the missing value |
backendUrl has a scheme | Dial or address errors in the log | It is a gRPC dial target, host:port, no https:// |
| Wrong address or port | connection refused | A self-hosted backend’s <release>-agent-ingest Service listens on 9090; behind the Gateway API listener or a TLS load balancer it is usually 443. KubeBolt Cloud’s is agent.kubebolt.io:443 |
| Egress blocked | Timeouts | The agent only ever dials outbound. Allow egress to the backend host and port |
| TLS mismatch | TLS handshake errors, or the connection closes before the first message | --set tls.enabled=true for any endpoint that terminates TLS (typically 443), and off for a plain 9090 |
| Token rejected | code = Unauthenticated desc = agent auth: invalid credentials | Missing, mistyped or expired token, against a backend that enforces agent auth (Cloud always does). Issue a new token in Administration → Agents & Ingest → Agent Tokens, replace the Secret, then kubectl -n kubebolt-system rollout restart ds/kubebolt-agent |
| Token revoked | code = PermissionDenied desc = agent auth: credential rejected | Same fix: a new token |
| Plan cap (Cloud) | code = ResourceExhausted desc = cluster cap reached (2/2) for plan "free" — remove a cluster or upgrade to add another | Delete a cluster you no longer use under Clusters, or change plan in Administration → Billing |
Then confirm the backend saw anything at all: Administration → Agents & Ingest → Activity shows samples per second, active series, connection events (an auth rejected count means the token) and a list of the agents currently connected. If your agent is absent from that list and nothing was rejected, the traffic never reached the backend — that is a network or address problem, not an auth one.
The agent is OOMKilled on a large cluster
Symptom. On a cluster connected through its agent, the agent pods restart
with OOMKilled during the first connect, or a list fails with
… exceeds N MiB, a quarter of this agent's memory limit (M MiB): raise the agent's resources.limits.memory.
Cause. The agent relays the API server’s responses and holds each one
whole. On a cluster with thousands of ReplicaSets, a list is tens of MB. The
agent’s default limit was 128Mi before 1.4.10; since 1.4.10 it is 256Mi,
and a body past a quarter of the limit is refused with the error above instead
of taking the pod down.
Fix. Upgrade the agent to 1.4.10, or raise its memory limit on its release:
helm upgrade kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
-n kubebolt-system --reset-then-reuse-values --set resources.limits.memory=512Mi
The OOMKilled and Memory Near Limit insights on an agent pod give this
command already filled in: twice the current limit, at least 256Mi, pinned to
the chart version the agent runs (for the promread pod, its own
agent.promRead.deployment.resources.limits.memory). More in
The Agent.
The admin password is lost
Symptom. You cannot log in to a self-hosted KubeBolt and nobody has the generated first-boot password. (On KubeBolt Cloud, use Forgot password? on the sign-in page.)
Cause. The password is printed to the API log exactly once, on the boot that seeds the admin user. A later restart does not reprint it.
Fix. First try the Secret: on an in-cluster install, the generated
first-boot password is also stored in kubebolt-admin-password in the release
namespace (it does not track later password changes, and it isn’t created when
you set the password yourself):
kubectl -n kubebolt get secret kubebolt-admin-password \
-o jsonpath='{.data.password}' | base64 -d ; echo
If the password was changed since, reset it. The helm upgrade path is the
simple one — the new pod resets the password on boot and then starts normally:
helm upgrade kubebolt \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
-n kubebolt --reset-then-reuse-values --set auth.resetAdminPassword='NEW-PASSWORD'
Log in with it, change it from the account menu, then clear the value:
helm upgrade kubebolt \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
-n kubebolt --reset-then-reuse-values --set auth.resetAdminPassword=
Running --reset-admin-password with kubectl exec inside the live pod does
not work: the running API holds the embedded database’s lock, so the reset
can’t open it. Without Helm, scale deployment/kubebolt-api to 0 and run the
reset as a one-shot Job on the same PVC (kubebolt-api --reset-admin-password=NEW-PASSWORD with KUBEBOLT_DATA_DIR=/data) — the
exact Job is in the chart README.
On the single binary or single container, stop the process and run
kubebolt --reset-admin-password=NEW-PASSWORD against the same data
directory. Minimum length is 8 characters.
Clear auth.resetAdminPassword on the next helm upgrade, as above. While
it is set, every new pod resets the password on boot and the value sits in
your release values.
On a fresh install, set the password instead of hunting for it:
--set auth.adminPassword=... (see Authentication). The first-boot
log line is only there on the boot that seeded the admin user:
kubectl -n kubebolt logs deployment/kubebolt-api | grep "Generated admin password"
The terminal will not open on an agent-connected cluster
Symptom. A pod’s terminal shows an error instead of a shell, and the pod
file browser and port-forward fail too. Since 2.6.7 the terminal says why: the
agent’s mode does not allow exec. Before, it printed
agent-proxy SPDY: expected 101 Switching Protocols, got 403.
Cause. The agent runs in reader mode, the chart’s default. It reads, and
the API server refuses pods/exec to it. On a cluster connected through a
kubeconfig, it is that context’s credentials that may not exec.
Fix. Upgrade the agent to the operator tier, which adds create on
pods/exec, pods/portforward and pods/eviction
(the agent’s RBAC tiers):
helm upgrade kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
-n kubebolt-system --reset-then-reuse-values --set rbac.mode=operator
Since 2.6.7 a refused terminal is not recorded as an opened session: the audit
trail keeps one open record with result: error and the cluster’s reason.
Kobi’s panel is missing
The Kobi button (bottom right) and ⌘J appear only when Kobi Copilot is
configured — an API key saved in Administration → AI (Kobi), or set with
KUBEBOLT_AI_API_KEY at boot. The API logs which at startup:
kubectl -n kubebolt logs deployment/kubebolt-api | grep -i "AI copilot"
AI copilot enabled or AI copilot disabled (KUBEBOLT_AI_API_KEY not set);
a key saved in the UI later doesn’t change that line. See
Enabling Kobi. On KubeBolt Cloud there is no key to
set: Kobi needs a confirmed email, and pauses when the month’s AI credits run
out (AI credits). If the panel opens and Kobi answers
but can’t read the cluster, that is the connector, not the copilot: read the
503 entry above.
Still stuck
Open an issue with the backend version, the agent version, and the log line you saw, at github.com/clm-cloud-solutions/kubebolt, or tell us at kubebolt.io/feedback.