Core Concepts
Architecture
Go backend with informer caches, VictoriaMetrics for history (bundled with the Helm chart and Compose), and an outbound-only agent. React frontend with live WebSocket updates.
On this page
System Diagram
(Go) (React 18 · TS · Vite 5 · Tailwind) Go Workspace
Monorepo with go.work containing four modules:
apps/api— Main backend server (entry:cmd/server/main.go)packages/agent— The cluster agent (its own release line,1.x)packages/proto— Generated gRPC/protobuf types shared by API and agentpackages/shared— Shared Go utilities
The two halves
Backend (per install): reads the Kubernetes API through shared informers,
evaluates insights, serves the REST/WebSocket API, and keeps state — users,
settings, insight episodes, security findings, the audit trail, Kobi
conversations — in an embedded BoltDB. Historical metrics go to
VictoriaMetrics: the Helm chart and the Compose stack run a single-node
instance next to the API; the single binary and single-container image don’t
bundle one, and read any VictoriaMetrics-compatible endpoint you point
KUBEBOLT_METRICS_STORAGE_URL at.
Agent (per cluster, optional): connects outbound over gRPC
(AgentChannel) and ships metric samples — from its built-in kubelet/node
collectors, a bundled vmagent scrape sidecar, or by reading your existing
Prometheus (promRead). It also carries Hubble network flows for the
Reliability tab and the Cluster Map’s Traffic layout,
OpenCost series for the Cost tab and, in reader or operator
mode, proxies Kubernetes API
calls for clusters the backend can’t reach directly — including exec,
port-forward and file-browser sessions in operator mode. See
Connecting clusters.
Key Backend Packages
All under apps/api/internal/.
| Package | Purpose |
|---|---|
cluster/manager.go | Cluster registry + lifecycle: kubeconfig contexts, agent-proxy and metrics-only clusters, display names, context switching; runs the insight evaluation loop (every 60s) |
cluster/connector.go | Shared informers + dynamic client; the cache sync fails after 45s without receiving anything (configurable) and carries on while pages arrive, up to 10 minutes; 15s rest timeout, debounced topology rebuild (2s) |
cluster/permissions.go | RBAC probing via SSAR (31 resource types), cluster-wide then namespace fallback, semaphore of 10 |
cluster/optional_crds.go | Re-checks optional CRDs (cert-manager, Argo CD, VPA, Cilium policies) every 2 minutes |
cluster/nslister.go | Multi-namespace lister wrappers for namespace-scoped ServiceAccounts |
cluster/graph.go | In-memory topology graph |
cluster/relationships.go | Edge detection: ownerRefs, selectors, Gateway parentRefs, volumes, env references |
agent/channel | gRPC AgentChannel: sample ingest, tunnels for exec/port-forward/files |
metrics/collector.go | Metrics Server polling (every 30s), per-namespace fallback, graceful degradation |
insights/engine.go | 24-rule evaluation engine |
insights/policy.go | Rule classes, thresholds and the per-rule policy layer |
insights/episodestore_bolt.go | Episode lifecycle on BoltDB: episodes, transitions, mutes, presence |
insights/policystore_bolt.go | Rule policy overrides on BoltDB |
helm/ | Read-only Helm releases decoded from their storage Secrets |
findings/ | Security & Compliance store — Trivy, Kyverno and Falco findings, persisted so they answer without a live connector |
copilot/ | Kobi: 41 tools, providers, conversations, memory/compact, usage analytics |
mcp/ | Read-only MCP server over Streamable HTTP and stdio |
auth/store.go, auth/jwt.go, auth/middleware.go | Users and refresh tokens on BoltDB, JWT issue/verify, role enforcement (viewer < editor < admin) |
auth/api_tokens_store.go | Long-lived REST tokens (kbs_ service, kbk_ key) with path scopes and the public-edge rejection |
websocket/hub.go | Broadcast hub, 4096 buffer, silent drops when no clients; broadcasts carry a reference (kind, namespace, name), never the object |
api/router.go | Chi router with requireConnector middleware |
Data Flow
- Manager reads kubeconfig contexts + the persistent registry → async connection (HTTP server binds immediately, returns 503 until connected)
- Permission probe: SSAR calls, cluster-wide then namespace fallback, ~2-5s
- Informers start only for permitted resources; namespace-scoped SAs get per-namespace informer factories with multi-lister aggregation
- Each informer’s initial LIST arrives in pages of 500 objects; watches are
unchanged. Every 5 minutes a cache audit asks for one object per type and
reads the count from
remainingItemCount, skipping the pass when the server does not report it - The Secrets cache leaves Helm’s release storage out: lists and counts read its metadata once a minute, and a release’s payload is read when a view or an insight needs it
- A connect that fails is retried after 30 s, then 1, 2 and 4 minutes, at most 5; an agent reconnecting or an explicit switch tries at once
- Dynamic client lists Gateway API resources (5s timeout, gracefully skipped) and the optional CRDs on demand
- Metrics Server polls every 30s → in-memory cache; agent samples land in
VictoriaMetrics and serve
query/query_range(PromQL). A batch VictoriaMetrics refuses for a passing reason (connection error, timeout, 5xx, 429) is held, up to 64 MiB, and written again oldest first with its own timestamps, so a VictoriaMetrics restart leaves no gap - The insights engine evaluates every 60s over the informer cache and the Metrics Server readings, and persists episodes to BoltDB
- REST API serves enriched resources with metrics injection, paginated (50/page); WebSocket broadcasts changes with debounced topology rebuilds
Deployment topology
The web UI can be served from a different origin than the API (split
UI/API, with KUBEBOLT_CORS_ORIGINS listing the UI’s origin), and the chart supports fronting the API with either a classic
Ingress or Gateway API (gatewayAPI.enabled) — with separate listeners
for the HTTP/WebSocket API and the agent’s gRPC channel.