KubeBolt docs
ENES GitHub

Security

Falco: runtime events

Falco is the only source that pushes into KubeBolt. Wire falcosidekick to the ingest endpoint with a cluster-scoped token — anything else is a 403.

On this page
  1. The three things it needs
  2. Install
  3. How KubeBolt finds it
  4. Verify
  5. What is stored, and what is not
  6. Noise

Falco watches every node’s syscalls and speaks up when something steps outside the expected: a shell inside a container, a sensitive file read, a binary dropped and executed. Events land in the Runtime tab of Security & Compliance.

Unlike Trivy and Kyverno, KubeBolt does not go looking for this data — Falco pushes it. It is the only source that works that way, and that shapes everything about the configuration. falcosidekick, the forwarder the Falco chart deploys next to it, POSTs each event to KubeBolt’s API; the agent’s tunnel does not carry it.

The three things it needs

ValueWhere it comes fromWithout it
Ingest URLSelf-hosted: https://<your-kubebolt>/api/v1/ingest/falco. Cloud: https://api.kubebolt.io/api/v1/ingest/falcofalcosidekick logs a connection error and drops the event
Ingest tokenAdministration → Agents & Ingest → Agent Tokens, cluster-scoped (a kb_… ingest token, not a REST API token)401 — nothing is accepted
cluster_idthe kube-system namespace UIDOptional. If set and it disagrees with the token, 403

The URL has to be reachable from inside the cluster. On Cloud that means outbound HTTPS to api.kubebolt.io — a different host from the agent’s agent.kubebolt.io, so an egress allowlist needs both. Self-hosted, when Falco runs in the same cluster as KubeBolt, the API Service works without leaving the cluster: http://<kubebolt-api-service>.<namespace>.svc.cluster.local:8080/api/v1/ingest/falco (the Service is kubebolt-api for a release named kubebolt).

The tenant is never sent — it comes from the token. A self-hosted install has a single tenant; on Cloud this is also what stops a sender from writing into another organization’s feed.

Self-hosted, the endpoint needs authentication enabled (KUBEBOLT_AUTH_ENABLED=true, auth.enabled=true in the chart — the default). Ingest tokens and the event store don’t exist without it, and the endpoint answers 503 runtime events are not available (persistence disabled).

The token must be cluster-scoped, or every event is rejected. Falco has no handshake. The KubeBolt agent announces its cluster when it connects, but a pushed event carries only the token, so an unscoped token produces events that cannot be attributed to a cluster. The API refuses rather than filing them half-attributed:

403  this ingest token is not scoped to a cluster; issue a cluster-scoped token
     for Falco so its events can be attributed

Issue a token for this specific cluster — in Issue token, pick the cluster under Cluster scope instead of “Any cluster”. Do not reuse the agent’s.

Issuing tokens takes the admin role. In the Cluster scope dropdown only a cluster that has already reported its id can be picked, so connect the cluster first; it defaults to the cluster selected in the top bar. Expires after (days) is optional — leave it blank for a token that never expires, and revoke it from the same page.

Get the cluster id — the identity KubeBolt uses everywhere:

kubectl get ns kube-system -o jsonpath='{.metadata.uid}'

Install

CLUSTER_ID=$(kubectl get ns kube-system -o jsonpath='{.metadata.uid}')
TOKEN=<your cluster-scoped ingest token>

# The token goes in a Secret, not in the Helm values (where anyone who can run
# `helm get values` reads it). falcosidekick loads it after its own Secret.
kubectl create namespace falco --dry-run=client -o yaml | kubectl apply -f -
kubectl -n falco create secret generic falco-kubebolt \
  --from-literal=WEBHOOK_CUSTOMHEADERS="Authorization:Bearer $TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm upgrade --install falco falcosecurity/falco -n falco \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set-string falcosidekick.config.existingSecret=falco-kubebolt \
  --set-string falcosidekick.config.webhook.address=https://<your-kubebolt>/api/v1/ingest/falco \
  --set-string falcosidekick.config.customfields="cluster_id:$CLUSTER_ID"

On Cloud, the address is https://api.kubebolt.io/api/v1/ingest/falco.

Use chart 9.2.1 or later (Falco 0.45.1+); on kernel 6.18 older versions do not start (see The driver).

ValueWhat it does
driver.kindHow Falco reads syscalls: modern_ebpf (recommended), ebpf or kmod
falcosidekick.enabledDeploys falcosidekick, which forwards each event
falcosidekick.config.existingSecretSecret falcosidekick loads its settings from — here, the Authorization header
falcosidekick.config.webhook.addressThe ingest URL
falcosidekick.config.customfieldsOptional cluster_id:<uid> check: the token decides the cluster, and a cluster_id that disagrees with it is refused
falcosidekick.config.webhook.minimumpriorityOptional. Forward only events at or above a priority, e.g. warning

To rotate the token, re-run the create secret line with the new one and restart falcosidekick (kubectl -n falco rollout restart deploy/falco-falcosidekick): it reads the Secret only when it starts. To check what it sends:

kubectl get secret -n falco falco-kubebolt \
  -o jsonpath='{.data.WEBHOOK_CUSTOMHEADERS}' | base64 -d

Passing the header in the values instead (webhook.customHeaders) puts the token in the release values, and it has to be customHeaders, with a capital H: that is the only spelling the falcosidekick chart’s template reads for the webhook, while other outputs in the same values.yaml (Elasticsearch, Loki) spell it customheaders. The lowercase key leaves the header empty with no warning at all, the POST goes out without a token, and the API answers 401 — and the obvious diagnosis, the wrong one, is that the token is bad.

The driver

modern_ebpf (CO-RE) needs kernel 5.8 or newer with BTF:

uname -r                        # 5.8+
ls /sys/kernel/btf/vmlinux      # must exist

If it is not available, the chart also offers ebpf and kmod.

Kernel 6.18 needs Falco 0.45.1 or later (chart 9.2.1+). Earlier versions with modern_ebpf crash-loop there: the BPF verifier rejects the signal_deliver program (R9 invalid mem access 'scalar'). Amazon EKS AL2023 node images run 6.18 from November 2026. Fixed in falcosecurity/libs (#3119); verified with chart 9.2.2 / Falco 0.45.2.

One sensor per kernel

Falco is a DaemonSet: one pod per node, each seeing its own kernel. Correct on any real cluster.

On kind or k3d it is not. The “nodes” are containers sharing a single Docker kernel, so every Falco sees the whole machine’s syscalls and the same event arrives twice — one copy with the Kubernetes identity resolved, another without it, from the node that does not host the pod. For a lab, pin it to one node:

helm upgrade falco falcosecurity/falco -n falco --reuse-values \
  --set-string nodeSelector."kubernetes\.io/hostname"=<your-node>

No detection is lost — the remaining sensor already sees those syscalls.

How KubeBolt finds it

By the label app.kubernetes.io/name=falco, falling back to pods named falco* in the falco or falco-system namespaces. The result is the Falco card under Administration → Agents & Ingest → Integrations, for the cluster selected in the top bar. The card counts Falco’s pods and their readiness; whether events actually arrive shows only on the Runtime tab and in falcosidekick’s log.

Verify

# 1. Trip a rule on purpose
kubectl exec -n <ns> deploy/<something> -- cat /etc/shadow

# 2. Delivery must report 202
kubectl logs -n falco -l app.kubernetes.io/name=falcosidekick --tail=5
#   → Webhook - POST OK (202)

The event lands in the Runtime tab in under a minute.

SymptomCause
401 missing Bearer tokenexistingSecret was left out of the install, or the Secret’s key is not exactly WEBHOOK_CUSTOMHEADERS; with the header in the values, lowercase customheaders
falcosidekick stuck in CreateContainerConfigErrorThe falco-kubebolt Secret doesn’t exist in the falco namespace. Create it and the pod starts on its own
401 invalid ingest tokenThe token is revoked, expired, or belongs to another install or organization
503 runtime events are not available (persistence disabled)Self-hosted with authentication disabled
403 not scoped to a clusterThe token was issued without a cluster
403 cluster_id does not matchThe token belongs to a different cluster than customfields
400 unrecognized falco payloadThe body is not a falcosidekick webhook event: not JSON, or neither rule nor output in it
connection refused or a timeoutThe URL is not reachable from the cluster (egress rules, wrong host, or the in-cluster Service name)
Nothing in Falco’s own logThe driver never loaded — check kernel and BTF
Falco in CrashLoopBackOff, R9 invalid mem access 'scalar' in its logFalco older than 0.45.1 on kernel 6.18 — upgrade the chart to 9.2.1+

What is stored, and what is not

An event stores what falcosidekick sends and nothing more: the rule, the priority, the time, Falco’s output line, the hostname, every field of the event’s output_fields — namespace and pod, the process and its command line, the user, the file — and the rule’s tags, including the MITRE ATT&CK technique, which says what was being attempted rather than merely which syscall fired.

No content is stored. Falco’s rules print metadata, not file contents or request bodies: the event says what was touched, never what was inside. Since 2.6.4 a credential in Falco’s output line or fields (a password or token passed as an argument, say) is hidden when the event is stored and again when it is read, with the same filter Kobi uses; ids and digests stay readable.

falcosidekick retries a failed POST with the same body. KubeBolt derives the event’s id from the payload, so a retry never lands as a second event.

Events are a point-in-time stream: they age out of the feed after 24 hours instead of being resolved. There is no state to close, which is why the detail view offers no “resolved” button — it would lie about what the system can do. They stay stored, and readable by Kobi, until the findings retention horizon, covered in Security and compliance.

Noise

Falco’s default profile is conservative and fairly talkative. Before silencing a rule, look at the tab’s Top 5 rules by hits panel: a single rule accounting for most of the feed is usually a noisy rule, not an incident. Tuning happens in Falco (customRules), not in KubeBolt — the source is authoritative. To cut volume at the sender instead, raise falcosidekick.config.webhook.minimumpriority.