Security
Falco: runtime events
Falco is the only source that pushes into KubeBolt. Wire falcosidekick to the ingest endpoint with a cluster-scoped token — anything else is a 403.
On this page
Falco watches every node’s syscalls and speaks up when something steps outside the expected: a shell inside a container, a sensitive file read, a binary dropped and executed. Events land in the Runtime tab of Security & Compliance.
Unlike Trivy and Kyverno, KubeBolt does not go looking for this data — Falco pushes it. It is the only source that works that way, and that shapes everything about the configuration. falcosidekick, the forwarder the Falco chart deploys next to it, POSTs each event to KubeBolt’s API; the agent’s tunnel does not carry it.
The three things it needs
| Value | Where it comes from | Without it |
|---|---|---|
| Ingest URL | Self-hosted: https://<your-kubebolt>/api/v1/ingest/falco. Cloud: https://api.kubebolt.io/api/v1/ingest/falco | falcosidekick logs a connection error and drops the event |
| Ingest token | Administration → Agents & Ingest → Agent Tokens, cluster-scoped (a kb_… ingest token, not a REST API token) | 401 — nothing is accepted |
cluster_id | the kube-system namespace UID | Optional. If set and it disagrees with the token, 403 |
The URL has to be reachable from inside the cluster. On Cloud that means
outbound HTTPS to api.kubebolt.io — a different host from the agent’s
agent.kubebolt.io, so an egress allowlist needs both. Self-hosted, when Falco
runs in the same cluster as KubeBolt, the API Service works without leaving the
cluster: http://<kubebolt-api-service>.<namespace>.svc.cluster.local:8080/api/v1/ingest/falco
(the Service is kubebolt-api for a release named kubebolt).
The tenant is never sent — it comes from the token. A self-hosted install has a single tenant; on Cloud this is also what stops a sender from writing into another organization’s feed.
Self-hosted, the endpoint needs authentication enabled
(KUBEBOLT_AUTH_ENABLED=true, auth.enabled=true in the chart — the default).
Ingest tokens and the event store don’t exist without it, and the endpoint
answers 503 runtime events are not available (persistence disabled).
The token must be cluster-scoped, or every event is rejected. Falco has no handshake. The KubeBolt agent announces its cluster when it connects, but a pushed event carries only the token, so an unscoped token produces events that cannot be attributed to a cluster. The API refuses rather than filing them half-attributed:
403 this ingest token is not scoped to a cluster; issue a cluster-scoped token
for Falco so its events can be attributedIssue a token for this specific cluster — in Issue token, pick the cluster under Cluster scope instead of “Any cluster”. Do not reuse the agent’s.
Issuing tokens takes the admin role. In the Cluster scope dropdown only a cluster that has already reported its id can be picked, so connect the cluster first; it defaults to the cluster selected in the top bar. Expires after (days) is optional — leave it blank for a token that never expires, and revoke it from the same page.
Get the cluster id — the identity KubeBolt uses everywhere:
kubectl get ns kube-system -o jsonpath='{.metadata.uid}'
Install
CLUSTER_ID=$(kubectl get ns kube-system -o jsonpath='{.metadata.uid}')
TOKEN=<your cluster-scoped ingest token>
# The token goes in a Secret, not in the Helm values (where anyone who can run
# `helm get values` reads it). falcosidekick loads it after its own Secret.
kubectl create namespace falco --dry-run=client -o yaml | kubectl apply -f -
kubectl -n falco create secret generic falco-kubebolt \
--from-literal=WEBHOOK_CUSTOMHEADERS="Authorization:Bearer $TOKEN" \
--dry-run=client -o yaml | kubectl apply -f -
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm upgrade --install falco falcosecurity/falco -n falco \
--set driver.kind=modern_ebpf \
--set falcosidekick.enabled=true \
--set-string falcosidekick.config.existingSecret=falco-kubebolt \
--set-string falcosidekick.config.webhook.address=https://<your-kubebolt>/api/v1/ingest/falco \
--set-string falcosidekick.config.customfields="cluster_id:$CLUSTER_ID"
On Cloud, the address is https://api.kubebolt.io/api/v1/ingest/falco.
Use chart 9.2.1 or later (Falco 0.45.1+); on kernel 6.18 older versions do not start (see The driver).
| Value | What it does |
|---|---|
driver.kind | How Falco reads syscalls: modern_ebpf (recommended), ebpf or kmod |
falcosidekick.enabled | Deploys falcosidekick, which forwards each event |
falcosidekick.config.existingSecret | Secret falcosidekick loads its settings from — here, the Authorization header |
falcosidekick.config.webhook.address | The ingest URL |
falcosidekick.config.customfields | Optional cluster_id:<uid> check: the token decides the cluster, and a cluster_id that disagrees with it is refused |
falcosidekick.config.webhook.minimumpriority | Optional. Forward only events at or above a priority, e.g. warning |
To rotate the token, re-run the create secret line with the new one and
restart falcosidekick (kubectl -n falco rollout restart deploy/falco-falcosidekick):
it reads the Secret only when it starts. To check what it sends:
kubectl get secret -n falco falco-kubebolt \
-o jsonpath='{.data.WEBHOOK_CUSTOMHEADERS}' | base64 -d
Passing the header in the values instead (webhook.customHeaders) puts the
token in the release values, and it has to be customHeaders, with a capital
H: that is the only spelling the falcosidekick chart’s template reads for the
webhook, while other outputs in the same values.yaml (Elasticsearch, Loki)
spell it customheaders. The lowercase key leaves the header empty with no
warning at all, the POST goes out without a token, and the API answers
401 — and the obvious diagnosis, the wrong one, is that the token is bad.
The driver
modern_ebpf (CO-RE) needs kernel 5.8 or newer with BTF:
uname -r # 5.8+
ls /sys/kernel/btf/vmlinux # must exist
If it is not available, the chart also offers ebpf and kmod.
Kernel 6.18 needs Falco 0.45.1 or later (chart 9.2.1+). Earlier versions with
modern_ebpf crash-loop there: the BPF verifier rejects the signal_deliver
program (R9 invalid mem access 'scalar'). Amazon EKS AL2023 node images run 6.18
from November 2026. Fixed in falcosecurity/libs
(#3119); verified with chart
9.2.2 / Falco 0.45.2.
One sensor per kernel
Falco is a DaemonSet: one pod per node, each seeing its own kernel. Correct on any real cluster.
On kind or k3d it is not. The “nodes” are containers sharing a single Docker kernel, so every Falco sees the whole machine’s syscalls and the same event arrives twice — one copy with the Kubernetes identity resolved, another without it, from the node that does not host the pod. For a lab, pin it to one node:
helm upgrade falco falcosecurity/falco -n falco --reuse-values \
--set-string nodeSelector."kubernetes\.io/hostname"=<your-node>
No detection is lost — the remaining sensor already sees those syscalls.
How KubeBolt finds it
By the label app.kubernetes.io/name=falco, falling back to pods named
falco* in the falco or falco-system namespaces. The result is the
Falco card under Administration → Agents & Ingest → Integrations, for
the cluster selected in the top bar. The card counts Falco’s pods and their
readiness; whether events actually arrive shows only on the Runtime tab and in
falcosidekick’s log.
Verify
# 1. Trip a rule on purpose
kubectl exec -n <ns> deploy/<something> -- cat /etc/shadow
# 2. Delivery must report 202
kubectl logs -n falco -l app.kubernetes.io/name=falcosidekick --tail=5
# → Webhook - POST OK (202)
The event lands in the Runtime tab in under a minute.
| Symptom | Cause |
|---|---|
401 missing Bearer token | existingSecret was left out of the install, or the Secret’s key is not exactly WEBHOOK_CUSTOMHEADERS; with the header in the values, lowercase customheaders |
falcosidekick stuck in CreateContainerConfigError | The falco-kubebolt Secret doesn’t exist in the falco namespace. Create it and the pod starts on its own |
401 invalid ingest token | The token is revoked, expired, or belongs to another install or organization |
503 runtime events are not available (persistence disabled) | Self-hosted with authentication disabled |
403 not scoped to a cluster | The token was issued without a cluster |
403 cluster_id does not match | The token belongs to a different cluster than customfields |
400 unrecognized falco payload | The body is not a falcosidekick webhook event: not JSON, or neither rule nor output in it |
connection refused or a timeout | The URL is not reachable from the cluster (egress rules, wrong host, or the in-cluster Service name) |
| Nothing in Falco’s own log | The driver never loaded — check kernel and BTF |
Falco in CrashLoopBackOff, R9 invalid mem access 'scalar' in its log | Falco older than 0.45.1 on kernel 6.18 — upgrade the chart to 9.2.1+ |
What is stored, and what is not
An event stores what falcosidekick sends and nothing more: the rule, the
priority, the time, Falco’s output line, the hostname, every field of the
event’s output_fields — namespace and pod, the process and its command line,
the user, the file — and the rule’s tags, including the MITRE ATT&CK
technique, which says what was being attempted rather than merely which
syscall fired.
No content is stored. Falco’s rules print metadata, not file contents or request bodies: the event says what was touched, never what was inside. Since 2.6.4 a credential in Falco’s output line or fields (a password or token passed as an argument, say) is hidden when the event is stored and again when it is read, with the same filter Kobi uses; ids and digests stay readable.
falcosidekick retries a failed POST with the same body. KubeBolt derives the event’s id from the payload, so a retry never lands as a second event.
Events are a point-in-time stream: they age out of the feed after 24 hours instead of being resolved. There is no state to close, which is why the detail view offers no “resolved” button — it would lie about what the system can do. They stay stored, and readable by Kobi, until the findings retention horizon, covered in Security and compliance.
Noise
Falco’s default profile is conservative and fairly talkative. Before silencing
a rule, look at the tab’s Top 5 rules by hits panel: a single rule
accounting for most of the feed is usually a noisy rule, not an incident.
Tuning happens in Falco (customRules), not in KubeBolt — the source is
authoritative. To cut volume at the sender instead, raise
falcosidekick.config.webhook.minimumpriority.