OpenShift

Blocker 1 — kubebolt-web crash-loops

A standard install leaves the web container in CrashLoopBackOff while everything else runs:

NAME                            READY   STATUS             RESTARTS
kubebolt-api-...                1/1     Running            0
kubebolt-victoriametrics-0      1/1     Running            0
kubebolt-web-...                0/1     CrashLoopBackOff   7

oc logs deployment/kubebolt-web

/docker-entrypoint.sh: Launching /docker-entrypoint.d/40-api-backend.sh
sed: can't create temp file '/etc/nginx/conf.d/default.confXXXXXX': Permission denied

Why

restricted-v2 assigns each pod a random UID from the namespace’s range (openshift.io/sa.scc.uid-range) and always puts it in GID 0. The web image is stock nginx:1.27-alpine, where every path nginx writes to is root:root 0755 — group-readable, not group-writable.

Four paths are denied under an arbitrary UID:

The sed is only the first one reached. Fixing it alone is not enough — nginx then dies one step later with mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied).

Two plausible theories the evidence rules out:

• It is not the volumes. PersistentVolumeClaims are unaffected: restricted-v2 supplies an fsGroup from the namespace range and the kubelet chowns the volume to it. Both the VictoriaMetrics PVC and the API’s /data PVC work untouched, which is exactly why those two pods are Running.
• A securityContext in the chart cannot fix it, and adding runAsUser actively breaks it: restricted-v2 uses the MustRunAsRange strategy, so a pod requesting a UID outside the namespace’s allocated range is rejected at admission. Under restricted-v2 the field must stay empty. The fix belongs in the image.

Workaround A — grant the anyuid SCC

Fastest path. Needs cluster-admin. The pod runs as root, exactly as it does on vanilla Kubernetes.

oc adm policy add-scc-to-user anyuid -z default -n kubebolt
oc rollout restart deployment/kubebolt-web -n kubebolt

-z default is correct today: the web Deployment sets no serviceAccountName, so it runs under the namespace’s default ServiceAccount while the API runs under the chart’s. That inconsistency is itself on the fix list.

This weakens the namespace’s security posture. Prefer B if your platform team will not grant anyuid.

Workaround B — keep restricted-v2

An initContainer renders the nginx config into an emptyDir, the main container bypasses the entrypoint scripts, and the three remaining write paths become emptyDir mounts — writable because the SCC’s fsGroup applies to them.

oc patch deployment/kubebolt-web -n kubebolt --type=strategic -p '
spec:
  template:
    spec:
      initContainers:
        - name: render-config
          image: ghcr.io/clm-cloud-solutions/kubebolt/web:2.1.0
          command: ["/bin/sh", "-c"]
          args:
            - sed "s|api:8080|${API_BACKEND}|g" /etc/nginx/conf.d/default.conf > /mnt/conf/default.conf
          env:
            - name: API_BACKEND
              value: kubebolt-api:8080
          volumeMounts:
            - name: nginx-conf
              mountPath: /mnt/conf
      containers:
        - name: web
          command: ["nginx", "-g", "daemon off;"]
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/conf.d
            - name: nginx-cache
              mountPath: /var/cache/nginx
            - name: nginx-log
              mountPath: /var/log/nginx
            - name: nginx-run
              mountPath: /run
      volumes:
        - name: nginx-conf
          emptyDir: {}
        - name: nginx-cache
          emptyDir: {}
        - name: nginx-log
          emptyDir: {}
        - name: nginx-run
          emptyDir: {}
'

Pin the initContainer image to the tag you installed, and set API_BACKEND to <release-name>-api:8080.

The agent has the opposite problem

The agent chart does set a security context, but pins a UID restricted-v2 will reject:

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 65532      # rejected: outside the namespace's UID range

Drop the UID at install time and let OpenShift assign one. runAsNonRoot: true still holds, because the assigned UID is never 0:

helm upgrade --install kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  --namespace kubebolt \
  --set podSecurityContext.runAsUser=null \
  --set backendUrl=<host:port>

--set-json 'podSecurityContext={}' does not work — Helm merges maps, so an empty map leaves the pinned UID in place. Only =null removes the key.

Nothing else in the agent needs elevated privileges: the DaemonSet declares no hostPath, hostNetwork or hostPID, and readOnlyRootFilesystem: true is compatible with restricted-v2 as-is.

Blocker 2 — the in-cluster context times out

Independent of the SCC problem. The API detects its in-cluster config correctly, reaches the API server, and finishes the permission probe — then, about 25 seconds later:

connect timed out after 25s for context in-cluster — agent may be stuck

The UI shows Cluster unreachable even though nothing is unreachable, and the message names an agent that is not in the picture.

The isolation that found it

The reporter removed only events from the get/list/watch rule of the kubebolt ClusterRole, left Pods and Nodes authorised, and restarted:

On that cluster, oc get events -A took ~32 s and GET /api/v1/events returned ~138 MiB. Nodes, Pods and Deployments answered in under a second each. DNS, authentication and every other authenticated call worked.

Why

Three things compound:

1. Events are unbounded in a way nothing else is. Everything else is bounded by what is deployed; Events are bounded by what happens. 138 MiB at a one-hour TTL is roughly 90,000 objects. OpenShift’s own operators are chatty before you add a workload.
2. One slow informer blocks every other one. The connector waits on the whole shared factory, so it returns only when the last informer is ready.
3. The outer deadline is shorter than the inner budget. The connect deadline is 25 s while the cache-sync budget is 45 s, so a sync progressing normally toward 30 s is killed at 25. The inner budget can never be spent.

Workarounds

• Drop events from the ClusterRole. What the reporter did. The cluster comes up immediately; the cost is that the Events tab and any insight that reads Events go empty, and the probe honestly reports one fewer resource type.
• Raise the connect timeout — KUBEBOLT_AGENT_PROXY_CONNECT_TIMEOUT (default 25s), or Administration → Agents & Ingest → Configuration. It works, but it is the wrong knob twice over: it is an agent-proxy resilience setting, and loosening it weakens the protection it exists for on clusters that do use agents.

Neither is a fix.

Reproducing without an OpenShift cluster

Run the image the way restricted-v2 would — any UID works as long as it is not in /etc/passwd and the GID is 0, which is the whole of what OpenShift does differently:

docker run --rm --user 1000700000:0 --entrypoint /bin/sh \
  ghcr.io/clm-cloud-solutions/kubebolt/web:2.1.0 -c '
    sed -i "s|api:8080|x:1|g" /etc/nginx/conf.d/default.conf
    ls -ld /etc/nginx/conf.d /var/cache/nginx /var/log/nginx /run
  '

Pending fixes

Tracked, not shipped:

1. Make the web image tolerate an arbitrary UID — chgrp -R 0 plus chmod -R g=u on the four write paths. Verified against a patched image: it runs as UID 1000700000:0, serves HTTP 200, substitutes API_BACKEND, and still works unchanged on vanilla Kubernetes and Docker Compose.
2. Make the agent’s runAsUser omittable, so the chart installs on OpenShift without =null.
3. Give the web Deployment a serviceAccountName, so SCC grants land on the right subject.
4. Expose optional podSecurityContext on the api and web Deployments, defaulting to empty. Never ship a hardcoded runAsUser.
5. Stop one informer blocking the rest — bound the Events informer, sync it asynchronously, or wait per-informer so a slow one degrades its own tab.
6. Make the outer connect deadline at least the inner sync budget.
7. Stop applying the agent-proxy connect deadline to the in-cluster path, and stop saying “agent may be stuck” where there is no agent.

References

• Managing security context constraints — the MustRunAsRange strategy and the namespace UID range
• Pod Security Standards in OpenShift 4.11+