Deployment Guides
Azure AKS Guide
Deploy KubeBolt on Azure Kubernetes Service, including AGIC ingress and Azure Monitor managed Prometheus.
On KubeBolt Cloud the only thing that runs in your AKS cluster is the agent, and the Add cluster wizard prints its command. Of this page, Azure Monitor managed Prometheus is the part that may apply to you; the rest is for running KubeBolt yourself.
Quick install
The chart works out of the box on AKS — it creates a ServiceAccount and a ClusterRole granting KubeBolt read access to the cluster plus the write verbs its actions use (exec, port-forward, scale, restart, edit, delete).
helm install kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--namespace kubebolt --create-namespace --wait
kubectl -n kubebolt get secret kubebolt-admin-password \
-o jsonpath='{.data.password}' | base64 -d ; echo
kubectl -n kubebolt port-forward svc/kubebolt 3000:80
Open http://localhost:3000 and log in as admin with the printed password
(other ways to get it: Quick Start).
AKS enables Metrics Server by default. Confirm with
kubectl get apiservice v1beta1.metrics.k8s.io.
Authentication
KubeBolt needs no Azure permissions to show and operate the cluster — that goes through the Kubernetes API. Workload Identity matters for one case only: reading Azure Monitor managed Prometheus with the agent, below.
Kobi’s Azure connector (open
source) does not use Workload Identity either: it reads Azure through the Azure
MCP Server, a second container in the API pod
(mcpConnectors.azure.sidecar.enabled) that signs in as a read-only service
principal kept in a Secret.
Azure RBAC for Kubernetes. If the cluster authorizes through Azure RBAC instead of native Kubernetes RBAC, the chart’s ClusterRole still applies — the two coexist, and the chart’s ServiceAccount uses native Kubernetes RBAC.
Networking. Both kubenet and Azure CNI work with no extra configuration.
Ingress
With the Application Gateway Ingress Controller:
helm install kubebolt \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
--namespace kubebolt --create-namespace \
--set ingress.enabled=true \
--set ingress.className=azure-application-gateway \
--set ingress.hosts[0].host=kubebolt.example.com \
--set ingress.hosts[0].paths[0].path=/ \
--set ingress.hosts[0].paths[0].pathType=Prefix
With another controller, swap the class: nginx for ingress-nginx, or
webapprouting.kubernetes.azure.com for AKS’s application routing add-on:
--set ingress.className=nginx
The Ingress fronts the UI and API (the web Service proxies /api and /ws).
Agents in other clusters dial the gRPC port, kubebolt-agent-ingest on 9090,
which you expose separately — see
The agent cannot reach the backend.
Instead of an Ingress, the chart can create a Gateway API Gateway with an
HTTPRoute for the UI and a GRPCRoute for agents (gatewayAPI.enabled=true,
mutually exclusive with ingress). Its defaults target Envoy Gateway
(GatewayClass eg, plus an Envoy ClientTrafficPolicy); with another
controller set gatewayAPI.gatewayClassName and gatewayAPI.forceHTTP1=false.
Pass web.enabled=true too, or the route skips the web UI and points at the
API alone. The chart README
has the full command.
Azure Monitor managed Prometheus
If your metrics already live in an Azure Monitor Workspace, deploy the optional agent in promread mode and a separate single-replica Deployment reads the metric families straight from the workspace, authenticating with Azure Workload Identity — no second scrape pipeline and no secret to rotate.
The cluster needs three things enabled: --enable-oidc-issuer,
--enable-workload-identity, and --enable-azure-monitor-metrics (which
provisions the workspace and the ama-metrics scrapers, node-exporter and
kube-state-metrics included). Each can be turned on later with
az aks update. Check them:
RG=my-aks-resource-group
CLUSTER=my-aks-cluster
az aks show -g "$RG" -n "$CLUSTER" --query oidcIssuerProfile.issuerUrl -o tsv # an https URL
az aks show -g "$RG" -n "$CLUSTER" --query securityProfile.workloadIdentity.enabled -o tsv # true
az aks show -g "$RG" -n "$CLUSTER" --query azureMonitorProfile.metrics.enabled -o tsv # true
1. Find the endpoint
Endpoints are workspace- and region-scoped, with a random suffix Azure picks at provisioning time, so look it up rather than guessing:
# The first workspace in the subscription; pick yours if there are several
AMW_ID=$(az resource list \
--resource-type "Microsoft.Monitor/accounts" \
--query "[0].id" -o tsv)
AMW_ENDPOINT=$(az resource show --ids $AMW_ID \
--query "properties.metrics.prometheusQueryEndpoint" -o tsv)
TOKEN=$(az account get-access-token \
--resource "https://prometheus.monitor.azure.com" \
--query accessToken -o tsv)
curl -s -H "Authorization: Bearer $TOKEN" \
"${AMW_ENDPOINT}/api/v1/query?query=count(up)"
A status: success response means the endpoint and the bearer-auth path work.
2. Create the managed identity and the federated credential
Azure needs the most wiring of the three clouds, because each piece lives in a different control plane:
LOCATION=$(az aks show -g "$RG" -n "$CLUSTER" --query location -o tsv)
NAMESPACE=kubebolt-system # where the agent goes; the Add cluster wizard's default
KSA=kubebolt-agent # the chart's ServiceAccount for a release named kubebolt-agent
UAMI=kubebolt-promread
# 1. The user-assigned managed identity.
az identity create --name "$UAMI" --resource-group "$RG" --location "$LOCATION"
UAMI_CLIENT_ID=$(az identity show -g "$RG" -n "$UAMI" --query clientId -o tsv)
UAMI_PRINCIPAL_ID=$(az identity show -g "$RG" -n "$UAMI" --query principalId -o tsv)
# 2. Read-only access to the workspace.
az role assignment create \
--assignee-object-id "$UAMI_PRINCIPAL_ID" \
--assignee-principal-type ServicePrincipal \
--role "Monitoring Data Reader" \
--scope "$AMW_ID"
# 3. Trust the cluster's OIDC issuer for that ServiceAccount.
OIDC_ISSUER=$(az aks show -g "$RG" -n "$CLUSTER" \
--query "oidcIssuerProfile.issuerUrl" -o tsv)
az identity federated-credential create \
--name "${UAMI}-fed" \
--identity-name "$UAMI" \
--resource-group "$RG" \
--issuer "$OIDC_ISSUER" \
--subject "system:serviceaccount:${NAMESPACE}:${KSA}" \
--audience api://AzureADTokenExchange
The subject must name the namespace and ServiceAccount the agent runs under; if
you install it elsewhere, change NAMESPACE before running this. Verify before
moving on — a missing role surfaces later as 403 Forbidden, a subject that
doesn’t match as 401 Unauthorized:
az role assignment list --assignee "$UAMI_PRINCIPAL_ID" --scope "$AMW_ID" \
--query "[].roleDefinitionName" -o tsv # → Monitoring Data Reader
az identity federated-credential show \
--name "${UAMI}-fed" --identity-name "$UAMI" -g "$RG" \
--query "{subject:subject,issuer:issuer,audiences:audiences}"
3. Install the agent in promread mode
The kubebolt-agent-token Secret carries the ingest token (Administration →
Agents & Ingest → Agent Tokens, or the wizard), under the token key:
BACKEND_URL=agent.kubebolt.io:443 # KubeBolt Cloud; self-hosted: your agent gRPC host:port
CLUSTER_NAME=my-aks-cluster
INGEST_TOKEN='paste-the-token-here'
kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
kubectl -n "$NAMESPACE" create secret generic kubebolt-agent-token \
--from-literal=token="$INGEST_TOKEN" --dry-run=client -o yaml | kubectl apply -f -
helm upgrade --install kubebolt-agent \
oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
--namespace "$NAMESPACE" \
--set backendUrl="$BACKEND_URL" --set tls.enabled=true \
--set cluster.name="$CLUSTER_NAME" \
--set auth.mode=ingest-token \
--set auth.ingestToken.existingSecret=kubebolt-agent-token \
--set scrape.enabled=false \
--set agent.promRead.enabled=true \
--set agent.promRead.url="${AMW_ENDPOINT}" \
--set agent.promRead.auth.mode=azureWorkloadIdentity \
--set serviceAccount.annotations."azure\.workload\.identity/client-id"="${UAMI_CLIENT_ID}" \
--set-string podLabels."azure\.workload\.identity/use"="true"
Notes that catch people out:
podLabels."azure\.workload\.identity/use"is what makes the Workload Identity webhook inject the token into the pod. Pass it with--set-string(a label value must be the string"true"). Without it the promread pod exits at start withpromread init failed.backendUrlis a gRPC dial target,host:port, no scheme. With a TLS-terminating proxy in front of the backend,tls.enabled=trueis required as well — omitting it looks like the host is unreachable. For an agent in the same cluster as a self-hosted backend, use the in-cluster Service (kubebolt-agent-ingest.kubebolt.svc.cluster.local:9090) without TLS.agent.promRead.urlis the base endpoint; the agent appends/api/v1/query_rangeitself.scrape.enabled=falsekeeps the bundledvmagentsidecar off (it and promread are mutually exclusive), not the DaemonSet’s kubelet collectors.- The chart auto-defers the in-agent node network and NodeStress (load +
PSI) collectors when
agent.promRead.auth.mode=azureWorkloadIdentity, because Azure’sama-metrics-nodealready scrapes node-exporter. You don’t needagent.deferNodeStress=trueoragent.deferNodeNetwork=trueby hand; drop them if you carried them forward from an older values file.
The Add cluster wizard builds most of this command: in its advanced options,
pick Read from existing Prometheus under Prometheus integration, auth
Azure Workload Identity, and add the azure.workload.identity/client-id
ServiceAccount annotation. It has no field for pod labels, so append the
--set-string podLabels… line yourself. See
Connecting clusters and Metrics.
4. Verify
kubectl -n "$NAMESPACE" get pods -l kubebolt.dev/role=promread # 1 pod, Running
kubectl -n "$NAMESPACE" get lease kubebolt-promread # HOLDER is that pod
kubectl -n "$NAMESPACE" logs -l kubebolt.dev/role=promread --tail=50 \
| grep -E "promread enabled|promread init failed|channel registered|matcher failed"
You want promread enabled … auth_mode=azureWorkloadIdentity and
channel registered. promread matcher failed with 401 points at the
federated credential’s subject, with 403 at the role assignment. In KubeBolt,
the Prometheus (read) card in Administration → Agents & Ingest →
Integrations turns Installed after the first poll.
Troubleshooting
API pod in CrashLoopBackOff. Check kubectl -n kubebolt logs -l app.kubernetes.io/component=api.
The usual cause is a missing ClusterRoleBinding — verify with
kubectl get clusterrolebinding | grep kubebolt.
Image pull errors behind a private ACR. Allow ghcr.io (and docker.io for
VictoriaMetrics), or mirror the images. The tags below are the ones chart
2.7.0 and agent chart 1.4.10 pull; match them to the charts you install:
ACR=myacr
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/api:2.7.0 --image kubebolt/api:2.7.0
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/web:2.7.0 --image kubebolt/web:2.7.0
az acr import --name $ACR --source docker.io/victoriametrics/victoria-metrics:v1.156.0-scratch \
--image victoriametrics/victoria-metrics:v1.156.0-scratch
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/agent:1.4.10 --image kubebolt/agent:1.4.10
Then point the charts at them: api.image.repository=$ACR.azurecr.io/kubebolt/api,
web.image.repository, metrics.storage.embedded.image.repository and, on the
agent chart, image.repository.
403 on some resources. Expected, and handled: KubeBolt degrades gracefully. Restricted resources are dimmed in the sidebar behind a “Limited access” banner. See RBAC & Permissions.