KubeBolt docs
ENES GitHub

Deployment Guides

Azure AKS Guide

Deploy KubeBolt on Azure Kubernetes Service, including AGIC ingress and Azure Monitor managed Prometheus.

On this page
  1. Quick install
  2. Authentication
  3. Ingress
  4. Azure Monitor managed Prometheus
  5. Troubleshooting

On KubeBolt Cloud the only thing that runs in your AKS cluster is the agent, and the Add cluster wizard prints its command. Of this page, Azure Monitor managed Prometheus is the part that may apply to you; the rest is for running KubeBolt yourself.

Quick install

The chart works out of the box on AKS — it creates a ServiceAccount and a ClusterRole granting KubeBolt read access to the cluster plus the write verbs its actions use (exec, port-forward, scale, restart, edit, delete).

helm install kubebolt oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  --namespace kubebolt --create-namespace --wait

kubectl -n kubebolt get secret kubebolt-admin-password \
  -o jsonpath='{.data.password}' | base64 -d ; echo

kubectl -n kubebolt port-forward svc/kubebolt 3000:80

Open http://localhost:3000 and log in as admin with the printed password (other ways to get it: Quick Start).

AKS enables Metrics Server by default. Confirm with kubectl get apiservice v1beta1.metrics.k8s.io.

Authentication

KubeBolt needs no Azure permissions to show and operate the cluster — that goes through the Kubernetes API. Workload Identity matters for one case only: reading Azure Monitor managed Prometheus with the agent, below.

Kobi’s Azure connector (open source) does not use Workload Identity either: it reads Azure through the Azure MCP Server, a second container in the API pod (mcpConnectors.azure.sidecar.enabled) that signs in as a read-only service principal kept in a Secret.

Azure RBAC for Kubernetes. If the cluster authorizes through Azure RBAC instead of native Kubernetes RBAC, the chart’s ClusterRole still applies — the two coexist, and the chart’s ServiceAccount uses native Kubernetes RBAC.

Networking. Both kubenet and Azure CNI work with no extra configuration.

Ingress

With the Application Gateway Ingress Controller:

helm install kubebolt \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt \
  --namespace kubebolt --create-namespace \
  --set ingress.enabled=true \
  --set ingress.className=azure-application-gateway \
  --set ingress.hosts[0].host=kubebolt.example.com \
  --set ingress.hosts[0].paths[0].path=/ \
  --set ingress.hosts[0].paths[0].pathType=Prefix

With another controller, swap the class: nginx for ingress-nginx, or webapprouting.kubernetes.azure.com for AKS’s application routing add-on:

  --set ingress.className=nginx

The Ingress fronts the UI and API (the web Service proxies /api and /ws). Agents in other clusters dial the gRPC port, kubebolt-agent-ingest on 9090, which you expose separately — see The agent cannot reach the backend.

Instead of an Ingress, the chart can create a Gateway API Gateway with an HTTPRoute for the UI and a GRPCRoute for agents (gatewayAPI.enabled=true, mutually exclusive with ingress). Its defaults target Envoy Gateway (GatewayClass eg, plus an Envoy ClientTrafficPolicy); with another controller set gatewayAPI.gatewayClassName and gatewayAPI.forceHTTP1=false. Pass web.enabled=true too, or the route skips the web UI and points at the API alone. The chart README has the full command.

Azure Monitor managed Prometheus

If your metrics already live in an Azure Monitor Workspace, deploy the optional agent in promread mode and a separate single-replica Deployment reads the metric families straight from the workspace, authenticating with Azure Workload Identity — no second scrape pipeline and no secret to rotate.

The cluster needs three things enabled: --enable-oidc-issuer, --enable-workload-identity, and --enable-azure-monitor-metrics (which provisions the workspace and the ama-metrics scrapers, node-exporter and kube-state-metrics included). Each can be turned on later with az aks update. Check them:

RG=my-aks-resource-group
CLUSTER=my-aks-cluster

az aks show -g "$RG" -n "$CLUSTER" --query oidcIssuerProfile.issuerUrl -o tsv                    # an https URL
az aks show -g "$RG" -n "$CLUSTER" --query securityProfile.workloadIdentity.enabled -o tsv       # true
az aks show -g "$RG" -n "$CLUSTER" --query azureMonitorProfile.metrics.enabled -o tsv            # true

1. Find the endpoint

Endpoints are workspace- and region-scoped, with a random suffix Azure picks at provisioning time, so look it up rather than guessing:

# The first workspace in the subscription; pick yours if there are several
AMW_ID=$(az resource list \
  --resource-type "Microsoft.Monitor/accounts" \
  --query "[0].id" -o tsv)

AMW_ENDPOINT=$(az resource show --ids $AMW_ID \
  --query "properties.metrics.prometheusQueryEndpoint" -o tsv)

TOKEN=$(az account get-access-token \
  --resource "https://prometheus.monitor.azure.com" \
  --query accessToken -o tsv)

curl -s -H "Authorization: Bearer $TOKEN" \
  "${AMW_ENDPOINT}/api/v1/query?query=count(up)"

A status: success response means the endpoint and the bearer-auth path work.

2. Create the managed identity and the federated credential

Azure needs the most wiring of the three clouds, because each piece lives in a different control plane:

LOCATION=$(az aks show -g "$RG" -n "$CLUSTER" --query location -o tsv)
NAMESPACE=kubebolt-system   # where the agent goes; the Add cluster wizard's default
KSA=kubebolt-agent          # the chart's ServiceAccount for a release named kubebolt-agent
UAMI=kubebolt-promread

# 1. The user-assigned managed identity.
az identity create --name "$UAMI" --resource-group "$RG" --location "$LOCATION"

UAMI_CLIENT_ID=$(az identity show -g "$RG" -n "$UAMI" --query clientId -o tsv)
UAMI_PRINCIPAL_ID=$(az identity show -g "$RG" -n "$UAMI" --query principalId -o tsv)

# 2. Read-only access to the workspace.
az role assignment create \
  --assignee-object-id "$UAMI_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Monitoring Data Reader" \
  --scope "$AMW_ID"

# 3. Trust the cluster's OIDC issuer for that ServiceAccount.
OIDC_ISSUER=$(az aks show -g "$RG" -n "$CLUSTER" \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

az identity federated-credential create \
  --name "${UAMI}-fed" \
  --identity-name "$UAMI" \
  --resource-group "$RG" \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:${NAMESPACE}:${KSA}" \
  --audience api://AzureADTokenExchange

The subject must name the namespace and ServiceAccount the agent runs under; if you install it elsewhere, change NAMESPACE before running this. Verify before moving on — a missing role surfaces later as 403 Forbidden, a subject that doesn’t match as 401 Unauthorized:

az role assignment list --assignee "$UAMI_PRINCIPAL_ID" --scope "$AMW_ID" \
  --query "[].roleDefinitionName" -o tsv          # → Monitoring Data Reader

az identity federated-credential show \
  --name "${UAMI}-fed" --identity-name "$UAMI" -g "$RG" \
  --query "{subject:subject,issuer:issuer,audiences:audiences}"

3. Install the agent in promread mode

The kubebolt-agent-token Secret carries the ingest token (Administration → Agents & Ingest → Agent Tokens, or the wizard), under the token key:

BACKEND_URL=agent.kubebolt.io:443   # KubeBolt Cloud; self-hosted: your agent gRPC host:port
CLUSTER_NAME=my-aks-cluster
INGEST_TOKEN='paste-the-token-here'

kubectl create namespace "$NAMESPACE" --dry-run=client -o yaml | kubectl apply -f -
kubectl -n "$NAMESPACE" create secret generic kubebolt-agent-token \
  --from-literal=token="$INGEST_TOKEN" --dry-run=client -o yaml | kubectl apply -f -

helm upgrade --install kubebolt-agent \
  oci://ghcr.io/clm-cloud-solutions/kubebolt/helm/kubebolt-agent \
  --namespace "$NAMESPACE" \
  --set backendUrl="$BACKEND_URL" --set tls.enabled=true \
  --set cluster.name="$CLUSTER_NAME" \
  --set auth.mode=ingest-token \
  --set auth.ingestToken.existingSecret=kubebolt-agent-token \
  --set scrape.enabled=false \
  --set agent.promRead.enabled=true \
  --set agent.promRead.url="${AMW_ENDPOINT}" \
  --set agent.promRead.auth.mode=azureWorkloadIdentity \
  --set serviceAccount.annotations."azure\.workload\.identity/client-id"="${UAMI_CLIENT_ID}" \
  --set-string podLabels."azure\.workload\.identity/use"="true"

Notes that catch people out:

The Add cluster wizard builds most of this command: in its advanced options, pick Read from existing Prometheus under Prometheus integration, auth Azure Workload Identity, and add the azure.workload.identity/client-id ServiceAccount annotation. It has no field for pod labels, so append the --set-string podLabels… line yourself. See Connecting clusters and Metrics.

4. Verify

kubectl -n "$NAMESPACE" get pods -l kubebolt.dev/role=promread   # 1 pod, Running
kubectl -n "$NAMESPACE" get lease kubebolt-promread              # HOLDER is that pod
kubectl -n "$NAMESPACE" logs -l kubebolt.dev/role=promread --tail=50 \
  | grep -E "promread enabled|promread init failed|channel registered|matcher failed"

You want promread enabled … auth_mode=azureWorkloadIdentity and channel registered. promread matcher failed with 401 points at the federated credential’s subject, with 403 at the role assignment. In KubeBolt, the Prometheus (read) card in Administration → Agents & Ingest → Integrations turns Installed after the first poll.

Troubleshooting

API pod in CrashLoopBackOff. Check kubectl -n kubebolt logs -l app.kubernetes.io/component=api. The usual cause is a missing ClusterRoleBinding — verify with kubectl get clusterrolebinding | grep kubebolt.

Image pull errors behind a private ACR. Allow ghcr.io (and docker.io for VictoriaMetrics), or mirror the images. The tags below are the ones chart 2.7.0 and agent chart 1.4.10 pull; match them to the charts you install:

ACR=myacr
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/api:2.7.0 --image kubebolt/api:2.7.0
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/web:2.7.0 --image kubebolt/web:2.7.0
az acr import --name $ACR --source docker.io/victoriametrics/victoria-metrics:v1.156.0-scratch \
  --image victoriametrics/victoria-metrics:v1.156.0-scratch
az acr import --name $ACR --source ghcr.io/clm-cloud-solutions/kubebolt/agent:1.4.10 --image kubebolt/agent:1.4.10

Then point the charts at them: api.image.repository=$ACR.azurecr.io/kubebolt/api, web.image.repository, metrics.storage.embedded.image.repository and, on the agent chart, image.repository.

403 on some resources. Expected, and handled: KubeBolt degrades gracefully. Restricted resources are dimmed in the sidebar behind a “Limited access” banner. See RBAC & Permissions.