Kubernetes OpenBao Secrets Operator

June 25, 2025

Build k8s cluster

Out of scope 🙃

Build OpenBao

Easy Mode: https://github.com/thejambavan/openbao-compose Ouroboros Mode: https://openbao.org/docs/platform/k8s/

Add BSO to k8s cluster

Install the secrets operator helm chart:

helm install vault-secrets-operator hashicorp/vault-secrets-operator --namespace openbao-secrets-operator --create-namespace

Unfortunately you can’t yet do this because the openbao version hasn’t been uploaded to the helm repo:

helm install openbao-secrets-operator openbao/openbao-secrets-operator --namespace openbao-secrets-operator --create-namespace

k8s configuration

Create an appropriate namespace and connect k8s ↔️ openbao

kubectl create namespace openbao

export K8S_TO_VAULT=bao.tunstall.in

curl -vk https://$K8S_TO_VAULT

Note: SSH port-forwarding tricks do not work here!

kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: openbao
  name: openbao-connection
spec:
  # address to the Vault server.
  address: https://$K8S_TO_VAULT:443
  # skipTLSVerify: true # enable if doing dev stuff
---
EOF

if you made a mistake and need to delete it:

kubectl delete VaultConnection openbao-connection -n openbao

kubectl describe --namespace openbao vaultconnection.secrets.hashicorp.com/openbao-connection

Service account

kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: openbao-to-k8s-sa
  namespace: openbao
---
apiVersion: v1
kind: Secret
metadata:
  name: openbao-to-k8s-sa-secret
  namespace: openbao
  annotations:
    kubernetes.io/service-account.name: openbao-to-k8s-sa
type: kubernetes.io/service-account-token
---
EOF

Role for the above service account

kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: openbao-to-k8s-sa-tokenreview-role
  namespace: openbao
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: openbao-to-k8s-sa
    namespace: openbao
EOF

Create k8s auth tokens

variously get k8s ca cert and auth token for vso. SSH port-forward tricks work here.

export VSO_CERT=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data."ca.crt"'| base64 -d)
export VSO_TOKEN=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data."token"'| base64 -d)

export VAULTSA_SECRET=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data')

verify they’re correct:

echo -e "${VSO_CERT}\n${VSO_TOKEN}\n${VAULTSA_SECRET}"

Give tokens to openbao

copypasta those vars to a bao host or something else with the bao cli binary and then configure k8s auth with them.

Configure bao env vars and credentials to point it at your cluster

export VAULT_ADDR="https://bao.tunstall.in"
export VAULT_SKIP_VERIFY=true # unnecessary if you've got a proper letsencrypt cert
bao login # you'll find ths root token in `/var/lib/docker/volumes/openbao_tls-certs/_data/init.json` if using the dev/test docker-compose yaml

$ bao auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

$ bao write auth/kubernetes/config token_reviewer_jwt="$VSO_TOKEN" kubernetes_host="https://k3s-test:6443" kubernetes_ca_cert="$VSO_CERT"
Success! Data written to: auth/kubernetes/config

You can just download the OpenBao binary (bao) and run it from anywhere that can reach the OpenBao cluster. https://openbao.org/downloads/

The k8s authentication method can be viewed but not configured from the UI.

Create OpenBao static secrets engine on `/static`

This differs from Vault, where the kv engine is named kv2. OpenBao never had a “version 1” of this, so it’s just called kv.

bao secrets enable -version=2 -path=static kv
Success! Enabled the kv secrets engine at: static/

Store an example secret

bao kv put static/exampleapp/creds username='testuser' password='password123'

======== Secret Path ========
static/data/exampleapp/creds

======= Metadata =======
Key                Value
---                -----
created_time       2025-06-25T09:51:27.515299225Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Set a policy to allow the secret to be read by k8s

bao policy write exampleapp-kv-read - << EOF
path "static/data/exampleapp/creds" {
  capabilities = ["read"]
}
EOF

Success! Uploaded policy: exampleapp-kv-read

Set an openbao role for k8s vso

bao write auth/kubernetes/role/openbao-role-static-exampleapp bound_service_account_names=vso-static-exampleapp-sa bound_service_account_namespaces=static-exampleapp policies=exampleapp-kv-read ttl=1h

Create k8s example secret mapped to a vault secret

namespace and service account

kubectl create namespace static-exampleapp

kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vso-static-exampleapp-sa
  namespace: static-exampleapp
---
EOF

VaultAuth resource

kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: openbao-auth-vso-static-exampleapp
  namespace: static-exampleapp
spec:
  vaultConnectionRef: openbao/openbao-connection
  method: kubernetes
  mount: kubernetes
  allowedNamespaces:
    - static-exampleapp
  kubernetes:
    role: openbao-role-static-exampleapp
    serviceAccount: vso-static-exampleapp-sa
---
EOF

VaultStaticSecret resource

kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: openbao-static-secret
  namespace: static-exampleapp
spec:
  vaultAuthRef: openbao-auth-vso-static-exampleapp
  mount: static
  type: kv-v2
  path:  exampleapp/creds
  refreshAfter: 300s
  destination:
    create: true
    name: vso-static-creds-from-vault
---
EOF

Retrieve Secret

At this point you can retrieve the native k8s secret called openbao-static-secret which is mapped to the vault secret static/exampleapp/creds:

$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"data\":{\"password\":\"asdfasdfasdfasdfasdf\",\"username\":\"testuser_version2\"},\"metadata\":{\"created_time\":\"2024-10-29T16:20:35.670041844Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":3}}",
  "password": "asdfasdfasdfasdfasdf",
  "username": "testuser_version2"

…and then after changing the secret in Bao and waiting for it to sync, getting the now-changed version. Note the incremented version key:

$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
  "_raw": "{\"data\":{\"password\":\"solarwinds123\",\"username\":\"test_user\"},\"metadata\":{\"created_time\":\"2024-10-29T16:23:50.338672408Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":4}}",
  "password": "solarwinds123",
  "username": "test_user"

Notes

You MUST HAVE connectivity between k8s 👉 OpenBao. SSH port forwards WILL NOT work unless you do heavy DNS trickery.
You MUST HAVE connectivity between OpenBao 👉 k8s. SSH port forwards WILL work for this, however bear in mind that ALL vault nodes need to be able to get to the k8s API otherwise you’ll see errors like this:

{"level":"error","ts":"2024-10-29T16:11:14Z","logger":"cachingClientFactory","msg":"Failed to get NewClientWithLogin","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","VaultStaticSecret":{"name":"openbao-static-secret","namespace":"static-exampleapp"},"namespace":"static-exampleapp","name":"openbao-static-secret","reconcileID":"4474bced-6d91-44c6-92e9-af0ce232c747","cacheKey":"kubernetes-9f0b4138ae774cf7ad4cd7","error":"Error making API request.\n\nURL: PUT https://bao.tunstall.in:443/v1/auth/kubernetes/login\nCode: 403. Errors:\n\n* permission denied"}

This is not actually a “permission denied” as such. What it actually means is “Vault/Bao is unable to reach k8s to perform an AuthNZ”