Kubernetes OpenBao Secrets Operator
Build k8s cluster
Out of scope 🙃
Build OpenBao
Easy Mode: https://github.com/thejambavan/openbao-compose Ouroboros Mode: https://openbao.org/docs/platform/k8s/
Add BSO to k8s cluster
Install the secrets operator helm chart:
helm install vault-secrets-operator hashicorp/vault-secrets-operator --namespace openbao-secrets-operator --create-namespace
Unfortunately you can’t yet do this because the openbao version hasn’t been uploaded to the helm repo:
helm install openbao-secrets-operator openbao/openbao-secrets-operator --namespace openbao-secrets-operator --create-namespace
k8s configuration
Create an appropriate namespace and connect k8s ↔️ openbao
kubectl create namespace openbao
export K8S_TO_VAULT=bao.tunstall.in
curl -vk https://$K8S_TO_VAULT
Note: SSH port-forwarding tricks do not work here!
kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
namespace: openbao
name: openbao-connection
spec:
# address to the Vault server.
address: https://$K8S_TO_VAULT:443
# skipTLSVerify: true # enable if doing dev stuff
---
EOF
if you made a mistake and need to delete it:
kubectl delete VaultConnection openbao-connection -n openbao
kubectl describe --namespace openbao vaultconnection.secrets.hashicorp.com/openbao-connection
Service account
kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: openbao-to-k8s-sa
namespace: openbao
---
apiVersion: v1
kind: Secret
metadata:
name: openbao-to-k8s-sa-secret
namespace: openbao
annotations:
kubernetes.io/service-account.name: openbao-to-k8s-sa
type: kubernetes.io/service-account-token
---
EOF
Role for the above service account
kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: openbao-to-k8s-sa-tokenreview-role
namespace: openbao
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: openbao-to-k8s-sa
namespace: openbao
EOF
Create k8s auth tokens
variously get k8s ca cert and auth token for vso. SSH port-forward tricks work here.
export VSO_CERT=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data."ca.crt"'| base64 -d)
export VSO_TOKEN=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data."token"'| base64 -d)
export VAULTSA_SECRET=$(kubectl get secret --namespace openbao openbao-to-k8s-sa-secret --output json | jq -r '.data')
verify they’re correct:
echo -e "${VSO_CERT}\n${VSO_TOKEN}\n${VAULTSA_SECRET}"
Give tokens to openbao
copypasta those vars to a bao host or something else with the bao cli binary and then configure k8s auth with them.
Configure bao env vars and credentials to point it at your cluster
export VAULT_ADDR="https://bao.tunstall.in"
export VAULT_SKIP_VERIFY=true # unnecessary if you've got a proper letsencrypt cert
bao login # you'll find ths root token in `/var/lib/docker/volumes/openbao_tls-certs/_data/init.json` if using the dev/test docker-compose yaml
$ bao auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
$ bao write auth/kubernetes/config token_reviewer_jwt="$VSO_TOKEN" kubernetes_host="https://k3s-test:6443" kubernetes_ca_cert="$VSO_CERT"
Success! Data written to: auth/kubernetes/config
You can just download the OpenBao binary (bao) and run it from anywhere that can reach the OpenBao cluster. https://openbao.org/downloads/
The k8s authentication method can be viewed but not configured from the UI.
Create OpenBao static secrets engine on /static
This differs from Vault, where the kv engine is named kv2. OpenBao never had a “version 1” of this, so it’s just called kv.
bao secrets enable -version=2 -path=static kv
Success! Enabled the kv secrets engine at: static/
Store an example secret
bao kv put static/exampleapp/creds username='testuser' password='password123'
======== Secret Path ========
static/data/exampleapp/creds
======= Metadata =======
Key Value
--- -----
created_time 2025-06-25T09:51:27.515299225Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
Set a policy to allow the secret to be read by k8s
bao policy write exampleapp-kv-read - << EOF
path "static/data/exampleapp/creds" {
capabilities = ["read"]
}
EOF
Success! Uploaded policy: exampleapp-kv-read
Set an openbao role for k8s vso
bao write auth/kubernetes/role/openbao-role-static-exampleapp bound_service_account_names=vso-static-exampleapp-sa bound_service_account_namespaces=static-exampleapp policies=exampleapp-kv-read ttl=1h
Create k8s example secret mapped to a vault secret
namespace and service account
kubectl create namespace static-exampleapp
kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vso-static-exampleapp-sa
namespace: static-exampleapp
---
EOF
VaultAuth resource
kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: openbao-auth-vso-static-exampleapp
namespace: static-exampleapp
spec:
vaultConnectionRef: openbao/openbao-connection
method: kubernetes
mount: kubernetes
allowedNamespaces:
- static-exampleapp
kubernetes:
role: openbao-role-static-exampleapp
serviceAccount: vso-static-exampleapp-sa
---
EOF
VaultStaticSecret resource
kubectl create -f - <<EOF
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: openbao-static-secret
namespace: static-exampleapp
spec:
vaultAuthRef: openbao-auth-vso-static-exampleapp
mount: static
type: kv-v2
path: exampleapp/creds
refreshAfter: 300s
destination:
create: true
name: vso-static-creds-from-vault
---
EOF
Retrieve Secret
At this point you can retrieve the native k8s secret called openbao-static-secret which is mapped to the vault secret static/exampleapp/creds:
$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
"_raw": "{\"data\":{\"password\":\"asdfasdfasdfasdfasdf\",\"username\":\"testuser_version2\"},\"metadata\":{\"created_time\":\"2024-10-29T16:20:35.670041844Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":3}}",
"password": "asdfasdfasdfasdfasdf",
"username": "testuser_version2"
…and then after changing the secret in Bao and waiting for it to sync, getting the now-changed version. Note the incremented version key:
$ kubectl get secret --namespace static-exampleapp vso-static-creds-from-vault -o json | jq ".data | map_values(@base64d)"
{
"_raw": "{\"data\":{\"password\":\"solarwinds123\",\"username\":\"test_user\"},\"metadata\":{\"created_time\":\"2024-10-29T16:23:50.338672408Z\",\"custom_metadata\":null,\"deletion_time\":\"\",\"destroyed\":false,\"version\":4}}",
"password": "solarwinds123",
"username": "test_user"
Notes
- You MUST HAVE connectivity between k8s 👉 OpenBao. SSH port forwards WILL NOT work unless you do heavy DNS trickery.
- You MUST HAVE connectivity between OpenBao 👉 k8s. SSH port forwards WILL work for this, however bear in mind that ALL vault nodes need to be able to get to the k8s API otherwise you’ll see errors like this:
{"level":"error","ts":"2024-10-29T16:11:14Z","logger":"cachingClientFactory","msg":"Failed to get NewClientWithLogin","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","VaultStaticSecret":{"name":"openbao-static-secret","namespace":"static-exampleapp"},"namespace":"static-exampleapp","name":"openbao-static-secret","reconcileID":"4474bced-6d91-44c6-92e9-af0ce232c747","cacheKey":"kubernetes-9f0b4138ae774cf7ad4cd7","error":"Error making API request.\n\nURL: PUT https://bao.tunstall.in:443/v1/auth/kubernetes/login\nCode: 403. Errors:\n\n* permission denied"}
This is not actually a “permission denied” as such. What it actually means is “Vault/Bao is unable to reach k8s to perform an AuthNZ”