Startower

Running HashiCorp Vault on Kubernetes: HA, Auto-Unseal, and Rotating Secrets

DevSecOps·10 min read·Gilang Javier

A Kubernetes Secret is not encrypted. It is base64, which is just encoding with extra steps. Anyone who can read etcd, or an old backup of it, or a snapshot that leaked to the wrong bucket, can read every database password, API key, and token you have ever stored. base64 decodes in one command.

Vault fixes this by moving secrets out of the cluster and handing them out under a policy, with a short lifetime and an audit trail. This guide builds a production-shaped Vault on Kubernetes: a highly available Raft cluster, auto-unsealed by a second Vault, delivering secrets into a running app, then rotating those secrets while the app keeps serving traffic. I ran every command below against a live kind cluster, so the output is real.

This is Part 1. Part 2 covers disaster recovery and migration.

What Vault actually gives you

Before the helm install, it helps to know what you’re signing up for. Vault is not just an encrypted key/value store; that’s the smallest part of it. The pieces you’ll reach for:

  • KV secrets. Versioned static secrets (secret/data/...) for the familiar case of putting a password somewhere safe.
  • Dynamic secrets. Vault creates a credential on request, unique per consumer, that expires on its own. No human ever sees a long-lived database password.
  • Encryption as a service. The transit engine encrypts and decrypts data for your app so the keys never leave Vault. Your database stores ciphertext.
  • Identity and policies. Every request authenticates (here, with a Kubernetes service account) and gets checked against a least-privilege policy.
  • Leasing, renewal, and revocation. Every secret has a lease. Vault can revoke one, or all of them, at once.
  • Audit. Every request and response goes to an audit device.

The rest of this guide wires up four of those: KV, dynamic database credentials, transit, and Kubernetes identity.

The topology we’re building

Two Vault deployments, one Postgres, one app:

  • vault-transit is a small single Vault that runs only the transit engine. Its job is to auto-unseal the main cluster.
  • vault is the real cluster: three Vault pods in HA mode using Integrated Storage (Raft), so there’s no external Consul to run. Each pod lands on its own node.
  • postgres plus a tiny vaultdemo app that reads its database credentials from Vault.

Here’s how the pieces connect. The app never talks to the database with a stored password; it authenticates to Vault and gets a fresh credential:

Vault on Kubernetes topologyThe apps namespace holds the vaultdemo app and Postgres. The vault namespace holds the Agent Injector and a three-node Vault HA cluster on Raft, with vault-0 as leader. The app authenticates to vault-0 with Kubernetes auth, the injector writes the secret into the app, and the app logs in to Postgres with a dynamic credential. A separate vault-transit instance auto-unseals the cluster.Vault on KubernetesWorkloadVaultThe unsealernamespace: appsvaultdemo appPostgresdynamic loginnamespace: vaultAgent InjectorVault HA · Raftvault-0 · leadervault-1vault-2vault-transitholds the unseal keyinject secretk8s authauto-unseal

Local vs production. I built this on a 4-node kind cluster (one control plane, three workers). To keep it laptop-sized I disabled TLS and shrank the resource requests. Both are called out inline. Keep TLS on and size the pods properly in production.

Install the chart

Add the repo and pin the chart version. Pinning matters: an unpinned helm install is not reproducible.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm search repo hashicorp/vault --versions | head
# chart 0.34.0 deploys Vault 2.0.3

First, the auto-unseal provider

A freshly started Vault is sealed: it holds ciphertext but not the key to decrypt it. Someone has to supply the unseal key on every start, and doing that by hand on every pod restart does not scale. So production uses auto-unseal, where an external service holds the key. Cloud KMS is the usual choice, but on a laptop there’s no KMS. The portable answer is transit auto-unseal: a second Vault hands out the unseal operation through its transit engine.

Install the transit Vault as a standalone server:

kubectl create namespace vault
helm install vault-transit hashicorp/vault \
  --namespace vault --version 0.34.0 \
  --values vault-transit-values.yaml

Initialize and unseal it once (this one uses Shamir keys, since someone has to unseal the unsealer), then create the key and a token the main cluster will use:

kubectl -n vault exec vault-transit-0 -- vault operator init \
  -key-shares=1 -key-threshold=1        # 1/1 is fine for the unsealer in a lab
kubectl -n vault exec vault-transit-0 -- vault operator unseal <key>

# enable transit, create the auto-unseal key, scope a policy to just that key
vault secrets enable transit
vault write -f transit/keys/autounseal
vault policy write autounseal - <<'EOF'
path "transit/encrypt/autounseal" { capabilities = ["update"] }
path "transit/decrypt/autounseal" { capabilities = ["update"] }
EOF
AUTOTOKEN=$(vault token create -orphan -policy=autounseal -period=24h -field=token)
kubectl -n vault create secret generic vault-transit-token --from-literal=token="$AUTOTOKEN"

That token is the only thing the main cluster needs to auto-unseal. It goes into a Kubernetes Secret, not into a config file.

Then, the HA Raft cluster

The main cluster’s values enable HA plus Raft, keep the chart’s default one-pod-per-node anti-affinity, and add a seal "transit" stanza pointing at the transit Vault. The token arrives as the VAULT_TOKEN environment variable, sourced from the Secret:

server:
  extraSecretEnvironmentVars:
    - envName: VAULT_TOKEN
      secretName: vault-transit-token
      secretKey: token
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        listener "tcp" {
          tls_disable = 1          # LOCAL ONLY: use TLS in production
          address     = "[::]:8200"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join { leader_api_addr = "http://vault-0.vault-internal:8200" }
          retry_join { leader_api_addr = "http://vault-1.vault-internal:8200" }
          retry_join { leader_api_addr = "http://vault-2.vault-internal:8200" }
        }
        seal "transit" {
          address    = "http://vault-transit.vault.svc:8200"
          key_name   = "autounseal"
          mount_path = "transit/"
        }
        service_registration "kubernetes" {}
helm install vault hashicorp/vault \
  --namespace vault --version 0.34.0 \
  --values vault-ha-values.yaml

Initialize and watch it auto-unseal

With auto-unseal configured, init produces recovery keys instead of unseal keys. You no longer feed keys in on startup:

kubectl -n vault exec vault-0 -- vault operator init \
  -recovery-shares=1 -recovery-threshold=1

The moment vault-0 is initialized, it calls the transit Vault, decrypts its root key, and unseals itself. The other two pods find the leader through retry_join and unseal the same way, with no human in the loop:

$ vault status
Seal Type            transit
Recovery Seal Type   shamir
Initialized          true
Sealed               false
HA Mode              active

$ vault operator raft list-peers
Node       Address                       State       Voter
vault-0    vault-0.vault-internal:8201   leader      true
vault-1    vault-1.vault-internal:8201   follower    true
vault-2    vault-2.vault-internal:8201   follower    true

$ vault operator raft autopilot state | grep -E 'Healthy|Failure'
Healthy:             true
Failure Tolerance:   1

Failure Tolerance: 1 is the payoff of three nodes: one can die and the cluster keeps serving. The Seal Type transit line confirms the auto-unseal path is live.

Give an app a secret

A secret nobody can reach is not useful. The delivery path has three parts: an auth method that proves who the pod is, a policy that says what it may read, and the Agent Injector that renders the secret into the pod.

The full login-to-connect exchange, using the pod’s own Kubernetes identity:

How a pod gets a credentialThe pod starts with inject annotations. The Agent Injector logs in to Vault with the pod's service-account token, Vault validates it through the Kubernetes TokenReview API, then issues a scoped token. The injector reads a dynamic database credential, Vault creates a short-lived Postgres role, and the injector writes the credential into the pod, which then connects to Postgres as that user.How a pod gets a credentialvaultdemo PodAgent InjectorVaultKubernetes APIPostgres1start with agent-inject annotations2login: service-account token, role "app"3TokenReview: is this token valid?4valid, app-sa in namespace apps5token scoped to policy "app"6read database/creds/app-dynamic7CREATE ROLE (short-lived user)8username and password9write /vault/secrets/db-creds10connect using the dynamic user

Enable Kubernetes auth and a KV secret:

vault secrets enable -path=secret kv-v2
vault kv put secret/app/config username=app_user password='S3cret-P@ss'

vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Write a least-privilege policy that reads exactly the paths the app needs and nothing else, plus a role binding a specific service account to it:

vault policy write app - <<'EOF'
path "secret/data/app/config"           { capabilities = ["read"] }
path "database/creds/app-dynamic"       { capabilities = ["read"] }
path "database/static-creds/app-static" { capabilities = ["read"] }
EOF

vault write auth/kubernetes/role/app \
  bound_service_account_names=app-sa \
  bound_service_account_namespaces=apps \
  policies=app ttl=1h

Gotcha I hit. I first set audience=vault on the role. The injected service-account token carries aud: ["https://kubernetes.default.svc.cluster.local"], so every login failed with invalid audience (aud) claim. Either drop audience (the service-account-name binding is the real control) or project a token whose audience matches. Note that vault write on an existing role does not clear a field you omit. I had to vault delete the role and recreate it.

The app opts in with pod annotations. The injector sees them, authenticates with the pod’s service account, and writes each secret to /vault/secrets/:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "app"
  vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-dynamic"
  vault.hashicorp.com/agent-inject-template-db-creds: |
    {{- with secret "database/creds/app-dynamic" -}}
    username={{ .Data.username }}
    password={{ .Data.password }}
    {{- end }}

My vaultdemo app reads that file and connects to Postgres. Its /healthz proves the whole chain end to end:

{
  "db_connected": true,
  "db_current_user": "v-kubernet-app-dyna-4oOhMvRhPqRe0Lsk923x-1784913608",
  "kv_loaded": true,
  "time": "2026-07-24T17:21:11Z"
}

The app never held a database password. It connected as a Vault-generated user that did not exist a minute earlier.

Rotate the secrets

This is where Vault earns its cost. Three rotation patterns, all validated against the running cluster.

Dynamic credentials: rotation by design

The database engine mints a new user per request, each with a short TTL. Two reads, two different users:

$ vault read -field=username database/creds/app-dynamic
v-root-app-dyna-k56ik07akzDedZXw0SUd-1784913724
$ vault read -field=username database/creds/app-dynamic
v-root-app-dyna-W5NOFVIBIU09Kv8lERGg-1784913724

Nothing to rotate manually. The credential expires and Vault drops the Postgres role. The app’s injected file updates when the lease renews, so a rotated credential shows up without a restart.

Static roles: scheduled rotation of an existing user

Sometimes you can’t hand out throwaway users; you have one fixed account. A static role lets Vault own that account’s password and rotate it on a schedule (rotation_period), or on demand:

# before
password             V-o4NEvUay5GhXLNsDWd
$ vault write -f database/rotate-role/app-static
# after: same user, new password, no app change
password             8-6N4mb1Oa81utUEXo67

Transit: rotate the encryption key, keep old ciphertext readable

Rotating an encryption key normally means re-encrypting everything. Transit versions the key, so a rotation is instant and old data still decrypts:

$ vault write -field=ciphertext transit/encrypt/app-data plaintext=$(echo -n 'hello startower' | base64)
vault:v1:uIyeePMFTiagf3UYGmIO0Arex...

$ vault write -f transit/keys/app-data/rotate     # now on version 2
$ vault write -field=plaintext transit/decrypt/app-data ciphertext="vault:v1:uIyee..." | base64 -d
hello startower

New writes use v2, and the vault:v1: ciphertext still decrypts. You rewrap on your own schedule instead of in a panicked migration.

Operational hardening checklist

The walkthrough disables a few things for the laptop. Before this goes anywhere real, work down this list. Every item is straight from Vault’s own production-hardening guidance:

  • Turn TLS back on. End to end, including the retry_join between Raft peers. tls_disable = 1 is for demos only.
  • Revoke the initial root token once setup is done. It’s a break-glass credential, not a daily driver.
  • Enable an audit device (vault audit enable file ...) so every request is logged.
  • Take Raft snapshots on a schedule (vault operator raft snapshot save) and before every upgrade, then store them off-cluster and test-restore them.
  • Upgrade in the right order. The chart’s updateStrategyType: OnDelete means pods don’t roll automatically; restart standbys first and the active leader last, so you never fail over to an older version.
  • Keep policies least-privilege and TTLs short. Grant paths, not wildcards.
  • Size the pods. The chart ships empty resource requests; set them.

What you have now

A Vault cluster that survives a node failure, unseals itself without a human, and hands your app credentials that rotate on their own. The database never holds a long-lived password, and neither does your app.

The one thing this setup can’t yet survive is losing the cluster itself: a bad snapshot restore, a region gone, a migration to new infrastructure. That’s Part 2: disaster recovery and migration for Vault, where the Raft snapshots above stop being a checkbox and become the plan. Then in Part 3 I break this cluster on purpose to find out which failures heal themselves and which need a human.