Vault Disaster Recovery Without Enterprise: Building a Warm Standby From Snapshots

Here is the sentence from Vault’s own documentation that should decide your backup strategy:

“If the seal mechanism or its keys are permanently deleted, then the Vault cluster cannot be recovered, even from backups.”

A perfect snapshot plus a lost unseal key is permanent data loss. The snapshot holds your secrets, encrypted by a root key, which is itself encrypted by something that lives outside the snapshot. Back up the cluster without backing up its unseal mechanism and you have a file nobody can ever open.

Part 1 built a Vault cluster on Kubernetes that unseals itself using a second Vault’s transit engine. That design has a flaw I did not call out at the time: the unsealer runs inside the very cluster it unseals. This post fixes the backup story around it.

Vault Community has no DR replication. That is an Enterprise feature. So I built a substitute: automated snapshots to off-cluster storage, and a second Kubernetes cluster that restores them on a schedule so it is always a few minutes behind and ready to take over. Then I measured what it actually gives you, and where it falls short of the thing it imitates.

Every command below ran against two live kind clusters. The numbers are real.

What we are building

Note the dotted lines. One transit Vault unseals both clusters, and that is not a shortcut. It is the only way the standby can open a snapshot the primary wrote.

The snapshot contains the seal, and you can prove it

snapshot inspect lists the storage keys inside a snapshot. Take one and look:

$ vault operator raft snapshot save /tmp/vault-dr.snap
$ vault operator raft snapshot inspect /tmp/vault-dr.snap
 ID           bolt-snapshot
 Size         32106
 Index        785

 Key Name                            Count      Size
 ----                                ----       ----
 sys/policy                          5          4.1KB
 core/keyring                        1          337B
 core/hsm                            1          162B
 ...

core/keyring is the encrypted keyring. core/hsm is the seal configuration. The snapshot knows how it was sealed and carries the encrypted root key, but not the key that decrypts it. That key is in the transit Vault, and if you lose it the bytes above are noise.

The backup pipeline

A token that can only take snapshots

The snapshot endpoint is not root-protected, so a narrow token is enough. HashiCorp does not publish a ready-made policy, so I wrote one and tested whether read alone really suffices:

# snapshot-agent: take snapshots, nothing else.
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}

It does, and the same token is correctly powerless to restore:

$ VAULT_TOKEN=$SNAP_TOKEN vault operator raft snapshot save /tmp/lp-test.snap
RESULT: SUCCESS

$ VAULT_TOKEN=$SNAP_TOKEN vault operator raft snapshot restore /tmp/lp-test.snap
Error installing the snapshot: Code: 403

Restoring gets its own policy with create and update on sys/storage/raft/snapshot plus snapshot-force. Keep the two apart. The thing that runs every few minutes should not be able to overwrite your cluster.

The CronJob nobody ships for you

Enterprise has scheduled snapshots built in. Community does not, and the Vault Helm chart has no CronJob template, so you author it. Mine runs in two stages sharing an emptyDir: an init container takes the snapshot, the main container ships it off-cluster.

spec:
  schedule: '*/2 * * * *' # every 2 minutes in the lab; hourly or daily in production
  jobTemplate:
    spec:
      template:
        spec:
          initContainers:
            - name: save
              image: hashicorp/vault:2.0.3
              command:
                - /bin/sh
                - -c
                - |
                  STAMP=$(date -u +%Y%m%dT%H%M%SZ)
                  vault operator raft snapshot save "/snap/vault-${STAMP}.snap"
              env:
                - name: VAULT_ADDR
                  value: http://vault-active.vault.svc:8200

Two details that matter. vault-active is the chart’s service for the current leader, so the job always talks to a node that can serve the request. And timestamped filenames sort lexicographically, which is what lets the restore side pick the newest one without parsing anything.

The upload container pushes to MinIO running outside both clusters, then a lifecycle rule handles retention:

$ mc ilm rule add --expire-days 7 store/vault-snapshots
Lifecycle configuration rule added with ID `d9hr2l806h1s74hvduo0`.

$ mc ls store/vault-snapshots/
[18:44:03 UTC]  32KiB  vault-20260724T184400Z.snap
[18:46:02 UTC]  32KiB  vault-20260724T184600Z.snap
[18:48:02 UTC]  32KiB  vault-20260724T184800Z.snap
[18:50:02 UTC]  32KiB  vault-20260724T185000Z.snap

Those two-minute gaps are your recovery point objective, written out as filenames. Whatever happened in the gap is gone.

The seal is the hard part

The standby has to reproduce the primary’s unseal mechanism, and an equivalent one does not count. It has to be the same key.

The option most people reach for is copying the transit key to the standby. Vault has transit/backup/<key> for exactly that, so I tried it:

$ vault read transit/backup/autounseal
Error reading transit/backup/autounseal: Code: 500

$ vault read transit/keys/autounseal
allow_plaintext_backup    false
exportable                false

Both flags are creation-time properties that cannot be enabled later. Part 1 created that key with the defaults, like every tutorial does, so the key can never leave that Vault. Whether your DR plan is even possible was decided the day you ran vault write -f transit/keys/autounseal.

If you want that option, you have to take it up front, and understand the trade:

# Decide this on day one. It cannot be changed afterwards.
vault write -f transit/keys/autounseal allow_plaintext_backup=true

A plaintext key backup is unseal material sitting in a file. It buys portability and gives up custody. That is a real trade-off, not a best practice.

Since the key cannot move, the standby uses the same transit Vault over the network. Both kind clusters are containers on the same Docker bridge, so cluster 1 exposes its transit Vault on a NodePort and cluster 2 dials the node’s container IP.

Prove the seal works before you install anything. If this fails, the standby will never unseal, and you want to find out now rather than after a restore:

$ kubectl --context kind-vault-dr -n vault run seal-check --image=hashicorp/vault:2.0.3 \
    --env="VAULT_ADDR=http://172.22.0.5:30530" --env="VAULT_TOKEN=$SEAL_TOKEN" \
    --command -- sh -c 'vault write -f transit/encrypt/autounseal plaintext=$(echo -n ok | base64)'

ciphertext     vault:v1:q6MhmYmCUcJ+bFWhtUrTgh5L22g1YPcF131sNbpqk/KUfbc=
SEAL REACHABLE: YES

One thing Part 1 got right by accident: the transit Vault is Shamir sealed. That stops the recursion. Its unseal keys are shares you can carry by hand, so the thing that unseals everything else does not itself depend on another service being alive.

Standing up the standby

The standby is a single Raft node with the same seal stanza pointing at the shared transit Vault. Initialize it and it unseals itself across the cluster boundary:

$ vault operator init -recovery-shares=1 -recovery-threshold=1
Recovery Key 1: <redacted>
Initial Root Token: <redacted>

$ vault status
Seal Type            transit
Sealed               false
HA Mode              active

Now restore. I expected to need -force, because that flag exists to bypass “checks ensuring the Autounseal or shamir keys are consistent with the snapshot data”. It was not needed:

--- restoring (force=false) ---
RESTORE EXIT: 0

That is the seal prerequisite passing, not being skipped. The consistency check asks whether this cluster’s seal can open the snapshot’s root key, and it can, because it is literally the same key. -force is for when the seal genuinely differs, and it does not make an unmatched seal work. It only stops Vault from warning you.

A restore replaces the cluster’s identity, not just its data. The root token I generated thirty seconds earlier stopped working, and the primary’s token started working:

# the standby's own root token
$ vault token lookup
Code: 403. Errors: * permission denied * invalid token

# the primary's root token, on the standby
$ vault token lookup
display_name        root
policies            [root]

$ vault kv get secret/dr/marker
state     before-snapshot

That has an operational consequence people trip over: every credential the standby needs must come from the primary, because each restore overwrites the auth state. I created the snapshot-restore policy on the primary, and it arrived in the standby inside the next snapshot. After that the narrow restore token worked there, and was still correctly refused when it reached for a secret:

$ vault policy list
snapshot-agent
snapshot-restore

$ VAULT_TOKEN=$RESTORE_TOKEN vault kv get secret/dr/marker
Code: 403

The failover drill

I wrote a new secret on the primary, let the pipeline capture it, then timed a refresh of the standby:

=== FAILOVER DRILL ===
RTO (fetch + restore): 5s

$ vault kv get secret/dr/marker   # on the standby
state     after-failover-test
note      rpo-probe

$ vault status
Sealed      false
HA Mode     active

Five seconds, and the standby never restarted. It was already unsealed and stayed that way. So the honest numbers for this design are an RPO of one snapshot interval and an RTO of a few seconds plus however long your traffic cutover takes. The cutover is the part I have not automated, and in real life it is usually the slow one.

The cluster is up. It is not yet correct.

A restored Vault is a Vault that believes things about the world that stopped being true.

Kubernetes auth carries the old cluster’s address. The config is restored verbatim, and kubernetes_host is load-bearing. Mine survived only because both kind clusters happen to expose the API server at the same in-cluster address. Point it somewhere unreachable and every workload login dies:

$ vault write auth/kubernetes/config kubernetes_host="https://10.255.255.1:443"
Success!

# a real pod, logging in
$ vault write auth/kubernetes/login role=app jwt="$JWT"
Code: 403. Errors: * permission denied

Migrate between real clusters and that address changes. Check it against the new cluster’s reality instead of assuming the restore handled it, and verify with an actual pod login rather than reading the config back.

Worth noting what did not break: I deliberately stored cluster 1’s CA certificate in the standby’s auth config, and logins kept working. Vault fell back to the certificate mounted in its own pod. Configuring Kubernetes auth without a stored CA or reviewer JWT, which is what Part 1 did, is what made the config portable.

Dynamic credentials drift away from reality. The standby’s lease table is a photograph of a moment. The database kept moving. After issuing one credential on the primary post-snapshot:

# Postgres, the source of truth
v-kubernet-app-dyna-Af3dvXmErxBtys2Eg1Ia-1784918960
v-root-app-dyna-WIG3MVB7gxp5ogXtec0L-1784919183

# leases the primary knows about        # leases the standby knows about
36CzWqC32FlPjT6pdKKi22Yt                m2wnHZ6sBV1sT4ma8DMmI1Py
VfQzNdlDPYMR6miOlGZStYta

Two users in the database, and a standby that has never heard of either. Promote it and those accounts become orphans: live logins that Vault will never revoke because it does not know they exist. Post-failover, reconcile the two lists and drop what has no lease. Short TTLs shrink the window; they do not close it.

What this is not

I built something that behaves like DR replication. It is not DR replication, and the gap is worth stating plainly.

Also Enterprise-only: scheduled snapshots, single-secret recovery from a snapshot, and online seal migration. If your RPO is measured in seconds, buy the licence. If it is measured in hours, this is a reasonable answer, and you now know exactly which parts of it you are responsible for.

The checklist that actually matters

• Back up the unseal mechanism, not just the cluster. It is part of the backup, and a snapshot without it cannot be opened.
• Decide key portability on day one, because allow_plaintext_backup cannot be turned on later.
• Put the unsealer outside the failure domain it unseals. Mine is still inside cluster 1, which is fine for a lab and wrong for production.
• Keep the unsealer Shamir sealed so the dependency chain terminates somewhere.
• Test the restore, not the backup. An untested snapshot is a hypothesis.
• Restore into isolation. A restored cluster starts revoking leases it thinks it owns, and those credentials may still be live in production.
• After a failover, reconcile the Kubernetes auth address, the orphaned dynamic credentials, and the static role passwords.

Snapshots are cheap and the pipeline above took an afternoon. The expensive part is finding out, on the worst day, that the key which opens them was only ever stored in the cluster you just lost.

A recovery plan you have never run is still a guess, so in Part 3 I break this cluster on purpose: killing nodes, wiping storage, and destroying quorum to see which failures heal themselves and which need a human.