Startower

Breaking Vault on Purpose: Four Failure Drills on Kubernetes

DevSecOps·10 min read·Gilang Javier

A backup you have never restored is a hypothesis. The same is true of a runbook. So before trusting the Vault cluster from Part 1 and the disaster recovery setup from Part 2, I spent an afternoon breaking it in four escalating ways and writing down exactly what happened.

The short version: two of the four failures fixed themselves in seconds, one needed a single command, and one required surgery on the storage volume. The interesting part is that the cluster lied to me about how healthy it was, and the standard runbook has a step whose reason only becomes clear when you skip it.

Everything below ran against the live three-node cluster. The timings and error messages are copied out of the terminal.

Which failure am I in?

Which failure am I in?A node is down. If its storage is intact, the pod restarts, auto-unseals and rejoins, so nothing is required. If the storage is gone but quorum still holds, it rebuilds and rejoins under the same node ID. If quorum is gone, recovery needs a peers.json file. If the node is never coming back, remove the peer and then wipe its data.Which failure am I in?A node is downStorage intact?Quorum still holds?Gone for good?Restarts, unseals, rejoinsnothing for you to doRebuilds and rejoinssame node ID, fresh volumepeers.json surgeryall servers stopped, survivors onlyremove-peer, then wipe its dataskip the wipe and it seals itselfyesyesnoyes

Drill A: kill a pod

The baseline: three nodes, Failure Tolerance: 1, vault-0 leading.

Deleting a follower is a non-event. The StatefulSet recreates it, the transit Vault unseals it, and it rejoins on its own:

$ kubectl delete pod vault-2
$ vault status                 # on the new vault-2, 23 seconds later
Seal Type    transit
Sealed       false
HA Mode      standby

Killing the leader is more interesting, because it forces an election. vault-1 took over before I could finish typing the next command:

$ kubectl delete pod vault-0    # deleted in 6s
$ vault status                  # asking vault-1
Sealed       false
HA Mode      active             # it promoted itself

# 13 seconds after the delete, vault-0 is back as a follower
$ vault operator raft list-peers
vault-0    vault-0.vault-internal:8201    follower    true
vault-1    vault-1.vault-internal:8201    leader      true
vault-2    vault-2.vault-internal:8201    follower    true

No human action either time. This is the payoff of auto-unseal that people underrate: without it, every one of these restarts would page someone to enter unseal keys.

Drill B: destroy a node and its storage

Deleting the pod and its volume is a harder failure. The node comes back as a blank slate with the same identity. The official guidance tells you to remove the peer first, then wipe its data, so I skipped that to see what the step is protecting me from.

$ kubectl delete pod vault-2
$ kubectl delete pvc data-vault-2

It healed itself. The StatefulSet provisioned a fresh volume, transit unsealed the new pod, and retry_join put it back in the cluster. Autopilot marked the cluster degraded for a few seconds, then recovered:

t+5s  Healthy=true  FailureTolerance=1

$ vault operator raft autopilot state
   vault-0    Healthy: true   Last Index: 4633
   vault-1    Healthy: true   Last Index: 4633
   vault-2    Healthy: true   Last Index: 4633

Matching Last Index across all three means the wiped node replayed the entire Raft log and is genuinely caught up. So in this configuration, with setNodeId: true and retry_join pointing at every peer, losing a node and its disk needs no intervention at all.

That raises the obvious question: what is remove-peer actually for?

The cluster will lie to you about resilience

remove-peer is for a node that is never coming back. Scaling the StatefulSet from three replicas to two simulates that, and the result is the part of this whole exercise I found most useful:

$ kubectl scale sts vault --replicas=2
$ vault operator raft list-peers
vault-0    vault-0.vault-internal:8201    follower    true
vault-1    vault-1.vault-internal:8201    leader      true
vault-2    vault-2.vault-internal:8201    follower    true    # gone, still listed

$ vault operator raft autopilot state
Healthy:             true
Failure Tolerance:   1

Two live nodes, and Vault reports that it can tolerate a failure. It cannot. Quorum of a three-member set is two, so losing one of the two survivors takes the cluster down. The reason it says otherwise is in the autopilot defaults:

$ vault operator raft autopilot get-config
Cleanup Dead Servers                  false
Dead Server Last Contact Threshold    24h0m0s
Min Quorum                            0
Server Stabilization Time             10s

Dead server cleanup is off by default, and the contact threshold before a server is even considered dead is a full day. Removing the departed peer does not reduce your resilience. It reveals it:

$ vault operator raft remove-peer vault-2
Peer removed successfully!

$ vault operator raft autopilot state
Healthy:             true
Failure Tolerance:   0     # the honest number

If you monitor one Vault metric, monitor this one, and know that it can be stale for 24 hours after a node dies.

The trap in the runbook

Now the part that explains why the official procedure orders the steps the way it does. I scaled back to three replicas, expecting vault-2 to rejoin as it had in Drill B. Its volume still held the Raft data from before I removed it. It never came back:

$ kubectl get pod vault-2
vault-2   0/1   Running

$ vault status
Initialized   true
Sealed        true

The log says exactly why:

[ERROR] core: shutting down core: error="node has been removed from the HA cluster"

A node that has been removed from the peer set refuses to run with its old storage, and it seals itself rather than rejoining. The fix is the step I had skipped, applied after the fact: wipe the data directory.

$ kubectl delete pod vault-2 && kubectl delete pvc data-vault-2
# rebuilt on a fresh volume, unsealed, back in the peer set in 9s

So “remove the peer, then delete its data” is not ceremony. Do the first without the second and you get a pod that boots, seals itself, and quietly stays out of your cluster.

Drill C: lose quorum

The real emergency. I destroyed two of the three nodes along with their volumes, leaving vault-0 alone.

Recovering from quorum lossWith two of three peers destroyed there is no quorum, so the surviving node refuses reads even though it is unsealed. The operator scales the StatefulSet to zero, which frees the volume, writes a peers.json recovery file listing only the survivor, then scales back to one. Vault reads the file at startup, rewrites its peer set, elects itself leader, and the data is intact eleven seconds later.Recovering from quorum lossOperatorStatefulSetvault-0 (survivor)Raft2 of 3 peers destroyed. No quorum.1try to serve a read2local node not active3scale to 0 (frees the volume)4write peers.json into /vault/data/raft/5scale to 16read peers.json, rewrite the peer set7entering leader staterecovered in 11s, data intact

A single surviving node is unsealed and useless. It cannot elect itself, so it sits in standby and refuses everything:

$ vault status
Sealed       false
HA Mode      standby

$ vault kv get secret/drill/canary
Code: 500. Errors:
* local node not active but active cluster node not found

That error is what quorum loss looks like from the client side. Note that the data is fine and the seal is fine. Vault simply will not act without a majority.

The recovery is a file called peers.json that tells Raft to forget the members that are gone. Vault reads it once at startup, so the servers have to be stopped while you write it. On Kubernetes that means scaling to zero, which conveniently also releases the volume so another pod can mount it:

kubectl scale sts vault --replicas=0

Then a throwaway pod mounts the survivor’s volume and drops the file in:

volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-vault-0
containers:
  - name: writer
    image: busybox:1.36
    command:
      - /bin/sh
      - -c
      - |
        cat > /vault/data/raft/peers.json <<'JSON'
        [
          {
            "id": "vault-0",
            "address": "vault-0.vault-internal:8201",
            "non_voter": false
          }
        ]
        JSON
    volumeMounts:
      - name: data
        mountPath: /vault/data

The id values are the node IDs from list-peers, which are the pod names because Part 1 set setNodeId: true. List only the nodes that still exist. Scale back to one replica and Vault does the rest:

[INFO] storage.raft: raft recovery initiated: recovery_file=peers.json
[INFO] storage.raft: raft recovery found new config: config="{[{Voter vault-0 vault-0.vault-internal:8201}]}"
[INFO] storage.raft: raft recovery deleted peers.json
[INFO] storage.raft: entering leader state: leader="Node at vault-0.vault-internal:8201 [Leader]"

Eleven seconds from scaling up to a working single-node cluster, and the data was exactly where I left it:

$ vault kv get secret/drill/canary
stage    pre-quorum-loss
value    alive

Two details worth keeping. Vault deletes peers.json after applying it, so it is a one-shot instruction rather than config you leave behind. And the recovered cluster is now a single node with no redundancy, so scaling back to three is part of the procedure, not an afterthought:

$ kubectl scale sts vault --replicas=3
Failure Tolerance restored: 1

One thing I checked because I assumed the opposite

I expected the snapshot pipeline from Part 2 to go dark during these outages. It did not. Snapshots kept landing in object storage every two minutes right through the drills, because the job targets the vault-active service and for most of the exercise some node was active.

It only fails when nothing is. With the StatefulSet at zero replicas, the backup job failed outright:

$ kubectl get job vault-snapshot-blackout
vault-snapshot-blackout   Failed   0/1

Which is the useful version of the warning: your backups survive rolling failures, and stop precisely at the moment of total loss. The last snapshot you get is the one taken before everything went down, so the recovery point is whatever your interval was, not the moment of the incident.

Drill D: undo a mistake

The last drill is the failure that actually happens on a Tuesday. Nothing crashed. Someone deleted the wrong path.

$ vault kv put secret/drill/payroll account=ACME-9931 amount=48500
$ # snapshot taken here
$ vault kv metadata delete secret/drill/payroll
Success! Data deleted (if it existed) at: secret/metadata/drill/payroll

$ vault kv get secret/drill/payroll
No value found at secret/data/drill/payroll

Restoring the snapshot into the same cluster brought it back in four seconds, and notably without -force:

--- restoring (force=false) ---
RESTORE EXIT: 0

$ vault kv get secret/drill/payroll
account    ACME-9931
amount     48500

-force exists to bypass the check that the cluster’s seal matches the snapshot’s. Restoring into the cluster that wrote the snapshot passes that check honestly, so the flag is unnecessary. If you find yourself reaching for it during a same-cluster rollback, something else is wrong.

The catch is scope. This is a whole-cluster rewind, not an undo button for one secret. Everything else moves back to snapshot time too, including tokens and leases. Recovering a single secret out of a snapshot is an Enterprise feature. On Community the options are restore the lot, or restore into an isolated cluster and copy the one value out by hand.

What I would put on the wall

  • A dead node with intact storage needs nothing from you. A dead node with wiped storage also needs nothing, as long as quorum holds.
  • Failure Tolerance can be stale for 24 hours. Cleanup of dead servers is off by default, so a departed node inflates the number until you remove it.
  • After remove-peer, wipe that node’s data before it comes back, or it will boot, seal itself, and log node has been removed from the HA cluster.
  • Quorum loss looks like local node not active but active cluster node not found on a perfectly healthy, unsealed node.
  • peers.json requires all servers stopped, lists only the survivors, and is consumed and deleted on first read.
  • A restore is a rewind of the entire cluster. Snapshot before you restore, so the mistake you are undoing is itself recoverable.

That closes this series: Part 1 built the cluster, Part 2 built the disaster recovery around it, and this post spent an afternoon trying to destroy both. The drills took a few hours. Learning any of this during a real incident would have cost considerably more.