# Vault Operations Guide

**Last Updated:** 2026-02-01  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date:** 2026-01-19  
**Status:** ✅ Complete  
**Purpose:** Day-to-day operations guide for Vault cluster

---

## Quick Reference

### Cluster Information

- **Cluster Nodes:** 3 (vault-phoenix-1, vault-phoenix-2, vault-phoenix-3)
- **API Endpoints:** http://192.168.11.200:8200 (8640), http://192.168.11.215:8200 (8641), http://192.168.11.202:8200 (8642)
- **Storage:** Raft (integrated)
- **Seal Type:** Shamir (5 keys, threshold 3)

---

## Daily Operations

### Health Checks

Run health check script:
```bash
./scripts/vault-health-check.sh
```

With cluster status:
```bash
VAULT_TOKEN=<root-token> ./scripts/vault-health-check.sh
```

### Check Cluster Status

```bash
ssh root@192.168.11.11 "pct exec 8640 -- bash -c 'export VAULT_ADDR=http://127.0.0.1:8200 && export VAULT_TOKEN=<token> && vault operator raft list-peers'"
```

### Check Node Status

```bash
# Node 1
ssh root@192.168.11.11 "pct exec 8640 -- vault status"

# Node 2
ssh root@192.168.11.12 "pct exec 8641 -- vault status"

# Node 3
ssh root@192.168.11.11 "pct exec 8642 -- vault status"
```

---

## Backup Operations

### Manual Backup

```bash
VAULT_TOKEN=<root-token> ./scripts/vault-backup.sh
```

### Automated Backups

Add to crontab:
```bash
# Daily backup at 2 AM
0 2 * * * cd /home/intlc/projects/proxmox && VAULT_TOKEN=<token> ./scripts/vault-backup.sh
```

### Restore from Backup

```bash
# On Vault node
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=<root-token>
vault operator raft snapshot restore /path/to/backup.snapshot
```

---

## Unsealing Operations

### Unseal a Node

```bash
# On the node
export VAULT_ADDR=http://127.0.0.1:8200
vault operator unseal <key-1>
vault operator unseal <key-2>
vault operator unseal <key-3>
```

### Unseal All Nodes

```bash
# Node 1
ssh root@192.168.11.11 "pct exec 8640 -- bash -c 'export VAULT_ADDR=http://127.0.0.1:8200 && vault operator unseal <key-1> && vault operator unseal <key-2> && vault operator unseal <key-3>'"

# Node 2
ssh root@192.168.11.12 "pct exec 8641 -- bash -c 'export VAULT_ADDR=http://127.0.0.1:8200 && vault operator unseal <key-1> && vault operator unseal <key-2> && vault operator unseal <key-3>'"

# Node 3
ssh root@192.168.11.11 "pct exec 8642 -- bash -c 'export VAULT_ADDR=http://127.0.0.1:8200 && vault operator unseal <key-1> && vault operator unseal <key-2> && vault operator unseal <key-3>'"
```

---

## Secret Management

### Create/Update Secret

```bash
vault kv put secret/phoenix/database/postgres \
  username=phoenix \
  password=new_password \
  host=db.example.com \
  port=5432 \
  database=phoenix
```

### Read Secret

```bash
vault kv get secret/phoenix/database/postgres
```

### List Secrets

```bash
vault kv list secret/phoenix/
```

### Delete Secret

```bash
vault kv delete secret/phoenix/old-secret
```

---

## Policy Management

### List Policies

```bash
vault policy list
```

### Read Policy

```bash
vault policy read phoenix-api-policy
```

### Update Policy

```bash
vault policy write phoenix-api-policy - <<EOF
# Updated policy content
path "secret/data/phoenix/api/*" {
  capabilities = ["read"]
}
EOF
```

---

## AppRole Management

### List AppRoles

```bash
vault list auth/approle/role
```

### Get Role ID

```bash
vault read auth/approle/role/phoenix-api/role-id
```

### Generate Secret ID

```bash
vault write -f auth/approle/role/phoenix-api/secret-id
```

### Rotate Secret ID

```bash
# Generate new secret ID
NEW_SECRET_ID=$(vault write -field=secret_id -f auth/approle/role/phoenix-api/secret-id)

# Update service configuration with new secret ID
# Then delete old secret IDs if needed
```

---

## Monitoring

### Enable Audit Logging

```bash
vault audit enable file file_path=/var/log/vault/audit.log
```

### View Logs

```bash
# Service logs
ssh root@192.168.11.11 "pct exec 8640 -- journalctl -u vault -f"

# Audit logs
ssh root@192.168.11.11 "pct exec 8640 -- tail -f /var/log/vault/audit.log"
```

### Metrics (if enabled)

```bash
curl http://192.168.11.200:8200/v1/sys/metrics?format=prometheus
```

---

## Troubleshooting

### Node Not Joining Cluster

1. Check network connectivity:
```bash
ping 10.160.0.40
ping 10.160.0.41
ping 10.160.0.42
```

2. Check Vault logs:
```bash
ssh root@192.168.11.11 "pct exec 8640 -- journalctl -u vault -n 50"
```

3. Verify configuration:
```bash
ssh root@192.168.11.11 "pct exec 8640 -- cat /etc/vault.d/vault.hcl"
```

### Service Won't Start

1. Check service status:
```bash
ssh root@192.168.11.11 "pct exec 8640 -- systemctl status vault"
```

2. Check configuration:
```bash
ssh root@192.168.11.11 "pct exec 8640 -- vault server -config=/etc/vault.d/vault.hcl -verify-only"
```

3. Check logs:
```bash
ssh root@192.168.11.11 "pct exec 8640 -- journalctl -u vault -n 100"
```

### Cluster Split-Brain

If cluster loses quorum:

1. Identify nodes with latest data
2. Remove failed nodes from cluster:
```bash
vault operator raft remove-peer <node-id>
```

3. Rejoin nodes:
```bash
# Nodes will auto-rejoin via retry_join configuration
```

---

## Maintenance

### Restart Node

```bash
# Stop node
ssh root@192.168.11.11 "pct stop 8640"

# Start node
ssh root@192.168.11.11 "pct start 8640"

# Unseal after restart
ssh root@192.168.11.11 "pct exec 8640 -- bash -c 'export VAULT_ADDR=http://127.0.0.1:8200 && vault operator unseal <key-1> && vault operator unseal <key-2> && vault operator unseal <key-3>'"
```

### Update Vault

1. Backup cluster
2. Update on one node at a time
3. Restart node
4. Unseal node
5. Verify cluster health
6. Repeat for other nodes

### Scale Cluster

To add a node:
1. Create new container
2. Install Vault
3. Configure with same cluster settings
4. Start Vault
5. Node will auto-join via retry_join

---

## Emergency Procedures

### Complete Cluster Failure

1. Restore from latest backup
2. Initialize new cluster if needed
3. Restore Raft snapshot
4. Unseal all nodes

### Lost Unseal Keys

If unseal keys are lost:
- Use recovery keys (if configured)
- Or reinitialize cluster (data will be lost)

### Data Corruption

1. Stop affected node
2. Restore from backup
3. Restart node
4. Verify data integrity

---

## Related Documentation

- [Phoenix Vault Integration Guide](PHOENIX_VAULT_INTEGRATION_GUIDE.md)
- [Vault TLS Configuration](VAULT_TLS_CONFIGURATION.md)
- [HashiCorp Vault Documentation](https://developer.hashicorp.com/vault/docs)

---

**Status:** ✅ Complete  
**Last Updated:** 2026-01-19