proxmox/docs/04-configuration/HSM_MIGRATION_GUIDE.md

# HSM Key Vault Migration Guide

**Last Updated:** 2026-01-31  
**Document Version:** 1.0  
**Status:** Active Documentation

---

**Date:** 2026-01-26  
**Status:** 🔴 **CRITICAL - IMMEDIATE ACTION REQUIRED**  
**Purpose:** Guide for migrating private keys and secrets from files to HSM/Key Vault

---

## Executive Summary

**Current Risk:** 🔴 **CRITICAL**

Private keys and sensitive secrets are currently stored in `.env` files and documentation, creating a significant security risk. This guide provides step-by-step instructions for migrating all secrets to a Hardware Security Module (HSM) or Key Vault system.

---

## 🔴 Critical Secrets Identified

### Private Keys (CRITICAL - Highest Priority)

**Locations Found:**
1. `smom-dbis-138/.env` - Deployer private key
2. `no_five/.env` - Private key (same as deployer)
3. `237-combo/.env` - Different private key
4. `loc_az_hci/smom-dbis-138/.env` - Deployer private key
5. `smom-dbis-138/services/*/.env` - Multiple service files
6. `docs/06-besu/T1_2_CREDENTIALS_VERIFIED.md` - Documented in markdown

**Risk Level:** 🔴 **CRITICAL**
- Complete compromise of blockchain accounts
- Unauthorized transaction signing
- Financial loss
- Reputation damage

---

## Migration Strategy

### Phase 1: Immediate Actions (Day 1)

#### Step 1: Identify All Private Keys

```bash
# Search for private keys in .env files
find . -name ".env" -type f -exec grep -l "PRIVATE_KEY" {} \;

# Search for private keys in documentation
find docs -name "*.md" -type f -exec grep -l "0x[0-9a-fA-F]\{64\}" {} \;
```

#### Step 2: Document Current Keys

Create a secure inventory (encrypted or in secure location):

```bash
# Create encrypted inventory
cat > /tmp/key-inventory.txt <<EOF
# Private Key Inventory - DO NOT COMMIT
# Created: $(date)

# Key 1: Deployer Key (ChainID 138)
Location: smom-dbis-138/.env
Address: [REDACTED - check blockchain explorer]
Purpose: Contract deployment
Status: ACTIVE - MIGRATE IMMEDIATELY

# Key 2: Service Key
Location: smom-dbis-138/services/*/.env
Purpose: Service operations
Status: ACTIVE - MIGRATE IMMEDIATELY

EOF

# Encrypt the inventory
gpg --encrypt --recipient your-email@example.com /tmp/key-inventory.txt
rm /tmp/key-inventory.txt
```

#### Step 3: Set Up HSM/Key Vault

**Option A: AWS KMS**
```bash
# Create KMS key
aws kms create-key --description "Blockchain Deployer Key" --key-usage SIGN_VERIFY

# Create alias
aws kms create-alias --alias-name alias/blockchain-deployer --target-key-id <key-id>
```

**Option B: Azure Key Vault**
```bash
# Create key vault
az keyvault create --name blockchain-keyvault --resource-group your-rg

# Create key
az keyvault key create --vault-name blockchain-keyvault --name deployer-key --kty EC --curve P-256
```

**Option C: HashiCorp Vault**
```bash
# Enable transit secrets engine
vault secrets enable transit

# Create encryption key
vault write -f transit/keys/blockchain-deployer
```

---

### Phase 2: Key Migration (Days 2-3)

#### Step 4: Import Keys to HSM

**⚠️ IMPORTANT:** Never export private keys from HSM. Use HSM for all operations.

**For AWS KMS:**
```bash
# Import key material (if supported by your HSM)
# Note: AWS KMS doesn't support importing EC private keys directly
# You may need to use AWS CloudHSM or generate new keys in KMS
```

**For Azure Key Vault:**
```bash
# Import key from file (if supported)
az keyvault key import \
  --vault-name blockchain-keyvault \
  --name deployer-key \
  --pem-file private-key.pem
```

**For HashiCorp Vault:**
```bash
# Import key (if supported)
vault write transit/keys/blockchain-deployer \
  type=ecdsa-p256 \
  exportable=false
```

#### Step 5: Update Application Code

**Before (Insecure):**
```typescript
const privateKey = process.env.PRIVATE_KEY; // ❌ Insecure
const wallet = new ethers.Wallet(privateKey);
```

**After (Secure):**
```typescript
// Use HSM for signing
import { KMSClient } from '@aws-sdk/client-kms';

const kmsClient = new KMSClient({ region: 'us-east-1' });
const keyId = 'arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012';

// Sign transaction using HSM
async function signWithHSM(message: string) {
  const signCommand = new SignCommand({
    KeyId: keyId,
    Message: Buffer.from(message),
    MessageType: 'RAW',
    SigningAlgorithm: 'ECDSA_SHA_256',
  });
  
  const response = await kmsClient.send(signCommand);
  return response.Signature;
}
```

#### Step 6: Remove Keys from Files

**⚠️ CRITICAL:** Only remove keys after:
1. Keys are successfully imported to HSM
2. Application code is updated to use HSM
3. All operations are tested with HSM
4. Backup of keys is securely stored (encrypted, offline)

```bash
# Remove private keys from .env files
find . -name ".env" -type f -exec sed -i '/^PRIVATE_KEY=/d' {} \;

# Remove from documentation
find docs -name "*.md" -type f -exec sed -i 's/0x[0-9a-fA-F]\{64\}/[PRIVATE_KEY_REDACTED]/g' {} \;
```

---

### Phase 3: Key Rotation (Days 4-5)

#### Step 7: Generate New Keys in HSM

If keys were exposed, generate new keys:

```bash
# AWS KMS - Generate new key
aws kms create-key --description "New Blockchain Deployer Key"

# Azure Key Vault - Generate new key
az keyvault key create --vault-name blockchain-keyvault --name deployer-key-new --kty EC

# HashiCorp Vault - Generate new key
vault write -f transit/keys/blockchain-deployer-new
```

#### Step 8: Update Deployed Contracts

If using new keys, update contract ownership:

```solidity
// Transfer ownership to new address
contract.transferOwnership(newOwnerAddress);
```

---

## API Tokens and Passwords Migration

### Cloudflare API Credentials

**Current Locations:**
- `proxmox/.env` - API key and tunnel token
- `scripts/fix-certbot-dns-propagation.sh` - Hardcoded token
- `scripts/install-shared-tunnel-token.sh` - Hardcoded tunnel token

**Migration Steps:**

1. **Create API tokens in Cloudflare** (not global API key)
   - Limit permissions to specific operations
   - Set expiration dates

2. **Store in Key Vault:**
```bash
# AWS Secrets Manager
aws secretsmanager create-secret \
  --name cloudflare/api-token \
  --secret-string "your-token"

# Azure Key Vault
az keyvault secret set \
  --vault-name blockchain-keyvault \
  --name cloudflare-api-token \
  --value "your-token"

# HashiCorp Vault
vault kv put secret/cloudflare api-token="your-token"
```

3. **Update scripts to use Key Vault:**
```bash
# Before
CLOUDFLARE_API_TOKEN="hardcoded-token"

# After
CLOUDFLARE_API_TOKEN=$(aws secretsmanager get-secret-value \
  --secret-id cloudflare/api-token \
  --query SecretString --output text)
```

---

### NPM (Nginx Proxy Manager) Credentials

**Current Locations:**
- `proxmox/.env` - Plain text password
- `scripts/create-npmplus-proxy.sh` - Hardcoded password hash
- `scripts/nginx-proxy-manager/update-npmplus-proxy-hosts-api.sh` - Hardcoded password

**Migration Steps:**

1. **Store in Key Vault**
2. **Update scripts** (already done for some scripts)
3. **Use API tokens instead of passwords** (if NPM supports it)

---

### Database Credentials

**Current Locations:**
- `dbis_core/.env` - DATABASE_URL with embedded password
- `explorer-monorepo/.env` - Database credentials

**Migration Steps:**

1. **Use Vault database secrets engine** (dynamic credentials)
2. **Separate password from connection string**
3. **Rotate credentials regularly**

---

## Implementation Checklist

### Immediate (Day 1)
- [ ] Identify all private keys
- [ ] Document current keys (encrypted)
- [ ] Set up HSM/Key Vault
- [ ] Create secure backup of keys

### Short Term (Days 2-3)
- [ ] Import keys to HSM
- [ ] Update application code to use HSM
- [ ] Test all operations with HSM
- [ ] Remove keys from files
- [ ] Remove keys from documentation

### Medium Term (Days 4-5)
- [ ] Generate new keys (if keys were exposed)
- [ ] Update contract ownership
- [ ] Rotate API tokens
- [ ] Update all scripts to use Key Vault

### Long Term (Week 2+)
- [ ] Set up automated key rotation
- [ ] Implement key rotation policies
- [ ] Set up monitoring and alerts
- [ ] Document HSM operations procedures

---

## Security Best Practices

### Key Management
1. **Never export private keys from HSM**
2. **Use HSM for all cryptographic operations**
3. **Rotate keys regularly** (annually or after exposure)
4. **Limit key access** (principle of least privilege)
5. **Monitor key usage** (audit logs)

### Access Control
1. **Use IAM roles** (not API keys) where possible
2. **Limit permissions** (minimum required)
3. **Set expiration dates** on tokens
4. **Rotate credentials regularly**

### Monitoring
1. **Enable audit logging** for all key operations
2. **Set up alerts** for unusual key usage
3. **Review access logs** regularly
4. **Monitor for exposed credentials** (GitHub, logs, etc.)

---

## Tools and Resources

### HSM/Key Vault Options
- **AWS KMS** - Managed key service
- **AWS CloudHSM** - Dedicated HSM
- **Azure Key Vault** - Managed key service
- **HashiCorp Vault** - Self-hosted secrets management
- **Google Cloud KMS** - Managed key service

### Migration Tools
- **git-secrets** - Prevent committing secrets
- **truffleHog** - Scan for secrets in git history
- **detect-secrets** - Detect secrets in codebase
- **AWS Secrets Manager Migration** - Migrate to AWS

### Documentation
- [AWS KMS Best Practices](https://docs.aws.amazon.com/kms/latest/developerguide/best-practices.html)
- [Azure Key Vault Best Practices](https://docs.microsoft.com/en-us/azure/key-vault/general/best-practices)
- [HashiCorp Vault Best Practices](https://learn.hashicorp.com/tutorials/vault/production-hardening)

---

## Emergency Procedures

### If Keys Are Exposed

1. **Immediately rotate keys**
2. **Revoke access** for exposed keys
3. **Monitor accounts** for unauthorized activity
4. **Review transaction history**
5. **Update all systems** with new keys
6. **Document incident** and lessons learned

### If HSM Becomes Unavailable

1. **Use backup HSM** (if available)
2. **Use encrypted backup keys** (last resort)
3. **Document downtime** and impact
4. **Review and improve** redundancy

---

## Status Tracking

**Current Status:** 🔴 **CRITICAL - IMMEDIATE ACTION REQUIRED**

**Next Review Date:** After Phase 1 completion

**Responsible Party:** [To be assigned]

---

**⚠️ WARNING:** Do not delay this migration. Private keys in files represent a critical security risk that could result in complete compromise of blockchain accounts and financial loss.