# Validator Transitions Runbook

## Overview

This runbook provides procedures for adding and removing validators from the network.

## Adding a Validator

### Prerequisites

- New validator keys generated
- Validator infrastructure ready
- Coordination with existing validators

### Procedure

1. **Generate Validator Keys**
   ```bash
   ./scripts/key-management/generate-validator-keys.sh 1
   ```

2. **Store Keys in Key Vault**
   ```bash
   az keyvault secret set --vault-name defi-oracle-kv --name validator-key-new --value <private-key>
   ```

3. **Update Genesis File**
   - Add validator address to extraData
   - Regenerate genesis file using Besu operator

4. **Update Static Nodes**
   - Add validator enode to static-nodes.json
   - Update all node configurations

5. **Deploy Validator**
   ```bash
   # Update StatefulSet replica count
   kubectl scale statefulset/besu-validator --replicas=5 -n besu-network
   ```

6. **Verify Validator**
   - Check validator is producing blocks
   - Verify validator is in validator set
   - Monitor validator performance

## Removing a Validator

### Prerequisites

- Coordination with validator operator
- Backup of validator data
- Maintenance window scheduled

### Procedure

1. **Plan Removal**
   - Notify all validators
   - Schedule maintenance window
   - Plan rollback procedure

2. **Update Genesis File**
   - Remove validator address from extraData
   - Regenerate genesis file

3. **Update Static Nodes**
   - Remove validator enode from static-nodes.json
   - Update all node configurations

4. **Remove Validator Pod**
   ```bash
   kubectl scale statefulset/besu-validator --replicas=3 -n besu-network
   ```

5. **Verify Removal**
   - Check validator set
   - Verify network is still producing blocks
   - Monitor network performance

## Validator Key Rotation

See [Key Rotation Script](../scripts/key-management/rotate-keys.sh) for detailed procedures.

## Troubleshooting

### Validator Not Producing Blocks

1. Check validator status
2. Verify validator keys
3. Check network connectivity
4. Verify IBFT configuration
5. Check logs

### Validator Not in Validator Set

1. Verify validator address in extraData
2. Check static nodes configuration
3. Verify network connectivity
4. Check validator keys

## Contacts

- **Validators**: validators@d-bis.org
- **Network Operators**: operators@d-bis.org

## References

- [Node Add/Remove Runbook](node-add-remove.md)
- [Parameter Change Runbook](parameter-change.md)
- [Key Management Scripts](../scripts/key-management/)