Files

defiQUG cb47cce074 Complete markdown files cleanup and organization

- Organized 252 files across project
- Root directory: 187 → 2 files (98.9% reduction)
- Moved configuration guides to docs/04-configuration/
- Moved troubleshooting guides to docs/09-troubleshooting/
- Moved quick start guides to docs/01-getting-started/
- Moved reports to reports/ directory
- Archived temporary files
- Generated comprehensive reports and documentation
- Created maintenance scripts and guides

All files organized according to established standards.

2026-01-06 01:46:25 -08:00

20 KiB

Raw Blame History

Comprehensive Infrastructure Review

Last Updated: 2025-12-27
Document Version: 1.0
Status: Active Documentation
Review Scope: All Tunnels, DNS Entries, Nginx Configurations, VMIDs

Executive Summary

This document provides a comprehensive review of:

✅ All Cloudflare Tunnels
✅ All DNS Entries
✅ All Nginx Configurations
✅ All VMIDs and Services
✅ Recommendations for Optimization

1. Cloudflare Tunnels Review

Active Tunnels

Tunnel Name	Tunnel ID	Status	Location	Purpose
`explorer.d-bis.org`	`b02fe1fe-cb7d-484e-909b-7cc41298ebe8`	✅ HEALTHY	VMID 102	Explorer/Blockscout
`rpc-http-pub.d-bis.org`	`10ab22da-8ea3-4e2e-a896-27ece2211a05`	⚠️ DOWN	VMID 102	RPC Services (needs config)
`mim4u-tunnel`	`f8d06879-04f8-44ef-aeda-ce84564a1792`	✅ HEALTHY	Unknown	Miracles In Motion
`tunnel-ml110`	`ccd7150a-9881-4b8c-a105-9b4ead6e69a2`	✅ HEALTHY	Unknown	Proxmox Host Access
`tunnel-r630-01`	`4481af8f-b24c-4cd3-bdd5-f562f4c97df4`	✅ HEALTHY	Unknown	Proxmox Host Access
`tunnel-r630-02`	`0876f12b-64d7-4927-9ab3-94cb6cf48af9`	✅ HEALTHY	Unknown	Proxmox Host Access

Current Tunnel Configuration (VMID 102)

Active Tunnel: rpc-http-pub.d-bis.org (Tunnel ID: 10ab22da-8ea3-4e2e-a896-27ece2211a05)

Current Routing (from logs):

rpc-ws-pub.d-bis.org → https://192.168.11.252:443
rpc-http-prv.d-bis.org → https://192.168.11.251:443
rpc-ws-prv.d-bis.org → https://192.168.11.251:443
rpc-http-pub.d-bis.org → https://192.168.11.252:443

⚠️ Issue: Tunnel is routing directly to RPC nodes instead of central Nginx

✅ Recommended Configuration:

All HTTP endpoints → http://192.168.11.21:80 (Central Nginx)
WebSocket endpoints → Direct to RPC nodes (as configured)

2. DNS Entries Review

Current DNS Records (from d-bis.org zone file)

A Records (Direct IPs)

Domain	IP Address(es)	Proxy Status	Notes
`api.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`besu.d-bis.org`	20.215.32.42, 70.153.83.83	✅ Proxied	DUPLICATE - Remove one
`blockscout.d-bis.org`	20.215.32.42, 70.153.83.83	✅ Proxied	DUPLICATE - Remove one
`d-bis.org` (root)	20.215.32.42, 20.215.32.15	✅ Proxied	DUPLICATE - Remove one
`docs.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`explorer.d-bis.org`	20.215.32.42, 70.153.83.83	✅ Proxied	DUPLICATE - Remove one
`grafana.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`metrics.d-bis.org`	70.153.83.83	❌ Not Proxied	Should use tunnel
`monitoring.d-bis.org`	70.153.83.83	✅ Proxied	Should use tunnel
`prometheus.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`tessera.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`wallet.d-bis.org`	70.153.83.83	✅ Proxied	Should use tunnel
`ws.d-bis.org`	20.8.47.226	❌ Not Proxied	Should use tunnel
`www.d-bis.org`	20.8.47.226	✅ Proxied	Should use tunnel

CNAME Records (Tunnel-based)

Domain	Target	Proxy Status	Notes
`rpc.d-bis.org`	`dbis138fdendpoint-cgergbcqb7aca7at.a03.azurefd.net`	✅ Proxied	Azure Front Door
`ipfs.d-bis.org`	`ipfs.cloudflare.com`	✅ Proxied	Cloudflare IPFS

Missing DNS Records (Should Exist)

Domain	Type	Target	Status
`rpc-http-pub.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`rpc-ws-pub.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`rpc-http-prv.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`rpc-ws-prv.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`dbis-admin.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`dbis-api.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`dbis-api-2.d-bis.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`mim4u.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing
`www.mim4u.org`	CNAME	`<tunnel-id>.cfargotunnel.com`	❌ Missing

3. Nginx Configurations Review

Central Nginx (VMID 105 - 192.168.11.21)

Status: ✅ Configured
Configuration: /data/nginx/custom/http.conf
Type: Nginx Proxy Manager (OpenResty)

Configured Services:

✅ explorer.d-bis.org → http://192.168.11.140:80
✅ rpc-http-pub.d-bis.org → https://192.168.11.252:443
✅ rpc-http-prv.d-bis.org → https://192.168.11.251:443
✅ dbis-admin.d-bis.org → http://192.168.11.130:80
✅ dbis-api.d-bis.org → http://192.168.11.150:3000
✅ dbis-api-2.d-bis.org → http://192.168.11.151:3000
✅ mim4u.org → http://192.168.11.19:80
✅ www.mim4u.org → 301 Redirect → mim4u.org

Note: WebSocket endpoints (rpc-ws-*) are NOT in this config (routing directly)

Blockscout Nginx (VMID 5000 - 192.168.11.140)

Status: ✅ Running
Configuration: /etc/nginx/sites-available/blockscout
Purpose: Local Nginx for Blockscout service

Ports:

Port 80: HTTP (redirects to HTTPS or serves content)
Port 443: HTTPS (proxies to Blockscout on port 4000)

Miracles In Motion Nginx (VMID 7810 - 192.168.11.19)

Status: ✅ Running
Configuration: /etc/nginx/sites-available/default
Purpose: Web frontend and API proxy

Ports:

Port 80: HTTP (serves static files, proxies API to 192.168.11.8:3001)

DBIS Frontend Nginx (VMID 10130 - 192.168.11.130)

Status: ✅ Running (assumed)
Purpose: Frontend admin console

RPC Nodes Nginx (VMIDs 2500, 2501, 2502)

Status: ⚠️ Partially Configured
Purpose: SSL termination and local routing

VMID 2500 (192.168.11.250):

Port 443: HTTPS RPC → 127.0.0.1:8545
Port 8443: HTTPS WebSocket → 127.0.0.1:8546

VMID 2501 (192.168.11.251):

Port 443: HTTPS RPC → 127.0.0.1:8545
Port 443: HTTPS WebSocket → 127.0.0.1:8546 (SNI-based)

VMID 2502 (192.168.11.252):

Port 443: HTTPS RPC → 127.0.0.1:8545
Port 443: HTTPS WebSocket → 127.0.0.1:8546 (SNI-based)

4. VMIDs Review

Infrastructure Services

VMID	Name	IP	Status	Purpose
100	proxmox-mail-gateway	192.168.11.32	✅ Running	Mail gateway
101	proxmox-datacenter-manager	192.168.11.33	✅ Running	Datacenter management
102	cloudflared	192.168.11.34	✅ Running	Cloudflare tunnel client
103	omada	192.168.11.30	✅ Running	Network management
104	gitea	192.168.11.31	✅ Running	Git repository
105	nginxproxymanager	192.168.11.26	✅ Running	Central Nginx reverse proxy
130	monitoring-1	192.168.11.27	✅ Running	Monitoring stack

Blockchain Services

VMID	Name	IP	Status	Purpose	Notes
5000	blockscout-1	192.168.11.140	✅ Running	Blockchain explorer	Has local Nginx
6200	firefly-1	192.168.11.7	✅ Running	Hyperledger Firefly	Web3 gateway

RPC Nodes

VMID	Name	IP	Status	Purpose	Notes
2500	besu-rpc-1	192.168.11.250	✅ Running	Core RPC	Located on ml110 (192.168.11.10)
2501	besu-rpc-2	192.168.11.251	✅ Running	Permissioned RPC	Located on ml110 (192.168.11.10)
2502	besu-rpc-3	192.168.11.252	✅ Running	Public RPC	Located on ml110 (192.168.11.10)

✅ Status: RPC nodes are running on ml110 (192.168.11.10), not on pve2.

Application Services

VMID	Name	IP	Status	Purpose
7800	sankofa-api-1	192.168.11.13	✅ Running	Sankofa API
7801	sankofa-portal-1	192.168.11.16	✅ Running	Sankofa Portal
7802	sankofa-keycloak-1	192.168.11.17	✅ Running	Sankofa Keycloak
7810	mim-web-1	192.168.11.19	✅ Running	Miracles In Motion Web
7811	mim-api-1	192.168.11.8	✅ Running	Miracles In Motion API

DBIS Core Services

VMID	Name	IP	Status	Purpose	Notes
10100	dbis-postgres-primary	192.168.11.100	✅ Running	PostgreSQL Primary	Located on ml110 (192.168.11.10)
10101	dbis-postgres-replica-1	192.168.11.101	✅ Running	PostgreSQL Replica	Located on ml110 (192.168.11.10)
10120	dbis-redis	192.168.11.120	✅ Running	Redis Cache	Located on ml110 (192.168.11.10)
10130	dbis-frontend	192.168.11.130	✅ Running	Frontend Admin	Located on ml110 (192.168.11.10)
10150	dbis-api-primary	192.168.11.150	✅ Running	API Primary	Located on ml110 (192.168.11.10)
10151	dbis-api-secondary	192.168.11.151	✅ Running	API Secondary	Located on ml110 (192.168.11.10)

✅ Status: DBIS Core containers are running on ml110 (192.168.11.10), not on pve2.

5. Critical Issues Identified

🔴 High Priority

Tunnel Configuration Mismatch
- Tunnel rpc-http-pub.d-bis.org is DOWN
- Currently routing directly to RPC nodes instead of central Nginx
- Action: Update Cloudflare dashboard to route HTTP endpoints to http://192.168.11.21:80
Missing DNS Records
- RPC endpoints (rpc-http-pub, rpc-ws-pub, rpc-http-prv, rpc-ws-prv) missing CNAME records
- DBIS services (dbis-admin, dbis-api, dbis-api-2) missing CNAME records
- mim4u.org and www.mim4u.org missing CNAME records
- Action: Create CNAME records pointing to tunnel
Duplicate DNS A Records
- besu.d-bis.org: 2 A records (20.215.32.42, 70.153.83.83)
- blockscout.d-bis.org: 2 A records (20.215.32.42, 70.153.83.83)
- explorer.d-bis.org: 2 A records (20.215.32.42, 70.153.83.83)
- d-bis.org: 2 A records (20.215.32.42, 20.215.32.15)
- Action: Remove duplicate records, keep single authoritative IP
RPC Nodes Location
- ✅ VMIDs 2500, 2501, 2502 found on ml110 (192.168.11.10)
- Action: Verify network connectivity from pve2 to ml110
DBIS Core Services Location
- ✅ VMIDs 10100-10151 found on ml110 (192.168.11.10)
- Action: Verify network connectivity from pve2 to ml110

🟡 Medium Priority

DNS Records Using Direct IPs Instead of Tunnels
- Many services use A records with direct IPs
- Should use CNAME records pointing to tunnel
- Action: Migrate to tunnel-based DNS
Inconsistent Proxy Status
- Some records proxied, some not
- Action: Standardize proxy status (proxied for public services)
Multiple Nginx Instances
- Central Nginx (105), Blockscout Nginx (5000), MIM Nginx (7810), RPC Nginx (2500-2502)
- Action: Consider consolidating or document purpose of each

🟢 Low Priority

Documentation Gaps
- Some VMIDs have incomplete documentation
- Action: Update documentation with current status
Service Discovery
- No centralized service registry
- Action: Consider implementing service discovery

6. Recommendations

Immediate Actions (Critical)

Fix Tunnel Configuration

# Update Cloudflare dashboard for tunnel: rpc-http-pub.d-bis.org
# Route all HTTP endpoints to central Nginx:
- explorer.d-bis.org → http://192.168.11.21:80
- rpc-http-pub.d-bis.org → http://192.168.11.21:80
- rpc-http-prv.d-bis.org → http://192.168.11.21:80
- dbis-admin.d-bis.org → http://192.168.11.21:80
- dbis-api.d-bis.org → http://192.168.11.21:80
- dbis-api-2.d-bis.org → http://192.168.11.21:80
- mim4u.org → http://192.168.11.21:80
- www.mim4u.org → http://192.168.11.21:80

Create Missing DNS Records
- Create CNAME records for all RPC endpoints
- Create CNAME records for DBIS services
- Create CNAME records for MIM services
- All should point to: <tunnel-id>.cfargotunnel.com
- Enable proxy (orange cloud) for all
Remove Duplicate DNS Records
- Remove duplicate A records for besu.d-bis.org
- Remove duplicate A records for blockscout.d-bis.org
- Remove duplicate A records for explorer.d-bis.org
- Remove duplicate A records for d-bis.org (keep 20.215.32.15)
Locate Missing VMIDs
- Find RPC nodes (2500-2502) on other Proxmox hosts
- Verify DBIS Core services (10100-10151) deployment status

Short-term Improvements

DNS Migration to Tunnels
- Migrate all A records to CNAME records pointing to tunnels
- Remove direct IP exposure
- Enable proxy for all public services
Tunnel Consolidation
- Consider consolidating multiple tunnels into single tunnel
- Use central Nginx for all HTTP routing
- Simplify tunnel management
Nginx Architecture Review
- Document purpose of each Nginx instance
- Consider if all are necessary
- Standardize configuration approach

Long-term Optimizations

Service Discovery
- Implement centralized service registry
- Automate DNS record creation
- Dynamic service routing
Monitoring and Alerting
- Monitor all tunnel health
- Alert on tunnel failures
- Track DNS record changes
Documentation
- Maintain up-to-date infrastructure map
- Document all service dependencies
- Create runbooks for common operations

7. Architecture Recommendations

Recommended Architecture

Internet
  ↓
Cloudflare (DNS + SSL Termination)
  ↓
Cloudflare Tunnel (VMID 102)
  ↓
Routing Decision:
  ├─ HTTP Services → Central Nginx (VMID 105:80) → Internal Services
  └─ WebSocket Services → Direct to RPC Nodes (bypass Nginx)

Key Principle:

HTTP traffic routes through central Nginx for unified management
WebSocket traffic routes directly to RPC nodes for optimal performance

Benefits

Single Point of Configuration: All HTTP routing in one place
Simplified Management: Easy to add/remove services
Better Security: No direct IP exposure
Centralized Logging: All traffic logs in one location
Easier Troubleshooting: Single point to check routing

8. Action Items Checklist

Critical (Do First)

Update Cloudflare tunnel configuration to route HTTP endpoints to central Nginx
Create missing DNS CNAME records for all services
Remove duplicate DNS A records
Locate and verify RPC nodes (2500-2502) - ✅ Found on ml110
Verify DBIS Core services deployment status - ✅ Found on ml110
Verify network connectivity from pve2 (192.168.11.12) to ml110 (192.168.11.10)

Important (Do Next)

Migrate remaining A records to CNAME (tunnel-based)
Standardize proxy status across all DNS records
Document all Nginx instances and their purposes
Test all endpoints after configuration changes

Nice to Have

Implement service discovery
Set up monitoring and alerting
Create comprehensive infrastructure documentation
Automate DNS record management

9. DNS Records Migration Plan

Current State (A Records - Direct IPs)

Many services use A records pointing to direct IPs. These should be migrated to CNAME records pointing to Cloudflare tunnels.

Migration Priority

High Priority (Public-facing services):

explorer.d-bis.org → CNAME to tunnel
rpc-http-pub.d-bis.org → CNAME to tunnel
rpc-ws-pub.d-bis.org → CNAME to tunnel
rpc-http-prv.d-bis.org → CNAME to tunnel
rpc-ws-prv.d-bis.org → CNAME to tunnel

Medium Priority (Internal services): 6. dbis-admin.d-bis.org → CNAME to tunnel 7. dbis-api.d-bis.org → CNAME to tunnel 8. dbis-api-2.d-bis.org → CNAME to tunnel 9. mim4u.org → CNAME to tunnel 10. www.mim4u.org → CNAME to tunnel

Low Priority (Monitoring/internal): 11. grafana.d-bis.org → CNAME to tunnel (if public access needed) 12. prometheus.d-bis.org → CNAME to tunnel (if public access needed) 13. monitoring.d-bis.org → CNAME to tunnel

Migration Steps

For each domain:

Create CNAME record: <subdomain> → <tunnel-id>.cfargotunnel.com
Enable proxy (orange cloud)
Wait for DNS propagation (1-5 minutes)
Test endpoint accessibility
Remove old A record (if exists)

10. Testing Plan

After implementing recommendations:

Test HTTP Endpoints:

curl https://explorer.d-bis.org/api/v2/stats
curl -X POST https://rpc-http-pub.d-bis.org \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}'
curl https://dbis-admin.d-bis.org
curl https://mim4u.org

Test WebSocket Endpoints:

wscat -c wss://rpc-ws-pub.d-bis.org
wscat -c wss://rpc-ws-prv.d-bis.org

Test Redirects:

curl -I https://www.mim4u.org  # Should redirect to mim4u.org

Verify Tunnel Health:
- Check Cloudflare dashboard for tunnel status
- Verify all tunnels show HEALTHY
- Check tunnel logs for errors

11. Summary of Recommendations

🔴 Critical (Fix Immediately)

Update Cloudflare Tunnel Configuration
- Tunnel: rpc-http-pub.d-bis.org (Tunnel ID: 10ab22da-8ea3-4e2e-a896-27ece2211a05)
- Action: Route all HTTP endpoints to http://192.168.11.21:80 (central Nginx)
- Keep WebSocket endpoints routing directly to RPC nodes
Create Missing DNS CNAME Records
- rpc-http-pub.d-bis.org → CNAME to tunnel
- rpc-ws-pub.d-bis.org → CNAME to tunnel
- rpc-http-prv.d-bis.org → CNAME to tunnel
- rpc-ws-prv.d-bis.org → CNAME to tunnel
- dbis-admin.d-bis.org → CNAME to tunnel
- dbis-api.d-bis.org → CNAME to tunnel
- dbis-api-2.d-bis.org → CNAME to tunnel
- mim4u.org → CNAME to tunnel
- www.mim4u.org → CNAME to tunnel
Remove Duplicate DNS A Records
- besu.d-bis.org: Remove one IP (keep single authoritative)
- blockscout.d-bis.org: Remove one IP
- explorer.d-bis.org: Remove one IP
- d-bis.org: Remove 20.215.32.42 (keep 20.215.32.15)

🟡 Important (Fix Soon)

Migrate A Records to CNAME (Tunnel-based)
- Convert remaining A records to CNAME records
- Point all to Cloudflare tunnel endpoints
- Enable proxy (orange cloud) for all public services
Verify Network Connectivity
- Test connectivity from pve2 (192.168.11.12) to ml110 (192.168.11.10)
- Ensure RPC nodes (2500-2502) are accessible from central Nginx
- Ensure DBIS services (10100-10151) are accessible from central Nginx

🟢 Optimization (Nice to Have)

Documentation Updates
- Update all service documentation with current IPs and locations
- Document network topology (pve2 vs ml110)
- Create service dependency map
Monitoring Setup
- Monitor all tunnel health
- Alert on tunnel failures
- Track DNS record changes

Architecture Documents

NETWORK_ARCHITECTURE.md ⭐⭐⭐ - Complete network architecture
PHYSICAL_HARDWARE_INVENTORY.md ⭐⭐⭐ - Physical hardware inventory
ORCHESTRATION_DEPLOYMENT_GUIDE.md ⭐⭐⭐ - Deployment orchestration
DOMAIN_STRUCTURE.md ⭐⭐ - Domain structure

Network Documents

../05-network/CLOUDFLARE_TUNNEL_ROUTING_ARCHITECTURE.md - Cloudflare tunnel routing
../05-network/CENTRAL_NGINX_ROUTING_SETUP.md - Central Nginx routing

Configuration Documents

../04-configuration/cloudflare/CLOUDFLARE_DNS_TO_CONTAINERS.md - DNS mapping to containers
../04-configuration/RPC_DNS_CONFIGURATION.md - RPC DNS configuration

Last Updated: 2025-12-27
Document Version: 1.0
Review Cycle: Quarterly

20 KiB Raw Blame History