explorer-monorepo/docs/REUSABLE_COMPONENTS_EXTRACTION_PLAN.md

# Reusable Components Extraction Plan

**Completion status (in-repo):** All libs are present under `backend/libs/` and `frontend/libs/`. Backend is wired to use **go-pgconfig** (API + indexer), **go-rpc-gateway** (Track 1). Frontend is wired to use **frontend-api-client** (services/api/client) and **frontend-ui-primitives** (all pages using Card, Table, Address). CI uses `submodules: recursive`; README documents clone with submodules. To publish as separate repos, copy each lib to its own repo and add as submodule.

**Review and test:** Backend handlers that need the DB use `requireDB(w)`; without a DB they return 503. Tests run with a nil DB and accept 200/503/404 as appropriate. Run backend tests: `go test ./...` in `backend/`. Frontend build: `npm run build` in `frontend/` (ESLint uses root `.eslintrc.cjs` and frontend `"root": true` in `.eslintrc.json`). E2E: `npm run e2e` from repo root (Playwright, default base URL https://explorer.d-bis.org; set `EXPLORER_URL` for local).

---

**Goal:** Extract reusable, non–Explorer-specific components into their own repositories under `/home/intlc/projects`, and link them back to this monorepo via **git submodules** using best practices.

**Scope:** Components that are generic (EVM/chain-agnostic, multi-tenant, or generic UI/infra) are candidates. Explorer-specific logic (Chain 138 routes, Blockscout integration, explorer branding) stays in `explorer-monorepo`.

---

## 1. Reusable vs Explorer-Specific (Summary)

| Category | Reusable (extract) | Explorer-specific (keep in monorepo) |
|--------|---------------------|--------------------------------------|
| **Backend Go** | Bridge/swap aggregators, chain adapters, DB config, wallet auth + tiered access, logging/metrics, generic cache/rate-limit, security middleware (generic part) | REST routes, track* endpoints, indexer (block/processor), Blockscout/etherscan compat, explorer API server |
| **Frontend** | Button, Card, Table, generic API client, optional Address (minimal) | Explorer pages, block/tx/address views, SPA `index.html`, explorer API calls |
| **Deployment** | Nginx/systemd/Cloudflare/fail2ban templates, generic verify scripts | Explorer deploy scripts, VMID 5000 / Blockscout-specific fixes |
| **Docs** | — | All current docs are explorer/deployment-specific |

---

## 2. Proposed New Repositories (under `/home/intlc/projects`)

All new repos live as siblings of `proxmox/explorer-monorepo`, e.g.:

- `/home/intlc/projects/go-bridge-aggregator`
- `/home/intlc/projects/go-chain-adapters`
- `/home/intlc/projects/go-pgconfig`
- `/home/intlc/projects/go-tiered-auth`
- `/home/intlc/projects/go-http-middleware`
- `/home/intlc/projects/go-logging`
- `/home/intlc/projects/go-rpc-gateway`
- `/home/intlc/projects/frontend-ui-primitives`
- `/home/intlc/projects/deployment-common`

Relationship to monorepo: **submodules** inside `explorer-monorepo` (e.g. `backend/vendor/go-bridge-aggregator` or `libs/go-bridge-aggregator`), so the monorepo stays the single checkout for development and CI.

---

## 3. Repository-by-Repository Plan

### 3.1 `go-bridge-aggregator`

- **Purpose:** Multi-provider bridge quote aggregation (chain-agnostic interface + implementations).
- **Source in monorepo:** `backend/bridge/` (all providers: `providers.go`, `ccip_provider.go`, `lifi_provider.go`, `socket_provider.go`, `squid_provider.go`, `symbiosis_provider.go`, `relay_provider.go`, `stargate_provider.go`, `hop_provider.go`).
- **Reusability:** Provider interface, `BridgeRequest`/`BridgeQuote`/`BridgeStep`, `Aggregator`, `GetBestQuote`. Chain IDs and integrator names should be config (env or struct), not hardcoded 138.
- **New repo path:** `/home/intlc/projects/go-bridge-aggregator`.
- **Suggested layout:** `go.mod` (module e.g. `github.com/yourorg/go-bridge-aggregator`), `provider.go`, `aggregator.go`, `providers/` (one file per provider). Dependencies: minimal (HTTP client, context).
- **Explorer coupling:** Explorer will depend on this module via Go module (or submodule + replace). CCIP “138 <-> 1” and integrator strings become config in explorer.

---

### 3.2 `go-chain-adapters`

- **Purpose:** Chain abstraction for EVM-compatible chains (block, tx, receipt, balance, code, gas).
- **Source:** `backend/chain/adapters/evm.go` (interface `ChainAdapter` + `EVMAdapter`).
- **Reusability:** No explorer references; only `go-ethereum`.
- **New repo path:** `/home/intlc/projects/go-chain-adapters`.
- **Suggested layout:** `go.mod`, `adapter.go` (interface), `evm/evm.go`. Other chains (e.g. non-EVM) can be added later in same repo.
- **Explorer coupling:** Explorer indexer and API use chain adapter; import this module.

---

### 3.3 `go-pgconfig`

- **Purpose:** Load PostgreSQL configuration from environment and build `pgxpool.Config`.
- **Source:** `backend/database/config/database.go` (and optionally read-replica).
- **Reusability:** No explorer references; only `pgx` and `os`.
- **New repo path:** `/home/intlc/projects/go-pgconfig`.
- **Suggested layout:** `go.mod`, `config.go` (DatabaseConfig, LoadDatabaseConfig, ConnectionString, PoolConfig, ReadReplicaConfig).
- **Explorer coupling:** Explorer `api/rest/cmd/main.go` and indexer use DB config; import this module.

---

### 3.4 `go-tiered-auth`

- **Purpose:** Tiered (track) access: wallet-based auth (nonce + JWT), optional “tier” level, and feature access by tier.
- **Source:** `backend/auth/` (auth.go, roles.go, wallet_auth.go), `backend/featureflags/flags.go`, and the auth/track parts of `backend/api/middleware/auth.go`.
- **Reusability:** Wallet nonce + JWT and “tier” (track) are generic; feature names in `featureflags` can be default map, overridable by consumer. Middleware uses `auth` + `featureflags`; can live in this repo or in `go-http-middleware` with dependency on this repo.
- **New repo path:** `/home/intlc/projects/go-tiered-auth`.
- **Suggested layout:** `go.mod`, `auth/` (wallet_auth, nonce storage interface), `featureflags/` (HasAccess, GetEnabledFeatures, configurable feature map), `middleware/` (RequireAuth, RequireTier, OptionalAuth). DB tables (e.g. `wallet_nonces`, `operator_roles`) stay defined in explorer migrations; this repo only needs interfaces (e.g. NonceStore, TierResolver).
- **Explorer coupling:** Explorer implements storage (existing DB); middleware and WalletAuth stay in explorer but can call into this lib for validation and JWT. Alternatively, move full WalletAuth + middleware here and pass DB and feature map from explorer.

---

### 3.5 `go-http-middleware`

- **Purpose:** Generic HTTP middleware: security headers (CSP, X-Frame-Options, etc.), CORS, compression, logging. No explorer URLs.
- **Source:** `backend/api/middleware/security.go` (generic headers only; CSP `connect-src` etc. are explorer-specific and should be supplied by explorer or config).
- **Reusability:** Security headers and CORS are generic; CSP should be a parameter or callback.
- **New repo path:** `/home/intlc/projects/go-http-middleware`.
- **Suggested layout:** `go.mod`, `security.go` (AddSecurityHeaders with configurable CSP), `cors.go` if extracted, `logging.go` if request logging is extracted. Explorer passes CSP string when wiring middleware.
- **Explorer coupling:** Explorer server uses this middleware with explorer-specific CSP.

---

### 3.6 `go-logging`

- **Purpose:** Structured logging (context, fields, levels).
- **Source:** `backend/logging/logger.go`.
- **Reusability:** No explorer references.
- **New repo path:** `/home/intlc/projects/go-logging`.
- **Suggested layout:** `go.mod`, `logger.go`. Optional: integrate with a standard logger interface (e.g. slog) for compatibility.
- **Explorer coupling:** Optional; explorer can keep using it or switch to stdlib/slog and deprecate this in monorepo.

---

### 3.7 `go-rpc-gateway`

- **Purpose:** Generic RPC gateway: in-memory (and optionally Redis) cache, in-memory (and optionally Redis) rate limiter, and a generic HTTP proxy to an upstream RPC URL. No route definitions or explorer semantics.
- **Source:** `backend/api/track1/cache.go`, `redis_cache.go`, `rate_limiter.go`, `redis_rate_limiter.go`, `rpc_gateway.go`. Exclude `endpoints.go` (explorer-specific handlers).
- **Reusability:** Cache and rate limiter are generic; RPC gateway is “forward request to RPC URL with cache and rate limit.”
- **New repo path:** `/home/intlc/projects/go-rpc-gateway`.
- **Suggested layout:** `go.mod`, `cache/` (memory + Redis), `ratelimit/` (memory + Redis), `gateway.go` (proxy to single upstream). Explorer track1 endpoints call this gateway and format responses (e.g. chain_id 138).
- **Explorer coupling:** Explorer imports gateway + cache + ratelimit; track1 endpoints remain in explorer and use this package.

---

### 3.8 `frontend-ui-primitives`

- **Purpose:** Reusable React UI components: Button, Card, Table. Optionally a minimal Address (truncate + copy) without explorer-specific links.
- **Source:** `frontend/src/components/common/Button.tsx`, `Card.tsx`, `Table.tsx`; optionally `frontend/src/components/blockchain/Address.tsx` (strip explorer-specific links/props if any).
- **Reusability:** No explorer API or routes; only Tailwind + clsx + React. Table is generic; Address can be “display + copy” only.
- **New repo path:** `/home/intlc/projects/frontend-ui-primitives`.
- **Suggested layout:** npm package (e.g. `@yourorg/ui-primitives`). Structure: `src/Button.tsx`, `Card.tsx`, `Table.tsx`, `Address.tsx`; export from `index.ts`; build with tsup or similar; peer deps: `react`, `clsx`. Tailwind: consumer app includes Tailwind and uses same class names, or package ships minimal CSS.
- **Explorer coupling:** Explorer frontend adds dependency (npm package or git submodule + workspace). If submodule: e.g. `frontend/libs/ui-primitives` and in package.json `"@yourorg/ui-primitives": "file:./libs/frontend-ui-primitives"`.

---

### 3.9 `frontend-api-client` (optional)

- **Purpose:** Generic API client: axios instance, interceptors, optional API key from storage, typed `ApiResponse<T>` and error shape.
- **Source:** `frontend/src/services/api/client.ts` (and possibly a minimal `types.ts` for ApiResponse/ApiError).
- **Reusability:** No explorer endpoints; base URL and headers are config.
- **New repo path:** `/home/intlc/projects/frontend-api-client`.
- **Suggested layout:** npm package exporting `createApiClient(baseURL, options?)` and types. Explorer uses `createApiClient(process.env.NEXT_PUBLIC_API_URL)` and keeps explorer-specific API modules (blocks, addresses, transactions) in monorepo.
- **Explorer coupling:** Explorer depends on this package; explorer-specific services import client from package and define endpoints locally.

---

### 3.10 `deployment-common`

- **Purpose:** Reusable deployment and ops snippets: nginx (generic reverse proxy, SSL, `location /api/` template), systemd unit templates, Cloudflare Tunnel sample config, fail2ban jail/config, and small generic “verify” scripts (e.g. curl health, check port).
- **Source:** `deployment/` – extract generic parts of `setup-nginx.sh`, `systemd/*.service`, `cloudflare/tunnel-config.yml`, `fail2ban/`. Omit explorer-specific env (VMID 5000, Blockscout, explorer.d-bis.org) or make them placeholders.
- **Reusability:** No explorer branding or VMIDs; only patterns (proxy to backend, systemd, tunnel, fail2ban).
- **New repo path:** `/home/intlc/projects/deployment-common`.
- **Suggested layout:** `nginx/`, `systemd/`, `cloudflare/`, `fail2ban/`, `scripts/` (e.g. verify-http-endpoint.sh). README with usage and variable placeholders.
- **Explorer coupling:** Explorer’s `deployment/` can include this repo as submodule (e.g. `deployment/common`) and copy or symlink templates, or reference them from docs.

---

## 4. Submodule Strategy (Best Practices)

- **Location of submodules:** Prefer a single directory for “vendored” or “lib” submodules to avoid clutter, e.g.:
  - **Option A:** `explorer-monorepo/libs/` with one submodule per repo (`libs/go-bridge-aggregator`, `libs/frontend-ui-primitives`, etc.).
  - **Option B:** Backend libs under `backend/libs/` and frontend under `frontend/libs/` (so Go and Node resolve paths naturally).
- **Recommendation:** Use **Option B** for simpler Go/Node resolution:
  - `backend/libs/go-bridge-aggregator`, `backend/libs/go-chain-adapters`, `backend/libs/go-pgconfig`, `backend/libs/go-tiered-auth`, `backend/libs/go-http-middleware`, `backend/libs/go-logging`, `backend/libs/go-rpc-gateway`.
  - `frontend/libs/frontend-ui-primitives` (and optionally `frontend/libs/frontend-api-client`).
  - `deployment/common` → `deployment-common` (or `deployment-common` at repo root if you prefer).

- **Adding a submodule (example):**
  ```bash
  cd /home/intlc/projects/proxmox/explorer-monorepo
  git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator
  ```
  Use SSH or HTTPS consistently; pin to a branch or tag (e.g. `main` or `v0.1.0`).

- **Go modules:** In explorer’s `backend/go.mod`, use `replace` to point at the local submodule during development:
  ```go
  replace github.com/yourorg/go-bridge-aggregator => ./libs/go-bridge-aggregator
  ```
  CI and contributors run `go mod tidy`; when publishing the library, depend on the published module version and remove `replace` for release.

- **Clone/update instructions:** Document in root README:
  ```bash
  git clone --recurse-submodules <explorer-monorepo-url>
  # or after clone:
  git submodule update --init --recursive
  ```

- **Publishing:** When a reusable repo is stable, publish it (Go: tag and push to GitHub; npm: publish to npm or private registry). Explorer can then depend on versioned releases and only use submodules for active development or private forks.

---

## 5. Phased Extraction Order

To minimize breakage and respect dependencies:

1. **Phase 1 – No dependency on other extracted pieces**
   - `go-pgconfig` (used by api + indexer)
   - `go-logging`
   - `go-chain-adapters`
   - `go-bridge-aggregator` (make chain IDs/config injectable)
   - `frontend-ui-primitives`
   - `deployment-common` (templates only)

2. **Phase 2 – May depend on Phase 1**
   - `go-rpc-gateway` (cache + rate limit + gateway; no dependency on auth)
   - `go-http-middleware` (security headers only)
   - `go-tiered-auth` (auth + featureflags + optional middleware; depends on DB interface, so explorer keeps DB)
   - `frontend-api-client` (optional)

3. **Phase 3 – Integration in explorer**
   - Replace internal packages with submodule or published module references.
   - Update explorer `backend/go.mod` and `frontend/package.json`.
   - Keep explorer-specific code (routes, indexer, Blockscout, SPA) in monorepo; ensure tests and CI pass.

---

## 6. What Stays in Explorer Monorepo

- **Backend:** All REST route handlers, track* endpoint logic, indexer (listener + processor + backfill), Blockscout/etherscan compatibility, explorer-specific config (chain_id 138, RPC URLs), and migrations (schema stays here; libs use interfaces or config).
- **Frontend:** All pages and views, `public/index.html` SPA, explorer API service modules (blocks, transactions, addresses), Next.js app and deployment config.
- **Deployment:** All explorer- and VMID 5000–specific scripts (`fix-502-blockscout.sh`, `complete-explorer-api-access.sh`, `deploy-frontend-to-vmid5000.sh`, etc.), and nginx/config that reference explorer.d-bis.org and Blockscout.
- **Docs:** All current documentation (API access, deployment, runbook, etc.).

---

## 7. Checklist for Each New Repo

- [ ] Create repo under `/home/intlc/projects/<repo-name>` (or under GitHub/GitLab first, then clone there).
- [ ] Add minimal `README.md`, `LICENSE`, and `.gitignore`.
- [ ] Copy only the reusable code; remove explorer-specific constants or make them config.
- [ ] Add `go.mod` / `package.json` with correct module/package name.
- [ ] Add tests where feasible (e.g. cache, rate limiter, chain adapter).
- [ ] Add to explorer-monorepo as submodule in chosen path (`backend/libs/...` or `frontend/libs/...` or `deployment/common`).
- [ ] Update explorer to use the new module (replace directives or npm dependency).
- [ ] Document in explorer README that submodules are required (`git clone --recurse-submodules`).

---

## 8. Submodule Best Practices (`.gitmodules` and clone)

- **One submodule = one entry in `.gitmodules`.** After adding submodules, the file will look like:
  ```ini
  [submodule "backend/libs/go-bridge-aggregator"]
    path = backend/libs/go-bridge-aggregator
    url = https://github.com/yourorg/go-bridge-aggregator.git
    branch = main
  ```
- **Use a consistent base URL** (SSH or HTTPS) for all submodules so clones work without reconfiguring.
- **Prefer `branch = main`** (or your default branch) so `git submodule update --remote` pulls the right branch; for releases you can pin by committing the submodule at a specific tag.
- **Clone explorer-monorepo with submodules (first time):**
  ```bash
  cd /home/intlc/projects
  git clone --recurse-submodules <explorer-monorepo-url> proxmox/explorer-monorepo
  ```
- **Existing clone – init and update submodules:**
  ```bash
  cd /home/intlc/projects/proxmox/explorer-monorepo
  git submodule update --init --recursive
  ```
- **Add a new submodule (example for local repo under projects):**
  ```bash
  cd /home/intlc/projects/proxmox/explorer-monorepo
  git submodule add -b main ../go-bridge-aggregator backend/libs/go-bridge-aggregator
  ```
  Or with a remote URL:
  ```bash
  git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator
  ```

---

## 9. Quick Reference: Repo Paths and Submodule Paths

| New repo (under `/home/intlc/projects`) | Suggested submodule path in explorer-monorepo |
|----------------------------------------|------------------------------------------------|
| `go-bridge-aggregator`                 | `backend/libs/go-bridge-aggregator`            |
| `go-chain-adapters`                    | `backend/libs/go-chain-adapters`              |
| `go-pgconfig`                          | `backend/libs/go-pgconfig`                    |
| `go-tiered-auth`                       | `backend/libs/go-tiered-auth`                 |
| `go-http-middleware`                   | `backend/libs/go-http-middleware`              |
| `go-logging`                           | `backend/libs/go-logging`                      |
| `go-rpc-gateway`                       | `backend/libs/go-rpc-gateway`                  |
| `frontend-ui-primitives`               | `frontend/libs/frontend-ui-primitives`         |
| `frontend-api-client`                  | `frontend/libs/frontend-api-client`            |
| `deployment-common`                    | `deployment/common`                            |

This plan keeps Explorer-specific behavior in the monorepo while moving generic building blocks into separate, reusable repos linked via submodules and optional published modules.