286 lines
19 KiB
Markdown
286 lines
19 KiB
Markdown
# Reusable Components Extraction Plan
|
||
|
||
**Completion status (in-repo):** All libs are present under `backend/libs/` and `frontend/libs/`. Backend is wired to use **go-pgconfig** (API + indexer), **go-rpc-gateway** (Track 1). Frontend is wired to use **frontend-api-client** (services/api/client) and **frontend-ui-primitives** (all pages using Card, Table, Address). CI uses `submodules: recursive`; README documents clone with submodules. To publish as separate repos, copy each lib to its own repo and add as submodule.
|
||
|
||
**Review and test:** Backend handlers that need the DB use `requireDB(w)`; without a DB they return 503. Tests run with a nil DB and accept 200/503/404 as appropriate. Run backend tests: `go test ./...` in `backend/`. Frontend build: `npm run build` in `frontend/` (ESLint uses root `.eslintrc.cjs` and frontend `"root": true` in `.eslintrc.json`). E2E: `npm run e2e` from repo root (Playwright, default base URL https://explorer.d-bis.org; set `EXPLORER_URL` for local).
|
||
|
||
---
|
||
|
||
**Goal:** Extract reusable, non–Explorer-specific components into their own repositories under `/home/intlc/projects`, and link them back to this monorepo via **git submodules** using best practices.
|
||
|
||
**Scope:** Components that are generic (EVM/chain-agnostic, multi-tenant, or generic UI/infra) are candidates. Explorer-specific logic (Chain 138 routes, Blockscout integration, explorer branding) stays in `explorer-monorepo`.
|
||
|
||
---
|
||
|
||
## 1. Reusable vs Explorer-Specific (Summary)
|
||
|
||
| Category | Reusable (extract) | Explorer-specific (keep in monorepo) |
|
||
|--------|---------------------|--------------------------------------|
|
||
| **Backend Go** | Bridge/swap aggregators, chain adapters, DB config, wallet auth + tiered access, logging/metrics, generic cache/rate-limit, security middleware (generic part) | REST routes, track* endpoints, indexer (block/processor), Blockscout/etherscan compat, explorer API server |
|
||
| **Frontend** | Button, Card, Table, generic API client, optional Address (minimal) | Explorer pages, block/tx/address views, SPA `index.html`, explorer API calls |
|
||
| **Deployment** | Nginx/systemd/Cloudflare/fail2ban templates, generic verify scripts | Explorer deploy scripts, VMID 5000 / Blockscout-specific fixes |
|
||
| **Docs** | — | All current docs are explorer/deployment-specific |
|
||
|
||
---
|
||
|
||
## 2. Proposed New Repositories (under `/home/intlc/projects`)
|
||
|
||
All new repos live as siblings of `proxmox/explorer-monorepo`, e.g.:
|
||
|
||
- `/home/intlc/projects/go-bridge-aggregator`
|
||
- `/home/intlc/projects/go-chain-adapters`
|
||
- `/home/intlc/projects/go-pgconfig`
|
||
- `/home/intlc/projects/go-tiered-auth`
|
||
- `/home/intlc/projects/go-http-middleware`
|
||
- `/home/intlc/projects/go-logging`
|
||
- `/home/intlc/projects/go-rpc-gateway`
|
||
- `/home/intlc/projects/frontend-ui-primitives`
|
||
- `/home/intlc/projects/deployment-common`
|
||
|
||
Relationship to monorepo: **submodules** inside `explorer-monorepo` (e.g. `backend/vendor/go-bridge-aggregator` or `libs/go-bridge-aggregator`), so the monorepo stays the single checkout for development and CI.
|
||
|
||
---
|
||
|
||
## 3. Repository-by-Repository Plan
|
||
|
||
### 3.1 `go-bridge-aggregator`
|
||
|
||
- **Purpose:** Multi-provider bridge quote aggregation (chain-agnostic interface + implementations).
|
||
- **Source in monorepo:** `backend/bridge/` (all providers: `providers.go`, `ccip_provider.go`, `lifi_provider.go`, `socket_provider.go`, `squid_provider.go`, `symbiosis_provider.go`, `relay_provider.go`, `stargate_provider.go`, `hop_provider.go`).
|
||
- **Reusability:** Provider interface, `BridgeRequest`/`BridgeQuote`/`BridgeStep`, `Aggregator`, `GetBestQuote`. Chain IDs and integrator names should be config (env or struct), not hardcoded 138.
|
||
- **New repo path:** `/home/intlc/projects/go-bridge-aggregator`.
|
||
- **Suggested layout:** `go.mod` (module e.g. `github.com/yourorg/go-bridge-aggregator`), `provider.go`, `aggregator.go`, `providers/` (one file per provider). Dependencies: minimal (HTTP client, context).
|
||
- **Explorer coupling:** Explorer will depend on this module via Go module (or submodule + replace). CCIP “138 <-> 1” and integrator strings become config in explorer.
|
||
|
||
---
|
||
|
||
### 3.2 `go-chain-adapters`
|
||
|
||
- **Purpose:** Chain abstraction for EVM-compatible chains (block, tx, receipt, balance, code, gas).
|
||
- **Source:** `backend/chain/adapters/evm.go` (interface `ChainAdapter` + `EVMAdapter`).
|
||
- **Reusability:** No explorer references; only `go-ethereum`.
|
||
- **New repo path:** `/home/intlc/projects/go-chain-adapters`.
|
||
- **Suggested layout:** `go.mod`, `adapter.go` (interface), `evm/evm.go`. Other chains (e.g. non-EVM) can be added later in same repo.
|
||
- **Explorer coupling:** Explorer indexer and API use chain adapter; import this module.
|
||
|
||
---
|
||
|
||
### 3.3 `go-pgconfig`
|
||
|
||
- **Purpose:** Load PostgreSQL configuration from environment and build `pgxpool.Config`.
|
||
- **Source:** `backend/database/config/database.go` (and optionally read-replica).
|
||
- **Reusability:** No explorer references; only `pgx` and `os`.
|
||
- **New repo path:** `/home/intlc/projects/go-pgconfig`.
|
||
- **Suggested layout:** `go.mod`, `config.go` (DatabaseConfig, LoadDatabaseConfig, ConnectionString, PoolConfig, ReadReplicaConfig).
|
||
- **Explorer coupling:** Explorer `api/rest/cmd/main.go` and indexer use DB config; import this module.
|
||
|
||
---
|
||
|
||
### 3.4 `go-tiered-auth`
|
||
|
||
- **Purpose:** Tiered (track) access: wallet-based auth (nonce + JWT), optional “tier” level, and feature access by tier.
|
||
- **Source:** `backend/auth/` (auth.go, roles.go, wallet_auth.go), `backend/featureflags/flags.go`, and the auth/track parts of `backend/api/middleware/auth.go`.
|
||
- **Reusability:** Wallet nonce + JWT and “tier” (track) are generic; feature names in `featureflags` can be default map, overridable by consumer. Middleware uses `auth` + `featureflags`; can live in this repo or in `go-http-middleware` with dependency on this repo.
|
||
- **New repo path:** `/home/intlc/projects/go-tiered-auth`.
|
||
- **Suggested layout:** `go.mod`, `auth/` (wallet_auth, nonce storage interface), `featureflags/` (HasAccess, GetEnabledFeatures, configurable feature map), `middleware/` (RequireAuth, RequireTier, OptionalAuth). DB tables (e.g. `wallet_nonces`, `operator_roles`) stay defined in explorer migrations; this repo only needs interfaces (e.g. NonceStore, TierResolver).
|
||
- **Explorer coupling:** Explorer implements storage (existing DB); middleware and WalletAuth stay in explorer but can call into this lib for validation and JWT. Alternatively, move full WalletAuth + middleware here and pass DB and feature map from explorer.
|
||
|
||
---
|
||
|
||
### 3.5 `go-http-middleware`
|
||
|
||
- **Purpose:** Generic HTTP middleware: security headers (CSP, X-Frame-Options, etc.), CORS, compression, logging. No explorer URLs.
|
||
- **Source:** `backend/api/middleware/security.go` (generic headers only; CSP `connect-src` etc. are explorer-specific and should be supplied by explorer or config).
|
||
- **Reusability:** Security headers and CORS are generic; CSP should be a parameter or callback.
|
||
- **New repo path:** `/home/intlc/projects/go-http-middleware`.
|
||
- **Suggested layout:** `go.mod`, `security.go` (AddSecurityHeaders with configurable CSP), `cors.go` if extracted, `logging.go` if request logging is extracted. Explorer passes CSP string when wiring middleware.
|
||
- **Explorer coupling:** Explorer server uses this middleware with explorer-specific CSP.
|
||
|
||
---
|
||
|
||
### 3.6 `go-logging`
|
||
|
||
- **Purpose:** Structured logging (context, fields, levels).
|
||
- **Source:** `backend/logging/logger.go`.
|
||
- **Reusability:** No explorer references.
|
||
- **New repo path:** `/home/intlc/projects/go-logging`.
|
||
- **Suggested layout:** `go.mod`, `logger.go`. Optional: integrate with a standard logger interface (e.g. slog) for compatibility.
|
||
- **Explorer coupling:** Optional; explorer can keep using it or switch to stdlib/slog and deprecate this in monorepo.
|
||
|
||
---
|
||
|
||
### 3.7 `go-rpc-gateway`
|
||
|
||
- **Purpose:** Generic RPC gateway: in-memory (and optionally Redis) cache, in-memory (and optionally Redis) rate limiter, and a generic HTTP proxy to an upstream RPC URL. No route definitions or explorer semantics.
|
||
- **Source:** `backend/api/track1/cache.go`, `redis_cache.go`, `rate_limiter.go`, `redis_rate_limiter.go`, `rpc_gateway.go`. Exclude `endpoints.go` (explorer-specific handlers).
|
||
- **Reusability:** Cache and rate limiter are generic; RPC gateway is “forward request to RPC URL with cache and rate limit.”
|
||
- **New repo path:** `/home/intlc/projects/go-rpc-gateway`.
|
||
- **Suggested layout:** `go.mod`, `cache/` (memory + Redis), `ratelimit/` (memory + Redis), `gateway.go` (proxy to single upstream). Explorer track1 endpoints call this gateway and format responses (e.g. chain_id 138).
|
||
- **Explorer coupling:** Explorer imports gateway + cache + ratelimit; track1 endpoints remain in explorer and use this package.
|
||
|
||
---
|
||
|
||
### 3.8 `frontend-ui-primitives`
|
||
|
||
- **Purpose:** Reusable React UI components: Button, Card, Table. Optionally a minimal Address (truncate + copy) without explorer-specific links.
|
||
- **Source:** `frontend/src/components/common/Button.tsx`, `Card.tsx`, `Table.tsx`; optionally `frontend/src/components/blockchain/Address.tsx` (strip explorer-specific links/props if any).
|
||
- **Reusability:** No explorer API or routes; only Tailwind + clsx + React. Table is generic; Address can be “display + copy” only.
|
||
- **New repo path:** `/home/intlc/projects/frontend-ui-primitives`.
|
||
- **Suggested layout:** npm package (e.g. `@yourorg/ui-primitives`). Structure: `src/Button.tsx`, `Card.tsx`, `Table.tsx`, `Address.tsx`; export from `index.ts`; build with tsup or similar; peer deps: `react`, `clsx`. Tailwind: consumer app includes Tailwind and uses same class names, or package ships minimal CSS.
|
||
- **Explorer coupling:** Explorer frontend adds dependency (npm package or git submodule + workspace). If submodule: e.g. `frontend/libs/ui-primitives` and in package.json `"@yourorg/ui-primitives": "file:./libs/frontend-ui-primitives"`.
|
||
|
||
---
|
||
|
||
### 3.9 `frontend-api-client` (optional)
|
||
|
||
- **Purpose:** Generic API client: axios instance, interceptors, optional API key from storage, typed `ApiResponse<T>` and error shape.
|
||
- **Source:** `frontend/src/services/api/client.ts` (and possibly a minimal `types.ts` for ApiResponse/ApiError).
|
||
- **Reusability:** No explorer endpoints; base URL and headers are config.
|
||
- **New repo path:** `/home/intlc/projects/frontend-api-client`.
|
||
- **Suggested layout:** npm package exporting `createApiClient(baseURL, options?)` and types. Explorer uses `createApiClient(process.env.NEXT_PUBLIC_API_URL)` and keeps explorer-specific API modules (blocks, addresses, transactions) in monorepo.
|
||
- **Explorer coupling:** Explorer depends on this package; explorer-specific services import client from package and define endpoints locally.
|
||
|
||
---
|
||
|
||
### 3.10 `deployment-common`
|
||
|
||
- **Purpose:** Reusable deployment and ops snippets: nginx (generic reverse proxy, SSL, `location /api/` template), systemd unit templates, Cloudflare Tunnel sample config, fail2ban jail/config, and small generic “verify” scripts (e.g. curl health, check port).
|
||
- **Source:** `deployment/` – extract generic parts of `setup-nginx.sh`, `systemd/*.service`, `cloudflare/tunnel-config.yml`, `fail2ban/`. Omit explorer-specific env (VMID 5000, Blockscout, explorer.d-bis.org) or make them placeholders.
|
||
- **Reusability:** No explorer branding or VMIDs; only patterns (proxy to backend, systemd, tunnel, fail2ban).
|
||
- **New repo path:** `/home/intlc/projects/deployment-common`.
|
||
- **Suggested layout:** `nginx/`, `systemd/`, `cloudflare/`, `fail2ban/`, `scripts/` (e.g. verify-http-endpoint.sh). README with usage and variable placeholders.
|
||
- **Explorer coupling:** Explorer’s `deployment/` can include this repo as submodule (e.g. `deployment/common`) and copy or symlink templates, or reference them from docs.
|
||
|
||
---
|
||
|
||
## 4. Submodule Strategy (Best Practices)
|
||
|
||
- **Location of submodules:** Prefer a single directory for “vendored” or “lib” submodules to avoid clutter, e.g.:
|
||
- **Option A:** `explorer-monorepo/libs/` with one submodule per repo (`libs/go-bridge-aggregator`, `libs/frontend-ui-primitives`, etc.).
|
||
- **Option B:** Backend libs under `backend/libs/` and frontend under `frontend/libs/` (so Go and Node resolve paths naturally).
|
||
- **Recommendation:** Use **Option B** for simpler Go/Node resolution:
|
||
- `backend/libs/go-bridge-aggregator`, `backend/libs/go-chain-adapters`, `backend/libs/go-pgconfig`, `backend/libs/go-tiered-auth`, `backend/libs/go-http-middleware`, `backend/libs/go-logging`, `backend/libs/go-rpc-gateway`.
|
||
- `frontend/libs/frontend-ui-primitives` (and optionally `frontend/libs/frontend-api-client`).
|
||
- `deployment/common` → `deployment-common` (or `deployment-common` at repo root if you prefer).
|
||
|
||
- **Adding a submodule (example):**
|
||
```bash
|
||
cd /home/intlc/projects/proxmox/explorer-monorepo
|
||
git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator
|
||
```
|
||
Use SSH or HTTPS consistently; pin to a branch or tag (e.g. `main` or `v0.1.0`).
|
||
|
||
- **Go modules:** In explorer’s `backend/go.mod`, use `replace` to point at the local submodule during development:
|
||
```go
|
||
replace github.com/yourorg/go-bridge-aggregator => ./libs/go-bridge-aggregator
|
||
```
|
||
CI and contributors run `go mod tidy`; when publishing the library, depend on the published module version and remove `replace` for release.
|
||
|
||
- **Clone/update instructions:** Document in root README:
|
||
```bash
|
||
git clone --recurse-submodules <explorer-monorepo-url>
|
||
# or after clone:
|
||
git submodule update --init --recursive
|
||
```
|
||
|
||
- **Publishing:** When a reusable repo is stable, publish it (Go: tag and push to GitHub; npm: publish to npm or private registry). Explorer can then depend on versioned releases and only use submodules for active development or private forks.
|
||
|
||
---
|
||
|
||
## 5. Phased Extraction Order
|
||
|
||
To minimize breakage and respect dependencies:
|
||
|
||
1. **Phase 1 – No dependency on other extracted pieces**
|
||
- `go-pgconfig` (used by api + indexer)
|
||
- `go-logging`
|
||
- `go-chain-adapters`
|
||
- `go-bridge-aggregator` (make chain IDs/config injectable)
|
||
- `frontend-ui-primitives`
|
||
- `deployment-common` (templates only)
|
||
|
||
2. **Phase 2 – May depend on Phase 1**
|
||
- `go-rpc-gateway` (cache + rate limit + gateway; no dependency on auth)
|
||
- `go-http-middleware` (security headers only)
|
||
- `go-tiered-auth` (auth + featureflags + optional middleware; depends on DB interface, so explorer keeps DB)
|
||
- `frontend-api-client` (optional)
|
||
|
||
3. **Phase 3 – Integration in explorer**
|
||
- Replace internal packages with submodule or published module references.
|
||
- Update explorer `backend/go.mod` and `frontend/package.json`.
|
||
- Keep explorer-specific code (routes, indexer, Blockscout, SPA) in monorepo; ensure tests and CI pass.
|
||
|
||
---
|
||
|
||
## 6. What Stays in Explorer Monorepo
|
||
|
||
- **Backend:** All REST route handlers, track* endpoint logic, indexer (listener + processor + backfill), Blockscout/etherscan compatibility, explorer-specific config (chain_id 138, RPC URLs), and migrations (schema stays here; libs use interfaces or config).
|
||
- **Frontend:** All pages and views, `public/index.html` SPA, explorer API service modules (blocks, transactions, addresses), Next.js app and deployment config.
|
||
- **Deployment:** All explorer- and VMID 5000–specific scripts (`fix-502-blockscout.sh`, `complete-explorer-api-access.sh`, `deploy-frontend-to-vmid5000.sh`, etc.), and nginx/config that reference explorer.d-bis.org and Blockscout.
|
||
- **Docs:** All current documentation (API access, deployment, runbook, etc.).
|
||
|
||
---
|
||
|
||
## 7. Checklist for Each New Repo
|
||
|
||
- [ ] Create repo under `/home/intlc/projects/<repo-name>` (or under GitHub/GitLab first, then clone there).
|
||
- [ ] Add minimal `README.md`, `LICENSE`, and `.gitignore`.
|
||
- [ ] Copy only the reusable code; remove explorer-specific constants or make them config.
|
||
- [ ] Add `go.mod` / `package.json` with correct module/package name.
|
||
- [ ] Add tests where feasible (e.g. cache, rate limiter, chain adapter).
|
||
- [ ] Add to explorer-monorepo as submodule in chosen path (`backend/libs/...` or `frontend/libs/...` or `deployment/common`).
|
||
- [ ] Update explorer to use the new module (replace directives or npm dependency).
|
||
- [ ] Document in explorer README that submodules are required (`git clone --recurse-submodules`).
|
||
|
||
---
|
||
|
||
## 8. Submodule Best Practices (`.gitmodules` and clone)
|
||
|
||
- **One submodule = one entry in `.gitmodules`.** After adding submodules, the file will look like:
|
||
```ini
|
||
[submodule "backend/libs/go-bridge-aggregator"]
|
||
path = backend/libs/go-bridge-aggregator
|
||
url = https://github.com/yourorg/go-bridge-aggregator.git
|
||
branch = main
|
||
```
|
||
- **Use a consistent base URL** (SSH or HTTPS) for all submodules so clones work without reconfiguring.
|
||
- **Prefer `branch = main`** (or your default branch) so `git submodule update --remote` pulls the right branch; for releases you can pin by committing the submodule at a specific tag.
|
||
- **Clone explorer-monorepo with submodules (first time):**
|
||
```bash
|
||
cd /home/intlc/projects
|
||
git clone --recurse-submodules <explorer-monorepo-url> proxmox/explorer-monorepo
|
||
```
|
||
- **Existing clone – init and update submodules:**
|
||
```bash
|
||
cd /home/intlc/projects/proxmox/explorer-monorepo
|
||
git submodule update --init --recursive
|
||
```
|
||
- **Add a new submodule (example for local repo under projects):**
|
||
```bash
|
||
cd /home/intlc/projects/proxmox/explorer-monorepo
|
||
git submodule add -b main ../go-bridge-aggregator backend/libs/go-bridge-aggregator
|
||
```
|
||
Or with a remote URL:
|
||
```bash
|
||
git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Quick Reference: Repo Paths and Submodule Paths
|
||
|
||
| New repo (under `/home/intlc/projects`) | Suggested submodule path in explorer-monorepo |
|
||
|----------------------------------------|------------------------------------------------|
|
||
| `go-bridge-aggregator` | `backend/libs/go-bridge-aggregator` |
|
||
| `go-chain-adapters` | `backend/libs/go-chain-adapters` |
|
||
| `go-pgconfig` | `backend/libs/go-pgconfig` |
|
||
| `go-tiered-auth` | `backend/libs/go-tiered-auth` |
|
||
| `go-http-middleware` | `backend/libs/go-http-middleware` |
|
||
| `go-logging` | `backend/libs/go-logging` |
|
||
| `go-rpc-gateway` | `backend/libs/go-rpc-gateway` |
|
||
| `frontend-ui-primitives` | `frontend/libs/frontend-ui-primitives` |
|
||
| `frontend-api-client` | `frontend/libs/frontend-api-client` |
|
||
| `deployment-common` | `deployment/common` |
|
||
|
||
This plan keeps Explorer-specific behavior in the monorepo while moving generic building blocks into separate, reusable repos linked via submodules and optional published modules.
|