# Reusable Components Extraction Plan **Completion status (in-repo):** All libs are present under `backend/libs/` and `frontend/libs/`. Backend is wired to use **go-pgconfig** (API + indexer), **go-rpc-gateway** (Track 1). Frontend is wired to use **frontend-api-client** (services/api/client) and **frontend-ui-primitives** (all pages using Card, Table, Address). CI uses `submodules: recursive`; README documents clone with submodules. To publish as separate repos, copy each lib to its own repo and add as submodule. **Review and test:** Backend handlers that need the DB use `requireDB(w)`; without a DB they return 503. Tests run with a nil DB and accept 200/503/404 as appropriate. Run backend tests: `go test ./...` in `backend/`. Frontend build: `npm run build` in `frontend/` (ESLint uses root `.eslintrc.cjs` and frontend `"root": true` in `.eslintrc.json`). E2E: `npm run e2e` from repo root (Playwright, default base URL https://explorer.d-bis.org; set `EXPLORER_URL` for local). --- **Goal:** Extract reusable, non–Explorer-specific components into their own repositories under `/home/intlc/projects`, and link them back to this monorepo via **git submodules** using best practices. **Scope:** Components that are generic (EVM/chain-agnostic, multi-tenant, or generic UI/infra) are candidates. Explorer-specific logic (Chain 138 routes, Blockscout integration, explorer branding) stays in `explorer-monorepo`. --- ## 1. Reusable vs Explorer-Specific (Summary) | Category | Reusable (extract) | Explorer-specific (keep in monorepo) | |--------|---------------------|--------------------------------------| | **Backend Go** | Bridge/swap aggregators, chain adapters, DB config, wallet auth + tiered access, logging/metrics, generic cache/rate-limit, security middleware (generic part) | REST routes, track* endpoints, indexer (block/processor), Blockscout/etherscan compat, explorer API server | | **Frontend** | Button, Card, Table, generic API client, optional Address (minimal) | Explorer pages, block/tx/address views, SPA `index.html`, explorer API calls | | **Deployment** | Nginx/systemd/Cloudflare/fail2ban templates, generic verify scripts | Explorer deploy scripts, VMID 5000 / Blockscout-specific fixes | | **Docs** | — | All current docs are explorer/deployment-specific | --- ## 2. Proposed New Repositories (under `/home/intlc/projects`) All new repos live as siblings of `proxmox/explorer-monorepo`, e.g.: - `/home/intlc/projects/go-bridge-aggregator` - `/home/intlc/projects/go-chain-adapters` - `/home/intlc/projects/go-pgconfig` - `/home/intlc/projects/go-tiered-auth` - `/home/intlc/projects/go-http-middleware` - `/home/intlc/projects/go-logging` - `/home/intlc/projects/go-rpc-gateway` - `/home/intlc/projects/frontend-ui-primitives` - `/home/intlc/projects/deployment-common` Relationship to monorepo: **submodules** inside `explorer-monorepo` (e.g. `backend/vendor/go-bridge-aggregator` or `libs/go-bridge-aggregator`), so the monorepo stays the single checkout for development and CI. --- ## 3. Repository-by-Repository Plan ### 3.1 `go-bridge-aggregator` - **Purpose:** Multi-provider bridge quote aggregation (chain-agnostic interface + implementations). - **Source in monorepo:** `backend/bridge/` (all providers: `providers.go`, `ccip_provider.go`, `lifi_provider.go`, `socket_provider.go`, `squid_provider.go`, `symbiosis_provider.go`, `relay_provider.go`, `stargate_provider.go`, `hop_provider.go`). - **Reusability:** Provider interface, `BridgeRequest`/`BridgeQuote`/`BridgeStep`, `Aggregator`, `GetBestQuote`. Chain IDs and integrator names should be config (env or struct), not hardcoded 138. - **New repo path:** `/home/intlc/projects/go-bridge-aggregator`. - **Suggested layout:** `go.mod` (module e.g. `github.com/yourorg/go-bridge-aggregator`), `provider.go`, `aggregator.go`, `providers/` (one file per provider). Dependencies: minimal (HTTP client, context). - **Explorer coupling:** Explorer will depend on this module via Go module (or submodule + replace). CCIP “138 <-> 1” and integrator strings become config in explorer. --- ### 3.2 `go-chain-adapters` - **Purpose:** Chain abstraction for EVM-compatible chains (block, tx, receipt, balance, code, gas). - **Source:** `backend/chain/adapters/evm.go` (interface `ChainAdapter` + `EVMAdapter`). - **Reusability:** No explorer references; only `go-ethereum`. - **New repo path:** `/home/intlc/projects/go-chain-adapters`. - **Suggested layout:** `go.mod`, `adapter.go` (interface), `evm/evm.go`. Other chains (e.g. non-EVM) can be added later in same repo. - **Explorer coupling:** Explorer indexer and API use chain adapter; import this module. --- ### 3.3 `go-pgconfig` - **Purpose:** Load PostgreSQL configuration from environment and build `pgxpool.Config`. - **Source:** `backend/database/config/database.go` (and optionally read-replica). - **Reusability:** No explorer references; only `pgx` and `os`. - **New repo path:** `/home/intlc/projects/go-pgconfig`. - **Suggested layout:** `go.mod`, `config.go` (DatabaseConfig, LoadDatabaseConfig, ConnectionString, PoolConfig, ReadReplicaConfig). - **Explorer coupling:** Explorer `api/rest/cmd/main.go` and indexer use DB config; import this module. --- ### 3.4 `go-tiered-auth` - **Purpose:** Tiered (track) access: wallet-based auth (nonce + JWT), optional “tier” level, and feature access by tier. - **Source:** `backend/auth/` (auth.go, roles.go, wallet_auth.go), `backend/featureflags/flags.go`, and the auth/track parts of `backend/api/middleware/auth.go`. - **Reusability:** Wallet nonce + JWT and “tier” (track) are generic; feature names in `featureflags` can be default map, overridable by consumer. Middleware uses `auth` + `featureflags`; can live in this repo or in `go-http-middleware` with dependency on this repo. - **New repo path:** `/home/intlc/projects/go-tiered-auth`. - **Suggested layout:** `go.mod`, `auth/` (wallet_auth, nonce storage interface), `featureflags/` (HasAccess, GetEnabledFeatures, configurable feature map), `middleware/` (RequireAuth, RequireTier, OptionalAuth). DB tables (e.g. `wallet_nonces`, `operator_roles`) stay defined in explorer migrations; this repo only needs interfaces (e.g. NonceStore, TierResolver). - **Explorer coupling:** Explorer implements storage (existing DB); middleware and WalletAuth stay in explorer but can call into this lib for validation and JWT. Alternatively, move full WalletAuth + middleware here and pass DB and feature map from explorer. --- ### 3.5 `go-http-middleware` - **Purpose:** Generic HTTP middleware: security headers (CSP, X-Frame-Options, etc.), CORS, compression, logging. No explorer URLs. - **Source:** `backend/api/middleware/security.go` (generic headers only; CSP `connect-src` etc. are explorer-specific and should be supplied by explorer or config). - **Reusability:** Security headers and CORS are generic; CSP should be a parameter or callback. - **New repo path:** `/home/intlc/projects/go-http-middleware`. - **Suggested layout:** `go.mod`, `security.go` (AddSecurityHeaders with configurable CSP), `cors.go` if extracted, `logging.go` if request logging is extracted. Explorer passes CSP string when wiring middleware. - **Explorer coupling:** Explorer server uses this middleware with explorer-specific CSP. --- ### 3.6 `go-logging` - **Purpose:** Structured logging (context, fields, levels). - **Source:** `backend/logging/logger.go`. - **Reusability:** No explorer references. - **New repo path:** `/home/intlc/projects/go-logging`. - **Suggested layout:** `go.mod`, `logger.go`. Optional: integrate with a standard logger interface (e.g. slog) for compatibility. - **Explorer coupling:** Optional; explorer can keep using it or switch to stdlib/slog and deprecate this in monorepo. --- ### 3.7 `go-rpc-gateway` - **Purpose:** Generic RPC gateway: in-memory (and optionally Redis) cache, in-memory (and optionally Redis) rate limiter, and a generic HTTP proxy to an upstream RPC URL. No route definitions or explorer semantics. - **Source:** `backend/api/track1/cache.go`, `redis_cache.go`, `rate_limiter.go`, `redis_rate_limiter.go`, `rpc_gateway.go`. Exclude `endpoints.go` (explorer-specific handlers). - **Reusability:** Cache and rate limiter are generic; RPC gateway is “forward request to RPC URL with cache and rate limit.” - **New repo path:** `/home/intlc/projects/go-rpc-gateway`. - **Suggested layout:** `go.mod`, `cache/` (memory + Redis), `ratelimit/` (memory + Redis), `gateway.go` (proxy to single upstream). Explorer track1 endpoints call this gateway and format responses (e.g. chain_id 138). - **Explorer coupling:** Explorer imports gateway + cache + ratelimit; track1 endpoints remain in explorer and use this package. --- ### 3.8 `frontend-ui-primitives` - **Purpose:** Reusable React UI components: Button, Card, Table. Optionally a minimal Address (truncate + copy) without explorer-specific links. - **Source:** `frontend/src/components/common/Button.tsx`, `Card.tsx`, `Table.tsx`; optionally `frontend/src/components/blockchain/Address.tsx` (strip explorer-specific links/props if any). - **Reusability:** No explorer API or routes; only Tailwind + clsx + React. Table is generic; Address can be “display + copy” only. - **New repo path:** `/home/intlc/projects/frontend-ui-primitives`. - **Suggested layout:** npm package (e.g. `@yourorg/ui-primitives`). Structure: `src/Button.tsx`, `Card.tsx`, `Table.tsx`, `Address.tsx`; export from `index.ts`; build with tsup or similar; peer deps: `react`, `clsx`. Tailwind: consumer app includes Tailwind and uses same class names, or package ships minimal CSS. - **Explorer coupling:** Explorer frontend adds dependency (npm package or git submodule + workspace). If submodule: e.g. `frontend/libs/ui-primitives` and in package.json `"@yourorg/ui-primitives": "file:./libs/frontend-ui-primitives"`. --- ### 3.9 `frontend-api-client` (optional) - **Purpose:** Generic API client: axios instance, interceptors, optional API key from storage, typed `ApiResponse` and error shape. - **Source:** `frontend/src/services/api/client.ts` (and possibly a minimal `types.ts` for ApiResponse/ApiError). - **Reusability:** No explorer endpoints; base URL and headers are config. - **New repo path:** `/home/intlc/projects/frontend-api-client`. - **Suggested layout:** npm package exporting `createApiClient(baseURL, options?)` and types. Explorer uses `createApiClient(process.env.NEXT_PUBLIC_API_URL)` and keeps explorer-specific API modules (blocks, addresses, transactions) in monorepo. - **Explorer coupling:** Explorer depends on this package; explorer-specific services import client from package and define endpoints locally. --- ### 3.10 `deployment-common` - **Purpose:** Reusable deployment and ops snippets: nginx (generic reverse proxy, SSL, `location /api/` template), systemd unit templates, Cloudflare Tunnel sample config, fail2ban jail/config, and small generic “verify” scripts (e.g. curl health, check port). - **Source:** `deployment/` – extract generic parts of `setup-nginx.sh`, `systemd/*.service`, `cloudflare/tunnel-config.yml`, `fail2ban/`. Omit explorer-specific env (VMID 5000, Blockscout, explorer.d-bis.org) or make them placeholders. - **Reusability:** No explorer branding or VMIDs; only patterns (proxy to backend, systemd, tunnel, fail2ban). - **New repo path:** `/home/intlc/projects/deployment-common`. - **Suggested layout:** `nginx/`, `systemd/`, `cloudflare/`, `fail2ban/`, `scripts/` (e.g. verify-http-endpoint.sh). README with usage and variable placeholders. - **Explorer coupling:** Explorer’s `deployment/` can include this repo as submodule (e.g. `deployment/common`) and copy or symlink templates, or reference them from docs. --- ## 4. Submodule Strategy (Best Practices) - **Location of submodules:** Prefer a single directory for “vendored” or “lib” submodules to avoid clutter, e.g.: - **Option A:** `explorer-monorepo/libs/` with one submodule per repo (`libs/go-bridge-aggregator`, `libs/frontend-ui-primitives`, etc.). - **Option B:** Backend libs under `backend/libs/` and frontend under `frontend/libs/` (so Go and Node resolve paths naturally). - **Recommendation:** Use **Option B** for simpler Go/Node resolution: - `backend/libs/go-bridge-aggregator`, `backend/libs/go-chain-adapters`, `backend/libs/go-pgconfig`, `backend/libs/go-tiered-auth`, `backend/libs/go-http-middleware`, `backend/libs/go-logging`, `backend/libs/go-rpc-gateway`. - `frontend/libs/frontend-ui-primitives` (and optionally `frontend/libs/frontend-api-client`). - `deployment/common` → `deployment-common` (or `deployment-common` at repo root if you prefer). - **Adding a submodule (example):** ```bash cd /home/intlc/projects/proxmox/explorer-monorepo git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator ``` Use SSH or HTTPS consistently; pin to a branch or tag (e.g. `main` or `v0.1.0`). - **Go modules:** In explorer’s `backend/go.mod`, use `replace` to point at the local submodule during development: ```go replace github.com/yourorg/go-bridge-aggregator => ./libs/go-bridge-aggregator ``` CI and contributors run `go mod tidy`; when publishing the library, depend on the published module version and remove `replace` for release. - **Clone/update instructions:** Document in root README: ```bash git clone --recurse-submodules # or after clone: git submodule update --init --recursive ``` - **Publishing:** When a reusable repo is stable, publish it (Go: tag and push to GitHub; npm: publish to npm or private registry). Explorer can then depend on versioned releases and only use submodules for active development or private forks. --- ## 5. Phased Extraction Order To minimize breakage and respect dependencies: 1. **Phase 1 – No dependency on other extracted pieces** - `go-pgconfig` (used by api + indexer) - `go-logging` - `go-chain-adapters` - `go-bridge-aggregator` (make chain IDs/config injectable) - `frontend-ui-primitives` - `deployment-common` (templates only) 2. **Phase 2 – May depend on Phase 1** - `go-rpc-gateway` (cache + rate limit + gateway; no dependency on auth) - `go-http-middleware` (security headers only) - `go-tiered-auth` (auth + featureflags + optional middleware; depends on DB interface, so explorer keeps DB) - `frontend-api-client` (optional) 3. **Phase 3 – Integration in explorer** - Replace internal packages with submodule or published module references. - Update explorer `backend/go.mod` and `frontend/package.json`. - Keep explorer-specific code (routes, indexer, Blockscout, SPA) in monorepo; ensure tests and CI pass. --- ## 6. What Stays in Explorer Monorepo - **Backend:** All REST route handlers, track* endpoint logic, indexer (listener + processor + backfill), Blockscout/etherscan compatibility, explorer-specific config (chain_id 138, RPC URLs), and migrations (schema stays here; libs use interfaces or config). - **Frontend:** All pages and views, `public/index.html` SPA, explorer API service modules (blocks, transactions, addresses), Next.js app and deployment config. - **Deployment:** All explorer- and VMID 5000–specific scripts (`fix-502-blockscout.sh`, `complete-explorer-api-access.sh`, `deploy-frontend-to-vmid5000.sh`, etc.), and nginx/config that reference explorer.d-bis.org and Blockscout. - **Docs:** All current documentation (API access, deployment, runbook, etc.). --- ## 7. Checklist for Each New Repo - [ ] Create repo under `/home/intlc/projects/` (or under GitHub/GitLab first, then clone there). - [ ] Add minimal `README.md`, `LICENSE`, and `.gitignore`. - [ ] Copy only the reusable code; remove explorer-specific constants or make them config. - [ ] Add `go.mod` / `package.json` with correct module/package name. - [ ] Add tests where feasible (e.g. cache, rate limiter, chain adapter). - [ ] Add to explorer-monorepo as submodule in chosen path (`backend/libs/...` or `frontend/libs/...` or `deployment/common`). - [ ] Update explorer to use the new module (replace directives or npm dependency). - [ ] Document in explorer README that submodules are required (`git clone --recurse-submodules`). --- ## 8. Submodule Best Practices (`.gitmodules` and clone) - **One submodule = one entry in `.gitmodules`.** After adding submodules, the file will look like: ```ini [submodule "backend/libs/go-bridge-aggregator"] path = backend/libs/go-bridge-aggregator url = https://github.com/yourorg/go-bridge-aggregator.git branch = main ``` - **Use a consistent base URL** (SSH or HTTPS) for all submodules so clones work without reconfiguring. - **Prefer `branch = main`** (or your default branch) so `git submodule update --remote` pulls the right branch; for releases you can pin by committing the submodule at a specific tag. - **Clone explorer-monorepo with submodules (first time):** ```bash cd /home/intlc/projects git clone --recurse-submodules proxmox/explorer-monorepo ``` - **Existing clone – init and update submodules:** ```bash cd /home/intlc/projects/proxmox/explorer-monorepo git submodule update --init --recursive ``` - **Add a new submodule (example for local repo under projects):** ```bash cd /home/intlc/projects/proxmox/explorer-monorepo git submodule add -b main ../go-bridge-aggregator backend/libs/go-bridge-aggregator ``` Or with a remote URL: ```bash git submodule add -b main https://github.com/yourorg/go-bridge-aggregator.git backend/libs/go-bridge-aggregator ``` --- ## 9. Quick Reference: Repo Paths and Submodule Paths | New repo (under `/home/intlc/projects`) | Suggested submodule path in explorer-monorepo | |----------------------------------------|------------------------------------------------| | `go-bridge-aggregator` | `backend/libs/go-bridge-aggregator` | | `go-chain-adapters` | `backend/libs/go-chain-adapters` | | `go-pgconfig` | `backend/libs/go-pgconfig` | | `go-tiered-auth` | `backend/libs/go-tiered-auth` | | `go-http-middleware` | `backend/libs/go-http-middleware` | | `go-logging` | `backend/libs/go-logging` | | `go-rpc-gateway` | `backend/libs/go-rpc-gateway` | | `frontend-ui-primitives` | `frontend/libs/frontend-ui-primitives` | | `frontend-api-client` | `frontend/libs/frontend-api-client` | | `deployment-common` | `deployment/common` | This plan keeps Explorer-specific behavior in the monorepo while moving generic building blocks into separate, reusable repos linked via submodules and optional published modules.