virtual-banker/README.md

Virtual Banker Submodule

AI digital twin human-like Virtual Banker with full video realism using Unreal Engine server-rendered avatars, real-time voice interaction, and an embeddable widget for portal sites.

Features

• Embeddable Widget: Drop-in widget for any portal site
• Real-time Voice: ASR (Speech-to-Text) and TTS (Text-to-Speech) with streaming
• Full Video Realism: Unreal Engine server-rendered avatar with PixelStreaming
• Multi-tenant: Different configs/brands/policies per tenant
• Secure Auth: JWT/SSO integration
• Accessible: WCAG-oriented (keyboard, screen reader, captions, reduced motion)
• Observable: Audit logs, safety rules, analytics

Architecture

virtual-banker/
├── backend/          # Go backend services
├── widget/          # React/TypeScript embeddable widget
├── avatar/          # Unreal Engine avatar service
├── database/        # Database migrations
└── deployment/      # Docker/Kubernetes configs

Quick Start

Prerequisites

• Docker and Docker Compose
• Go 1.21+ (for backend development)
• Node.js 20+ (for widget development)
• PostgreSQL 16+ with pgvector extension
• Redis

Configuration

• Copy .env.example to .env and set DATABASE_URL, REDIS_URL, and optionally TTS vars (TTS_BASE_URL, TTS_API_KEY, TTS_VOICE_ID) for ElevenLabs or Phoenix. See backend/tts/README.md for TTS backend selection and Phoenix endpoint swap.

Development Setup

1. Start infrastructure (uses existing postgres/redis from main monorepo):

cd deployment
docker-compose up -d postgres redis

2. Run database migrations:

cd database
psql -U explorer -d explorer -f migrations/001_sessions.up.sql
psql -U explorer -d explorer -f migrations/002_conversations.up.sql
psql -U explorer -d explorer -f migrations/003_tenants.up.sql
psql -U explorer -d explorer -f migrations/004_vector_extension.up.sql
psql -U explorer -d explorer -f migrations/005_user_profiles.up.sql

3. Start backend API:

cd backend
go run main.go

4. Build and serve widget:

cd widget
npm install
npm run build
# Serve dist/ via CDN or static server

Widget Integration

Basic Integration

Add the widget loader script to your HTML:

<script src="https://cdn.example.com/virtual-banker/widget.js"
        data-tenant-id="your-tenant-id"
        data-user-id="user-123"
        data-auth-token="jwt-token"
        data-api-url="https://api.example.com"
        data-avatar-enabled="true"></script>
<div id="virtual-banker-widget"></div>

Programmatic Control

// Open widget
window.VirtualBankerWidgetAPI.open();

// Close widget
window.VirtualBankerWidgetAPI.close();

// Set context
window.VirtualBankerWidgetAPI.setContext({
  route: '/account',
  accountId: 'acc-123'
});

// Update auth token
window.VirtualBankerWidgetAPI.setAuthToken('new-jwt-token');

API Endpoints

Create Session

POST /v1/sessions
{
  "tenant_id": "tenant-123",
  "user_id": "user-456",
  "auth_assertion": "jwt-token"
}

Refresh Token

POST /v1/sessions/{id}/refresh-token

End Session

POST /v1/sessions/{id}/end

Implementation Status

Phase 0: Foundation & Widget ✅

• Backend session service
• REST API endpoints
• Database migrations
• Embeddable widget (React/TypeScript)
• Basic chat UI
• Theming system
• Accessibility features

Phase 1: Voice & Realtime ✅

• WebRTC infrastructure
• ASR service integration (mock + interface for Deepgram)
• TTS service integration (mock + interface for ElevenLabs)
• Conversation orchestrator
• Barge-in support

Phase 2: LLM & RAG ✅

• LLM gateway (mock + interface for OpenAI)
• RAG service with pgvector
• Tool framework
• Banking integrations

Phase 3: Avatar System ✅

• Unreal Engine setup documentation
• Render service structure
• Animation controller (visemes, expressions, gestures)

Phase 4: Memory & Observability ✅

• Memory service
• Observability (tracing, metrics)
• Safety/compliance filters

Phase 5: Enterprise Features (In Progress)

• Multi-tenancy support
• Tenant admin console (UI)
• Advanced compliance tools
• Usage analytics dashboard

Next Steps

1. Integrate Real Services: Replace mocks with actual API integrations:

   • Deepgram or Google STT for ASR
   • ElevenLabs or Azure TTS for TTS
   • OpenAI or Anthropic for LLM
   • Connect to existing banking services

2. Complete WebRTC: Implement full WebRTC signaling and media streaming

3. Unreal Setup: Set up actual Unreal Engine project with digital human

4. Testing: Add unit tests, integration tests, E2E tests

5. Production Deployment: Configure for production with proper secrets, monitoring, scaling

Documentation

• Architecture
• API Reference
• Widget Integration
• Deployment

License

MIT