virtual-banker/README.md

# Virtual Banker Submodule

AI digital twin human-like Virtual Banker with full video realism using Unreal Engine server-rendered avatars, real-time voice interaction, and an embeddable widget for portal sites.

## Features

- **Embeddable Widget**: Drop-in widget for any portal site
- **Real-time Voice**: ASR (Speech-to-Text) and TTS (Text-to-Speech) with streaming
- **Full Video Realism**: Unreal Engine server-rendered avatar with PixelStreaming
- **Multi-tenant**: Different configs/brands/policies per tenant
- **Secure Auth**: JWT/SSO integration
- **Accessible**: WCAG-oriented (keyboard, screen reader, captions, reduced motion)
- **Observable**: Audit logs, safety rules, analytics

## Architecture

```
virtual-banker/
├── backend/          # Go backend services
├── widget/          # React/TypeScript embeddable widget
├── avatar/          # Unreal Engine avatar service
├── database/        # Database migrations
└── deployment/      # Docker/Kubernetes configs
```

## Quick Start

### Prerequisites

- Docker and Docker Compose
- Go 1.21+ (for backend development)
- Node.js 20+ (for widget development)
- PostgreSQL 16+ with pgvector extension
- Redis

### Configuration

- Copy `.env.example` to `.env` and set `DATABASE_URL`, `REDIS_URL`, and optionally TTS vars (`TTS_BASE_URL`, `TTS_API_KEY`, `TTS_VOICE_ID`) for ElevenLabs or Phoenix. See `backend/tts/README.md` for TTS backend selection and Phoenix endpoint swap.

### Development Setup

1. **Start infrastructure** (uses existing postgres/redis from main monorepo):
```bash
cd deployment
docker-compose up -d postgres redis
```

2. **Run database migrations**:
```bash
cd database
psql -U explorer -d explorer -f migrations/001_sessions.up.sql
psql -U explorer -d explorer -f migrations/002_conversations.up.sql
psql -U explorer -d explorer -f migrations/003_tenants.up.sql
psql -U explorer -d explorer -f migrations/004_vector_extension.up.sql
psql -U explorer -d explorer -f migrations/005_user_profiles.up.sql
```

3. **Start backend API**:
```bash
cd backend
go run main.go
```

4. **Build and serve widget**:
```bash
cd widget
npm install
npm run build
# Serve dist/ via CDN or static server
```

## Widget Integration

### Basic Integration

Add the widget loader script to your HTML:

```html
<script src="https://cdn.example.com/virtual-banker/widget.js"
        data-tenant-id="your-tenant-id"
        data-user-id="user-123"
        data-auth-token="jwt-token"
        data-api-url="https://api.example.com"
        data-avatar-enabled="true"></script>
<div id="virtual-banker-widget"></div>
```

### Programmatic Control

```javascript
// Open widget
window.VirtualBankerWidgetAPI.open();

// Close widget
window.VirtualBankerWidgetAPI.close();

// Set context
window.VirtualBankerWidgetAPI.setContext({
  route: '/account',
  accountId: 'acc-123'
});

// Update auth token
window.VirtualBankerWidgetAPI.setAuthToken('new-jwt-token');
```

## API Endpoints

### Create Session
```
POST /v1/sessions
{
  "tenant_id": "tenant-123",
  "user_id": "user-456",
  "auth_assertion": "jwt-token"
}
```

### Refresh Token
```
POST /v1/sessions/{id}/refresh-token
```

### End Session
```
POST /v1/sessions/{id}/end
```

## Implementation Status

### Phase 0: Foundation & Widget ✅
- [x] Backend session service
- [x] REST API endpoints
- [x] Database migrations
- [x] Embeddable widget (React/TypeScript)
- [x] Basic chat UI
- [x] Theming system
- [x] Accessibility features

### Phase 1: Voice & Realtime ✅
- [x] WebRTC infrastructure
- [x] ASR service integration (mock + interface for Deepgram)
- [x] TTS service integration (mock + interface for ElevenLabs)
- [x] Conversation orchestrator
- [x] Barge-in support

### Phase 2: LLM & RAG ✅
- [x] LLM gateway (mock + interface for OpenAI)
- [x] RAG service with pgvector
- [x] Tool framework
- [x] Banking integrations

### Phase 3: Avatar System ✅
- [x] Unreal Engine setup documentation
- [x] Render service structure
- [x] Animation controller (visemes, expressions, gestures)

### Phase 4: Memory & Observability ✅
- [x] Memory service
- [x] Observability (tracing, metrics)
- [x] Safety/compliance filters

### Phase 5: Enterprise Features (In Progress)
- [x] Multi-tenancy support
- [ ] Tenant admin console (UI)
- [ ] Advanced compliance tools
- [ ] Usage analytics dashboard

## Next Steps

1. **Integrate Real Services**: Replace mocks with actual API integrations:
   - Deepgram or Google STT for ASR
   - ElevenLabs or Azure TTS for TTS
   - OpenAI or Anthropic for LLM
   - Connect to existing banking services

2. **Complete WebRTC**: Implement full WebRTC signaling and media streaming

3. **Unreal Setup**: Set up actual Unreal Engine project with digital human

4. **Testing**: Add unit tests, integration tests, E2E tests

5. **Production Deployment**: Configure for production with proper secrets, monitoring, scaling

## Documentation

- [Architecture](./docs/ARCHITECTURE.md)
- [API Reference](./docs/API.md)
- [Widget Integration](./docs/WIDGET_INTEGRATION.md)
- [Deployment](./docs/DEPLOYMENT.md)

## License

MIT