Add full monorepo: virtual-banker, backend, frontend, docs, scripts, deployment
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
defiQUG
140
docs/API.md
Normal file
140
docs/API.md
Normal file
@@ -0,0 +1,140 @@
|
||||
# Virtual Banker API Reference
|
||||
|
||||
## Base URL
|
||||
|
||||
```
|
||||
http://localhost:8081
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
All requests (except health check) require authentication via JWT token in the `Authorization` header:
|
||||
|
||||
```
|
||||
Authorization: Bearer <jwt-token>
|
||||
```
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Health Check
|
||||
|
||||
```
|
||||
GET /health
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy"
|
||||
}
|
||||
```
|
||||
|
||||
### Create Session
|
||||
|
||||
```
|
||||
POST /v1/sessions
|
||||
```
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"tenant_id": "tenant-123",
|
||||
"user_id": "user-456",
|
||||
"auth_assertion": "jwt-token",
|
||||
"portal_context": {
|
||||
"route": "/account",
|
||||
"account_id": "acc-789"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"session_id": "sess-abc123",
|
||||
"ephemeral_token": "ephemeral-token-xyz",
|
||||
"config": {
|
||||
"theme": {
|
||||
"primaryColor": "#0066cc"
|
||||
},
|
||||
"avatar_enabled": true,
|
||||
"greeting": "Hello! How can I help you today?",
|
||||
"allowed_tools": ["get_account_status", "create_ticket"],
|
||||
"policy": {
|
||||
"max_session_duration_minutes": 30,
|
||||
"rate_limit_per_minute": 10,
|
||||
"require_consent": true
|
||||
}
|
||||
},
|
||||
"expires_at": "2024-01-20T15:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Refresh Token
|
||||
|
||||
```
|
||||
POST /v1/sessions/{session_id}/refresh-token
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"ephemeral_token": "new-ephemeral-token",
|
||||
"expires_at": "2024-01-20T15:35:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### End Session
|
||||
|
||||
```
|
||||
POST /v1/sessions/{session_id}/end
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ended"
|
||||
}
|
||||
```
|
||||
|
||||
## Error Responses
|
||||
|
||||
All errors follow this format:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": "Error message",
|
||||
"message": "Detailed error description"
|
||||
}
|
||||
```
|
||||
|
||||
### Status Codes
|
||||
|
||||
- `200 OK` - Success
|
||||
- `201 Created` - Resource created
|
||||
- `400 Bad Request` - Invalid request
|
||||
- `401 Unauthorized` - Authentication required
|
||||
- `404 Not Found` - Resource not found
|
||||
- `500 Internal Server Error` - Server error
|
||||
|
||||
## WebRTC Signaling
|
||||
|
||||
WebRTC signaling is handled via WebSocket (to be implemented in Phase 1):
|
||||
|
||||
```
|
||||
WS /v1/realtime/{session_id}
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
Rate limits are enforced per tenant and user:
|
||||
- Default: 10 requests per minute per user
|
||||
- Configurable per tenant
|
||||
|
||||
Rate limit headers:
|
||||
```
|
||||
X-RateLimit-Limit: 10
|
||||
X-RateLimit-Remaining: 9
|
||||
X-RateLimit-Reset: 1642680000
|
||||
```
|
||||
|
||||
166
docs/ARCHITECTURE.md
Normal file
166
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# Virtual Banker Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
The Virtual Banker is a multi-layered system that provides a digital human banking experience with full video realism, real-time voice interaction, and embeddable widget capabilities.
|
||||
|
||||
## System Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Client Layer │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ Embeddable Widget (React/TypeScript) │ │
|
||||
│ │ - Chat UI │ │
|
||||
│ │ - Voice Controls │ │
|
||||
│ │ - Avatar View │ │
|
||||
│ │ - WebRTC Client │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Edge Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ CDN │ │ API Gateway │ │ WebRTC │ │
|
||||
│ │ (Widget) │ │ (Auth/Rate) │ │ Gateway │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Core Services │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Session │ │ Orchestrator │ │ LLM Gateway │ │
|
||||
│ │ Service │ │ │ │ │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ RAG Service │ │ Tool/Action │ │ Safety/ │ │
|
||||
│ │ │ │ Service │ │ Compliance │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Media Services │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ ASR Service │ │ TTS Service │ │ Avatar │ │
|
||||
│ │ (Streaming) │ │ (Streaming) │ │ Renderer │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Data Layer │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ Vector DB │ │
|
||||
│ │ (State) │ │ (Cache) │ │ (pgvector) │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Voice Turn Flow
|
||||
|
||||
1. **User speaks** → Widget captures audio via microphone
|
||||
2. **Audio stream** → WebRTC gateway → ASR service
|
||||
3. **ASR** → Transcribes to text (partial + final)
|
||||
4. **Orchestrator** → Sends transcript to LLM with context
|
||||
5. **LLM** → Generates response + tool calls + emotion tags
|
||||
6. **TTS** → Converts text to audio stream
|
||||
7. **Avatar** → Generates visemes, expressions, gestures
|
||||
8. **Widget** → Plays audio, displays captions, animates avatar
|
||||
|
||||
### Text Turn Flow
|
||||
|
||||
1. **User types** → Widget sends text message
|
||||
2. **Orchestrator** → Processes message (same as step 4+ above)
|
||||
|
||||
## Components
|
||||
|
||||
### Backend Services
|
||||
|
||||
#### Session Service
|
||||
- Creates and manages sessions
|
||||
- Issues ephemeral tokens
|
||||
- Loads tenant configurations
|
||||
- Tracks session state
|
||||
|
||||
#### Conversation Orchestrator
|
||||
- Maintains conversation state machine
|
||||
- Routes messages to appropriate services
|
||||
- Handles barge-in (interruptions)
|
||||
- Synchronizes audio/video
|
||||
|
||||
#### LLM Gateway
|
||||
- Multi-tenant prompt templates
|
||||
- Function/tool calling
|
||||
- Output schema enforcement
|
||||
- Model routing
|
||||
|
||||
#### RAG Service
|
||||
- Document ingestion and embedding
|
||||
- Vector similarity search
|
||||
- Reranking
|
||||
- Citation formatting
|
||||
|
||||
#### Tool/Action Service
|
||||
- Tool registry and execution
|
||||
- Banking service integrations
|
||||
- Human-in-the-loop confirmations
|
||||
- Audit logging
|
||||
|
||||
### Frontend Widget
|
||||
|
||||
#### Components
|
||||
- **ChatPanel**: Main chat interface
|
||||
- **VoiceControls**: Push-to-talk, hands-free, volume
|
||||
- **AvatarView**: Video stream display
|
||||
- **Captions**: Real-time captions overlay
|
||||
- **Settings**: User preferences
|
||||
|
||||
#### Hooks
|
||||
- **useSession**: Session management
|
||||
- **useConversation**: Message handling
|
||||
- **useWebRTC**: WebRTC connection
|
||||
|
||||
### Avatar System
|
||||
|
||||
#### Unreal Engine
|
||||
- Digital human character
|
||||
- Blendshapes for visemes/expressions
|
||||
- Animation blueprints
|
||||
- PixelStreaming for video output
|
||||
|
||||
#### Render Service
|
||||
- Controls Unreal instances
|
||||
- Manages GPU resources
|
||||
- Streams video via WebRTC
|
||||
|
||||
## Security
|
||||
|
||||
- JWT/SSO authentication
|
||||
- Ephemeral session tokens
|
||||
- PII redaction
|
||||
- Content filtering
|
||||
- Rate limiting
|
||||
- Audit trails
|
||||
|
||||
## Accessibility
|
||||
|
||||
- WCAG 2.1 AA compliance
|
||||
- Keyboard navigation
|
||||
- Screen reader support
|
||||
- Captions (always available)
|
||||
- Reduced motion support
|
||||
- ARIA labels
|
||||
|
||||
## Scalability
|
||||
|
||||
- Stateless services (behind load balancer)
|
||||
- Redis for session caching
|
||||
- PostgreSQL for persistent state
|
||||
- GPU cluster for avatar rendering
|
||||
- CDN for widget assets
|
||||
|
||||
213
docs/DEPLOYMENT.md
Normal file
213
docs/DEPLOYMENT.md
Normal file
@@ -0,0 +1,213 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose
|
||||
- PostgreSQL 16+ with pgvector extension
|
||||
- Redis 7+
|
||||
- (Optional) Kubernetes cluster for production
|
||||
|
||||
## Development Deployment
|
||||
|
||||
### 1. Database Setup
|
||||
|
||||
Run migrations:
|
||||
|
||||
```bash
|
||||
cd virtual-banker/database
|
||||
psql -U explorer -d explorer -f migrations/001_sessions.up.sql
|
||||
psql -U explorer -d explorer -f migrations/002_conversations.up.sql
|
||||
psql -U explorer -d explorer -f migrations/003_tenants.up.sql
|
||||
psql -U explorer -d explorer -f migrations/004_vector_extension.up.sql
|
||||
```
|
||||
|
||||
### 2. Start Services
|
||||
|
||||
Using the main docker-compose.yml:
|
||||
|
||||
```bash
|
||||
cd deployment
|
||||
docker-compose up -d virtual-banker-api virtual-banker-widget
|
||||
```
|
||||
|
||||
Or using the virtual-banker specific compose file:
|
||||
|
||||
```bash
|
||||
cd virtual-banker/deployment
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### 3. Verify
|
||||
|
||||
Check health:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8081/health
|
||||
```
|
||||
|
||||
Access widget:
|
||||
|
||||
```
|
||||
http://localhost:8082
|
||||
```
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**Backend API:**
|
||||
```bash
|
||||
DATABASE_URL=postgres://user:pass@host:5432/db
|
||||
REDIS_URL=redis://host:6379
|
||||
PORT=8081
|
||||
```
|
||||
|
||||
**Widget CDN:**
|
||||
- Deploy to CDN (Cloudflare, AWS CloudFront, etc.)
|
||||
- Configure CORS headers
|
||||
- Enable gzip compression
|
||||
|
||||
### Docker Compose Production
|
||||
|
||||
```yaml
|
||||
services:
|
||||
virtual-banker-api:
|
||||
image: your-registry/virtual-banker-api:latest
|
||||
environment:
|
||||
- DATABASE_URL=${DATABASE_URL}
|
||||
- REDIS_URL=${REDIS_URL}
|
||||
deploy:
|
||||
replicas: 3
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2'
|
||||
memory: 2G
|
||||
```
|
||||
|
||||
### Kubernetes Deployment
|
||||
|
||||
Example deployment:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: virtual-banker-api
|
||||
spec:
|
||||
replicas: 3
|
||||
selector:
|
||||
matchLabels:
|
||||
app: virtual-banker-api
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: virtual-banker-api
|
||||
spec:
|
||||
containers:
|
||||
- name: api
|
||||
image: your-registry/virtual-banker-api:latest
|
||||
env:
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: virtual-banker-secrets
|
||||
key: database-url
|
||||
ports:
|
||||
- containerPort: 8081
|
||||
resources:
|
||||
requests:
|
||||
memory: "1Gi"
|
||||
cpu: "1"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2"
|
||||
```
|
||||
|
||||
## Scaling
|
||||
|
||||
### Horizontal Scaling
|
||||
|
||||
- **API Service**: Stateless, scale horizontally
|
||||
- **Widget CDN**: Use CDN for global distribution
|
||||
- **Avatar Renderer**: GPU-bound, scale based on concurrent sessions
|
||||
|
||||
### Vertical Scaling
|
||||
|
||||
- **Database**: Increase connection pool, add read replicas
|
||||
- **Redis**: Use Redis Cluster for high availability
|
||||
- **Avatar Renderer**: Allocate more GPU resources
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Health Checks
|
||||
|
||||
- API: `GET /health`
|
||||
- Widget: `GET /health` (nginx)
|
||||
|
||||
### Metrics
|
||||
|
||||
- Session creation rate
|
||||
- Active sessions
|
||||
- API latency
|
||||
- Error rates
|
||||
- Avatar render queue
|
||||
|
||||
### Logging
|
||||
|
||||
- Structured logging (JSON)
|
||||
- Log aggregation (ELK, Loki, etc.)
|
||||
- Audit logs for compliance
|
||||
|
||||
## Security
|
||||
|
||||
### Network
|
||||
|
||||
- Use internal networks for service communication
|
||||
- Expose only necessary ports
|
||||
- Use TLS for external communication
|
||||
|
||||
### Secrets
|
||||
|
||||
- Store secrets in secret management (Vault, AWS Secrets Manager)
|
||||
- Rotate tokens regularly
|
||||
- Use ephemeral tokens for sessions
|
||||
|
||||
### Compliance
|
||||
|
||||
- Enable audit logging
|
||||
- Implement data retention policies
|
||||
- PII redaction in logs
|
||||
- Encryption at rest and in transit
|
||||
|
||||
## Backup & Recovery
|
||||
|
||||
### Database
|
||||
|
||||
- Regular PostgreSQL backups
|
||||
- Point-in-time recovery
|
||||
- Test restore procedures
|
||||
|
||||
### Redis
|
||||
|
||||
- Enable persistence (AOF/RDB)
|
||||
- Regular snapshots
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Session creation fails:**
|
||||
- Check database connection
|
||||
- Verify tenant exists
|
||||
- Check JWT validation
|
||||
|
||||
**Widget not loading:**
|
||||
- Check CORS configuration
|
||||
- Verify CDN is accessible
|
||||
- Check browser console for errors
|
||||
|
||||
**Avatar not displaying:**
|
||||
- Verify WebRTC connection
|
||||
- Check avatar renderer service
|
||||
- Verify GPU resources available
|
||||
|
||||
158
docs/WIDGET_INTEGRATION.md
Normal file
158
docs/WIDGET_INTEGRATION.md
Normal file
@@ -0,0 +1,158 @@
|
||||
# Widget Integration Guide
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Include the Widget Script
|
||||
|
||||
Add the widget loader script to your HTML page:
|
||||
|
||||
```html
|
||||
<script src="https://cdn.example.com/virtual-banker/widget.js"
|
||||
data-tenant-id="your-tenant-id"
|
||||
data-user-id="user-123"
|
||||
data-auth-token="jwt-token"
|
||||
data-api-url="https://api.example.com"
|
||||
data-avatar-enabled="true"></script>
|
||||
<div id="virtual-banker-widget"></div>
|
||||
```
|
||||
|
||||
### 2. Configuration Options
|
||||
|
||||
| Attribute | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `data-tenant-id` | Yes | Tenant identifier |
|
||||
| `data-user-id` | No | User identifier (for authenticated sessions) |
|
||||
| `data-auth-token` | No | JWT token for authentication |
|
||||
| `data-api-url` | No | API base URL (default: http://localhost:8081) |
|
||||
| `data-avatar-enabled` | No | Enable/disable avatar (default: true) |
|
||||
|
||||
## Programmatic API
|
||||
|
||||
### Methods
|
||||
|
||||
```javascript
|
||||
// Open widget
|
||||
window.VirtualBankerWidgetAPI.open();
|
||||
|
||||
// Close widget
|
||||
window.VirtualBankerWidgetAPI.close();
|
||||
|
||||
// Minimize widget
|
||||
window.VirtualBankerWidgetAPI.minimize();
|
||||
|
||||
// Set context (page/route information)
|
||||
window.VirtualBankerWidgetAPI.setContext({
|
||||
route: '/account',
|
||||
accountId: 'acc-123',
|
||||
productId: 'prod-456'
|
||||
});
|
||||
|
||||
// Update authentication token
|
||||
window.VirtualBankerWidgetAPI.setAuthToken('new-jwt-token');
|
||||
```
|
||||
|
||||
## PostMessage Events
|
||||
|
||||
Listen for widget events from the parent window:
|
||||
|
||||
```javascript
|
||||
window.addEventListener('message', (event) => {
|
||||
if (event.data.source === 'virtual-banker-widget') {
|
||||
switch (event.data.type) {
|
||||
case 'ready':
|
||||
console.log('Widget is ready');
|
||||
break;
|
||||
|
||||
case 'session_started':
|
||||
console.log('Session ID:', event.data.payload.sessionId);
|
||||
break;
|
||||
|
||||
case 'action_requested':
|
||||
console.log('Action:', event.data.payload.action);
|
||||
console.log('Params:', event.data.payload.params);
|
||||
// Handle action (e.g., open ticket, schedule appointment)
|
||||
break;
|
||||
|
||||
case 'action_completed':
|
||||
console.log('Action completed:', event.data.payload);
|
||||
break;
|
||||
|
||||
case 'handoff_to_human':
|
||||
console.log('Handoff reason:', event.data.payload.reason);
|
||||
// Show human agent interface
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Sending Messages to Widget
|
||||
|
||||
Send commands to the widget from the parent window:
|
||||
|
||||
```javascript
|
||||
const widget = document.getElementById('virtual-banker-widget');
|
||||
if (widget && widget.contentWindow) {
|
||||
widget.contentWindow.postMessage({
|
||||
type: 'open',
|
||||
source: 'virtual-banker-host'
|
||||
}, '*');
|
||||
}
|
||||
```
|
||||
|
||||
Available commands:
|
||||
- `open` - Open the widget
|
||||
- `close` - Close the widget
|
||||
- `minimize` - Minimize the widget
|
||||
- `setContext` - Update context
|
||||
- `setAuthToken` - Update auth token
|
||||
|
||||
## Styling
|
||||
|
||||
The widget can be styled via CSS:
|
||||
|
||||
```css
|
||||
#virtual-banker-widget {
|
||||
position: fixed;
|
||||
bottom: 20px;
|
||||
right: 20px;
|
||||
width: 400px;
|
||||
height: 600px;
|
||||
z-index: 9999;
|
||||
border-radius: 8px;
|
||||
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.2);
|
||||
}
|
||||
|
||||
#virtual-banker-widget.minimized {
|
||||
height: 60px;
|
||||
width: 200px;
|
||||
}
|
||||
```
|
||||
|
||||
## Theming
|
||||
|
||||
Theme can be configured per tenant via the backend API. The widget will automatically apply the theme from the session configuration.
|
||||
|
||||
## Accessibility
|
||||
|
||||
The widget is built with accessibility in mind:
|
||||
- Full keyboard navigation
|
||||
- Screen reader support
|
||||
- ARIA labels
|
||||
- Captions always available
|
||||
- Reduced motion support (respects OS preference)
|
||||
|
||||
## Browser Support
|
||||
|
||||
- Chrome/Edge 90+
|
||||
- Firefox 88+
|
||||
- Safari 14+
|
||||
- Mobile browsers (iOS Safari, Chrome Mobile)
|
||||
|
||||
## Security
|
||||
|
||||
- Content Security Policy (CSP) compatible
|
||||
- No direct secrets in browser
|
||||
- Ephemeral session tokens only
|
||||
- CORS configured for embedding
|
||||
|
||||
1137
docs/openapi.yaml
Normal file
1137
docs/openapi.yaml
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user