virtual-banker/avatar/unreal/README.md

Unreal Engine Avatar Setup

This directory contains the Unreal Engine project for the Virtual Banker avatar.

Prerequisites

• Unreal Engine 5.3+ (or 5.4+ recommended)
• PixelStreaming plugin enabled
• Digital human character asset (Ready Player Me, MetaHuman, or custom)

Setup Instructions

1. Create Unreal Project

1. Open Unreal Engine Editor
2. Create new project:
   • Template: Blank
   • Blueprint or C++: Blueprint (or C++ if custom code needed)
   • Target Platform: Desktop
   • Quality: Maximum
   • Raytracing: Enabled (optional, for better quality)

2. Enable PixelStreaming

1. Edit → Plugins
2. Search for "Pixel Streaming"
3. Enable the plugin
4. Restart Unreal Editor

3. Import Digital Human

1. Import your digital human character:

   • Ready Player Me: Use their Unreal plugin
   • MetaHuman: Use MetaHuman Creator
   • Custom: Import FBX/glTF with blendshapes

2. Set up blendshapes for visemes:

   • Import viseme blendshapes (aa, ee, oh, ou, mbp, etc.)
   • Map to animation system

4. Configure PixelStreaming

1. Edit → Project Settings → Plugins → Pixel Streaming
2. Configure:
   • Streamer Port: 8888
   • WebRTC Port Range: 8888-8897
   • Enable WebRTC

5. Set Up Animation Blueprint

1. Create Animation Blueprint for avatar

2. Set up state machine:

   • Idle
   • Speaking (viseme-driven)
   • Gesturing
   • Expressions

3. Connect viseme blendshapes to animation graph

6. Create Control Blueprint

1. Create Blueprint Actor for avatar control
2. Add functions:
   • SetVisemes(VisemeData)
   • SetExpression(Valence, Arousal)
   • SetGesture(GestureType)
   • SetGaze(Target)

7. Build and Package

1. Package project for Linux (for server deployment):
   • File → Package Project → Linux
   • Or use command line:

     UnrealEditor-Cmd.exe -run=UnrealVersionSelector -project="path/to/project.uproject" -game -cook -package -build
     

Deployment

The packaged Unreal project should be deployed to a GPU-enabled server with:

• NVIDIA GPU (RTX 3090+ recommended)
• CUDA drivers
• Sufficient VRAM (8GB+ per instance)

Integration

The renderer service (avatar/renderer/service.go) controls Unreal instances via:

• Process management (start/stop instances)
• WebSocket communication (animation parameters)
• PixelStreaming WebRTC streams

Notes

• Each active session requires one Unreal instance
• GPU resources should be allocated per instance
• Consider using Unreal's multi-instance support for scaling