98 lines
2.5 KiB
Markdown
98 lines
2.5 KiB
Markdown
|
# Unreal Engine Avatar Setup
|
||
|
|
|
||
|
|
This directory contains the Unreal Engine project for the Virtual Banker avatar.
|
||
|
|
|
||
|
|
## Prerequisites
|
||
|
|
|
||
|
|
- Unreal Engine 5.3+ (or 5.4+ recommended)
|
||
|
|
- PixelStreaming plugin enabled
|
||
|
|
- Digital human character asset (Ready Player Me, MetaHuman, or custom)
|
||
|
|
|
||
|
|
## Setup Instructions
|
||
|
|
|
||
|
|
### 1. Create Unreal Project
|
||
|
|
|
||
|
|
1. Open Unreal Engine Editor
|
||
|
|
2. Create new project:
|
||
|
|
- Template: Blank
|
||
|
|
- Blueprint or C++: Blueprint (or C++ if custom code needed)
|
||
|
|
- Target Platform: Desktop
|
||
|
|
- Quality: Maximum
|
||
|
|
- Raytracing: Enabled (optional, for better quality)
|
||
|
|
|
||
|
|
### 2. Enable PixelStreaming
|
||
|
|
|
||
|
|
1. Edit → Plugins
|
||
|
|
2. Search for "Pixel Streaming"
|
||
|
|
3. Enable the plugin
|
||
|
|
4. Restart Unreal Editor
|
||
|
|
|
||
|
|
### 3. Import Digital Human
|
||
|
|
|
||
|
|
1. Import your digital human character:
|
||
|
|
- Ready Player Me: Use their Unreal plugin
|
||
|
|
- MetaHuman: Use MetaHuman Creator
|
||
|
|
- Custom: Import FBX/glTF with blendshapes
|
||
|
|
|
||
|
|
2. Set up blendshapes for visemes:
|
||
|
|
- Import viseme blendshapes (aa, ee, oh, ou, mbp, etc.)
|
||
|
|
- Map to animation system
|
||
|
|
|
||
|
|
### 4. Configure PixelStreaming
|
||
|
|
|
||
|
|
1. Edit → Project Settings → Plugins → Pixel Streaming
|
||
|
|
2. Configure:
|
||
|
|
- Streamer Port: 8888
|
||
|
|
- WebRTC Port Range: 8888-8897
|
||
|
|
- Enable WebRTC
|
||
|
|
|
||
|
|
### 5. Set Up Animation Blueprint
|
||
|
|
|
||
|
|
1. Create Animation Blueprint for avatar
|
||
|
|
2. Set up state machine:
|
||
|
|
- Idle
|
||
|
|
- Speaking (viseme-driven)
|
||
|
|
- Gesturing
|
||
|
|
- Expressions
|
||
|
|
|
||
|
|
3. Connect viseme blendshapes to animation graph
|
||
|
|
|
||
|
|
### 6. Create Control Blueprint
|
||
|
|
|
||
|
|
1. Create Blueprint Actor for avatar control
|
||
|
|
2. Add functions:
|
||
|
|
- SetVisemes(VisemeData)
|
||
|
|
- SetExpression(Valence, Arousal)
|
||
|
|
- SetGesture(GestureType)
|
||
|
|
- SetGaze(Target)
|
||
|
|
|
||
|
|
### 7. Build and Package
|
||
|
|
|
||
|
|
1. Package project for Linux (for server deployment):
|
||
|
|
- File → Package Project → Linux
|
||
|
|
- Or use command line:
|
||
|
|
```
|
||
|
|
UnrealEditor-Cmd.exe -run=UnrealVersionSelector -project="path/to/project.uproject" -game -cook -package -build
|
||
|
|
```
|
||
|
|
|
||
|
|
## Deployment
|
||
|
|
|
||
|
|
The packaged Unreal project should be deployed to a GPU-enabled server with:
|
||
|
|
- NVIDIA GPU (RTX 3090+ recommended)
|
||
|
|
- CUDA drivers
|
||
|
|
- Sufficient VRAM (8GB+ per instance)
|
||
|
|
|
||
|
|
## Integration
|
||
|
|
|
||
|
|
The renderer service (`avatar/renderer/service.go`) controls Unreal instances via:
|
||
|
|
- Process management (start/stop instances)
|
||
|
|
- WebSocket communication (animation parameters)
|
||
|
|
- PixelStreaming WebRTC streams
|
||
|
|
|
||
|
|
## Notes
|
||
|
|
|
||
|
|
- Each active session requires one Unreal instance
|
||
|
|
- GPU resources should be allocated per instance
|
||
|
|
- Consider using Unreal's multi-instance support for scaling
|
||
|
|
|