metaverseDubai/docs/FACIAL_CAPTURE.md

Facial Capture - MetaHuman Animator

Overview

This document describes the facial capture workflow using MetaHuman Animator for the Dubai Metaverse project.

MetaHuman Animator

Overview

MetaHuman Animator is a tool for capturing facial animation from video or live performance and applying it to MetaHumans in Unreal Engine.

Requirements

• MetaHuman Animator Plugin: Enable in Unreal Engine
• Camera: High-quality camera (webcam, DSLR, or phone)
• Lighting: Good lighting conditions
• Calibration: Facial calibration process

---

Setup

Plugin Installation

1. Enable Plugin:

   • Edit > Plugins
   • Search for "MetaHuman Animator"
   • Enable plugin
   • Restart editor

2. Verify Installation:

   • Check plugin is active
   • Verify MetaHuman Animator tools are available

Camera Setup

1. Camera Selection:

   • Webcam: Basic setup
   • DSLR: Higher quality
   • Phone: Convenient option

2. Lighting:

   • Even Lighting: Avoid harsh shadows
   • Front Lighting: Light from front
   • Avoid Backlighting: Avoid strong backlighting

3. Positioning:

   • Distance: 2-3 feet from camera
   • Angle: Face camera directly
   • Stability: Keep head still

---

Calibration

Calibration Process

1. Start Calibration:

   • Open MetaHuman Animator
   • Start calibration process
   • Follow on-screen instructions

2. Calibration Steps:

   • Neutral Expression: Maintain neutral expression
   • Head Movement: Follow head movement instructions
   • Facial Expressions: Perform requested expressions
   • Completion: Complete calibration

3. Calibration Validation:

   • Review calibration quality
   • Re-calibrate if needed
   • Save calibration

---

Recording

Live Performance

1. Start Recording:

   • Begin live performance
   • Speak dialogue
   • Perform expressions

2. Performance Tips:

   • Clear Speech: Speak clearly
   • Natural Expressions: Use natural facial expressions
   • Head Movement: Natural head movement
   • Eye Contact: Look at camera

3. Stop Recording:

   • Stop recording when complete
   • Review performance
   • Process if satisfied

Video Recording

1. Record Video:

   • Record video of performance
   • Ensure good quality
   • Follow recording guidelines

2. Import Video:

   • Import video to MetaHuman Animator
   • Process video
   • Extract facial animation

---

Processing

Animation Processing

1. Process Performance:

   • MetaHuman Animator processes performance
   • Extracts facial animation
   • Generates animation data

2. Review Animation:

   • Review processed animation
   • Check quality
   • Refine if needed

3. Export Animation:

   • Export animation to Unreal Engine
   • Apply to MetaHuman
   • Test in level

---

Application

Apply to MetaHuman

1. Select MetaHuman:

   • Select MetaHuman in level
   • Open MetaHuman Animator

2. Apply Animation:

   • Apply captured animation
   • Preview animation
   • Adjust if needed

3. Integration:

   • Integrate with dialogue system
   • Sync with audio
   • Test in context

---

Best Practices

Recording

1. Quality: Ensure good recording quality
2. Lighting: Use good lighting
3. Stability: Keep head stable
4. Natural: Use natural expressions

Processing

1. Calibration: Ensure good calibration
2. Review: Review processed animation
3. Refinement: Refine if needed
4. Testing: Test in context

---

Troubleshooting

Common Issues

Issue: Calibration fails

• Solution: Check lighting conditions
• Solution: Ensure face is clearly visible
• Solution: Follow calibration instructions carefully

Issue: Animation quality poor

• Solution: Improve recording quality
• Solution: Re-calibrate
• Solution: Check lighting

Issue: Animation not applying

• Solution: Check MetaHuman Animator plugin
• Solution: Verify animation export
• Solution: Check MetaHuman setup

---

Use Cases

NPC Dialogue

• Purpose: Animate NPCs during dialogue
• Implementation: Capture dialogue performance
• Application: Apply to MetaHuman NPCs

Cinematic Sequences

• Purpose: Animate characters in cinematics
• Implementation: Capture performance for cinematics
• Application: Apply to cinematic MetaHumans

---

Documentation

Performance Documentation

Document each performance:

• Purpose: What the performance is for
• MetaHuman: Which MetaHuman it's applied to
• Context: Where it's used
• Quality: Animation quality notes

---

Version: 1.0 Last Updated: [Current Date]

Note: Facial capture is optional. NPCs can function without facial capture using standard animation.