Real-Time Webcam to VRM Retargeting
Can a browser drive a 3D avatar using only a webcam?
I tested MediaPipe Holistic to see if it can estimate body, face, and hand movements to control a VRM 1.0 avatar in real time. I ran this test on an Apple M1 Max using CPU and WASM.
The results show that a standard webcam can drive a 3D avatar without expensive motion-capture gear.
Test Results: • Hardware: Apple M1 Max • Input: 1280x720 webcam stream • Effective speed: 17.3 FPS • Mean inference time: 49.17 ms • Success: All 141 detected pose frames updated the VRM skeleton
How it works: The system uses a multi-stage pipeline to track 553 points:
- 33 points for body pose
- 478 points for face mesh (including irises)
- 21 points for each hand
The process follows these steps:
- MediaPipe detects landmarks from the webcam
- Coordinates convert to Three.js space
- Data maps to 34 VRM bones and expressions
- Smoothing algorithms reduce jitter by interpolating movements
Retargeting works by aligning the direction between two landmarks with the VRM bone direction. For example, it maps the shoulder to the elbow to set the arm rotation. Facial coefficients map directly to avatar expressions like eye blinking or jaw opening.
Limitations:
- Speed: 17.3 FPS is good for demos but not smooth enough for professional 60 FPS motion capture.
- Rotation: This is a directional approximation. It does not solve exact twist for forearms or ankles.
- Complexity: The test did not prove stable simultaneous tracking of the face and both hands at once due to short testing duration.
This setup provides a lightweight way to create in-browser avatar demos and gesture interfaces.