The Challenge
Developed during my tenure at ADabisc Future Qatar, the objective was to engineer a responsive, real-time facial capture system capable of driving the expressions of 3D characters.
I selected the Intel RealSense SR300 camera for its combination of structured coded-light depth measurement and infrared (IR) sensing, enabling robust depth discrimination under variable ambient lighting.
Engine Integration & Pipeline
- Unity Native Plugin: Integrated the Intel RealSense SDK into the Unity Game Engine via custom C++ native wrappers, unblocking high-throughput sensory pipeline streaming directly to engine memory.
- Landmark Tracking: Used the RealSense SDK face module in color + depth mode to track facial landmarks, head pose, and expressions, grouping the landmarks by face region (eyes, brows, nose, mouth, jaw) and smoothing them over time.
Calibration & Fast Training Pipeline
To solve the non-linear mapping between facial landmarks and 3D mesh blend shape weights, I first used my own C++ neural network, then switched to Dlib kernel ridge regression with a Radial Basis Function (RBF) kernel, which gave the same results. Dlib trains one model per blend shape, split into eye, brow, and mouth/jaw regions:
- Dataset Collection: The subject performs a sequence of designated facial expressions in front of the RealSense camera. Coordinate displacements are captured and stored directly to disk.
- Rapid Convergence: The models train on the captured pose dataset in 30 to 60 seconds, allowing instant calibration per actor.
Real-time Performance
Once trained, the system performs real-time inference, dynamically mapping live landmark positions directly to blend shape weights on the target 3D model with interactive framerates.