3D face modeling for multiview facial action detection

Abstract

Facial action unit (AU) detection struggles with the lack of 3D information from the common way of detection, using 2D images. This paper leverages 3D face reconstruction models, parametric facial data, and other multi-view techniques to tackle this problem. Using a state-of-the-art AU detection model, we show that incorporating 3D data has various advantages, such as privacy preservation, enhanced detection at various sharp angles, and improved performance. Our fusion model, which combines typical image-based AU detection with a model that is designed for FLAME parameter embeddings, achieves greater results than other current state-of-the-art models on the BP4D dataset. In this paper, we present and compare these various models in great detail.