Avatar scanning and personalization from consumer devices

Abstract

This thesis presents methods for creating and personalizing realistic mesh-based avatars from smartphone input data. In the first part, we present an avatar reconstruction method that enables non-expert users to generate high-quality avatars using only consumer hardware, i.e., smartphones. To the best of our knowledge, this is the first fully automated end-to-end system that integrates guided smartphone capture with automated reconstruction to generate personalized, realistic, and VR-ready full-body avatars without requiring additional user input or preprocessing. Our reconstruction system consists of an iOS application that visually guides users through the scanning process and a server that automatically computes realistic avatars using photogrammetry, landmark detection, and template fitting. While this setup allows accessible avatar creation, the system is prone to motion-induced errors during scanning. We address this by reducing the effects of micro-motion at the photogrammetry stage and adapting the reconstruction method accordingly. Specifically, we compute partial point clouds from sub-scans using photogrammetry, incorporate silhouette constraints to improve fitting robustness, and apply inverse rendering to refine the texture on a per-image basis.

The resulting avatars accurately match the scanned person's shape. However, animating the person's face is only possible with generic blendshapes. While this is sufficient to create new facial expressions, the resulting facial expression may lead to incongruent social cues. To address this, the second part of this thesis focuses on the personalization and animation of facial expressions. Building on the example-based facial rigging (EBFR) approach by Li et al. (2010), we introduce a method for personalizing blendshapes of realistic 3D avatars using smartphone input data. We developed an application to automatically capture example facial expressions. From these, we compute $52$ ARKit-compatible blendshapes using a position-based EBFR formulation and seamless partial deformation transfer. The resulting personalized blendshapes enable users to create faithful face animations. Finally, we evaluate the plausibility and naturalness of these facial animations by comparing different blendshape animation methods, including synthesized, tracked, and static (non-animated) facial expressions.