Differentiable algorithms with data-driven parameterization in 3D vision

Abstract

This thesis is concerned with designing and analyzing efficient differentiable data flow for representations in the field of 3D vision and applying it to different 3D vision tasks. To this end, the topic is looked upon from the perspective of differentiable algorithms, a more general variant of Deep Learning, utilizing the recently emerged tools in the field of differentiable programming. Contributions are made in the subfields of Graph Neural Networks (GNNs), differentiable matrix decompositions and implicit neural functions, which serve as important building blocks for differentiable algorithms in 3D vision. The contributions include SplineCNN, a neural network consisting of operators for continuous convolution on irregularly structured data, Local Spatial Graph Transformers, a GNN to infer local surface orientations on point clouds, and a parallel GPU solver for Eigendecomposition on a large number of symmetric matrices. For all methods, efficient forward and backward GPU implementations are provided. Consequently, two differentiable algorithms are introduced, composed of building blocks from these concept areas. The first algorithm, Differentiable Iterative Surface Normal Estimation, is an iterative algorithm for surface normal estimation on unstructured point clouds. The second algorithm, Group Equivariant Capsule Networks, is a version of capsule networks grounded in group theory for unsupervised pose estimation and, in general, for inferring disentangled representations from 2D and 3D data. The thesis concludes that a favorable trade-off in the metrics of efficiency, quality and interpretability can be found by combining prior geometric knowledge about algorithms and data types with the representational power of Deep Learning.