Leveraging data-based inductive biases in deep learning

Abstract

Deep learning has revolutionised pattern recognition and decision-making, yet its success crucially depends on the inductive biases embedded in models implicit assumptions that guide learning toward useful solutions. These biases determine not only what a model can learn, but also how efficiently it can do so. Despite their importance, the sources and effects of inductive biases are rarely viewed in a more general manner. This dissertation investigates how inductive biases can be systematically identified and leveraged to improve learning across diverse domains. It formalises a distinction between two complementary kinds of inductive biases: (i) model-based inductive biases, which are encoded in architectures, loss functions, and regularization terms; and (ii) data-based inductive biases, which derive directly from structural or statistical regularities in the data. While the former has driven much of modern progress from convolutional networks to attention-based models, the latter remains comparatively underexplored despite its potential to capture task-specific invariances that are difficult to encode architecturally. The central contribution of this thesis is to define data-based inductive biases, which can be used to complement and enhance model-based inductive biases to create more robust and generalisable models. For this, a framework is introduced to describe how data-based inductive biases can be extracted and integrated into existing models, enhancing learning without the need for architectural redesign or extensive supervision. Latent variable modeling plays a key role in this framework, providing a principled means of discovering and encoding data structure as an additional learning signal. The chapters in this thesis demonstrate this principle across multiple domains. Cyclophobic Reinforcement Learning introduces a hand-crafted data-based inductive bias via hierarchical latent spaces over environment views, improving exploration by discouraging cyclic trajectories across these hierarchical perspectives. The Power of LLM-Generated Synthetic Data for Online Political Discussions uses synthetic data generated by large language models to define a controllable latent space, identifying ambiguous real-world samples that can be removed to reduce annotation cost or manually labelled to refine the model’s decision boundary. Just Cluster It: An Approach for Exploration in High Dimensions Using Clustering and Pre-Trained Representations shows how clustering DINO representations with Gaussian mixture models enables learning a latent density model over the state space, yielding more informative pseudo-count estimates for intrinsic motivation in 3D environments. Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representations demonstrates how augmenting DINOv2 representations with object-centric latent features learned through a Variational Autoencoder mitigates limitations of foundation models in representing non-geometric features such as colour and texture. Across these applications, the integration of data-derived structure consistently enhances performance and interpretability, showing that data-based inductive biases are a fundamental component to model learning.