We present a sparse representation of model uncertainty for Deep Neural Networks (DNNs) where the parameter posterior is approximated with an inverse formulation of the Multivariate Normal Distribution (MND), also known as the information form. The key insight of our work is that the information matrix, i.e. the inverse of the covariance matrix tends to be sparse in its spectrum. Therefore, dimensionality reduction techniques such as low rank approximations (LRA) can be effectively exploited. To achieve this, we develop a novel sparsification algorithm and derive a cost-effective analytical sampler. As a result, we show that the information form can be scalably applied to represent model uncertainty in DNNs. Our exhaustive theoretical analysis and empirical evaluations on various benchmarks show the competitiveness of our approach over the current methods.


    Access

    Download


    Export, share and cite



    Title :

    Estimating Model Uncertainty of Neural Networks in Sparse Information Form


    Contributors:

    Conference:



    Publication date :

    2020-07-13


    Type of media :

    Conference paper


    Type of material :

    Electronic Resource


    Language :

    English




    Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes

    Lee, Jongseok / Feng, Jianxiang / Humt, Matthias et al. | German Aerospace Center (DLR) | 2021

    Free access


    AIAA-2006-1923 Uncertainty Quantification During Integral Validation: Estimating Parametric, Model Form, and Solution Contributions

    Logan, R. / Nitta, C. / Chidester, S. et al. | British Library Conference Proceedings | 2006


    Estimating Uncertainty of Work Zone Capacity using Neural Network Models

    Bian, Zilin / Ozbay, Kaan | Transportation Research Record | 2019