The current scalable Bayesian methods for Deep Neural Networks (DNNs) often rely on the Fisher Information Matrix (FIM). For the tractable computations of the FIM, the Kronecker-Factored Approximate Curvature (K-FAC) method is widely adopted, which approximates the true FIM by a layer-wise block-diagonal matrix, and each diagonal block is then Kronecker-factored. In this paper, we propose an alternative formulation to obtain the Kronecker-factored FIM. The key insight is to cast the given FIM computations into an optimization problem over the sums of Kronecker products. In particular, we prove that this formulation is equivalent to the best rank-one approximation problem, where the well-known power iteration method is guaranteed to converge to an optimal rank-one solution - resulting in our novel algorithm: the Kronecker-Factored Optimal Curvature (K-FOC). In a proof-of-concept experiment, we show that the proposed algorithm can achieve more accurate estimates of the true FIM when compared to the K-FAC method.


    Zugriff

    Download


    Exportieren, teilen und zitieren



    Titel :

    Kronecker-Factored Optimal Curvature


    Beteiligte:
    Schnaus, Dominik (Autor:in) / Lee, Jongseok (Autor:in) / Triebel, Rudolph (Autor:in)

    Kongress:



    Erscheinungsdatum :

    2021-12-14


    Medientyp :

    Aufsatz (Konferenz)


    Format :

    Elektronische Ressource


    Sprache :

    Englisch




    Design Enhancement Through Human Factored Function Allocation

    Welch, D. / IEEE / AIAA | British Library Conference Proceedings | 1994


    FACMAC: Factored multi−agent centralised policy gradients

    Peng, B / Rashid, T / Schroeder de Witt, CA et al. | BASE | 2021

    Freier Zugriff

    Parallelization of the Factored Implicit Finite Difference Technique

    Abraham N. Varghese / Peter E. Raadt | AIAA | 1993


    How quality is factored into the design process

    Berke,M.I. / Chrysler,US | Kraftfahrwesen | 1989


    How quality is factored into the design process

    Burke, M.I. | Tema Archiv | 1989