Analytical Probability Distributions and Exact Expectation-Maximization for Deep Generative Networks
Randall Balestriero,Sebastien PARIS,Richard Baraniuk
Deep Generative Networks (DGNs) with probabilistic modeling of their outputand latent space are currently trained via Variational Autoencoders(VAEs).In the absence of a known analytical form for the posterior andlikelihood expectation, VAEs resort to approximations, including(Amortized) Variational Inference (AVI) and Monte-Carlosampling.We exploit the Continuous Piecewise Affinepropertyof modern DGNs to derive their posterior and marginaldistributions as well as the latters first two moments. These findings enable us to derive an analytical Expectation-Maximization (EM) algorithm for gradient-free DGN learning.We demonstrate empirically that EM training of DGNs produces greaterlikelihood than VAE training.Our new framework will guide the design of new VAE AVI that better approximates the true posterior and open new avenues to apply standard statistical tools for model comparison, anomaly detection, and missing data imputation.
