- LiteVAE: A Lightweight and Efficient Variational Autoencoder for Latent Diffusion Models
Authors: Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, Romann M. Weber
Abstract: Advances in latent diffusion models (LDMs) have revolutionized high-resolution image generation, yet the design space of autoencoders at the heart of these systems remains underexplored. In this paper, we present LiteVAE, a family of autoencoders for LDMs that leverages the 2D discrete wavelet transform to provide improved scalability and computational efficiency over standard variational autoencoders (VAEs) without sacrificing output quality. We also explore LiteVAE's training methodology and decoder architecture, and propose several enhancements that improve training dynamics and reconstruction quality. Our baseline LiteVAE model matches the quality of established VAEs for current LDMs by reducing the encoder parameters by a factor of 6, leading to faster training and reduced GPU memory requirements. Meanwhile, our larger-scale models outperform VAEs of comparable complexity across all metrics evaluated (rFID, LPIPS, PSNR, and SSIM).
2. Poisson Variational Autoencoder
Authors: Hadi Vafaii, Dekel Galor, Jacob L. Yates
Summary: Variational autoencoders (VAEs) use Bayesian inference to interpret sensory inputs, mirroring processes occurring in both the ventral (Higgins et al., 2021) and dorsal (Vafaii et al., 2023) pathways in primate vision. Despite their success, traditional VAEs rely on continuous latent variables, which deviate significantly from the discrete nature of biological neurons. Here, we develop a novel architecture, the Poisson VAE (P-VAE), that combines the principles of predictive coding with a VAE that encodes inputs into discrete spike counts. Combining Poisson-distributed latent variables with predictive coding introduces a metabolic cost term in the model loss function, suggesting a relationship with sparse coding, which we empirically validate. Furthermore, we analyze the geometry of the learned representations and contrast the P-VAE with alternative VAE models. We find that the P-VAE encodes inputs in a relatively high dimensionality, facilitating linear separation of categories in downstream classification tasks and significantly improving sample efficiency (by a factor of 5). Our work provides an interpretable computational framework for studying brain-like sensory processing and paves the way for a deeper understanding of perception as an inferential process.
