Efficient Fréchet Distributional Decoder Alignment for Latent Generative Models

Xianghong Fang, Jiajun Zhu, Esma Aimeur, Dehan Kong, Tim G. J. Rudner

Fréchet Distributional (FD) objectives provide an effective distribution-level signal for post-training generative models. For latent generative models, however, existing FD-based post-training methods optimize only the latent generator while keeping the decoder frozen: the decoder is trained on reconstruction latents obtained from real images but deployed on generator-produced latents at generation time. When these latent distributions differ, this \textbf{decoder train-generation mismatch} means that the decoder is deployed on inputs drawn from a distribution it was not directly trained on. We introduce \textbf{Fréchet Distributional Decoder Alignment (FDDA)}, an efficient post-training method that freezes the latent generator and directly optimizes the decoder on generation-time latents using an FD objective. A single epoch of FDDA consistently improves gFID and gFDr$^6$ across diverse flow-based and autoregressive latent generative models. Across all 11 evaluated models, FDDA reduces gFDr$^6$ by 34.0-59.6\% from the original pretrained models and provides a further 10.2-57.2\% reduction after generator-side FD post-training. FDDA also reduces total wall-clock time by factors…