Knowledge Diversion: A Novel Paradigm for Pre-training Decomposable Image Generation Models

Yucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang, Xin Geng +1

Large-scale pre-trained models have become foundational to modern machine learning, yet conventional pre-training objectives primarily optimize source-domain performance without explicitly considering downstream requirements. Consequently, transferable and task-specific knowledge remains entangled within a shared parameter space, limiting selective knowledge reuse and efficient downstream adaptation. This limitation is particularly consequential for image-generation models, which must accommodate diverse conditions and domains. To address this challenge, we introduce Knowledge Diversion (KDiv), a condition-aware pre-training paradigm for learning inherently decomposable generative models. Before pre-training, KDiv factorizes randomly initialized weights into rank-one Knowledge Units, which are organized into learngenes and tailors. During pre-training, continuous condition-aware routing consolidates knowledge shared across conditions in learngenes while directing condition-dependent variations to tailors. This structured organization enables learngenes to capture transferable generative priors and tailors to develop condition-specific expertise, allowing the resulting model to be…