Updating diffusion models in an incremental setting would be practical in real-world applications yet computationally challenging. We present a novel learning strategy of Concept Neuron Selection, a simple yet effective approach to perform personalization in a continual learning scheme. CNS uniquely identifies neurons in diffusion models that are closely related to the target concepts. In order to mitigate catastrophic forgetting problems while preserving zero-shot text-to-image generation ability, CNS finetunes concept neurons in an incremental manner and jointly preserves knowledge learned of previous concepts. Evaluation of real-world datasets demonstrates that CNS achieves state-of-the-art performance with minimal parameter adjustments, outperforming previous methods in both single and multi-concept personalization works. CNS also achieves fusion-free operation, reducing memory storage and processing time for continual personalization.
(a) Identify neurons highly responsive to the target concept (base neurons).
(b) Search for the neurons that consistently activate across many unrelated prompts with diverse calibration set (general neurons).
(c) Remove general neurons from the base neurons.
(d) Only fine-tune concept neurons and special token standing for the concept.
Note that only Continual Diffusion and CNS are capable of performing continual personalization, while Mix-of-Show and Orthogonal Adaptation require to keep LoRAs for each concept for personalization. It can be seen that our personalized outputs match concepts learned across different time, alleviating appearance leakage and catastrophic forgetting problems.
In addition to the alignment-based metrics of CLIP-I and CLIP-T, we provide the computation estimates for different personalization methods. Note that memory requirements for GPU/CPU and computation time for multi-concept personalization indicate the additional costs for fusing concept weights previously learned.
@InProceedings{Liao_2025_ICCV,
author = {Liao, Yu-Chien and Chen, Jr-Jen and Huang, Chi-Pin and Lin, Ci-Siang and Wu, Meng-Lin and Wang, Yu-Chiang Frank},
title = {Continual Personalization for Diffusion Models},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {15511-15520}
}