ScoreMix: A Scalable Augmentation Strategy for Training GANs With Limited Data

Jie Cao, Mandi Luo, Junchi Yu, Ming Hsuan Yang, Ran He

Research output: Contribution to journalArticlepeer-review


Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. To facilitate GAN training, current methods propose to use data-specific augmentation techniques. Despite the effectiveness, it is difficult for these methods to scale to practical applications. In this work, we present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks. We first produce augmented samples using the convex combinations of the real samples. Then, we optimize the augmented samples by minimizing the norms of the data scores, <italic>i.e.</italic>, the gradients of the log-density functions. This procedure enforces the augmented samples close to the data manifold. To estimate the scores, we train a deep estimation network with multi-scale score matching. For different image synthesis tasks, we train the score estimation network using different data. We do not require the tuning of the hyperparameters or modifications to the network architecture. The ScoreMix method effectively increases the diversity of data and reduces the overfitting problem. Moreover, it can be easily incorporated into existing GAN models with minor modifications. Experimental results on numerous tasks demonstrate that GAN models equipped with the ScoreMix method achieve significant improvements.

Original languageEnglish
Pages (from-to)1-16
Number of pages16
JournalIEEE transactions on pattern analysis and machine intelligence
Publication statusAccepted/In press - 2022

Bibliographical note

Publisher Copyright:

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'ScoreMix: A Scalable Augmentation Strategy for Training GANs With Limited Data'. Together they form a unique fingerprint.

Cite this