| Download | - View author's version: Over-training with Mixup may hurt generalization (PDF, 4.7 MiB)
|
|---|
| Author | Search for: Liu, Zixuan; Search for: Wang, Ziqiao; Search for: Guo, Hongyu1; Search for: Mao, Yongyi |
|---|
| Affiliation | - National Research Council of Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | Eleventh International Conference on Learning Representations (ICLR) 2023, May 1-5, 2023, Kigali, Rwanda [Hybrid] |
|---|
| Physical description | 20 p. |
|---|
| Subject | Mixup; generalization; overfitting; regularization |
|---|
| Abstract | Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation. |
|---|
| Publication date | 2023-05-05 |
|---|
| Publisher | The Conference
OpenReview |
|---|
| In | |
|---|
| Other version | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | 64a48c99-034a-4a0b-a1c7-b30e985ff93e |
|---|
| Record created | 2023-08-11 |
|---|
| Record modified | 2023-08-14 |
|---|
