Reading List for Techniques in Training Neural Networks

1st week: Data augmentation in Computer Vision and Natural Language Processing

• Barret Zoph, Ekin D. Cubuk, Golnaz Ghiasi, Tsung-Yi Lin, Jonathon Shlens, Quoc V. Le: Learning Data Augmentation Strategies for Object Detection. ECCV (27) 2020: 566-583

• Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woo-Myoung Park: GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation. EMNLP 2021: 2225-2239

2nd week: Data augmentation in Graphs and Time Series

• Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, Liang Wang: Graph Contrastive Learning with Adaptive Augmentation. WWW 2021: 2069-2080

• Jinsung Yoon, Daniel Jarrett, Mihaela van der Schaar: Time-series Generative Adversarial Networks. NeurIPS 2019: 5509-5519

3rd week: Normalization Methods

• Tim Salimans, Diederik P. Kingma: Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. NIPS 2016: 901

• Chunjie Luo, Jianfeng Zhan, Xiaohe Xue, Lei Wang, Rui Ren, Qiang Yang: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks. ICANN (1) 2018: 382-391

4th week: Batch size Optimization

• Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. ICLR 2020

• Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le: Don't Decay the Learning Rate, Increase the Batch Size. ICLR 2018

5th week: Optimization algorithms: Gradient Descent methods

• Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur: Sharpness-aware Minimization for Efficiently Improving Generalization. ICLR 2021

• Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha C. Dvornek, Xenophon Papademetris, James S. Duncan: AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. NeurIPS 2020

6th week: Optimization algorithms: Evolutionary methods

• Shimin Li, Huiling Chen, Mingjing Wang, Ali Asghar Heidari, Seyedali Mirjalili: Slime mould algorithm: A new method for stochastic optimization. Future Gener. Comput. Syst. 111: 300-323 (2020)

• Seyedali Mirjalili, Andrew Lewis: The Whale Optimization Algorithm. Adv. Eng. Softw. 95: 51-67 (2016)

7th week: Initialization of Neural Network

• Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015: 1026-1034

• Ari S. Morcos, Haonan Yu, Michela Paganini, Yuandong Tian: One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. NeurIPS 2019: 4933-4943

8th week: Curriculum Learning

• Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, Koray Kavukcuoglu: Automated Curriculum Learning for Neural Networks. ICML 2017: 1311-1320

• Guy Hacohen, Daphna Weinshall: On The Power of Curriculum Learning in Training Deep Networks. ICML 2019: 2535-2544

9th week: Incremental/Online Learning

• Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu: Large Scale Incremental Learning. CVPR 2019: 374-382

• Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, Karteek Alahari: End-to-End Incremental Learning. ECCV (12) 2018: 241-257

10th week: Convergence Analysis in Deep Learning

• Sanjeev Arora, Nadav Cohen, Noah Golowich, Wei Hu: A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. ICLR 2019

• Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song: A Convergence Theory for Deep Learning via Over-Parameterization. ICML 2019: 242-252

11th week: The Generalization of Neural Networks

• Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals: Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64(3): 107-115 (2021)

• Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song: A Convergence Theory for Deep Learning via Over-Parameterization. ICML 2019: 242-252

12th week: Fairness in Neural Networks

• Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, Vicente Ordonez: Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations. ICCV 2019: 5309-5318

• Yi Chern Tan, L. Elisa Celis: Assessing Social and Intersectional Biases in Contextualized Word Representations. NeurIPS 2019: 13209-13220