Deep Learning: Why Now?

Rise of Deep Learning

The fundamental ideas behind deep learning and neural networks have been around for decades. The surge in its popularity has been driven by several factors.

Key Questions

• Why has deep learning become significant now?
• What has driven the recent successes?

Drivers Behind the Rise of Deep Learning

1. Amount of Data:

   • Digitization of society has led to the creation of vast amounts of data ↑.
   • Human digital activity, inexpensive cameras in phones, and the Internet of Things have contributed to this data surge.

2. Performance with Increased Data:

   • Traditional learning algorithms (like SVM and logistic regression) plateau after a certain data point.
   • Neural networks, especially large ones, continue to improve with more data.

3. Computation Scale:

   • Neural network size (more hidden units and parameters) and data scale drive progress in deep learning.
   • Today, one of the most reliable ways to improve neural networks is to train bigger networks or use more data.

4. Computation and Algorithmic Advancements:

   • The ability to train large neural networks efficiently using GPUs or CPUs has contributed significantly.
   • Algorithmic innovations, like switching from the sigmoid activation function to the ReLU function, have made training faster. (yield faster gradient descent)

5. Rapid Iteration in Neural Network Design:

   • Fast computation allows for quick feedback on neural network designs, enabling faster improvements.

Future of Deep Learning

• Continual increase in data with growing digital footprints.
• Advancements in computational capabilities, including specialized hardware like GPUs.
• Continuous improvements in algorithms driven by an active deep learning research community.

Conclusion

Deep learning, powered by data, computational advances, and algorithmic innovations, will continue to advance in the foreseeable future.

What will the variable m denote in this course?

• Number of training examples