Introduction to Gradient Descent Algorithm

"It's about to go down! 👇"
11 January 2022

In this series, we are going to learn one of the well-known optimization algorithms, namely Gradient Descent algorithm. However, we will go through the series gradually, so that the readers will not get overwhelmed with lots of details, especially the mathematical notations and formulas. First, we are going to start from Batch Gradient Descent. Slowly, we are going to customize the algorithm bit by bit to get to the most advanced version of Gradient Descent, which is Adam.

Here is the list of algorithms that we are going to cover:

  1. Batch Gradient Descent
  2. Mini Batch Gradient Descent
  3. Stochastic Gradient Descent
  4. SGD with Momentum (Coming Soon)
  5. SGD with Nesterov & Momentum (Coming Soon)
  6. AdaGrad (Coming Soon)
  7. RMSProp (Coming Soon)
  8. Adam (Coming Soon)
  9. AdaMax (Coming Soon)
  10. AdaGradDelta (Coming Soon)
  11. Nadam (Coming Soon)
  12. AdamW (Coming Soon)

By the end of this series, we will be able to write those optimization algorithms from scratch in Python.