00:03Hello, everyone. Welcome back to the lecture series on foundations of deep learning concepts
00:09and applications. In our last session, we have discussed about the activation functions
00:15and the loss functions in detail, the various types of activation and loss functions. And
00:20today in this session, we will discuss about the optimization of machine learning models.
00:26What exactly optimization means? What are the algorithms that we use for optimization of
00:31machine learning models? So, that is the topic of discussion for today's session.
00:36So, in the contents, we have the most important algorithm of optimization which is called as
00:43the gradient descent algorithm. So, we will discuss in detail what is this gradient descent
00:48algorithm in today's session. Now, to start with before I go into what is
00:54gradient descent algorithm, we will try to understand what exactly optimization algorithms
01:00do in neural network. What is the meaning of optimization and what is its significance in
01:06neural networks? So, to start with the process of minimizing or maximizing any mathematical
01:13expression is called optimization. Optimizers are algorithms or methods used to change the
01:20attributes of the neural network such as weights and biases to reduce the losses and to provide
01:26the most accurate results possible. So, we all know in machine learning, what we try to do is we try
01:35our
01:35model to predict some result. And we want the prediction to be as close as possible to the true output.
01:43Now, this optimization or optimizing algorithms, they help the neural network to updates the weights and biases
01:54in such a way that the output becomes as close as to the ground truth ok. Now, we will come
02:02to what is
02:03this gradient descent algorithm. So, as I told optimization algorithms in machine learning are mathematical techniques
02:12used to adjust models parameter to minimize errors and improve accuracy and gradient descent algorithm is heavily applied in deep
02:20learning.
02:21So, it is one of the most used algorithms for optimization in deep learning. Now, we will see what exactly
02:28is this gradient descent.
02:30So, we have a cartoon here. If we see at the cartoon, there is a cyclist which is at the
02:39top of a hill
02:40and he wants to come down and come down to a point where it is not a hill means from
02:48the top to the bottom he wants to come down.
02:51He cannot see the entire landscape, he just sees what is there below his foot and or he can feel
02:57that one
02:58and he wants to come down. Now, when we want to descend from the top of a hill,
03:06how do you choose the correct path? You can choose various paths, but some of them may take very long
03:12time to reach, some of them you may not be able to reach the you know the ground of the
03:19hill or the base
03:20of the hill. So, how do you choose the correct path? So, the same thing you can think when we
03:28are discussing
03:29about gradient descent algorithm. How do we think about that one? So, gradient descent is a first
03:36order iterative minimization method. It iteratively updates parameters to minimize the loss function.
03:43So, just as I was telling the cyclist is trying to come down to the base of the hill and
03:50his every
03:51location while he is coming down, it gives the measure how much he is away from the you know base
03:59of the hill. And if he changes the his position iteratively at one point of time, he will come
04:07down to the hill ok. The same thing we will see in gradient descent. How exactly it is? So,
04:13as I told it is a first order iterative minimization method. Now, what exactly we are minimizing here?
04:19We are minimizing the loss function which is nothing but it returns the error between the predicted
04:25outcome and the actual outcome ok. So, that is the loss or error of any algorithm of any model.
04:33And we are trying to minimize this loss in an iterative way ok. And while trying to do this,
04:42we use gradients or derivatives to decide the direction of change.
04:47Which means in which direction should I move? So, that I come down to the base of the hill
04:52without spending too much of time. So, that is done in gradient descent algorithm also using gradients
04:59or derivatives. It computes partial derivatives of the loss function with respect to every weight
05:07in the neural network using the chain rule of calculus. Now, these are all you know textbook
05:12definitions. We will now try to see what exactly step by step what is a derivative, what is a
05:19great you know gradient, what is partial derivative and how these are used to update the weights and
05:26biases in a neural network which finally affects the loss function that is to reduce the loss function
05:33ok. So, we will see how this entire process is done using gradients, derivatives and partial derivatives ok.
05:40Now, we are talking about or we will be talking about parameters in a neural network.
05:49When we talk about updating any parameter in a neural network, we mainly talk about the weights and
05:55biases because those are the parameters which have an effect on the loss function.
06:01Now, we will just look into as I told one by one, we will look into what is a derivative,
06:06what is a partial derivative, what is a gradient and then we will proceed to understand what is the
06:12gradient descent algorithm. Now, suppose we have a loss function L which is a function of the weight
06:20and it is given by 3W square. The derivative if we take that is derivative of L with respect to
06:29the
06:29weight W is equal to 6W. Now, suppose my loss function L is a function of weight and bias,
06:38W and B and it is given by 3W square plus B square something ok. Then the partial derivative,
06:45what do we mean by partial derivative? Partial derivative means when there is a function,
06:50suppose here L which is a function of 2 parameters W and B in this case, then we take the
06:58derivative of
06:59L with respect to one of the parameters keeping the other constant that we call as partial derivative ok.
07:07So, if we take the partial derivative of L with respect to W, then we get 6W ok. Similarly,
07:15if we take the partial derivative of L with respect to B keeping W constant, then we get 2W. Now,
07:23what is this gradient, we understood what is normal derivative, partial derivative.
07:27Now, we have to understand what is this gradient. Gradient is actually a vector which comprises
Comments