00:03less than f of x for all values of x. Now what happens here? Once again I will come
00:10back to this local minima picture and we will see there are 2 things that can happen here.
00:17When we have a non-convex loss function and we have this kind of a graph, then when we
00:29are coming down the valley, the algorithm may get stuck here because this is also looks
00:37like a minima. And we know mathematically any minima at any minima point the gradient
00:43is 0 ok. So, at this point in the local minima also the gradient will be 0, but this is
00:51not
00:51the global minima. That means, there is more chance of the error getting reduced, but we
00:56are stuck here ok. Now this is one thing that may come. Another thing is there is something
01:02called as the saddle point. Now what is this saddle point? Saddle point means suppose you
01:07have a local minima and then you have a global minima and you have something like this and
01:12in one side of this there is it is going down and in one side it is going up. And
01:17here it
01:17is like a straight you know kind of a region flat kind of a region where if you draw the
01:24gradient the gradient is 0, but it is not a local minima also. So, this is called as
01:30a saddle point. And what happens when we if we get stuck in a place where the gradient is
01:350, what is the most important or what is the flip side of getting the gradient equal to 0
01:44in this case, what is the disadvantage not flip side disadvantage. So, at the saddle point
01:49the gradient is 0, but it is not a local minima and what we see here is the weight updation
01:55which is your weight in the next iteration is equal to the weight in your present iteration
02:00minus eta into gradient of L. Now if your gradient is 0, then this factor becomes 0 and then your
02:07weight updation Wt plus 1 is equal to Wt. That means, your weight updation stops that means,
02:14your model stops learning. So, this is you know you know a problem which we might face
02:24when we are working with non-convex loss functions. For convex loss functions it will always go into
02:30the point of convergence will be the local global minima. But in this case it can get stuck in
02:36local minima or at a saddle point ok. So, gradient descent may settle in a local minima or as in
02:43a saddle point in case of non-convex loss function ok. Now, the question arises how to solve this
02:51problem if we are getting stuck in local minima or at a saddle point. This will be the topic
02:56of discussion in a subsequent lecture of mine. So, with this one I come to the end of today's
03:02session. We will quickly have a summarization of what all we have done today, what all we
03:09have discussed today. So, we started with what exactly optimization means. Optimization is
03:15the process of maximizing or minimizing any mathematical function that we got to know.
03:20And then what is the function of this, what is the reason why optimization is so important
03:25in machine learning or deep learning because we always want the error or the loss from our
03:32model to get reduced. So, we want to minimize the loss from the error from the model. And
03:38the algorithm that we learnt today is the gradient descent algorithm which is the backbone of most
03:46all of the deep learning models heavily deployed in deep learning. And then what we understood
03:54what exactly is gradient descent trying to do. Gradient descent is trying to find out a structured
04:00way of movement in the weights and biases plane. That means, weight updation and bias updation
04:07will happen in a structured way such that the loss after the weight the loss from the net network
04:15after the weight updation is reduced compared to your previous iteration. And this happens in
04:20an iterative way. It is the first order iterative process because we take the first order gradient
04:25of the weights of the loss function with respect to the weights and biases. And we have seen
04:30what is partial derivative, how the gradient comes, how the entire mathematical formulation
04:36of gradient descent algorithm happens. And then we have seen how the weight updations happened
04:41w t plus 1 in the t plus 1 1th time is equal to w t minus eta into gradient
04:48of l. Now, that
04:49eta is called as the learning rate. And there are constraints if we choose learning rate to
04:55be very small, there are constraints if we choose learning rate to be very large. For very small
05:00learning rates it takes a lot of time to converge to the minimum point. Whereas, if it is too high
05:06then it drastically changes the weights and what happens is it oscillates and we will miss
05:14the minimum point, it oscillates and it does not come to the minimum, it diverges. So, in
05:20that case we need to choose some learning rate which is optimal ok, which is neither too high
05:26nor too low. And then we have spoken about convex and non-convex loss functions, how is the landscape
05:33of convex loss functions look like, what is the landscape of non-convex loss functions look like,
05:38what is the concept of global minima and local minima in case of non-convex loss functions,
05:46what is saddle point. And what happens when in case of optimization of non-convex loss functions,
05:52what happens when we get stuck in a local minima or a saddle point ok. So, because there the weight
05:59updation gets stops. So, the learning of the model actually halts. So, with this I come to the
06:07conclusion of my today's session. I hope you all have understood the gradient descent algorithm.
06:15And in the next session I will be discussing about the back propagation which in which we do all these
06:23partial derivatives and take the partial derivative of loss functions, whatever I have told in this
06:28session we will look into how mathematically they are done. So, we will have a full session on back
06:35propagation algorithm. And then in the subsequent sessions we will look into how to solve this
06:41problem of getting stuck in local minima or saddle point. So, with that one I conclude today's session.
06:49Happy learning to all of you till we meet in the next session. Thank you all.
06:58I think it's time for now.
07:00We'll see you again next week.
07:00Bye-bye.
07:00Now, we'll be catching up with some of these.
07:01.
Comments