00:03gradient descent algorithm. So, this is how the gradient descent algorithm works ok. Now,
00:12so if I want to show the algorithm, so we start at t equal to 0, maximum iterations
00:17we take as, we take it usually a very high value of 500 or 1000 and while t is less
00:24than
00:24this max iterations keep doing you know updating the weights using these equations and then
00:33you know when the number of iterations are reached you come out.
00:36So, that is the algorithm and usually we keep the number of iterations large value, so that
00:41we can continue this process of weight updation can continue till it converges ok. So, converging
00:49means it comes to the lowest point, it comes to the minimum error point ok. Now, we will
00:57come to something which is called as the learning rate in gradient descent, what is this learning
01:02rate. Now, we have seen this is the way the weights are updated, w t plus 1 at the time
01:08t plus 1 is equal to at the time t minus eta gradient of the loss function with respect
01:16to the weights. Now, this eta is called as the learning rate.
01:21Now, this eta can take any value, it can take 1, 2, 3, 0.1, 0.2, 0.3 any
01:26value, only thing
01:26we have told that it cannot take a negative value ok. Now, suppose I have a loss function
01:34in the y axis suppose weights are there in the x axis and we have you know we start at
01:39a random point in the curve and we slowly we come down to a point, this is my point of
01:47convergence,
01:48this we called as the you know convergence point which is the minimum point ok, this is
01:53my point of convergence. Now, if my learning rate is very small, what happens is we require
02:02a lot of steps before we reach to the minimum point. So, it takes you know a lot of time
02:08to
02:08come to the convergence. Suppose my learning rate is very high like this ok. So, we start
02:15from here and it goes in this you know this learning rate is very high instead of coming
02:20like this drastically it goes here, then again from here it goes here, again it goes here.
02:25So, what happens is in this case if the learning rate is high drastic updates may lead to oscillations
02:33and can never converge. So, it leads to oscillations we can see that it is not coming to the point
02:37of convergence. So, if it is the point of convergence it is never coming there ok, it is just oscillating.
02:43So, we miss the minimum ok. And then this is something where we have an optimal learning
02:49rate where it takes you know some value of eta which is not too small or not too large and
02:57we get into the point of convergence or minima point very fast ok. So, basically this eta the
03:06learning parameter learning rate is a hyper parameter in neural network that controls how quickly
03:12the model learns during the training. And usually eta values are chosen to be 0.01, 0.03, 0.1,
03:200.3, 0.3 etcetera ok. Those are the usual default values that are chosen.
03:24Now, so we have learnt about till now we have learnt about gradient descent, we have learnt about
03:29weight updations in gradient descent algorithm, how the weight updations happen, then what
03:35is the learning rate. Now, we will come to something called as the or some concept about
03:42convex and non-convex loss functions. Now, we will try to see what is a convex function,
03:47what is a non-convex function. Suppose we have a graph fx equal to x square and it looks
03:52like this. Now, if we take 2 points in the graph and we try to join them, what we see
03:59is a line connecting 2 points on the curve always lies above the curve, if you take any
04:04point it will be above the curve ok. So, this type of curves or functions are called as convex
04:09function. Whereas, if we take a function like this x is this one fx and it is like this and
04:15you take 2 points and you join them, it is not always that it will be above the curve. So,
04:23these type of functions they are called as non-convex functions. Now, if we see this non-convex function
04:30and if we compare with the convex function, this convex function is having only one minima
04:42point ok. So, there are multiple value in a non-convex function.
04:48Now what happens if my loss function is of this type ok. If it is a convex function, very
04:55good we have only one minima point and we will try to reach there. But if my loss function
04:59is a non-convex function, what are the things that may come into picture, we will try to see.
05:05And here I have told that gradient descent converges to global minima in case of a convex function.
05:10We will look into what is a global minima, what is a local minima and then again we will
05:14come back to this point. So, as I was telling this is a non-convex loss function and if you
05:21see these are called the local minima because they are like a valley, but they are not the
05:29point where you get the minimum of the curve. So, wherever you are getting the minimum of
05:35the entire region that is called as the global minima and the other valley points we call
05:40as local minima. Similarly, it is the global maxima and these in between you know the crests
05:49they are called as a local maxima. So, this is very much possible when we have a landscape
05:57of non-convex loss function. So, for global minima what mathematically how we can write
06:03it as if x star is your point of global minima, then f of x star is less than f
06:10of x for all
06:11x minus x star mod, that means in the vicinity point. So, that means, in a vicinity point this
06:11is all x belonging to r ok. So, it is minima over all possible values of f, if this is
06:17x
06:17then all possible values of x your f x star is the minimum. Now, what about local minima
06:26how do I define it mathematically? So, if x star is my local minima in that case f of x
06:33star
06:33is less than equal to f of x for all x minus x star mod that means, in a vicinity
06:40point which
06:41is less than delta and if there exist a delta greater than 0. What does it mean? For all
06:48points close to x star your f of x is greater than f of x star ok. So, this is
06:53the concept
06:54of local minima and for global minima if f of x star is less than f of x for all
07:02values of
07:02x. Now, what happens here? Once again I will come back to this local minima picture and we
07:10will see there are two things that can happen here. When we have a non-convex loss function
07:17and we have this kind of a graph. Then, when we are coming down the valley, the you know
07:29the algorithm may get stuck here, because it is .
Comments