00:04Now, just for convenience let us call this delta theta equal to u, then the loss function
00:10l theta plus theta into delta theta I can write as l of theta plus eta into nu. There
00:17is another extra bracket that has come. Now, according to Taylor series, if we have a function
00:41like l which is a function of theta plus eta nu, u that can be written as a series like
00:49this, which is equal to your l of theta plus eta into u transpose. What is my u? u is
00:56a
00:56vector, which is nothing but your delta theta. And inside delta theta, what was there? Delta
01:03theta was having the components or having the you know the terms delta w and delta b ok.
01:11So, u transpose then the gradient of l plus eta square by 2 factorial into u transpose gradient
01:23of the gradient of l and so on the series goes on. Now, if we have eta to be a
01:34very small
01:34value, then eta square eta cube, these will be extremely small. In that case, what we can
01:41do is we can neglect these terms. And what we want to keep as an approximation is l theta
01:48plus eta u is equal to l theta plus eta into u transpose into gradient of l theta ok. So,
01:58u transpose into gradient, this is a vector multiplication and eta is a small value ok. Now, this is my
02:09new
02:10loss. Now, this is my new loss and what I want my new loss should be lesser than my old
02:19loss, which is
02:20my l of theta. So, the condition that needs to be satisfied here is l of theta plus nu u
02:26should be
02:27less than l theta ok. For that one, what should we have here? This term should be negative ok. Now,
02:36we want this term to be maximum negative because if it is maximum negative, then only my new loss
02:44function will be much lesser than my previous loss function ok. Now, how do we make this term negative,
02:52that we have to see. This is the term in question now. Here, eta cannot be negative. Now, why eta
03:00cannot
03:00be negative? If we go back and look into what exactly my eta was, eta was basically nothing but the
03:11small
03:12change that I was doing in the direction. So, this was my thetas direction and this was my eta into
03:23delta
03:23theta. Now, if I make eta negative, the direction will change which I do not want ok. So, this I
03:31do not want. So,
03:32I cannot make eta to be negative. Then what am I left with? This is a vector multiplication between
03:40the transpose and the gradient of l ok, which is a dot product between two vectors ok. Now, this dot
03:51product
03:52can become maximum negative when the angle between u and gradient of l is equal to 180 degree. So, that
04:02is the place where my dot product becomes negative, maximum negative. So, according to gradient descent,
04:10if we want to reduce the loss, we should move in a direction which is opposite to the gradient of
04:18the
04:18loss function ok. So, this is one of the very important points of gradient descent algorithm that
04:25if we want to reduce the loss, we should move in a direction which is opposite to the direction in
04:30which the gradient of the loss is ok. Now, we will see after gradient descent how my weights are updated.
04:38So, suppose we have the weights at a time t plus 1, then that is equal to w t at
04:46the time t the weights minus eta gradient of l with respect to you know weights that is my new
04:57weight.
04:59So, at time t plus 1 if I see I will have my previous iterations weights minus there is a
05:10factor eta which
05:11we will look into what exactly it is what it is called as and what are the values that we
05:17should
05:17take for eta and then we have the gradient of loss function ok. Similarly, the for the change in biases
05:25also this is the similar equation we can follow and we are moving in a negative direction. So,
05:32we have this minus term negative to the gradient of the loss function ok. Now, if we put these two
05:42in a vector w at the time t plus 1, the bias at the time t plus 1 equal to
05:50the weight at time t,
05:51the bias at time t these are two vectors minus eta into the gradient ok with respect to weights and
06:00biases at time w t and b t sorry it should go up at time w t b t ok.
06:13Now, my final equation is theta t plus
06:171 equal to theta t minus eta gradient of l l theta where what is my theta t plus 1
06:28theta t plus 1 is
06:29coming from here which is weights at the time t plus 1 and biases at the time t plus 1.
06:34So,
06:35this entire you know thing is coming in theta t plus 1 whereas, your theta t is nothing but this
06:40one
06:40omega w t and b t that is the weights and biases at the time step t. So, this is
06:46my theta t and then
06:47eta which is the learning rate and then this vector is coming as gradient of l with l theta ok.
06:54So,
06:55this is the weight updation or the parameter updation equation in gradient descent algorithm.
07:01So, this is how the gradient descent algorithm works ok. Now, so, if I want to show the algorithm,
07:10so we start at t equal to 0 maximum iterations we take as we take it usually a very high
07:17value of 500
07:17or 1000 and while t is less than this max iterations keep doing you know updating the weights using these
07:27equations and then you know when the number of iterations are reached you come out. So, that is the
Comments