00:03with respect to one of the parameters keeping the other constant that we call as partial
00:08derivative ok. So, if we take the partial derivative of L with respect to W then we
00:15get 6W ok. Similarly, if we take the partial derivative of L with respect to B keeping
00:22W constant then we get 2B. Now, what is this gradient we understood what is normal derivative
00:29partial derivative. Now, we have to understand what is this gradient. Gradient is actually
00:34a vector which is which comprises of the partial derivatives of the function with respect to
00:42its parameters with respect to all the parameters. So, here partial derivative of L with respect
00:47to weight W and partial derivative of loss function with respect to the B bias ok both
00:54are there. And it comes as a vector and that is called
00:59as the gradient. Now, suppose I have a function with 10 variables, in this case the number of
01:05variables were 2. Suppose we have a function of 10 variables, what will be the dimension
01:10of the gradient vector? It will be a 10 dimensional vector ok. Because we have to take the partial
01:15derivative derivative of the function with respect to all the variables that are there. Now, we
01:23will see a little bit you know the intuition what exactly gradient descent algorithm is trying
01:29to do. Suppose we have this x, y, z coordinate and we have weights in one coordinate bias in another
01:39coordinate and the z axis is your loss function ok. And suppose I have this you know free-flying
01:47you know surface which is a loss or error surface. Now, my aim of using this gradient descent algorithm
01:58is to minimize the loss function that means, I need to change the weights and biases in such
02:05a way. So, that the loss function goes to its minimum value ok.
02:10Now, we can randomly select next point in the weights and biases plane to reduce the loss.
02:17Because we do not know which direction, should I increase the weight, should I decrease the
02:21weight, should I increase the bias, should I decrease the bias, I have no idea. So, it is
02:25just a randomly we are trying to reduce the loss ok. Now, if we use gradient descent algorithm,
02:33we get a structured way to move in a certain direction, which will definitely reduce the
02:40loss ok. So, gradient descent algorithm actually gives me a structured way to understand in
02:48which direction I should move, so that my loss gets reduced. Now, we will see how this
02:58gradient descent algorithm is working ok. So, let theta be a vector of the parameters of
03:04the neural network. Now, what are these parameters? As I told, whenever we tell about the parameters
03:09of a neural network, we speak about the weights and the biases ok. So, let theta be equal to
03:16is a vector of parameters of the neural network that is weights and biases ok. Now, we start
03:22from a random value of weights and biases and calculate the loss function. So, we have seen
03:27previous in our previous discussions also, how to calculate the loss if you know the weights
03:34and biases value ok. So, from there we first we take some random values of weights and biases
03:40and calculate the loss function. Now, what we need to do? We need to update these weights
03:46and biases in an iterative way, so that the new loss function is lesser than the old loss function.
03:53That is what we want to do right, because at the end we want to reduce or minimize the loss
03:58function ok. So, that our predicted output is almost you know same as the actual output.
04:07Now, let the change in theta be delta theta. So, you have theta which is a vector of w and
04:16b and you
04:18are changing theta in such a way the change is delta theta, where delta theta is nothing
04:23but a change in the weight delta w and change in the biases delta b. So, delta theta also is
04:29a vector
04:29vector of the changes in the vector diagram. Suppose this is the direction of my theta and this is the
04:41direction of my you know delta theta, this is the change. Now, if I add them theta plus delta theta,
04:49then we get something using the parallelogram law of vector addition. So, this is the direction where
04:55my theta nu will you know reside ok. Now, suppose I we all know that for any vector we have
05:07direction
05:08as well as magnitude. Now, suppose I do not want theta to change by a huge amount. So, my delta
05:17theta,
05:18I do not want to be a drastic change, I want it to be a small amount. So, in that
05:24case what can
05:25we do? We can multiply this delta theta with a very small value which we call as eta and in
05:33this figure
05:34you can see this is my thetas direction, if this is my eta into delta theta, then this is my
05:40theta nu.
05:41So, what I have done is here previously my the original parameters of the network is embedded in
05:49theta which is w and b. Then I try to change the parameters little amount which is my delta theta
05:57which is once again a vector of delta w and delta b and then I take a small magnitude of
06:05that delta
06:06theta by multiplying with by it with a factor eta and then taking the summation which is my delta which
06:15is my theta nu. So, my theta nu equal to theta plus eta into delta theta where theta nu is
06:22a function of
06:23my nu weight and nu bias and theta is a small value. Now, my next question is what in which
06:33direction
06:34should delta theta be ok. Should I go up, should I go down, which direction means weights can be increased,
06:40can be decreased, can be decreased same for biases which direction should I go. Now, suppose my old loss
06:48not suppose we have already told the loss function here is l which is a function of theta and the
06:55nu loss
06:55function is a function of theta plus eta into delta theta ok. Now, just for convenience let us call
07:03this delta theta equal to u, then the loss function l theta plus eta into delta theta I can write
07:10as l of
07:12theta plus eta into nu. There is another extra bracket that has come, I have to remove that one from
07:19there.
07:29So, just striking it out ok. Now,
Comments