00:03Hello, everyone. Welcome back to the lecture series on foundations of deep learning concepts
00:09and applications. In our last session, we have discussed about gradient descent algorithm
00:15and we have got to know that for the weight updation in the neural network, we need to
00:21find out the gradient of the loss function. Now in today's session, we will study the
00:28algorithm through which this gradient of loss function is found mathematically and this
00:35is called as the back propagation algorithm in neural network. So, today's my session will
00:40be entirely focused on back propagation in neural network.
00:45To start with, let us have this setup. So, I have an input layer and then in the input
00:55layer, there are input neurons x1, x2. Then I have the hidden layer which are shown here
01:04z1, 1, z2, 1, z3, 1. Now, if we see the edges connecting x1 to z1, x1 to z2 or
01:16x1 to z3,
01:17we are having the weights w11, w12 with a superscript 1. That means, this 1 in the superscript shows
01:26that they are belonging to the first layer ok. So, after this hidden layers z1 calculation
01:34which is nothing but the weighted sum of the inputs, then we have given an activation function
01:40which is the sigmoid activation function here, which operates over z11, z21, z31 and then whatever
01:51is the output, again those goes to the output layer neuron. There is only one neuron in this
01:58output layer and the associated weights are given by w1, 1, 2, w21, 2 and w3, 1, 2. So, here
02:05the subscript superscript 2 means that it is in the second layer and z12 means it is in
02:12the output neuron ok. And then z12 goes through once again through the sigmoid activation function
02:18and then we get the output as y ok. So, this is a toy neural network that I have taken
02:25for
02:25explaining the back propagation algorithm ok. And we know the loss function is actually the
02:34difference between the actual value and the your predicted value correct. So, the actual
02:42value minus y if y is your predicted value and t is if it is your actual value then t
02:48minus
02:48y is your loss ok. Now, we will have this toy neural network schematic of this one here in
02:59all throughout the slides and then we will do the mathematical calculation side by side.
03:03For simplicity, I have taken the loss function here. So, I will just write it here. So, I have
03:15taken the loss error is equal to the mean squared error which may change in your actual application.
03:31Of course, it will change here for simplicity I have taken. And which is nothing but 1 by n,
03:35we all know summation over i equal to 1 to n and then we have y i minus y i
03:43hat square ok. So,
03:47and one half term we will add here for mathematical convenience. And what we know here is 1 by 2
03:55n,
03:55now it becomes then summation i equal to 1 to n. Your y i is nothing but your ground truth
04:02which
04:02is your t minus your predicted value which is your y square ok. This equation we will keep as equation
04:10number 1. Now, what is n? n is the number of observations we all know and half I have told
04:17that we have taken for some mathematical convenience. Now, according to gradient descent,
04:30gradient descent we need to calculate the gradient of the loss function right. So, gradient of L with
04:41respect to L which is a function of W and B and this we require for weight updation ok for
04:47weight updation.
04:55Now, back propagation algorithm is a structured way to get this gradient of L with respect to your weights ok.
05:07So, what we will write here is gradient of L with respect to the weights is equal to partial derivative
05:15of L with respect to L with respect to the weights when L is a function of both weight and
05:22bias.
05:23Now, if you see in this figure there are lot of weights ok. So, we have w111 then w121 w221
05:34and then
05:35we have w112 w212 w312. So, there are a number of weights are associated with this network ok. So,
05:43in order to calculate the gradient of the loss function here we need to calculate the partial
05:53derivative of the loss function with respect to all the weights ok. So, and also before I start into
06:02the calculations we will see that as loss is calculated here. So, it is you know the loss comes
06:10here correct loss equal to your target minus your whatever your predicted 1 by n summation whatever
06:17ok. So, as it is calculated here what we do is we will try or we will do the propagation
06:28of the error
06:29from the last part of the neural network towards in the backward direction ok. Now, we will try to find
06:38out
06:38what is the gradient of the loss function with respect to this weight w112 ok. So, we will calculate that
06:48one. So, let us calculate let us calculate partial derivative of L with respect to
Comments