00:03So, this is a drawback of ReLU activation function and that drawback or the limitation
00:10can be handled or avoided using something called as the Leaky ReLU activation function.
00:17Now, what is this Leaky ReLU if we see here, Leaky ReLU if you see for the positive x values
00:25it goes it outputs the same x value.
00:27For negative values it does not take it to 0, however, it gives some 0.01 into x.
00:36So, Leaky ReLU if the function is x then it goes through Leaky ReLU function and then it
00:44gives x if x is greater than 0 and it is equal to 0.01x if x is less than
00:52equal to 0.
00:53That means, it is not allowing it to go to 0 even in the negative values of x.
00:59So, this way the dying ReLU problem can be overcome.
01:06Till now, we have seen all the various types of activation functions and now we will go
01:13into the loss functions which decides how much your predicted output is different from your
01:22actual or ground truth output.
01:25So, what is this loss function?
01:28Loss function measures how well a machine learning models predicted output match the true output
01:33level.
01:36The loss function is used to optimize the model by minimizing the loss which means that the
01:42model makes fewer mistakes on the training data.
01:44So, the main goal of any neural network is to optimize the model in such a way that the
01:51loss is minimized.
01:53So, if we have input data and it is going through the neural network and it is giving some output
02:00in the output neuron, we have some predicted output which is y pred and then we have some
02:06the true output which is the ground truth and the difference between them is the loss which
02:13is you know function of the predicted output as well as the ground truth value.
02:18So, now we will see what are the different kind of loss functions that are used for different
02:23kind of applications.
02:25We will first see what is the loss function for regression problem.
02:29For regression problem, we have something called as the mean squared error.
02:33What is it?
02:34Mean squared error equal to 1 by n summation over i equal to 1 to n yi minus y hat
02:42whole square
02:43where yi is the actual value, yi hat is the predicted value and n is the number of observations.
02:52There is another error term which is called as the or error function which we call as the
03:01mean absolute error and this mean absolute error is equal to 1 by n summation over i equal
03:09to 1 to n the absolute value between yi and yi hat whereas, yi is the actual value, yi hat
03:18is the predicted value, n is the number of observations.
03:21Now, if we see in both the equations, we see the error is either taken as a squared value or
03:30it is an absolute value.
03:31Now, why it is taken as a squared or an absolute value in machine learning, what is the reason
03:37for that one?
03:39To understand that one, let us try to understand what kind of values can come in error.
03:46Suppose I am predicting the price of a house and the actual price is say 82 lakhs, the predicted
03:57price can be 100 lakhs or it can be 60 lakhs.
04:03That means, yi minus yi hat can either be positive or negative.
04:10Now, if there are equal number of data points which are having positive error to the number
04:17of data points which are having negative error, then these two will cancel each other and we
04:23will get that there is no error which is not actually the case.
04:28So that is the reason to avoid this, we take either the squared error or we take the absolute
04:35value of the error.
04:41Now this mean squared error is very sensitive to outliers.
04:46Now what do I mean by outliers?
04:48Let us have a look.
04:49If we have data points like this in your training data set and there is one data which is lying
04:57very far away from the entire group of data points, then we call this one as the outlier.
05:04Now, we will see with a small example how the mean squared error is sensitive to outliers.
05:12So, suppose my error values are 0.1, 0.5 and 0.3, when we do the MSC calculation mean
05:28squared
05:28error calculation, we will see MSC is equal to three data points I have taken.
05:34So, 1 by 3 into 0.1 square plus 0.5 square plus 0.3 square that is equal to
05:501 by 3 0.01 plus
05:560.25 plus 0.09 which is equal to 1 by 3 if we do the addition 0.35 which
06:09is equal to 0.116.
06:15Now, these are the three error values, ok.
06:20That means, yi minus yi hat is for the first observation it is 0.1, then for the second
06:27observation it is 0.5 and for the third observation it is 0.3.
06:31Now this is the value for mean squared error.
06:33Now, if we do this calculation for mean absolute error, what do we get?
06:38It is 1 by 3 into 0.1 plus 0.5 plus 0.3 which is equal to one third
06:55of 0.9 and that is equal
06:58to 0.3. Now, you see here in this case your mean absolute error is greater than your mean
07:08squared error. Now, take the case when the errors are large. So, suppose we take the values of
07:17errors, the errors are say 1, 5 and 3 for three data points. So, if you do the MSC calculation,
07:27what will come? 1 third 1 square plus 5 square.
Comments