00:04is tangent hyperbolic or we call it as tan h activation function.
00:08The function looks like this and it is fx equal to e to the power of x minus e to
00:13the
00:13power of minus x divided by e to the power of x plus e to the power of minus x.
00:18If we see the graph it is like an s shaped curve centered at 0 and if we once again
00:25we
00:25take some values of x from minus 5 to plus 5, we see the fx values vary between minus
00:331 or very near to minus 1 to very near to plus 1.
00:37And if you see the center value which is 0, the fx value also is 0.
00:43That means this function output is a 0 centered output, tan hyperbolic activation function
00:51gives 0 centered output and it is very much used in the hidden layers of a neural network
00:58as it makes the mean of the hidden layers come to 0 or very near to 0.
01:03Now this mean of the data points that are coming out or mean of the values that are coming from
01:10the hidden layer, if they are centered at 0, then what happens is it helps in learning in
01:18the subsequent layers and that is the reason why tan h hyperbolic activation function is
01:25used in the hidden layers because of its 0 centered output value.
01:32Then we come to another highly used activation function which is the softmax activation function.
01:40The softmax activation function looks like this, the mathematical form is like this, I will
01:45explain.
01:46So, suppose the output neuron of a multi class classification problem, we are taking the output
01:56from the neurons of a multi class classification problem.
01:59In multi class classification problem, there will be the number of neurons in the output will
02:04be equal to the number of classes that are present in your training data.
02:08So, suppose you have a training data in which you have 5 different types of output and then
02:15your output layer will have 5 neurons and these are the values that are coming out of the
02:19neurons before the activation function works.
02:23And these values will go through the softmax activation function whose mathematical form
02:29is e to the power of zi divided by summation of j equal to 1 by 1 to k e
02:35to the power of zj.
02:36And when we do this one, for each and every value of this output layers, neurons values, we get something
02:46like this.
02:47And these are basically the probability values.
02:50That means, the probability of this class is the maximum.
02:57So, this is what activation softmax activation function does.
03:02And if we see, it is used in the output layer of a multi class classification problem.
03:08Suppose you have a training data in which you have multiple number of fruits like bananas,
03:15apples, oranges, green grapes and black grapes.
03:20Now, there are 5 classes of fruits that are present in your basket.
03:25Now, during the inference time that means, when you are showing your model some unknown image
03:31like an apples image, the image goes to the multi class classifier.
03:38And the classifier gives the probability of the different kind of fruits that are present
03:44in the basket.
03:45Like probability of green grapes is very low, probability of banana is say 0.02, probability
03:51of apple is 0.9 which is very high probability, probability of black grapes is 0.01, probability
03:57of orange is 0.02.
03:59So, these probability values actually gives you the or gives the predicted class from the model.
04:10Next we come to something called as the rectified linear unit or ReLU activation function.
04:16Now, what is this?
04:17If we see this graph, we will see that the output is in one portion of the graph, in the
04:28negative
04:28portion it is 0 and in the positive portion it is equal to x.
04:32So, the function looks like this, x, fx equal to x if x is greater than 0 and it is
04:390 if x
04:40is less than equal to 0.
04:42That means, what it makes all the input, all the negative inputs to 0 and passes only the
04:52positive inputs to the next neuron.
04:55And this is the default activation function for many type of neural networks because it
05:00is computationally efficient as it does a thresholding operation.
05:05Means, it is telling that if the x value is or the neurons output value is greater than
05:120, then only it will be passed to the next layer.
05:16If its value is in the negative range, then we will not pass it, we will stop it.
05:21So, that makes the computation much more efficient as half of the neurons outputs are not going
05:31to the next layer.
05:33It introduces sparsity in the network as only fraction of neurons activate at a point of time.
05:38So, that is very computationally effective.
05:42And also, it helps in back propagation as its derivative is either 0 or 1.
05:48Because when the x value is in the positive range, then the derivative is 1 and when it
05:55is 0 in the negative range, the derivative is 0 and that helps in the back propagation
05:59function, which is required for weight updation.
06:05Then, there even though ReLU is good and it is the default activation function for lot of
06:13our neural networks, there is a very strong limitation or drawback of ReLU activation function
06:20which is called as the dying ReLU problem.
06:23Now, what exactly is this dying ReLU problem?
06:25Sometimes, neurons can become inactive and only output 0.
06:31And this will happen when very large negative values, they result in 0 gradient leading to
06:39neurons that are never activated and cannot learn further.
06:43So, when the gradient becomes 0 for very large negative values, then the neurons stop learning.
06:50And when the neurons stop learning, we call it as a dying ReLU problem.
06:55Then those neurons become inactive and the learning cannot proceed further.
06:59So, this is a drawback of ReLU activation function and that drawback or the limitation can be handled
07:08or avoided using something called as the leaky ReLU activation function.
07:13Now, what is this leaky ReLU?
07:15If we see here, leaky ReLU if you see for the positive x values, it goes, it outputs the same
07:23x value.
07:24For negative values, it does not take it to 0.
07:27However, it gives sum 0.01 into x.
07:32So, liquid, liquid.
Comments