00:03Hello everyone, welcome back to our lecture series on foundations of deep learning concepts
00:09and applications.
00:11In our last session, we have discussed about the multilayer perceptrons, they are working
00:17and then we have touched upon little bit on shallow and deep neural networks.
00:22And, we have mentioned about activation function and loss function in the last session.
00:29However, in today's session, we will go in detail about activation function and loss
00:36functions which are an integral part of multilayer perceptrons.
00:41So, we will have a look into the contents of today's session, we will start with introduction
00:47to activation functions, what exactly they are, then type of activation functions, then
00:53we will look into loss functions and its types and then we will touch upon how to design a
00:59neural network.
01:01So, we will start with what is an activation function.
01:06In feed forward propagation, the activation function is a mathematical function that is
01:12applied to a neurons output and it decides whether that output should be passed to the next layer
01:19or not.
01:20So, basically, it acts as a gate between one layer neurons to the other layer neurons because
01:27it decides whether that neuron should be fired or not based on the information that is coming
01:32from the neuron.
01:34It transformed the summed weighted input from the node into an output value which is to be
01:41fed to the next hidden layer or as output.
01:44So, there is a transformation that happens when activation function works on any input.
01:50These are the backbone of neural network enabling them to capture non-linear relationship in data.
01:56So, this is the most significant part of activation function.
02:00It captures the non-linear relationship in any data.
02:03Now, if we see real world data like images, text, speech, this kind of complex data, they
02:12always have a non-linear relationship between the input and the output.
02:17If we are not able to capture the non-linearity in the relationship, then we will not be able
02:22to predict it properly.
02:23Without non-linearity, a neural network would only function as a simple linear regression
02:29model.
02:30So, how many ever hidden layers we put, it will do only a linear regression models work.
02:36It will not capture any non-linearity in the data.
02:40So, that is the main function, why activation functions are used in neural networks.
02:46And as we have seen in the last lecture itself, so this is the place where activation function
02:53works.
02:54So, in the hidden layers, after the weighted sum of the inputs are done and it is added to
02:58the biases, then activation function works on that input.
03:02And once again, this happens in all the hidden layers and also in the output layer.
03:07So, that is the place where activation function works.
03:12Now, we will have a quick look into the different kinds of activation functions.
03:19To start with, there are two types of activation functions, linear activation function and non-linear
03:25activation functions.
03:27Now, there are various kinds in non-linear activation functions like sigmoid activation function, tanh
03:33activation function, softmax activation function, relu activation function and leaky relu activation
03:40function.
03:41Now, one by one, we will look what are these things.
03:44So, we will start with linear activation function.
03:47It is also known as an identity function and the graph, if you plot fx versus x or fz versus
03:55z, you see that fz have the same value as z.
04:02So, this is basically whatever we are passing through the function, the same value is coming
04:07as the output.
04:09So, the range of possible outcomes here can vary from minus infinity to plus infinity and
04:15it is used in the output layer for regression where we do not want any transformation in the
04:21output value per neuron.
04:23So, these are basically used in the output layer of regression problems.
04:29Then, we start with the non-linear activation functions which are very much used in neural
04:33networks.
04:35So, we will start with the non-linear activation function which is the sigmoid activation function.
04:40And if we see, this is the graph for the non-linear activation function and the mathematical
04:46form is fx equal to 1 by 1 plus e to the power of minus x.
04:50If we see in the right hand side, there is a table which gives for different values of
04:54x how the fx is changing.
04:57If we see for negative values of x, this fx value is very near to 0 and for positive values
05:05of x, it is very near to 1.
05:10And if we see a very high negative value and a high positive value, we see for very high
05:17negative value, the function gives a value which is very very close to 0 and for a high
05:24positive value, the function gives some value which is very very close to 1.
05:29That means, whatever be the values of x, after sigmoid activation function works upon those
05:36values, we get the output in between 0 and 1.
05:40Now, this 0 and 1 values basically they are the probability values that are associated with
05:47any kind of classification problem.
05:50So, this activation function, specifically the sigmoid activation function is used for binary
05:58image classifier, where there is only one output neuron is there and there we want whether an
06:08item or an image belongs to one class or not.
06:10Like for example, here I have given there is a dog's image and there is a horse's image.
06:18Now, suppose I want to classify this image, these 2 images whether they are dog or not a dog,
06:25then I will send it through the binary image classifier and we will get the output.
06:30For a dog's image we will get the output as a dog and for the horse's image I will get
06:36an output as not a dog.
06:37And this dog and not a dog is basically coming from the probability value that is coming out
06:44of this activation function which is the sigmoid activation function.
06:50So, the sigmoid activation function is used in the output layer for binary classifier.
06:56Then we come to the next type of non-linear activation function which is tangent hyperbolic or we call
07:02it as tan h activation function.
07:05The function looks like this and it is fx equal to e to the power of x minus e to
07:10the power
07:10of minus x divided by e to the power of x plus e to the power of minus x.
07:15If we see the graph, it is like an s shaped curve centered at 0.
07:20And if we once again we take some values of x from minus 5 to plus 5, we see the
07:26fx values
07:28vary between minus 1 or very near to minus 1 to very near to.
Comments