00:030.775, 0.116, 0.039 and 0.070 where we see the output probability for you know the classes
00:12are given here the dog class is the highest one. And if we take the difference between
00:19the output probability values P and the true ground truth where is the dog class is 1 and
00:25the others are 0. So, the difference between them gives the categorical cross entropy loss.
00:31So, suppose if there are three classes and the true class is the second class that means
00:37the true labels are 010 and suppose the predictions are 0.1, 0.7 and 0.2 in that case
00:44the loss is
00:45minus summation i equal to 1 to c yi into log yi hat and if you do it and we
00:52will get you
00:52know like this calculation you can pause and see the calculation for your own benefit and
00:57that is how the categorical cross entropy losses are calculated.
01:01Then loss function for multi-label classification, for multi-label classification with multiple
01:08classes binary cross entropy loss is computed for each of the classes independently and then
01:13they are summed over all the classes to obtain the total loss. So, you have, so these are
01:19images where you have a cat, a dog and a butterfly and this is the image where only the cat
01:25and
01:25the dog are present, the butterfly is missing. Now, what happens is for each, these are each,
01:30each one of them are having a label. So, cat label is there, dog label is there and the
01:35butterfly is there. So, we use sigmoid activation function for each output node. So, there will
01:40be three output nodes here and we use sigmoid activation function for each of these output nodes
01:45and use binary cross entropy loss for each of these classes and then sum over all the classes
01:51to obtain the total loss. So, we will have a look into cool back labeler divergence loss
01:57which we call as the KL divergence loss. So, what exactly is this KL divergence loss? In
02:02machine learning especially in models involving neural networks, KL divergence is often applied
02:07in task where the models outputs are probability distributions. Example in generative models, so
02:13where the output is not a probability value but a distribution. So, in that case we use
02:18KL divergence loss. So, we will have a look how this difference in the probability distribution
02:25looks like. Suppose, I have a graph x is the value in the x direction and probability density
02:31in the y axis and I have two probability distributions. The red one is P of x and the blue
02:42one is Q of
02:43x and then we see the difference between you know whatever is there marked in red that is the KL
02:50divergence between P and Q distribution and whatever is marked in blue here that is the KL
02:56divergence between Q and P ok. And now, we will see by minimizing KL divergence the model ensures
03:03that its predicted probability distribution closely aligns with the true distribution. That means,
03:09we would like to predict or the models prediction of the probability distribution should match the
03:16actual probability distribution. So, from that perspective the KL divergence will give how much
03:21difference in probability distributions of the predicted one and the original one is.
03:27Now, in KL divergence loss, the formula goes like D KL between P and Q is summation over i,
03:34P i log P i by Q i for discrete values of i and it is an integration P i
03:39log P i by Q i for continuous values
03:42of i and where KL divergence between P and Q is between the it is the Kuhlbach-Lebeler divergence between
03:52two probability distributions P and Q, we are trying to get the difference between them and P i is the
04:00probability of an event i under the distribution P, Q i is the probability of an event i under the
04:05distribution Q. And these are the rules that is D KL P or the KL divergence between P and Q
04:14is always
04:15greater than 0, KL divergence between P and Q is equal to 0 only if P equal to Q and
04:21KL divergence between P
04:22and Q is not equal to the KL divergence between Q and P. And then quickly we will look into
04:28how to design a
04:29neural network architecture. So, in the neural network we have input layers and the number of
04:35input neurons is equal to the number of features in the data set. So, suppose you have a data set
04:40like
04:40this where sepal length, sepal width, petal length, petal width they are the features and the different
04:46classes of flowers are the you know that is the target variable. So, the number of input neurons in
04:53this case will be 4. In hidden layers, how many neurons we should put that depends on the task
05:01that we are doing. So, it will be few layers for simple task and many layers for a complex high
05:06dimensional task ok. And then we come to the output layer. If it is a binary classification problem,
05:13then it is one neuron if like detection of spam or not spam emails. If it is a multi-class
05:21classification,
05:22the number of neurons equal to number of classes in the multi-class classification. In this figure,
05:27there are 3 classes. So, 3 neurons will be there. If it is you know multi-label classification,
05:33number of neurons will be equal to number of labels for the number of labels that are present
05:37in your data set. Like here there are 3 labels, the cat, the bird and there is a flower. And
05:45there are
05:453 classes cat, bird and flower. And this is if that only the cat is present, it is 1 0
05:520 label.
05:53It is if only the bird is present, it is 0 1 0. Similarly, if cat and bird both are
05:59present,
05:59it is 1 1 0. And if the bird and flower is present, it is 0 1 1. And here
06:08when all the things are present,
06:09it is 1 1 1 level. Now, we have come to almost to the end. There are some examples of
06:16real world
06:17problems where we use neural networks like handwriting, handwritten digit recognition.
06:22There is a data set called as the MNIST data set where you have to recognize 10 digits 0 to
06:309. So,
06:30you have 10 output neurons and the activation function is softmax and the loss function is
06:35categorical cross entropy. Then spam email detection output, it is a binary classification problem,
06:40which is having 1 neuron. Activation function is sigmoid and the loss function is binary cross
06:46entropy. Image tagging like cat, dog, bird can co-exist that is number of neurons is 3 1 per
06:52label. Activation function is sigmoid and binary cross entropy loss is the loss function. Stock price
06:58prediction output neuron will be 1. Activation will be a linear activation function and the loss
07:04function should be mean squared error loss. So, with this one, we come to the end of today's session.
07:12Before I come to the summary, I would like to give some reference books for you all to, you know,
07:19have them as your reference or text books. So, the first book is, you know,
07:26Bishop CM and Bishop H Deep Learning Foundations and Concepts from Springer Nature and the other book is
07:35go through the fall, which is, you know,
07:35like, the fact you will get the more important to the process, which is, you know,
07:36the same on the slide. The first book may be made of materials like this,
07:36you know, theIME gene as well as the other group isério literally,
07:36because it is always a problem. So, the second book is, you know,
07:36the second book, is the same-and-and-and- completed night,
Comments