- 21 hours ago
Standard deviation can sound intimidating, but it’s a practical way to understand how spread out data is—and this beginner-friendly statistics lesson makes the idea approachable. Explore how standard deviation relates to the mean and bell curve, and how values within one, two, or three standard deviations can help reveal common patterns and potential outliers.
Follow a hands-on Python example using order values: calculate deviations from the average, square them, find the variance, and take the square root to get standard deviation. The lesson also compares a manual calculation with Pandas and uses a visualization to show how the results help identify unusual data points.
A useful introduction to standard deviation, variance, descriptive statistics, and data analysis with Python for students, aspiring data analysts, and anyone learning statistics. Follow along for a practical statistics tutorial, or use it as a clear refresher on measuring data spread and spotting possible anomalies.
Follow a hands-on Python example using order values: calculate deviations from the average, square them, find the variance, and take the square root to get standard deviation. The lesson also compares a manual calculation with Pandas and uses a visualization to show how the results help identify unusual data points.
A useful introduction to standard deviation, variance, descriptive statistics, and data analysis with Python for students, aspiring data analysts, and anyone learning statistics. Follow along for a practical statistics tutorial, or use it as a clear refresher on measuring data spread and spotting possible anomalies.
Category
📚
LearningTranscript
00:00What's going on everybody? Welcome back to another video. Today we are continuing our
00:04statistics series and today we're looking at standard deviation. Now standard deviation was
00:09always something that kind of scared me if I'm being completely honest. It was being thrown
00:14around in like meetings and stuff and I was personally kind of intimidated. I'm like that
00:18looks scary. But when I actually started using it I'm like oh this actually makes a lot of sense and
00:24is pretty easy. But it's just kind of a scary term I think and so maybe you're like me maybe
00:30you're
00:31not. Maybe you think I'm you know overreacting a bit probably. But let's look at what standard
00:36deviation actually is. So what we're looking at right here is a classic bell curve and right in
00:40the middle we have our average or our mean. Now at the bottom we have different numbers. We have
00:47different standard deviations. We have plus one plus two and plus three and that little symbol right
00:52next to it is just the greek letter sigma which kind of stands for standard deviation. This is all
00:57just how spread out the data is from that center value. So that plus one sigma is the first step
01:05of
01:05standard deviation. This is where 68 percent of our value is falling within one standard deviation of
01:12that middle value. Then we have another step and so we have our second standard deviation. That's where
01:1813.5 percent of all of our values fall within that range. And then lastly we have our third step
01:25and
01:25sometimes you can even have more. It's possible. But here we have two percent. So only two percent are
01:31falling between the second and the third standard deviations of our data. Now let's get hands-on and
01:37see how we can calculate these levels of standard deviation. Now if you're following along this is the
01:42exact same notebook as we were working on before you can find this data set down below. You don't
01:49have to write out everything we're about to do by hand. I think following along is perfectly fine.
01:54Again we're using python for this and I'm going to walk you through each step of determining the
01:59standard deviation. The first thing we're going to do is just look at our data set. We're working
02:03with a different data set here. Here we have the order id, customer id, order date, the category,
02:08how many items they ordered, their order value, their shipping costs, and days to deliver. Now the
02:15one that we're going to be working with in this lesson is just order value. Keep it really simple.
02:20Now all I've done and this is just some pre-work is I've determined our values are going to be
02:25our
02:25orders, order value. So it's just order value. That's the data that we're going to be working with
02:30and then the average is those values and we're taking the average. So all we've determined is the mean.
02:36So if I run this right here, our average is 106.18 essentially. That's all we have. We have our
02:45average which is 106.18. Now what we're going to do is we're going to start step by step getting
02:54this
02:54standard deviation. So let's start with our very first point. So step one is we just need to find
03:01out. I'm actually going to write this out. Step one, find out how far each order, so every single
03:09value, how far that is from the average. That's what we need to do first. It's really easy. So we're
03:16going to call this deviations. We're going to say that's equal to the values minus the mean.
03:24So we're taking each individual value and we're subtracting the average. So let's actually pull
03:31this up. We'll see what this looks like. So let's run this. So our deviations are going to look like
03:37this. Some are going to be negative. Some are going to be positive. So if the value right here is
03:431,000
03:44and then we're subtracting 100 or whatever that number was, then we're going to have like 900. But if this
03:50value is 10 and we're subtracting 100, we're going to get like minus 90. So some are positive or above
03:56that average line and some are negative or they're below the average line. That's what it looked like
04:01in our bell curve before. Some are above the average line, some are below the average line. So we have
04:06our deviations right here. Now, step two, and let me write it out for us. Step two is just to
04:14square
04:16the deviation. Now, why do we do this? It's because we actually want to get rid of these
04:20negatives. So let's go ahead and square this. We're going to take our deviations and we're going to
04:26square it. We're just going to do star star two, which means to the power of two. If we run
04:32this,
04:32it's going to look like this. So we're just squaring them. Now that's in scientific notation,
04:37but that's no big deal for what we're about to do. So we'll call this squared underscore
04:43deviations. And there we go. Now let's move on to step three. Let me give us some room here.
04:48So now we're going to do step three. And step three is just to average the squared deviations,
04:57which we just did. This is what's called variance. Think of this as like the average squared distance
05:04from the mean. And don't worry, it's going to make a large value. We know this. This variance takes
05:10all of these numbers, whether it's high, low, everything. And then we're putting that into
05:15one single number, the average of that number. So we're taking the squared devs right here
05:21and we're getting the mean. So we're saying dot the mean. That's our average. Let's run this.
05:26So when we square this number and we get our squared devs and we look at the average of those
05:33squared
05:33numbers, we're getting 4,626. That's a big number, but don't worry. We squared it initially,
05:40to basically get rid of those negatives in a way. But now we're about to use the square root
05:45to bring it back down to its original units or, you know, the dollars from right down here.
05:52So that's step four. We'll do step four. We're going to square it. That is just bringing it back
06:01down. Again, we're not doing anything crazy here. Let's call this our variance. We're going to say
06:06variance. I'm going to run that. So now we want to take that variance and get the square root of
06:13it.
06:14So we want to actually call something called numpy. We're going to say the square root. That's just,
06:19numpy is just a library that we're reading in for the statistics lesson. And we're going to get the
06:26variance right here. And let's take a look at it. So this says our square root of our variance is
06:3168.
06:31So this is going to be our standard deviation manually done. Now I'm about to go a little bit
06:37further in depth into this and we're going to look at some visualizations. But let's run this.
06:44If we come right down here, and again, we calculated this by hand, but we could just use the function
06:51for this. Let's do values.std. That's standard deviation. If we run this, and I need to do
06:59std, I did stf. If I run this, you'll notice that this is 68.01839. And I'm going to call
07:07this
07:10std pandas. So if I come right here, I'm just going to say print std manual, and I'm going to
07:18do the
07:19same thing for the pandas. You'll see how close they are. And I can just copy it over. It'll be
07:28a
07:28little bit faster. All right, let's run this. So now you will see that our two numbers are incredibly
07:34close. I mean, to basically two decimal points, they're almost the exact same thing. Now this
07:40function may be doing another slight variation of this, but it's essentially getting us to the same
07:47place. All this means, right, we calculated this. What does this mean? That this value, we're going
07:55to place it against this standard deviation and see how many steps away from the average
08:00is this value. So if I print out, and I'm going to do print the mean here. If our value
08:09is 106.17,
08:11and we add 68 to it, that gives us about 174. So this 194 is going to be past one
08:19standard deviation
08:20from the mean. Now I'm going to pull something over really quick. You don't have to write this
08:25out. I just wanted to visualize it. Let's go down here, and let's run this. Now what we get right
08:31here, and I'll walk you through the code really quick. It's not anything crazy complicated, but all
08:36we're really doing is we're saying, okay, if it's within one standard deviation, we're basically
08:42saying, and you can see it right here, that if we take the average and add 68 to it, it's
08:46$38 to $174.
08:49This is our range for it to be within one standard deviation, and we're just calculating that out
08:55right here. And then we're saying, okay, take away another 68. Boom. Took away another 68. Add another 68
09:03on this end. And boom. That's within two standard deviations. Now there is some theory to this,
09:09and there are some, you know, generalized rules, although these aren't 100% accurate, and we can
09:14actually see that over here, is that in general, the theory is that about 68% of your values are
09:21going
09:21to fall within one standard deviation, 95 within two standard deviations, and 93 within three standard
09:30deviations. If most of our values are in this green section, those are really common values.
09:37If we get values that are, you know, in this orange, that's, you know, fairly normal. It's going to be
09:41most of our values. But then even here, we have a few values that are going to go over that,
09:47and those
09:47are going to be, you know, kind of our extreme ones. Because if we have thousands of orders, and 99
09:53.9%
09:55or however, 99.7% falls within these red, and it's outside of that, they may be something
10:01of concern. It could technically be an outlier, depending on, you know, your data. So in practice,
10:08when we're using this within data, this is really useful. Now, you don't have to calculate it by hand
10:12like we did in here. You can usually just do this. But understanding how it actually works behind the
10:18scenes is really important, because then you can start looking at your data in a different way. You can
10:22say, okay, why do I have this one value way over here, when that's like an eighth standard deviation
10:29apart? Let's go dig into that data and see what's going on with it. Maybe it's a legitimate customer,
10:34maybe it's some error in the data. But you can use these standard deviations to identify patterns in
10:39your data, and maybe anomalies or outliers in that data as well. This is also used for other things.
10:45And we're actually going to be getting into that later on in this series, how we use standard
10:50deviations in different parts of statistics. If you enjoyed this lesson, be sure to like and
10:56subscribe. I also have an entire statistics course on Analyst Builder. I will leave a link down in the
11:01description. And with that being said, I will see you in the next lesson.