- 2 hours ago
Category
📚
LearningTranscript
00:00Hello, everyone. Welcome back to FabTrader, where we talk all things trading, investing, and building wealth.
00:04Today's video is something that's been in popular demand from many of you in the community,
00:08especially those of you who are just starting in algo trading or learning Python.
00:11One of the first questions that I get asked is, how do I download NSE historical data automatically,
00:16both EOD as well as intraday? By intraday, I mean the 1 minute, 5 minute, 15, 30, 60 minutes type
00:21of data
00:22for stocks, futures, and option symbols. For most experienced algo traders, they get this data
00:27directly from their broker APIs. However, for people who are starting new and don't have a broker
00:30API setup, they need this type of utility. Well, the good news is, per your request, I've created
00:35a simple Python utility that can download the NSE historical data. So whether you're backtesting
00:40strategies or just want to analyze price action, this tool gives you clean, reliable market data
00:44straight from the official NSE website, and it's completely free as well. In this video, I'll walk
00:48you through the utility, its code, and I'll show you also how to use it step by step. You can
00:52copy this
00:52code for free and use it as you please. So stick around and smash that like button if you haven't
00:57done so already.
00:59If this is your first time here, welcome. My name is Vivek and I'm a financial independent
01:02algo trader. This channel is all about building a community of algo traders. We discuss everything
01:06about algo trading using Python, building and backlisting trading strategies, market updates,
01:09and much more. Please do visit our community website fabtrader.in. Also, do check out my
01:13other YouTube channel, Fab Wealth, where I talk about my own financial independence journey
01:17and share tools, methods, and strategies that help me achieve my financial freedom. Thank you.
01:25A few days back, I had written this blog article. I also did a video on this where I had
01:29created
01:30an NSE utility in Python, using which I think you can do a bunch of activities like, for
01:35example, tracking live price. You can download bar copies. You can extract corporate actions,
01:39corporate announcements, a whole lot of things that you can basically fetch directly from the
01:44NSE website. And this is that video. If you haven't seen this video or read this article, I'll
01:48provide the link in the description. So take a look at it. The video and the blog article
01:53was received quite well. A lot of people told me that they are now starting to use this on
01:57daily basis, and they found it very useful. And as part of that video, comments and people
02:02who pinged me told me that if I can do something on the historic data as well, because most people
02:07are looking to download NSE historic data, both the EOD data as well as the intraday data,
02:11which is the one minute, five minute, 15 minute type data, not just for the stock data. They also
02:16wanted, you know, stock options and then futures data as well. So they had asked me if I can create
02:22a similar utility where I can download this data from the NSE. It will be very useful to them. So
02:27based on popular demand, I've created a simple tool that will do just that. So the details that I'm going
02:32to be talking through in this video, all that information is available in this blog article,
02:35and I'll post the link in the description so you can read through this. And then what this tool would
02:41do
02:41is basically, just as you guys asked, it'll get historical data, not just for the underlying
02:47and the dsys, but also for the futures and options data for those stocks and indices. The article gives
02:54you a very detailed step by step process on how to install this and how to use this. And this
02:58is the
02:58entire code. You can just simply copy this and then run it as you wish. You don't really need anything
03:03apart from just pandas and Python installed on your machine. Before I walk you through the Python
03:08code and the actual Python implementation, let's just start from the fundamentals, right? What
03:12exactly are we trying to do here? And how are we getting this data? So I'm sure you are aware
03:16of
03:16this website, which is charting.nsc.com, which is the official NSE website. And then this basically
03:21has charts for all the instruments that are in question. All that the Python program that we are
03:26trying to do is trying to extract data from this chart. And that's where you're getting your EOD data
03:30and all the intraday data, right? At a very high level, this is exactly what that Python code is
03:35doing. For people who are slightly interested in technically understanding how this is done,
03:39I know a couple of subscribers had asked me that you provide code, which is all good,
03:43but try and explain how this code works also, so that we can learn a thing or two. So for
03:48people who
03:48are slightly more interested technically on how this works, what we are trying to do is that if any
03:52website, if you go, it basically works as a series of requests and response, right?
03:58So if you press F12 from any web screen that you're currently in, you would see something
04:02like this. And then when you go into network, this is where you see how this particular client,
04:07which is your browser, is talking to the server, which in this case is the NSE server, right? How
04:12these two items basically talk to each other can be seen from this window. So at a high level, how
04:18this
04:18works is the browser or the client basically sends a web request to the server in a specific format that
04:24the server can understand. And then when the server gets that request, it basically responds back with
04:29the actual data. So in this case, if you really see, these are the server requests that are going
04:33out to the server from this particular web browser, and it is sending the data in this particular format
04:37where it is telling the server as to which specific script code or the script code in this case, Nifty,
04:4326,000, as everybody knows, this is the script code for Nifty. And then what time frame the data is
04:49required. In this case, the interval is one minute. As you can see here, the time interval is one. So
04:53it
04:53specifically sends this information in a JSON format that the server can understand, and the server
04:57basically serves that page with the data that is required. So in the Python program, what we're
05:02trying to do is just emulate as if a real user is trying to request NSE for data. In this
05:07case,
05:08there is no real web browser or anything involved, but we're using Python program to basically emulate
05:12as if a user is trying to get this request done through a web browser. And that's exactly how we
05:17are
05:17trying to send that request, and we get that data back from the server, and we trap that data, and
05:21that's how we
05:22get our historical data. If you haven't joined our Telegram group, please do so. Share market insights,
05:27algo trading tips, and new video notifications. And this way, you can stay up to date with our
05:30community news and events. Now with that background, now let's look at the Python
05:35implementation. Maybe if you look at the code, it'll make a lot more sense now. So this is the code.
05:40Right now, from an intraday perspective, these are all the timelines, the time frames that it
05:44sells, you know, kind of supports, which is one minute, three minutes, five minutes, 10 minutes, 15,
05:4730 minutes, and one hour, which is 60 minutes, the daily, one weekend, one month. So this is all the
05:52data that you can basically extract for any stock, any indices, including, you know, futures or option
05:57symbols, even stock options, including, right? So that's what this does. The way it is implemented
06:02is we basically have created a class where the key functionality is basically already coded in.
06:07This is the class. For people who are not familiar with class, class is one of the most powerful data
06:11structures within Python. If you're familiar with object-oriented programming, I'm sure you've
06:15heard of class. If I want to give you a very crass example on what this really means, assume that
06:20you're building plastic dolls or silicone dolls, right? To build a doll like that, what you first do
06:26is you create a mold first, right? The mold is basically the class here. And then once the mold is
06:31ready, you can create as many dolls as you want using that mold. And these dolls are basically the
06:36objects that you're creating based out of the class that you're creating, right? So the class becomes a
06:40mold and your dolls becomes the object. In this case, we are creating a class called the nsemasterdata.
06:46And like I mentioned, what this is trying to do is that it's basically trying to access these URLs
06:51and then emulating as if a user is sending the request to the nsewebs servers. And then the server
06:57responds with the actual data that we want, which is this data. And then once the data comes through,
07:01we are basically extracting the data that we want, convert that into a pandas data frame,
07:05and we download it and use it that we wanted. I hope that kind of makes sense and hope I'm
07:09not
07:10confusing you. But let's just move on to how this code really works. So at the start, I told you
07:15about the nsemasterdata being the class and then you create an instant called nse. This is where this
07:19becomes the mold and nse becomes the doll that you're creating. So the nse is the object or the
07:23instance that you're creating based on the mold, which is the class, right? So in this case, nse comes
07:27to life here as an object. First thing is, before we even download anything, we'll have to download the
07:32entire nse symbol master, which is both including the nse as well as the nfo. This step is very important,
07:37you'll have to run it every time. All it does is basically downloads the entire symbol master for
07:40nse and nfo and creates a cross-reference file. And then for downloading any data, of course,
07:46you need the start date and end date. In this case, I'm just taking, just for testing purposes,
07:50six days. So end date is today, and my start date is six days prior to the end date, which
07:54is basically
07:55the six day window is what I'm considering. So feel free to change this number to whatever time frame that
08:00you want. And accordingly, the data will be downloaded. Before we download the historic data, there is one
08:05additional utility that I've created here, which can be a bit of use. Not all of us know the entire
08:10you know, the symbol list and its names by heart. So this is basically a simple search facility.
08:15So you can, you can give even the entire full name of this symbol or partially whatever you know,
08:21and then you can tell which exchange that you're basically looking at. And then when you try to run
08:25this, it'll give you the entire list of the symbol code from nse. In this case, I've given nifty bank,
08:30and there is only one symbol there within nse, which is this nifty bank, which is again, very clear,
08:36very familiar 26009 is the bank nifty's symbol. So it basically searches and gives you the details
08:40along with the script code. Similarly, you can do a search on NFO, the FNO symbols as well. So in
08:46this
08:46case, I've given the partial bank nifty 25th April, but I want to extract all symbols that kind of roughly
08:51match this wildcard. So when I run this, this gives me the entire list of all the, you know,
08:57the option symbols for bank nifty. So using this, you can basically zero in copy and then run
09:02historic download on any specific code that you want.
09:07Now that you understand how the symbol search utility works, let's get to the meat of the
09:11video, which is how to actually download the historic data, right? So I've given samples snippet
09:15for every type here. And then I've just printed the first two rows so that, you know, you can, you
09:20get to see the actual output also. I've run this already. So let's quickly take a look at, you know,
09:24what are the things that we can do here in this case, I'm basically downloading the EOD data for
09:28Nifty. Here I've given interval as one day. That's why you're getting the EOD data for Nifty,
09:33which is here, right? So you typically get the open, high, low, close, and volume for Nifty 50 in
09:37this case. Moving on, this is index data, but this is for a lower timeframe, timeframe, which is one
09:44minute data. So I'm basically having bank nifty data for one minute data, which is again, the underlying
09:48data index for it. So this is how it basically prints for every minute. It gives you the data.
09:54for one minute. And then similarly, you can do it for, in this case, I'm doing it for the TCS,
09:59which is again, the underlying stock alone for 10 minutes. So this is where the TCS 10 minutes data
10:03is printed. You can again extend this to, you know, index futures. In this case, Nifty 50 futures is
10:10what I'm taking, the April futures, that also is downloaded. I've given a one hour data here. So that
10:14is downloaded. And then if you want the stock futures, again, you can give the stock name and then
10:19the future symbol that you want. And in this case, Reliance data is printed. I've asked for one hour
10:24data and one hour data is what we get here. Similarly, if you go for index options, given
10:31bank nifty index options, you know, the 50,000 PE, and that's downloaded for five minutes. And then
10:37if you also want the stock options that is also available in this case, the TCS stock options
10:41for 3000 CE, and that is also available here. So I've given a flavor of each of the functionalities
10:46or the symbol types that you can download so that, you know, you can run it and then, you know,
10:50try it out for yourself. There are a couple of items that I would like to remind you. Number one
10:55is
10:55NSE, the data is not very clean. I mean, in the sense that, you know, for example, if
11:00sometimes what happens is instead of 9.15, it gives you dates, timestamps as 9, 14, 59 seconds,
11:06right? I've done my best to clean up all of that to ensure that, you know, you get a proper
11:11timestamp just the way you usually get it on, you know, trading view. So I've done most of the cleanup,
11:15you know, using a tampered with the code a little bit, and then cleaned up all of that issues for
11:19you. But I would still like to warn you that for some of the instruments, especially, you know,
11:24stock options, where there's not a lot of volume and all that, you sometimes might get some weird
11:28timestamps here. So take a closer look at it and then fix it, you know, accordingly. The second
11:33aspect is for majority of the stock options, you know, when you run it, it will give you an empty
11:38one.
11:38So if there is not enough activity for a specific expiry or a strike, you know, you will get an
11:42empty
11:42data frame. So please make sure. So in case there is no data coming from some of your options
11:47instruments, don't blame me. That's how NSE is, right? Because I've seen some of the comments
11:52saying that your code is bullshit, and it's not working and all that. So I try to do my best
11:57here
11:57to give you a cleaner code. But typically, the data is something that's coming from NSE,
12:01and I can't do much about it. So don't blame me, right? Hope you like this tool. A lot of
12:07effort has
12:07gone into doing this. So I mean, I would really appreciate if you can like and subscribe,
12:11because almost 85% of the viewers don't subscribe to the channel. So if you can really do that,
12:15that will encourage me to do more and more of these type of tools that will be useful to you.
12:19So do try this out and let me know how this goes. And if there are any issues,
12:22just put those, you know, in the comment section, I'll definitely get to it. Until then,
12:27thanks for watching, and I'll see you on another video. Bye bye.
12:30If you genuinely found this video useful, please consider subscribing and liking the video.
12:33And I will see you soon in another video. And until then, take care and happy trading.