- 7 hours ago
Learn how to use Beautiful Soup’s find() and find_all() methods to locate and extract specific information from HTML with Python. This practical web scraping tutorial shows how to request a web page, parse its HTML, and search for elements such as div, paragraph, table, and heading tags.
See how tag attributes like class can narrow your results, how find() returns a single match while find_all() collects multiple matches, and how to extract and clean text with .text and .strip(). A clear introduction to Python web scraping, HTML parsing, and BeautifulSoup selectors for beginners who want to find targeted page content and prepare data for further analysis.
See how tag attributes like class can narrow your results, how find() returns a single match while find_all() collects multiple matches, and how to extract and clean text with .text and .strip(). A clear introduction to Python web scraping, HTML parsing, and BeautifulSoup selectors for beginners who want to find targeted page content and prepare data for further analysis.
Category
📚
LearningTranscript
00:00Hello, everybody. In this lesson, we're going to be taking a look at find and find all. Really,
00:05we're going to be looking at a ton of different things in this lesson. This is where we really
00:08start digging in, seeing how we can extract specific information from our webpage. But in
00:14order to do that, let's set everything up where we actually bring in the HTML like we did in the
00:18last lesson. And we're just going to write all this out one more time just for practice, if
00:23nothing else. And then we'll get into actually getting that information from the HTML. So we're
00:28going to start by saying from VS4 import beautiful soup. There we go. And import requests. We'll go
00:39ahead and run this. Then we're going to come up here, grab our HTML or sorry, our URLs. We'll say
00:46URL is equal to, and we'll have that right here. Now we need to say page is equal to, and
00:54then we'll
00:54do requests dot get, and then we'll put in our URL right here. And we're going to come over here
00:59and
00:59run this. And lastly, we need to say soup. We'll say soup is equal to beautiful soup. There we go.
01:07And then within our parentheses, we need to specify the page dot text because we need that and our
01:13parser, which is HTML. And there we go. And let's go ahead and run this. Let's print it out, make
01:20sure
01:20it's working. And there we go. So we have our soup right here. All this should look really similar to
01:29our last lesson. And so now we've brought in our HTML from our page. We have a lot, a lot,
01:34a lot of
01:35information in here. Now, really quickly, let's come over and let's inspect our webpage.
01:42Now in here, we have a ton of information, right? We have a bunch of different tags and classes and
01:48all
01:48these other things. But how do we actually use these? Well, that's where the find and find all
01:54is going to come into play. And they're pretty similar. And you'll see that in just a little bit.
01:58But let's say we want to take one of these tags. And let's come down. Let's say we just want
02:05to take
02:05this div tag. Now, there's going to be a lot of different div tags in our HTML. But let's just
02:12come
02:13right here. Let's go down. And let's say we're going to call soup. We're going to say soup. That's
02:18all of our information. We're going to say dot find. Now, within our parentheses, we can specify
02:22a lot of different things. But we're going to keep it really simple right now. We're just going to say
02:27div. Let's go ahead and run this. What this is going to bring up is the very first div tag
02:33in our
02:33HTML. And that's going to be this information right here. Now, let's copy this. I'm going to do the exact
02:39same thing, except we're going to say find underscore all. Now, let's run this. Now, we're
02:47going to have a ton more information. Really all find and find all do is that they find the
02:54information. Now, find is only going to find the first response in our HTML. That's the div class
03:00container. Let's go back up to the top. That's our div class container. But find all is going to find
03:06all of them. So it'll put it in this list for you. So it's going to have this first one
03:11and it goes down to this forward slash div, which should be right here. And then we have
03:17a comma, which separates our next div tag. So that is how we can use it. Now, what if we
03:23want to specify one of these div tags? We pulled in a ton of them, but we want to just
03:27look for
03:27one of them. Well, this is something where the class comes in handy because right now we have
03:32class is equal to container, class is equal to colmd-12. I don't know what these are off the top
03:39of my head, but usually they'll be somewhat unique and we can use these to help us specify what we're
03:46looking for. For example, just kind of glancing at this, we can also use this a tag if we wanted
03:51to
03:51look at this. So we could say, oh, we're looking for these href. So we have an href here and
03:56this
03:57right down here, we have this href as well, which again, if you remember from a previous lesson,
04:02that stands for a hyperlink. Now, something like the class or the href or these IDs, these are all
04:09attributes. So we can specify or kind of filter down based off of these. Now let's try it. So what
04:15we can do is we can do class first, and this is kind of the default within something like find
04:20all
04:20is you can even do class underscore. We can come right back up. We have this div and then here's
04:26our
04:26class. So again, we have to have the div and the class. If we took this a tag, this is
04:31an a tag,
04:32which would go right here with the class of something like nav link or something like nav link
04:37again down here. We need to specify that more, but we have our div. So we'll say coal MD 12
04:43right here.
04:45And let's go ahead and run this. And now it's going to pull in just that information. Now we're still
04:50getting a list because we have multiple of these. So this div class, coal MD dash 12 doesn't just
04:56happen once. If we scroll down, we'll see it multiple times, something like right here,
05:03or actually, let me see right here. So here's this comma, then here's our next one. So we have two
05:08of
05:09these div tags with a class of coal dash MD dash 12. And in each of these, we have different
05:16information. This looks like a paragraph with this P tag right here. And let's scroll back up.
05:23So I also think we should try out doing something like this P tag. Typically these P tags stand for
05:28paragraphs or they have text information in them. Let's try to P tag really quickly and let's just
05:34see what we get. And let's run this. And it looks like we get multiple P tags. Now, if we
05:40come back here,
05:40you can see that there's this information and it's this information that we're pulling in.
05:45And I'm just noticing that from right here. And then we have this information right here.
05:51It looks like there's one more, which is this href, which looks like this open source. So data via,
05:57and then that hyperlink or that link right there. So we have three different P tags. Now, just to
06:04verify and make sure that that's correct, what we could do is come over here. We're going to click on
06:09this paragraph and it's going to take us to that P tag where the class is equal to lead. Let's
06:15come
06:15over here and look at this paragraph. Now we have another P tag right over here where the class is
06:21equal to Glyphicon, Glyphicon slash education. I have no idea what that means. And then we'll go to
06:28our last one, which is right here, where the P tag is equal to, we have a tag, href, class,
06:36and a bunch of other information. So let's say we just wanted to pull in this paragraph right here.
06:42Let's go here and see how we can specify this information. So it looks like P where the class
06:47is equal to lead. That looks like it's going to be unique to just that one. So if we come
06:52down here,
06:53we're going to say comma, and it was class. So you can do a class underscore is equal to,
07:00and then we're going to say lead. Let's try running this. And we're just pulling in that
07:06information. Now let's say we actually want to pull in this paragraph. We actually want this text
07:11right here. And this is a very real use case. You know, let's say I'm trying to pull in some
07:16information or a paragraph of text. Well, let's copy this. And what we're going to then do is say,
07:23dot text. And let's run this. Now we're going to get an error right here. And this is a very
07:28common
07:28error because we're trying to use find all. Unfortunately, find all does not have a text
07:36attribute. We actually need to change this to find. Typically when I'm working with these find and
07:42find alls, I'm using find all most of the time until I want to start extracting text. Then when I
07:48specify it, I'll change this back to find just like this. Now let's try this. And now we're getting
07:54in parentheses, this information. Now this is all wonky and needs to definitely be cleaned up a little
08:00bit, but if we go back up, it's no longer in a list and we no longer have things like
08:06these P tags
08:07in here or this class attribute. So we're really just trying to pull out this information. Now, again,
08:14this does not look perfect. We could even try to do something like dot strip. Look like there's
08:19some white space and that cleans it up a little bit. This definitely looks a little better. And
08:25we could definitely go in here and clean this up more. But just for an example, this is how we
08:30can
08:31then extract that information. Now let's look at one more example. This is some information. This is
08:36what we're going to do kind of our little mini project in the next lesson on. Let's say we wanted
08:40to take all this information. Well, what if we wanted to pull in something like the team name?
08:45That's going to be in right here in this tr tag. And each of these tr tags have th tags
08:51underneath
08:52them. So if we scroll down, you'll notice that each row is this tr tag. So let's go ahead and
08:59search
09:00for, let's do th. Let's just search for that first. So let's come right back up here. Let's use
09:06this find all and we'll get rid of this text for right now. And let's just say we want to
09:15look for
09:16the tr. Is that what we said we were looking for? No, th. So let's say we're looking for th.
09:22Let's go
09:22ahead and run this. So we're going to have underneath this th, we have team name, year, wins, losses,
09:28and notice these are all the titles. So these titles are the only ones with these th tags.
09:35If we go down, you'll notice that the data is actually td tags. So now let's go back and look
09:42for td. We'll say d. And this is going to be a lot longer. We have a lot of information,
09:49but these are
09:49all the rows of data. Let's see if we can just get one piece of this data. We're going to
09:55get back.
09:55We want just this team name. That's all we're trying to pull in for now. And then we'll try
10:00to get this row. And then in the next lesson, we're going to try to get all of this information,
10:05make it look really nice. And then we'll put it into a pandas data frame. So let's just get this
10:10team name right now. Let's go ahead. We're going to say th. Let's run this. And we have this th.
10:18And now that we know we're getting this information in, we can do find, let's run this.
10:26So there's our team name. I'm just going to say dot text. And again, we can do dot strip just
10:34like
10:34that. And bam, we have our team name. So you can kind of start getting the idea of how we're
10:40pulling
10:40this information out. We're really just specifying exactly what we're seeing in this HTML. And what's
10:46really, really helpful. And, you know, it's something that I do all the time is I'm inspecting
10:51it. I'm just kind of searching like, what do I want? What piece of information do I want? Then I
10:56go ahead and click on it. And then I'm looking, you know, where is this sitting in the hierarchy?
11:00It's within the body. It's within this table with the class of table. Then it's down here where
11:06this tr tag and then this td tag. So I'm looking kind of at the hierarchy and I'm specifying exactly
11:11what I'm looking for. So that is what we're going to look at in today's lesson. That's how we can
11:16use
11:16find and find all. We were able to look at classes and tags and attributes and variable strings,
11:23which is this right here, getting that text and variable strings. And we were to look at find and
11:28find all and how it's pulling that information in and how we can specify exactly what we're looking for.
11:33Now in the next lesson, which is definitely going to be the most exciting one, we're going to try to
11:37pull in all of this information. So every single thing, because we'll be able to put all this
11:43information into a data frame, which then we can use pandas to really search and manipulate that data
11:49within that data frame. So with that being said, that is the end of this lesson. If you liked this
11:53video, be sure to like, and subscribe. I will see you in the next lesson.
12:08I'll see you in the next lesson.