Python guide to Requests and BeautifulSoup

137

    In this guide I share how I learned to use the Requests and BeautifulSoup libraries in Python to make HTTP requests and perform web scraping to analyze the content of a web page.

    Introduction

    Requests It is a Python library that allows you to make HTTP requests easily and efficiently. On the other hand, BeautifulSoup It is a library that facilitates the analysis and extraction of data from HTML and XML documents.

    Facility

    To use Requests and BeautifulSoup, you must first install them. You can do this by running the following commands in your terminal:

    ```bash pip install requests pip install bs4 ```
    

    Requests Bookstore

    Make an HTTP request

    With the Requests module, we use the method get() To obtain the content of a web page. For example, to obtain the content of the Wikipedia page in Spanish, we can run the following code:

    ```python
    import requests response = requests.get(''https://es.wikipedia.org/wiki/Wikipedia:Portada'')
    ```
    

    In this case, we will obtain an object of type Response which contains the content of the web page. To access the content, we can use the attribute text:

    ```python print(response.text) ```
    

    And that's how we get the content of the web page in plain text format.

    To obtain the HTTP response status code, we can use the attribute status_code:

    ```python
    print(response.status_code) ``` A small note: the codes from state from HTTP responses indicate whether an HTTP request HE has been successfully completed. For example, the code from state `200` indicates that the application HE has completed correctly, while that the code from status `404` indicates that the Web page No has been found.
    

    We can also obtain the HTTP response headers using the attribute headers:

    ```python print(response.headers) ```
    

    And that's how we get a dictionary with the HTTP response headers.

    To use the content of the page in question as a variable, we can use the method content:

    ```python content = response.content ```
    

    This method, exactly, encodes the content of the web page in binary format.

    Once we have the content of the web page stored in a variable, we can use the BeautifulSoup library to parse the content and extract the information that interests us.

    BeautifulSoup Bookstore

    Parsing the content of a web page

    To use BeautifulSoup, we must first import it:

    ```python
    from bs4 import BeautifulSoup
    

    Once imported, we can create an object of type BeautifulSoup based on the content of the website:

    ```Python
    soup = BeautifulSoup(content, ''lxml'') ``` CONTENT PARAMETER: content of the pwebsite in Binary format. PARAMETER ''lxml'': specifies he analyzer that HE will use to parse he content of the pwebsite. In this case, we will use he lxml parser, but we could also use he html.parser
    

    Now that we have the web page content parsed, we can use BeautifulSoup's methods to extract the information we are interested in.

    BeautifulSoup Methods

    soup.title

    We will use BeautifulSoup's methods to extract the information we need from the webpage. For example, to get the webpage title, we can use the method title:

    ```python print(soup.title) ``` The title method returns the title of the web page, including the HTML tags.
    

    soup.find() y soup.find_all()

    Another very useful method from BeautifulSoup is find() y find_all(). These methods allow us to search for HTML elements within the content of a web page.

    In the following example, we will use the method find_all() To get all the links on the website:

    links = soup.find_all(""to"") #links It is a list of links
    print(''print(soup.find_all("a"))'")
    print(links)
    print(""\n"")
    

    A quick note: to identify links in HTML, the tag is used , That's why we use the parameter "a" in the find_all() method.

    soup.prettify()

    To print the content of the web page in a more readable way, we can use the method prettify():

    print(soup.prettify())
    

    The prettify() method returns the content of the web page in a tree structure, making it easier to read.

    Differences between response.text, BeautifulSoup(response.text, 'lxml') y soup.prettify().

    import requests from bs4 import BeautifulSoup
    
    response = requests.get('https://es.wikipedia.org/wiki/Wikipedia:Cover')
    content = response.content
    soup = BeautifulSoup(content, 'lxml')
    prettyify = soup.prettify()
    
    • RESPONSE: The text method returns the content of the web page in plain text format.
    • SOUP: When we use BeautifulSoup(response.text, 'lxml'), we create a BeautifulSoup object from the plain text content of the webpage. This object is (internally) like a tree containing all the HTML elements of the webpage, structured in its own way.
    • PRETTIFY: The `prettify()` method returns the soup with a tree structure. Essentially, it allows us to better understand the structure of the `BeautifulSoup` object by indenting the text and adding line breaks according to the tree structure.

    Practical example

    To continue with the example and recap what we've seen so far, let's search for all the links containing the string "about" on Google's homepage:

    import requests
    from bs4 import BeautifulSoup result = requests.get(""https://www.google.com/"")
    print(result.status_code) # 200, is correct
    
    src = result.content # web page content in binary format
    soup = BeautifulSoup(src, ''lxml'') # We parse the content of the web page
    
    links = soup.find_all(""to"") # we searched for all the links on the website
    for link in links: # for each link in the link list
        if ""About"" in link.text:
            print(link) # We print the link
            print(\n) # we print a line break
            print(link.attrs[''href'']) # we obtain the value of the href attribute
    

    Quick note: to get the value of an HTML attribute we can use the method attrs and specify the name of the attribute we want to obtain. In this case, we want to obtain the value of the attribute href, which contains the link URL.

    Attributes of HTML elements

    HTML element attributes are pairs of values used to configure the behavior of HTML elements. For example, the attribute href It is used to specify the URL of a link. Below we will see the most common attributes:

    First, if we want to observe all the attributes of an HTML element, we can use the method attrs:

    link = soup.find(""to"")
    print(link.attrs)
    

    This will show us a dictionary with all the attributes of the HTML element.

    href

    We've seen this before; it's used to specify the URL of a link.

    <a href=""https://www.google.com/"">Google</a>
    

    To get the value of any HTML attribute in Python, we will use the method attrs and we will specify the name of the attribute we want to obtain:

    link = soup.find(""to"")
    print(link.attrs[''href'']) # https://www.google.com/
    

    id

    It is used to specify a unique identifier for an HTML element.

    <p id=""paragraph"">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>
    

    In the same way as before, we will obtain the attribute value using Python. id as follows:

    p = soup.find(""p"")
    print(p.attrs[''id'']) # paragraph
    

    src

    It is used to specify the URL of an image.

    <img src=""https://germanmallo.com/wp-content/uploads/2024/05/googlelogo_color_272x92dp.png"">
    
    img = soup.find(""img"")
    print(img.attrs[''src'']) # https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png
    
    ✉ SHALL WE TALK?

    Let's talk

    If we share interests in artificial intelligence, data, or software development, I'd be happy to exchange ideas. You can also contact me for professional matters.

    Read the blog