BloG

A Beginner’s Guide to HTTP Python Requests

Nov 24, 2022

Every part is accessible on the Web through requests. For those who need information from an online page in your Python application, you would like an online request. In this text, we’ll dig into Python requests. We’ll have a look at how an online request is structured and how one can make a Python request. By the top, you’ll find a way to make use of the Python requests library, which makes the entire process easier.

An Introduction to HTTP Requests

To exchange data on the Web, we firstly need a communication protocol. The protocol used after we browse the Web is the Hypertext Transfer Protocol, or HTTP. HTTP uses TCP as a transport protocol, since it needs reliable transport, and only TCP can guarantee that.

Let’s say there’s a resource we want — such an HTML page, on an online server situated somewhere on the earth. We would like to access this resource or, in other words, we wish to have a look at that page in our web browser. The very first thing we have now to do is make an HTTP request. HTTP is a client–server protocol, which implies that the requests are initiated by the client.

After the server receives the requests, it processes them and returns an appropriate response.

The server might reply in other ways. It would send the resource we requested, or reply with status codes if something doesn’t go as expected.

In every communication protocol, the data must be in specific fields. That’s because each the client and the server should know how one can interpret the request or response. In the following sections, we’ll have a look at how an HTTP request and an HTTP response are built. We’ll also discuss the role of a very powerful fields.

The HTTP request

Some of the vital design features of HTTP is that it’s human readable. Which means that, after we have a look at an HTTP request, we will easily read every thing, even when there’s loads of complexity under the hood. One other feature of HTTP is that it’s stateless. Which means that there’s no link between two requests served one after the opposite. The HTTP protocol doesn’t remember anything of the previous request. This means that every request must contain every thing that the server must perform the request.

A sound HTTP request must contain the next elements:

  • an HTTP method — resembling GET or POST
  • the version of the HTTP protocol
  • the trail of the resource to fetch

Then, we can even add some optional headers that specify additional information concerning the sender or the message. One example of a typical HTTP request header is the User-Agent or the natural language the client prefers. Each of those optional headers give information concerning the client that’s making the request.

That is an example of an HTTP message, and we will clearly understand all of the fields specified:

~~~http
GET / HTTP/1.1
Host: www.google.com
Accept-Language: en-GB,en;q=0.5
~~~

The primary line specifies the request type and the version of the HTTP protocol. Then we specify the Host and the language accepted by the client that’s sending the request. Often, the messages are for much longer, but this provides a touch of what they appear like.

The HTTP response

Now that we have now an idea of what an HTTP request looks like, we will go on and see the HTTP response.

An HTTP response often comprises the next elements:

  • the version of the HTTP protocol
  • a standing code, with a descriptive short-message
  • an inventory of HTTP headers
  • a message body containing the requested resource

Now that we’ve introduced the fundamental elements you would like, it’s price making a summary before taking the following step. It ought to be clear by now that, every time a client wants to speak with an HTTP server, it must create and send an HTTP request. Then, when the server receives it, it creates and sends an HTTP response.

We’re finally able to introduce the Python requests library.

The Python requests Library

The Python requests library means that you can send Python HTTP requests — from basic to complicated ones. The Python requests library abstracts the complexities of creating complex Python requests, providing an easy-to-use interface. In the following sections, we’ll see how one can create easy Python requests and interpret the response. We’ll also see among the features provided by the Python requests library.

Installing Python requests

First, we want to put in the Python requests library. Let’s install it using pip:

$ pip install requests

Once the Python requests library is installed accurately, we will start using it.

Our first GET request with Python requests

The very first thing we have now to do is to create a Python file. In this instance, we call it web.py. Inside this source file, insert this code:

import requests

URL = “https://www.google.com”
resp = requests.get(URL)

print(resp)

This program makes a GET request for Google. If we run this program, we’ll probably get this output:

$ python web.py
<Response [200]>

So, what does this mean?

We talked concerning the status code earlier. This output is telling us that our request has been received, understood and processed successfully. There are other codes as well, and we will list a number of of probably the most common:

  • 301 Moved Permanently. This can be a redirection message. The URL of the resource we were searching for has been moved. The brand new URL comes with the response.

  • 401 Unauthorized. This means a client error response. On this case, the server is telling us that we must authenticate before proceeding with the request.

  • 404 Not found. This means a client error response too. Particularly, which means the server can’t find the resource we were searching for.

What if we wish to conditionally check the status, and supply different actions based on the status code? Well, we will easily do that:

import requests

URL = “https://www.google.com/blah”
resp = requests.get(URL)

if resp.status_code == 200:
print(“Okay, all good!”)
elif resp.status_code == 301:
print(“Ops, the resource has been moved!”)
elif resp.status_code == 404:
print(“Oh no, the resource wasn’t found!”)
else:
print(resp.status_code)

If we run the script now, we’ll get something different. Have a attempt to see what we get. 😉

If we also want the descriptive short message that comes with each status code, we will use resp.reason. Within the case of a 200 status code, we’ll simply get OK.

Inspecting the response of the Python request

At this point, we all know how one can make a basic Python request. After the request, we wish the response, right?

Within the previous section, we saw how one can get the status code of the response. Now, we wish to read the body of the response, which is the actual resource we requested. To do that, we want to make use of resp.content. Let’s say that we’re searching for the Google home page.

That is what we get after we run the script:

b’<!doctype html><html itemscope=“” itemtype=“http://schema.org/WebPage” lang=“en”><head><meta content=“text/html; [...]

I’ve added […] above since the resource we get — which is a text/html document — is simply too long to be printed. By how much? We will use len(resp.content) to get this information. Within the case above, it was 13931 bytes — definitely an excessive amount of to be printed here!

Making use of APIs

Certainly one of the the reason why the Python requests library became so popular is since it makes interacting with APIs very easy. For this instance, we’ll use an easy API for predicting an individual’s age, given their name. This API known as Agify.

That is the code for the instance:

import requests
import json

URL = “https://api.agify.io/?name=Marcus”
resp = requests.get(URL)

if resp.status_code == 200:
encoded = resp.json()
print(encoded[‘age’])
else:
print(resp.status_code)

On this case, we wish to know the age of an individual whose name is Marcus. Once we have now the response, if the status code is 200, we interpret the end in JSON using resp.json(). At this point, we have now a dictionary-like object, and we will print the estimated age.

The estimated age of Marcus is 41 years old.

HTTP headers provide additional information to each parties of an HTTP conversation. In the next example, we’ll see how we will change the headers of an HTTP GET request. Particularly, we’ll change the User-Agent and the Accept-Language headers. The User-Agent tells the server some details about the appliance, the operating system and the seller of the requesting agent. The Accept-Language header communicates which languages the client is capable of understand.

That is our easy code snippet:

import requests

URL = “https://www.google.com”
custom_headers = {‘Accept-Language’: ‘fr-CH, fr;q=0.9, en;q=0.8, de;q=0.7, *;q=0.5’, ‘User-Agent’: ‘Mozilla/5.0 (Linux; Android 12; SM-S906N Construct/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.119 Mobile Safari/537.36’}
resp = requests.get(URL, headers=custom_headers)

if resp.status_code == 200:

print(resp.content[:100])
else:
print(resp.status_code)

If every thing goes right, it’s best to get something like this:

$ <!doctype html><html lang=“fr”><head><meta charset=“UTF-8”><meta content=“width=device-width,mini [...]

In this instance, we’ve modified the User-Agent, pretending that our request comes from Mozilla Firefox. We’re also saying that our operating system is Android 12 and that our device is a Samsung Galaxy S22.

Since we’ve printed the primary 100 characters of the response above, we will see that the HTML page we’ve received is in French.

Conclusion

In this text, we talked concerning the HTTP protocol, with a transient theoretical introduction. Then we checked out the Python requests library. We saw how one can write basic Python HTTP requests and how one can customize them in line with our needs.

I hope you’ll find this library and this text useful to your projects.

Related reading: