Python, create a network request

To create a network request in Python you can use the urllib standard library package. It ships with Python, so there’s nothing to install.

Create a request using:

from urllib import request
url = 'https://dog.ceo/api/breeds/list/all'

response = request.urlopen(url)
content = response.read()

print(content)

You can also use the with statement to simplify. It closes the connection for you when the block ends:

from urllib import request
url = 'https://dog.ceo/api/breeds/list/all'

with request.urlopen(url) as response:
   content = response.read()

print(content)

The response object also tells you how the request went. Its status attribute holds the HTTP status code, which is 200 when everything worked:

print(response.status)
# 200

The body returned by read() is a sequence of bytes, as you will notice because the response is wrapped in a b'' string:

b'{"message":{"affenpinscher":[],"african":[],"airedale":[],"akita":[],"appenzeller":[],"australian":["shepherd"],"basenji":[],"beagle":[],"bluetick":[],"borzoi":[],"bouvier":[],"boxer":[],"brabancon":[],"briard":[],"buhund":["norwegian"],"bulldog":["boston","english","french"]},"status":"success"}'

Bytes are the raw data that traveled over the network. To work with it as text, decode it to a UTF-8 encoded string using content.decode('utf-8').

This gets the HTML content from my website flaviocopes.com:

from urllib import request

url = 'https://flaviocopes.com'

with request.urlopen(url) as response:
   content = response.read().decode('utf-8')

print(content)

Parsing a JSON response

Most APIs answer with JSON, and you can parse it using the json standard library module:

from urllib import request
import json

url = 'https://dog.ceo/api/breeds/list/all'

with request.urlopen(url) as response:
   content = response.read()

data = json.loads(content)
print(data['status'])

json.loads() turns the response into a Python dictionary, so you can access the fields with the usual square bracket syntax.

Adding query parameters

If you need to specify query parameters, use the urlencode() function from the urllib.parse module to build the query string:

from urllib import request, parse
url = 'https://api.thecatapi.com/v1/images/search'

parms = {
    'limit' : 5,
    'page' : 1,
    'order' : 'Desc'
}

querystring = parse.urlencode(parms)

with request.urlopen(url + '?' + querystring) as response:
   content = response.read().decode('utf-8')

print(content)

urlencode() also takes care of escaping special characters, like spaces, so you don’t have to.

Handling errors

Here’s the thing that catches everyone the first time: when the server answers with an error status like 404 or 500, urlopen() does not return the response. It raises an exception.

Wrap the call in a try block if the request can fail:

from urllib import request, error

try:
    with request.urlopen('https://dog.ceo/api/missing') as response:
        content = response.read()
except error.HTTPError as e:
    print(e.code, e.reason)
    # 404 Not Found

This is the built-in urllib package.

For convenience purposes, you might want to use the requests package, not part of the Python standard library but quite popular.