This post has already been read 207 times!

Python urllib

  • We can use Python urllib to get website content in python program.
  • We can also use it to call REST web services.
  • We can make GET and POST http requests.
  • This module allows us to make HTTP as well as HTTPS requests.
  • We can send request headers and also get information about response headers.
Python urllib GET example
import urllib.request
response = urllib.request.urlopen('https://www.wikipedia.org')
print(response.read())

Response read() method returns the byte array. Above code will print the HTML data returned by the Wikipedia home page. It will not be in human readable format, but we can use some HTML parser to extract useful information from it.

Python urllib request with header

import urllib.request

# Request with Header Data to send User-Agent header
url = 'https://www.journaldev.com'

headers = {}
headers['User-Agent'] = 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17'

request = urllib.request.Request(url, headers=headers)
resp = urllib.request.urlopen(request)
print(resp.read())

Leave a Reply

Post Navigation