How can I fix "403 Forbidden" errors when calling APIs using Python 'Requests'?
I needed to parse a site, but I got a 403 Forbidden error. Here is the code: url = 'http://worldagnetwork.com/' result = requests.get(url) print(result.content.decode()) The output is: <html> <head><title>403 Forbidden</title></head> <body bgcolor="white"> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> What is the problem?
It seems the page rejects GET requests that do not identify a User-Agent. I visited the page with a browser (Chrome) and copied the User-Agent header of the GET request (look in the Network tab of the developer tools): import requests url = 'http://worldagnetwork.com/' headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'} result = requests.get(url, headers=headers) print(result.content.decode()) # <!doctype html> # <!--[if lt IE 7 ]><html class="no-js ie ie6" lang="en"> <![endif]--> # <!--[if IE 7 ]><html class="no-js ie ie7" lang="en"> <![endif]--> # <!--[if IE 8 ]><html class="no-js ie ie8" lang="en"> <![endif]--> # <!--[if (gte IE 9)|!(IE)]><!--><html class="no-js" lang="en"> <!--<![endif]--> # ...
Get this solution programmatically \u2014 free, no authentication.
curl https://depscope.dev/api/error/6f145e73fcbea9b7f6042993ccea7f79da3430252eeb23fe59189a25114709c2