I'm using python requests library, I need send a request without a user-agent header. I found this question, but it's for Urllib2.
I'm trying to simulate an Android app which does this when calling a private API.
I try to set User-Agent to None as in the following code, but it doesn't work. It still sends User-Agent: python-requests/2.27.1.
Is there any way?
headers = requests.utils.default_headers()
headers['User-Agent'] = None
requests.post(url, *args, headers=headers, **kwargs)
The requests library is built on top of the urllib3 library. So, when you pass None User-Agent header to the requests's post method, the urllib3 set their own default User-Agent
import requests
r = requests.post("https://httpbin.org/post", headers={
"User-Agent": None,
})
print(r.json()["headers"]["User-Agent"])
Output
python-urllib3/1.26.7
Here the urllib3 source of connection.py
class HTTPConnection(_HTTPConnection, object):
...
def request(self, method, url, body=None, headers=None):
if headers is None:
headers = {}
else:
# Avoid modifying the headers passed into .request()
headers = headers.copy()
if "user-agent" not in (six.ensure_str(k.lower()) for k in headers):
headers["User-Agent"] = _get_default_user_agent()
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
So, you can monkey patch it to disable default User-Agent header
import requests
from urllib3 import connection
def request(self, method, url, body=None, headers=None):
if headers is None:
headers = {}
else:
# Avoid modifying the headers passed into .request()
headers = headers.copy()
super(connection.HTTPConnection, self).request(method, url, body=body, headers=headers)
connection.HTTPConnection.request = request
r = requests.post("https://httpbin.org/post", headers={
"User-Agent": None,
})
print(r.json()["headers"])
Output
{
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate',
'Content-Length': '0',
'Host': 'httpbin.org',
'X-Amzn-Trace-Id': 'Root=1-61f7b53b-26c4c8f6498c86a24ff05940'
}
Also, consider to provide browser-like User-Agent like this Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36. Maybe it solves your task with less effort
For urllib3 >= 1.26.0 (aka, requests >= 2.25.0), you can make use of urllib3.util.SKIP_HEADER to suppress User-Agent, Accept-Encoding and Host header. See https://github.com/psf/requests/issues/5671#issuecomment-1006735307 for an example and https://github.com/urllib3/urllib3/pull/2018 for upstream changes.
PS: You may want to use urllib3 >= 1.26.3 to avoid a minor issue spamming connection.py: BytesWarning: Comparison between bytes and string. See https://github.com/urllib3/urllib3/issues/2071.
