According to the API, the headers can all be passed in with requests.get():
Copyimport requests
r = requests.get("http://www.example.com/", headers={"Content-Type":"text"})
Answer from cwallenpoole on Stack OverflowAccording to the API, the headers can all be passed in with requests.get():
Copyimport requests
r = requests.get("http://www.example.com/", headers={"Content-Type":"text"})
This answer taught me that you can set headers for an entire session:
Copys = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})
# Both 'x-test' and 'x-test2' are sent
s.get('http://httpbin.org/headers', headers={'x-test2': 'true'})
Bonus: Sessions also handle cookies
Videos
How do I rotate headers to avoid detection?
Maintain a list of realistic header sets and randomly select one per request. Vary the User-Agent, Accept-Language, and Referer values. Combine this with proxy rotation for better results.
When should I use browser automation instead of requests?
Use browser automation tools like Playwright or Selenium when you need to load JavaScript, handle clicks, or when you're blocked by TLS fingerprinting. While these tools solve the TLS problem, they are slower and use more computer resources. ScrapFly offers both simple requests and full browser automation.
Why do my headers work in testing but fail in production?
This is usually because you're sending too many requests. In testing, your low number of requests doesn't trigger extra anti-bot checks. In production, more requests trigger TLS fingerprinting. Even with perfect headers, a different TLS fingerprint will get you blocked. ScrapFly solves this by using real browser TLS profiles.