I am trying to teach myself some basic web scraping. Using Python's requests module, I was able to grab html for various websites until I tried this: More on stackoverflow.com

Stack Overflow | The World’s Largest Online Community for Developers More on stackoverflow.com

requests-html requests-html (at least the .render() function) is doing a similar thing to selenium, with a lot of the same performance impact: it's invoking a whole browser to render the page UI. It does have the advantage that you can get away without .render() in cases where you don't need it, so is preferable in that respect, but the main thing those websites are advocating is to simply use the plain requests.get() approach, without having to get a web browser involved in the process at all. I think selenium often becomes a bit of a newbie trap, because it works in a way that's pretty familiar to everyone (we've all used a browser), and so viewing a site as being about clicking on links and manipulating UI elements then looking at the result is an easy way to understand scraping. And as sites grow more dynamic and complex, sometimes it's the simplest (but usually not best) way to get some information, especially if you're unfamiliar with what goes on behind the scenes. But if we drop a level of abstraction and view websites not as UI elements, but as simply fetching and sending data to and from the server, scraping becomes orders of magnitude more efficient, and sometimes even easier. Even on dynamic webpages that build their content dynamically with javascript, that code needs to get that data from somewhere, and often a quick look at the network tab when loading the page will show you where, often in a format much easier and more consistent to parse than messing with complex html. But a lot of time people reach for selenium as a first option, even if they could accomplish the same goal with a simple get request, simply because it's their default. In reality, I think selenium should be the option of last resort - used only when regular get requests are too difficult or tedious to figure out (generally when a site is basically actively using anti-spidering countermeasures), and requests-html.render() is only really one step behind that (but is much better in that it presents this as an optional tool you can selectively apply, rather than the interface through which you do everything). More on reddit.com

Well, requests for starters. Its simple and easy to get hold of. But the prob is that it doesn't fetch the Javascript content which is mostly on E-commerce websites. For that theoretically requests-html is for. However, i had some trouble with it (unable to do SSL Handshake and couldn't resolve it) Scrapy is a bit advanced but is very powerful. Selenium isn't that efficient as it loads the webpage through a browser, but does the job. TL:DR: if you have small and simple scraping job, use requests. If the task is simple and data is huge, still would suggest requests. If going through small data on E-commerce or JS websites, selenium does the job. If large data, scrapy is pretty efficient. More on reddit.com