Features of Web scraping
Retrieving data from websites using crawlers is called web scraping or parsing. It is used to copy Internet resources, such as message boards, online stores, job search sites, etc. But not everything is as simple as it might seem.
Scraping is a rather complex and time-consuming process that requires knowledge of programming in the Python language. It is necessary to create a script for the crawler (program it to crawl certain pages) and write regular expressions or use XPath to get the necessary data (product name, price, description, parameters of product, image, etc.) Each website requires an individual approach and script, which was used for one Internet resource and will not be suitable for parsing another. Moreover site developers periodically change the html-code, after the scripts stop working normally. It is not convenient to correct the code every time and test it again, and if the script was not written by you then it becomes impossible.
Modern sites often use AJAX requests to load information while browsing a web page. For example, Facebook loads the news feed while scrolling. In such situations, it will not work to get information using CURL, only a real browser is required for web scraping of such sites. Selenium is used for automatic browser control. The Python script sends commands to the browser through Selenium and receives the necessary text, or html-code.
Also worth mentioning that scripts written in Python often crash while working with Selenium. You ran the script in the evening in the hope that you will receive fresh data in the morning, but it stopped working as soon as you got up from your desktop. Then you need to write another script that will monitor the work of the first one. As a result, all this will turn into a cumbersome parsing project and this is only for one site.
From what has been stated above, we can conclude that the standard parsing process with Python requires significant improvements. On this site, you will find an elegant and stable solution to this problem.
