Description: a Python library for pulling data out of HTML and XML files. Has less power and complexity than Scrapy, but a more accessible interface.
Type: HTML and XML parser
Programming Language(s): Python
Skill Level(s): Intermediate
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: a NodeJS library for parsing and manipulating HTML and XML. Best for developers familiar and comfortable with jQuery.
Type: HTML and XML parser for Node.js
Programming Language(s): NodeJS
Skill Level(s): Intermediate, Advanced
Compatibility: Windows, MacOS, Linux, Unix, more
Cost: Free and open-source

Description: a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites.
Type: web scraping framework
Programming Language(s): Ruby
Skill Level(s): Advanced
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: a Node.js library which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. Puppeteer runs in headless mode by default, but can be configured to run in full ("headful") Chrome/Chromium.
Type: Chromium API
Programming Language(s): NodeJS
Skill Level(s): Advanced
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: a simple but effective HTTP client library for the Python.
Type: HTTP client library
Skill Level(s): Intermediate
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: a package for scraping (or harvesting) data from HTML web pages, inspired by libraries like Beautiful Soup
Type: R package
Programming Language(s): R
Skill Level(s): Intermediate
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: a fast, high-level web crawling & scraping framework for Python.
Type: web-crawling framework
Programming Language(s): Python
Skill Level(s): Intermediate, Advanced
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: an open source project for a range of tools and libraries that enable browser automation. It is usually used for automated testing, but it can be used for scraping data from dynamic web pages with elements that are not visible until the page is fully loaded or the user interacts with them.
Type: software testing framework for web applications
Programming Language(s): C#, Groovy, Java, JavaScript (Node.js), Perl, PHP, Python, Ruby, Scala
Skill Level(s): Intermediate
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source

Description: an interactive command-line tool for retrieving files using HTTP, HTTPS, FTP and FTPS that can also
be called from scripts, cron jobs, and terminals without X-Windows support.
Type: command-line FTP client / HTTP client
Skill Level(s): Intermediate
Compatibility: Windows, MacOS, Linux
Cost: Free and open-source