Web scraping is the process of identifying and extracting different types of text from a website, such as author names or article titles, through a programming language or a web tool. In using web scrapers, researchers must consider a website's possible copyright restrictions. In some cases, websites only allow the use of proprietary APIs for text mining.  

Out-of-the-Box*

• Octoparse
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, search bars, etc.

• Parsehub
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, search bars, etc.

• NCapture
  For scraping web content to import into NVivo

• Outwit (Hub, Images, Docs, Email Sourcer)
  For scraping simple and dynamic websites (using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, CAPTCHA tests, etc), news media, social media sites, images, documents, and email

• Portia
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, etc.

• Import.io
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, etc.

• Webhose.io
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, etc.

• Spinn3r
  For social media, blogs, news sites, videos, RSS feeds, and live web content

• Data Scraper
  For simple HTML and CSS websites

• Web scraper
  For scraping simple and dynamic websites using HTML, CSS, AJAX technologies, JavaScript, redirects, cookies, etc.

• Scraper
  For scraping tables on For simple HTML websites

• wget
  For retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols; non-interactive commandline tool, so it may easily be called from scripts.

• OpenRefine
  For fetching, exploring, cleaning, transforming, reconciling and matching data

* For a detailed comparison of these tools, see the Non-Coding Web Scraping Tools chart.  

Programmatic

 Python 

• Scrapy
  For extracting the data you need from websites

• Beautiful Soup
  For parsing and extracting data from HTML and XML documents

• Pattern
  For webscraping (Google, Wikipedia, Twitter, Facebook, generic RSS, etc.), web crawling, HTML DOM parsing, part-of-speech tagging, n-gram search, sentiment analysis, vector space modeling, clustering, classification (KNN, SVM, Perceptron), graph centrality and visualization

R

• rvest
  For scraping (or harvesting) data from web pages

• selectr
  For translating CSS selectors to XPath expressions

• “Web Scraping: The Comprehensive Guide for 2020." n.d. ProWebScraper.  

• Turkel, William J., and Adam Crymble. 2012. “Downloading Web Pages with Python.” Programming Historian, July.

• Champagne, Ashley. "Beautiful Soup Workflow." n.d. Github. 

• Parikshit. "Web Scraping with R." 2020. Scraping Bee. May 5, 2020.

• "Web Scraping." n.d. Gender Novels Project. 

• "GNU Wget 1.21.1-dirty Manual"

• Baxter, Richard. “How to Download, Install and Use WGET in Windows 10.” BuiltVisible, January 1, 2019.

• Pot, Justin. “How to Use Wget, the Ultimate Command Line Downloading Tool.” How-To Geek, November 26, 2016.

• Williamson, Evan Peter. “Fetching and Parsing Data from the Web with OpenRefine.” Programming Historian, August 12, 2017.

   

• Boeing, Geoff, and Paul Waddell. “New Insights into Rental Housing Markets Across the United States: Web Scraping and Analyzing Craigslist Rental Listings.” Journal of Planning Education and Research 37, no. 4 (December 2017): 457–476. 

• Meschenmoser, Philipp, Norman Meuschke, Manuel Hotz, and Bela Gipp. 2016. “Scraping Scientific Web Repositories: Challenges and Solutions for Automated Content Extraction.” D-Lib Magazine 22, no. 9-10.