ScrapingBee logo

The ScrapingBee Blog

We help you get better at web-scraping: detailed tutorials, case studies and writings by industry experts.

Don't know where to begin?

Check out our most popular articles.

Python Web scraping without getting blocked Python Web scraping in Python Python Web scraping in NodeJS Python Web scraping in R Python Web scraping in C++ Python Web scraping in C# Python Web scraping in Php Python Web scraping in Scala Python Java web scraping Book Python Web scraping in Ruby Python Web scraping in Go Python Web scraping in Elixir Python Web scraping in Rust Python Web scraping in Perl Python No code web scraping Python No code competitor monitoring
How to submit a form with Puppeteer?
6 min read

How to submit a form with Puppeteer?

Submitting a form with Puppeteer can be tricky. In this article, we will show the different way to submit a simple form, but also with attachments.

Shadid Haque
11 min read

The Best Ruby HTTP clients for 2021

This article will discuss the best HTTP clients in Ruby. Faraday, HTTParty, Rest-client...it can be hard to choose the best one.

Sylwia vargas
7 min read

How to download a file with Puppeteer?

There are different ways to download a file with Puppeteer. Unfortunately, all of the different techniques are not well documented. We will go through several practical examples of file dowloading with Puppeteer.

Kevin Sahin
11 min read

Scraping the web with Playwright

Learn how to scrape the web with Playwright. We will see the different ways to scrape the web with Playwright through lots of example.

Kevin Sahin
12 min read

Top 5 Rotating and Residential Proxies for Web Scraping

In this post you will find the 5 best rotating and residential proxies for Web Scraping. 1) ScrapingBee 2) Luminati 3) Oxylabs 4) Smartproxy 5) Crawlera

Kevin Sahin
15 min read

The 5 Best Free Proxy Lists for Web Scraping

This post will show you the top 5 free proxy lists available on the web: ScrapingBee, ProxyScrape, FreeproxylistCZ and more.

Kevin Sahin
11 min read

How to extract data from a website? Ultimate guide to pull data from any website

This article will teach you to extract data from the web easily. There are many ways to pull data from websites, from web scraping tools to in-house pipeline. This post will show you exactly what's the best solution to your web scraping problem.

Kevin Sahin
7 min read

HTTP headers with axios

This post will show you how to send HTTP headers with Axios. We will see different examples with GET and POST requests on how to set your headers with Axios.

Kevin Sahin
8 min read

Web Scraping with node-fetch

In this tutorial we will see how to use the node-fetch package for web scraping. We will learn what the fetch API is and the different ways to use the package.

Kevin Sahin
10 min read

Charles proxy for web scraping

This post will show you how to use Charles proxy to reverse-engineer hidden and private APIs on websites and mobile applications.

Kevin Sahin
4 min read

How to use a proxy with node-fetch?

In this tutorial we will see how to use a proxy with the node-fetch package. We will also discuss on how to choose the right proxy provider.

Pierre de Wulf
7 min read

How to execute JavaScript with Scrapy?

Learn how to use Scrapy with website using JavaScript rendering.

Ari Bajo
5 min read

'JMAP (YC S10) Linux Inside is hiring': the quest for the best Hacker News title

Analysing 2.8 millions Hacker News posts titles in order to generate the one that would perform the best, statistically speaking.

Pierre de Wulf
5 min read

Serverless Web Scraping With Aws Lambda and Java

In this post we are going to deploy a Java web scraper using AWS Lambda function and the serverless framework. Cloud functions are a really good fit for web scraping tasks.

Kevin Sahin
4 min read

Scraping single page applications with Python.

Learn how to scrape single page application with Python. Scraping Javascript pages with Python can be tricky, this tutorial will show you how do to it.

Kevin Sahin
Previous
Next