Contents

The 5 Best mobile and 4g proxy providers for web scraping

Kevin Sahin Kevin Sahin

Kevin has been working in the web scraping industry for 10 years before co-founding ScrapingBee. He is also the author of the Java Web Scraping Handbook.

Blog post header

In this article, we will look at the 5 best mobile and 4g proxy providers for web scraping. We will not only look at the different features they offer, but also perform a real-world test that includes the performance, speed, success and error rate on some of the most popular websites: Google, Amazon, Instagram and the top 1000 Alexa rank.

There are many types of proxies that you can use for web scraping. Depending on the website you are extracting data from, it can be interesting or even mandatory to use mobile proxies or 4g proxies.

Mobile proxies are great to bypass rate-limits because the IP blocks assigned to mobile network operators generally have a great reputation. Using 3G/4G proxies minimizes the risk of getting blocked or encounter CAPTCHAS while extracting data from third-party websites.

They can be mandatory when scraping social media websites or mobile-first websites.

There are lots of players in this field, and it can be hard to choose one. The main features that I looked at for this list are:

  • price
  • performance (speed, reliability)
  • success rate on different websites

1) Scrapingbee

Yes… I know it sounds like an ad, and if you looked at previous our blog post about free proxies it sounds like “déjà-vu” but even with an unbiased opinion we're the best service on this list :)

Scrapingbee is a simple API, where you can get the raw HTML of any website with a simple API call. We offer different types of proxies, residential, data center and mobile so that you never get blocked again while scraping the web.

With Scrapingbee you don't have to deal with proxy management or headless browser.

Our pricing starts at $29/mo, we have a $99/mo plan for the startup plan, and of course we can offer custom pricings for higher needs.

Results (full benchmark & methodology)

Website errors blocked success average time
Instagram 45 0 955 3.35
Google 80 0 920 8.30
Amazon 22 0 978 3.34
Top 1000 Alexa 5 0 995 3.34

2) Airproxy

Airproxy is an interesting provider on this list. They are very transparent about where their mobile proxies are coming from as you can see in this video.

https://vimeo.com/414328430

Basically they rent sim cards from Italian mobile carriers and let you use it. The cost is high, it's 87€ (~$100) per month per IP. You can request an IP change every 15mn. The authentication is user/password based, which is great when you are deploying your scrapers on cloud providers without a fixed IP address.

While those proxies are great for social media scraping and automation, it's not that great for other use cases. For example, the proxies are blocked by default on everything that isn't social media or Google. On Google the rate limits are a big problem. On my test, I the IP get blocked after a few thousands request. And since you can request a new IP after 15mn, it can be a problem is you need high volume.

On the other hand, we found their proxies very reliable, and the speed is awesome.

Results (full benchmark)

1000 requests were sent for each website:

Website errors blocked success average time
Instagram 10 15 975 2.66
Google 0 1000 0
Amazon 1000 0 0
Top 1000 Alexa 1000 0 0

3) Luminati

Luminati is the biggest player in the proxy industry. They have all kinds of different proxy types, Data center, Residential and of course, mobile proxies. They get their mobile proxy with a “brilliant” idea:

Luminati offers an SDK for mobile developers in order to monetize apps. Instead of running ads, they can monetize their mobile apps by installing the Luminati SDK and letting Luminati share a small part of users’ bandwitdth. They claim to have access to 72 million mobile IP addresses that way, all around the world.

The different options with Luminati mobile proxies are insane:

  • Country-level targeting
  • City-level targeting
  • Carrier (AT&T, Verizon…)

Luminati offers an extensive dashboard, and a great documentation. The major problem is the price. You will have to pay $42/gb of bandwidth, and it goes very quickly. Let's say you're scraping a website with a 3mb pages, that consume 1mb of bandwitdth (after Gzip/Brotli compression), it will cost you $42 per 1000 pages.

Results (full benchmark)

1000 requests were sent for each website:

Website errors blocked success average time
Instagram 17 16 967 5.66
Google 32 32 968 1.32
Amazon 113 0 887 8.62
Top 1000 Alexa 47 0 953 5.50

Overall the results are good, but we didn't expect so many errors on Amazon. We suspect that they don't assign the same IP pools on every website.

4) Smartproxy

Smartproxy is another big player in the mobile proxy game. They don't have a pure mobile proxy offering, but most of their proxies are from mobile devices. Here are the different options Smart proxy offers:

  • Country level targeting
  • City level targeting in the US
  • Sticky sessions
  • Great customer service (they often answer in less then 3 minutes)

The bandwidth cost is much lower than Luminati and the performance is similar in our benchmark.

Results (full benchmark)

1000 requests were sent for each website:

Website errors blocked success average time
Instagram 8 62 931 3.57
Google 99 0 901 5.09
Amazon 82 4 914 5.65
Top 1000 Alexa 139 0 861 5.31

5) Oxylabs

Oxylabs is a residential proxy provider. One of the biggest ones. They have residential proxies in most countries. As with SmartProxy, they claim that most of their residential proxies are mobile IPs, and the other part comes directly from Internet service providers.

They are aware that the residential proxy acquisition process is a sensitive subject, and they set explicit contractual obligations with their proxy providers, in order to make sure that the end-users are aware and that consent is documented.

Here are some features they offer:

  • Residential IPs (mostly mobile)
  • City and country-level targeting

Oxylab is blocking Google by default so we couldn't measure it. For Instagram, we were disappointed by the results, as about 30% of requests were blocked.

Results (full benchmark)

1000 requests were sent for each website:

Website errors blocked success average time
Instagram 4 332 664 5.04
Google - - -
Amazon 47 17 936 5.75
Top 1000 Alexa 120 4 876 4.47

Benchmark

Now that we have our list of mobile and 4g proxy providers, it's time to benchmark against real websites. Generally, mobile proxies are great to extract data from social media, so we have included Instagram in the benchmark. We used a few Instagram profiles from well-known celebrities. The other domains that we tested are Google, Amazon and the top 1000 Alexa rank (the list of the most visited domains in the world).

In order to do that, we made a small script that uses one of the proxy providers, and perform one thousand requests on the target domain. For each response, we look for HTTP codes on the proxy server (5**), but also HTTP codes that indicate the request was blocked (403, 429). We also checked for known block messages, for example Amazon is sending 200 HTTP codes with a CAPTCHA inside the HTML, Instagram is redirecting to the login page when they block a request etc.

You can find the full benchmark code here: https://github.com/ScrapingBee/freeproxylist-blogpost/blob/master/benchmark_mobile_proxies.py

Instagram

proxy provider errors blocked success average time
Airproxy 10 15 975 2.66
Luminati 17 16 967 5.66
Oxylabs 4 332 664 5.04
Smartproxy 8 62 931 3.57
Scrapingbee 45 0 955 3.3

Google

proxy provider errors blocked success average time
Airproxy - - -
Luminati 32 32 968 1.32
Oxylabs - - -
Smartproxy 99 0 901 5.09
Scrapingbee 80 0 920 8.30

*Using Scrapingbee Google API

Amazon

proxy provider errors blocked success average time
Airproxy - - -
Luminati 113 0 887 8.62
Oxylabs 47 17 936 5.75
Smartproxy 82 4 914 5.65
Scrapingbee 22 0 978 3.34

Top 300 Alexa Rank

proxy provider errors blocked success average time
Airproxy - - -
Luminati 47 0 953 5.50
Oxylabs 120 4 876 4.47
Smartproxy 139 0 861 5.31
Scrapingbee 5 0 995 3.34

Analysis

This was a quick benchmark of the most serious mobile providers on the market. One of the difficult things when choosing a mobile proxy provider is that lots of them are built for social media automation and not web scraping. When you scrape the web, you want as many clean IP addresses as possible, and reasonnable bandwidth price. With social media automation/bots, 4g proxy provider tends to offer only one IP address, that doesn't necessarily change.

This will get you quickly blocked on websites with IP rate-limits restrictions like Google or Amazon. That's why you need to be careful when choosing a provider. The first 1000 requests can work, and then you start getting blocked.

The other big issue is the price. The providers with the biggest amount of proxies, like Luminati and Oxylabs are charging a very high price for the bandwidth. They have the best options and are very flexible (especially Luminati), but the price can be prohibitive.

As you can see in the benchmark, there are significant differences between those providers, especially regarding the success rate, error rate, speed, and price. That's all the things you will need to consider before making a decision.

If you liked this article, don't hesitate to check out this one about the best mobile proxies for web scraping.

Happy Scrapping :)

Tired of getting blocked while scraping the web? Our API handles headless browsers and rotates proxies for you.