In this article, we will look at the top five proxy list websites out there and perform a benchmark. The idea is not only to talk about the different features they offer, but also to test the reliability with a real world test. We will look and compare the response times, errors and success rates on popular websites like Google and Amazon.
There is a proxy type to match any specific needs you might have, but you can always start with a free proxy server. This is especially true if you want to use it as a proxy scraper.
A free proxy server is a proxy you can connect to without needing special credentials and there are plenty online to choose from. The most important thing you need to consider is the source of the proxy. Since proxies take your information and re-route it through a different IP address, they still have access to any internet requests you make.
While there are a lot of reputable free proxies available for web scraping, there are just as many proxies that are hosted by hackers or government agencies. This is still a third-party you are sending your requests to, so they have a chance to see all of the unencrypted data that comes from your computer or phone.
You might want to gather information through web scraping and not want websites tracking your bots or maybe you need to bypass rate limits. No matter what reasons you want your privacy online, there's a way for you to get it.
You can get around many of these privacy concerns by using a proxy server. Proxies help keep your online activity secure by routing all of your requests through a different IP address. That means when websites try to track you, they won't be able to because they don't have the original IP address your request came from.
When you do find a trustworthy free proxy, there are still some issues with using them. They could return responses incredibly slow if there are many users on the proxy at the same time. Some of them are unreliable and might disappear without warning and never come back. Proxies can also inject ads into the data that gets returned to your computer.
In the context of web scraping, most users start out with a free proxy. Usually you aren't sending any sensitive information with your requests so many people feel comfortable using them for this purpose. Even though you aren't sending any personal information, you still might not want a website to know that you are scraping it for its data.
You could be doing market research and you want to learn more about your competition through web scraping. Many users don't want a website to know about that kind of activity. One big reason users turn to free proxies for web scraping is because they might not plan to do it often. You can get the information you need from a site and then disconnect from the proxy without any issues.
While free proxies are great for web scraping, they are still unsecure. A malicious proxy could alter the HTML of the page you requested and give you false information. You also have the risk that the proxy you are currently using can disconnect at any time without warning. The proxy IP address you're using could get blocked by websites if there are a lot of people using it for malicious reasons.
Free proxies have their uses and there are thousands of lists available with free proxy IP addresses and their statuses. Some lists have higher quality proxies than others and you also have the option to use specific proxy services. You'll learn about several of these lists and services to help you get started in your search for the best option for your proxy scraper.
I know I know… It sounds a bit pushy to immediately talk about our service but this article isn't an ad. We put a lof of time and effort into benchmarking those services, and I think it is fair to compare those free proxy lists to ScrapingBee API.
If you're going to use a proxy for web scraping, ScrapingBee is one of the best services. While some of the best features are in the paid version, you can get 1000 free credits when you sign up. What makes this service stand out is that even free users have access to support and the IP addresses you have access to are more secure.
One of the coolest features is that they have rotating proxies so that you can get around rate-limiting websites, which helps you hide your proxy scraper bots and lowers the chance you'll get blocked by a website.
You can also find code snippets in Python, NodeJS, PHP, Go, and several for web scrapers. ScrapingBee even has its own API that makes it even easier to do web scraping. You don't have to worry about security leaks or the proxy running slow because access to the proxy servers is limited.
You can customize things like your geolocation, the headers that get forwarded, cookies that are sent in the requests, and they automatically block ads and images to speed up your requests.
Another cool thing is that if your requests return a status code other than 200, you don't get charged for that credit. Unless you get a response, you don't have to pay for the request.
Even though the free proxy scrapers you can use with ScrapingBee are great, if you plan on using proxy scraping a lot you will need to upgrade to a paid plan. Then of course, if you have any problem you can get in touch with the team to find out what happened.
With the free proxies on the lists below, you won't have any support at all. You'll be responsible for making sure your information is secure and you'll have to deal with IP addresses getting blocked and requests returning painfully slow as more users connect to the same proxy.
|Top 300 Alexa||5||0||995||3.34|
If you're looking for a list of completely free proxies, Proxyscrape is one of the leading free proxy lists available. One really cool feature is that you can download the list of proxies to a
.txt file. This can be useful if you want to run a lot of proxy scrapers at the same time on different IP addresses.
You can even filter the free proxy lists by country, level of anonymity, and whether they use an SSL connection. This lets you find the kind of proxy you want to use faster than many other lists where you have to scroll down a page, looking through table columns.
Proxyscrape even has different kinds of proxies available. You still have access to HTTP proxies and you can find lists of Socks4 and Socks5 proxies. There aren't as many filters available for Socks4 and Socks5 lists, but you can select the country you want to use.
The Proxyscrape API currently works with Python and there are only four types of API requests you can make. An important thing to remember is that none of the proxies on any of the lists you get from this website are guaranteed to be secure. Free proxies can be hosted by anyone or any entity, so you will be using these proxies at your own risk.
They do have a premium service available where they host datacenter proxies and these are typically more secure than the free ones. They do more monitoring on these proxies to make sure that you have consistent uptime and that the IP addresses don't get added to blocklists.
Another nice tool they have is an online proxy checker. This lets you enter the IP addresses of some of the free proxies you've found and test them to see if they are still working. When you're trying to do web scraping you want to make sure that your proxy doesn't disconnect in the middle of the process and this is one way you can keep an eye on the connection.
|Top 300 Alexa||551||1||448||13.60|
free-proxy.cz is one of the original free proxy list sites. There hasn't been much maintenance on the website so it still has the user interface of an early 2000's website, but if you're just looking for free proxies it has a large list. One thing you'll find here that's different from other proxy list sites is a list for free web proxies.
Web proxies are usually run on server-side scripts like PHProxy, Glype, or CGIProxy. The list is also pre-filtered for duplicates so there aren't any repeating IP addresses. The list of other proxy servers they have in their database is unique.
On the homepage there is a table with all of the free proxies they have found. You can filter the proxies by country, protocol, and anonymity level. You can sort the filtered table by the proxy speed, uptime, response time, and the last time the status was checked. The table shows paginated results, so taking advantage of the sort function will save you some time.
There's also a “proxies by category” tool below the table that lets you look at the free proxies by country and region. This makes it easier to go through the table of results and find exactly what you need. This is the best way to navigate this list of free proxies because there are thousands available.
Another useful tool on this site is the “Your IP Address Info” button at the top of the page. It will tell you everything about the IP address you are using to connect to the website. It'll show you the location, proxy variables, and other useful information on your current connection. It even goes as far as showing your location on Google Maps. This a good way to test a proxy server you're connected to.
This site doesn't offer any premium or paid services so connecting to any free proxies you find here has no guarantees of always being online or any security measures to protect your proxy scraping activities.
|Top 300 Alexa||742||0||258||12.73|
GatherProxy (proxygather.com) is another great option for finding free proxy lists. It's a bit more organized than many of the lists you'll find online. You can find proxies based on country or port number. There are anonymous proxies and web proxies. They have a separate section for socks lists.
There are several free tools this site offers as well, like a free proxy scraper. You can download the tool, although it hasn't been updated in a few years. It's a good starting point if you are trying to build a proxy scraper or do web scraping in general. There is also an embed plugin for GatherProxy that lets you add a free proxy list to your own website if that would be useful for you.
If you want to check your IP address or browser information, they also have a tool to show you that information. It's not as detailed as the IP address information you see on free-proxy.cz, but it still gives you enough information to find what you need.
Another tool you can find on this site is the proxy checker. It lets you find, filter, and check the status of millions of proxies. You can export all of the proxies you find using this tool into a number of different formats, like CSV. There are some great videos on GatherProxy that show you how to use these tools.
The main difference between this site and a lot of the others is that you have to enter an email address before you can browse through the lists of free proxies they have. It's still a completely free service, but you have to sign up and get login credentials. Once you do that, you'll be able to see the tables of free proxies and sort them by a number of parameters.
You also have the option to download the free proxy lists after you sort and filter them based on your search criteria. One nice feature they have is that they auto-update the proxy lists constantly so you don't have to worry about getting a list of stale IP addresses.
(At the time of writing, this service was down)
Freeproxylists service is very simple to use. The homepage brings up a table of all of the free proxy that has been found. Like many of the other sites in this post, you can sort the table by country, port number, uptime, and other parameters. The results are paginated, so you'll have to click through multiple pages to see everything available.
It does have a straight-forward filtering function at the top of the page so you can limit the number of results shown in the table. If using a proxy from a specific country is a concern, then you can go to the “By Country” page of the site. It'll show you a list of all of the countries the free proxies represent and the number of proxies available for that country.
One downside is that you won't be able to download this proxy list from this website. This is probably one of the more basic free proxy lists you'll find online for your web scrapers. This service does have a good reputation compared to the thousands of other lists available so the proxies you find here at least work.
Even with a decent reputation as a site for free proxy lists, always remember that there is a risk involved with using proxies hosted by entities you don't know. This list is updated frequently if you check back occasionally, but they don't share how often the list is updated. You'll find the free proxies here, but it would be best to use a different tool to check if the proxy you want to use is still available.
There is an email address available on the site if you have questions, although you shouldn't expect the fastest response time. Unlike some of the other free proxy sites, there aren't any paid or premium versions of the proxy lists or any additional tools, like proxy scrapers.
|Top 300 Alexa||483||0||517||10.90|
Now that we saw the different free proxies available on the market, it is time to test them against different websites. The benchmark is simple. We made a script that collects free proxies from this list (it has to be dynamic and get the latest proxy, since the list changes every few hours on those websites.). Then we have a set of URLs for some popular websites like Google, Amazon, Instagram and 300 URLs from the top 1000 Alexarank. We then go to each URL using the proxy list and record the response time / HTTP code and eventual blocking behavior on the website.
For example, Google will send a 429 HTTP code if they block an IP, Amazon will return a 200 HTTP code with a Captcha in the body, and Instagram will redirect you to the login page.
You can find the script here: https://github.com/ScrapingBee/freeproxylist-blogpost
We ran the script on each proxy list with different websites, 1000 requests each time and came to this result.
|proxy list||errors||blocked||success||average time|
|proxy list||errors||blocked||success||average time|
*Using Scrapingbee Google API
|proxy list||errors||blocked||success||average time|
|proxy list||errors||blocked||success||average time|
The biggest issue with all these proxies was the error rate on the proxy. Timeouts, network error, HTTPS… you name it. Then especially for Google and Instagram, most of the requests were blocked with the “working” proxies (meaning proxies that don't produce timeouts, network errors). This can be explained by the fact that Google is heavily scraped by tools like Scrapebox / Screaming Frog spider. These are SEO tools used to get keywords suggestion, scrape Google and generate SEO reports. They have a built-in mechanism to gather those free proxy lists, and lots of SEO people use them. So those proxies are over-used on Google and get blocked very often.
Overall, beside Scrapingbee of course, Freeproxylists.net seems to have the best proxies, but as you can see it's not that great either.
When you are trying to use web scraping to get information about competitors, find email addresses, or get data from a website, using a proxy will help you protect your identity and avoid adding your true IP address to any blocklists. Proxy scrapers help you keep your bots secure and crawling pages for as long as you need.
While there are numerous lists of free proxies online, not all of them contain the same quality of proxies. You still have to be aware of the risks that come with using free proxies. There's a chance you could connect to one hosted by a hacker or government agency or just someone trying to insert their ads into every response that is returned from any website. That's why it's good to use free proxy services from websites you trust.
Having a list of free proxies does give you the advantage of not dealing with blocklists because if an IP address does get blocked, you can move on to another proxy without much hassle. If you need to use the same IP address multiple times for your web scraping, then it will be worth the investment to pay for a service that has support and manages their own proxies so you don't have to worry about them going down at the worst time.