How to use a Proxy with HttpClient in C#

How to use a Proxy with C# HttpClient

In this article, you will learn about how to use C#'s HttpClient library behind a proxy. HttpClient comes by default with C# and provides asynchronous requests out of the box which makes it a very attractive option as an HTTP client. Developers such as yourself prefer to use proxies while making web requests to retain anonymity and to prevent their primary IP from being blocked due to excessing web scraping. Whatever your reasons might be, you will be well equipped to use authenticated and unauthenticated proxies with HttpClient after going through this tutorial.

Creating a new C# project

Let's start by creating a new C# project. I am using Visual Studio for Mac and creating a new console application using .net 6.

Once the new project is created, you will be greeted by the new console application template in the editor:

// See https://aka.ms/new-console-template for more information
Console.WriteLine("Hello, World!");

If you haven't used top-level statements like this before and are wondering where the Main declaration is then it might be a good idea to read more about this new template in the official docs. We will be using the latest features in C# and .Net in this tutorial.

Making a basic HTTP request

You can create a basic HTTP request using HttpClient like this:

using var client = new HttpClient();

var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);

The first line instantiates an HttpClient object. The next line uses the GetStringAsync method to send a GET request to https://api.ipify.org/, and the last line prints the result.

Save the file and run the code. You should see your public IP in the terminal like below:

Using a proxy with HttpClient

You can use a basic, unauthenticated, proxy with HttpClient like this:

using System.Net;

var proxy = new WebProxy
{
    Address = new Uri($"http://172.104.241.29:8081"),
    BypassProxyOnLocal = false,
    UseDefaultCredentials = false,
};

// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
    Proxy = proxy,
};

// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);

You need to create a custom WebProxy object with the proxy details, pass that to a custom HttpClientHandler, and finally pass that handler to HttpClient. I also disable SSL verification as some free proxies cause an SSL verification error if you try to connect to them securely.

Multiple sources list free public proxies. I took the above proxy from a similar list hosted by Free Proxy List. Running the code above should result in the proxy host being printed in the terminal:

172.104.241.29

Using an authenticated proxy with HttpClient

You can use authenticated proxies with HttpClient as well that require a user/pass combination. Just create a new NetworkCredential Object and pass that to WebProxy. HttpClient will automatically use these credentials when connecting to the proxy server.

using System.Net;

var proxy = new WebProxy
{
    Address = new Uri($"http://172.104.241.29:8081"),
    BypassProxyOnLocal = false,
    UseDefaultCredentials = false,
    
    // Proxy credentials
    Credentials = new NetworkCredential(
        userName: proxyUserName,
        password: proxyPassword)
};

// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
    Proxy = proxy,
};

// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);

If you run the above code (after replacing the placeholder proxy details with working proxy details), it should print a similar output as before:

172.104.241.29

Rotating proxies with requests

It is always a good idea to rotate your proxies with successive requests. This decreases the likelihood of any one proxy being blacklisted. You can do this by simply defining a list of proxies and using a random proxy with each new request.

using System.Net;

List<Dictionary<string, string>> proxy_list = new List<Dictionary<string, string>> {
    new Dictionary<string, string> {
        {"protocol", "http"},
        {"host", "204.185.204.64"},
        {"port", "8080"},
    },
    //...
    new Dictionary<string, string> {
        {"protocol", "http"},
        {"host", "132.129.121.148"},
        {"port", "8080"},
    },
    new Dictionary<string, string> {
        {"protocol", "http"},
        {"host", "154.129.98.156"},
        {"port", "8080"},
    },
    new Dictionary<string, string> {
        {"protocol", "http"},
        {"host", "211.129.132.150"},
        {"port", "8080"},
    },
    new Dictionary<string, string> {
        {"protocol", "http"},
        {"host", "164.129.114.111"},
        {"port", "8080"},
    }
};

// Generate a random number between 0 and 4 (inclusive)
Random rnd = new Random();
var randomIndex = rnd.Next(proxy_list.Count);

var proxy = new WebProxy
{
    Address = new Uri($"{proxy_list[randomIndex]["protocol"]}://{proxy_list[randomIndex]["host"]}:{proxy_list[randomIndex]["port"]}"),
    BypassProxyOnLocal = false,
    UseDefaultCredentials = false,
};

// Now create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
    Proxy = proxy,
};

// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);

You can alter the code above to meet your specific needs. On each run, HttpClient will use a random proxy from the proxy_list.

Using ScrapingBee proxies

If you have done some decent amount of web scraping then you know how difficult it is to get a list of premium unblocked proxies. ScrapingBee helps you get around that problem by allowing you to use their premium IPs for web scraping. While the most powerful features of the platform are available only in the paid version, you do get access to 1000 free credits for making proxied requests. They only charge you credits when they return a successful response.

Let's look at a quick example of how you can use ScrapingBee. Go to the ScrapingBee website and sign up for an account:

After successful signup, you will be greeted with the default dashboard. Copy your API key from this page and start modifying the code in the app.js file:

You can make a request via ScrapingBee using HttpClient like this:

using System.Net;

var proxy = new WebProxy
{
    Address = new Uri($"http://proxy.scrapingbee.com:8886"),
    BypassProxyOnLocal = false,
    UseDefaultCredentials = false,
    
    // Proxy credentials
    Credentials = new NetworkCredential(
        userName: "YOUR_API_KEY",
        password: "render_js=False&premium_proxy=True")
};

// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
    Proxy = proxy,
};

// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);

Note: Don't forget to replace YOUR_API_KEY with your API key from the ScrapingBee dashboard.

The password field is used to configure the various parameters supported by the ScrapingBee API. In this example, I am disabling JS rendering and asking ScrapingBee to use premium proxies.

Running the above code should print a random IP address in the terminal. Successive requests will print different IP addresses as ScrapingBee automatically rotates IP addresses so you don't have to do that on your own. ScrapingBee's proxy mode is so economical and easy to use that there is no reason not to give it a try!

Conclusion

In this article, you learned how to use a proxy with HttpClient in C#. You saw how you can use an authenticated proxy and how you can write some basic C# code to use a random proxy from a list. You also learned that our web scraping API can take a lot of pain out of this process of using proxies and abstracts away most of the complexity of finding reliable, unblocked proxies, and rotating them.

Even though Python is more famous for web scraping, C# isn't too far behind either. There are multiple libraries like the HTML Agility Pack library that provide support for parsing and extracting data from HTML documents. Give web scraping in C# a go and let us know about your experience!

Additional resources

• The 5 Best Free Proxy Lists for Web Scraping
• Top 5 Rotating and Residential Proxies for Web Scraping
• Getting started with ScrapingBee and C#
• Web Scraping in C#