Getting Started with Web Scraping in C# using ScrapingBee

In this tutorial, you'll learn how to use ScrapingBee for C# website scraping. You'll make an API request, scrape a website with C#, and save the returned HTML.

1. Make a Web Scraping API Request in C#

The following C# web scraping example sends a request to ScrapingBee, retrieves the HTML of the target webpage, and prints the response to the console:

using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
namespace Test
{
    class Program
    {
        private static readonly string BASE_URL =
            "https://app.scrapingbee.com/api/v1/";
        private static readonly string API_KEY =
            "YOUR-API-KEY";
        public static async Task<string> Get(string url)
        {
            string uri =
                $"{BASE_URL}?url={Uri.EscapeDataString(url)}";
            using HttpClient client = new HttpClient();
            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Bearer", API_KEY);
            HttpResponseMessage response =
                await client.GetAsync(uri);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync();
        }
        public static async Task Main(string[] args)
        {
            Console.WriteLine("Response Body:");
            string html = await Get("https://example.com");
            Console.WriteLine(html);
        }
    }
}

In the above code, Get() builds the request with your target URL and sends your API key in the Authorization header as a Bearer token. It then sends the request with HttpClient and returns the scraped HTML as a string.

The Main() method calls Get() with the target URL and prints the returned HTML to the console.

2. C# Web Scraping Example

The following C# web scraping example scrapes the ScrapingBee blog and saves the returned HTML locally.

using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
namespace Test
{
    class Program
    {
        private static readonly string BASE_URL =
            "https://app.scrapingbee.com/api/v1/";
        private static readonly string API_KEY =
            "YOUR-API-KEY";
        public static async Task<string> Get(string url)
        {
            string uri =
                $"{BASE_URL}?url={Uri.EscapeDataString(url)}";
            using HttpClient client = new HttpClient();
            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Bearer", API_KEY);
            HttpResponseMessage response =
                await client.GetAsync(uri);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync();
        }
        public static async Task Main(string[] args)
        {
            string targetUrl =
                "https://www.scrapingbee.com/blog/";
            string output = await Get(targetUrl);
            string path = "./ScrapingBeeBlog.html";
            await File.WriteAllTextAsync(path, output);
            Console.WriteLine($"Saved response to {path}");
        }
    }
}

This request sends the ScrapingBee blog URL to the API, which retrieves the page and returns its HTML. The C# code then saves the response locally as ScrapingBeeBlog.html.

To reuse this C# website scraping setup, replace the target URL with the page you want to scrape. You can then parse, process, or store the returned HTML as needed.

Note for production scraping: These examples create a new HttpClient for a single request to keep the code simple. If you're building a production scraping application that makes multiple requests, it's better to reuse an HttpClient instance rather than creating a new one for every request.

Go back to tutorials