Retry failed requests in C#

For most websites, your initial requests will be successful with ScrapingBee. However, some requests may fail for different reasons. For example, 400, 401, and 403 errors usually indicate problems with the request or account, so retrying them won't fix the issue. Other errors, such as 500, can be temporary and may be worth retrying in C#.

In this tutorial, you'll learn how to add retry logic in C# for temporary 500 status failures. The example below requests the ScrapingBee blog, waits between failed attempts, and saves the response as an HTML file when the request succeeds.

using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;

namespace ScrapingBeeRetry
{
    class Program
    {
        private static readonly string BASE_URL =
            "https://app.scrapingbee.com/api/v1/";

        private static readonly string API_KEY =
            "YOUR-API-KEY";

        static async Task Main(string[] args)
        {
            string targetUrl = "https://www.scrapingbee.com/blog/";
            string encodedUrl = Uri.EscapeDataString(targetUrl);

            string url = $"{BASE_URL}?url={encodedUrl}";

            const int MAX_ATTEMPTS = 5;

            using HttpClient client = new HttpClient
            {
                Timeout = TimeSpan.FromSeconds(30)
            };

            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Bearer", API_KEY);

            for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++)
            {
                try
                {
                    using HttpResponseMessage response =
                        await client.GetAsync(url);

                    if (response.IsSuccessStatusCode)
                    {
                        string content =
                            await response.Content.ReadAsStringAsync();

                        string path = "./ScrapingBeeBlog.html";
                        await File.WriteAllTextAsync(path, content);

                        Console.WriteLine("Done!");
                        return;
                    }

                    Console.WriteLine(
                        $"Request failed with status code: {response.StatusCode}");

                    if ((int)response.StatusCode != 500)
                    {
                        Console.WriteLine("This error is not retryable.");
                        return;
                    }
                }
                catch (TaskCanceledException)
                {
                    Console.WriteLine("Request timed out.");
                }
                catch (HttpRequestException ex)
                {
                    Console.WriteLine($"Request failed: {ex.Message}");
                }

                if (attempt < MAX_ATTEMPTS)
                {
                    int delay = (int)Math.Pow(2, attempt) * 1000;

                    Console.WriteLine(
                        $"Retrying in {delay / 1000} seconds...");

                    await Task.Delay(delay);
                }
            }

            Console.WriteLine(
                $"Request failed after {MAX_ATTEMPTS} attempts.");
        }
    }
}

Here, MAX_ATTEMPTS is set to 5, meaning the code makes up to five total attempts. If the request succeeds, the loop stops immediately and the response body is saved to ScrapingBeeBlog.html. If it fails with a 500 error, the C# code retries the request.

The TaskCanceledException and HttpRequestException handlers cover request timeouts and network-related errors.

The example above also uses exponential backoff between attempts, so the delay increases after each failed attempt: 2 seconds, 4 seconds, 8 seconds, and then 16 seconds. If all five attempts fail, the program prints a final failure message.

Go back to tutorials