Retry failed requests in Python

For most websites, your first requests will always be successful, however, it’s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won’t charge you for the request.

In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:

from scrapingbee import ScrapingBeeClient # Importing SPB's client
client = ScrapingBeeClient(api_key='YOUR-API-KEY') # Initialize the client with your API Key, and using screenshot_full_page parameter to take a screenshot!

MAX_RETRIES = 5 # Setting the maximum number of retries if we have failed requests to 5.

for _ in range(MAX_RETRIES):
    response = client.get("", params={'screenshot': True}) # Scrape!

    if response.ok: # If we get a successful request
        with open("./screenshot.png", "wb") as f:
            f.write(response.content) # Save the screenshot in the file "screenshot.png"
            break # Then get out of the retry loop
    else: # If we get a failed request, then we continue the loop
Go back to tutorials