499 status code - what is it and how to avoid it?

A 499 status code refers to "client closed request" error. This is a client-side code where the client did not wait long enough for the server to respond. It generally occurs in reverse proxy setups where NGINX is acting as a reverse proxy for a UWSGI or similar upstream server and did not wait long enough for the server to return the response.

If the website is working fine under normal settings then the chances are that your requests might be getting identified as being automated. In such a scenario, make sure you are using undetected-chromedriver or a similar tool and pairing it up with proxies from a reliable proxy provider like ScrapingBee. Or better yet, use ScrapingBee's web scraping API and let us handle the task of not getting blocked. This should help solve the issue.

Related Web scraping blocked web scraping questions: