A 429 status code refers to the Too Many Requests error. It might be thrown by the server if the user has made excessive requests in a short amount of time and the server is using rate-limiting. The best way to avoid this error is to do either of these two things:
- Throttle your requests. Make sure you are making only a few requests in a given timeframe so as not to hit the rate-limit
- Distribute your requests across proxies so that they all go from different IPs and don't trigger the rate-limit
For the second option, you can use ScrapingBee's reliable proxies to make sure they aren't part of any blocklist. Or better yet, use ScrapingBee's web scraping API and let us handle the task of not getting blocked.