Scraper doesn't see the data I see in the browser - why?

This issue can often show up when you are using an HTML parser like BeautifulSoup or lxml instead of a browser engine via Selenium or Puppeteer. The data you are seeing in the browser might be getting generated via client-side JavaScript after the page load. BeautifulSoup, lxml, and similar HTML parsing libraries do not execute JavaScript.

There are two options to solve this issue:

  1. Use a browser automation framework like Selenium or Puppeteer and execute the JavaScript before attempting data extraction
  2. Search for required data in the <script> tags. Most of the time, the required data is hidden inside <script> tags as JavaScript variables and then rendered on the page after the page load

Related Data Parsing web scraping questions: