Scrolling via page API

Some pages load more content only after you click “Load more results” or scroll and wait. In reality, the page often fetches additional results from its own API. If ScrapingBee can’t load those results, you can target the site’s API URL directly.

Here’s how to do that using this URL as an example: https://www.reuters.com/technology

*Note that the JavaScript Scenario has a maximum execution time limit of 40 seconds. Requests exceeding this limit will result in a timeout: https://www.scrapingbee.com/documentation/js-scenario/#timeout

1. Open the page and inspect it. In DevTools, go to the Network tab.

2. Click Load more articles. You’ll see a new request appear in the Network panel.

3. Copy the Request URL. In this case, it exposes an API endpoint that returns JSON.

Example request URL:

https://www.reuters.com/pf/api/v3/content/fetch/articles-by-section-alias-or-id-v1?query={"arc-site"%3A"reuters"%2C"fetch_type"%3A"collection"%2C"offset"%3A20%2C"section_id"%3A"%2Ftechnology%2F"%2C"size"%3A9%2C"website"%3A"reuters"}&d=301&mxId=00000000&_website=reuters

Opening that URL returns the content in JSON.

In practice, you may not need all the parameters. From our testing, this is sufficient:

https://www.reuters.com/pf/api/v3/content/fetch/articles-by-section-alias-or-id-v1?query={"arc-site":"reuters","fetch_type":"collection","offset":47,"section_id":"/technology/","size":9,"website":"reuters"}

On Reuters’ side, content is paginated using the offset and size parameters. The maximum appears to be 20 items per page (controlled by size), starting from offset 0. If you want page 2, use offset 19; for page 3, use offset 39, and so on.

Keep in mind that every site is different. This approach works for Reuters, but other sites can use different endpoints, parameters, and pagination rules.