Scrolling and loading more content via a JS scenario

Certain websites may require you to scroll in order to load more results on the page or within a specific element.



This is a quick guide on how to achieve different scrolling behaviors using JavaScript scenario.
*Note that the JavaScript Scenario has a maximum execution time limit of 40 seconds. Requests exceeding this limit will result in a timeout: https://www.scrapingbee.com/documentation/js-scenario/#timeout

1. Scrolling a Specific Element

Some page elements, such as tables or graphs, may contain content that only becomes visible after scrolling.

You can scroll a specific element using the following JavaScript function:

{
  "evaluate": "document.querySelector('#example').scrollTop = document.querySelector('#example').scrollHeight"
}

In this function, replace '#example' with the appropriate CSS selector for the element you wish to scroll.

2. Setting Up Continuous Scrolling

In some cases, not all content on a site is loaded initially—it may require continuous scrolling to fully load. This can be achieved with the following function:

{
  "evaluate": "let lastHeight = 0, stableTime = 0, interval = 700; let scroller = setInterval(() => { window.scrollBy(0, 1500); let currentHeight = document.body.scrollHeight; if (currentHeight !== lastHeight) { lastHeight = currentHeight; stableTime = 0; } else { stableTime += interval; if (stableTime >= 5000) clearInterval(scroller); } }, interval);"
}

In this script:

• window.scrollBy(0, 1500) scrolls the page down by 1500 pixels.
• interval = 700 sets the scroll delay to 700 milliseconds.
• The script automatically stops if no new content is loaded after 5 seconds, as defined by stableTime >= 5000.

3. Scrolling the Page and Clicking to Load More Content

Some pages load additional content only after clicking a “Load more” or “Show more” button.



The following function finds the element and clicks it repeatedly until all content is loaded:

{
  "evaluate": "(async()=>{let s='#example';while(1){let b=document.querySelector(s);if(!b||getComputedStyle(b).display==='none')break;b.scrollIntoView();b.click();await new Promise(r=>setTimeout(r,1000));}})();"
}

In this case, set s = '#example' to match the CSS selector of the button. The function scrolls to the button, clicks it, waits 1 second, and repeats the process until the button is no longer visible or present on the page.