How to Wait for Page to Load in Playwright?

Wait for the specific page content you need with page.wait_for_selector() or Playwright's built-in auto-waiting. Turn to page.wait_for_load_state() when you need a specific browser load state, and keep fixed timeouts as a fallback.

Let's look at these methods to wait for page to load in Playwright, so you can use the one that best works for your task.

1. Wait for a Selector to Load in Playwright

You can use the wait_for_selector method to make the playwright wait for the page to load before fetching the HTML.

By default, Playwright waits for navigation and many page interactions, but it doesn't wait for asynchronous XHR (XMLHttpRequest) or AJAX requests that run after the initial page load. If the data you need is loaded later, wait for an element that confirms the content has finished loading before parsing the page.

Here is a sample code that searches for the hottest sneakers on Pinterest and then waits for the pins to show up before saving a screenshot of the page.

Before running the code below, install Playwright and its browser binaries:

pip install playwright
playwright install

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Launch a Chromium browser
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Open the Pinterest search page
    page.goto(
        "https://www.pinterest.com/search/pins/?rs=ac&len=2&q=hottest%20sneakers"
    )

    # Wait until the pins appear
    page.wait_for_selector("div[data-grid-item=true]")

    # Save a screenshot
    page.screenshot(path="pinterest.png")
    browser.close()

2. Playwright's Auto-Wait for Page Load

Unlike many other browser automation frameworks, Playwright automatically waits for the element you're interacting with before clicking it, taking a screenshot, or performing other actions. This is another way to wait for a page to load in Playwright without adding an explicit wait. Let's get the Pinterest screenshot using this method:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Launch a Chromium browser
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Open the Pinterest search page
    page.goto(
        "https://www.pinterest.com/search/pins/?rs=ac&len=2&q=hottest%20sneakers"
    )

    # Wait for the search results to appear
    pins = page.locator("div[data-test-id=search-feed]")
    pins.wait_for()

    # Retrieve the HTML after the content has loaded
    html = page.content()
    browser.close()

When you call locator.wait_for(), Playwright automatically waits for that element to load before performing any action. If the element doesn't appear within the timeout period, Playwright throws a timeout error.

3. Wait for Page to Load With a Fixed Timeout

You can use the page.wait_for_timeout to wait for a specified number of milliseconds. This is a straightforward method for pages that have a complicated load mechanism or HTML structure. Let's see the code:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Launch a Chromium browser
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Open the Pinterest search page
    page.goto(
        "https://www.pinterest.com/search/pins/?rs=ac&len=2&q=hottest%20sneakers"
    )

    # Wait for 10 seconds
    page.wait_for_timeout(10_000)

    # Save a screenshot
    page.screenshot(path="pinterest.png")
    browser.close()

Fixed timeouts are usually the least reliable option because they either slow down every run or fail when the page takes longer to load than expected. Use them only when the page doesn't provide a reliable selector, load state, or network response to wait for.

4. Playwright Network-Based Waiting

Playwright offers several methods to wait based on certain network conditions. For scraping, the most commonly used one is page.wait_for_load_state('networkidle'). This method waits until the page finishes all network requests triggered during the page load and the network becomes idle again. You can also use it to wait for data to load after triggering a scroll or click. Let's see some demo code below:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Launch a Chromium browser
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    # Open the Pinterest search page
    page.goto(
        "https://www.pinterest.com/search/pins/?rs=ac&len=2&q=hottest%20sneakers"
    )

    # Wait until the page finishes its network activity
    page.wait_for_load_state("networkidle")

    # Save a screenshot
    page.screenshot(path="pinterest.png")
    browser.close()

Caution: Playwright's own documentation discourages using networkidle. Pages with polling, ads, or analytics may never become idle, causing wait_for_load_state("networkidle") to wait until it times out. In that case, use the other methods discussed in this tutorial.

You can also wait for the load or domcontentloaded load state instead of networkidle. In addition, Playwright offers the wait_for_request() and wait_for_response() methods when you need to wait for a specific network request or response.

Related Playwright web scraping questions: