Docs

How to find elements by CSS selector in Selenium?

Selenium is a popular browser automation framework that is also used for scraping data using headless browsers. While using Selenium, one of the most popular things to do is use CSS selectors to select particular HTML elements to interact with or extract data from.

Using Browser Developer Tools To Find CSS Selectors

To scrape content or fill in forms using Selenium, we first need to know the CSS selector of the HTML element we'll be working with. To find the CSS selector, we need to go through the HTML structure of the web page, which could be confusing and cumbersome. Most modern browsers provide developer tools to make this easier.

For example, in Chrome we can get the CSS selectors using the following steps:

  1. Right-click the element of interest, and select 'Inspect' from the drop-down menu
  2. This will open up the HTML tree in a sidebar, with the element selected in the code.
  3. Right-click the element in the tree, select "Copy" > "Copy Selector".
  4. The CSS selector for that element will now be copied to the clipboard and can be pasted anywhere.

A detailed explanation of this process is available in our tutorial on extracting CSS selectors using Chrome. Now that we know how to find the CSS selectors we need, let's proceed to set up Selenium and use these selectors.

Installation And Setup Of Selenium

To get Selenium up and running, we first need to download the browser and its corresponding driver. For Chrome, we can download the browser first, and get the ChromeDriver from https://developer.chrome.com/docs/chromedriver/downloads.

Next, we can install the Python bindings for Selenium using pip, uv, or any other package management system:

$ pip install selenium
OR
$ uv add selenium

Selenium: find_element vs find_elements

You can find elements by CSS selectors in Selenium by utilizing the find_element and find_elements methods. find_element returns the first occurrence of the CSS selector being used, while find_elements returns all elements of the page that match the selector. Now that we've understood the difference between the two functions, let's look at some code next.

CSS Selectors Examples With Selenium

Here is some example code that navigates to the ScrapingBee website and extracts some data using CSS selectors:

from selenium import webdriver
from selenium.webdriver.common.by import By

DRIVER_PATH = '/path/to/chromedriver'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)

# Open Scrapingbee's website
driver.get("http://www.scrapingbee.com")

# Get the first h1 element using find_element
first_h1 = driver.find_element(By.CSS_SELECTOR, "h1")

# Get all p elements using find_elements
all_p_elements = driver.find_elements(By.CSS_SELECTOR, "p")

# Get all a elements with the class of btn
all_btn_elements = driver.find_elements(By.CSS_SELECTOR, "a.btn")

# Get all internal links
all_internal_links = driver.find_elements(By.CSS_SELECTOR, "a[href^='/']")

# Get all spans containing an icon
features = driver.find_elements(By.CSS_SELECTOR, "span:has(> i)")

How To Print Text And Attributes From Elements

Once we've selected elements, we can get the text from these elements using the .text attribute or get other HTML attributes using the get_attribute function. Continuing the code we wrote above, let's print some information from the elements we selected:

# Get the textContent of the h1 element
h1_value = first_h1.text

# The total number of p elements on the page
p_total = len(all_p_elements)

# The total number of a.btn elements on the page
btn_elements_count = len(all_btn_elements)

print(h1_value)
print(p_total)
print(btn_elements_count)

# Print all internal links
for a in all_internal_links:
    print(a.get_attribute('href'))

# Print feature list (spans with icons)
for span in features:
    print(span.text)

Alternatives to Selenium for selecting elements with CSS

Selenium is not the only tool that supports CSS selectors. CSS selectors can also be used with other libraries and frameworks in Python. We've listed a few of these below.

  • BeautifulSoup: This is the most popular Python library used to parse HTML strings and work with the HTML elements inside them. We cover this in depth in our BeautifulSoup tutorial.
  • Scrapy: Scrapy is a popular, opinionated, beginner-friendly web crawling framework for Python. It uses a spidering (graph traversal) algorithm, starting with one page, and discovering new pages to visit on each page it visits. We discuss this tool in detail on our blog about easy web scraping with Scrapy.
  • LXML + CSSSelect: LXML is a low-level XML processing engine that can be used to work with HTML data. The cssselect module allows using CSS selectors with the LXML library. Together, the setup can be used for efficient scraping, but there could be some issues with parsing improperly formatted HTML documents.

Related Selenium web scraping questions: