Search Engine Scraping Tutorial With ScrapingBee

Search engine scraping has become an essential method for many businesses, digital marketers, and researchers to gather information. It is an excellent data extraction method when you need to analyze a large number of competitor websites. With web scraping, you can extract information on market trends and make informed decisions on pricing strategies using the data extracted from SERPs.

In this tutorial, I’ll show you how to perform search engine scraping safely and efficiently using ScrapingBee’s web data extraction tool. You’ll learn how to extract structured data from major search engines like Google and Bing without worrying about getting blocked, managing proxies, or dealing with CAPTCHAs. Let's dive in!

Quick Answer (TL;DR)

Let me show you how to set up a Google Search API that can reliably extract data using our handy tool. Simply pick a relevant Google search page and use the code below:

import requests

def scrape_google_search(api_key, query):
    url = 'https://app.scrapingbee.com/api/v1/search'
    params = {
        'api_key': api_key,
        'engine': 'google',
        'query': query
    }

    response = requests.get(url, params=params)
    data = response.json()

    results = []
    for result in data.get('organic_results', []):
        results.append({
            'title': result.get('title'),
            'link': result.get('link'),
            'snippet': result.get('snippet')
        })
    return results

# Example usage
API_KEY = 'YOUR_API_KEY'
query = 'how to scrape Google search results'
for r in scrape_google_search(API_KEY, query):
    print(f"{r['title']} → {r['link']}")

It's really that simple. Our platform is built to take away all the tedious technical configurations from web scraping, allowing you to collect relevant data.

Don't forget to check out the Scraping Search API documentation for more details on our advanced features.

What Is Search Engine Scraping?

Search engine scraping is an automated data collection process that allows scraping websites such as Google or Bing. It works by sending requests to search engines with a specialized web scraping tool.

The web scraper then returns results in a structured format for analysis or storage. The process can be further simplified by using artificial intelligence to distill insights from the public data you collect.

Web scraping is particularly useful for businesses and researchers who need:

• SEO monitoring: Tracking keyword rankings and analyze SERP features

• Competitive analysis: Monitoring competitor pricing, visibility, and content strategy. It's especially usefull for large e-commerce websites

• Market research: Identifying trends, user behavior, and consumer interests based on search patterns, and performing sentiment analysis

• Lead generation: Finding potential customers or partners through targeted searches

• Content research: Discovering popular topics and questions providing insights relevant to your industry

Web scraping gives you access to the publicly available data that appears in search results, including titles, URLs, snippets, and sometimes additional elements like featured snippets. With the right web scraping tools, you can even collect data from complex websites, such as Google Maps.

The extracted search data can provide valuable insights that inform business decisions, pricing strategies, and content creation. Instead of manually checking search results, automated scraping saves time and delivers more comprehensive data sets.

Check out this article to learn more about what web scraping is.

Why Search Engines Are Hard to Scrape

Search engines implement sophisticated anti-bot measures to prevent automated scraping, making them among the most challenging web pages to extract data from.

Here are the main obstacles you’ll face when attempting web scraping.

1. CAPTCHAs and verification challenges: Search engines quickly detect unusual patterns and trigger “I’m not a robot” verification or more complex puzzles that stop scraping tools.

2. IP blocking: Make too many HTTP requests from the same IP address, and you’ll quickly get blocked from web scraping, sometimes for hours or days.

3. Rate limiting: Search engines restrict the number of queries you can make in a given timeframe to prevent server overload.

4. JavaScript rendering: Modern search results rely heavily on JavaScript to display content, which basic web scrapers can’t handle properly.

5. Dynamic layout changes: Search engines frequently alter their HTML structure – a headache for any web crawler.

6. User-agent detection: Using the wrong user-agent or not rotating them can trigger blocks as search engines identify bot-like behavior.

7. Browser fingerprinting: Advanced techniques identify scrapers based on browser characteristics that differ from those of real users.

These challenges make web scraping without getting blocked particularly difficult when targeting search engines. However, there's a solution. Let me introduce it below.

How ScrapingBee Data Scraper Solves These Challenges

ScrapingBee provides a specialized solution for search engine scraping that handles all the technical challenges mentioned above. Instead of building and maintaining your own web scraper, you can use ScrapingBee’s API to extract search data without issues.

Here’s how the platform addresses each challenge:

Proxy rotation and management: Our solution maintains a large pool of residential and datacenter proxies, automatically rotating them to prevent IP blocking. Your requests come from different IP addresses, making detection much harder.

Headless browser rendering: For JavaScript-heavy pages, the platform uses real Chrome browsers to render the content fully, ensuring you get the complete search results just as a human would see them.

CAPTCHA handling: The system is designed to minimize CAPTCHA triggers, and when they do appear, our API can solve many of them automatically without interrupting your data collection.

Request throttling: ScrapingBee manages request rates intelligently to avoid triggering rate limits while maximizing the data you can collect.

Structured data extraction: Instead of dealing with complex HTML parsing, you receive data in a clean, structured format that’s ready to use in Google Sheets.

Browser fingerprinting protection: The headless browsers are configured to appear as regular users, reducing the chance of being detected as a bot.

In my experience, managing proxies and handling CAPTCHAs are the most frustrating parts of search engine scraping. That's why I prefer tools that can solve this for me.

How to Scrape Search Engines With ScrapingBee

Now let me show you how to scrape both Google and Bing search results using ScrapingBee’s API. While the process is similar for both search engines, there are some differences in the parameters and response structure that we’ll cover.

Before we start coding, you’ll need to set up your environment:

1. Create an account: Sign up at ScrapingBee.com to get your API key. You'll get a free trial with enough credits to test the examples in this tutorial.

2. Install Python and required packages: Make sure you have Python 3.6+ installed. We’ll use the requests library to make API calls, which you can install with:

pip install requests

1. Understand the API structure: ScrapingBee’s search API uses a dedicated endpoint (/api/v1/search) that accepts parameters to customize your search queries.

For both Google and Bing, you can customize various aspects of your search, including:

• The search query

• Number of results to return

• Language and country settings

• Device type (mobile or desktop)

• SafeSearch settings

In the following sections, we’ll look at specific examples for Google and Bing.

Example: Scraping Google Search Results

Let’s dive into scraping Google Search engine. The process involves understanding the endpoint, setting the required parameters, making the request, and parsing the results.

Step 1: The Search Endpoint

Our Search API provides a dedicated endpoint for extracting search engine results directly from Google, Bing, and other web pages.

Here's the endpoint:

https://app.scrapingbee.com/api/v1/search

By using this endpoint, you don’t need to worry about proxy rotation, solving CAPTCHAs, or writing your own data scraping logic.

Step 2: Configure the Required Parameters

When making a request, you must include the following parameters:

• api_key: Your personal API key.

• engine: The search engine you want to query (e.g., Google).

• query: The keyword or phrase you want to search for.

He're a list of additional parameters:

• num: Number of results to return (default is 10, max is typically 100)

• hl: Language code for the search results

• gl: Country code to localize search results

• location: More specific geographic targeting

• safe: SafeSearch filtering level

These parameters allow you to customize your search to match exactly what you’d see if you were searching manually from a specific location and language setting.

Step 3: Make the Request

Here’s a sample request to the endpoint with multiple parameters:

import requests

params = {
    'api_key': 'YOUR_API_KEY',
    'engine': 'google',
    'query': 'best Python scraping libraries',
    'num': 20,  # Get 20 results
    'hl': 'en',  # Language: English
    'gl': 'us',  # Country: United States
    'location': 'San Francisco,California',  # More specific location
    'safe': 'active'  # Enable SafeSearch
}

Step 4: Parse the Results

Once the request is sent, you can process the JSON response and extract the relevant information:

response = requests.get('https://app.scrapingbee.com/api/v1/search', params=params)
data = response.json()

for result in data.get('organic_results', []):
    print(f"Title: {result['title']}")
    print(f"Link: {result['link']}")
    print(f"Snippet: {result.get('snippet')}\n")

The response contains structured data in JSON format, so data points map cleanly to titles, links, and snippets. The organic_results field contains a list of search results, each with fields like title, link, and snippet.

By the way, if Google changes its layout, check out the Google SERP Scraping Tutorial for updated code and additional tips.

Example: Scraping Bing Search Results

Our API supports Bing through the same /search endpoint by setting the engine parameter. The process is very similar to scraping Google, with a few parameter differences.

Here's the code example you should use:

import requests

params = {
    'api_key': 'YOUR_API_KEY',
    'engine': 'bing',
    'query': 'web scraping in Python',
    'num': 15,  # Get 15 results
    'hl': 'en',  # Language
    'gl': 'gb',  # Country: Great Britain
    'safe': 'moderate'  # SafeSearch mode
}
response = requests.get('https://app.scrapingbee.com/api/v1/search', params=params)
data = response.json()

for result in data.get('organic_results', []):
    print(f"{result['title']} → {result['link']}")
    print(f"Snippet: {result.get('snippet')}\n")

The key differences when scraping Bing compared to Google are:

1. The engine parameter is set to ‘bing’ instead of ‘google’

2. Bing’s response structure is slightly different, though ScrapingBee normalizes this to make it consistent

3. Some parameters, like safe have different accepted values

You might wonder: why scrape Bing in addition to Google? Different search engines can provide different results, and some markets have higher Bing usage. For comprehensive market research, collecting data from multiple search engines gives you a more complete picture.

For more details on Bing scraping, check out our guide explaining how to scrape Bing.

Best Practices for Search Engine Scraping

Ideally, you should be scraping responsibly. The goal here is to extract data without getting blocked, without overwhelming the target site. How should you do it? Make sure to follow these best practices:

Respect Rate Limits

Even though our API handles proxy rotation, it’s still important to space out your requests when data scraping:

import time
import requests

queries = ["web scraping", "data extraction", "python automation"]
results = []

for query in queries:
    params = {
        'api_key': 'YOUR_API_KEY',
        'engine': 'google',
        'query': query
    }
    response = requests.get('https://app.scrapingbee.com/api/v1/search', params=params)
    results.append(response.json())
    time.sleep(1)  # Add a small delay between requests

Store Results Efficiently

For one-time scraping, storing results in a CSV file works well, with little coding required:

import csv
import requests

def save_to_csv(results, filename):
    with open(filename, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=['title', 'link', 'snippet'])
        writer.writeheader()
        for result in results:
            writer.writerow(result)

params = {
    'api_key': 'YOUR_API_KEY',
    'engine': 'google',
    'query': 'data extraction tools'
}
response = requests.get('https://app.scrapingbee.com/api/v1/search', params=params)
data = response.json()

results = []
for result in data.get('organic_results', []):
    results.append({
        'title': result.get('title'),
        'link': result.get('link'),
        'snippet': result.get('snippet')
    })

save_to_csv(results, 'search_results.csv')

For larger projects or ongoing monitoring, consider using a database to store and query your web data more efficiently.

Use Specific Queries

More specific queries often yield more relevant results and are less likely to trigger anti-scraping measures. Here's an example:

# Too broad
params = {'query': 'shoes'}

# Better
params = {'query': 'men\'s running shoes under $100'}

Monitor Your Credit Usage

Keep track of your API credit usage to avoid unexpected interruptions:

def check_credits(api_key):
    url = f'https://app.scrapingbee.com/api/v1/account?api_key={api_key}'
    response = requests.get(url)
    data = response.json()
    return data.get('credit')

credits = check_credits('YOUR_API_KEY')
print(f"Remaining credits: {credits}")

By following these best practices, you’ll extract data reliably while being a good citizen of the web. Remember that the goal is to collect web data, not to overload servers or violate terms of service.

Common Errors and How to Fix Them

When scraping search engines, you might encounter some common errors. Here’s how to identify and fix them:

1. Invalid API Key

Error Response:

{
  "error": "Invalid API key"
}

Solution:
Double-check your API key. Make sure you’re copying the entire key without any extra spaces.

# Incorrect
api_key = 'ABCD1234 '  # Notice the trailing space

# Correct
api_key = 'ABCD1234'

2. Exceeded Credits

Error Response:

{
  "error": "You don't have enough credits"
}

Solution:
Check your credit balance in the dashboard. If you’ve run out, you’ll need to upgrade your plan or purchase additional credits. Visit ScrapingBee pricing to see your options.

# Check your credit balance before making requests
def check_credits(api_key):
    url = f'https://app.scrapingbee.com/api/v1/account?api_key={api_key}'
    response = requests.get(url)
    data = response.json()
    return data.get('credit')

credits = check_credits('YOUR_API_KEY')
if credits < 10:
    print("Warning: Low credit balance!")

3. Invalid Parameters

Error Response:

{
  "error": "Invalid parameter: engin"
}

Solution:
Check for typos in your parameter names. The correct parameter for specifying the search engine is engine, not engin. This happens more often than you think!

4. Network Errors

Error:
Connection timeouts or network-related exceptions.

Solution:
Implement retry logic to handle temporary network issues:

import requests
from requests.exceptions import RequestException
import time

def make_request_with_retry(url, params, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = requests.get(url, params=params)
            response.raise_for_status()  # Raise exception for 4XX/5XX responses
            return response.json()
        except RequestException as e:
            if attempt == max_retries - 1:  # Last attempt
                print(f"Failed after {max_retries} attempts: {e}")
                raise
            wait_time = 2 ** attempt  # Exponential backoff
            print(f"Attempt {attempt + 1} failed, retrying in {wait_time} seconds...")
            time.sleep(wait_time)

Start Scraping Search Engines With ScrapingBee

Now that you understand how to scrape search engine results, it’s time to put this knowledge into practice. ScrapingBee makes web scraping accessible to everyone, from individual developers to large enterprises, by handling all the technical challenges that typically make this task difficult.

If you're not yet convinced, here's what makes our scraping technology effective:

1. No infrastructure to maintain: Forget about managing proxies, browser farms, or CAPTCHA solvers. Web scraping can be done in just a few clicks.

2. Reliable results: Get consistent, structured data from search engines without worrying about layout changes or anti-bot measures.

3. Cost-effective solution: Save development time and resources by using a specialized API rather than building your own scraping infrastructure.

4. Flexible parameters: Customize your data extraction with language, location, device type, and other parameters to get exactly the data you need.

5. Multiple search engines: Scrape Google, Bing, and other websites using the same API and code structure.

Ready to launch your web scraper? Sign up for ScrapingBee and get 1,000 free API calls to test the service. No credit card is required to get started, and you can upgrade to a paid plan once you’re ready to scale your data scraping operations.

Frequently Asked Questions (FAQs)

Is scraping search engines legal?

Search engine scraping exists in a legal gray area. While accessing publicly available data isn't illegal itself, it may violate search engines' terms of service. Always check robots.txt files, respect rate limits, and avoid scraping sensitive or personal information. Consider consulting legal advice for your specific use case.

Can I scrape search engines without proxies?

Technically, yes, but it's rarely effective. Search engines quickly block IPs making multiple automated requests. Without proxies, you'll likely encounter CAPTCHAs or IP bans after just a few queries. Using a service like ScrapingBee eliminates this problem by handling proxy rotation automatically.

How many requests can I make with ScrapingBee?

ScrapingBee offers various plans with different request limits. Their free trial includes 1,000 credits, while paid plans range from 10,000 to millions of monthly requests. Each search query typically costs 1-5 credits, depending on complexity. Custom enterprise plans are available for higher volumes.

Does ScrapingBee support other search engines besides Google?

Yes, ScrapingBee supports multiple search engines. Besides Google, you can scrape Bing, Yahoo, Yandex, and DuckDuckGo using the same API structure with just a parameter change. This makes it easy to compare results across different search engines for comprehensive market research.