LangChain

Overview

LangChain is a framework for building applications powered by language models — chains, agents, and tools that call out to external systems. The official langchain-scrapingbee package wraps every ScrapingBee API as a LangChain Tool, so an agent can scrape pages, search Google, look up Amazon/Walmart products, or query YouTube as part of its reasoning loop.

Unlike the Zapier, Make, and n8n integrations, which run inside a visual no-code builder, this is a Python package you install into your own codebase.

Prerequisites

You need:

• Python >=3.9,<4.0
• A ScrapingBee account and an API key from your dashboard

Installation

Install the package from PyPI:

pip install -U langchain-scrapingbee

Then set your API key as an environment variable:

export SCRAPINGBEE_API_KEY="your-api-key"

Or set it at runtime in Python, and/or pass it explicitly per tool instead of relying on the environment:

import os, getpass

if not os.environ.get("SCRAPINGBEE_API_KEY"):
    os.environ["SCRAPINGBEE_API_KEY"] = getpass.getpass("Enter your ScrapingBee API key: ")

api_key = os.environ["SCRAPINGBEE_API_KEY"]

Every tool's constructor takes api_key explicitly — there's no global configuration step; you pass it when you instantiate each tool (e.g. ScrapeUrlTool(api_key=api_key)).

How the tools behave (read this first)

All 10 tools share the same conventions, with two exceptions (CheckUsageTool and YouTubeMetadataTool, noted below). Understanding these up front will save you from the most common confusion with this package:

• Every tool call makes a real ScrapingBee API request and consumes credits, regardless of any other argument.
• Results are always saved to disk first, in a fresh, timestamped subfolder under results_folder (default: "scraping_results"), e.g. scraping_results/20260706_083012/amazon_search_iphone_16.json. This happens unconditionally — there's no way to opt out of the file write.
• return_content (default False) only controls what .invoke(...) returns — not whether the API call or file save happens, and not whether anything prints to your terminal. False returns a short "saved successfully" summary with the file path and a few extracted highlights; True also inlines the full content/JSON in that return string. In a script, you still need print(result) to see it — assigning to a variable alone produces no output. This design exists to save LLM tokens when an agent doesn't need to read the full result directly (e.g. it can hand the file path to another tool instead).
• A scraping_metadata.jsonl log is appended in the same timestamped subfolder on every call, recording the timestamp, input, params, and result type — useful for auditing what an agent actually did across a session.
• params and headers dict arguments tolerate stringified input. If a model passes JSON, a Python dict-literal, or a URL-query string (key=value&key2=value2) instead of a real dict, the tool attempts to parse it automatically. When calling a tool directly from your own code, always pass a real Python dict — don't rely on this fallback.

CheckUsageTool is the one exception to all of the above — it takes no arguments, makes no file writes, and returns the raw usage response immediately.

YouTubeMetadataTool doesn't accept a params dict at all — just video_id.

Common parameters

Every tool below (except CheckUsageTool, which has none, and YouTubeMetadataTool, which has no params) accepts the exact same three parameters in addition to its own primary argument. They're documented once here instead of repeated in every tool's table:

ScrapeUrlTool additionally accepts headers (dict, default {}) and custom_filename (str, default None) — see its section below.

Available tools

The package exposes 10 tools, one per ScrapingBee endpoint. Each section below covers one tool's arguments, its full list of supported params keys, and a working code example.

ScrapeUrlTool

Scrapes any public URL — the most flexible tool, wrapping the full HTML API.

from langchain_scrapingbee import ScrapeUrlTool

scrape_tool = ScrapeUrlTool(api_key=api_key)
result = scrape_tool.invoke({
    "url": "http://httpbin.org/html",
    "params": {"render_js": False},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content) — here, params accepts any HTML API parameter (see table below).

If the response is binary (screenshot, PDF, image), the tool detects the content type automatically, saves the binary file with the correct extension, and returns file info instead of trying to inline binary data.

Supported params keys (the most commonly used — all standard HTML API parameters work):

The proxy ladder goes cheap → expensive: start with no proxy flag, add premium_proxy: true if blocked, escalate to stealth_proxy: true only if still blocked. See credit costs for exact pricing per option.

GoogleSearchTool

Searches Google — classic, news, maps, or images — wrapping the Google API.

from langchain_scrapingbee import GoogleSearchTool

google_search_tool = GoogleSearchTool(api_key=api_key)
result = google_search_tool.invoke({
    "search": "What is LangChain?",
    "params": {"search_type": "news", "country_code": "gb"},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content).

Supported params keys:

Image searches get special handling: base64-encoded images are decoded and saved as individual image files, plain image URLs are written to image_links.txt, and the full JSON response is saved alongside — all in the same timestamped results folder.

Known quirk: if you don't pass search_type, the API still performs a normal ("classic") organic search, but the tool labels it Type: web in the summary and filename instead of classic. This is cosmetic only — the actual search and saved data are correct.

CheckUsageTool

Checks your ScrapingBee credit balance and concurrency usage. This is the simplest tool — no arguments, no file writes, no return_content.

from langchain_scrapingbee import CheckUsageTool

usage_tool = CheckUsageTool(api_key=api_key)
print(usage_tool.invoke({}))

It maps directly to the ScrapingBee usage endpoint and returns the raw JSON response (credits used/remaining, concurrency limits, account status) immediately.

AmazonSearchTool

Searches Amazon for products, wrapping the Amazon Search API.

from langchain_scrapingbee import AmazonSearchTool

amazon_search_tool = AmazonSearchTool(api_key=api_key)
result = amazon_search_tool.invoke({
    "query": "iphone 16",
    "params": {"domain": "co.uk", "device": "desktop"},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content).

Supported params keys:

AmazonProductTool

Gets detailed info and reviews for a single Amazon product, wrapping the Amazon Product API.

from langchain_scrapingbee import AmazonProductTool

amazon_product_tool = AmazonProductTool(api_key=api_key)
result = amazon_product_tool.invoke({
    "query": "B0DPDRNSXV",
    "params": {"domain": "com", "device": "desktop"},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content) — params accepts the same keys as AmazonSearchTool above (domain, country, currency, language, device, zip_code, light_request, add_html, autoselect_variant).

WalmartSearchTool

Searches Walmart for products, wrapping the Walmart Search API.

from langchain_scrapingbee import WalmartSearchTool

walmart_search_tool = WalmartSearchTool(api_key=api_key)
result = walmart_search_tool.invoke({
    "query": "iphone",
    "params": {"sort_by": "price_low", "max_price": 500},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content).

Supported params keys:

WalmartProductTool

Gets detailed info and reviews for a single Walmart product, wrapping the Walmart Product API.

from langchain_scrapingbee import WalmartProductTool

walmart_product_tool = WalmartProductTool(api_key=api_key)
result = walmart_product_tool.invoke({
    "product_id": "454408250",
    "params": {"domain": "com", "delivery_zip": "72716"},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content) — params accepts domain, device, delivery_zip, store_id, light_request, add_html (same meanings as WalmartSearchTool above).

ChatGPTTool

Sends a prompt to ChatGPT, wrapping the ChatGPT API.

from langchain_scrapingbee import ChatGPTTool

chatgpt_tool = ChatGPTTool(api_key=api_key)
result = chatgpt_tool.invoke({
    "prompt": "Explain the benefits of renewable energy in 100 words",
    "params": {"search": True},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content).

Supported params keys:

YouTubeMetadataTool

Gets title, description, views, likes, channel info, and more for a video, wrapping the YouTube Metadata API. This is the simplest YouTube tool — no params dict.

from langchain_scrapingbee import YouTubeMetadataTool

youtube_metadata_tool = YouTubeMetadataTool(api_key=api_key)
result = youtube_metadata_tool.invoke({"video_id": "dQw4w9WgXcQ", "return_content": True})
print(result)

Plus results_folder and return_content from the common parameters — no params dict.

YouTubeSearchTool

Searches YouTube for videos, channels, or playlists, wrapping the YouTube Search API.

from langchain_scrapingbee import YouTubeSearchTool

youtube_search_tool = YouTubeSearchTool(api_key=api_key)
result = youtube_search_tool.invoke({
    "search": "python programming tutorial",
    "params": {"hd": True, "sort_by": "view_count"},
    "return_content": True,
})
print(result)

Plus the common parameters (params, results_folder, return_content).

Supported params keys:

Example: agent with tools

Pass these LangChain tools to a LangGraph agent and let the model decide which ones to call and in what order. This example uses Gemini via langchain-google-genai — install it alongside langgraph (not included in langchain-scrapingbee itself) and set GOOGLE_API_KEY:

pip install langchain-google-genai langgraph
export GOOGLE_API_KEY="your-google-api-key"

import os
from langchain_scrapingbee import (
    ScrapeUrlTool,
    GoogleSearchTool,
    AmazonSearchTool,
    AmazonProductTool,
)
from langchain_google_genai import ChatGoogleGenerativeAI
from langgraph.prebuilt import create_react_agent

scrapingbee_api_key = os.environ["SCRAPINGBEE_API_KEY"]

llm = ChatGoogleGenerativeAI(temperature=0, model="gemini-2.5-flash")

tools = [
    ScrapeUrlTool(api_key=scrapingbee_api_key),
    GoogleSearchTool(api_key=scrapingbee_api_key),
    AmazonSearchTool(api_key=scrapingbee_api_key),
    AmazonProductTool(api_key=scrapingbee_api_key),
]

agent = create_react_agent(llm, tools)

user_input = """
Search for "harry potter" book on Amazon, then get the product
details for the top result using its ASIN.
"""

for step in agent.stream({"messages": user_input}, stream_mode="values"):
    step["messages"][-1].pretty_print()

This example was tested end-to-end: the agent called AmazonSearchTool for "harry potter book", read ASIN 1546148507 from the result, then called AmazonProductTool to return full product details — no manual chaining required. Because each tool's description embeds its own supported params, a capable model can also pick sensible filters (e.g. sort_by, device, country_code) without you hardcoding them.

Credits and limits

Each tool call consumes ScrapingBee credits at the same rate as the underlying API call — the package adds no surcharge. GoogleSearchTool costs whatever the Google API costs; ScrapeUrlTool with params: {"premium_proxy": True} costs whatever the HTML API costs with that flag. See credit costs for the exact per-endpoint pricing.

Because an agent decides which tools to call and how many times, a single user prompt can trigger multiple ScrapingBee requests. Call CheckUsageTool during development (and periodically in production) to catch runaway loops before they burn through credits.

For ScrapeUrlTool, the proxy ladder goes cheap → expensive: start without premium_proxy or stealth_proxy. If the site blocks you, retry with params: {"premium_proxy": True}. If still blocked, escalate to params: {"stealth_proxy": True}. Only step up when you need to — each step costs more credits.

Troubleshooting

.invoke(...) only prints a "saved successfully" message, not the actual content

This is expected when return_content is False (the default) — every tool writes the full result to a file on disk unconditionally (the API call and credit spend happen either way), and returns only a short file-info summary in the string. Pass "return_content": True in the .invoke(...) call to include the actual scraped data, search results, or API response in the return value — then call print(result) to display it in a script. Without print(result), assigning the return value to a variable produces no terminal output even when return_content is True.

A scrape returns empty or blocked content

The target site is blocking ScrapingBee's default scraper. Pass params={"premium_proxy": True} to ScrapeUrlTool; if still blocked, escalate to params={"stealth_proxy": True}. Each step up costs more credits — see credit costs.

Agent calls the wrong tool, or calls it in a loop

Tighten your prompt or system message to be explicit about which tool to use and when to stop. If the agent keeps mispicking tools or looping, try a more capable model — tool selection and stopping behavior improve noticeably with stronger models. Use CheckUsageTool to confirm whether the agent is actually making excessive ScrapingBee calls.

Resources

• langchain-scrapingbee on GitHub — source code
• langchain-scrapingbee on PyPI — installation and version history
• Web Scraping With LangChain & ScrapingBee — ScrapingBee's own walkthrough
• HTML API · Google API · Data Extraction · JavaScript Scenario · Amazon API · Walmart API · ChatGPT API · YouTube APIs
• ScrapingBee dashboard — manage your API key, view usage, upgrade your plan
• Knowledge Base — searchable troubleshooting articles