A deep research workflow is only as good as the sources it can retrieve. Open systems like Local Deep Research, gpt-researcher, and LangChain's deep agents use a similar loop: a search API finds URLs, then a downloader retrieves each one. That retrieval stage is where things break. A documentation site returns a 403. A pricing page rendered entirely with JavaScript comes back empty. Raw HTML floods the model's context with navigation and ad markup instead of the paragraph it needed.
This guide replaces that retrieval stage with a web scraping API, and spends real time on the part most guides skip: what happens when a request fails, and how to tell a real page from a block page before you hand it to a model.
Key takeaways
- Reading the modern web reliably is a scraping problem: a search API finding URLs doesn't help if the retrieval step can't get real content back from them.
- Several open deep-research systems, including Local Deep Research and gpt-researcher, use a variant of the same loop: search finds URLs, a downloader retrieves each one.
- Three failure modes account for most of the trouble: 403s from bot protection, JavaScript pages that return an empty shell, and raw HTML that wastes context tokens.
- A 200 status code doesn't guarantee usable content. Block pages and empty shells can also return 200, so status code alone isn't enough validation.
- Different failures need different handling: fail fast on bad requests, retry rate limits and server errors, and escalate to a stronger proxy only when there's an actual sign of blocking.
How deep research agents read the web (and where it breaks)
A deep research workflow runs in a loop. A planner breaks the question into smaller sub-questions, a search API returns candidate URLs, a retrieval stage downloads each document and pulls out its text, and the model synthesizes a cited report. A recent survey of deep research agent architectures documents this same pattern academically, reviewing information acquisition strategies and contrasting API-based retrieval methods with browser-based exploration. Local Deep Research, gpt-researcher, and several other open systems use a variant of this loop, though the details of search and synthesis differ between them.
Most of the trouble concentrates at the retrieval stage.
A site returns a 403 to a request that doesn't present like a browser. Or a single-page React application returns a nearly empty HTML shell, since nothing ran the JavaScript that would have populated it, so the text never arrives no matter how good the parsing code downstream is. Even a document that loads correctly is still mostly navigation, ads, and boilerplate on many sites, which crowds out the tokens that mattered.
None of these announce themselves reliably. A 200 status code means the HTTP request completed, not that the body contains the source's actual content: a blocked request, a JavaScript-only page, and a working page can all return 200, and a check that stops at the status code can't tell them apart. Local Deep Research documents this trade-off directly: it identifies itself truthfully and skips stealth or detection-evasion techniques, so sites that actively block automated access won't be retrieved. That's a defensible choice for a tool built around user privacy, and it also means sites with bot protection stay out of reach unless whatever sits behind Local Deep Research's search layer can get past them.
This affects gpt-researcher and LangChain-based research agents the same way, since all three hand the same job — turning a URL into usable text — to whatever HTTP client and parser sits behind the search step.
Why "just use requests and BeautifulSoup" fails your agent
The common DIY approach is a plain HTTP GET plus an HTML parser, and its failures don't surface where you'd expect. gpt-researcher's default scraper falls back to roughly this pattern when no dedicated scraping backend is configured: a requests call parsed with BeautifulSoup, no JavaScript execution, wrapped in broad exception handling that returns empty text on failure instead of raising an error. A 403 block, a page that needs JavaScript to render, and a network timeout all become empty content the same way, and that empty content flows into the report as if the source said nothing at all. gpt-researcher's own documentation recommends a managed scraping backend for production use, for exactly this reason: plain requests and BeautifulSoup can't handle bot protection or JavaScript rendering on their own.
Two more failure modes compound this. A page that relies heavily on JavaScript — a product listing page that builds its content client-side, for instance — returns an empty shell to any client that can't execute scripts, so there's nothing to extract in the first place. And on many sites, raw HTML is dominated by navigation, ads, and boilerplate markup, which wastes the context window and complicates chunking before a single useful sentence reaches the model.
None of these three failure modes look different from a working request at the point where most workflows check for success: a status code. That's part of why they go unnoticed until someone reads the actual report and finds a section built on nothing. For more on why plain requests get blocked in the first place, see how to scrape without getting blocked.
The fix: a web scraping API as your agent's read-a-URL tool
A web scraping API can run JavaScript, handle many common blocking mechanisms, and hand back clean text or structured JSON from a single call, replacing the fragile downloader described above. For the broader picture of AI extraction patterns in Python beyond this one use case, see AI web scraping with Python.
Tested environment:
- Python 3.10.4, scrapingbee SDK 2.0.2 (original API verification); re-verified 2026-08-15 on Python 3.14.3 with scrapingbee SDK 2.1.1. The code below calls client.html_api(), which needs scrapingbee SDK 2.1.1 or newer: it replaces client.get(), deprecated in 2.1.1 and scheduled for removal in 3.0.0.
- Live-tested against the real ScrapingBee API, August 2026:
- fetch_url_minimal / return_page_markdown, against example.com and a full Wikipedia article; both returned clean Markdown.
- extract_fields (ai_extract_rules), against a real blog post URL; returned JSON matching the requested schema.
- A direct stealth_proxy=true call against an accessible page returned 200 with real content, confirming Stealth works mechanically; this doesn't by itself prove Stealth defeated an active block, since the test page wasn't blocking to begin with.
- Redirect and content-type validation, confirmed via ScrapingBee's Spb-resolved-url and Spb-content-type response headers against a domain with a real apex-to-www redirect (microsoft.com), with no false positive on that routine redirect.
- Retry and backoff behavior (network exceptions, 408, 5xx) is covered by scenario tests against mocked responses. The 429 path was additionally verified live: two bursts of 20 concurrent requests against the real API both triggered a genuine 429, and Retry-After was absent in both, confirming the code's fallback to exponential backoff is the path that actually runs in practice, not just in mocks.
- The fetch_url control flow below, which separates rendering escalation from proxy escalation, adds a low-confidence path for short-but-real pages, and corroborates markers and redirects with content length, has been fully verified as of 2026-08-15:
- All 9 classification/escalation scenarios (network exception, an unhandled status code, Retry-After as an HTTP-date, a routine www-redirect, a short-but-real page, a non-text content-type, a block marker inside long real text, the same marker inside a short response, and a login-shaped redirect on another host) pass against mocked client.html_api responses, alongside 2 tests confirming cached_fetch_url never caches a failure and always caches a success.
- 3 live calls against the real API: Wikipedia succeeded on the cheapest tier (confidence="high", no escalation); example.com behaved exactly as designed (confidence="low", never reaching Premium or Stealth); extract_fields against a real Wikipedia URL returned JSON matching the requested schema. Total cost: 25 credits.
- mypy and pyright, run independently against fetch_url.py, both originally flagged two real issues beyond the expected scrapingbee stubs warning: a latent UnboundLocalError in extract_fields() (if client.html_api() failed before response was assigned, the except clause referencing it crashed instead of printing the intended message), and a type mypy couldn't pin down in _EXPECTED_TYPES from mixing a plain type with a tuple[type, ...] without an explicit annotation. Both are fixed here: the html_api() call and response.raise_for_status() now sit in separate try blocks, and _EXPECTED_TYPES carries an explicit dict[str, type | tuple[type, ...]] annotation. With both fixes in place, mypy and pyright report zero errors beyond the unavoidable scrapingbee stubs warning.
- The Local Deep Research adapter below has been verified against a real, currently-installed copy, 1.10.3, commit 3475653, checked 2026-08-15, not just the documented extension pattern. Two things changed since the pattern was written; the code below and the note after it reflect the fix.
Set up ScrapingBee in your agent
Get a free API key (1,000 credits, no credit card), then install the Python SDK:
pip install scrapingbee
import os
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
If you'd rather skip the SDK dependency, a plain HTTP GET to the same endpoint works too. Send your key as a Bearer token rather than as an api_key query parameter, which ScrapingBee now deprecates:
curl "https://app.scrapingbee.com/api/v1?url=YOUR-URL" \
-H "Authorization: Bearer YOUR-API-KEY"
Fetch a page as clean, LLM-ready markdown
Instead of handing your own parser raw HTML to fight with, ask ScrapingBee to return structured Markdown directly:
def fetch_url_minimal(url: str) -> str:
"""Minimal example. See fetch_url() further down for a version with
real error handling; don't use this version as-is in anything that
matters."""
response = client.html_api(
url,
method="GET",
params={"render_js": "true", "return_page_markdown": "true"},
)
response.raise_for_status()
return response.content.decode("utf-8")
ScrapingBee's markdown scraper, triggered with return_page_markdown, strips navigation, cookie banners, and script tags, and hands back headings, paragraphs, and lists instead. return_page_text does the same for a plain text version when structure doesn't matter.
Here's the difference on an actual page. Raw HTML for a typical article starts with boilerplate before any article text shows up:
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8">
<script src="/analytics.js" async></script>...
<nav><ul><li><a href="/">Home</a></li><li><a href="/about">About</a></li>...</ul></nav>
<div class="cookie-banner">This site uses cookies. <button>Accept</button></div>
<article><h1>The Actual Article Title</h1>
<p>The article's first real sentence starts here...</p>
return_page_markdown cuts straight to this:
# The Actual Article Title
The article's first real sentence starts here...
Markdown headings also give a retrieval pipeline natural chunk boundaries, which raw HTML doesn't provide as cleanly; a chunker working against unstructured HTML tends to split content mid-tag rather than at a sentence or section break.
This example enables render_js unconditionally and skips validating the response entirely. fetch_url, built later in this guide under "Handle failures and control cost," validates what comes back, retries the failures worth retrying, and never mixes an error message into the field a research agent reads as page content.
Get past anti-bot walls and JavaScript
render_js=true runs the document through a real headless browser, which addresses empty-shell responses from pages rendered with JavaScript. For sites with active bot protection, ScrapingBee offers two escalating proxy tiers on top of that: Premium for sites that block plain datacenter IPs, and Stealth for the hardest targets — an aggressively fingerprinted e-commerce catalog or ticketing site, for example. Stealth only works with JavaScript rendering enabled.
The credit costs: a standard render_js request costs 5 credits, Premium costs 10 credits without rendering or 25 with it, and Stealth costs 75. ScrapingBee's mode=auto parameter can pick a configuration automatically, but it can't be combined with render_js, premium_proxy, stealth_proxy, or transparent_status_code in the same request (ScrapingBee rejects that combination with a 400), and mode=auto only works on GET requests. It decides based on whether a request succeeds, not on whether the body it got back is a real page or an empty shell. That last exclusion matters once transparent_status_code is part of fetch_url's standard params: the two are mutually exclusive, so mode=auto isn't a usable alternative to the escalation logic in this article. If cost matters more than detecting soft blocks, ScrapingBee's max_cost parameter caps how many credits Auto-Mode is allowed to spend on a single request. fetch_url, built later in this guide, handles that distinction itself by treating rendering and proxy tier as two separate decisions: a short or empty response justifies trying render_js once, since that's the signature of an unrendered page, while only a confirmed block (a 403, or a redirect to what looks like a login or captcha page) justifies paying for Premium or Stealth.
Extract structured fields with AI extraction
When your workflow needs structured data rather than prose — a publication date, an author, a price, a table of results — ScrapingBee's AI extraction describes what you want in plain English via ai_query and returns JSON, with no selectors to write or maintain:
response = client.html_api(
url,
method="GET",
params={
"render_js": "true",
"ai_query": "publication date, author name, and article summary",
},
)
response.raise_for_status()
answer = response.content.decode("utf-8")
print(answer)
ai_query returns a free-form answer in plain text, shaped by however the model chose to phrase it, not a fixed JSON schema. A plain decode is the honest way to read it; reach for ai_extract_rules instead (covered next) when you need the same fields back in a predictable, typed shape across many sources.
Like the minimal example above, this always requests render_js to keep the snippet self-contained; in practice, run extraction through the production fetch_url from later in this guide if you want it to start cheap and escalate only when needed.
ai_extract_rules is the schema-based counterpart, useful when you want a fixed, typed shape across many sources instead of a one-off prompt. Checking the status code isn't enough here either: AI extraction will pull something out of an error page (a 404 page still has a title), and a JSON parse succeeding doesn't confirm the result actually matches the schema you asked for — a field can come back missing, null, or the wrong type while the JSON itself stays valid. extract_fields below checks both:
from __future__ import annotations
import os
import json
import requests
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
_EXPECTED_TYPES: dict[str, type | tuple[type, ...]] = {
"string": str,
"list": list,
"boolean": bool,
"number": (int, float),
"item": dict,
}
def _type_ok(rule_type: str, value) -> bool:
"""bool is a subclass of int in Python; without this explicit check,
a boolean value would pass a "number" rule too."""
expected_type = _EXPECTED_TYPES.get(rule_type)
if expected_type is None or value is None:
return True
if rule_type == "number" and isinstance(value, bool):
return False
return isinstance(value, expected_type)
def extract_fields(url: str, rules: dict) -> dict | None:
"""Fetch a URL and extract structured fields via ScrapingBee's AI extraction.
Returns a dict of extracted fields, or None on failure. Passing `rules`
as a plain dict works with the official SDK (it serializes the
parameter for you); a raw requests.get() call would need
json.dumps(rules) instead.
Validates that the result actually matches rules across all five ai_extract_rules types (string, list, number, boolean, item), not only the two this function originally checked. Booleans need a special case: Python's bool is a subclass of int, so a value of True would otherwise pass a "number" check too.
A JSON parse succeeding doesn't guarantee that, since a missing
or mistyped field can still be valid JSON.
"""
try:
response = client.html_api(
url,
method="GET",
params={
"render_js": "true",
"ai_extract_rules": rules,
},
)
except Exception as exc:
print(f"extract_fields({url}): {type(exc).__name__}")
return None
try:
response.raise_for_status()
except requests.exceptions.HTTPError:
print(f"extract_fields({url}): ScrapingBee returned HTTP {response.status_code}")
return None
try:
data = json.loads(response.content.decode("utf-8"))
except json.JSONDecodeError:
print(f"extract_fields({url}): response was not valid JSON")
return None
for field, rule in rules.items():
if field not in data:
print(f"extract_fields({url}): missing expected field '{field}'")
return None
value = data[field]
if not _type_ok(rule.get("type"), value):
print(f"extract_fields({url}): '{field}' is {type(value).__name__}, expected {rule['type']}")
return None
return data
# Example usage
if __name__ == "__main__":
article_rules = {
"title": {"description": "the title of the article", "type": "string"},
"author": {"description": "the author's name", "type": "string"},
"publication_date": {"description": "when the article was published", "type": "string"},
"key_points": {"description": "the main takeaways or key points of the article", "type": "list"},
}
fields = extract_fields("https://example.com/some-article", article_rules)
if fields:
print(fields.get("title"))
Guard the example call behind if name == "main": — without it, this fires a real (billable) request the moment anything does from extract_fields import extract_fields, which is exactly the kind of surprise this article argues against building into a research pipeline.
Both ai_query and ai_extract_rules add 5 credits on top of the base request cost. The optional ai_selector parameter narrows the AI's attention to one region of the page (defined with a CSS selector), which improves accuracy and cuts processing time when you already know roughly where the data lives. See the data extraction docs for the full rule syntax.
Wire it in as a tool your agent calls
How this plugs in depends on the framework. For a LangChain-based agent, it's a tool that wraps a fetch function. For gpt-researcher, it's a custom scraper class that replaces the default BeautifulSoup path. Local Deep Research has a documented extension point worth showing concretely, since generic advice ("just wire it in") is exactly what's missing from most guides on this topic; the caveats below are as important as the code.
Local Deep Research's search sources all extend BaseSearchEngine, which splits retrieval into two phases: _get_previews() finds candidate URLs and short snippets, and _get_full_content() fetches the full text for whichever previews survive relevance filtering. That second phase is the one to replace. The inner search engine (SearXNG, Wikipedia, or whichever one currently finds your URLs) needs to be constructed inside the class, not passed in ready-made: Local Deep Research's own factory builds engines by name and keyword arguments, not by injecting already-built instances through the registry.
# src/local_deep_research/web_search_engines/engines/search_engine_scrapingbee.py
from typing import Any, Dict, List
from ..search_engine_base import BaseSearchEngine, Sensitivity
from ..search_engine_factory import create_search_engine
# fetch_url is the function built later in this guide.
from .fetch_url import fetch_url
class ScrapingBeeSearchEngine(BaseSearchEngine):
"""Wraps an existing search engine's previews with a reliable
retrieval step. The search step (finding URLs) is delegated to
whichever engine you name; only the content-fetching step is
replaced."""
is_public = True
is_generic = True
# BaseSearchEngine defaults this to SENSITIVE (fail closed) — every
# built-in generic web engine (DuckDuckGo, Brave, arXiv, ...) overrides
# it to NON_SENSITIVE, since it's sending queries out over the public
# web, not touching anything private.
egress_sensitivity = Sensitivity.NON_SENSITIVE
def __init__(self, inner_engine_name: str = "searxng", **kwargs):
super().__init__(**kwargs)
# create_search_engine() requires settings_snapshot for any
# non-retriever engine — every real call site in Local Deep
# Research threads it through explicitly, and it raises
# RuntimeError if it's missing.
self.inner_engine = create_search_engine(
inner_engine_name,
settings_snapshot=self.settings_snapshot,
programmatic_mode=self.programmatic_mode,
)
def _get_previews(self, query: str) -> List[Dict[str, Any]]:
return self.inner_engine._get_previews(query)
def _get_full_content(
self, relevant_items: List[Dict[str, Any]]
) -> List[Dict[str, Any]]:
results = []
for item in relevant_items:
result = item.copy()
fetch_result = fetch_url(item["link"])
result["content"] = fetch_result.content if fetch_result.success else ""
result["full_content"] = result["content"]
if not fetch_result.success:
result["fetch_error"] = fetch_result.error
results.append(result)
return results
Registering it means adding a branch to create_search_engine() in web_search_engines/search_engine_factory.py. Local Deep Research's own extension guide documents this as the "Modify Factory" pattern, but its own abridged example (# ... existing code ...) skips two things that matter on a real, current install:
# In search_engine_factory.py, inside create_search_engine() — place this
# branch right after the built-in retriever-registry check (the one that
# returns a RetrieverSearchEngine) and before the config lookup ("Extract
# search engine configs from settings snapshot"). Later than that, an
# unrecognized engine_name is rejected with ValueError("Unknown search
# engine ...") before this branch would ever run.
if engine_name.lower() == "scrapingbee":
from .engines.search_engine_scrapingbee import ScrapingBeeSearchEngine
return ScrapingBeeSearchEngine(
inner_engine_name=kwargs.pop("inner_engine_name", "searxng"),
# max_results isn't a plain local variable at this point in the
# function; it only gets assigned later, after the config lookup
# this branch runs before. Read it from kwargs using pop, not get:
# this branch also splats **kwargs below, and get would leave
# max_results in kwargs to be passed a second time, raising
# TypeError. LDR's own retriever branch
# (search_engine_factory.py:125) can safely use get because it
# doesn't forward **kwargs; this one does, so pop is the one
# that's safe here.
max_results=kwargs.pop("max_results", 10),
settings_snapshot=settings_snapshot,
programmatic_mode=programmatic_mode,
**kwargs,
)
Once that's in place, search_tool="scrapingbee" selects it for a research run. Returning this early skips the egress policy gate (evaluate_engine) that every config-registered engine goes through further down in this function, which is fine for a personal script but worth knowing if that policy layer matters for your deployment.
Two things worth being upfront about. First, _get_previews() reaches into the inner engine's own two-phase API, which are protected methods, not a documented public interface for "search but don't fetch yet." Local Deep Research doesn't currently expose one; this is the trade-off of only replacing half of an existing engine rather than writing a fully independent one. Second, this adapter has been verified against a real, currently-installed copy, 1.10.3, commit 3475653, checked 2026-08-15, not just the documented pattern, and the code above already reflects what that check found: create_search_engine() now hard-requires settings_snapshot (it raises RuntimeError without one), and BaseSearchEngine now carries an egress-classification pair (egress_sensitivity defaults to the fail-closed SENSITIVE, which every built-in generic engine overrides). Internal APIs like these move faster than the public quick_summary()/search_tool interface does, so if your installed version differs from 1.10.3, re-confirm this against it before depending on it in production.
For gpt-researcher and LangChain agents, the same fetch_url() function plugs in wherever each framework expects a scraper or tool callable; the retrieval logic itself doesn't change between frameworks, only the adapter shape.
Handle failures and control cost
Not every failure means the same thing, and treating them the same is what causes both wasted spend and false negatives. A bad request needs neither a retry nor an escalation, since nothing will fix it. A timeout or a rate limit is worth retrying on the same configuration, since it says nothing about whether the target is blocking you. Only a real sign of blocking — a 403, a matched interstitial phrase, or a redirect to what looks like a login or captcha page — justifies paying for a stronger proxy. And a short response on its own is ambiguous: example.com's real page is genuinely under 200 characters, so treating "short" as an automatic failure means rejecting a page that loaded correctly and burning credits escalating toward a block that isn't there.
Getting a real 403 to reach _attempt at all requires transparent_status_code=true: without it, ScrapingBee returns a 500 for anything outside 200-299 or 404, so a hard block looks identical to a server error and only gets retried, never escalated. Turning it on changes the billing model too. With transparent_status_code set, every request is considered successful and billed, including the ones that come back blocked.
fetch_url below keeps those cases separate. A small helper, _attempt, makes one (possibly retried) request at a fixed set of parameters and reports back one of five outcomes: a validated success, a weakly-validated "short or wrong type" result, a confirmed block, a terminal failure that nothing will fix, or a transient failure whose retries ran out. fetch_url itself only has two escalation moves: try rendering once if the plain request looked weak, and climb the proxy ladder only in response to a confirmed block.
import os
import time
import random
import requests
from dataclasses import dataclass
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from typing import Optional
from urllib.parse import urlsplit
from scrapingbee import ScrapingBeeClient
client = ScrapingBeeClient(api_key=os.environ["SCRAPINGBEE_API_KEY"])
MIN_CONTENT_LENGTH = 200
MAX_RETRIES_PER_ATTEMPT = 2
BASE_BACKOFF_SECONDS = 1.5
MAX_RETRY_AFTER_SECONDS = 30
# Interstitial phrases, only treated as evidence when corroborated by a
# second signal (see _classify_content): an article discussing anti-bot
# systems can legitimately contain "verify you are human" mid-paragraph
# while being thousands of characters long; an actual challenge page is
# both short and tends to open with the phrase as its heading.
BLOCK_PAGE_MARKERS = (
"checking your browser",
"attention required",
"verify you are human",
"enable javascript and cookies to continue",
)
# A hostname change alone (example.com -> www.example.com, a shortener,
# a migrated domain, a regional mirror) is routine, not suspicious. Only
# treat a redirect as block evidence when the resolved path also matches
# one of these hints.
REDIRECT_BLOCK_PATH_HINTS = ("login", "signin", "sign-in", "captcha", "challenge", "denied", "blocked")
@dataclass
class FetchResult:
"""Outcome of a fetch attempt.
Check `success` before reading `content`. Failure details live in
`error` and `status_code`, never mixed into the content field, so a
caller can't accidentally index or cite an error message as if it
were retrieved text. `confidence` is "low" when the page only passed
validation weakly (short, with no rendering left to try), so a
caller can choose to trust it less rather than treating it the same
as a normal, fully validated page.
"""
success: bool
content: str = ""
status_code: Optional[int] = None
error: str = ""
confidence: str = "high"
def _normalize_host(url: str) -> str:
host = urlsplit(url).netloc.lower()
return host[4:] if host.startswith("www.") else host
def _parse_retry_after(value: str) -> float:
"""Retry-After is either delta-seconds ("120") or an HTTP-date
("Wed, 21 Oct 2026 07:28:00 GMT"); handle both, and cap the result so
a malformed or hostile value can't stall the process indefinitely."""
try:
seconds = float(value)
except ValueError:
try:
target = parsedate_to_datetime(value)
if target.tzinfo is None:
target = target.replace(tzinfo=timezone.utc)
seconds = (target - datetime.now(timezone.utc)).total_seconds()
except (TypeError, ValueError):
seconds = BASE_BACKOFF_SECONDS
return max(0.0, min(seconds, MAX_RETRY_AFTER_SECONDS))
def _is_redirect_block(requested_url: str, resolved_url: str) -> bool:
if _normalize_host(resolved_url) == _normalize_host(requested_url):
return False
path = urlsplit(resolved_url).path.lower()
return any(hint in path for hint in REDIRECT_BLOCK_PATH_HINTS)
def _first_line(text: str) -> str:
"""Best-effort page title from returned Markdown: the first heading
or, failing that, the first non-blank line. return_page_markdown
doesn't preserve an HTML <title> tag, so this is the closest
available stand-in, used only to corroborate a marker match, never
as a classification signal by itself."""
for line in text.splitlines():
line = line.strip()
if line:
return line.lstrip("#").strip()
return ""
def _classify_content(text: str, headers, requested_url: str) -> str:
"""Returns "ok", "marker", "redirected", "content_type", or "short".
"marker" and "redirected" are strong evidence of an actual block.
"content_type" means the response isn't a text resource at all,
unsupported rather than blocked; no proxy tier or rendering fixes
that. "short" is weak evidence: plenty of real pages are
legitimately this short. Still a heuristic, not a guarantee: it
doesn't check expected page structure beyond the title line, which
would need per-source knowledge this generic function doesn't have.
"""
stripped = text.strip()
lowered = stripped.lower()
title = _first_line(stripped).lower()
marker_in_body = any(marker in lowered for marker in BLOCK_PAGE_MARKERS)
marker_in_title = any(marker in title for marker in BLOCK_PAGE_MARKERS)
is_short_ish = len(stripped) < MIN_CONTENT_LENGTH * 3
# A marker alone isn't enough: a real article can mention "verify
# you are human" in passing. Require a second, independent signal,
# either the whole response is short (an interstitial page has
# little else on it), or the marker itself is what the page opens
# with (a real article's title doesn't repeat its own body text).
if marker_in_body and (is_short_ish or marker_in_title):
return "marker"
resolved_url = headers.get("Spb-resolved-url", "")
if resolved_url and _is_redirect_block(requested_url, resolved_url):
return "redirected"
content_type = headers.get("Spb-content-type", "")
if content_type and not content_type.split(";")[0].strip().startswith("text/"):
return "content_type"
if len(stripped) < MIN_CONTENT_LENGTH:
return "short"
return "ok"
@dataclass
class AttemptResult:
"""Outcome of a single _attempt() call.
`outcome`discriminates which of the other fields carry meaning, so a caller doesn't have to infer, from outcome alone, whether to read text, status, or message. Naming the fields lets mypy and pyright verify that mapping directly, instead of taking it on faith.
- "success" / "weak" -> `text` holds the page content
- "blocked" -> `status` holds the response status code
- "terminal" / "stopped" -> `status` and `message` describe the failure
"""
outcome: str
text: str = ""
status: Optional[int] = None
message: str = ""
def _attempt(url: str, params: dict) -> AttemptResult:
"""One (possibly retried) request at a fixed parameter set.
outcome meanings:
- "success" validated content, ready to use
- "weak" content came back, but validation was weak (short or wrong type)
- "blocked" strong evidence of an active block
- "terminal" not retriable or escalatable: bad request, source missing, or unsupported type
- "stopped" transient failures exhausted their retries; do not escalate
"""
last_status: Optional[int] = None
last_message = "exhausted retries"
for attempt_num in range(MAX_RETRIES_PER_ATTEMPT):
try:
response = client.html_api(url, method="GET", params=params, timeout=30)
except requests.exceptions.RequestException as exc:
last_message = f"{type(exc).__name__}: {exc}"
if attempt_num < MAX_RETRIES_PER_ATTEMPT - 1:
time.sleep(BASE_BACKOFF_SECONDS * (2 ** attempt_num) + random.uniform(0, 0.5))
continue
return AttemptResult("stopped", status=last_status, message=last_message)
status = response.status_code
last_status = status
# Nothing about the request can be fixed by retrying or
# escalating: a bad URL, bad parameters, or bad credentials.
if status in (400, 401, 422):
return AttemptResult(
"terminal", status=status, message="request rejected: check URL, parameters, or credentials"
)
# The source genuinely isn't there. A normal, expected outcome.
if status in (404, 410):
return AttemptResult("terminal", status=status, message="source not available")
# A likely block, reported directly by the status code.
if status == 403:
return AttemptResult("blocked", status=status)
# Timeouts, rate limits, and server errors are transient by
# nature. Retry the same configuration; none of these says
# anything about whether the target is blocking the request.
if status in (408, 429) or 500 <= status < 600:
last_message = f"transient error {status}"
if attempt_num < MAX_RETRIES_PER_ATTEMPT - 1:
if status == 429:
retry_after = response.headers.get("Retry-After")
delay = _parse_retry_after(retry_after) if retry_after else BASE_BACKOFF_SECONDS * (2 ** attempt_num)
else:
delay = BASE_BACKOFF_SECONDS * (2 ** attempt_num)
time.sleep(delay + random.uniform(0, 0.5))
continue
return AttemptResult("stopped", status=status, message=last_message)
# Anything else outside 2xx (a stray 3xx, 402, 405, 409, 418,
# 451, ...) is treated as terminal rather than silently falling
# through to content validation as if it were a success.
if not (200 <= status < 300):
return AttemptResult("terminal", status=status, message=f"unexpected status {status}")
text = response.content.decode("utf-8", errors="replace")
classification = _classify_content(text, response.headers, url)
if classification == "ok":
return AttemptResult("success", text=text)
if classification in ("marker", "redirected"):
return AttemptResult("blocked", status=status)
if classification == "content_type":
return AttemptResult("terminal", status=status, message="response is not a text resource")
return AttemptResult("weak", text=text) # "short"
return AttemptResult("stopped", status=last_status, message=last_message)
def fetch_url(url: str) -> FetchResult:
"""Fetch a URL through ScrapingBee.
Rendering and proxy tier escalate on different evidence. A weak
(short) plain response gets one rendering attempt, since that's the
signature of an unrendered single-page app; if it's still weak after
rendering, it comes back as a low-confidence success rather than
being chased with an expensive proxy. Premium and Stealth escalate
only after a confirmed block: a 403, a matched interstitial phrase,
or a redirect to a login- or captcha-shaped URL.
Every request includes transparent_status_code=true: without it,
ScrapingBee returns a 500 for anything outside 200-299/404, so a real
403 would look identical to a server error and never reach the
"blocked" branch below. The trade-off is billing: with
transparent_status_code on, every request is considered successful
and billed, including the ones that come back blocked.
"""
result = _attempt(url, {"return_page_markdown": "true", "transparent_status_code": "true"})
if result.outcome == "success":
return FetchResult(success=True, content=result.text, status_code=200)
if result.outcome in ("terminal", "stopped"):
return FetchResult(success=False, status_code=result.status, error=result.message)
if result.outcome == "weak":
rendered = _attempt(url, {"return_page_markdown": "true", "render_js": "true", "transparent_status_code": "true"})
if rendered.outcome == "success":
return FetchResult(success=True, content=rendered.text, status_code=200)
if rendered.outcome == "weak":
return FetchResult(success=True, content=rendered.text, status_code=200, confidence="low")
if rendered.outcome in ("terminal", "stopped"):
return FetchResult(success=False, status_code=rendered.status, error=rendered.message)
# rendered.outcome == "blocked" — fall through to the proxy tier
# loop below, same as a "blocked" result from the very first
# attempt would.
for tier_params in ({"premium_proxy": "true"}, {"stealth_proxy": "true"}):
params = {
"return_page_markdown": "true",
"render_js": "true",
"transparent_status_code": "true",
**tier_params,
}
tier = _attempt(url, params)
if tier.outcome == "success":
return FetchResult(success=True, content=tier.text, status_code=200)
if tier.outcome == "weak":
return FetchResult(success=True, content=tier.text, status_code=200, confidence="low")
if tier.outcome in ("terminal", "stopped"):
return FetchResult(success=False, status_code=tier.status, error=tier.message)
return FetchResult(success=False, status_code=None, error="blocked even on Stealth")
_fetch_cache: dict = {}
def cached_fetch_url(url: str) -> FetchResult:
"""Like fetch_url, but only caches successful results. A transient
failure (a timeout, a momentary rate limit) is retried on the next
call instead of being remembered as a permanent failure for the
rest of the process."""
if url in _fetch_cache:
return _fetch_cache[url]
result = fetch_url(url)
if result.success:
_fetch_cache[url] = result
return result
Transient failures (a network exception, 408, 429, or 5xx) that run out of retries return "stopped", which fetch_url turns into an immediate failure, not an escalation: a timeout is not evidence of blocking, so it shouldn't cost more to find that out. A 403 is trusted immediately, since it's the site's own explicit signal. A redirect only counts as a block when the resolved path also looks like a login or captcha destination, since a bare hostname change (example.com to www.example.com, a shortener, a regional mirror) is routine. A marker match only counts when it's corroborated by a second signal — the response is short, or the marker also shows up in the page's own title — since a long article can legitimately mention "verify you are human" mid-paragraph without opening with it as a heading. And a non-text Spb-content-type is treated as unsupported and returned as a failure immediately, since no amount of rendering or proxying turns a PDF into an HTML article.
cached_fetch_url only stores successes for the same reason: caching a transient failure would make it permanent for the rest of the process, so a source that failed once because of a momentary rate limit would stay unreachable even after the rate limit passed. It holds the cache in memory for the life of the process; a longer-running agent that persists across sessions would want something backed by a file or a database instead.
On billing: with transparent_status_code enabled, every response fetch_url gets back is billable, not just 200, 404, and 410. Calling all of those "successful" gets confusing once 404 and a blocked 403 are both in the billable set, since neither is a success in the way a 200 is, just a request ScrapingBee completed and charged for.
What clean retrieval does for answer quality and cost
Clean Markdown removes navigation, ads, and boilerplate before a single token reaches the model. That means lower token cost per document, and the model working from the article itself instead of an empty shell or a page full of menu links. The same logic applies well beyond research workflows; any pipeline collecting data for machine learning benefits from clean, structured input for similar reasons.
Scrape responsibly for research
A research workflow should behave like a good web citizen: scrape public data only, respect robots.txt and reasonable rate limits, and skip anything that sits behind a login. The retry logic in fetch_url builds in part of this: it backs off with exponential delay on every retry. In two live bursts of 20 concurrent requests that did trigger a real 429, ScrapingBee never included a Retry-After header, so the code doesn't lean on that header being present. It's read opportunistically when it happens to show up, but exponential backoff is the fallback the retry logic actually depends on. cached_fetch_url, defined above, covers the rest: call it instead of fetch_url directly, and the same URL requested twice in one run (a source cited from two different sub-questions, for instance) is fetched once. Attribute sources in the final report too, which a deep research system does anyway as part of producing a citable answer.
Scraping publicly available data is generally lawful in many jurisdictions, but a site's Terms of Service can restrict it, and the rules vary by jurisdiction. This isn't legal advice, and if your use case is consequential or runs at large scale, it's worth a real legal review. For a research workflow specifically, respecting rate limits is also the practical choice: a system that backs off is less likely to trip a blocking system in the first place. If you're weighing which scraping tool fits a responsible workflow at research scale, this comparison of AI web scraping tools runs through the options.
Give your research agent web access it can trust
ScrapingBee runs JavaScript, handles many common blocking mechanisms, and returns clean text or structured JSON in one call. Drop it into Local Deep Research, gpt-researcher, or a LangChain agent you built yourself in place of the fragile fetch step, and the research loop has real pages to work from far more often than it did before.
→ Get your free ScrapingBee API key
Web scraping for AI agents FAQs
How do deep research agents get web data?
In a loop. A planner breaks the question into sub-questions, a search API returns candidate URLs, a retrieval stage downloads and extracts each source's text, and the model synthesizes a cited report. That retrieval stage is where reliability problems concentrate, since it has to read arbitrary documents on the open web, where bot protection and JavaScript rendering get in the way.
Why does my agent get 403s or empty pages?
Because a plain HTTP request is easy for a site to distinguish from a real browser — missing headers, cookies, or TLS fingerprint behavior a browser would normally send. Sites return a 403 based on that pattern or on a suspicious request rate, and JavaScript-heavy pages return an empty shell to a downloader that can't run JavaScript. Many scrapers also skip checking the status code, so a block silently becomes empty text the system reasons over anyway.
What is the best way to give an LLM clean web content?
Convert each source to clean Markdown or plain text at the retrieval stage, so the model gets the article without the navigation, ads, and boilerplate that a lot of sites carry. A web scraping API that returns Markdown handles this in one call, with no selectors to maintain, though it's worth validating the response rather than trusting the status code alone.
Can I use ScrapingBee with gpt-researcher or LangChain?
Yes. For LangChain, wrap a fetch function as a tool. For gpt-researcher, register a custom scraper class in place of the default BeautifulSoup path. For Local Deep Research, subclass BaseSearchEngine, override _get_full_content(), and add it to create_search_engine() in search_engine_factory.py, as shown earlier in this guide. In each case, the planner and synthesizer stay as they are; only the retrieval step changes.
How much does scraping for an agent cost?
It depends on how hard the target is. A plain request is cheapest, JavaScript rendering costs more, and the Stealth tier for the toughest bot protection costs the most; AI extraction adds a small amount on top of any of those. ScrapingBee treats 200, 404, and 410 responses as billable; most outright failures aren't. Check the pricing page for current numbers.
Do I need JavaScript rendering for research scraping?
Only for sources that need it. Many pages return their content in the initial HTML, so a plain request is enough. Single-page apps and many modern sites load content with JavaScript instead, so a plain request comes back empty; for those, turn on JavaScript rendering. Escalating per source, rather than rendering everything by default, keeps the cost down on sources that didn't need it.
