Fast search is basically the backbone for many modern AI setup now. Agents, chatbots, RAG loops, analytics tools, even custom LLM training — all of them need fresh web data to stay useful. Your model can be a genius and your prompts can be perfect, but if the info feeding it is stale, the whole thing falls apart.
And this is where the pain usually kicks in. The product is growing, users are happy, everything looks good, but the search layer is the part that keeps slowing things down. Captchas, IP blocks, flaky scrapers, random breakages, the classic "why did our SERP job die again?" Even when it behaves, it's often slow, fragile, and eats way too much engineering time.
That's why search isn't an optional add-on anymore. It's core infrastructure. And if that part of the stack can't stay fast or reliable, the rest of your AI system feels it right away. In this article we'll walk through what Fast Search API is, why it matters, and how to use it.
If you ever need a full-fledged web scraping API (browser automation, proxy rotation, or JavaScript rendering), ScrapingBee is here to save the day!
TL;DR
Fast search API gives you real-time SERP data (organic + news when available) as clean JSON in under a second, with zero data retention and no scraping infrastructure. Great for AI agents, chatbots, LLM training loops, analytics, and competitive intelligence.
Quick test with cURL:
curl "https://app.scrapingbee.com/api/v1/fast_search?api_key=YOUR_KEY&search=ai%20news%20today"
That's all you need to get structured SERP results instantly.
What is Fast Search?
Fast search is a simple API endpoint that gives you real search results — the same stuff you'd see in the browser — in JSON format, almost instantly. You don't need any browsers or scraping tricks; it's a direct way to ask "what does search engine show for this query right now?" and get the answer back. It's built for AI teams, devs, and data folks who need fresh, structured search data.
In short: it's a real-time search layer you can drop into any AI or data workflow without building or maintaining your own search infrastructure.
The problem: Search pipelines are slowing AI down
Fast search sounds simple on paper, but most teams know the real story: SERP scraping is a house of cards. One captcha and the whole thing collapses. One rate limit spike and your agents freeze. One IP block and your "real-time" system suddenly feels like dial-up. It's basically trying to run a race while dragging a shopping cart with a busted wheel. Possible? Sure. Fun? Absolutely not.
Most teams hit the same problems:
1. Constant breakages
- Captchas out of nowhere
- IP blocks when traffic spikes
- Scrapers drifting out of sync with page changes
- That "mystery failure" you only notice when users complain
Keeping this layer alive is a full-time job no one actually wants. (I definitely don't want that job, trust me.)
2. Latency that kills workflows
Agents don't wait. Chatbots don't wait. And stakeholders sure as hell don't wait. If your pipeline relies on news, finance, or anything time-sensitive, slow SERP pulls turn the whole system into a slideshow.
3. Messy, inconsistent data
Raw HTML, missing snippets, broken metadata... You can clean it, but feeding that into an LLM is like handing someone a crumpled receipt and asking for a full report. They'll do something, but it won't be great.
4. Compliance and privacy overhead
The moment you store queries or logs, you enter a whole new world of:
- retention rules
- privacy audits
- "wait, who has access to this?" conversations
Every extra layer adds risk you didn't ask for. Put all of this together, and search stops being a simple "grab some results" step. It becomes the bottleneck that slows your entire AI stack down.
What AI teams actually need from search
Most teams aren't asking for "yet another SERP scraping API." They want fast search that actually feels like part of their AI stack. Something that doesn't break every other morning or demand a sacrificial engineer just to babysit it. And when you look at what they deal with day to day, the wishlist is pretty simple:
- Structured SERP data. Titles, links, snippets — ready to use. Not raw HTML that feels like someone handed you a box of tangled cables and said "figure it out."
- Real-time speed. Agents don't think in minutes. If your search call takes longer than a second, the whole flow feels like waiting for a friend who keeps saying "I'm on my way" while still at home putting on shoes.
- Zero retention. No logs, no stored queries, no "wait, do we have to run this by legal?" moments. Just fetch results and move on with your life.
- Easy integration. A simple request you can drop in without pulling core engineers off real work. No home-built scrapers, no proxy juggling, no half-broken code stitched together from five repos.
That's the baseline. Not hype, not magic: just the things you need to keep AI products fast, fresh, and actually pleasant to maintain.
Introducing Fast Search API
Alright, let's speak plainly. Fast Search API is a real-time SERP endpoint for people who just want search to work, not turn into a side quest you never signed up for. It's built for AI teams, data engineers, backend folks, and anyone wiring search straight into agents, RAG loops, dashboards, or training jobs.
One call → structured SERP JSON → under a second → zero retention. That's the whole thing. No puppeteer farms, no captcha roulette, no "why is our parser exploding again?" nights. Just fast search that gives your AI the data it needs without dragging the rest of your system down.
How Fast Search API works at a high level
The Search API is a thin, fast layer sitting on top of a large web-data backend you never have to touch. One request in — real-time SERP data out. At a high level, here's what you get:
- Real-time organic results + top news stories. The two data types most AI workflows rely on, returned in one go.
- Structured response built for LLMs and analytics. Rank, title, link, snippet, extensions, and a clean array of news items. No HTML cleanup, no parsing glue code.
- Pagination for deeper search. Up to 10 pages per query (around 100 results total), depending on how many pages search engine exposes for that term.
- Sub-second latency and stable performance. Tuned for fast pulls under normal load, so agents and RAG loops don't stall.
- Backed by our web-data collection ecosystem. All the anti-bot handling, scaling, and reliability work happens behind the scenes — you never see it, and you never manage it.
- Zero data retention. User queries and results aren't kept. You get your JSON response and that's it.
Put together, it's basically search that behaves the way AI teams always wanted: fast, predictable, and not another system you have to babysit.
Key use cases and examples
- Answer engines & AI chatbots. You can ground your model's answers with real-time results and fresh news, so it doesn't spit out yesterday's facts. For example, an agent checking "latest Nvidia earnings" can grab live SERP + top stories in under a second and answer without guessing.
- LLM training & data ingestion. If you're running continuous training, evals, or any automated enrichment, fast search gives you a clean stream of structured SERP data you can drop straight into your pipeline. Daily pulls of a few hundred or thousand queries become simple.
- Research & competitive intelligence. Teams tracking markets, brands, or competitors can hit fast search on a schedule and always know what's moving. Checking something like "best payroll software" every few hours gives you up-to-date ranking shifts you can push right into your dashboards.
What this unlocks for your team
- Faster time-to-market. You stop burning cycles on building or babysitting a SERP layer. Just plug in fast search and ship features instead of fighting scrapers and proxy setups.
- More accurate, up-to-date outputs. Agents and models finally work with what's happening right now instead of relying on whatever they remember from old pretraining data.
- Way less infra and maintenance pain. No scrapers to patch, no captchas to dodge, no IP pools to rotate, no retention risks to explain to legal. The entire search layer disappears from your to-do list.
How to get started
Getting rolling with Fast Search API is dead simple. No SDK maze, no weird setup flow, just an API key and one request.
1. Authenticate
Pass your ScrapingBee API key as a query param. That's literally it.
You can find your API key in the ScrapingBee dashboard.
2. Make a basic request
Here's the simplest possible curl example:
curl "https://app.scrapingbee.com/api/v1/fast_search?api_key=YOUR_KEY&search=latest%20ai%20news"
You get back a structured JSON payload with organic results, top stories, ranks, links, and snippets. Clean SERP data ready for LLMs, agents, or any analytics pipeline.
Find full documentation at scrapingbee.com/documentation/fast-search/.
3. Do a few fast workflow checks
A few quick experiments to see the value right away:
- Drop it into your chatbot or RAG loop so responses pull live info instead of stale memory.
- Add a scheduled fast search call to your analytics or BI pipeline.
- Build a lightweight CI dashboard that checks key industry queries every hour.
- From there, it's basically: send query → get clean SERP → keep building.
A practical Fast Search Python demo you can run in 5 minutes
You've probably got the itch to try this already, so let's stop talking and ship a tiny script. It shows the full loop end-to-end: call the API, get structured SERP data back, turn it into a clean "context" block you can paste into an LLM, and also export JSONL for training, eval, or ingestion.
Step 1: Setup and load your API key from .env
We'll keep secrets out of the code and load SCRAPINGBEE_API_KEY from a .env file. So, create a .env file:
SCRAPINGBEE_API_KEY=your_key_here
Install dependencies:
pip install requests python-dotenv
Let's also add all the necessary imports and constants into the fast_search_demo.py file:
"""
ScrapingBee Fast Search -> normalized docs -> context.txt + JSONL (Python 3.12+)
What it does:
- Calls ScrapingBee Fast Search API
- Normalizes results into stable "Document" objects
- Writes a prompt-ready context.txt
- Appends a JSONL record for ingestion (training/eval/monitoring)
Requirements:
pip install requests python-dotenv
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
import json
import logging
import os
import time
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
import requests
from dotenv import load_dotenv
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
FAST_SEARCH_URL = "https://app.scrapingbee.com/api/v1/fast_search"
DEFAULT_TIMEOUT_S = 10
# Hard-coded search settings (keep it simple).
QUERY = "ai news today"
COUNTRY_CODE = "us"
LANGUAGE = "en"
PAGE = 1
MAX_CONTEXT_ITEMS = 8
OUTDIR = Path(".")
CONTEXT_PATH = OUTDIR / "context.txt"
JSONL_PATH = OUTDIR / "serp_results.jsonl"
# Remove common invisible characters and normalize NBSP -> space.
_TRANSLATION = str.maketrans(
{
"\u00A0": " ", # NBSP
"\u200B": None, # zero-width space
"\u200C": None, # zero-width non-joiner
"\u200D": None, # zero-width joiner
"\uFEFF": None, # BOM / zero-width no-break space
}
)
You also might be interested in checking our Python Web Scraping tutorial with examples.
Step 2: Make the request to the Search API
Add a new fast_search() function into the fast_search_demo.py file:
def fast_search(
session: requests.Session,
*,
api_key: str,
query: str,
country_code: str,
language: str,
page: int,
timeout: tuple[float, float],
) -> dict[str, Any]:
"""
Call ScrapingBee Fast Search API and return parsed JSON.
Notes:
- `query` is what you'd type into the search engine.
- `country_code`, `language`, `page` localize/paginate.
"""
params = {
"api_key": api_key,
"search": query,
"country_code": country_code,
"language": language,
"page": page,
}
resp = session.get(FAST_SEARCH_URL, params=params, timeout=timeout)
resp.raise_for_status()
payload = resp.json()
if not isinstance(payload, dict):
raise ValueError("Unexpected JSON payload type (expected object).")
return payload
def build_session(timeout_s: int) -> tuple[requests.Session, tuple[float, float]]:
"""
Build a requests session with retries.
Returns:
(session, timeout_tuple)
"""
session = requests.Session()
retry = Retry(
total=3,
connect=3,
read=3,
backoff_factor=0.4,
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=("GET",),
raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10)
session.mount("https://", adapter)
session.mount("http://", adapter)
# requests supports a (connect, read) timeout tuple
timeout = (min(5, timeout_s), float(timeout_s))
return session, timeout
def main() -> None:
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger("fast_search_demo")
load_dotenv()
api_key = os.getenv("SCRAPINGBEE_API_KEY")
if not api_key:
raise SystemExit("Missing SCRAPINGBEE_API_KEY. Add it to your .env file.")
session, timeout = build_session(DEFAULT_TIMEOUT_S)
started = time.time()
payload = fast_search(
session,
api_key=api_key,
query=QUERY,
country_code=COUNTRY_CODE,
language=LANGUAGE,
page=PAGE,
timeout=timeout,
)
took_ms = int((time.time() - started) * 1000)
Key points:
- We define
fast_search()— a small wrapper around the Search API. You pass a query and a few settings, and you get JSON back. - We build a reusable
requestssession With retries, timeouts, and connection pooling so you don't get random failures. - We load the API key from
.envKeeps secrets out of your code and makes the script easy to reuse.
Step 3: Parse the response into clean "documents"
Before we use the data anywhere, we need to turn the raw API payload into something your model or pipeline can actually work with. Organic results and top stories look similar but not identical, so this step just flattens everything into one list of documents. One format, same fields, no surprises. After this, anything you build (LLM prompts, embeddings, dashboards, JSONL) becomes way easier.
@dataclass(slots=True, frozen=True)
class Document:
source: str
rank: int | None
title: str
url: str
snippet: str
def is_useful(self) -> bool:
return bool(self.title) and bool(self.url) and is_http_url(self.url)
def is_http_url(value: str) -> bool:
"""Basic sanity check: keep only http(s) URLs."""
try:
p = urlparse(value)
except ValueError:
return False
return p.scheme in {"http", "https"} and bool(p.netloc)
def to_int(value: Any) -> int | None:
"""Best-effort int conversion (rank/position fields can be messy)."""
if value is None:
return None
try:
return int(value)
except (TypeError, ValueError):
return None
def pick_first(*values: Any) -> Any:
"""Return the first non-empty value (not None / not empty string)."""
for v in values:
if v is None:
continue
if isinstance(v, str) and not v.strip():
continue
return v
return None
def normalize_text(value: Any) -> str:
"""
Normalize a single text field.
SERP payloads can be inconsistent: sometimes fields are missing, empty, or not strings.
We normalize defensively so downstream code doesn't care.
- Accept Any (sometimes SERP fields are not strings)
- Best-effort conversion for lists/objects
- Strip invisibles + collapse whitespace
"""
if value is None:
return ""
if not isinstance(value, str):
if isinstance(value, list):
parts = [v for v in value if isinstance(v, str) and v.strip()]
value = " ".join(parts) if parts else ""
else:
value = str(value)
cleaned = value.translate(_TRANSLATION)
return " ".join(cleaned.split())
def parse_documents(payload: dict[str, Any]) -> list[Document]:
"""
Normalize the API response into a stable list of Document objects.
Snippet fields are not guaranteed. We try multiple keys because SERP payloads can vary.
"""
docs: list[Document] = []
for r in (payload.get("organic") or []):
if not isinstance(r, dict):
continue
snippet_raw = pick_first(
r.get("description"),
r.get("snippet"),
r.get("content"),
r.get("text"),
)
docs.append(
Document(
source="organic",
rank=to_int(r.get("rank")),
title=normalize_text(r.get("title")),
url=normalize_text(r.get("link")),
snippet=normalize_text(snippet_raw),
)
)
for r in (payload.get("top_stories") or []):
if not isinstance(r, dict):
continue
snippet_raw = pick_first(
r.get("description"),
r.get("snippet"),
r.get("content"),
r.get("text"),
)
docs.append(
Document(
source="top_story",
rank=to_int(pick_first(r.get("rank"), r.get("position"))),
title=normalize_text(r.get("title")),
url=normalize_text(pick_first(r.get("link"), r.get("url"))),
snippet=normalize_text(snippet_raw),
)
)
useful = [d for d in docs if d.is_useful()]
# Keep ordering predictable: top stories first, then organic; both sorted by rank when possible.
def sort_key(d: Document) -> tuple[int, int]:
source_prio = 0 if d.source == "top_story" else 1
rank = d.rank if d.rank is not None else 10_000
return (source_prio, rank)
useful.sort(key=sort_key)
return useful
What's happening here:
- Normalize everything (organic + top stories) into one
Documentformat. - Clean titles, URLs, and snippets so downstream code doesn't choke on weird fields.
- Pick the best snippet from several possible keys.
- Drop anything useless (empty titles, bad URLs).
- Sort results in a predictable order (top stories first, then organic by rank).
Then in main(), after you build organic and top_stories, you can do:
def main() -> None:
# get the payload ...
docs = parse_documents(payload)
log.info("Status: %s | Latency: ~%sms", payload.get("status"), took_ms)
log.info(
"Organic: %s | Top stories: %s | Parsed docs: %s",
len(payload.get("organic") or []),
len(payload.get("top_stories") or []),
len(docs),
)
Now you've got one list that's easy to reuse in the next step, where we export a prompt-ready context.txt and a JSONL stream for ingestion.
Step 4: Export useful outputs (context.txt + JSONL for ingestion)
Now that you've turned the raw payload into documents, this is the part where the data becomes actually useful. Most AI teams want two outputs right away:
- A readable context block they can drop straight into an LLM or agent step.
- A JSONL record they can feed into training, eval, or monitoring pipelines.
This step sets up both so you can plug Fast Search into real workflows, not just print data to the console.
Build a prompt-ready context.txt
The code below takes your normalized documents and turns them into a compact, LLM-friendly block. One bullet per result, optional snippet, and a clean format that plays nicely with prompts and agent memory.
def build_context(docs: list[Document], *, max_items: int) -> str:
"""
Build a prompt-friendly context block.
Output stays readable:
- One bullet per doc
- Snippet printed only when present
"""
lines: list[str] = []
for d in docs[:max_items]:
rank = d.rank if d.rank is not None else "-"
if d.snippet:
lines.append(f"- [{d.source} #{rank}] {d.title} ({d.url})\n {d.snippet}")
else:
lines.append(f"- [{d.source} #{rank}] {d.title} ({d.url})")
return "\n".join(lines).strip()
Write it out inside main():
def main() -> None:
# parse documents, print logs...
context = build_context(docs, max_items=MAX_CONTEXT_ITEMS)
CONTEXT_PATH.write_text(context + "\n", encoding="utf-8")
log.info("Wrote %s", CONTEXT_PATH)
You can plug it straight into your LLM pipeline later.
Append a JSONL record for ingestion
Here we generate the "pipeline-friendly" output. JSONL is the simplest way to store search results for training, datasets, eval sets, analytics, scheduled ETL, or CI watchers. The helper just writes a single JSON record to the file, and main() packs in the useful fields:
def append_jsonl(path: Path, record: dict[str, Any]) -> None:
"""
Append one JSON record per line (JSONL).
"""
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a", encoding="utf-8") as f:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
Use it inside main():
def main() -> None:
# parse documents, print logs, generate context ...
record = {
"ts": int(time.time()),
"query": QUERY,
"latency_ms": took_ms,
"docs": [asdict(d) for d in docs],
"raw_status": payload.get("status"),
"raw_url": normalize_text(payload.get("url")),
}
append_jsonl(JSONL_PATH, record)
log.info("Appended %s", JSONL_PATH)
This gives you two outputs in one run:
- context.txt → something your LLM or agent can use immediately
- serp_results.jsonl → something your pipelines can consume later
Most AI teams don't need anything fancier to start wiring Fast Search into real workflows.
Final code
And so here's the final code version:
#!/usr/bin/env python3
"""
ScrapingBee Fast Search -> normalized docs -> context.txt + JSONL (Python 3.12+)
What it does:
- Calls ScrapingBee Fast Search API
- Normalizes results into stable "Document" objects
- Writes a prompt-ready context.txt
- Appends a JSONL record for ingestion (training/eval/monitoring)
Requirements:
pip install requests python-dotenv
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
import json
import logging
import os
import time
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
import requests
from dotenv import load_dotenv
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
FAST_SEARCH_URL = "https://app.scrapingbee.com/api/v1/fast_search"
DEFAULT_TIMEOUT_S = 10
# Hard-coded search settings (keep it simple).
QUERY = "ai news today"
COUNTRY_CODE = "us"
LANGUAGE = "en"
PAGE = 1
MAX_CONTEXT_ITEMS = 8
OUTDIR = Path(".")
CONTEXT_PATH = OUTDIR / "context.txt"
JSONL_PATH = OUTDIR / "serp_results.jsonl"
# Remove common invisible characters and normalize NBSP -> space.
_TRANSLATION = str.maketrans(
{
"\u00A0": " ", # NBSP
"\u200B": None, # zero-width space
"\u200C": None, # zero-width non-joiner
"\u200D": None, # zero-width joiner
"\uFEFF": None, # BOM / zero-width no-break space
}
)
def normalize_text(value: Any) -> str:
"""
Normalize a single text field.
SERP payloads can be inconsistent: sometimes fields are missing, empty, or not strings.
We normalize defensively so downstream code doesn't care.
- Accept Any (sometimes SERP fields are not strings)
- Best-effort conversion for lists/objects
- Strip invisibles + collapse whitespace
"""
if value is None:
return ""
if not isinstance(value, str):
if isinstance(value, list):
parts = [v for v in value if isinstance(v, str) and v.strip()]
value = " ".join(parts) if parts else ""
else:
value = str(value)
cleaned = value.translate(_TRANSLATION)
return " ".join(cleaned.split())
def is_http_url(value: str) -> bool:
"""Basic sanity check: keep only http(s) URLs."""
try:
p = urlparse(value)
except ValueError:
return False
return p.scheme in {"http", "https"} and bool(p.netloc)
def pick_first(*values: Any) -> Any:
"""Return the first non-empty value (not None / not empty string)."""
for v in values:
if v is None:
continue
if isinstance(v, str) and not v.strip():
continue
return v
return None
def to_int(value: Any) -> int | None:
"""Best-effort int conversion (rank/position fields can be messy)."""
if value is None:
return None
try:
return int(value)
except (TypeError, ValueError):
return None
@dataclass(slots=True, frozen=True)
class Document:
source: str
rank: int | None
title: str
url: str
snippet: str
def is_useful(self) -> bool:
# Snippet can be empty; title+url is the real minimum.
return bool(self.title) and bool(self.url) and is_http_url(self.url)
def build_session(timeout_s: int) -> tuple[requests.Session, tuple[float, float]]:
"""
Build a requests session with retries.
Returns:
(session, timeout_tuple)
"""
session = requests.Session()
retry = Retry(
total=3,
connect=3,
read=3,
backoff_factor=0.4,
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=("GET",),
raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10)
session.mount("https://", adapter)
session.mount("http://", adapter)
# requests supports a (connect, read) timeout tuple
timeout = (min(5, timeout_s), float(timeout_s))
return session, timeout
def fast_search(
session: requests.Session,
*,
api_key: str,
query: str,
country_code: str,
language: str,
page: int,
timeout: tuple[float, float],
) -> dict[str, Any]:
"""
Call ScrapingBee Fast Search API and return parsed JSON.
Notes:
- `query` is what you'd type into the search engine.
- `country_code`, `language`, `page` localize/paginate.
"""
params = {
"api_key": api_key,
"search": query,
"country_code": country_code,
"language": language,
"page": page,
}
resp = session.get(FAST_SEARCH_URL, params=params, timeout=timeout)
resp.raise_for_status()
payload = resp.json()
if not isinstance(payload, dict):
raise ValueError("Unexpected JSON payload type (expected object).")
return payload
def parse_documents(payload: dict[str, Any]) -> list[Document]:
"""
Normalize the API response into a stable list of Document objects.
Snippet fields are not guaranteed. We try multiple keys because SERP payloads can vary.
"""
docs: list[Document] = []
for r in (payload.get("organic") or []):
if not isinstance(r, dict):
continue
snippet_raw = pick_first(
r.get("description"),
r.get("snippet"),
r.get("content"),
r.get("text"),
)
docs.append(
Document(
source="organic",
rank=to_int(r.get("rank")),
title=normalize_text(r.get("title")),
url=normalize_text(r.get("link")),
snippet=normalize_text(snippet_raw),
)
)
for r in (payload.get("top_stories") or []):
if not isinstance(r, dict):
continue
snippet_raw = pick_first(
r.get("description"),
r.get("snippet"),
r.get("content"),
r.get("text"),
)
docs.append(
Document(
source="top_story",
rank=to_int(pick_first(r.get("rank"), r.get("position"))),
title=normalize_text(r.get("title")),
url=normalize_text(pick_first(r.get("link"), r.get("url"))),
snippet=normalize_text(snippet_raw),
)
)
useful = [d for d in docs if d.is_useful()]
# Keep ordering predictable: top stories first, then organic; both sorted by rank when possible.
def sort_key(d: Document) -> tuple[int, int]:
source_prio = 0 if d.source == "top_story" else 1
rank = d.rank if d.rank is not None else 10_000
return (source_prio, rank)
useful.sort(key=sort_key)
return useful
def build_context(docs: list[Document], *, max_items: int) -> str:
"""
Build a prompt-friendly context block.
Output stays readable:
- One bullet per doc
- Snippet printed only when present
"""
lines: list[str] = []
for d in docs[:max_items]:
rank = d.rank if d.rank is not None else "-"
if d.snippet:
lines.append(f"- [{d.source} #{rank}] {d.title} ({d.url})\n {d.snippet}")
else:
lines.append(f"- [{d.source} #{rank}] {d.title} ({d.url})")
return "\n".join(lines).strip()
def append_jsonl(path: Path, record: dict[str, Any]) -> None:
"""
Append one JSON record per line (JSONL).
"""
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a", encoding="utf-8") as f:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
def main() -> None:
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger("fast_search_demo")
load_dotenv()
api_key = os.getenv("SCRAPINGBEE_API_KEY")
if not api_key:
raise SystemExit("Missing SCRAPINGBEE_API_KEY. Add it to your .env file.")
session, timeout = build_session(DEFAULT_TIMEOUT_S)
started = time.time()
payload = fast_search(
session,
api_key=api_key,
query=QUERY,
country_code=COUNTRY_CODE,
language=LANGUAGE,
page=PAGE,
timeout=timeout,
)
took_ms = int((time.time() - started) * 1000)
# Debug tip (optional): if snippets look empty, inspect raw fields.
# first = (payload.get("organic") or [{}])[0]
# print("description type:", type(first.get("description")), "value:", repr(first.get("description"))[:200])
# print("snippet type:", type(first.get("snippet")), "value:", repr(first.get("snippet"))[:200])
docs = parse_documents(payload)
log.info("Status: %s | Latency: ~%sms", payload.get("status"), took_ms)
log.info(
"Organic: %s | Top stories: %s | Parsed docs: %s",
len(payload.get("organic") or []),
len(payload.get("top_stories") or []),
len(docs),
)
# 1) Prompt-ready context
context = build_context(docs, max_items=MAX_CONTEXT_ITEMS)
CONTEXT_PATH.write_text(context + "\n", encoding="utf-8")
log.info("Wrote %s", CONTEXT_PATH)
# 2) JSONL record
record = {
"ts": int(time.time()),
"query": QUERY,
"latency_ms": took_ms,
"docs": [asdict(d) for d in docs],
"raw_status": payload.get("status"),
"raw_url": normalize_text(payload.get("url")),
}
append_jsonl(JSONL_PATH, record)
log.info("Appended %s", JSONL_PATH)
if __name__ == "__main__":
main()
Here's an example from the context.txt file:
- [top_story #1] Accel doubles down on Fibr AI as agents turn static websites into one-to-one experiences (https://techcrunch.com/2026/02/04/accel-doubles-down-on-fibr-ai-as-agents-turn-static-websites-into-one-to-one-experiences/)
- [top_story #2] Musk's SpaceX and xAI merge to make world's most valuable private company (https://www.bbc.com/news/articles/cq6vnrye06po)
And here's JSONL:
{"ts": 1770219098, "query": "ai news today", "latency_ms": 972, "docs": [{"source": "organic", "rank": 7, "title": "Artificial intelligence - BBC News", "url": "https://www.bbc.com/news/topics/ce1qrvleleqt", "snippet": "9 hours ago · A study by Bangor University says a third of young people would rather speak to AI than a friend. 2 days ago."}], "raw_status": "done", "raw_url": "RAQ_QUERY_URL_HERE"}
Ready to try ScrapingBee?
Fast search shouldn't be a headache. With the ScrapingBee Fast Search API you get clean, structured SERP data in under a second — no home-built scrapers, no proxy juggling, no captcha fights, no odd infrastructure side quests. Just one simple request, a predictable JSON response, and results your agents, models, and pipelines can actually use.
If search is part of your AI stack (and these days it always is) this gives you the speed, structure, and compliance you need without owning any of the messy plumbing behind it.
If you want to try it yourself, spin up a free trial and grab your first 1000 credits as a gift. The setup takes a minute, and you'll see real value on the first request.
Fast Search: Frequently asked questions
What is Fast Search API, in plain English?
It's a super-lightweight SERP endpoint: you send a query, it gives you real-time search results (organic + news when available) as clean JSON. No scraping setup, no proxies, no captchas.
Why would I use this instead of scraping results myself?
Because scraping is brittle as hell. Maintaining headless browsers, IP pools, captchas, parsing, breakage fixes: all of that turns into a full-time distraction. Our Search API gives you the data nearly instantly without owning any of that infrastructure.
What are the main use cases?
- Grounding AI agents and chatbots with fresh search results
- Feeding structured data into LLM training/eval loops
- Competitive intelligence and market monitoring
- Dashboards and analytics that need up-to-date info
- Any workflow where "latest results" actually matter
How fast is it really?
Under normal load you get responses in less than 1 second. It's built for real-time agent workflows, not background scraping jobs.
Does it store my queries or data?
Nope. Fast Search follows a zero-retention model for user queries and results. You get the response, and nothing you send is kept afterward. Solid for compliance and privacy-sensitive pipelines.
How many results can I get?
Each page gives you the top organic results + top stories when search engine shows them. You can paginate up to 10 pages (about ~100 results depending on the query).
Ilya is an IT tutor and author, web developer, and ex-Microsoft/Cisco specialist. His primary programming languages are Ruby, JavaScript, Python, and Elixir. He enjoys coding, teaching people and learning new things. In his free time he writes educational posts, participates in OpenSource projects, tweets, goes in for sports and plays music.
