# ScrapingBee HTML API

> Fetch arbitrary URLs through ScrapingBee, with JavaScript rendering, proxy controls, browser interactions, screenshots, downloads, and data extraction.

## Endpoint and authentication

- Endpoint: `https://app.scrapingbee.com/api/v1/`
- Required input: `url`, including `http` or `https`.
- Recommended authentication: `Authorization: Bearer YOUR_API_KEY`.
- The `api_key` query parameter remains supported for backward compatibility but is deprecated for new integrations.
- URL-encode the target URL in raw HTTP requests; client libraries and `curl --data-urlencode` handle this safely.
- The endpoint accepts `GET`, `POST`, and `PUT`. For `POST` and `PUT`, send the target `url` and ScrapingBee forwards the request body transparently.

```bash
curl --get 'https://app.scrapingbee.com/api/v1/' \
  --header "Authorization: Bearer $SCRAPINGBEE_API_KEY" \
  --data-urlencode 'url=https://example.com'
```

## Core behavior

- JavaScript rendering is enabled by default.
- Failed target URL requests are retried for up to 30 seconds.
- Browser execution order is `wait_for` → `wait` → `js_scenario`.
- JSON-valued request parameters (`js_scenario`, `extract_rules`, and `ai_extract_rules`) must be stringified when sent in a GET query string.
- Non-HTML downloads are limited to 2 MB per request. Use `render_js=false` for files.

## Rendering and browser controls

| Parameter | Default / values | Important behavior |
| --- | --- | --- |
| `render_js` | `true` | Headless-browser rendering; set `false` for raw HTML or files. |
| `wait` | `0`–`35000` ms | Runs after `wait_for`. |
| `wait_for` | CSS or XPath selector | A selector beginning with `/` is XPath. |
| `wait_browser` | `domcontentloaded` | Also accepts `load`, `networkidle0`, and `networkidle2`. |
| `js_scenario` | JSON object | Runs after waits. |
| `block_ads` | `false` | Only affects JavaScript-rendered requests. |
| `block_resources` | `true` | Blocks images and CSS; set `false` when they matter. |
| `window_width`, `window_height` | `1920`, `1080` viewport dimensions | Only meaningful with `render_js=true`. |

`screenshot=true` requires JavaScript rendering and forces `block_resources=false`. By default it captures the viewport; use `screenshot_selector` for a CSS-scoped screenshot or `screenshot_full_page=true` for a full-page image. Set `json_response=true` when both screenshot and HTML are needed.

## Proxy and automatic modes

| Parameter | Behavior |
| --- | --- |
| `mode=auto` | GET-only automatic escalation through the cost ladder below. Charges only the configuration that succeeds; a total failure costs 0. |
| `max_cost` | Integer `>= 1`; caps Auto-Mode's attempted cost and requires `mode=auto`. |
| `premium_proxy=true` | Residential proxy; 10 credits without JavaScript and 25 with JavaScript. |
| `stealth_proxy=true` | Currently requires JavaScript and costs 75 credits per successful API call. A stealth tier without JavaScript is documented as coming soon, not as an available configuration. |
| `country_code` | Lower-case ISO 3166-1 geolocation code. |
| `own_proxy` | Custom proxy; source documentation specifies user/password/host/port syntax and defaults the port to `1080` when omitted. |
| `custom_google=true` | Referenced by some integration documentation for direct Google domains or subdomains, but absent from the current HTML API parameter contract. Treat it as undocumented and version-sensitive; prefer the dedicated Google Search API for structured SERPs. |

Auto-Mode tries the first successful tier, from cheapest to most expensive: rotating without JavaScript (1), rotating with JavaScript (5), premium without JavaScript (10), premium with JavaScript (25), then stealth with JavaScript (75). It changes only `render_js`, `premium_proxy`, and `stealth_proxy`; it never adds or adjusts `js_scenario`, `wait_for`, `wait`, `block_resources`, country targeting, headers, or cookies. Pass required page-specific controls yourself.

`max_cost` applies only to Auto-Mode and prevents it from trying a tier above that amount; for example, `max_cost=25` prevents escalation to stealth. Do not combine `mode=auto` with `render_js`, `premium_proxy`, `stealth_proxy`, or `transparent_status_code`; the API returns `400`.

Stealth proxy does not support `infinite_scroll`, custom headers, custom cookies, `timeout`, or `evaluate_results` in a JSON response.

`own_proxy` syntax is `<protocol><username>:<password>@<host>:<port>`; the protocol is optional and an omitted port defaults to `1080`.

`custom_google` is neither defined nor priced in the current HTML API parameter contract. Tool documentation conflicts on its price (15 versus 20 credits), so verify both support and billing in the dashboard before using it. See [reference notes](https://www.scrapingbee.com/llms/reference-notes.txt).

## Headers, sessions, and response handling

| Parameter | Behavior |
| --- | --- |
| `forward_headers=true` | Forward caller headers plus ScrapingBee headers. In raw HTTP, set this flag and prefix each forwarded header with `Spb-`. Python and Node clients auto-forward supplied headers without the prefix or this flag. |
| `forward_headers_pure=true` | Forward only caller headers; useful only with `render_js=false`. In raw HTTP, still prefix each forwarded header with `Spb-`; Python and Node clients do not need the prefix. |
| `cookies` | Semicolon-separated cookies; `name` and `value` are required, with optional `domain`, `path`, and `expires` attributes separated by commas. |
| `session_id` | Reuses an IP for 5 minutes. Use a random integer from `0` to `10000000`. |
| `timeout` | `140000` ms by default; valid range is `1000`–`140000` ms. Allow a 0.5-second margin, and change it cautiously because it can reduce success rate. |
| `device` | `desktop` (default) or `mobile`. |
| `tag` | Arbitrary response-header label; it does not affect scraping. |
| `transparent_status_code=true` | Return the target status and body directly; every request is billed and target `500`s are not retried. |

Without `transparent_status_code`, target statuses other than `2xx` or `404` are returned as ScrapingBee `500`.

| Parameter | Result |
| --- | --- |
| `json_response` | `false` by default; set `true` for a JSON envelope rather than the raw target body. |
| `return_page_source=true` | HTML before JavaScript execution; unnecessary when `render_js=false`. |
| `return_page_markdown=true` | Main page content as Markdown. |
| `return_page_text=true` | Main page content as plain text. |
| `scraping_config` | Apply a saved dashboard configuration by name. |

The JSON envelope can include `headers`, `cost`, `initial-status-code`, `resolved-url`, `type`, `body`, `screenshot`, `cookies`, `evaluate_results`, `iframes`, `xhr`, `js_scenario_report`, and `metadata`. `body` is base64-encoded for downloaded binary content, and `type` distinguishes `html`, `json`, and binary response types.

## Cost and status rules

| Configuration | Credits |
| --- | --- |
| Rotating proxy, `render_js=false` | 1 |
| Rotating proxy, `render_js=true` (default) | 5 |
| Premium proxy, `render_js=false` | 10 |
| Premium proxy, `render_js=true` | 25 |
| Stealth proxy, `render_js=false` | Coming soon; not currently available |
| Stealth proxy, `render_js=true` | 75 |
| `ai_query` and/or `ai_extract_rules` | +5 |

| Status | Billed | Meaning |
| --- | --- | --- |
| `200` | Yes | Success |
| `400` | No | Invalid request |
| `401` | No | No more credit available |
| `404`, `410` | Yes | Target missing or gone |
| `413` | No | Download too large |
| `429` | No | Concurrency limit |
| `500` | No | Miscellaneous error; retry |

Response headers include `Spb-cost`, `Spb-initial-status-code`, and `Spb-resolved-url`. `Spb-auto-cost` appears only with `mode=auto`. Target headers and cookies are prefixed with `Spb-`.

AI extraction adds 5 credits to the underlying configuration: a default JavaScript-rendered request with `ai_query` costs 10 credits, while premium proxy with JavaScript and AI extraction costs 30 credits.

## Data extraction

Use `extract_rules` when the page structure is known. Use AI extraction for semantic or structurally irregular fields; `ai_query` and `ai_extract_rules` add 5 credits. `extract_rules` and `ai_extract_rules` return the requested structured JSON as the normal response body rather than the target page HTML. Full selector, output, nested-schema, table, AI-schema, and `ai_selector` guidance: [data extraction](https://www.scrapingbee.com/llms/data-extraction.txt).

## JavaScript scenarios

Pass a stringified JSON object in `js_scenario`. Instructions run sequentially after `wait_for` and `wait`; the full instruction contract, strict-mode behavior, JSON debugging report, and 40-second limit are in [JavaScript scenarios](https://www.scrapingbee.com/llms/js-scenario.txt).

## Proxy transport and country targeting

For proxy-compatible clients, see [proxy mode](https://www.scrapingbee.com/llms/proxy-mode.txt), including endpoints, credentials, TLS handling, forwarded-header behavior, and proxy-mode Google billing.

Use lower-case ISO 3166-1 values with `country_code`. Classic and premium proxy coverage differs; classic unsupported values silently fall back to `us`. Use [premium country codes](https://www.scrapingbee.com/llms/country-codes-premium.txt) or [classic country codes](https://www.scrapingbee.com/llms/country-codes-classic.txt). The HTML API also documents country targeting with stealth proxies.