# ScrapingBee CLI

> Use the official ScrapingBee command-line interface for scraping, crawling, exports, schedules, and dedicated APIs.

## Installation and authentication

```bash
uv tool install scrapingbee-cli
# or
pip install scrapingbee-cli
```

- Verify installation with `scrapingbee --version`.
- Authenticate interactively with `scrapingbee auth`.
- Authenticate non-interactively with `scrapingbee auth --api-key YOUR_API_KEY` or `SCRAPINGBEE_API_KEY`.
- The CLI also reads `SCRAPINGBEE_API_KEY` from a `.env` file in the current directory. Use `scrapingbee auth --show` to reveal the stored-key location.
- Use `scrapingbee logout` to remove the stored key.

## Commands

| Command | Purpose |
| --- | --- |
| `scrapingbee scrape URL` | HTML API; API parameter names map to CLI flags, such as `render_js` → `--render-js`. |
| `scrapingbee crawl` | Quick same-domain crawl, sitemap crawl, or Scrapy-project spider. |
| `scrapingbee export` | Merge batch or crawl output as NDJSON, text, or CSV. |
| `scrapingbee schedule` | Cron-based scheduled jobs; gated advanced feature. |
| `scrapingbee usage` | Credit and concurrency JSON (`max_api_credit`, `used_api_credit`, `max_concurrency`, `current_concurrency`, `renewal_subscription_date`). |
| `scrapingbee google`, `fast-search` | Google and Fast Search APIs. |
| `scrapingbee amazon-product`, `amazon-pricing`, `amazon-search` | Amazon APIs. |
| `scrapingbee walmart-product`, `walmart-search` | Walmart APIs. |
| `scrapingbee youtube-search`, `youtube-metadata` | YouTube APIs. |
| `scrapingbee chatgpt`, `gemini` | LLM APIs. |

```bash
scrapingbee scrape "https://example.com" --return-page-markdown true
scrapingbee amazon-product B08N5WRWNW --output-file product.json
scrapingbee chatgpt "Summarize the latest AI news" --search true
```

## Important behavior

- Quote URLs with double quotes and JSON or path values with single quotes.
- For LLM pipelines, use `--return-page-markdown true`, or `--chunk-size` and `--chunk-overlap` to emit NDJSON chunks.
- `--smart-extract` supersedes deprecated `--extract-field` and `--fields`; use spaces around its `|` and `&` operators.
- Batch input accepts text, CSV, or stdin. `--concurrency 0` chooses the plan limit; retry defaults to three with exponential backoff.
- Quick and sitemap crawling support HTML API parameters. In project-spider mode, configure ScrapingBee parameters in `ScrapingBeeRequest`; CLI API flags are rejected.
- `--escalate-proxy` is CLI-only and retries `403` or `429` through premium then stealth proxy.
- Shell-execution features (`--post-process`, `--on-complete`, `schedule`, and REPL `!command`) require `SCRAPINGBEE_ALLOW_EXEC=1` and `scrapingbee auth --unsafe`. Restrict allowed commands in automation.

## Scrape contract and presets

- `scrape` maps HTML API parameters to hyphenated CLI flags, for example `render_js` → `--render-js`. For enum values, both hyphens and underscores are accepted.
- All boolean flags accept explicit `true` or `false`; a bare flag is also accepted. An omitted flag is not sent, so the API default applies.
- Use `-H/--header` repeatedly for custom headers, `-X/--method` for `GET`, `POST`, or `PUT`, and `-d/--data` for a POST/PUT body.
- `--scraping-config NAME` loads a saved dashboard configuration. Inline flags override the saved setting. Names are case-sensitive and limited to alphanumeric characters, hyphens, and underscores.
- `--output-file` infers an extension unless you supply one; use `--force-extension` to override it and `--overwrite` to suppress an overwrite prompt.

Presets only fill flags not supplied explicitly:

| Preset | Result |
| --- | --- |
| `screenshot` | Viewport screenshot with JavaScript rendering. |
| `screenshot-and-html` | Full-page screenshot and HTML in one JSON response. |
| `fetch` | Raw fast fetch with JavaScript disabled. |
| `extract-links` | JSON extraction of all anchor `href` values. |
| `extract-emails` / `extract-phones` | Extract `mailto:` / `tel:` links. |
| `scroll-page` | JavaScript infinite-scroll scrape. |

`--escalate-proxy` retries a `403` or `429` first with premium proxy, then with stealth proxy. It is distinct from HTML API Auto-Mode and should be used only when a normal request is blocked.

## Interactive REPL

Run bare `scrapingbee` to open the REPL. All normal commands work there, `Ctrl+C` cancels an in-progress command, and `:set key=value` establishes session defaults that inline flags override.

| Meta-command | Behavior |
| --- | --- |
| `:help` / `:?` | Show commands and key bindings. |
| `:set`, `:unset`, `:reset`, `:show` | Manage session defaults. `:unset *` clears all. |
| `:view`, `:view PATH`, `:view crawl` | Open the last result, a file, or latest crawl log in a pager; `r` toggles raw/pretty output. |
| `:list` | List active scheduled jobs. |
| `:clear` | Clear the scrollback. |
| `:q`, `exit`, `quit` | Exit. |

If no key is configured, the REPL asks for a masked API key. `auth` and `logout` work in the REPL; `auth --unsafe` must run from a regular shell. Existing output files require confirmation unless `--overwrite` is passed.

## Smart Extract

`--smart-extract` is client-side extraction over auto-detected JSON, HTML, XML, CSV, NDJSON, Markdown, or plain text. It works on single-item `scrape`, Google/Fast Search, Amazon, Walmart, YouTube Search/Metadata, ChatGPT, and Gemini commands.

| Syntax | Meaning |
| --- | --- |
| `.key`, `(literal-key)` | Navigate object keys. |
| `[0]`, `[-1]`, `[0:5]`, `[0,2,4]` | Select array items. |
| `[keys]`, `[values]` | Get dictionary keys or values. |
| `...key`, `...*glob*` | Recursively search at any depth. |
| `~N` | Return the ancestor N levels above a match. |
| `[=text]`, `[=*glob*]`, `[=/regex/]`, `[!=pattern]` | Filter values. |
| `[key=pattern]`, `[key!=pattern]`, `[*=pattern]` | Filter objects by a property. |
| `path1 \| path2`, `path1 & path2` | OR / AND. Spaces around the operator are required, and the operators cannot be mixed. |

Output modes are: a single path (one raw value per line), comma-separated named blocks such as `'{titles:organic_results[0:3].title},{urls:organic_results[0:3].url}'`, or a JSON schema such as `'{"titles":"organic_results[0:3].title"}'`.

- In `scrape` batch mode it works for individual-file and CSV output, but not NDJSON streams.
- For non-`scrape` batch commands it is single-item only. Use deprecated `--extract-field` or `--post-process`/`jq` until batch support lands.

## Crawls, batches, and exports

- `crawl URL` is a same-domain quick crawl; `--from-sitemap` accepts a sitemap URL or path and follows sitemap indexes to depth 2; `crawl SPIDER --project DIR` runs a Scrapy project.
- `--max-depth 0` and `--max-pages 0` mean unlimited. `--max-pages` limits saved files, not discovery fetches; `--save-pattern` keeps only matching pages while still discovering links.
- Use `--include-pattern` / `--exclude-pattern`, `--allowed-domains`, or `--allow-external-domains` to scope a crawl. Combine external domains with a maximum page limit.
- Quick/sitemap crawls accept HTML API parameters. In project mode, set API parameters in `ScrapingBeeRequest`; the CLI rejects scrape API flags and `--scraping-config`, but accepts infrastructure controls such as concurrency, delay, and autothrottle.
- Crawl accepts client-side output/extraction flags for command parity, but `--output-file`, `--smart-extract`, `--extract-field`, and `--fields` do not change crawl output. Use API-side extraction or `export`/`jq` after the crawl.
- In non-HTML/extraction modes (screenshots, API-side extraction, text), a two-phase discovery flow can issue extra low-cost HTML requests before saving the requested output. The CLI prompts before this cost risk; use `--confirm yes` only after budgeting for it.
- Crawl output is numbered files plus `manifest.json`; `--resume` skips previously fetched URLs. `--concurrency 0` picks plan concurrency and is capped by `--max-pages`.

`--input-file` enables batch mode for scraper commands: `.txt` has one item per line, `.csv` uses `--input-column` (name or zero-based index), and `-` reads stdin. Default output is numbered files plus a manifest; `--output-format csv` or `ndjson` produces a merged stream/file. Use `--deduplicate`, `--sample`, and `--resume` to control spend and recover interrupted jobs; bare `scrapingbee --resume` finds incomplete batches in the current directory and prints resume commands. `--retries` defaults to 3 and `--backoff` to 2.0 (2s, 4s, 8s); `--verbose` prints status, cost, and resolved URL. `--update-csv` enriches a CSV in place, flattening JSON to dot-notation columns. `--post-process` applies to file/NDJSON output, not `--update-csv`.

`export` combines batch/crawl files into `ndjson` (default), text, or CSV. It annotates output with manifest URLs, supports nested JSON flattening (`--flatten`, default maximum depth 5), selected CSV columns, and CSV-row deduplication.

## Scheduling and shell-execution safety

- `schedule` creates `cron` jobs only on macOS/Linux; it is unavailable on Windows.
- Use `--every 5m`, `1h`, or `2d`; seconds are rounded down and the minimum interval is one minute. `--list`, `--stop NAME`, and `--stop all` manage jobs.
- Schedules log under `~/.config/scrapingbee-cli/logs/` and store metadata in `~/.config/scrapingbee-cli/schedules.json`.
- `--post-process`, `--on-complete`, `schedule`, and REPL `!command` execute local shell commands. Enable them only after setting `SCRAPINGBEE_ALLOW_EXEC=1` and running `scrapingbee auth --unsafe`.
- Set `SCRAPINGBEE_ALLOWED_COMMANDS` to a comma-separated allowlist of command prefixes in unattended environments. `logout` removes both the stored API key and unsafe verification flag.