# ScrapingBee HTML API Data Extraction

> Use `extract_rules` for CSS/XPath-based extraction and `ai_query` or `ai_extract_rules` for semantic extraction. These are capabilities of the HTML API and add 5 credits only for AI extraction.

## Request rules

- Send rules through the HTML API endpoint: `https://app.scrapingbee.com/api/v1/`.
- `extract_rules`, `ai_extract_rules`, and `js_scenario` are JSON values; stringify them in a GET query string.
- Choose CSS/XPath extraction when the page structure is known and stable.
- Choose AI extraction when the desired field is semantic or the structure is irregular.

## Response shape

- `extract_rules` and `ai_extract_rules` return a JSON object keyed by the output-rule names; their normal response body is the extracted data rather than the raw page HTML.
- With `json_response=true`, use the HTML API JSON envelope and read the extracted response body from it.
- The source does not publish a stable response schema for top-level `ai_query`; inspect the returned body rather than assuming it has the same shape as `ai_extract_rules`.

## CSS/XPath `extract_rules`

The compact notation maps an output key to a CSS/XPath selector:

```json
{
  "title": "h1",
  "link": "a@href"
}
```

This is equivalent to:

```json
{
  "title": {"selector": "h1", "output": "text", "type": "item"},
  "link": {"selector": "a", "output": "@href", "type": "item"}
}
```

### Rule fields

| Field | Behavior |
| --- | --- |
| `selector` | CSS or XPath selector. A selector beginning with `/` is treated as XPath by default. |
| `selector_type` | `auto` (default), `css`, or `xpath`; use it to override automatic detection. |
| `output` | `text` (default), `text_relevant`, `markdown_relevant`, `html`, `@attribute`, `table_json`, or `table_array`. |
| `type` | `item` (first match, default) or `list` (all matches). |
| `clean` | `true` by default; normalizes whitespace. Set `false` to retain whitespace. |

`text_relevant` and `markdown_relevant` remove scripts, CSS, headers, and footers to keep main content; both are beta and useful for AI training or LLM context.

### Tables and nested output

- Use `table_json` for a correctly formatted table with a header row; each row becomes an object keyed by header names.
- Use `table_array` for a table without a reliable header row; each row becomes an array of cells.
- Use nested `output` under a list container to extract repeated structured items.

```json
{
  "articles": {
    "selector": ".card",
    "type": "list",
    "output": {
      "title": ".post-title",
      "href": {"selector": ".post-title", "output": "@href"},
      "description": ".post-description"
    }
  }
}
```

### Common patterns

Extract all links:

```json
{"all_links": {"selector": "a", "type": "list", "output": "@href"}}
```

Extract page text:

```json
{"text": "body"}
```

Extract email links:

```json
{
  "email_addresses": {
    "selector": "a[href^='mailto']",
    "output": "@href",
    "type": "list"
  }
}
```

The email pattern only finds `mailto:` links. To find plain-text emails, extract page text or links and apply a client-side regular expression.

## AI extraction

`ai_query` is a top-level natural-language question:

```text
ai_query=What is the product price?
```

`ai_selector` is an optional CSS selector that focuses AI on a relevant part of the page, improving accuracy and reducing processing time:

```text
ai_query=price of the product
ai_selector=#product-details
```

`ai_extract_rules` returns structured output. Short notation maps keys to descriptions:

```json
{
  "title": "title of the blog post",
  "summary": "a five-sentence summary of the blog post"
}
```

Advanced notation supports `string` (default), `list`, `number`, `boolean`, and `item`; use `enum` to constrain allowed values:

```json
{
  "price": {"description": "current price in dollars", "type": "number"},
  "in_stock": {"description": "whether the product is available", "type": "boolean"},
  "shipping": {
    "description": "shipping details",
    "type": "item",
    "output": {
      "delivery_time": "estimated delivery in days",
      "shipping_cost": "shipping cost in dollars"
    }
  },
  "size": {
    "description": "product size",
    "type": "list",
    "enum": ["XS", "S", "M", "L", "XL"]
  }
}
```

AI extraction (`ai_query` and `ai_extract_rules`) adds 5 credits to the regular HTML API cost.