Documentation

How to use ScrapingBee

Overview

Here is the list of the different parameters you can use with the ScrapingBee API.

name [type](default) Description
api_key [string] required Your API key: Get your API key learn more
url [string] required The URL of the page you want to scrape learn more
block_ads [boolean] (false) Block ads on the page you want to scrape learn more
block_resources [boolean] (true) Block images and CSS on the page you want to scrape learn more
cookies [string] ("") Pass custom cookies to the webpage you want to scrape learn more
country_code [string] ("") Premium proxy geolocation. learn more
custom_google [boolean] (false) Set to true to scrape Google learn more
device [string] ("desktop") Control the device the request will be sent from learn more
extract_rules [stringified JSON] ("") Data extraction from CSS selectors learn more
forward_headers [boolean] (false) Forward particular headers to the webpage, as well as other headers generated by ScrapingBee. learn more
forward_headers_pure [boolean] (false) Forward only particular headers to the webpage, and nothing else. learn more
json_response [bool] (false) Wrap response in JSON. learn more
js_snippet [base64 encoded string] JavaScript snippet to execute (clicking on a button, scrolling ...) learn more
js_scroll [boolean] (false) Scrolling to the end of the page before returning your results learn more
js_scroll_wait [integer] (1000) The time to wait between each scroll learn more
js_scroll_count [integer] (1) The number of scrolls you want to make learn more
premium_proxy [boolean] (false) Use premium proxies to bypass difficult to scrape websites (10-25 credits/request) learn more
render_js [boolean] (True) Render the JavaScript on the page with a headless browser (5 credits/request) learn more
return_page_source [boolean] (false) Return the original HTML before the JavaScript rendering learn more
screenshot [boolean] (false) Return a screenshot of the page you want to scrape. learn more
screenshot_full_page [boolean] (false) Return a screenshot of the full page you want to scrape. learn more
session_id [integer] ("") Route multiple API requests through the same IP address. learn more
transparent_status_code [boolean] (false) Transparently return the same HTTP code of the page requested. learn more
timeout [int] (140 000) Timeout for your requests. learn more
wait [integer] (0) Additional time in ms for JavaScript to render learn more
wait_for [string] ("") CSS selector to wait for in the DOM. learn more
window_height [int] (1080) Height, in pixel, of the viewport used to render the page you want to scrape. learn more
window_width [int] (1920) Width, in pixel, of the viewport used to render the page you want to scrape. learn more

Getting Started

ScrapingBee is meant to be the easiest scraping API available on the web.

To scrape a web page, you only need two things:

The following cURL command is an example of a simple GET API call to scrape the URL defined in the query string variable "YOUR-URL":


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL"
         
# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get('YOUR-URL')

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

The API will then respond with the raw HTML content of the target URL:

<html>
  <head>
     ...
  </head>
  <body>
     ...
  </body>
</html>          

Every URL that failed will be tried as many times as possible for 30 seconds.

Be aware of the maximum 30-second timeout when your code calls the API.

Headers and cookies returned by the target website are prefixed with Spb- (for ScraPingBee).

API Key: api_key [string] required

All requests are authenticated using your private API key.

To get access to your API key, create an account here and confirm your email address.

URL: url [string] required

This parameter is the full URL including the protocol (with http/https) of the page to extract data from.

You must encode your URL. For example, the + character is encoded to %20. Consult your programming language documentation for functions that encode URLs

Several examples of URL encoding can be seen below:


sudo apt-get install gridsite-clients
urlencode "YOUR URL"
         

import urllib.parse
encoded_url = urllib.parse.quote("YOUR URL")

encoded_url = encodeURIComponent("YOUR URL")

String encoded_url = URLEncoder.encode("YOUR URL", "UTF-8");

require 'uri'
encoded_url = URI::encode("YOUR URL")

<?php

$url_encoded = urlencode("YOUR URL");

?>

package main

import (
	"net/url"
)

func main() {
	encoded_url := url.QueryEscape("YOUR URL")
}

Javascript Rendering: render_js [boolean] (default=true)

By default, ScrapingBee fetches the URL to scrape via a headless browser that will execute the JavaScript code on the page. This is the default behavior and costs 5 credits per request.

This can be useful for scraping a Single Page Application built with frameworks such as React.js, Angular.js, JQuery or Vue.

To fetch the URL without using a headless browser, use the render_js=false parameter in the GET request.

The following is an example with a dummy Single Page Application (SPA):

If you use render_js=true (default behavior)


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL"
         
# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get('YOUR-URL')

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

The following result is returned:

<html>
  <head>
     ...
  </head>
  <body>
     <content>
     </content>
     <content>
     </content>
     <content>
     </content>
      <content>
     </content>
     <content>
     </content>
  </body>
</html>             

Using render_js=false


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&render_js=false"
         
# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'YOUR-URL',
    params={'render_js': False}
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
        'render_js': 'false',
    }
}).then(function (response) {
    // handle success
    console.log(response);
}))

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&render_js=false")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&render_js=false')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&render_js=false');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&render_js=false", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

The following result is returned:

<html>
  <head>
  ..
  </head>
  <body>
  </body>
</html>          

JavaScript Execution: js_snippet [base64 encoded string]

ScrapingBee API can execute arbitrary JavaScript code inside a Headless Chrome instance.

This can be useful for example if you need to perform a scroll where an infinite scroll triggers AJAX requests to load more elements.

It's also useful where you need to simulate clicking a button before specific information is displayed.

To execute JavaScript in a headless Chrome browser, add the parameter js_snippet with your snippet encoded in base64.

If you need help encoding your JavaScript snippet in base64 you can find more information below based on your language of choice:


echo  'YOUR JS SNIPPET' | base64
         

import base64
base64_snippet = base64.b64encode("YOUR JS SNIPPET".encode()).decode()

'use strict';

let js_snippet = 'YOUR JS SNIPPET';
let buff = new Buffer(js_snippet);
let base64_snippet = buff.toString('base64');

import org.apache.commons.codec.binary.Base64;

byte[] encodedBytes = Base64.encodeBase64("YOUR JS SNIPPET".getBytes());
String base64_snippet = new String(encodedBytes);

require "base64"

base64_snippet = Base64.encode64('YOUR JS SNIPPET')


<?php

$str = 'YOUR JS SNIPPET';
$base64_snippet = base64_encode($str);

?>

package main

func main() {
    base64_snippet := b64.StdEncoding.EncodeToString([]byte("YOUR JS SNIPPET"))
}

Snippet Examples

The following JavaScript code snippets are examples of ways to work with scrolling, buttons and other event-based actions on a Single Page Application.


// Scrolling to the end page
Take a look at our Javascript Scroll new feature

// Scrolling to the end page after 1 second (to wait for the page to load for example)
Take a look at our Javascript Scroll new feature

// Clicking on a button
document.getElementById('my-button-selector').click();

// Clicking on a button after 1 second (to wait for the page to load for example)
setTimeout(function(){document.getElementById('my-button-selector').click();}, 1000);
                    

You can find a sandbox below to help you encode any JavaScript snippet you would like to run:

js_snippet=

Wait for a fixed amount of time: wait [integer] (default=0)

Some code-heavy websites need time to fully "render". To direct ScrapingBee to wait before it returns the fully rendered HTML, use the wait parameter with a value in milliseconds between 0 and 35000.

The ScrapingBee headless browsers will then wait the duration of the time set in milliseconds before returning the page's HTML.

If you need some help setting this up, do not hesitate to contact us.

Wait for CSS selector: wait_for [string] (default='')

It's sometimes necessary to wait for a particular element to appear in the DOM before ScrapingBee returns the HTML content.

Our headless browsers will wait for the CSS selector passed in the parameter before returning the HTML.

For example, to wait for the element <div class="loading-done"></div> use wait_for=.loading-done in your request.

Premium Proxy: premium_proxy [boolean] (default=false)

For some hard-to-scrape websites, you may need to use premium proxies (or residential proxies). These proxies are rarely blocked and we recommend trying premium proxies when you receive error codes or difficult to scrape websites, like search engines, social networks, or hard to scrape E-commerce websites.

To scrape these sites, you need to add the parameter premium_proxy=true.

Each request with this parameter will count as 25 API credits with Javascript enabled. If used without JavaScript rendering it will cost 10 credits

Downloading Picture and Files

The API will transparently download images, PDF or anything that is not HTML.

We recommend downloading files with render_js=false.

There is a 2 MB limit per request.

Proxy Mode (Beta)

ScrapingBee also offers a proxy front-end to the API. This can make integration with third-party tools easier. The Proxy mode only changes the way you access ScrapingBee. The ScrapingBee API will then handle requests just like any standard request.

Request cost, return code and default parameters will be the same as a standard no-proxy request.

We recommend disabling Javascript rendering in proxy mode, which is enabled by default. The following credentials and configurations are used to access the proxy mode:

  • HTTP address: proxy.scrapingbee.com:8886
  • HTTPS address: proxy.scrapingbee.com:8887
  • Socks5 address: socks.scrapingbee.com:8888
  • Username: YOUR-API-KEY
  • Password: PARAMETERS

Important : Replace PARAMETERS with our supported API parameters. If you don't know what to use, you can begin by using render_js=False.

As an alternative, you can use URLs like the following:

{
    "url_http": "http://YOUR-API-KEY:PARAMETERS@proxy.scrapingbee.com:8886",
    "url_https": "https://YOUR-API-KEY:PARAMETERS@proxy.scrapingbee.com:8887",
    "url_socks5": "socks5://YOUR-API-KEY:PARAMETERS@socks.scrapingbee.com:8888",
}

Important: if you try to scrape Google with this mode, each requests will cost 20 credits.

To read more on how to integrate the proxy-mode with your favorite tools and language, check out the our dedicated-page.

Data extraction from CSS selectors

If you want to extract data from pages and don't want to parse the HTML on your side, you can add extraction rules to your API call.

The simplest way to use JSON rules is to use the following format

{"key_name" : "css_selector"} 

If you wish to extract the title, subtitle and intro of our blog, you will just need to use those rules.

{
    "title" : "h1",
    "subtitle" : "#subtitle",
}

And this will be the JSON response

{
    "title" : "The ScrapingBee Blog",
    "subtitle" : "We help you get better at web-scraping: detailed tutorial, case studies and writing by industry experts",
}

Important: extraction rules are JSON formatted, and in order to pass them to a GET request, you need to stringify them.

Here is how to extract the above information in your favorite language.

# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'https://www.scrapingbee.com/blog',
    params={
        'extract_rules':{"title": "h1", "subtitle": "#subtitle"},
    },
)
print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'https://www.scrapingbee.com/blog',
        'extract_rules': '{"title":"h1","subtitle":"#subtitle"}',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

require 'net/http'
require 'net/https'
require 'uri'

# Classic (GET )
def send_request
    extract_rules = URI::encode('{"title": "h1", "subtitle": "#subtitle"}')
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=https://www.scrapingbee.com/blog&extract_rules=' + extract_rules)

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
$extract_rules = urlencode('{"title": "h1", "subtitle": "#subtitle"}');

curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=https://www.scrapingbee.com/blog&extract_rules=' . $extract_rules);

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
    "net/url"
)

func sendClassic() {
	// Create client
	client := &http.Client{}


    // Stringify rules
    extract_rules := url.QueryEscape(`{"title": "h1", "subtitle": "#subtitle"}`)
	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=https://www.scrapingbee.com/blog&extract_rules=" + extract_rules, nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

We've just described the easiest and quickest way to use this feature. If you want to read more about it, check out our full guide.

Screenshot: screenshot [boolean] (default=false)

If you want to get a screenshot of the page you want to scrape, use the screenshot=True parameter.

Screenshots are only available when using render_js=True.

When using screenshot=True, block_resources will automatically be set to False so the browser will load images and CSS before taking the screenshot.


 curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&screenshot=True" > ./screenshot.png
         
#  Install the Python ScrapingBee library:
# pip install scrapingbee

client = ScrapingBeeClient(api_key='YOUR-API-KEY')

response = client.get(
    'YOUR-URL',
    params={
        'screenshot': True,
    }
)

if response.ok:
    with open("./screenshot.png", "wb") as f:
        f.write(response.content)


// npm install scrapingbee
const fs = require('fs');
const scrapingbee = require('scrapingbee');

async function screenshot(url, path) {
    var client = new scrapingbee.ScrapingBeeClient('YOUR-API-KEY');
    var response = await client.get({
        url: url,
        params: {
            screenshot: true, // Take a screenshot
            screenshot_full_page: true, // Specify that we need the full height
            window_width: 375, // Specify a mobile width in pixel
        }
    });

    fs.writeFileSync(path, response.data);
}

screenshot('YOUR-URL', './screenshot.png').catch((e) =>
    console.log('A problem occurs : ' + e.message)
);

Screenshot full page: screenshot_full_page [boolean] (default=false)

By default, the screenshot will only capture the portion of the page that is visible in the browser's viewport.

If you need a screenshot of the full page from the target website use screenshot_full_page=True

If you need to change the size of the browser's viewport before taking a screenshot you can do it using window_width and window_height parameters. (learn more).

HTML to PDF conversion

Coming soon

Session: session_id [integer]

All API requests using the same session_id will be routed through the same IP address for a duration of 5 minutes.

We advice that you use a random integer between 0 and 10,000,000 everytime you wish to generate a new session_id.

Here is how to use one:


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&session_id=123456"
         
# Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'YOUR-URL',
    params={'session_id': False}
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
        'session_id': '123456',
    }
}).then(function (response) {
    // handle success
    console.log(response);
}))

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&session_id=123456")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&session_id=123456')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&session_id=123456');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR_URL&session_id=123456", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

Timeout (in ms): timeout [integer] (default=140 000)

The maximum number of ms, comprise between 1000 and 140000, ScrapingBee will wait before returning you a results, use timeout=45000 in your API call to modify it.

Changing it could have a negative impact on your success rate.

Viewport dimension: window_width [int] (default=1920) / window_height [int] (default=1080)

If you need to change the dimension of the browser's viewport (window) when scraping the target page you can use the window_width and window_height parameters.

Only useful when using render_js=True.

Important: There will be a 0.5 second margin of error between the timeout used in your API call and the actual maximum duration of this API call.

⭐️ Advanced Usage

Geolocation: country_code [string] (default='')

In addition to premium proxies, you can also choose the proxy country from the following list of countries using the parameter country_code=COUNTRY_CODE.

To use premium proxies from Germany for example you need to set both premium_proxy=true and country_code=de parameters in your API call.


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&premium_proxy=true&country_code=de"
         
#  Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'YOUR-URL',
    params={
        'premium_proxy': 'true',
        'country_code':'de'
    },
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
        'premium_proxy': 'true',
        'country_code':'de'
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&premium_proxy=true&country_code=de")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&premium_proxy=true&country_code=de')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&premium_proxy=true&country_code=de');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&premium_proxy=true&country_code=de", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

The following is the list of the most popular supported country codes using ISO 3166-1 format ).

The whole list of supported country codes can be found here.

country_code Country Name
au Australia
br Brazil
ca Canada
fr France
de Germany
gr Greece
il Israel
in India
it Italy
mx Mexico
nl Netherlands
pe Peru
pl Poland
ru Russia
es Spain
se Sweden
ua Ukraine
us UnitedStates
ve Venezuela
gb UnitedKingdom

Header Forwarding: forward_headers [boolean] (default=false)

You might need to forward specific headers to the website that you want to scrape.

In order to forward headers, you must set forward_headers to true and then pass your custom headers.

You must then prefix the headers to forward to the website with "Spb-" (for ScraPingBee).

This prefix will be trimmed by ScrapingBee and headers will be forwarded to the target web page.

Example :

If you want to send the header Accept-Language: En-US, add the header: Spb-Accept-Language: En-US and the parameter forward_headers=true to the request sent to the ScrapingBee API.

Note :

If you are using the ScrapingBee Python library, no need to prefix headers with "Spb-"

or to use forward_headers=True.

curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true" \
  -H "Spb-Accept-Language:En-US"  
         
#  Install the Python ScrapingBee library:
# `pip install scrapingbee`

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'http://httpbin.org/headers?json',
    headers={'Accept-Language': 'En-US'}
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)
send_request()

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'http://httpbin.org/headers?json',
        'forward_headers': 'true',
    } ,
    headers:{
        "Scn-Accept-Language": "En-US",
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true")

      // Add headers
        .addHeader("Spb-Accept-Language", "En-US")

      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)
    # Add headers
    req.add_field "Spb-Accept-Language", "En-US"

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// set headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Spb-Accept-Language: En-US',

]);

// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true", nil)

// Headers
        req.Header.Add("Spb-Accept-Language", "En-US")

	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

Please note that in order to make your request look like real ones, ScrapingBee adds several headers to all API requests. Use forward_headers_pure=True to avoid this behavior.

In the above example, we are scraping httpbin.org/headers?json, a page that demonstrates and displays the headers it received.

The following is the response from the above code. Note the Accept-Language header in the response.

{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7",
    "Host": "httpbin.org",
    "Accept-Language": "En-US" # <-- Your header
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36",
  }
}                   

Pure Header Forwarding: forward_headers_pure [boolean] (default=false)

If you want to forward specific headers to the website that you want to scrape, and don't need ScrapingBee to add any headers to your request, you should use forward_headers_pure=True.

You must then prefix the headers to forward to the website with "Spb-" (for ScraPingBee).

This prefix will be trimmed by ScrapingBee and headers will be forwarded to the target web page.

Example :

If you want to send the header Accept-Language: En-US, add the header: Spb-Accept-Language: En-US and the parameter forward_headers=true to the request sent to the ScrapingBee API.

Note :

If you are using the ScrapingBee Python library, no need to prefix headers with "Spb-"

or to use forward_headers=True.

curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true" \
  -H "Spb-Accept-Language:En-US"  
         
#  Install the Python ScrapingBee library:
# `pip install scrapingbee`

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'http://httpbin.org/headers?json',
    headers={'Accept-Language': 'En-US'}
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)
send_request()

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'http://httpbin.org/headers?json',
        'forward_headers': 'true',
    } ,
    headers:{
        "Scn-Accept-Language": "En-US",
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true")

      // Add headers
        .addHeader("Spb-Accept-Language", "En-US")

      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)
    # Add headers
    req.add_field "Spb-Accept-Language", "En-US"

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// set headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Spb-Accept-Language: En-US',

]);

// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fheaders%3Fjson&forward_headers=true", nil)

// Headers
        req.Header.Add("Spb-Accept-Language", "En-US")

	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

In the above example, we are scraping httpbin.org/headers?json, a page that demonstrates and displays the headers it received.

The following is the response from the above code. Note the Accept-Language header in the response.

{
  "headers": {
    "Accept-Encoding": "gzip, deflate", # Technical header sent with all requests
    "Host": "httpbin.org", # Technical header sent with all requests
    "Accept-Language": "En-US" # <-- Your header, and only your headers
  }
}                   

This parameter is only useful when using render_js=False.

Custom Cookies: cookies [string] (default='')

You can pass custom cookies to the target webpages

To do this you must pass a cookie string in the cookies parameter.

ScrapingBee currently only handles the name and value of custom cookies. If you want to set multiple cookies, separate them with ;.

Example with cookie_name_1=cookie_value1;cookie_name_2=cookie_value_2:


 curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2"
         
#  Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'http://httpbin.org/cookies?json',
     cookies= {"cookie_name_1":"cookie_value1","cookie_name_2":"cookie_value_2"},
)
print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content))

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'http://httpbin.org/cookies?json',
        'render_js': 'false',
        'cookies': 'cookie_name_1=cookie_value_1',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=http%3A%2F%2Fhttpbin.org%2Fcookies%3Fjson&cookies=cookie_name_1%3Dcookie_value1%3Bcookie_name_2%3Dcookie_value_2", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

In the above example, ScrapingBee is set to scrape httpbin.org/cookies?json, a page that displays the cookies it received.

The following will be returned by the above request.

{
  "cookies": {
    "cookie_name_1": "cookie_value1",
    "cookie_name_2": "cookie_value_2"
  }
}                   

Device: device [string] (default=desktop)

Choose the kind of device that will send the request to the server. Only two choices are available, desktop, the default, and mobile.

Google: custom_google [boolean] (default=false)

If you need to scrape webpage on Google main domain (google.com) or subdomains (news.google.com, scholar.google.com, etc ...) you'll need to use custom_google=True.

Important: each request using custom_google=True will cost you 20 credits.

Page Source: return_page_source [boolean] (default=false)

To have HTML returned by the server and unaltered by the browser (before the JavaScript execution), use return_page_source=true

This parameter is unnecessary if JavaScript rendering is disabled.

Blocking Ads: block_ads [boolean] (default=false)

By default, ScrapingBee does not block ads. To avoid scraping them (e.g.,to speed up your request), use block_ads=true

This parameter is unnecessary if JavaScript rendering is disabled.

Blocking Images and CSS: block_resources [boolean] (default=true)

By default, and to speed up requests, ScrapingBee blocks all images and CSS in the scraped page, but to scrape them, use block_resources=false

This parameter is unnecessary if JavaScript rendering is disabled.

Transparent HTTP status code: transparent_status_code [boolean] (default=false)

By default, ScrapingBee will return an HTTP 500 whenever the requested URL returns something other than: a 200-299 or a 404.

To get same body and status code as the requested URL in any circumstances, use the transparent_status_code=true parameter.

When this parameter is set to true:

  • Every requests you make will be considered successful and will cost some amount of credits.
  • ScrapingBee will not retry the request multiple time if the request returns a 500 error.

POST / PUT

To send a POST / PUT request, send the POST / PUT request to the main endpoint with your api_key and url parameter.

Data will be forwarded transparently to the target web page.

Headers and cookies will also be returned.

Below is an example using httpbin.org, a service mirroring HTTP requests sent to the site.

curl -X "POST" "https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY" \
     -H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
     --data-urlencode "KEY_1=VALUE_1"
         
# Install the Python Requests library:
# `pip install requests`

import requests


def send_request():
    # Post
    # POST https://app.scrapingbee.com/api/v1

    try:
        response = requests.post(
            url="https://app.scrapingbee.com/api/v1",
            params={
                "url": "https://httpbin.org/anything",
                "api_key": "YOUR-API-KEY",
            },
            headers={
                "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
            },
            data={
                "KEY_1": "VALUE_1",
            },
        )
        print('Response HTTP Status Code: {status_code}'.format(
            status_code=response.status_code))
        print('Response HTTP Response Body: {content}'.format(
            content=response.content))
    except requests.exceptions.RequestException:
        print('HTTP Request failed')
// request Post
(function(callback) {
    'use strict';

    const httpTransport = require('https');
    const responseEncoding = 'utf8';
    const httpOptions = {
        hostname: 'app.scrapingbee.com',
        port: '443',
        path: '/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY',
        method: 'POST',
        headers: {"Content-Type":"application/x-www-form-urlencoded; charset=utf-8"}
    };
    httpOptions.headers['User-Agent'] = 'node ' + process.version;


    const request = httpTransport.request(httpOptions, (res) => {
        let responseBufs = [];
        let responseStr = '';

        res.on('data', (chunk) => {
            if (Buffer.isBuffer(chunk)) {
                responseBufs.push(chunk);
            }
            else {
                responseStr = responseStr + chunk;
            }
        }).on('end', () => {
            responseStr = responseBufs.length > 0 ?
                Buffer.concat(responseBufs).toString(responseEncoding) : responseStr;

            callback(null, res.statusCode, res.headers, responseStr);
        });

    })
    .setTimeout(0)
    .on('error', (error) => {
        callback(error);
    });
    request.write("KEY_1=VALUE_1")
    request.end();


})((error, statusCode, headers, body) => {
    console.log('ERROR:', error);
    console.log('STATUS:', statusCode);
    console.log('HEADERS:', JSON.stringify(headers));
    console.log('BODY:', body);
});

import java.io.IOException;
import org.apache.http.client.fluent.*;
import org.apache.http.entity.ContentType;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Post (POST )

    try {

      // Create request
      Content content = Request.Post("https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY")

      // Add headers
      .addHeader("Content-Type", "application/x-www-form-urlencoded; charset=utf-8")

      // Add body
      .bodyForm(Form.form()
      .add("KEY_1", "VALUE_1")
      .build())

      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}
require 'net/http'
require 'net/https'

# Post (POST )
def send_request
  uri = URI('https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY')

  # Create client
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_PEER
  data = {
    "KEY_1" => "VALUE_1",
  }
  body = URI.encode_www_form(data)

  # Create Request
  req =  Net::HTTP::Post.new(uri)
  # Add headers
  req.add_field "Content-Type", "application/x-www-form-urlencoded; charset=utf-8"
  # Set body
  req.body = body

  # Fetch Request
  res = http.request(req)
  puts "Response HTTP Status Code: #{res.code}"
  puts "Response HTTP Response Body: #{res.body}"
rescue StandardError => e
  puts "HTTP Request failed (#{e.message})"
end
<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

// set headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
  'Content-Type: application/x-www-form-urlencoded; charset=utf-8',
]);

// form body
$body = [
  'KEY_1' => 'VALUE_1',
];
$body = http_build_query($body);

// set body
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $body);

// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"bytes"
)

func sendPost() {
	// Post (POST https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY)

	params := url.Values{}
	params.Set("KEY_1", "VALUE_1")
	body := bytes.NewBufferString(params.Encode())

	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("POST", "https://app.scrapingbee.com/api/v1?url=https:%2F%2Fhttpbin.org%2Fanything&api_key=YOUR-API-KEY", body)

	// Headers
	req.Header.Add("Content-Type", "application/x-www-form-urlencoded; charset=utf-8")

	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
	  fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

The following is the received response:


{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "KEY_1": "VALUE_1"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "13",
    "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
    "Host": "httpbin.org",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0",
    "X-Amzn-Trace-Id": "Root=1-5eb416d8-65c76a78a809bb07614d7cfe"
  },
  "json": null,
  "method": "POST",
  "origin": "23.250.56.153",
  "url": "https://httpbin.org/anything"
}
                    

Scrolling

Scrolling to the end of the screen before receiving results can be enabled by using the js_scroll=true parameter in your GET request.

To use this parameter, JavaScript rendering must be enabled.

Sometimes, especially when dealing with infinite scroll, it is necessary to do multiple scrolls before receiving results.

To perform multiple scrolls, use these two parameters: js_scroll_count is the number of scrolls to perform. The default value is 1 . And js_scroll_wait is the number of milliseconds to wait between each scroll. The default value for this parameter is 1000.

The following examples demonstrate scenarios that will show you what happens when you use those three parameters.

    #js_scroll=true
    - waiting 1000 milliseconds
    - scrolling to the end of the screen
    - waiting 1000 milliseconds
    - returning the result

    #js_scroll=true, js_scroll_wait=2000
    - waiting 2000 milliseconds
    - scrolling to the end of the screen
    - waiting 2000 milliseconds
    - returning the result

    #js_scroll=true, js_scroll_wait=2000, js_scroll_count=2
    - waiting 2000 milliseconds
    - scrolling to the end of the screen
    - waiting 2000 milliseconds
    - scrolling to the end of the screen
    - waiting 2000 milliseconds
    - returning the result

The following list provides examples when using scroll parameters with the ScrapingBee API:


curl "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&js_scroll=true&js_scroll_wait=2000&js_scroll_count=2"
         
#  Install the Python ScrapingBee library:
# pip install scrapingbee

from scrapingbee import ScrapingBeeClient

client = ScrapingBeeClient(api_key='YOUR-API-KEY')
response = client.get(
    'YOUR-URL',
    params={
        'js_scroll_wait': '2000',
        'js_scroll_count': '2',
    },
     cookies= {"cookie_name_1":"cookie_value1","cookie_name_2":"cookie_value_2"},
)

print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content))

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1', {
    params: {
        'api_key': 'YOUR-API-KEY',
        'url': 'YOUR-URL',
        'js_scroll': 'true',
        'js_scroll_wait': '2000',
        'js_scroll_count': '2',
    }
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&js_scroll=true&js_scroll_wait=2000&js_scroll_count=2")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&js_scroll=true&js_scroll_wait=2000&js_scroll_count=2')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&js_scroll=true&js_scroll_wait=2000&js_scroll_count=2');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/?api_key=YOUR-API-KEY&url=YOUR-URL&js_scroll=true&js_scroll_wait=2000&js_scroll_count=2", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

JSON Response: json_response [boolean] (default=false)

If you are planning to integrate ScrapingBee with third-party tools that only accept JSON response, or want to intercept the response of some XHR / Ajax requests, you can send your API call with json_response=True.

The following is the received response when using this parameter:


{
  # Headers sent by the server
  "headers": {
    "Date": "Fri, 16 Apr 2021 15:03:54 GMT",
    ...
    "Access-Control-Allow-Credentials": "true"
  },
  # Credit cost of your request
  "cost": 1,
  # Initial status code of the server
  "initial-status-code": 200,
  # Resolved URL (following redirection)
  "resolved-url": "https://httpbin.org/",
  # Type of the response "html" or "json" or "b64_bytes" for file, image, pdf,...
  "type": "html",
  # Content of the answer. Content will be base 64 encoded if is a file, image, pdf,...
  "body": "<html>... </body>"
  # Cookies sent back by the server
  'cookies': [
    {
        "name": "cookie_name",
        "value": "cookie_value",
        "domain": "test.com",
        ...
    },
    ...
  ],
  # XHR / Ajax requests sent by the browser
  "xhr": [
    {
      # URL
      "url": "https://",
      # status code of the server
      "status_code": 200,
      # Method of the request
      "method": "POST",
      # Headers of the XHR / Ajax request
      "headers": {
        "pragma": "no-cache",
        ...
      },
      # Response of the XHR / Ajax request
      "body": "2d,x"
    },
    ...
  ]
}

If the requested content is json, then the answers will look like this:


{
  # Headers sent by the server
  "headers": {
    "Date": "Fri, 16 Apr 2021 15:13:02 GMT",
    ...
    "Access-Control-Allow-Credentials": "true"
  },
  # Credit cost of your request
  "cost": 1,
  # Initial status code of the server
  "initial-status-code": 200,
  # Resolved URL (following redirection)
  "resolved-url": "https://httpbin.org/anything?json",
  # Type of the response "html" of "json"
  "type": "json",
  # Content of the answer
  "body": {
    "args": {
        ....
    }
  }
  # XHR / Ajax requests sent by the browser
  "xhr": [
    ...
  ]
}

Credit cost for your requests

Each ScrapingBee plan provides a certain amount of API credits per month.

It costs 1 to 25 credits to make requests to the ScrapingBee API. The credit cost depends on the parameters used with your API calls.

Here is a breakdown of ScrapingBee API credit costs:

Feature used API credit cost
Rotating Proxy without JavaScript rendering 1
Rotating Proxy with JavaScript rendering (default) 5
Premium Proxy without JavaScript rendering 10
Premium Proxy with JavaScript rendering 25

Usage endpoint

To programmatically monitor credit consumption and concurrency usage use /usage endpoint.

Calls to this endpoint will not increase concurrency, but you can only call it 6 times per minute.

Please note that the results are available in real-time.


curl "https://app.scrapingbee.com/api/v1/usage?api_key=YOUR-API-KEY"
         

#  Install the Python Requests library:
# `pip install requests`
import requests

def send_request():
    response = requests.get(
        url="https://app.scrapingbee.com/api/v1/usage",
        params={
            "api_key": "YOUR-API-KEY",
        },

    )
    print('Response HTTP Status Code: ', response.status_code)
    print('Response HTTP Response Body: ', response.content)
send_request()

// request Axios
const axios = require('axios');

axios.get('https://app.scrapingbee.com/api/v1/usage', {
    params: {'api_key': 'YOUR-API-KEY'}
}).then(function (response) {
    // handle success
    console.log(response);
})

import java.io.IOException;
import org.apache.http.client.fluent.*;

public class SendRequest
{
  public static void main(String[] args) {
    sendRequest();
  }

  private static void sendRequest() {

    // Classic (GET )

    try {

      // Create request
      Content content = Request.Get("https://app.scrapingbee.com/api/v1/usage?api_key=YOUR-API-KEY")



      // Fetch request and return content
      .execute().returnContent();

      // Print content
      System.out.println(content);
    }
    catch (IOException e) { System.out.println(e); }
  }
}

require 'net/http'
require 'net/https'

# Classic (GET )
def send_request
    uri = URI('https://app.scrapingbee.com/api/v1/usage?api_key=YOUR-API-KEY')

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)
    puts "Response HTTP Status Code: #{ res.code }"
    puts "Response HTTP Response Body: #{ res.body }"
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

send_request()

<?php

// get cURL resource
$ch = curl_init();

// set url
curl_setopt($ch, CURLOPT_URL, 'https://app.scrapingbee.com/api/v1/usage?api_key=YOUR-API-KEY');

// set method
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');

// return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);



// send the request and save response to $response
$response = curl_exec($ch);

// stop if fails
if (!$response) {
  die('Error: "' . curl_error($ch) . '" - Code: ' . curl_errno($ch));
}

echo 'HTTP Status Code: ' . curl_getinfo($ch, CURLINFO_HTTP_CODE) . PHP_EOL;
echo 'Response Body: ' . $response . PHP_EOL;

// close curl resource to free up system resources
curl_close($ch);

?>

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
)

func sendClassic() {
	// Create client
	client := &http.Client{}

	// Create request
	req, err := http.NewRequest("GET", "https://app.scrapingbee.com/api/v1/usage?api_key=YOUR-API-KEY", nil)


	parseFormErr := req.ParseForm()
	if parseFormErr != nil {
		fmt.Println(parseFormErr)
	}

	// Fetch Request
	resp, err := client.Do(req)

	if err != nil {
		fmt.Println("Failure : ", err)
	}

	// Read Response Body
	respBody, _ := ioutil.ReadAll(resp.Body)

	// Display Results
	fmt.Println("response Status : ", resp.Status)
	fmt.Println("response Headers : ", resp.Header)
	fmt.Println("response Body : ", string(respBody))
}

func main() {
    sendClassic()
}

Results.

{
    "max_api_credit": 20000000,
    "used_api_credit": 3704332,
    "max_concurrency": 200,
    "current_concurrency": 1
}                   

Response Status Code

The following list of HTTP codes are returned by ScrapingBee.

Code Billed? Meaning Solution
200 Yes Successful Call
400 No Bad request Incorrect parameters or parameters type. See the message in the response body.
401 No No more credit available Please upgrade your plan or contact sales.
404 Yes Requested URL not found Provide a valid URL.
413 No File too large Request a smaller file.
429 No Too many concurrent requests. Please upgrade your plan or contact sales.
500 No Misc error Please retry, and see the message in the response body.

Response Headers

The following is the list of additional HTTP headers returned by ScrapingBee.

Name Meaning
Spb-cost Request cost in credits.
Spb-initial-status-code The initial status code returned by the scraped page.
Useful when the page redirects.
Spb-resolved-url The resolved URL of the scraped page.
Useful when the page redirects.