Getting started with ScrapingBee's NodeJS SDK

In this tutorial, we will see how you can integrate ScrapingBee’s API with NodeJS using our Software Development Kit (SDK), and use it to scrape web pages. As such, we will cover these topics:

• Install ScrapingBee’s NodeJS SDK
• Create your first API request. Let’s get started!

1. Install the SDK

Before using an SDK, we will have to install the SDK. And we can do that using this command: npm install scrapingbee.

Or npm install -g scrapingbee if you want to install ScrapingBee for the whole system.

2. Create your first API request

The general structure of an API request made in NodeJS will always look like this:

const scrapingbee = require('scrapingbee'); // Importing SPB's SDK

async function get(url) {
  var client = new scrapingbee.ScrapingBeeClient('YOUR-API-KEY'); // Initialize the client with your API Key
  var response = await client.get({
    url: url,
    params: {
    },
  })
}
my_request = get('YOUR-URL');

And you can do whatever you want with the response variable! For example:

my_request.then(function (response) {
    console.log("Status Code:", response.status) // Print request status code
    var decoder = new TextDecoder();
    var text = decoder.decode(response.data); // Decode request content
    console.log("Response content:", text); // Print the content
}).catch((e) => console.log('A problem occurs : ' + e));

Let’s create a tool that saves the HTML code of ScrapingBee’s blog!

const scrapingbee = require('scrapingbee'); // Import ScrapingBee's SDK
const fs = require('fs');

async function save_html_content(url, path) {
  var client = new scrapingbee.ScrapingBeeClient('YOUR-API-KEY'); // New ScrapingBee client
  var response = await client.get({
    url: url,
    params: { // No parameters to use
    },
  })
  fs.writeFileSync(path, response.data); // Save the response content to a file
}

save_html_content('https://scrapingbee.com/blog', './blog.html').catch((e) =>
    console.log('A problem occurs : ' + e.message)
);

There are some parameters that a request can make use of, like forward_headers, screenshots, premium_proxy, js_scenario, wait, extract rules, etc… You can find the full list of parameters with their explanations here: Documentation - Overview