In this tutorial, we will see how you can integrate ScrapingBee’s API with Python using our Software Development Kit (SDK), and use it to scrape web pages. As such, we will cover these topics:
- Install ScrapingBee’s Python SDK
- Create your first API request.
Let's get started!
1. Install the SDK
Before using an SDK, we will have to install the SDK. And we can do that using this command:
pip install scrapingbee
Or pip3 install scrapingbee if you’re using multiple versions of Python
2. Create your first API request
The general structure of an API request made in Python will always look like this:
from scrapingbee import ScrapingBeeClient # Importing SPB's client
client = ScrapingBeeClient(api_key='YOUR-API-KEY') # Initialize the client with your API Key
response = client.get("URL_TO_SCRAPE", params={}) # Scrape!
And you can do whatever you want with the response variable! For example:
print('Response HTTP Status Code: ', response.status_code)
print('Response HTTP Response Body: ', response.content)
Let’s create a tool that takes a screenshot of ScrapingBee’s blog!
from scrapingbee import ScrapingBeeClient # Importing SPB's client
client = ScrapingBeeClient(api_key='YOUR-API-KEY') # Initialize the client with your API Key, and using screenshot_full_page parameter to take a screenshot!
response = client.get("https://www.scrapingbee.com/blog", params={'screenshot': True}) # Scrape!
if response.ok: # If we get a successful request
with open("./screenshot.png", "wb") as f:
f.write(response.content) # Save the screenshot in the file "screenshot.png"
else:
print(response.content)
There are more parameters that a request can make use of, like forward_headers, premium_proxy, js_scenario, wait,etc… You can find the full list with their explanations here: Documentation - Overview.
Go back to tutorials