ScrapingBee can handle several scraping requests at the same time. You don't have to wait for one request to finish before starting the next, which makes scraping a large number of pages much faster.
The number of requests you can run at once depends on your plan. You can see your limit under "Concurrency Usage" in the ScrapingBee dashboard. Exceeding your account's concurrency limit returns HTTP 429, so allow room for any other scripts using the same account.
One way to make parallel requests in Ruby is to use threads. This tutorial shows how to send multiple ScrapingBee requests concurrently with Thread.new.
Prerequisites
You need:
- Ruby installed
- A ScrapingBee API key
- The
addressablegem
You can find your API key in your ScrapingBee dashboard after signing up.
Install the required gem:
gem install addressable
Send Concurrent Requests with Threads
Create a file named concurrent_requests.rb and add:
require 'net/http'
require 'addressable/uri'
def send_request(user_url)
uri = Addressable::URI.parse('https://app.scrapingbee.com/api/v1/')
api_key = 'YOUR-API-KEY'
uri.query_values = {
'api_key' => api_key,
'url' => user_url
}
uri = URI(uri)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
puts "Sending request to #{user_url}"
request = Net::HTTP::Get.new(uri)
response = http.request(request)
puts "Completed #{user_url} with status #{response.code}"
rescue StandardError => e
puts "Request failed for #{user_url}: #{e.message}"
end
urls = [
'https://www.scrapingbee.com/blog/',
'https://www.scrapingbee.com/documentation/',
'https://www.scrapingbee.com/pricing/'
]
threads = urls.map do |url|
Thread.new { send_request(url) }
end
threads.each(&:join)
puts 'Process complete'
Replace YOUR-API-KEY with the API key from your ScrapingBee dashboard.
The scraped HTML is available in response.body, which you can save or parse inside send_request.
Run the script:
ruby concurrent_requests.rb
Example output:
Sending request to https://www.scrapingbee.com/blog/
Sending request to https://www.scrapingbee.com/documentation/
Sending request to https://www.scrapingbee.com/pricing/
Completed https://www.scrapingbee.com/blog/ with status 200
Completed https://www.scrapingbee.com/pricing/ with status 200
Completed https://www.scrapingbee.com/documentation/ with status 200
Process complete
Each URL is handled in a separate thread, allowing requests to overlap. threads.each(&:join) waits for all threads to finish before printing Process complete. The order of the messages may vary between runs.
Scrape More Pages
To scrape more pages, add their URLs to the urls list. The rest of the script stays the same.
The script starts every request in the list at once, so keep the number of URLs within your plan's concurrency limit. For a longer list, send the URLs in batches.
Replace the thread creation and join block with:
urls.each_slice(5) do |batch|
threads = batch.map do |url|
Thread.new { send_request(url) }
end
threads.each(&:join)
end
This sends up to five requests concurrently and waits for each batch to finish before starting the next. Set the batch size to your plan's concurrency limit or lower.
Go back to tutorials