Make concurrent requests in Ruby

ScrapingBee can handle several scraping requests at the same time. You don't have to wait for one request to finish before starting the next, which makes scraping a large number of pages much faster.

The number of requests you can run at once depends on your plan. You can see your limit under "Concurrency Usage" in the ScrapingBee dashboard. Exceeding your account's concurrency limit returns HTTP 429, so allow room for any other scripts using the same account.

One way to make parallel requests in Ruby is to use threads. This tutorial shows how to send multiple ScrapingBee requests concurrently with Thread.new.

Prerequisites

You need:

  • Ruby installed
  • A ScrapingBee API key
  • The addressable gem

You can find your API key in your ScrapingBee dashboard after signing up.

Install the required gem:

gem install addressable

Send Concurrent Requests with Threads

Create a file named concurrent_requests.rb and add:

require 'net/http'
require 'addressable/uri'

def send_request(user_url)
  uri = Addressable::URI.parse('https://app.scrapingbee.com/api/v1/')
  api_key = 'YOUR-API-KEY'
  uri.query_values = {
    'api_key' => api_key,
    'url' => user_url
  }
  uri = URI(uri)

  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_PEER

  puts "Sending request to #{user_url}"
  request = Net::HTTP::Get.new(uri)
  response = http.request(request)

  puts "Completed #{user_url} with status #{response.code}"
rescue StandardError => e
  puts "Request failed for #{user_url}: #{e.message}"
end

urls = [
  'https://www.scrapingbee.com/blog/',
  'https://www.scrapingbee.com/documentation/',
  'https://www.scrapingbee.com/pricing/'
]

threads = urls.map do |url|
  Thread.new { send_request(url) }
end

threads.each(&:join)

puts 'Process complete'

Replace YOUR-API-KEY with the API key from your ScrapingBee dashboard.

The scraped HTML is available in response.body, which you can save or parse inside send_request.

Run the script:

ruby concurrent_requests.rb

Example output:

Sending request to https://www.scrapingbee.com/blog/
Sending request to https://www.scrapingbee.com/documentation/
Sending request to https://www.scrapingbee.com/pricing/
Completed https://www.scrapingbee.com/blog/ with status 200
Completed https://www.scrapingbee.com/pricing/ with status 200
Completed https://www.scrapingbee.com/documentation/ with status 200
Process complete

Each URL is handled in a separate thread, allowing requests to overlap. threads.each(&:join) waits for all threads to finish before printing Process complete. The order of the messages may vary between runs.

Scrape More Pages

To scrape more pages, add their URLs to the urls list. The rest of the script stays the same.

The script starts every request in the list at once, so keep the number of URLs within your plan's concurrency limit. For a longer list, send the URLs in batches.

Replace the thread creation and join block with:

urls.each_slice(5) do |batch|
  threads = batch.map do |url|
    Thread.new { send_request(url) }
  end

  threads.each(&:join)
end

This sends up to five requests concurrently and waits for each batch to finish before starting the next. Set the batch size to your plan's concurrency limit or lower.

Go back to tutorials