Web Scraping with Ruby: Nokogiri, HTTParty, and the Rest of the 2026 Toolkit

The short answer for 2026: HTTParty or Faraday to fetch, Nokogiri to parse, Ferrum when the page needs JavaScript, Mechanize when you need a stateful session. That's the whole stack for most jobs. Everything else in the Ruby scraping ecosystem is either a wrapper around those four or a project that stopped shipping years ago — and a lot of published Ruby scraping advice still recommends the latter.

This guide covers the fetch layer (including the timeout and SSL settings that actually bite in production), parsing with Nokogiri (including why its install errors are mostly historical now), sessions, JavaScript rendering, PDFs, and the point where the gem stops being the problem.

Key Takeaways

• Nokogiri::HTML is the HTML4/libxml2 parser. Use Nokogiri.HTML5(...) instead — it's the WHATWG-spec parser, so it builds the same DOM your browser does on modern markup.
• Most Nokogiri "failed to build native extension" advice is obsolete. Precompiled gems have shipped since 1.11 and musl/Alpine builds since 1.16. If you're still compiling, something is forcing a source build — usually your lockfile or force_ruby_platform.
• HTTParty's timeout: is not a deadline. It sets per-socket open and read timeouts; a server that dribbles bytes can keep a request alive far past the number you set.
• Don't reach for verify: false. An SSL failure while scraping is nearly always an outdated CA bundle, a missing intermediate certificate, or a corporate MITM proxy — all of which have fixes that keep verification on.
• Ferrum talks to Chrome over CDP with no WebDriver in the loop — the fastest way to render JS in Ruby, and actively developed (0.17.2, March 2026).
• Watir is effectively frozen: last gem release 7.3.0 in August 2023, last commit May 2024. Don't start new work on it.
• Ruby 3.2 hit end of life on 2026-04-01 and 3.3 is security-fixes-only. Target 3.4 or newer — selenium-webdriver 4.46 already requires Ruby >= 3.3.

Which Ruby scraping gem should you use?

Decision rule: start with HTTParty or Faraday plus Nokogiri. Add Mechanize only when you find yourself hand-rolling a cookie jar. Reach for Ferrum only after you've confirmed the data isn't in the initial HTML — check by disabling JavaScript in DevTools and reloading, or by looking for the underlying JSON endpoint in the Network tab. A browser is roughly two orders of magnitude more expensive per page than an HTTP request.

Is Ruby still a good language for web scraping?

For fetching and parsing, yes — Nokogiri is a mature libxml2/gumbo binding, and Ruby's threading model handles network-bound work well (MRI releases the GVL during blocking I/O, so a thread pool gives you real concurrency on requests).

Where Ruby is genuinely behind Python: there's no equivalent of Scrapy's full crawling framework with the same ecosystem depth, and browser automation has fewer options — no first-party Playwright support, for example. If you're standing up a large distributed crawler from scratch, that's a real consideration. If you're adding scraping to an existing Rails app, the gems below are perfectly adequate and you keep your deployment, job queue, and models.

The HTTP layer: Net::HTTP, HTTParty, Faraday

Ruby's stdlib Net::HTTP needs no gem and is worth knowing because everything else sits on top of it:

require "net/http"

UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " \
     "(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"

uri = URI("https://example.com/products")

response = Net::HTTP.start(uri.host, uri.port,
                           use_ssl: true,
                           open_timeout: 5,
                           read_timeout: 15) do |http|
  http.request(Net::HTTP::Get.new(uri, "User-Agent" => UA))
end

raise "HTTP #{response.code}" unless response.is_a?(Net::HTTPSuccess)
html = response.body

Note what Net::HTTP does not do for you: it won't follow redirects, and it won't decompress a gzip response unless you leave the Accept-Encoding header alone (set it manually and you own the decompression). Those two omissions are why most people reach for a gem.

URI.open from open-uri shows up in every beginner tutorial. Skip it for scraping: it doesn't let you set a timeout cleanly, and passing user-controlled input to it is a known command-injection vector.

HTTParty is the shortest path from URL to body:

require "httparty"

response = HTTParty.get(
  "https://example.com/products",
  headers: { "User-Agent" => UA, "Accept-Language" => "en-US,en;q=0.9" },
  timeout: 15
)

raise "HTTP #{response.code}" unless response.success?
html = response.body

It follows redirects by default (cap it with limit:), decompresses gzip, and parses JSON automatically when the response content type says so — response.parsed_response gives you a Hash without a JSON.parse call.

Faraday costs a few more lines and gives you a middleware stack, which is what you want once retries stop being optional:

require "faraday"
require "faraday/retry"   # Faraday 2.x moved retry OUT of core — separate gem

conn = Faraday.new(
  headers: { "User-Agent" => UA },
  request: { timeout: 15, open_timeout: 5 }
) do |f|
  f.request :retry,
            max: 3,
            interval: 0.5,
            backoff_factor: 2,                            # 0.5s, 1s, 2s
            retry_statuses: [429, 500, 502, 503, 504],
            exceptions: [Faraday::TimeoutError, Faraday::ConnectionFailed]
  f.response :raise_error                                 # 4xx/5xx become exceptions
end

html = conn.get("https://example.com/products").body

The Faraday 2 gotcha: f.request :retry raises Faraday::Error unless you add gem "faraday-retry" and require it. Retry middleware lived in Faraday core in 1.x and was extracted in 2.0, which is why so many older snippets fail on a fresh install.

Handling timeouts in HTTParty

A scraper without timeouts eventually hangs forever on one slow host and takes the worker with it. HTTParty gives you three levels of control.

Per request — one number covering both connect and read:

HTTParty.get("https://example.com/quick", timeout: 5)
HTTParty.get("https://example.com/heavy", timeout: 30)

Per class, with default_timeout, so every call from that client inherits it:

require "httparty"

class ProductScraper
  include HTTParty
  base_uri "https://example.com"
  default_timeout 15
end

Split connect and read, which is what you actually want for scraping — connecting should be fast, reading a big page can legitimately be slow:

class ProductScraper
  include HTTParty

  default_options.update(
    open_timeout:  5,    # TCP + TLS handshake
    read_timeout: 20,    # waiting for response data
    write_timeout: 10    # sending the request body (Ruby 2.6+)
  )
end

Each maps to a distinct exception, and catching them separately tells you whether the host is unreachable or just slow:

begin
  response = ProductScraper.get("/products")
rescue Net::OpenTimeout
  # Couldn't connect — host down, DNS failure, or your IP is being dropped
  retry_later
rescue Net::ReadTimeout
  # Connected fine, response never finished — raise read_timeout or narrow the request
  retry_with_longer_timeout
rescue Net::WriteTimeout
  # Rare outside large POST bodies
  raise
end

The gotcha worth internalising: read_timeout is not a deadline for the whole request. It's the maximum wait for the next chunk of data. A server that sends a byte every ten seconds resets the clock each time, so a read_timeout: 20 request can run for minutes. If you need a hard ceiling — and in a Sidekiq worker with a job timeout, you do — put a total-time budget outside the client rather than trusting the socket timeout alone:

require "timeout"

# Net::HTTP's read_timeout bounds each read, not the whole exchange.
# Timeout.timeout bounds the exchange, at the cost of raising from an
# arbitrary point in the stack — so close the connection and don't reuse it.
Timeout.timeout(45) do
  ProductScraper.get("/enormous-report")
end

Sensible starting points: 5s connect, 15–30s read for HTML pages, and a total budget around double the read timeout. Retry on Net::OpenTimeout and Net::ReadTimeout with exponential backoff — but cap the retries, because a host that times out three times in a row is usually rate-limiting you rather than having a bad day.

SSL verification: the bypass question, answered honestly

Sooner or later a scrape dies with OpenSSL::SSL::SSLError: certificate verify failed, and the first search result tells you to turn verification off. Every Ruby HTTP client makes that a one-liner:

# Don't ship any of these.
HTTParty.get(url, verify: false)
Faraday.new(url: url, ssl: { verify: false })

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE

Here's why that's a worse trade in a scraper than in most code. Disabling verification means you no longer know who you're talking to — anyone positioned on the network path can present their own certificate, serve you whatever HTML they like, and you'll parse it and write it to your database as fact. A scraper's entire output is data you then act on. Silently accepting substituted data is the specific failure you can least afford, and unlike a browser there's no padlock icon for anyone to notice.

certificate verify failed almost always has one of four causes, and all four have a fix that keeps verification on:

1. Your CA bundle is stale or missing. Common in slim Docker images and on old macOS Rubies. Install the system trust store and point OpenSSL at it:

RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates

# Check what OpenSSL is actually trusting
puts OpenSSL::X509::DEFAULT_CERT_FILE   # e.g. /usr/lib/ssl/cert.pem
puts File.exist?(OpenSSL::X509::DEFAULT_CERT_FILE)

The SSL_CERT_FILE and SSL_CERT_DIR environment variables override those defaults process-wide, which is the least invasive fix when you can't change the image.

2. The server doesn't send its intermediate certificate. A misconfiguration on their side, and browsers paper over it by fetching the missing intermediate themselves. Ruby won't. Verify with openssl s_client -connect host:443 -showcerts — if the chain is incomplete, download the intermediate and add it to a bundle you control:

store = OpenSSL::X509::Store.new
store.set_default_paths                                  # keep the system roots
store.add_file("config/certs/missing-intermediate.pem")  # plus the one they omit

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.cert_store = store

3. A corporate proxy is intercepting TLS. Zscaler, Netskope and friends re-sign every connection with their own root. Add that root to your trust store — it's the same fix as above, and it's the correct one, not a workaround.

4. The certificate really is self-signed or expired. Trust that one certificate explicitly rather than turning verification off for the whole process:

HTTParty.get(url, ssl_ca_file: "config/certs/that-one-host.pem")

If you genuinely must skip verification — a staging box you own, a one-off script — scope it to the single host and never to a class-wide default:

# Bad: every request this class ever makes is now unverified, including the
# ones someone adds in six months against a different host.
class Scraper
  include HTTParty
  default_options.update(verify: false)
end

# Less bad: opt in per call, for one known host, with the reason written down.
INTERNAL = "https://staging.internal.example.com"
HTTParty.get("#{INTERNAL}/report", verify: false)  # self-signed, internal network only

Custom SSL options in HTTParty

Beyond turning verification on and off, HTTParty exposes the OpenSSL knobs you need for hosts with unusual requirements. These are the ones worth knowing:

require "httparty"

class SecureScraper
  include HTTParty
  base_uri "https://partner-api.example.com"

  default_options.update(
    verify: true,                                    # keep this on
    ssl_ca_file: ENV.fetch("PARTNER_CA_BUNDLE"),     # a single PEM bundle
    ssl_version: :TLSv1_2                            # minimum protocol version
  )
end

Mutual TLS — where the server demands a certificate from you — shows up on partner and bank APIs:

response = HTTParty.get(
  "https://partner-api.example.com/feed",
  pem: File.read(ENV.fetch("CLIENT_CERT_PEM")),   # cert + key concatenated
  pem_password: ENV.fetch("CLIENT_CERT_PASSWORD"),
  verify: true
)

Two things people get wrong here. pem: takes the file's contents, not its path — passing a path gives you a confusing OpenSSL parse error. And certificates belong in a secrets manager or a mounted volume, never in the repository; the ENV.fetch above is deliberate, since ENV[] returning nil fails much later and much less clearly.

Pinning ssl_version is occasionally necessary for legacy hosts, but pin a floor, not a ceiling. Forcing :TLSv1_3 breaks against any server that doesn't support it yet, which is a self-inflicted outage.

Parsing with Nokogiri

Nokogiri is where every Ruby scraping stack converges: it wraps libxml2 for XML and HTML4, and gumbo for HTML5, and gives you both CSS selectors and XPath over the result.

The single most valuable correction to make to old Ruby scraping code:

require "nokogiri"

doc = Nokogiri::HTML(html)    # HTML4 parser (libxml2) — HTML is an alias for HTML4
doc = Nokogiri.HTML5(html)    # WHATWG HTML5 parser (gumbo) — use this

Nokogiri::HTML has been an alias for Nokogiri::HTML4 since 1.12. The HTML4 path uses libxml2's parser, which predates the HTML5 spec and recovers from broken markup differently than a browser does — nested tables, <template>, and unclosed tags are where you notice. Nokogiri.HTML5 implements the WHATWG parsing algorithm, so the tree you query matches the DOM you inspected in DevTools.

Extraction, with the failure modes handled:

doc = Nokogiri.HTML5(html)

products = doc.css("li.product").map do |node|
  {
    name:  node.at_css("h2")&.text&.strip,
    price: node.at_css(".price")&.text.to_s[/[\d,.]+/]&.delete(",")&.to_f,
    url:   URI.join("https://example.com", node.at_css("a")&.attr("href").to_s).to_s,
    sku:   node["data-sku"]
  }
end.reject { |p| p[:name].nil? }

Three things worth copying:

• at_css returns one node or nil; css returns a NodeSet. Calling .text on an empty NodeSet gives you "", not an error — which is how silent data loss happens. Prefer at_css + &. so a layout change gives you nil you can filter on.
• node["href"] reads an attribute and returns a String directly. The .attribute("href").value form you see in older posts blows up with NoMethodError on missing attributes.
• URI.join for relative links. Never string-concatenate a base URL with an href.

CSS or XPath?

CSS selectors cover most cases and read better. XPath earns its keep for the axes CSS can't express — walking upward to a parent, or selecting relative to a sibling's text:

# "The <td> following the <th> whose text is 'SKU'" — one XPath, no CSS equivalent
sku = doc.at_xpath("//th[normalize-space(text())='SKU']/following-sibling::td[1]")&.text

# Nokogiri lets you mix: `.css` and `.xpath` on the same document, and
# `.search` accepts either.
rows = doc.search("table.spec tr")

Our XPath cheat sheet has the patterns worth memorising. One Nokogiri-specific note: XPath 1.0 is all libxml2 supports, so matches(), ends-with() and the other 2.0 functions aren't available — contains() and starts-with() are.

Namespaced XML (sitemaps, RSS, Atom) trips people up constantly. Either declare the namespace or strip it:

doc = Nokogiri::XML(xml)
doc.remove_namespaces!                     # blunt, but fine for scraping
urls = doc.xpath("//url/loc").map(&:text)

Memory on large documents

Nokogiri::HTML5 builds the entire tree in memory, and libxml2 allocates outside Ruby's heap — so a worker parsing 20 MB pages can grow its RSS in a way that GC.start doesn't obviously fix. Two mitigations:

• Parse in a block scope and let the document go out of scope before the next iteration; don't hold a document in a long-lived instance variable.
• For genuinely large XML (multi-hundred-megabyte feeds, sitemap indexes), stream it with Nokogiri::XML::Reader or a SAX parser instead of building a DOM:

Nokogiri::XML::Reader(File.open("huge-feed.xml")).each do |node|
  next unless node.name == "item" && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT

  item = Nokogiri::XML(node.outer_xml)     # one small document at a time
  puts item.at_xpath("//title")&.text
end

Nokogiri installation errors: what still breaks in 2026

Nokogiri has a reputation for being painful to install, and for most of the last decade it was earned — it compiles libxml2, libxslt and zlib, and any missing header meant a wall of C output. That reputation is now mostly historical. Nokogiri has shipped precompiled native gems since 1.11, covering Linux (glibc), macOS and Windows on both Intel and ARM, with musl builds — the ones Alpine needs — added in 1.16.

On a supported platform, this should just work with no system packages at all:

gem install nokogiri

If it doesn't, the useful question isn't "which --with-xml2-lib flag do I need" — it's "why is it compiling at all?" Almost always, one of these:

Your Gemfile.lock doesn't list your platform. Bundler only installs a precompiled gem for a platform recorded in the lockfile. Build on a Mac, deploy to Linux, and Bundler falls back to compiling from source in the container:

bundle lock --add-platform x86_64-linux      # or aarch64-linux on ARM servers
bundle lock --add-platform x86_64-linux-musl # Alpine

force_ruby_platform is set. This tells Bundler to ignore precompiled gems entirely. It's occasionally set deliberately and much more often inherited from a copy-pasted Dockerfile:

bundle config get force_ruby_platform    # check
bundle config unset force_ruby_platform  # usually the fix

You're on Alpine with Nokogiri older than 1.16. No musl gem existed, so it compiled. Upgrade Nokogiri, or if you're pinned to an old version, install the build dependencies:

RUN apk add --no-cache build-base libxml2-dev libxslt-dev zlib-dev

Your Ruby is older than the precompiled gems support. Native gems are built per Ruby minor version. On an EOL Ruby you fall off the precompiled path and back into compiling — one more reason to stay on a supported release.

gem install nokogiri --platform=ruby forces a source build. This flag appears in a lot of old advice as a fix; today it's the opposite of one.

If you truly do need to compile — an unusual architecture, a security policy requiring system libraries — the modern invocation is short:

# Debian/Ubuntu
sudo apt-get install -y build-essential libxml2-dev libxslt1-dev zlib1g-dev
bundle config build.nokogiri --use-system-libraries
bundle install

# macOS
xcode-select --install
brew install libxml2 libxslt
bundle config build.nokogiri --use-system-libraries

And verify what you ended up with, because a working build and a precompiled build look identical from the outside:

require "nokogiri"
puts Nokogiri::VERSION_INFO
# => {"warnings"=>[], "nokogiri"=>{"version"=>"1.19.4", ...},
#     "libxml"=>{"source"=>"packaged", ...}}

"source" => "packaged" means the vendored library — the fast, reproducible path. "system" means you compiled against whatever libxml2 the machine had, which is where version-skew bugs come from.

Sessions, logins, and forms: Mechanize

Mechanize bundles an HTTP client, a cookie jar, and Nokogiri into an object that remembers where it's been:

require "mechanize"

agent = Mechanize.new
agent.user_agent_alias = "Mac Safari"
agent.history_added = proc { sleep 0.5 }   # rate limit every navigation

login = agent.get("https://example.com/login")
form = login.form_with(action: /login/)
form.field_with(name: "email").value    = ENV.fetch("SCRAPER_EMAIL")
form.field_with(name: "password").value = ENV.fetch("SCRAPER_PASSWORD")
dashboard = agent.submit(form)

# Cookies persist automatically across subsequent requests
report = agent.get("https://example.com/reports/monthly")
rows   = report.search("table.data tr")   # Nokogiri under the hood

history_added is the underrated feature: it fires on every page load, so one line gives you a global delay without threading a sleep through your crawl logic. Mechanize is maintained (2.14.0, January 2025; commits through May 2026) and is the right tool whenever authentication or multi-step forms are involved. Our Mechanize guide goes deeper on form handling.

If you'd rather not add the dependency, HTTParty can carry a session too — it just doesn't manage the jar for you:

login = HTTParty.post("https://example.com/login",
                      body: { email: ..., password: ... },
                      follow_redirects: false)

cookie = login.headers["set-cookie"]
HTTParty.get("https://example.com/reports/monthly",
             headers: { "Cookie" => cookie })

That works for a single-cookie login and stops being pleasant the moment there's a CSRF token, a redirect chain, or cookie expiry. That's the line where Mechanize pays for itself.

What Mechanize can't do is run JavaScript — it has no JS engine at all. If a form submits over fetch() and updates the DOM, Mechanize sees nothing.

Scraping JavaScript pages: Ferrum

Ferrum drives Chrome over the Chrome DevTools Protocol directly — no ChromeDriver, no WebDriver process, no webdrivers gem. It's the modern default for headless browser work in Ruby:

require "ferrum"
require "nokogiri"

browser = Ferrum::Browser.new(
  headless: true,
  timeout: 20,
  window_size: [1366, 768],
  browser_options: { "disable-blink-features" => "AutomationControlled" }
)

page = browser.create_page
page.headers.set_overrides(user_agent: UA)

# Block what you don't parse — images and trackers are most of the page weight
page.network.blacklist = [
  %r{\.(png|jpe?g|gif|webp|svg|woff2?)(\?|$)},
  /googletagmanager\.com/,
  /doubleclick\.net/
]

begin
  page.go_to("https://example.com/spa-products")
  page.network.wait_for_idle(timeout: 10)   # settle XHR, not an arbitrary sleep

  raise "blocked: #{page.network.status}" unless page.network.status == 200

  doc = Nokogiri.HTML5(page.body)           # hand the rendered DOM to Nokogiri
  puts doc.css("li.product").size
ensure
  browser.quit                              # always kill the Chrome process
end

Two habits that matter more than the gem choice:

• network.wait_for_idle instead of sleep. It waits for in-flight connections to drain, so slow pages still work and fast pages don't cost you three wasted seconds each.
• network.blacklist aborts requests matching your patterns. Dropping images, fonts, and analytics typically cuts page load time and bandwidth substantially — and you were never going to parse them.

If your team already writes Capybara specs, Cuprite is the same engine behind Capybara's DSL (visit, find, all), which makes scraping code look like your test suite. It's a thinner layer over Ferrum, not a different browser.

What about Watir, Selenium, and Kimurai?

This is where most Ruby scraping articles are out of date, so here's the current state with dates:

Watir — avoid for new projects. Version 7.3.0 shipped in August 2023 and the GitHub repo's last commit was May 2024. It's a wrapper over selenium-webdriver (declared as ~> 4.2), and Selenium has released dozens of versions since. Nothing is broken today, but an unmaintained wrapper over a fast-moving dependency is a maintenance bill waiting to arrive.

selenium-webdriver — fine, just not the first choice. 4.46.0 (July 2026) is actively maintained and requires Ruby >= 3.3. Use it when you need a browser Ferrum can't drive (Firefox, Safari) or a Selenium Grid. For Chrome-only scraping, Ferrum has less machinery in the path. And delete the webdrivers gem: its README tells you outright to stop requiring it on Selenium 4.11+, because Selenium Manager now downloads and manages drivers itself.

Kimurai — genuinely revived, verify before committing. The framework was dormant at 1.4.0 from January 2019 through late 2025; 2.0.0 landed in December 2025 and 2.2.0 in January 2026, now requiring Ruby >= 3.2 and building on Capybara ~> 3.40. It gives you Scrapy-like spider classes, built-in throttling, and pipeline hooks. The caveat is that the revival is recent and the repo's last activity was January 2026 — run your own smoke test before making it load-bearing.

Scraping data out of PDFs

Plenty of the data worth collecting — regulatory filings, price lists, council minutes — is published as PDF. Ruby handles this with pdf-reader, which is pure Ruby and needs no native dependencies:

# Gemfile: gem "pdf-reader"
require "pdf-reader"
require "httparty"
require "stringio"

# PDF::Reader takes an IO, so you never have to touch disk
pdf = HTTParty.get("https://example.com/report.pdf", timeout: 30)
reader = PDF::Reader.new(StringIO.new(pdf.body))

puts reader.page_count
puts reader.info                              # title, author, producer

text = reader.pages.map(&:text).join("\n\n")

From there it's ordinary string work — regular expressions against text get you invoice numbers, dates and totals. Handle the two errors that matter:

begin
  reader = PDF::Reader.new(StringIO.new(pdf.body))
rescue PDF::Reader::MalformedPDFError
  # Truncated download or an HTML error page served with a .pdf URL —
  # check the response content type before parsing
rescue PDF::Reader::UnsupportedFeatureError
  # Usually encryption; PDF::Reader.new(io, password: "...") if you have it
end

Three limitations to plan around:

• Tables lose their structure. page.text returns text in reading order, so a three-column table arrives as a run-on line. When column position matters, page.runs gives you positioned text runs with x/y coordinates that you can bucket into columns yourself.
• Scanned PDFs contain no text at all. If page.text comes back empty or nearly so, the page is an image. You need OCR: rasterize with pdftoppm (from poppler) or ImageMagick, then run Tesseract via the rtesseract gem. Budget for accuracy well below 100% and validate the numbers you extract.
• It's CPU-bound. A few hundred pages is real work. Put PDF parsing in a background job, never in a request cycle.

For bulk extraction where you only need text, shelling out to poppler's pdftotext -layout is often faster than pure Ruby and preserves column alignment better — worth benchmarking against pdf-reader before you commit.

Concurrency without a framework

Scraping is I/O-bound, and MRI releases the GVL during blocking I/O, so a plain thread pool works well:

require "faraday"

def fetch_all(urls, concurrency: 5)
  queue   = Queue.new
  results = Queue.new
  urls.each { |u| queue << u }

  workers = Array.new(concurrency) do
    Thread.new do
      conn = Faraday.new(headers: { "User-Agent" => UA },
                         request: { timeout: 15 })
      loop do
        url = begin
          queue.pop(true)       # non-blocking; raises when the queue is empty
        rescue ThreadError
          break
        end

        begin
          results << [url, conn.get(url).body]
        rescue Faraday::Error => e
          warn "#{url}: #{e.class}"
        end
        sleep 0.2 + rand(0.3)   # jittered politeness delay
      end
    end
  end

  workers.each(&:join)
  Array.new(results.size) { results.pop }
end

Each thread gets its own Faraday connection — connection objects aren't guaranteed thread-safe to share. Start at 5 concurrent requests and only go higher against infrastructure you own or a service you're paying for.

When the gems aren't the problem

Past a certain point your scraper stops failing because of parsing and starts failing because the target doesn't want automated traffic. The escalation ladder, in order of effort:

1. A real User-Agent and matching headers. The cheapest fix, and it resolves a surprising share of 403s — see User-Agent rotation.
2. Slow down. Concurrency 2, one second between requests. Rate limits are the most common invisible block.
3. Change IPs. Datacenter proxies first; move to residential only when datacenter IPs get blocked, since they cost far more per request. Our guide to proxy types covers the tradeoff.
4. Render in a real browser with a plausible fingerprint — headless Chrome with default settings is detectable on its own.

Before scaling any of this, read the target's terms and robots.txt. Is web scraping legal covers the actual case law rather than the usual hand-waving; the summary is that it depends heavily on what data you take and how you got it.

Using WebScraping.AI from Ruby

Steps 3 and 4 above are infrastructure work: proxy pools, browser fleets, fingerprint maintenance. If that isn't the product you're building, our API does the fetch and hands your Ruby code the HTML — and there's a first-party gem for it (this site is a Rails app, so the Ruby client is one we use ourselves):

# Gemfile: gem "webscraping_ai", "~> 4.0"
require "webscraping_ai"
require "nokogiri"

client = WebScrapingAI::Client.new(api_key: ENV.fetch("WEBSCRAPING_AI_API_KEY"))

# Rendered HTML through a rotating proxy — then parse as usual
html = client.html(
  "https://example.com/spa-products",
  js: true,
  wait_for: "li.product",       # wait for a selector, not a fixed timeout
  proxy: "residential",
  country: "us"
)
products = Nokogiri.HTML5(html).css("li.product")

# Or skip selectors entirely and describe the fields you want
data = client.fields(
  "https://example.com/product/42",
  fields: {
    title: "Product title",
    price: "Current price in USD, numbers only",
    in_stock: "true or false, whether the item is purchasable"
  }
)
# => {"title" => "...", "price" => "39.99", "in_stock" => "true"}

Errors come back as a typed hierarchy, so retry logic stays readable:

begin
  client.html(url, js: true)
rescue WebScrapingAI::RateLimitError
  sleep 1
  retry
rescue WebScrapingAI::GatewayTimeoutError
  client.html(url, js: true, timeout: 30_000)   # page needed longer
end

Requests are priced in credits: 1 for a plain datacenter fetch, 5 with JavaScript rendering, 10 and 25 for residential without and with JS, 50 for stealth, and +5 when you use the AI extraction endpoints. Failed requests don't cost credits. The free tier is 2,000 credits a month with no credit card; full parameter reference is in the docs, and the AI extraction endpoints are documented there too. Teams running this for price monitoring usually pair it with a Sidekiq job per target.

Frequently asked questions

What is the best Ruby gem for web scraping? Nokogiri, for parsing — it's in every Ruby scraping stack regardless of how you fetch. For fetching, HTTParty if you want the shortest possible code and Faraday if you want middleware for retries and logging. There is no single gem that does the whole job well; Ruby's ecosystem composes small ones.

Why does installing Nokogiri fail, and how do I fix it? In 2026 a failing install almost always means Bundler is compiling from source instead of using a precompiled gem. Check that your platform is in Gemfile.lock (bundle lock --add-platform x86_64-linux), that force_ruby_platform isn't set, and that you're on Nokogiri 1.16 or newer if you're on Alpine. Installing libxml2-dev is the fallback, not the first step.

How do I set a timeout in HTTParty? Pass timeout: per request, default_timeout for a whole class, or open_timeout / read_timeout / write_timeout separately via default_options.update. Rescue Net::OpenTimeout and Net::ReadTimeout separately — they tell you different things. Remember read_timeout bounds each read, not the total request, so wrap the call if you need a hard ceiling.

How do I bypass SSL certificate verification in Ruby? HTTParty.get(url, verify: false), Faraday.new(ssl: { verify: false }), or http.verify_mode = OpenSSL::SSL::VERIFY_NONE on Net::HTTP. But bypassing it means anyone on the network path can feed your scraper fabricated data, so fix the cause first: update your CA bundle, add the missing intermediate certificate, or trust your corporate proxy's root. If you must skip it, do so for one named host, never as a class-wide default.

How do I use custom SSL certificates with HTTParty? ssl_ca_file: for a custom CA bundle, ssl_ca_path: for a hashed directory, and pem: plus pem_password: for client certificates in mutual TLS. pem: takes the file contents, not a path — that's the most common mistake.

How do I scrape JavaScript-rendered pages in Ruby? Use Ferrum, which drives headless Chrome over CDP with no WebDriver dependency, and pass page.body to Nokogiri once page.network.wait_for_idle returns. Before that, check whether the page fetches its data from a JSON endpoint you can call directly — that's faster and far more stable than driving a browser.

Can I extract data from PDFs with Ruby? Yes — pdf-reader is pure Ruby, takes any IO object, and gives you page.text per page plus page.runs when you need x/y positions to rebuild table columns. Scanned PDFs have no text layer and need OCR (rasterize, then Tesseract via rtesseract).

Is Nokogiri::HTML the same as Nokogiri.HTML5? No. Nokogiri::HTML is an alias for Nokogiri::HTML4, which uses libxml2's pre-HTML5 parser. Nokogiri.HTML5 uses the WHATWG-conformant gumbo parser and builds the same tree a browser would. Use HTML5 for anything scraped off the modern web.

Is Watir still maintained? Not actively. The last release, 7.3.0, was August 2023 and the repository's last commit was May 2024. It still works against selenium-webdriver ~> 4.2, but new Ruby browser automation should start with Ferrum or selenium-webdriver directly.

Can I scrape from inside a Rails app? Yes — that's one of Ruby's practical advantages. Put the fetch in an ActiveJob/Sidekiq worker rather than a request cycle, persist results with ActiveRecord, and cache raw HTML so a parser bug doesn't mean re-fetching everything.