diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e112f2..8e35974 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `RubyProxyHeaders::Typhoeus` sends proxy headers with libcurl `CURLOPT_PROXYHEADER` (`:proxyheader` and `headeropt: :separate` on Typhoeus 1.7 / Ethon 0.19) and exposes the CONNECT response, including `X-ProxyMesh-IP`, as `proxy_response_headers`. Those headers are parsed from the first block of `response_headers`. Typhoeus's `response.headers` stays the origin response. + +### Changed + +- Development dependencies are now `typhoeus` `~> 1.7` and `ethon` `~> 0.19`. The gem still has no runtime dependency on Typhoeus; applications that use the adapter should add `gem 'typhoeus', '~> 1.7'`. + +### Fixed + +- HTTPS requests through `RubyProxyHeaders::Typhoeus` no longer call `RubyProxyHeaders::NetHTTP.request`, which is not defined. + ## [0.1.0] - 2024-02-28 ### Added diff --git a/DEFERRED.md b/DEFERRED.md index c2babae..242c4f7 100644 --- a/DEFERRED.md +++ b/DEFERRED.md @@ -1,14 +1,6 @@ # Deferred / not fully supported -## Typhoeus / Ethon - -**Status:** Not implemented in this gem. - -[Ethon](https://github.com/typhoeus/ethon) does not expose `CURLOPT_PROXYHEADER` (or equivalent) in its `Ethon::Easy` option mapping as of ethon 0.18.x. Adding CONNECT response header capture would require Ethon/libcurl changes or a custom C extension. - -**Workaround:** Use `Net::HTTP` + {RubyProxyHeaders::NetHTTP.patch!} or Faraday with `ruby_proxy_headers_net_http` adapter. - ---- +Typhoeus is supported. Ethon 0.19 exposes `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt`, and `RubyProxyHeaders::Typhoeus` uses them. See [docs/typhoeus.md](docs/typhoeus.md). ## Mechanize diff --git a/Gemfile.lock b/Gemfile.lock index fdfa65a..0df34f1 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -18,7 +18,7 @@ GEM csv (3.3.6) diff-lcs (1.6.2) domain_name (0.6.20240107) - ethon (0.18.0) + ethon (0.19.1) ffi (>= 1.15.0) logger excon (1.6.0) @@ -139,8 +139,8 @@ GEM rubyntlm (0.6.5) base64 timeout (0.6.1) - typhoeus (1.6.0) - ethon (>= 0.18.0) + typhoeus (1.7.1) + ethon (>= 0.19.1, < 0.20.0) unicode-display_width (3.3.0) unicode-emoji (~> 4.3) unicode-emoji (4.3.0) @@ -166,6 +166,7 @@ PLATFORMS DEPENDENCIES bundler (>= 2.4) + ethon (~> 0.19) excon (~> 1.4) faraday (~> 2.14) faraday-net_http (~> 3.4) @@ -175,7 +176,7 @@ DEPENDENCIES rspec (~> 3.12) rubocop (~> 1.50) ruby-proxy-headers! - typhoeus (~> 1.6) + typhoeus (~> 1.7) webmock (~> 3.18) yard (~> 0.9) @@ -189,7 +190,7 @@ CHECKSUMS csv (3.3.6) sha256=aba61e7e507a66f03d45cb1f3c4b6359861c3504038b422962875dce099e4456 diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962 domain_name (0.6.20240107) sha256=5f693b2215708476517479bf2b3802e49068ad82167bcd2286f899536a17d933 - ethon (0.18.0) sha256=b598afc9f30448cb068b850714b7d6948e941476095d04f90a4ac65b8d6efcb2 + ethon (0.19.1) sha256=f309dec67cfdefaf96b3d1ee0ed4f186eac518f131340f607bb3df3b4e13ddc4 excon (1.6.0) sha256=ebe2ab83c055ef8e117c62c91a3535b966a59a64868188219d49ab90cd83b65f faraday (2.15.0) sha256=3be0eff0187ed913a1b4f679e9fcfe845b542cdd4f75b76cd6181a4ae1335fff faraday-net_http (3.5.0) sha256=2e7094c34e38754ee048cc3df47376f8657005d9375f8473bce1c3b526a22e36 @@ -249,7 +250,7 @@ CHECKSUMS ruby-proxy-headers (0.2.3) rubyntlm (0.6.5) sha256=47013402b99ae29ee93f930af51edaec8c6008556f4be25705a422b4430314f5 timeout (0.6.1) sha256=78f57368a7e7bbadec56971f78a3f5ecbcfb59b7fcbb0a3ed6ddc08a5094accb - typhoeus (1.6.0) sha256=bacc41c23e379547e29801dc235cd1699b70b955a1ba3d32b2b877aa844c331d + typhoeus (1.7.1) sha256=86dcb6d787ee08e34947c9436b24af3c4b02880dc83c466b6ca4dbfc53c2ee7f unicode-display_width (3.3.0) sha256=4b7aa66a4b11db50f6f7e98411215cd0dd10eaecb665d36eec5bcacb3fa0b613 unicode-emoji (4.3.0) sha256=11c02fa73290378c066bb0562cd4c87d8e1b706fbbe0059ca12746a4244de8ce uri (1.1.2) sha256=89292e98eabbef6248b0a0ed5e7e398e152f83f17560023a6d94a7f492b169fb diff --git a/IMPLEMENTATION_PRIORITY.md b/IMPLEMENTATION_PRIORITY.md index 9655af8..82bfbec 100644 --- a/IMPLEMENTATION_PRIORITY.md +++ b/IMPLEMENTATION_PRIORITY.md @@ -42,9 +42,13 @@ Prioritized roadmap for extension modules, aligned with [javascript-proxy-header --- -## Phase 4 — Typhoeus / Ethon (**deferred**) +## Phase 4 — Typhoeus / Ethon (**done**) -Ethon does not expose libcurl `CURLOPT_PROXYHEADER` in its option layer. See [DEFERRED.md](DEFERRED.md). +Ethon 0.19 exposes `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt`. `RubyProxyHeaders::Typhoeus` sends proxy headers on CONNECT with `headeropt: :separate` and reads `X-ProxyMesh-IP` from the first block of `response_headers`. + +**File:** `lib/ruby_proxy_headers/typhoeus.rb` + +Typhoeus ~> 1.7 and Ethon ~> 0.19 are development dependencies. The gem does not depend on them at runtime. --- @@ -65,8 +69,8 @@ Uses `net-http-persistent`; needs dedicated integration. See [DEFERRED.md](DEFER ## Testing -`bundle exec ruby test/test_proxy_headers.rb` with `PROXY_URL` set. Modules: `net_http`, `faraday`, `httparty`, `excon` (excon is a smoke test; CONNECT response headers are not asserted). +`bundle exec ruby test/test_proxy_headers.rb` with `PROXY_URL` set. Modules: `net_http`, `faraday`, `httparty`, `excon`, `typhoeus` (excon is a smoke test; CONNECT response headers are not asserted). `spec/typhoeus_spec.rb` covers the Typhoeus CONNECT path against a local mock proxy. --- -*Updated: March 2026* +*Updated: October 2026* diff --git a/LIBRARY_RESEARCH.md b/LIBRARY_RESEARCH.md index bd94a2f..d7f7cb3 100644 --- a/LIBRARY_RESEARCH.md +++ b/LIBRARY_RESEARCH.md @@ -11,10 +11,10 @@ This document analyzes the Ruby HTTP / scraping stack used in [proxy-examples](h | **HTTParty** | No — built on `Net::HTTP` | **High:** global or per-class `Net::HTTP` instances after `patch!`; or subclass `Connection` if needed | | **Mechanize** | No — uses `Net::HTTP` / persistent connections internally | **Medium–high:** ensure Mechanize’s internal HTTP object gets the same patched behavior and header accessors | | **Excon** | **Send:** yes — `:ssl_proxy_headers` on connection. **Read CONNECT response:** not exposed on origin `Excon::Response` | **Partial:** use `:ssl_proxy_headers` for sends; reading `X-ProxyMesh-IP` from CONNECT needs upstream Excon changes (see [DEFERRED.md](DEFERRED.md)) | -| **Typhoeus / Ethon** | Partial — libcurl has proxy options | **Medium:** map headers to `CURLOPT_PROXYHEADER` / related options (libcurl version dependent); capture CONNECT response via callbacks or debug hooks where supported | +| **Typhoeus / Ethon** | **Send and read:** `:proxyheader` and `:headeropt` since Ethon 0.19 / Typhoeus 1.7 | **Implemented:** `RubyProxyHeaders::Typhoeus` sets `headeropt: :separate` and parses the CONNECT block from `response_headers` | | **Nokogiri** | N/A (XML/HTML parser only) | **N/A** — proxying is whatever HTTP client fetches the document (usually `Net::HTTP`) | -As with Node and Python stacks, **none** of the high-level Ruby clients expose a first-class “proxy CONNECT request/response headers” API; extensions must hook **below** the HTTP library (stdlib `Net::HTTP`), **inside** Faraday’s adapter, or **at** the libcurl / Excon layer. +As with Node and Python stacks, most high-level Ruby clients do not expose a first-class “proxy CONNECT request/response headers” API. Extensions hook **below** the HTTP library (stdlib `Net::HTTP`), **inside** Faraday’s adapter, or **at** the libcurl / Excon layer. Typhoeus 1.7 / Ethon 0.19 is used directly via `:proxyheader`. ## Connection flow (HTTPS over HTTP proxy) @@ -56,9 +56,9 @@ Own implementation of proxy and TLS; does not use `Net::HTTP#connect`. ## Typhoeus / Ethon (libcurl) -libcurl can send additional headers to the proxy; exact options depend on libcurl version (e.g. proxy header lists). CONNECT response header visibility may require `CURLOPT_DEBUGFUNCTION` or version-specific features. +**Status:** Implemented for Typhoeus 1.7+ / Ethon 0.19+. -**Feasibility:** Medium — needs research per libcurl version and Ethon API surface. +Ethon maps `CURLOPT_PROXYHEADER` to `:proxyheader` and `CURLOPT_HEADEROPT` to `:headeropt` (`:unified` / `:separate`). `RubyProxyHeaders::Typhoeus` puts caller proxy headers on that list with `:separate`, so CONNECT does not also receive application headers. The CONNECT response, including `X-ProxyMesh-IP`, is the first header block of `response_headers` and is exposed as `proxy_response_headers`. There is no runtime gem dependency; applications install `typhoeus` ~> 1.7 themselves. libcurl 7.37 or newer is required. ## Nokogiri @@ -68,4 +68,4 @@ Parsing only; no network layer. --- -*Last updated: March 2026* +*Last updated: October 2026* diff --git a/README.md b/README.md index 97cb3c6..a87bccc 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Most Ruby HTTP clients use `Net::HTTP`, Faraday, or libcurl without exposing: 1. Extra headers on the `CONNECT` request to the proxy. 2. Headers from the proxy’s `CONNECT` response (often discarded after the tunnel is established). -This library adds opt-in support for **Net::HTTP**, **Faraday** (2.x via `faraday-net_http`), **HTTParty**, and documents **Excon**’s built-in `:ssl_proxy_headers` for sends. +This library adds opt-in support for **Net::HTTP**, **Faraday** (2.x via `faraday-net_http`), **HTTParty**, and **Typhoeus** (1.7+ / Ethon 0.19+), and documents **Excon**’s built-in `:ssl_proxy_headers` for sends. ## Why teams use this @@ -30,7 +30,7 @@ Or add to your `Gemfile`: gem 'ruby-proxy-headers' ``` -The `Net::HTTP` patch is pure Ruby. Install **faraday**, **faraday-net_http**, **httparty**, and/or **excon** when you use those integrations. +The `Net::HTTP` patch is pure Ruby. Install **faraday**, **faraday-net_http**, **httparty**, **typhoeus** (~> 1.7), and/or **excon** when you use those integrations. Typhoeus is not a runtime dependency of this gem. ## Net::HTTP @@ -94,6 +94,31 @@ HTTParty.get( puts RubyProxyHeaders.proxy_connect_response_headers['X-ProxyMesh-IP'] ``` +## Typhoeus + +Requires [Typhoeus](https://github.com/typhoeus/typhoeus) 1.7 or newer (Ethon 0.19+, which exposes `CURLOPT_PROXYHEADER`) and libcurl 7.37 or newer. Add Typhoeus to the application Gemfile; this gem does not depend on it at runtime. + +```ruby +gem 'typhoeus', '~> 1.7' +gem 'ruby-proxy-headers' +``` + +```ruby +require 'ruby_proxy_headers/typhoeus' + +response = RubyProxyHeaders::Typhoeus.get( + 'https://api.ipify.org?format=json', + proxy: ENV.fetch('PROXY_URL'), + proxy_headers: { 'X-ProxyMesh-Country' => 'US' }, # optional + headers: { 'Accept' => 'application/json' } +) + +puts response.body +puts response.proxy_response_headers['x-proxymesh-ip'] +``` + +Proxy headers are passed as libcurl's proxy header list (`:proxyheader`) with `:headeropt` set to `:separate`, so CONNECT carries those headers and application headers stay on the tunneled request. `proxy_response_headers` is the proxy's CONNECT response (the first block of `response_headers`). `response.headers` remains the origin response. + ## Excon (send-only; CONNECT response headers) Excon supports **sending** extra headers on CONNECT with `:ssl_proxy_headers`. Reading `X-ProxyMesh-IP` from the CONNECT response is **not** exposed on the origin response object — see [DEFERRED.md](DEFERRED.md). diff --git a/docs/getting-started.md b/docs/getting-started.md index 252153c..dbe155f 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -71,6 +71,20 @@ Choose the module that matches your HTTP library: ) ``` +=== "Typhoeus" + + ```ruby + require 'ruby_proxy_headers/typhoeus' + + response = RubyProxyHeaders::Typhoeus.get( + 'https://example.com', + proxy: 'http://user:pass@proxy:8080', + proxy_headers: { 'X-Custom-Header' => 'value' } + ) + ``` + + Requires Typhoeus ~> 1.7 (Ethon ~> 0.19). Proxy header names on the response are lowercase. + ### 3. Access Proxy Response Headers ```ruby diff --git a/docs/index.md b/docs/index.md index dd26ae8..b11dc97 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,7 +35,7 @@ Standard Ruby HTTP libraries support basic proxy authentication but **do not** a | Faraday | `RubyProxyHeaders::Faraday` | ✅ | | HTTParty | `RubyProxyHeaders::HTTParty` | ✅ | | HTTP.rb | `RubyProxyHeaders::HTTPGem` | ✅ | -| Typhoeus | `RubyProxyHeaders::Typhoeus` | ✅ | +| Typhoeus 1.7+ | `RubyProxyHeaders::Typhoeus` | ✅ | | Excon | `RubyProxyHeaders::Excon` | ✅ | | RestClient | `RubyProxyHeaders::RestClient` | ✅ | diff --git a/docs/testing.md b/docs/testing.md index 51a3c3e..2861327 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -20,7 +20,7 @@ bundle exec rspec spec/connection_spec.rb ## Running Integration Tests -Integration tests require an actual proxy server. +Integration tests require an actual proxy server. The Typhoeus case is included. `spec/typhoeus_spec.rb` also drives Typhoeus 1.7 through a local mock proxy (CONNECT receives proxy headers only, and `X-ProxyMesh-IP` comes back on `proxy_response_headers`) and does not need `PROXY_URL`. ### Environment Variables @@ -42,7 +42,7 @@ export SEND_PROXY_VALUE='US' ruby test/test_proxy_headers.rb # Run specific tests -ruby test/test_proxy_headers.rb net_http faraday +ruby test/test_proxy_headers.rb net_http faraday typhoeus # List available tests ruby test/test_proxy_headers.rb -l diff --git a/docs/typhoeus.md b/docs/typhoeus.md index 60b9a23..ebd986c 100644 --- a/docs/typhoeus.md +++ b/docs/typhoeus.md @@ -1,25 +1,32 @@ # Typhoeus -[Typhoeus](https://typhoeus.github.io/) is a fast HTTP client that wraps libcurl. The `RubyProxyHeaders::Typhoeus` module provides proxy header support. +[Typhoeus](https://typhoeus.github.io/) is an HTTP client that wraps libcurl through [Ethon](https://github.com/typhoeus/ethon). `RubyProxyHeaders::Typhoeus` sends custom proxy headers on HTTPS `CONNECT` and reads the proxy's CONNECT response. -## Installation +## Requirements + +- Typhoeus 1.7 or newer (`gem 'typhoeus', '~> 1.7'`), which depends on Ethon `>= 0.19`, `< 0.20` +- Ethon 0.19 or newer, for `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt` (`:unified` / `:separate`) +- libcurl 7.37 or newer (libcurl 8.x is what the mock-proxy spec runs against) + +`ruby-proxy-headers` does not declare a runtime dependency on Typhoeus or Ethon. Add Typhoeus to the application Gemfile when you use this adapter. Typhoeus 1.7 has its own breaking changes; see Typhoeus's `UPGRADE.md` before upgrading an existing app. ```ruby -gem 'typhoeus' -gem 'ruby_proxy_headers' +gem 'typhoeus', '~> 1.7' +gem 'ruby-proxy-headers' ``` -**Note:** Typhoeus requires libcurl to be installed on your system. +**Note:** Typhoeus requires libcurl to be installed on the system. ## Basic Usage ```ruby -require 'ruby_proxy_headers' +require 'ruby_proxy_headers/typhoeus' response = RubyProxyHeaders::Typhoeus.get( 'https://api.ipify.org?format=json', proxy: 'http://user:pass@proxy.example.com:8080', - proxy_headers: { 'X-ProxyMesh-Country' => 'US' } + proxy_headers: { 'X-ProxyMesh-Country' => 'US' }, + headers: { 'Accept' => 'application/json' } ) puts "Status: #{response.code}" @@ -27,6 +34,8 @@ puts "Body: #{response.body}" puts "Proxy IP: #{response.proxy_response_headers['x-proxymesh-ip']}" ``` +`proxy_headers` are copied into a libcurl proxy-header list and `:headeropt` is set to `:separate`. On HTTPS, libcurl sends that list only on `CONNECT`. Headers passed in `:headers` go to the origin and are not copied onto the CONNECT request. + ## API Reference ### `RubyProxyHeaders::Typhoeus.get(url, options)` @@ -38,8 +47,9 @@ Make a GET request. - `url` (String) - Target URL - `options` (Hash): - `:proxy` (String) - Proxy URL (required) - - `:proxy_headers` (Hash) - Headers to send to proxy - - `:headers` (Hash) - Request headers + - `:proxy_headers` (Hash) - Headers to send to the proxy + - `:headers` (Hash) - Headers to send to the origin + - Other Typhoeus / Ethon options (`:timeout`, `:body`, and so on) are passed through **Returns:** `ProxyResponse` @@ -65,36 +75,40 @@ Make a request with any HTTP method. ## Response Object ```ruby -response.code # HTTP status code +response.code # HTTP status code of the origin response response.body # Response body -response.headers # Response headers -response.proxy_response_headers # Proxy CONNECT response headers +response.headers # Origin response headers (last header block) +response.proxy_response_headers # Proxy CONNECT response headers (lowercase names) response.success? # True if 2xx status ``` -## Performance Note +libcurl appends every header block to `response_headers`. The proxy's CONNECT response is the first block, for example: + +```http +HTTP/1.1 200 Connection established +X-ProxyMesh-IP: 203.0.113.10 +``` + +Typhoeus's `response.headers` is only the last block, the origin response, so `X-ProxyMesh-IP` is not in `response.headers`. This adapter parses that first block when its status line says `Connection established`. -Typhoeus is built on libcurl, which has native support for `CURLOPT_PROXYHEADER`. While the current implementation uses our core connection handler for HTTPS proxy headers, a future version may leverage libcurl's native support for improved performance. +## Example: Several countries -## Example: Parallel Requests +Each helper call performs one request. Repeat it to send a different proxy header: ```ruby -require 'ruby_proxy_headers' - -hydra = Typhoeus::Hydra.new +require 'json' +require 'ruby_proxy_headers/typhoeus' countries = %w[US UK DE FR JP] results = {} countries.each do |country| - # Note: For parallel requests, use standard Typhoeus - # Our module is for sequential requests with proxy headers response = RubyProxyHeaders::Typhoeus.get( 'https://api.ipify.org?format=json', - proxy: ENV['PROXY_URL'], + proxy: ENV.fetch('PROXY_URL'), proxy_headers: { 'X-ProxyMesh-Country' => country } ) - + results[country] = { ip: JSON.parse(response.body)['ip'], proxy_ip: response.proxy_response_headers['x-proxymesh-ip'] diff --git a/lib/ruby_proxy_headers/typhoeus.rb b/lib/ruby_proxy_headers/typhoeus.rb index 3810086..9384175 100644 --- a/lib/ruby_proxy_headers/typhoeus.rb +++ b/lib/ruby_proxy_headers/typhoeus.rb @@ -1,8 +1,27 @@ # frozen_string_literal: true +require_relative '../ruby_proxy_headers' + +begin + require 'typhoeus' +rescue LoadError => e + raise LoadError, + 'typhoeus ~> 1.7 is required for ruby_proxy_headers/typhoeus ' \ + "(#{e.message})" +end + module RubyProxyHeaders - # Typhoeus/Ethon integration for proxy headers support. - # Typhoeus wraps libcurl, which has native support for CURLOPT_PROXYHEADER. + # Typhoeus adapter using libcurl's proxy-header list. + # + # Typhoeus 1.7 / Ethon 0.19 expose +:proxyheader+ (`CURLOPT_PROXYHEADER`) + # and +:headeropt+ (`CURLOPT_HEADEROPT`). With +headeropt: :separate+, + # proxy headers are sent only to the proxy. On HTTPS that is the CONNECT + # request; +:headers+ stay on the tunneled origin request. + # + # libcurl records the CONNECT response as the first header block of + # +response_headers+. Typhoeus's +response.headers+ is the last block (the + # origin response) only. This adapter parses the CONNECT block into + # +proxy_response_headers+ with lowercase names, matching Net::HTTP. # # @example # response = RubyProxyHeaders::Typhoeus.get( @@ -10,62 +29,131 @@ module RubyProxyHeaders # proxy: 'http://user:pass@proxy:8080', # proxy_headers: { 'X-ProxyMesh-Country' => 'US' } # ) - # puts response.proxy_response_headers + # puts response.proxy_response_headers['x-proxymesh-ip'] # module Typhoeus - # Make a GET request with proxy headers. - # @param url [String] Target URL - # @param proxy [String] Proxy URL - # @param proxy_headers [Hash] Custom headers to send to proxy - # @param options [Hash] Additional Typhoeus options - # @return [ProxyResponse] - def self.get(url, proxy:, proxy_headers: {}, **options) - request(:get, url, proxy: proxy, proxy_headers: proxy_headers, **options) - end + CONNECT_ESTABLISHED = /connection established/i - # Make a POST request with proxy headers. - def self.post(url, proxy:, proxy_headers: {}, body: nil, **options) - request(:post, url, proxy: proxy, proxy_headers: proxy_headers, body: body, **options) - end + class << self + # Make a GET request with proxy headers. + # @param url [String] Target URL + # @param proxy [String] Proxy URL + # @param proxy_headers [Hash] Custom headers to send to the proxy + # @param options [Hash] Additional Typhoeus options + # @return [ProxyResponse] + def get(url, proxy:, proxy_headers: {}, **options) + request(:get, url, proxy: proxy, proxy_headers: proxy_headers, **options) + end + + # Make a POST request with proxy headers. + # @param url [String] Target URL + # @param proxy [String] Proxy URL + # @param proxy_headers [Hash] Custom headers to send to the proxy + # @param body [String, nil] Request body + # @param options [Hash] Additional Typhoeus options + # @return [ProxyResponse] + def post(url, proxy:, proxy_headers: {}, body: nil, **options) + request(:post, url, proxy: proxy, proxy_headers: proxy_headers, body: body, **options) + end + + # Make a request with proxy headers. + # @param method [Symbol] HTTP method + # @param url [String] Target URL + # @param proxy [String] Proxy URL + # @param proxy_headers [Hash] Custom headers to send to the proxy + # @param options [Hash] Additional Typhoeus options + # @return [ProxyResponse] + def request(method, url, proxy:, proxy_headers: {}, **options) + typhoeus_options = options.merge(proxy: proxy, method: method) + apply_proxy_headers!(typhoeus_options, proxy_headers || {}) + + response = ::Typhoeus::Request.new(url, typhoeus_options).run + ProxyResponse.new(response, extract_proxy_response_headers(response)) + end + + private + + # libcurl does not copy the CURLOPT_PROXYHEADER list. The Slist wrapper + # stays referenced by the options hash passed to Typhoeus::Request, which + # outlives the easy-handle reset Typhoeus performs before #run returns. + def apply_proxy_headers!(typhoeus_options, proxy_headers) + return if proxy_headers.empty? + + ensure_proxyheader_support! + typhoeus_options[:proxyheader] = proxyheader_slist(proxy_headers) + typhoeus_options[:headeropt] = :separate + end + + def ensure_proxyheader_support! + curl_options = ::Ethon::Curl.easy_options(nil) + return if curl_options.key?(:proxyheader) && curl_options.key?(:headeropt) + + raise LoadError, + 'RubyProxyHeaders::Typhoeus requires Ethon >= 0.19 (:proxyheader and :headeropt) ' \ + 'and libcurl with CURLOPT_PROXYHEADER. Install typhoeus ~> 1.7 and ethon ~> 0.19.' + end + + def proxyheader_slist(proxy_headers) + lines = proxy_headers.map { |name, value| proxy_header_line(name, value) } + pointer = ::Ethon::Curl::Slist.build(lines) + raise ArgumentError, 'failed to build the proxy header list' if pointer.nil? || pointer.null? - # Make a request with proxy headers. - # @param method [Symbol] HTTP method - # @param url [String] Target URL - # @param proxy [String] Proxy URL - # @param proxy_headers [Hash] Custom headers to send to proxy - # @param options [Hash] Additional Typhoeus options - # @return [ProxyResponse] - def self.request(method, url, proxy:, proxy_headers: {}, **options) - require 'typhoeus' - - uri = URI.parse(url) - - # For HTTPS, we need custom handling since Typhoeus doesn't expose CURLOPT_PROXYHEADER - if uri.scheme == 'https' && !proxy_headers.empty? - # Use our core connection for now - # TODO: Extend Ethon to expose CURLOPT_PROXYHEADER for native support - response = RubyProxyHeaders::NetHTTP.request( - method, - url, - proxy: proxy, - proxy_headers: proxy_headers, - headers: options[:headers], - body: options[:body] - ) - return response - end - - # For HTTP or no proxy headers, use standard Typhoeus - typhoeus_options = options.merge( - proxy: proxy, - method: method - ) - - response = ::Typhoeus::Request.new(url, typhoeus_options).run - ProxyResponse.new(response, {}) + ::Ethon::Curl::Slist.new(pointer) + end + + def proxy_header_line(name, value) + RubyProxyHeaders.validate_header!(name, value) + "#{name}: #{value}" + end + + def extract_proxy_response_headers(response) + block = connect_response_block(response) + return {} if block.nil? + + parse_header_block(block) + end + + def connect_response_block(response) + first = first_http_block(response) + return nil if first.nil? + + status = first.lstrip.lines.first.to_s + status.match?(CONNECT_ESTABLISHED) ? first : nil + end + + def first_http_block(response) + return nil unless response.respond_to?(:response_headers) + + raw = response.response_headers + return nil if raw.nil? || raw.empty? + + raw.to_s.gsub("\r\n", "\n").split("\n\n").find { |block| block.lstrip.start_with?('HTTP/') } + end + + def parse_header_block(block) + headers = {} + block.each_line do |line| + name, value = header_pair(line) + next if name.nil? + + key = name.downcase + headers[key] = headers.key?(key) ? "#{headers[key]}, #{value}" : value + end + headers + end + + def header_pair(line) + stripped = line.strip + return if stripped.empty? || stripped.start_with?('HTTP/') + + name, value = stripped.split(':', 2) + return if name.nil? || value.nil? + + [name.strip, value.strip] + end end - # Response wrapper with proxy headers accessor + # Response wrapper with proxy headers accessor. class ProxyResponse attr_reader :proxy_response_headers diff --git a/ruby-proxy-headers.gemspec b/ruby-proxy-headers.gemspec index e69bf9b..2e89af9 100644 --- a/ruby-proxy-headers.gemspec +++ b/ruby-proxy-headers.gemspec @@ -32,6 +32,11 @@ Gem::Specification.new do |s| s.add_development_dependency 'faraday-net_http', '~> 3.4' s.add_development_dependency 'httparty', '~> 0.24' s.add_development_dependency 'mechanize', '~> 2.14' - s.add_development_dependency 'typhoeus', '~> 1.6' + # Optional at runtime. The Typhoeus adapter needs 1.7+ so Ethon exposes + # CURLOPT_PROXYHEADER (:proxyheader) and :headeropt. Callers install typhoeus + # themselves; ethon is a direct dev dependency because the adapter builds the + # proxy header slist through Ethon::Curl::Slist. + s.add_development_dependency 'ethon', '~> 0.19' + s.add_development_dependency 'typhoeus', '~> 1.7' s.add_development_dependency 'webmock', '~> 3.18' end diff --git a/spec/support/typhoeus_connect_proxy.rb b/spec/support/typhoeus_connect_proxy.rb new file mode 100644 index 0000000..ee7c306 --- /dev/null +++ b/spec/support/typhoeus_connect_proxy.rb @@ -0,0 +1,162 @@ +# frozen_string_literal: true + +require 'openssl' +require 'socket' + +# Local HTTP proxy plus TLS origin used to observe CONNECT bytes. +# The proxy records the CONNECT request and answers with X-ProxyMesh-IP, +# then splices the tunnel to the origin. The origin records the HTTP request +# that arrived inside TLS. +module TyphoeusConnectLab + class Origin + attr_reader :port, :request, :error + + def initialize + @request = nil + @error = nil + @tcp = TCPServer.new('127.0.0.1', 0) + @port = @tcp.addr[1] + cert, key = self.class.certificate + context = OpenSSL::SSL::SSLContext.new + context.cert = cert + context.key = key + context.verify_mode = OpenSSL::SSL::VERIFY_NONE + @ssl = OpenSSL::SSL::SSLServer.new(@tcp, context) + @thread = Thread.new { accept_loop } + end + + def stop + @tcp.close + @thread.join(2) + end + + def self.certificate + key = OpenSSL::PKey::EC.generate('prime256v1') + name = OpenSSL::X509::Name.parse('/CN=127.0.0.1') + cert = OpenSSL::X509::Certificate.new + cert.version = 2 + cert.serial = 1 + cert.subject = name + cert.issuer = name + cert.public_key = key + now = Time.now + cert.not_before = now - 60 + cert.not_after = now + 3600 + factory = OpenSSL::X509::ExtensionFactory.new + factory.subject_certificate = cert + factory.issuer_certificate = cert + cert.add_extension(factory.create_extension('basicConstraints', 'CA:FALSE', true)) + cert.add_extension(factory.create_extension('subjectAltName', 'IP:127.0.0.1', false)) + cert.sign(key, OpenSSL::Digest.new('SHA256')) + [cert, key] + end + + private + + def accept_loop + loop do + client = @ssl.accept + handle(client) + end + rescue IOError, Errno::EBADF, OpenSSL::SSL::SSLError + nil + rescue StandardError => e + @error = e + end + + def handle(client) + data = +'' + data << client.readpartial(4096) until data.include?("\r\n\r\n") + @request = data + body = 'origin-ok' + client.write( + "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n" \ + "Content-Length: #{body.bytesize}\r\nConnection: close\r\n\r\n#{body}" + ) + ensure + client&.close + end + end + + class Proxy + attr_reader :port, :connect_request, :error + + def initialize + @connect_request = nil + @error = nil + @tcp = TCPServer.new('127.0.0.1', 0) + @port = @tcp.addr[1] + @thread = Thread.new { accept_loop } + end + + def stop + @tcp.close + @thread.join(2) + end + + private + + def accept_loop + loop do + client = @tcp.accept + handle(client) + end + rescue IOError, Errno::EBADF + nil + rescue StandardError => e + @error = e + end + + def handle(client) + upstream = nil + head, rest = read_head(client) + @connect_request = head + client.write( + "HTTP/1.1 200 Connection established\r\nX-ProxyMesh-IP: 203.0.113.10\r\n\r\n" + ) + host, port = connect_target(head) + upstream = TCPSocket.new(host, port) + upstream.write(rest) unless rest.empty? + splice(client, upstream) + ensure + upstream&.close + client&.close + end + + def read_head(sock) + data = +'' + data << sock.readpartial(1024) until data.include?("\r\n\r\n") + head, rest = data.split("\r\n\r\n", 2) + [head, rest.to_s] + end + + def connect_target(head) + target = head.lines.first.to_s.split[1].to_s + host, _, port = target.rpartition(':') + [host, Integer(port)] + end + + def splice(left, right) + threads = [Thread.new { copy(left, right) }, Thread.new { copy(right, left) }] + threads.each do |thread| + thread.join + rescue StandardError + nil + end + end + + def copy(src, dst) + IO.copy_stream(src, dst) + rescue IOError, SystemCallError + nil + ensure + close_quietly(dst) + end + + def close_quietly(io) + io.close + rescue IOError, SystemCallError + nil + end + end +end diff --git a/spec/typhoeus_spec.rb b/spec/typhoeus_spec.rb new file mode 100644 index 0000000..d121b4e --- /dev/null +++ b/spec/typhoeus_spec.rb @@ -0,0 +1,144 @@ +# frozen_string_literal: true + +require 'spec_helper' +require 'ruby_proxy_headers/typhoeus' +require 'timeout' +require_relative 'support/typhoeus_connect_proxy' + +RSpec.describe RubyProxyHeaders::Typhoeus do + let(:proxy) { 'http://127.0.0.1:9' } + + def header_response(*blocks) + raw = blocks.join("\r\n\r\n") + ::Typhoeus::Response.new(response_code: 200, response_headers: raw, response_body: 'origin-ok') + end + + describe '.request' do + it 'sends proxy headers on a separate proxyheader list and reads the CONNECT block' do + response = header_response( + "HTTP/1.1 200 Connection established\r\nX-ProxyMesh-IP: 203.0.113.10", + "HTTP/1.1 302 Found\r\nLocation: https://example.test/next", + "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nX-ProxyMesh-IP: from-origin" + ) + captured = nil + allow(::Typhoeus::Request).to receive(:new) do |_url, options| + captured = options + instance_double(::Typhoeus::Request, run: response) + end + + result = described_class.get( + 'https://example.test/resource', + proxy: proxy, + proxy_headers: { 'X-ProxyMesh-Country' => 'US' }, + headers: { 'X-App-Secret' => 'should-not-leak' } + ) + + expect(captured[:proxy]).to eq(proxy) + expect(captured[:method]).to eq(:get) + expect(captured[:headeropt]).to eq(:separate) + expect(captured[:headers]).to include('X-App-Secret' => 'should-not-leak') + expect(captured[:proxyheader]).to be_a(::Ethon::Curl::Slist) + expect(captured[:proxyheader].to_a).to eq(['X-ProxyMesh-Country: US']) + expect(result.code).to eq(200) + expect(result.proxy_response_headers).to eq('x-proxymesh-ip' => '203.0.113.10') + expect(result.headers['x-proxymesh-ip']).to eq('from-origin') + expect(result.headers['content-type']).to eq('text/plain') + end + + it 'does not set proxyheader when no proxy headers are given' do + response = header_response( + "HTTP/1.1 200 Connection established\r\nX-ProxyMesh-IP: 203.0.113.4", + "HTTP/1.1 200 OK\r\nContent-Type: application/json" + ) + captured = nil + allow(::Typhoeus::Request).to receive(:new) do |_url, options| + captured = options + instance_double(::Typhoeus::Request, run: response) + end + + result = described_class.post( + 'https://example.test/items', + proxy: proxy, + body: 'payload' + ) + + expect(captured[:method]).to eq(:post) + expect(captured[:body]).to eq('payload') + expect(captured).not_to have_key(:proxyheader) + expect(captured).not_to have_key(:headeropt) + expect(result.proxy_response_headers['x-proxymesh-ip']).to eq('203.0.113.4') + end + + it 'ignores header blocks that are not a CONNECT response' do + response = header_response( + "HTTP/1.1 302 Found\r\nLocation: https://example.test/next", + "HTTP/1.1 200 OK\r\nContent-Type: text/plain" + ) + allow(::Typhoeus::Request).to receive(:new) do |_url, _options| + instance_double(::Typhoeus::Request, run: response) + end + + result = described_class.get('http://example.test/', proxy: proxy, proxy_headers: {}) + + expect(result.proxy_response_headers).to eq({}) + end + + it 'rejects proxy header values that could inject extra headers' do + expect(::Typhoeus::Request).not_to receive(:new) + + expect do + described_class.get( + 'https://example.test/', + proxy: proxy, + proxy_headers: { 'X-ProxyMesh-Country' => "US\r\nX-Injected: yes" } + ) + end.to raise_error(ArgumentError, /header value.*invalid/) + end + end + + describe 'local CONNECT proxy', :network do + around do |example| + WebMock.allow_net_connect! + example.run + ensure + WebMock.disable_net_connect! + end + + it 'sends proxy headers only on CONNECT and returns X-ProxyMesh-IP' do + origin = TyphoeusConnectLab::Origin.new + proxy_server = TyphoeusConnectLab::Proxy.new + + result = Timeout.timeout(10) do + described_class.get( + "https://127.0.0.1:#{origin.port}/tunnel", + proxy: "http://127.0.0.1:#{proxy_server.port}", + proxy_headers: { 'X-ProxyMesh-Country' => 'US' }, + headers: { 'X-App-Secret' => 'should-not-leak' }, + ssl_verifypeer: false, + ssl_verifyhost: 0, + http_version: :httpv1_1, + timeout: 5 + ) + end + + connect = proxy_server.connect_request.to_s + origin_request = origin.request.to_s + + expect(connect).to start_with('CONNECT 127.0.0.1:') + expect(connect).to include('X-ProxyMesh-Country: US') + expect(connect).not_to include('X-App-Secret') + expect(connect).not_to include('should-not-leak') + expect(origin_request).to include('X-App-Secret: should-not-leak') + expect(origin_request).not_to include('X-ProxyMesh-Country') + expect(result.code).to eq(200) + expect(result.body).to eq('origin-ok') + expect(result.proxy_response_headers['x-proxymesh-ip']).to eq('203.0.113.10') + expect(result.headers['x-proxymesh-ip']).to be_nil + expect(proxy_server.error).to be_nil + expect(origin.error).to be_nil + ensure + proxy_server&.stop + origin&.stop + end + end +end diff --git a/test/test_proxy_headers.rb b/test/test_proxy_headers.rb index ec8e9c5..fa806f7 100644 --- a/test/test_proxy_headers.rb +++ b/test/test_proxy_headers.rb @@ -8,7 +8,7 @@ # SEND_PROXY_HEADER, SEND_PROXY_VALUE — optional extra headers on CONNECT # # Usage: -# bundle exec ruby test/test_proxy_headers.rb [-v] [net_http faraday httparty excon] +# bundle exec ruby test/test_proxy_headers.rb [-v] [net_http faraday httparty excon typhoeus] require 'json' require 'uri' @@ -139,6 +139,29 @@ def test_httparty(verbose:) { ok: false, error: e.message } end +def test_typhoeus(verbose:) + require 'ruby_proxy_headers/typhoeus' + + proxy_url = env_proxy_url + raise 'Set PROXY_URL' unless proxy_url + + test_url = ENV.fetch('TEST_URL', 'https://api.ipify.org?format=json') + proxy_header = ENV.fetch('PROXY_HEADER', 'X-ProxyMesh-IP') + h = send_headers_from_env + + response = RubyProxyHeaders::Typhoeus.get( + test_url, + proxy: normalized_proxy_url(proxy_url), + proxy_headers: h + ) + return { ok: false, error: "HTTP #{response.code}" } unless response.code.to_i.between?(200, 299) + + val = find_header(response.proxy_response_headers, proxy_header) + assert_proxy_header(val, proxy_header, verbose: verbose, label: 'typhoeus') +rescue StandardError => e + { ok: false, error: e.message } +end + def test_excon(verbose:) require 'ruby_proxy_headers/excon' @@ -173,7 +196,8 @@ def test_excon(verbose:) 'net_http' => method(:test_net_http), 'faraday' => method(:test_faraday), 'httparty' => method(:test_httparty), - 'excon' => method(:test_excon) + 'excon' => method(:test_excon), + 'typhoeus' => method(:test_typhoeus) }.freeze def print_help