Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `RubyProxyHeaders::Typhoeus` sends proxy headers with libcurl `CURLOPT_PROXYHEADER` (`:proxyheader` and `headeropt: :separate` on Typhoeus 1.7 / Ethon 0.19) and exposes the CONNECT response, including `X-ProxyMesh-IP`, as `proxy_response_headers`. Those headers are parsed from the first block of `response_headers`. Typhoeus's `response.headers` stays the origin response.

### Changed

- Development dependencies are now `typhoeus` `~> 1.7` and `ethon` `~> 0.19`. The gem still has no runtime dependency on Typhoeus; applications that use the adapter should add `gem 'typhoeus', '~> 1.7'`.

### Fixed

- HTTPS requests through `RubyProxyHeaders::Typhoeus` no longer call `RubyProxyHeaders::NetHTTP.request`, which is not defined.

## [0.1.0] - 2024-02-28

### Added
Expand Down
10 changes: 1 addition & 9 deletions DEFERRED.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,6 @@
# Deferred / not fully supported

## Typhoeus / Ethon

**Status:** Not implemented in this gem.

[Ethon](https://github.com/typhoeus/ethon) does not expose `CURLOPT_PROXYHEADER` (or equivalent) in its `Ethon::Easy` option mapping as of ethon 0.18.x. Adding CONNECT response header capture would require Ethon/libcurl changes or a custom C extension.

**Workaround:** Use `Net::HTTP` + {RubyProxyHeaders::NetHTTP.patch!} or Faraday with `ruby_proxy_headers_net_http` adapter.

---
Typhoeus is supported. Ethon 0.19 exposes `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt`, and `RubyProxyHeaders::Typhoeus` uses them. See [docs/typhoeus.md](docs/typhoeus.md).

## Mechanize

Expand Down
13 changes: 7 additions & 6 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ GEM
csv (3.3.6)
diff-lcs (1.6.2)
domain_name (0.6.20240107)
ethon (0.18.0)
ethon (0.19.1)
ffi (>= 1.15.0)
logger
excon (1.6.0)
Expand Down Expand Up @@ -139,8 +139,8 @@ GEM
rubyntlm (0.6.5)
base64
timeout (0.6.1)
typhoeus (1.6.0)
ethon (>= 0.18.0)
typhoeus (1.7.1)
ethon (>= 0.19.1, < 0.20.0)
unicode-display_width (3.3.0)
unicode-emoji (~> 4.3)
unicode-emoji (4.3.0)
Expand All @@ -166,6 +166,7 @@ PLATFORMS

DEPENDENCIES
bundler (>= 2.4)
ethon (~> 0.19)
excon (~> 1.4)
faraday (~> 2.14)
faraday-net_http (~> 3.4)
Expand All @@ -175,7 +176,7 @@ DEPENDENCIES
rspec (~> 3.12)
rubocop (~> 1.50)
ruby-proxy-headers!
typhoeus (~> 1.6)
typhoeus (~> 1.7)
webmock (~> 3.18)
yard (~> 0.9)

Expand All @@ -189,7 +190,7 @@ CHECKSUMS
csv (3.3.6) sha256=aba61e7e507a66f03d45cb1f3c4b6359861c3504038b422962875dce099e4456
diff-lcs (1.6.2) sha256=9ae0d2cba7d4df3075fe8cd8602a8604993efc0dfa934cff568969efb1909962
domain_name (0.6.20240107) sha256=5f693b2215708476517479bf2b3802e49068ad82167bcd2286f899536a17d933
ethon (0.18.0) sha256=b598afc9f30448cb068b850714b7d6948e941476095d04f90a4ac65b8d6efcb2
ethon (0.19.1) sha256=f309dec67cfdefaf96b3d1ee0ed4f186eac518f131340f607bb3df3b4e13ddc4
excon (1.6.0) sha256=ebe2ab83c055ef8e117c62c91a3535b966a59a64868188219d49ab90cd83b65f
faraday (2.15.0) sha256=3be0eff0187ed913a1b4f679e9fcfe845b542cdd4f75b76cd6181a4ae1335fff
faraday-net_http (3.5.0) sha256=2e7094c34e38754ee048cc3df47376f8657005d9375f8473bce1c3b526a22e36
Expand Down Expand Up @@ -249,7 +250,7 @@ CHECKSUMS
ruby-proxy-headers (0.2.3)
rubyntlm (0.6.5) sha256=47013402b99ae29ee93f930af51edaec8c6008556f4be25705a422b4430314f5
timeout (0.6.1) sha256=78f57368a7e7bbadec56971f78a3f5ecbcfb59b7fcbb0a3ed6ddc08a5094accb
typhoeus (1.6.0) sha256=bacc41c23e379547e29801dc235cd1699b70b955a1ba3d32b2b877aa844c331d
typhoeus (1.7.1) sha256=86dcb6d787ee08e34947c9436b24af3c4b02880dc83c466b6ca4dbfc53c2ee7f
unicode-display_width (3.3.0) sha256=4b7aa66a4b11db50f6f7e98411215cd0dd10eaecb665d36eec5bcacb3fa0b613
unicode-emoji (4.3.0) sha256=11c02fa73290378c066bb0562cd4c87d8e1b706fbbe0059ca12746a4244de8ce
uri (1.1.2) sha256=89292e98eabbef6248b0a0ed5e7e398e152f83f17560023a6d94a7f492b169fb
Expand Down
12 changes: 8 additions & 4 deletions IMPLEMENTATION_PRIORITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,13 @@ Prioritized roadmap for extension modules, aligned with [javascript-proxy-header

---

## Phase 4 — Typhoeus / Ethon (**deferred**)
## Phase 4 — Typhoeus / Ethon (**done**)

Ethon does not expose libcurl `CURLOPT_PROXYHEADER` in its option layer. See [DEFERRED.md](DEFERRED.md).
Ethon 0.19 exposes `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt`. `RubyProxyHeaders::Typhoeus` sends proxy headers on CONNECT with `headeropt: :separate` and reads `X-ProxyMesh-IP` from the first block of `response_headers`.

**File:** `lib/ruby_proxy_headers/typhoeus.rb`

Typhoeus ~> 1.7 and Ethon ~> 0.19 are development dependencies. The gem does not depend on them at runtime.

---

Expand All @@ -65,8 +69,8 @@ Uses `net-http-persistent`; needs dedicated integration. See [DEFERRED.md](DEFER

## Testing

`bundle exec ruby test/test_proxy_headers.rb` with `PROXY_URL` set. Modules: `net_http`, `faraday`, `httparty`, `excon` (excon is a smoke test; CONNECT response headers are not asserted).
`bundle exec ruby test/test_proxy_headers.rb` with `PROXY_URL` set. Modules: `net_http`, `faraday`, `httparty`, `excon`, `typhoeus` (excon is a smoke test; CONNECT response headers are not asserted). `spec/typhoeus_spec.rb` covers the Typhoeus CONNECT path against a local mock proxy.

---

*Updated: March 2026*
*Updated: October 2026*
10 changes: 5 additions & 5 deletions LIBRARY_RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ This document analyzes the Ruby HTTP / scraping stack used in [proxy-examples](h
| **HTTParty** | No — built on `Net::HTTP` | **High:** global or per-class `Net::HTTP` instances after `patch!`; or subclass `Connection` if needed |
| **Mechanize** | No — uses `Net::HTTP` / persistent connections internally | **Medium–high:** ensure Mechanize’s internal HTTP object gets the same patched behavior and header accessors |
| **Excon** | **Send:** yes — `:ssl_proxy_headers` on connection. **Read CONNECT response:** not exposed on origin `Excon::Response` | **Partial:** use `:ssl_proxy_headers` for sends; reading `X-ProxyMesh-IP` from CONNECT needs upstream Excon changes (see [DEFERRED.md](DEFERRED.md)) |
| **Typhoeus / Ethon** | Partial — libcurl has proxy options | **Medium:** map headers to `CURLOPT_PROXYHEADER` / related options (libcurl version dependent); capture CONNECT response via callbacks or debug hooks where supported |
| **Typhoeus / Ethon** | **Send and read:** `:proxyheader` and `:headeropt` since Ethon 0.19 / Typhoeus 1.7 | **Implemented:** `RubyProxyHeaders::Typhoeus` sets `headeropt: :separate` and parses the CONNECT block from `response_headers` |
| **Nokogiri** | N/A (XML/HTML parser only) | **N/A** — proxying is whatever HTTP client fetches the document (usually `Net::HTTP`) |

As with Node and Python stacks, **none** of the high-level Ruby clients expose a first-class “proxy CONNECT request/response headers” API; extensions must hook **below** the HTTP library (stdlib `Net::HTTP`), **inside** Faraday’s adapter, or **at** the libcurl / Excon layer.
As with Node and Python stacks, most high-level Ruby clients do not expose a first-class “proxy CONNECT request/response headers” API. Extensions hook **below** the HTTP library (stdlib `Net::HTTP`), **inside** Faraday’s adapter, or **at** the libcurl / Excon layer. Typhoeus 1.7 / Ethon 0.19 is used directly via `:proxyheader`.

## Connection flow (HTTPS over HTTP proxy)

Expand Down Expand Up @@ -56,9 +56,9 @@ Own implementation of proxy and TLS; does not use `Net::HTTP#connect`.

## Typhoeus / Ethon (libcurl)

libcurl can send additional headers to the proxy; exact options depend on libcurl version (e.g. proxy header lists). CONNECT response header visibility may require `CURLOPT_DEBUGFUNCTION` or version-specific features.
**Status:** Implemented for Typhoeus 1.7+ / Ethon 0.19+.

**Feasibility:** Medium — needs research per libcurl version and Ethon API surface.
Ethon maps `CURLOPT_PROXYHEADER` to `:proxyheader` and `CURLOPT_HEADEROPT` to `:headeropt` (`:unified` / `:separate`). `RubyProxyHeaders::Typhoeus` puts caller proxy headers on that list with `:separate`, so CONNECT does not also receive application headers. The CONNECT response, including `X-ProxyMesh-IP`, is the first header block of `response_headers` and is exposed as `proxy_response_headers`. There is no runtime gem dependency; applications install `typhoeus` ~> 1.7 themselves. libcurl 7.37 or newer is required.

## Nokogiri

Expand All @@ -68,4 +68,4 @@ Parsing only; no network layer.

---

*Last updated: March 2026*
*Last updated: October 2026*
29 changes: 27 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Most Ruby HTTP clients use `Net::HTTP`, Faraday, or libcurl without exposing:
1. Extra headers on the `CONNECT` request to the proxy.
2. Headers from the proxy’s `CONNECT` response (often discarded after the tunnel is established).

This library adds opt-in support for **Net::HTTP**, **Faraday** (2.x via `faraday-net_http`), **HTTParty**, and documents **Excon**’s built-in `:ssl_proxy_headers` for sends.
This library adds opt-in support for **Net::HTTP**, **Faraday** (2.x via `faraday-net_http`), **HTTParty**, and **Typhoeus** (1.7+ / Ethon 0.19+), and documents **Excon**’s built-in `:ssl_proxy_headers` for sends.

## Why teams use this

Expand All @@ -30,7 +30,7 @@ Or add to your `Gemfile`:
gem 'ruby-proxy-headers'
```

The `Net::HTTP` patch is pure Ruby. Install **faraday**, **faraday-net_http**, **httparty**, and/or **excon** when you use those integrations.
The `Net::HTTP` patch is pure Ruby. Install **faraday**, **faraday-net_http**, **httparty**, **typhoeus** (~> 1.7), and/or **excon** when you use those integrations. Typhoeus is not a runtime dependency of this gem.

## Net::HTTP

Expand Down Expand Up @@ -94,6 +94,31 @@ HTTParty.get(
puts RubyProxyHeaders.proxy_connect_response_headers['X-ProxyMesh-IP']
```

## Typhoeus

Requires [Typhoeus](https://github.com/typhoeus/typhoeus) 1.7 or newer (Ethon 0.19+, which exposes `CURLOPT_PROXYHEADER`) and libcurl 7.37 or newer. Add Typhoeus to the application Gemfile; this gem does not depend on it at runtime.

```ruby
gem 'typhoeus', '~> 1.7'
gem 'ruby-proxy-headers'
```

```ruby
require 'ruby_proxy_headers/typhoeus'

response = RubyProxyHeaders::Typhoeus.get(
'https://api.ipify.org?format=json',
proxy: ENV.fetch('PROXY_URL'),
proxy_headers: { 'X-ProxyMesh-Country' => 'US' }, # optional
headers: { 'Accept' => 'application/json' }
)

puts response.body
puts response.proxy_response_headers['x-proxymesh-ip']
```

Proxy headers are passed as libcurl's proxy header list (`:proxyheader`) with `:headeropt` set to `:separate`, so CONNECT carries those headers and application headers stay on the tunneled request. `proxy_response_headers` is the proxy's CONNECT response (the first block of `response_headers`). `response.headers` remains the origin response.

## Excon (send-only; CONNECT response headers)

Excon supports **sending** extra headers on CONNECT with `:ssl_proxy_headers`. Reading `X-ProxyMesh-IP` from the CONNECT response is **not** exposed on the origin response object — see [DEFERRED.md](DEFERRED.md).
Expand Down
14 changes: 14 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,20 @@ Choose the module that matches your HTTP library:
)
```

=== "Typhoeus"

```ruby
require 'ruby_proxy_headers/typhoeus'

response = RubyProxyHeaders::Typhoeus.get(
'https://example.com',
proxy: 'http://user:pass@proxy:8080',
proxy_headers: { 'X-Custom-Header' => 'value' }
)
```

Requires Typhoeus ~> 1.7 (Ethon ~> 0.19). Proxy header names on the response are lowercase.

### 3. Access Proxy Response Headers

```ruby
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Standard Ruby HTTP libraries support basic proxy authentication but **do not** a
| Faraday | `RubyProxyHeaders::Faraday` | ✅ |
| HTTParty | `RubyProxyHeaders::HTTParty` | ✅ |
| HTTP.rb | `RubyProxyHeaders::HTTPGem` | ✅ |
| Typhoeus | `RubyProxyHeaders::Typhoeus` | ✅ |
| Typhoeus 1.7+ | `RubyProxyHeaders::Typhoeus` | ✅ |
| Excon | `RubyProxyHeaders::Excon` | ✅ |
| RestClient | `RubyProxyHeaders::RestClient` | ✅ |

Expand Down
4 changes: 2 additions & 2 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ bundle exec rspec spec/connection_spec.rb

## Running Integration Tests

Integration tests require an actual proxy server.
Integration tests require an actual proxy server. The Typhoeus case is included. `spec/typhoeus_spec.rb` also drives Typhoeus 1.7 through a local mock proxy (CONNECT receives proxy headers only, and `X-ProxyMesh-IP` comes back on `proxy_response_headers`) and does not need `PROXY_URL`.

### Environment Variables

Expand All @@ -42,7 +42,7 @@ export SEND_PROXY_VALUE='US'
ruby test/test_proxy_headers.rb

# Run specific tests
ruby test/test_proxy_headers.rb net_http faraday
ruby test/test_proxy_headers.rb net_http faraday typhoeus

# List available tests
ruby test/test_proxy_headers.rb -l
Expand Down
58 changes: 36 additions & 22 deletions docs/typhoeus.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,41 @@
# Typhoeus

[Typhoeus](https://typhoeus.github.io/) is a fast HTTP client that wraps libcurl. The `RubyProxyHeaders::Typhoeus` module provides proxy header support.
[Typhoeus](https://typhoeus.github.io/) is an HTTP client that wraps libcurl through [Ethon](https://github.com/typhoeus/ethon). `RubyProxyHeaders::Typhoeus` sends custom proxy headers on HTTPS `CONNECT` and reads the proxy's CONNECT response.

## Installation
## Requirements

- Typhoeus 1.7 or newer (`gem 'typhoeus', '~> 1.7'`), which depends on Ethon `>= 0.19`, `< 0.20`
- Ethon 0.19 or newer, for `:proxyheader` (`CURLOPT_PROXYHEADER`) and `:headeropt` (`:unified` / `:separate`)
- libcurl 7.37 or newer (libcurl 8.x is what the mock-proxy spec runs against)

`ruby-proxy-headers` does not declare a runtime dependency on Typhoeus or Ethon. Add Typhoeus to the application Gemfile when you use this adapter. Typhoeus 1.7 has its own breaking changes; see Typhoeus's `UPGRADE.md` before upgrading an existing app.

```ruby
gem 'typhoeus'
gem 'ruby_proxy_headers'
gem 'typhoeus', '~> 1.7'
gem 'ruby-proxy-headers'
```

**Note:** Typhoeus requires libcurl to be installed on your system.
**Note:** Typhoeus requires libcurl to be installed on the system.

## Basic Usage

```ruby
require 'ruby_proxy_headers'
require 'ruby_proxy_headers/typhoeus'

response = RubyProxyHeaders::Typhoeus.get(
'https://api.ipify.org?format=json',
proxy: 'http://user:pass@proxy.example.com:8080',
proxy_headers: { 'X-ProxyMesh-Country' => 'US' }
proxy_headers: { 'X-ProxyMesh-Country' => 'US' },
headers: { 'Accept' => 'application/json' }
)

puts "Status: #{response.code}"
puts "Body: #{response.body}"
puts "Proxy IP: #{response.proxy_response_headers['x-proxymesh-ip']}"
```

`proxy_headers` are copied into a libcurl proxy-header list and `:headeropt` is set to `:separate`. On HTTPS, libcurl sends that list only on `CONNECT`. Headers passed in `:headers` go to the origin and are not copied onto the CONNECT request.

## API Reference

### `RubyProxyHeaders::Typhoeus.get(url, options)`
Expand All @@ -38,8 +47,9 @@ Make a GET request.
- `url` (String) - Target URL
- `options` (Hash):
- `:proxy` (String) - Proxy URL (required)
- `:proxy_headers` (Hash) - Headers to send to proxy
- `:headers` (Hash) - Request headers
- `:proxy_headers` (Hash) - Headers to send to the proxy
- `:headers` (Hash) - Headers to send to the origin
- Other Typhoeus / Ethon options (`:timeout`, `:body`, and so on) are passed through

**Returns:** `ProxyResponse`

Expand All @@ -65,36 +75,40 @@ Make a request with any HTTP method.
## Response Object

```ruby
response.code # HTTP status code
response.code # HTTP status code of the origin response
response.body # Response body
response.headers # Response headers
response.proxy_response_headers # Proxy CONNECT response headers
response.headers # Origin response headers (last header block)
response.proxy_response_headers # Proxy CONNECT response headers (lowercase names)
response.success? # True if 2xx status
```

## Performance Note
libcurl appends every header block to `response_headers`. The proxy's CONNECT response is the first block, for example:

```http
HTTP/1.1 200 Connection established
X-ProxyMesh-IP: 203.0.113.10
```

Typhoeus's `response.headers` is only the last block, the origin response, so `X-ProxyMesh-IP` is not in `response.headers`. This adapter parses that first block when its status line says `Connection established`.

Typhoeus is built on libcurl, which has native support for `CURLOPT_PROXYHEADER`. While the current implementation uses our core connection handler for HTTPS proxy headers, a future version may leverage libcurl's native support for improved performance.
## Example: Several countries

## Example: Parallel Requests
Each helper call performs one request. Repeat it to send a different proxy header:

```ruby
require 'ruby_proxy_headers'

hydra = Typhoeus::Hydra.new
require 'json'
require 'ruby_proxy_headers/typhoeus'

countries = %w[US UK DE FR JP]
results = {}

countries.each do |country|
# Note: For parallel requests, use standard Typhoeus
# Our module is for sequential requests with proxy headers
response = RubyProxyHeaders::Typhoeus.get(
'https://api.ipify.org?format=json',
proxy: ENV['PROXY_URL'],
proxy: ENV.fetch('PROXY_URL'),
proxy_headers: { 'X-ProxyMesh-Country' => country }
)

results[country] = {
ip: JSON.parse(response.body)['ip'],
proxy_ip: response.proxy_response_headers['x-proxymesh-ip']
Expand Down
Loading