apache/stormcrawler

http.basicauth credentials are attached to every request, whatever the host

#2089

· rzo1 opened

on Aug 29, 2026

A response header named set-header becomes a request header directive

#2090

· rzo1 opened

on Aug 29, 2026

Response headers can populate the reserved verbatim request and response metadata keys

#2091

· rzo1 opened

on Aug 29, 2026

CharsetIdentification retries the meta charset lookup ten bytes at a time by recursing, and the call sits outside JSoupParserBolt's catch block

#2092

· rzo1 opened

on Aug 29, 2026

AbstractLLMTextExtractor pastes page HTML into the prompt unescaped and returns the reply unchecked

#2093

· rzo1 opened

on Aug 29, 2026

Playwright browser context stops validating certificates as soon as a proxy is configured

#2094

· rzo1 opened

on Aug 29, 2026

Playwright protocol applies neither the IP filter nor the URL filters to the requests it makes

#2095

· rzo1 opened

on Aug 29, 2026

Tika ParserBolt puts no limit on the extracted text and no time limit on the parse

#2097

· rzo1 opened

on Aug 29, 2026

URLFrontier channels are always plaintext and the module has no TLS option

#2098

· rzo1 opened

on Aug 29, 2026

Response headers named metrics.* are fed straight into the fetcher metric registry, and a non-numeric one fails the fetch

#2099

· rzo1 opened

on Aug 29, 2026

Gzipped sitemap files are expanded with no size bound, and SiteMapParserBolt reads Content-Type under a key that is never present

#2100

· rzo1 opened

on Aug 29, 2026

HostURLFilter memoizes the source host in plain fields shared by all fetcher threads

#2101

· rzo1 opened

on Aug 29, 2026

SCProxy.toString() rebuilds the proxy URL with user and password, and the protocol logs it

#2102

· rzo1 opened

on Aug 29, 2026

OpenSearch addresses default to http and Basic credentials are registered for any host

#2103

· rzo1 opened

on Aug 29, 2026

[Improvement] Add configuration to stop parsing PDFs after X pages

enhancement

#1901

· jnioche opened

on May 8, 2026

Scope credential headers in http.custom.headers to hosts

#2162

· rzo1 opened

on Sep 15, 2026

Navigation Menu

Uh oh!

4.0.0

http.basicauth credentials are attached to every request, whatever the host

A response header named set-header becomes a request header directive

Response headers can populate the reserved verbatim request and response metadata keys

CharsetIdentification retries the meta charset lookup ten bytes at a time by recursing, and the call sits outside JSoupParserBolt's catch block

AbstractLLMTextExtractor pastes page HTML into the prompt unescaped and returns the reply unchecked

Playwright browser context stops validating certificates as soon as a proxy is configured

Playwright protocol applies neither the IP filter nor the URL filters to the requests it makes

Tika ParserBolt puts no limit on the extracted text and no time limit on the parse

URLFrontier channels are always plaintext and the module has no TLS option

Response headers named metrics.* are fed straight into the fetcher metric registry, and a non-numeric one fails the fetch

Gzipped sitemap files are expanded with no size bound, and SiteMapParserBolt reads Content-Type under a key that is never present

HostURLFilter memoizes the source host in plain fields shared by all fetcher threads

SCProxy.toString() rebuilds the proxy URL with user and password, and the protocol logs it

OpenSearch addresses default to http and Basic credentials are registered for any host

#2134 fetcher.thread.timeout: cancel the okhttp call, bounded helper pool for other protocols

#315 Adaptative URL filter to normalize URLs based on canonical tag

#2092 Bound the meta charset look-ahead and catch charset detection failures in JSoupParserBolt

[Improvement] Add configuration to stop parsing PDFs after X pages

#2094 Make ignoring HTTPS errors in the Playwright protocol configurable and independent of the proxy

Scope credential headers in http.custom.headers to hosts

#2089 Send http.basicauth credentials only to the hosts in http.basicauth.hosts

#2162 Send credential headers from http.custom.headers only to the hosts in http.custom.headers.hosts

#2095 Apply the IP address filter to the requests made by the Playwright browser

Uh oh!

List view