[jehrr]

Problem

On 2026-10-15, --mode transfers read 20 rows. The page had no next link, and ?page=2 returned the same rows as page 1. The run still reported pages_completed: 2, status: complete.

I measured the listing that day with free requests: Playwright, system Chromium, no proxy, no key.

• The default view is filtered, not capped. /statistik/neuestetransfers applies the site's own market-value floor. Its filter form and tab links carry minMarktwert=500000. That day 20 transfers matched, which is less than one 25-row page. So the page had no pager and no rel=next. That is the whole filtered listing.
• The unfiltered view paginates normally. /transfers/neuestetransfers/statistik?land_id=0&verein_land_id=0&wettbewerb_id=alle&minMarktwert=0&maxMarktwert=500000000 had 4 pages: 25, 25, 25 and 2 rows, 77 transfers. Each page has a plain &page=N next link that keeps the filters. Pages 1 to 3 share no transfer ids.
• A page past the end returns the last page again, with HTTP 200.
  • ?page=2 of the default view served page 1.
  • ?page=5, ?page=401 and ?page=9999 of the unfiltered view all served page 4. The pager still marked page 4 as current.
• A second problem showed up during the live checks. Twice in one minute the edge answered with a 155-byte nginx 502 Bad Gateway.
  • The first time it replaced page 1 after the AWS WAF challenge had cleared. The challenge's status is discarded there, as it must be. So nothing but the body showed the failure, and the page classified as empty.
  • The run went on to pages 2 to 5 and reported 52 of 77 rows as complete.

Resulting behaviour

• Repeated pages are not counted. A later page whose rows were all seen on earlier pages is never counted as completed, and its rows are dropped as duplicates. What it means depends on the page before it:

  • No next link on the page before: the listing ended. stop_reason: listing_ended, exit 0. The sidecar gets repeated_page, repeat_of_page and listing_pages.
  • A next link on the page before: the site promised a page and served an old one. stop_reason: page_repeated, exit 6, status: partial.

• Where the rule applies:

  • all four entry points (Playwright, pyppeteer, Selenium, Scraper API);
  • Playwright's --concurrency path. It decides afterwards in page order (page_flow.settle_repeats), and it stops dispatching once a page brings nothing new.

• Next links come from the HTML. Each page's next link is read from the served HTML by a shared helper (product_parser.next_page_link), so every engine gives the same page the same answer.

• A bare nginx 5xx page is a failed page, never an empty one, in every engine, with or without a status.

• Docs:

  • README and TROUBLESHOOTING describe the default filter (dated), the unfiltered URL and the two new stop reasons.
  • CHANGELOG [Unreleased] has a behaviour-change note and two Fixed entries.

• Canary: added a --mode transfers --pages 3 run on the unfiltered URL. It turns red unless:

  • the exit code is 0;
  • at least 2 pages were served and more than 25 rows came back;
  • the status is complete and stop_reason is completed or listing_ended;
  • transfer ids are unique and rows are in page order;
  • the core fields are filled.

  It runs even when the market-values checks failed. Its artifacts are uploaded.

Evidence

Live runs, 2026-10-15 (Playwright, system Chromium, no proxy, no key):

Run	Exit	Result
--mode transfers --pages 3 (default view)	0	listing_ended, 1 page counted, 20 rows, repeated_page: 2, repeat_of_page: 1
--url <unfiltered> --pages 6	0	listing_ended, 4 pages, 77 unique rows, repeated_page: 5, repeat_of_page: 4
the same with --concurrency 2	0	same 77 transfer ids in the same order
before the 502 fix, same unfiltered run	0	page 1 a 502, 52 rows, status: complete

Offline suite: PYTHONDONTWRITEBYTECODE=1, __pycache__ cleared before each run.

Venv	Passed	Failed
core only	723	0
playwright	772	0
pyppeteer	756	0
selenium	754	0

ci_checks.py --all, pytest and py_compile are all OK.

New fixtures are real captures from that day, cut to the grid's first rows plus the pager, the head links and the keys div:

• the default view, page 1 and the ?page=2 repeat;
• unfiltered pages 1 and 2, the last page, and a page past the end;
• the 502.

New checks:

• value assertions on the captures: names, ids, ages, a EUR 2,000,000 disclosed fee, an undisclosed fee kept null, next links;
• the PageSequence rule on its own;
• each entry point's real scrape() fed the captures: the one-page listing, the 3-page listing with a past-the-end page, and a planted case where page 1 links onward but page 2 serves page 1;
• a source check that every engine's _fetch_one_page records the next link;
• the 502 marker matches none of the served, 404 or WAF captures. It also matched none of 63 local captures except the two 502s.

Planted-fault controls. Each one turned the suite red on the named checks, and the suite was green again once restored:

Planted fault	Failed checks
no repeat detection	13
every repeat read as the end	4 (planted case in Scraper API, Playwright, Playwright --concurrency)
listing_ended not complete	7
Selenium counts the repeated page	3
pyppeteer drops the next-link line	1 (source check)
Playwright --concurrency skips settle_repeats	3 (no_new_products, 3 pages counted)
Scraper API ignores page 1's next link	1
502 body read as empty	2 (incl. rc=0 complete, the old behaviour)

Not changed

• The default --mode transfers URL is still the site's own default view, with its EUR 500k floor. Switching the default to the unfiltered listing would change what the mode means, so that is left to the owner.
• A 5xx with a visible status already classified as blocked (exit 3 on page 1). A body-only 5xx now gets the same classification. A separate "server error" state with its own retry policy is not added here.
• No version bump, tag or release.

Second commit: --min-value and a site 5xx is a server error (owner decision)

--mode transfers --min-value EUR (0 = no floor) is available in all four entry points.

• What it reads: the filtered path /transfers/neuestetransfers/statistik?land_id=0&verein_land_id=0&wettbewerb_id=alle&minMarktwert=EUR&maxMarktwert=500000000. Any filters a typed --url already carries are kept.
• Without the flag: the site's own default view, as before.
• Rejected with exit 2: a negative or non-integer value, or the flag with another --mode.
• Sidecar: records min_market_value_eur and market_value_filter. The filter is url, or site_default when the floor is the site's own and not in the URL.
• Canary: the transfers step now passes --min-value 0 instead of a hand-built URL, and checks the recorded floor.

A target-site 5xx is a server error, not a block. This covers an HTTP 5xx, and the edge's 155-byte nginx 502 page when no status is visible.

• Retry: the page is fetched again on the same exit up to 2 times, --retry-delay x1 then x2 apart. A retry does not rotate the proxy, does not buy a solve and does not reset the page's single solve allowance.
• Then: the page fails with stop_reason: target_http_5xx, exit 5 on page 1 and exit 6 on a later page.
• Where: all three browser engines and the Scraper API client.
• Exceptions: a 5xx whose body names a challenge vendor still goes through the challenge path. A 4xx is still a refusal.

Evidence for this commit

• Real-code runs: each entry point's real scrape() and real _fetch_one_page ran against a scripted site serving the real 502 capture. Only the browser or the Scraper API call was replaced (fake page objects for Playwright, pyppeteer and Selenium).

  • Two 502s, then served: exit 0, page 1 fetched 3 times.
  • 502 on every try: exit 5, target_http_5xx, 3 fetches.
  • Page 2 a 502 on every try: exit 6, pages_failed: [2].

• Parse checks of --min-value in all four entry points: 0, 500000, -1, x, and the flag with market-values.

• Sidecar checks for --min-value 0 and for the default view.

• Live run (2026-10-16), --mode transfers --min-value 0 --pages 6: exit 0, listing_ended, 4 pages, 78 rows, min_market_value_eur: 0. The default view was exit 0, 20 rows, market_value_filter: site_default.

• Offline suite: no failures in any venv.

  Venv	Passed
  core	749
  playwright	807
  pyppeteer	791
  selenium	791

  ci_checks --all and pytest also pass.

• Planted-fault controls: each turned the suite red on the named checks, and the suite was green again once restored.

  Planted fault	Failed checks
  502 body read as blocked	9
  5xx status read as blocked	1
  PageAttempts never repeats	4
  Selenium does not retry a 5xx	3
  pyppeteer does not fail a 5xx page	2
  Playwright counts a 5xx as blocked	2
  failure_reason forgets 5xx	2
  Scraper API does not retry a 5xx	3
  Selenium ignores --min-value	3
  Scraper API omits the filter from the sidecar	2
  --min-value unbounded (pyppeteer)	2

The "Not changed" items about the default URL and 5xx-as-blocked above are now resolved by this commit.

🤖 Generated with Claude Code