Skip to content

fix(python): show and move top-level downloads left in staging after a restart - #826

Open
JAAdventurer wants to merge 2 commits into
nitrobass24:developfrom
JAAdventurer:fix/staged-downloads-after-restart
Open

JAAdventurer wants to merge 2 commits into
nitrobass24:developfrom
JAAdventurer:fix/staged-downloads-after-restart

Conversation

@JAAdventurer

@JAAdventurer JAAdventurer commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With staging on, a top-level download left in staging is lost track of after a restart.

Why

  • While a download sits in staging (finished but not yet moved, or stopped partway), it stays visible only through pending_completion, which keeps the active scanner on it. That set lives in memory.
  • After a restart it's empty, and the local scan only covers local_path, so:
    • A finished but unmoved download is still recorded in downloaded_file_names with no local copy, so it shows as Deleted and is never moved.
    • A stopped partial shows no local size.

Fix

  • When a pair's context is built with staging on, pending_completion is seeded from the top-level entries in that pair's staging dir. lftp's <name>.lftp temp file counts as <name>, and <name>.lftp-pget-status is skipped.
  • The names are also handed to the active scanner at that point, so its first scan covers them. Without that, the first remote scan could arrive first, and the row would briefly show Deleted, which drops the name from pending_completion.
  • From there the existing flow takes over. A finished download goes DOWNLOADED and gets its move. A stopped partial shows its staged size until it's re-queued or deleted.

Persist authority waits for a remote listing (found in the live test)

  • _check_persist_authority read a missing remote size as "deleted remotely", so a persisted name with any local copy was marked Downloaded.
  • Before the first remote scan, though, the size is only unknown. Every top-level job is persisted when it ends, Stops included. So once the seeding made a stopped partial visible at startup, it read as Downloaded in that window and was moved to local_path as if complete.
  • ModelBuilder now skips persist authority until it has received a remote listing. Auto-queue only acts on rows with a remote size, so it's unaffected in that window.
  • Side effect: with staging off, persisted downloads show as Default for the few seconds before the first remote scan, then Downloaded as before.

Skipped, and why

Contract notes

  • No changes to the persist format, the web API, or the lftp set stream.
  • Behaviour: after a restart, downloads left in staging show their staged size, and finished ones are moved. Needs a CHANGELOG "Fixed" line at release.
  • Tests: the "remote absent" cases in test_build_state_persist_with_auto_delete_remote now set an empty remote listing. Before, they set none, which no longer means "absent".
  • Integration with feat: move nested downloads out of staging #824: once both are in, a top-level folder whose staging copy holds only nested downloads mustn't be seeded, or its staging copy would hide its final-folder copy. feat: move nested downloads out of staging #824 handles that when it picks this change up.

Test plan

  • uv run ruff check . and uv run ruff format --check .
  • uv run pyright
  • pytest tests/unittests: new TestStagedTopLevelNames and wiring tests (pending_completion seeded, scanner handed the names, nothing with staging off) and test_persist_authority_waits_for_a_remote_listing fail before their fixes
  • pytest tests/integration (only test_extract fails locally: no archive tools installed; CI covers it)
  • Live restart on a real seedbox (run with feat: move nested downloads out of staging #824 merged in):
    • A stopped top-level partial stays in staging and shows its size. Before the persist-authority fix, it was moved to the final dir.
    • A finished but unmoved top-level download, put back into staging while the container was stopped, is moved after the restart.

…a restart

pending_completion, which keeps the active scanner on a download in
staging until it moves, is in memory only. After a restart the local scan
(local_path only) couldn't see what was left in staging: a finished but
unmoved download, recorded as downloaded with no local copy, showed as
Deleted and was never moved; a stopped partial showed no size.

Seed pending_completion from the pair's staging dir when its context is
built (lftp's "<name>.lftp" temp file counts as <name>), and hand the
names to the active scanner then so its first scan covers them, ahead of
the first remote scan.
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 22 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 23b8b8c and d0e57ce.

Files selected for processing: 6

  • src/python/controller/controller.py
  • src/python/controller/model_builder.py
  • src/python/controller/pair_context.py
  • src/python/tests/unittests/test_controller/test_controller_lftp_wiring.py
  • src/python/tests/unittests/test_controller/test_model_builder.py
  • src/python/tests/unittests/test_controller/test_pair_context.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…sting

_check_persist_authority read a missing remote size as "deleted remotely"
and marked a persisted name with any local copy Downloaded. Before the
first remote scan the size is only unknown. With downloads left in
staging now visible right after a restart, a stopped top-level partial
(persisted when its job ended) read as Downloaded in that window and was
moved to local_path as if complete. Skip persist authority until the
builder has had a remote listing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant