Skip to content

feat(server): add periodic background cleanup for storage temp staging, upload chunks, and recycle bin (#249, #248) - #253

Open
allenporter wants to merge 7 commits into
mainfrom
fix/storage-cleanup-249
Open

allenporter wants to merge 7 commits into
mainfrom
fix/storage-cleanup-249

Conversation

@allenporter

Copy link
Copy Markdown
Owner

Description

Resolves #249 and #248 by introducing scheduled background garbage collection for orphaned storage temp staging files, abandoned multi-part upload chunks, and recycle bin entries, as well as fixing cascading soft-deletes and purge logic.

Problem

  1. Orphaned .tmp staging files: LocalBlobStorage.put() stages writes in <storage_root>/temp/<hex>.tmp before an atomic rename. Server crashes (SIGKILL, host restarts, OOM) orphaned these staging files permanently.
  2. Abandoned upload chunks: Multi-part uploads store parts as <object_name>.part.<N> in the user data bucket, only deleting them upon final chunk merge. Abandoned/interrupted uploads leave partial chunks behind indefinitely.
  3. Recycle bin permanent delete & cascading soft-delete: Folder soft-deletion did not cascade to descendants, and purge_recycle only dropped the index row without deleting the underlying UserFileDO records or blob data.
  4. Dead code: FileService.temp_dir and the storage_root constructor parameter were unused.

Solution

  • Storage Cleanup (supernote/server/services/storage_cleanup.py):
    • Added BlobStorage.cleanup_staging() and BlobStorage.cleanup_chunks() with path traversal protection.
    • Multipart chunks are evaluated by grouping by base object and checking max(mtime) of the session to ensure in-flight active uploads are never pruned prematurely.
    • StorageCleanupService executes an immediate startup sweep (ttl=0) to clear crash debris before starting the periodic background polling loop.
  • Recycle Bin Cleanup (supernote/server/services/recycle_cleanup.py):
    • Fixed VFS.delete_node() to recursively soft-delete child nodes under folders.
    • Fixed VFS.purge_recycle() to delete underlying file records and blob storage content.
    • Added automated background retention-based purge job and manual admin trigger endpoint /api/admin/recycle/cleanup.
  • Architectural Standards & Conformance:
    • Removed dead storage_root parameter and unused temp_dir from FileService.
    • Encapsulated chunk wire pattern parsing in supernote/server/utils/paths.py.
    • Injected primitive configuration dependencies explicitly into cleanup services.
    • Enforced strict bidirectional 1:1 test parity across all modules.

Verification

  • ./script/lint: All 9 pre-commit hooks passed cleanly with 0 diagnostics.
  • tests/test_init.py: 1:1 bidirectional structure conformance check passed with an empty allowlist.
  • ./script/test: 676 tests passed, 54 snapshots passed, 0 failures.

…d chunks, and recycle bin (#249) (#248)

- Add StorageCleanupService for periodic pruning of orphaned staging .tmp files and abandoned multipart upload chunks
- Add startup sweep for staging temp debris left from server crashes
- Add RecycleBinCleanupService for periodic automated recycle bin purging
- Fix VFS soft delete recursion for folder hierarchies
- Fix VFS recycle purge to delete underlying UserFileDO rows and blob content
- Decouple chunk wire parsing into server utils/paths
- Remove unused FileService.temp_dir and dead storage_root parameter
- Add configuration settings, Prometheus counters, and admin manual trigger endpoints
- Enforce strict 1:1 test parity across all modified/new server modules
@codecov-commenter

codecov-commenter commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.73882% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.19%. Comparing base (874869e) to head (f3f6fe5).

Files with missing lines Patch % Lines
supernote/server/services/blob.py 81.81% 16 Missing ⚠️
supernote/server/services/storage_cleanup.py 89.15% 9 Missing ⚠️
supernote/server/services/vfs.py 96.69% 4 Missing ⚠️
supernote/server/services/file.py 93.87% 3 Missing ⚠️
supernote/server/services/recycle_cleanup.py 97.84% 2 Missing ⚠️
supernote/server/services/processor.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #253      +/-   ##
==========================================
+ Coverage   90.98%   91.19%   +0.21%     
==========================================
  Files         114      116       +2     
  Lines       11809    12338     +529     
==========================================
+ Hits        10744    11252     +508     
- Misses       1065     1086      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add periodic cleanup for orphaned storage/temp staging files and abandoned upload chunks

2 participants