Skip to content

Add S3-compatible storage source - #2919

Closed
kurob1993 wants to merge 2 commits into
PixelPlayerHQ:masterfrom
kurob1993:feature/s3-storage-source
Closed

kurob1993 wants to merge 2 commits into
PixelPlayerHQ:masterfrom
kurob1993:feature/s3-storage-source

Conversation

@kurob1993

Copy link
Copy Markdown

Re-opened from a new fork: #2916 was auto-closed when the source fork was detached from the network. Same commits, nothing changed.

Summary

Adds a new streaming source: S3-compatible object storage (Cloudeka, Cloudflare R2, MinIO, AWS S3, etc.), built entirely on the existing OkHttp stack — no new dependencies.

  • Auth: AWS SigV4 signing implemented from scratch (S3SigV4), validated against AWS test vectors in plain-JVM unit tests. Credentials stored in EncryptedSharedPreferences.
  • Listing: ListObjectsV2 with pagination; users pick bucket prefixes (folders) to sync.
  • Playback: presigned GET URLs (path-style, 1h TTL) streamed through the existing CloudStreamProxy with a 55-minute resolved-URL cache. Presigned URLs are exempt from the private-IP block (signature marker), private endpoints still protected.
  • Folder sync: ListObjectsV2 results are parsed (artist/title from Artist - Title.ext filenames, album from top-level prefix) and mirrored into the unified library.
  • Background metadata queue: new S3MetadataWorker (WorkManager, unique, network-constrained) probes embedded tags via presigned range fetch + MediaMetadataRetriever — title, artist, album, duration and album artwork — 4 files in parallel, so the initial sync stays fast. Live progress banner on the dashboard shows remaining files.
  • Library: SourceType.S3, DB v45 (s3_songs, s3_folders tables + album_art_uri / metadata_probed columns). Queue snapshots keep original cloud URIs so playback survives proxy restarts; stale 127.0.0.1 entries are dropped on restore.
  • UI: login form (endpoint/bucket/keys/region/prefix), folder dashboard with edit connection, per-folder re-sync, sync-all, logout. Full ID (id/en) strings.

Testing

  • S3SigV4Test: 6 tests against official AWS SigV4 test vectors (header + presigned) on plain JVM.
  • Manually verified end-to-end against a Cloudeka S3 bucket: folder listing, sync, background tag probe (title/artist/album/artwork), playback through the stream proxy, queue restore across app restarts.

🤖 Generated with Claude Code

- Streaming from any S3-compatible endpoint (Cloudeka etc.) using
  ListObjectsV2 + SigV4 signing and presigned GET URLs (path-style)
- Local stream proxy with cached resolved URLs (55 min) and 1h presign TTL
- Folder-based sync: pick bucket prefixes, sync to Room (DB v45),
  mirrored into the unified library
- Background metadata queue (S3MetadataWorker): embedded tags, duration
  and album artwork probed via presigned range fetch after sync,
  4 files in parallel; live progress banner on the dashboard
- Editable saved connection, logout, per-folder re-sync
- Queue snapshots keep original cloud URIs so playback survives
  proxy restarts; stale 127.0.0.1 entries are dropped on restore
- SigV4 unit tests against AWS test vectors (plain JVM)
- CloudStreamProxy: dedicated upstream client with 15s connect /
  30s read timeouts instead of the shared 8s API client; slow clouds
  (Cloudeka) routinely exceeded 8s to first byte, especially while
  the metadata probe queue was running
- resolveCloudUri: retry once after 400ms before falling back —
  proxy warm-up races and network blips previously left a raw
  s3:// URI for ExoPlayer and failed with "unknown protocol"
- Log each S3 resolve failure path (offline / proxy not ready /
  null resolution) for easier diagnosis
@kurob1993

Copy link
Copy Markdown
Author

Superseded by #2921 (moved to the okuru-id org fork).

@kurob1993 kurob1993 closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant