(migrated from: https://gitlab.com/christophfink/flickrhistory/-/issues/12)
Implement an additional type of worker thread that downloads all images (in reduced quality and resolution?) to an S3-style storage (e.g. CSC Allas)
Needs to keep file lists to check whether images already were downloaded, and has to somehow work around the 500,000 file labels per bucket restrictions of S3 (can we append to remote tars?)
(migrated from: https://gitlab.com/christophfink/flickrhistory/-/issues/12)
Implement an additional type of worker thread that downloads all images (in reduced quality and resolution?) to an S3-style storage (e.g. CSC Allas)
Needs to keep file lists to check whether images already were downloaded, and has to somehow work around the 500,000 file labels per bucket restrictions of S3 (can we append to remote tars?)