Tidy up GAE space

[MonikaSvata]

Description

Tidy up GAE space

Definition of Done

• Review the current stages of what is happening in GAE and how important is data there
• Agree the hierarchy of the order in which files can be deleted
• There is monitoring that check the amount of space used by compressed HL7 files or how much space is free.
• Implement the deletions as appropriate

Dependencies

Comments

[thompson318]

Overlap with #49. Which will enable us to monitor for low disk space.

[jeremyestein]

Thoughts on deletion hierarchy:

Data	Size	Replaceability	Comment	Suggested deletion policy
compressed HL7	Large	Cannot be replaced		Never delete unless under duress. If disk gets low, try to find more disk, or delete old stuff first if we have confidence that our uploaded data will not need to be re-processed. Archives are indexed by time and bed, but ALL variables are mixed together, so targeted deletion is only possible by time and bed.
original CSV	Large but see #15	Can be reprocessed from HL7	can be useful for debugging	Delete after 30 days or if disk gets low; consider targeted deletion (see #79)
original parquet	Small	Can be reprocessed from CSV	parquets are generated in a single step from CSV	Delete after 30 days or if disk gets low; consider targeted deletion (see #79)
pseduonymised parquet	Small	Can be reprocessed from CSV	parquets are generated in a single step from CSV	Delete after 30 days or if disk gets low; consider targeted deletion (see #79)
pre-upload tar files	Small	Is an intermediate file for upload	Not needed except for (short term) debugging	Delete after 30 days or if disk gets low; consider deleting immediately after upload

Retention times for file types that are only useful for debugging could be shortened when we have more confidence in the pipeline.

[skeating]

That seems reasonable to start with. Agree we can reassess once the pipeline is fully functional.

[thompson318]

Seems reasonable to me. I agree that once we get up and running we should shorten retention times for all but the hl7.

[MonikaSvata]

Turn on and leave in the dry run mode to see how much gets deleted.