Description
When running de-id on the GAE, snakemake processes use several gigabytes of memory. It also seems to gradually increase as it runs, suggesting some kind of memory leak. Are we keeping handles to the data from each open file perhaps?
Mitigation for now is to set SNAKEMAKE_CORES=1 so at least there's only one such process. Although it's not clear to me if it's the main snakemake process or the ones it spawns per job (or both) that are responsible.
Definition of Done
Description
When running de-id on the GAE, snakemake processes use several gigabytes of memory. It also seems to gradually increase as it runs, suggesting some kind of memory leak. Are we keeping handles to the data from each open file perhaps?
Mitigation for now is to set SNAKEMAKE_CORES=1 so at least there's only one such process. Although it's not clear to me if it's the main snakemake process or the ones it spawns per job (or both) that are responsible.
Definition of Done