Add checkpointing documentation page or section

[b-reyes]

It is often the case that checkpointing can be beneficial for long running workflows. We currently do not have any documentation on checkpointing. If possible, we should add a section or page about checkpointing.

[b-reyes]

Items should also be included that explain how certain jobs can be restarted from the last point. For example, when training a network, you can save the network state at certain intervals and then load that network from that state later on.

[mohalkh5]

This is a great idea and definitely something that would be valuable to include.
One thought is that checkpointing can get fairly broad depending on how deep we want to go. If we are aiming for a more detailed and comprehensive treatment, this could become quite extensive.
If that is the case, it might make sense to consider breaking this out into a separate story.

[monaghaa]

I appreciate the value of checkpointing. However, I my experience has been that the need for checkpointing applies to a small subset of our users, and they typically have very specific needs that would be difficult to document adequately in RTD. Given this, adding checkpointing documentation may not provide actionable information for most users, and therefore may not reduce UST's case/consult load.

I propose we instead we take a more minimalist approach and add an FAQ that adapts the checkpointing documentation in the Blanca section of RTD. The current text (which would be removed from the Blanca docs if moved to an FAQ) is:

Checkpointing
Given that preemptable jobs can request wall times up to 24 hours in duration, there is the possibility that users may lose results if they do not checkpoint. Checkpointing is the practice of incrementally saving computed results such that – if a job is preempted, killed, canceled or crashes – a given software package or model can continue from the most recent checkpoint in a subsequent job, rather than starting over from the beginning. For example, if a user implements hourly checkpointing and their 24 hour simulation job is preempted after 22.5 hours, they will be able to continue their simulation from the most recent checkpoint data that was written out at 22 hours, rather than starting over. Checkpointing is an application-dependent process, not something that can be automated on the system end; many popular software packages have checkpointing built in (e.g., ‘restart’ files). In summary, users of the preemptable QoS should implement checkpointing if at all possible to ensure they can pick up where they left off in the event their job is preempted.

[monaghaa]

Team discussed this and thinks it would worth having some specific examples for checkpointing. Possible outline for a page:

• Title: "Reducing Walltimes" or "Working within HPC wall clock constraints"
• Definition of checkpointing (brief)
• Example of Checkpointing itself, particularly in the context of PyTorch
• Slurm context -- how to set up job dependencies / pipelines in Slurm. Michael has a section in "job arrays" that could be adapted to this.