Skip to content

About

Tools/utilities for using UW's Hyak (HPC) cluster.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

12 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Tools/Utilities for using UW's Hyak Cluster

This repository contains tools for using UW's Hyak (HPC) Cluster.

General Hyak Information

The official Hyak documentation contains instructions for most basic use cases (submitting jobs, examining allocations, etc.).

This Repository

This repository contains a few useful items for Hyak:

  • The config/ directory contains (optional) configuration files that can be useful on Hyak.
  • The scripts/ directory contains BASH scripts. These should be already installed in the bin directory in the nbenlab's /gscratch space. If you use the bash_profile in the config/ directory, the commands should be available to you automatically.

Hyak SSH Setup

SSH is the preferred method of connecting to and interacting with Hyak. I recommend the following setup. In your ~/.ssh/config file, make sure the following block appears (make sure to copy indentation):

Host hyak
  HostName klone.hyak.uw.edu
  User <YOUR-UW-NETID>

Be sure to replace the <YOUR-UW-NETID> with your actual NetID. For more information see this page about the ssh-config file format.

If you need to forward a port from a worker node back to the computer you're using to connect to Hyak, you can add another line at the end:

Host hyak
  HostName klone.hyak.uw.edu
  User <YOUR-UW-NETID>
  LocalForward 8989 /mmfs1/home/<YOUR-UW-NETID>/.hyak-tunnels/connection.sock

The final line connects the port 8989 on your local computer to the unix domain socket file in the ~/.hyak-tunnels/connection.sock file. You'll need to create the ~/.hyak-tunnels directory. When using Jupyter or similar tools that need a port, you can direct them to open a unix domain socket as ${HOME}/.hyak-tunnels/connection.sock instead.

The nbenlab Hyak Account

The nbenlab has a GPU slice in Hyak with partition name gpu-l40s that contains the following resources:

  • 32 CPUs
  • 2 GPUs
  • 364GB or RAM

These resources are reserved for the lab, but other users can use them when we're not actively using them. When running commands like salloc, sbatch, and srun for nbenlab projects, make sure to set the account to nbenlab and the partition to gpu-l40s (e.g., salloc -A nbenlab -p gpu-l40s ...)

The lab also has 1TB of storage in /gscratch/nbenlab. For temporary storage, please use /gscratch/scrubbed (e.g., /gscratch/scrubbed/nbenlab or /gscratch/scrubbed/<YOUR-NETID>). If you need to use a /gscratch/scrubbed directory that doesn't exist you should have permission to create it, but if you create the /gscratch/scrubbed/nbenlab directory, please make sure the group for this directory is nbenlab, that the permissions allow group read-write-execute, that and the group sticky bit is set:

mkdir -p /gscratch/scrubbed/nbenlab
chgrp nbenlab /gscratch/scrubbed/nbenlab
chmod 775 /gscratch/scrubbed/nbenlab
chmod g+S /gscratch/scrubbed/nbenlab

Using Apptainer

Hyak severely limits the number of files you are allowed to store in your home directory or in the /gscratch directories. To get around this limitation, and to be able to run jobs in containerized installation environments, please use apptainer. Apptainer is a Docker-like tool that runs as your user-ID instead of as root and thus ha different security requirements. It is often used on HPC systems like Hyak where users can't run Docker due to root access problems.

The Hyak documentation on apptainer covers its use on Hyak specifically.

About

Tools/utilities for using UW's Hyak (HPC) cluster.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages