SLURM Simulator

Automated, reproducible multi-node SLURM cluster for local HPC learning and testing, provisioned on Vagrant + VirtualBox virtual machines.

[TOC]

Overview

SLURM Simulator brings up a small SLURM (Simple Linux Utility for Resource Management) workload-manager cluster on your own machine using Vagrant-managed Ubuntu VMs. A single config.sh file drives shell scripts that install SLURM and MUNGE, generate slurm.conf, slurmdbd.conf, and /etc/hosts, and start the controller and worker daemons — giving you a working HPC scheduler sandbox without physical hardware.

What is this? A Vagrant-based provisioning setup that builds a real SLURM cluster (controller + worker nodes) inside local virtual machines.
Who is it for? Students, HPC administrators, and developers who want to learn SLURM, test job scripts, or experiment with scheduler and accounting configuration on a laptop.
What does it solve? It removes the manual, error-prone steps of installing and configuring SLURM by hand, and it needs no access to a production cluster.
When to use it: learning SLURM, practicing sbatch/srun/squeue/sacct, or testing slurm.conf changes before applying them to a real cluster.
When NOT to use it: it is a local sandbox — not a production HPC deployment, and not a discrete-event scheduler simulator.

Key features

One-command cluster bring-up with vagrant up.
Central config.sh for node names, IPs, CPUs, and memory.
Automatic generation of slurm.conf, slurmdbd.conf, and /etc/hosts.
MUNGE authentication configured automatically.
Job accounting via slurmdbd + MySQL (sacct).
Example sbatch job scripts under jobs/.

Repository layout

Path	Purpose
`slurm_ex1/`	Single-controller example with the full auto-configuration scripts
`slurm_ex2_3_nodes/`	Three-node cluster example
`slurm_template/`	Reusable template for building your own cluster
`*/slurm_config/`	`config.sh` plus generators for `slurm.conf`, `slurmdbd.conf`, and `hosts`
`*/jobs/`	Sample `sbatch` batch job scripts

Quickstart

Requirements: Vagrant and VirtualBox. The VMs run Ubuntu 22.04 (jammy).

git clone https://github.com/MSKazemi/SLURM_Simulator.git
cd SLURM_Simulator/slurm_ex1

# Edit slurm_config/config.sh to set node names, IPs, CPUs, and memory
vagrant up            # provisions the VM and installs + configures SLURM

vagrant ssh           # log into the controller node
sinfo                 # view cluster / partition state
sbatch jobs/job1.sh   # submit an example job
squeue                # check the queue

The manual, step-by-step reference below documents each command the automation runs, so you can also install and configure SLURM by hand.

MUNGE

MUNGE (MUNGE Uid 'N' Gid Emporium) is an authentication service used by SLURM for secure communication between nodes.

sudo apt update
sudo apt install -y munge libmunge-dev libmunge2

Configure MUNGE

Create MUNGE Key

sudo /usr/sbin/create-munge-key
sudo /usr/sbin/create-munge-key -f

sudo chown munge: /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

Manually Create the MUNGE Key

sudo dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
sudo chown -R munge: /etc/munge
sudo chmod 0700 /etc/munge
sudo chown munge: /etc/munge/munge.key
sudo chown munge:munge /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

Permission denied error

sudo bash -c 'dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key'
sudo chown -R munge: /etc/munge
sudo chmod 0700 /etc/munge
sudo chmod 0400 /etc/munge/munge.key

Start MUNGE

sudo systemctl start munge
sudo systemctl enable munge
sudo systemctl status munge

Logs MUNGE

sudo mkdir -p /var/log/munge
sudo chown -R munge: /var/log/munge
sudo chmod 0755 /var/log/munge

sudo mkdir -p /var/run/munge
sudo chown -R munge: /var/run/munge
sudo chmod 0755 /var/run/munge

dpkg -l | grep munge

Install SLURM

sudo apt install -y slurm-wlm

Configure SLURM

Create SLURM User

Add the all nodes name in `/etc/hosts` file.

127.0.0.1	localhost
192.168.56.10	sl1
127.0.0.1	Node1
127.0.0.1	Node2
127.0.0.1	Node3
127.0.0.1	Node4

SLURM Configuration

/etc/slurm-llnl/slurm.conf /etc/slurm/slurm.conf

sudo chown slurm: /etc/slurm-llnl/slurm.conf
sudo chmod 644 /etc/slurm-llnl/slurm.conf

sudo systemctl start slurmctld
sudo systemctl enable slurmctld
sudo systemctl start slurmd
sudo systemctl enable slurmd
sudo systemctl status slurmctld
sudo systemctl status slurmd

Verfiy the SLURM installation Configuration

ps aux | grep slurm

Set the SLURM user on all nodes.

id slurm
sudo useradd -m -s /bin/bash -u <UID> -g <GID> slurm

SLURM commands

sinfo
squeue
sinfo -N
sinfo -l
sinfo -p debug
sinfo -N -l
sinfo -N -l -o "%N %c %m %d %f %G %z"
sinfo -N -l -o "%N %c %m %d %f %G %z" | grep Node
sinfo -N -l -o "%N %c %m %d %f %G %z" | grep Node | awk '{print $1}'
sinfo -N -l -o "%N %c %m %d %f %G %z" | grep Node | awk '{print $2}'
squeue -u slurm --start -l -o "%A %j %u %T %M %D %C %R %S"
squeue -u slurm --start -l -o "%A %j %u %T %M %D %C %R %S" | grep PENDING
squeue -u slurm --start -l -o "%A %j %u %T %M %D %C %R %S" | grep RUNNING
squeue -u slurm --start -l -o "%A %j %u %T %M %D %C %R %S" | grep COMPLETED


srun --nodes=1 --ntasks=1 --partition=debug --time=00:01:00 --wait=0 --pty /bin/bash
scancel <job_id>
scancel -u <username>
scancel --state=PENDING
scancel --state=RUNNING
scancel --state=COMPLETED


scontrol reconfigure
scontrol update NodeName=Node[1-10] State=IDLE


scontrol show job <job_id>
scontrol show partition
scontrol show nodes
scontrol show config
scontrol show state
scontrol show licenses
scontrol show topology
scontrol show switches
scontrol show network
scontrol show job <job_id>
scontrol show job <job_id> | grep NodeList
scontrol show job <job_id> | grep JobState

Slurm Logs

sudo mkdir -p /var/log/
sudo touch /var/log/slurmctld.log
sudo chown slurm: /var/log/slurmctld.log

sudo tail -f /var/log/slurmctld.log
sudo tail -f /var/log/slurmd.log

Manual Node Creation

sudo scontrol create NodeName=Node1 CPUs=1 RealMemory=1048 State=IDLE

Database Daemon (slurmdbd)

sudo apt-get install slurmdbd

sudo systemctl start slurmdbd
sudo systemctl enable slurmdbd
sudo systemctl status slurmdbd

SACCT

sudo apt update
sudo apt install mysql-server

slurmdbd.conf /etc/slurm-llnl/slurmdbd.conf

sudo systemctl start mysql
sudo systemctl enable mysql

Database name: slurm_acct_db

sudo mysql -u root -p

Initialize SLURM Database Schema:

If this is the first time you’re setting up SLURM accounting, you need to initialize the database schema. SLURM provides a script, slurmdbd.sql, that can be used to create the necessary tables. your_db_password: 123456

CREATE DATABASE slurm_acct_db;
CREATE USER 'slurm'@'localhost' IDENTIFIED BY '<your_db_password>';
GRANT ALL PRIVILEGES ON slurm_acct_db.* TO 'slurm'@'localhost';
FLUSH PRIVILEGES;
EXIT;

Then, load the schema into the database:

mysql -u <your_user> -p slurm_acct_db < /usr/share/doc/slurmdbd/slurmdbd.sql

/etc/slurm/slurm.conf

/etc/slurm/slurmdbd.conf

DbdHost=localhost
DbdPort=6819
StorageHost=localhost
StorageType=accounting_storage/mysql
StorageUser=slurm
StoragePass=<your_db_password>

ALTER USER 'slurm'@'localhost' IDENTIFIED BY 'new_password';
FLUSH PRIVILEGES;
EXIT;

sudo nano /etc/slurm/slurmdbd.conf

sudo systemctl start slurmdbd
sudo systemctl enable slurmdbd
sudo systemctl status slurmdbd

sudo systemctl restart slurmdbd
sudo systemctl restart slurmctld

sudo tail -f /var/log/slurmdbd.log
sudo tail -f /var/log/slurmctld.log
sudo tail -f /var/log/slurm/slurmctld.log

sacct --starttime=2023-09-01 --endtime=2023-09-25
sacct --state=COMPLETED

sacct -o JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode,Submit,Start,End,Elapsed,MaxRSS
sacct -o JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode,Submit,Start,End,Elapsed,MaxRSS,NodeList

sacct -o JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode,Submit,Start,End,Elapsed,MaxRSS,NodeList

sacct -o JobID,JobName,Partition,Account,AllocCPUS,State,ExitCode,Submit,Start,End,Elapsed,MaxRSS,NodeList,AllocNodes,CPUTime,ReqNodes
sacct --format=JobID,JobName,Partition,State,AllocCPUS,Start,End,Elapsed,ExitCode
sacct --format=JobID,JobName,Partition,State,AllocCPUS,MaxRSS,MaxVMSize,Elapsed,ExitCode

USER: slurm

slurmd is Running as slurm

ps aux | grep slurmctld
ps aux | grep slurmdbd
ps aux | grep slurmd

For chechking the status of the slurm services

sudo systemctl status slurmctld
sudo systemctl status slurmdbd
sudo systemctl status slurmd

sudo systemctl restart slurmctld
sudo systemctl restart slurmdbd
sudo systemctl restart slurmd

sudo systemctl restart munged
sudo systemctl restart slurmdbd
sudo systemctl restart slurmctld
sudo systemctl restart slurmd

sudo systemctl start munged
sudo systemctl enable munged
sudo systemctl status munged


sudo systemctl restart slurmdbd
sudo systemctl restart slurmctld
sudo systemctl restart slurmd



sudo systemctl daemon-reload
sudo systemctl restart slurmd

sudo systemctl daemon-reload
sudo systemctl restart slurmd

Job Submission

echo "hostname" | sbatch

Legal notice and responsible use

The materials in this repository are made available without warranties to the extent permitted by applicable law. Please read the Disclaimer and Responsible-Use Notice before relying on or deploying them. This notice does not change any existing software license or limit rights that cannot legally be waived.

Name		Name	Last commit message	Last commit date
slurm_ex1		slurm_ex1
slurm_ex2_3_nodes		slurm_ex2_3_nodes
slurm_template		slurm_template
.gitignore		.gitignore
CITATION.cff		CITATION.cff
DISCLAIMER.md		DISCLAIMER.md
LICENSE		LICENSE
README.md		README.md
llms.txt		llms.txt

Navigation Menu

Repository navigation

About

Repository files navigation

SLURM Simulator

Overview

Key features

Repository layout

Quickstart

MUNGE

Configure MUNGE

Create MUNGE Key

Manually Create the MUNGE Key

Permission denied error

Start MUNGE

Logs MUNGE

Install SLURM

Configure SLURM

Create SLURM User

Add the all nodes name in `/etc/hosts` file.

SLURM Configuration

Set the SLURM user on all nodes.

SLURM commands

Slurm Logs

Manual Node Creation

Database Daemon (slurmdbd)

SACCT

Initialize SLURM Database Schema:

USER: slurm

For chechking the status of the slurm services

Job Submission

Legal notice and responsible use

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

SLURM Simulator

Overview

Key features

Repository layout

Quickstart

MUNGE

Configure MUNGE

Create MUNGE Key

Manually Create the MUNGE Key

Permission denied error

Start MUNGE

Logs MUNGE

Install SLURM

Configure SLURM

Create SLURM User

Add the all nodes name in /etc/hosts file.

SLURM Configuration

Set the SLURM user on all nodes.

SLURM commands

Slurm Logs

Manual Node Creation

Database Daemon (slurmdbd)

SACCT

Initialize SLURM Database Schema:

USER: slurm

For chechking the status of the slurm services

Job Submission

Legal notice and responsible use

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Add the all nodes name in `/etc/hosts` file.