Computation node¶

How to send jobs to CALMODULIN¶

We will connect via SSH as follows:

$ ssh username@calmodulin.lc.ehu.es
$ ssh username@u036898.lc.ehu.es

What will log us into the system:

 ____  _        ____                            _   _
| __ )(_) ___  / ___|___  _ __ ___  _ __  _   _| |_(_)_ __   __ _
|  _ \| |/ _ \| |   / _ \| '_ ` _ \| '_ \| | | | __| | '_ \ / _` |
| |_) | | (_) | |__| (_) | | | | | | |_) | |_| | |_| | | | | (_| |
|____/|_|\___/ \____\___/|_| |_| |_| .__/ \__,_|\__|_|_| |_|\__, |
                                   |_|                       |___/
Unveiling the secrets of life through data

Welcome aritzleo

 Date: Wed Sep 23 23:51:55 CEST 2026

 Hostname:     calmodulin
 CPU Model:    Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz

 Number of procs:     48
 Total Memory:        790680476 kB
 Free Memory:         784220620 kB
 Available Memory:    784071288 kB

 cluster related issues --> aritz.leonardo@ehu.eus

Specifications¶

Compute Node	# nodes	Processor	# of cores	memory (GB)	Accelerator
u036898	1	Intel Xeon Gold 6240R	48	800	1x NVIDIA A40

Working Spaces¶

Ideally, each researcher would create a user directory in each of these spaces:

Role	mount point	Size
scratch	/scratch	3.8 TB
almacenamiento	/bigdisk	7.3 TB

Use /scratch partition for calculations (SSD) and /bigdisk (HDD) to store final calculations.

Sending Jobs¶

The resource manager is SLURM and you can find more information on its use in this page.

QoS and partitions¶

QoS/Partition	Priority	MaxWall	MaxNodesPU	MaxJobsPU	MaxSubmitPU	MaxTRES
compute	-	INFINITY	1

The columns mean the following:

MaxWall: Maximum time that a job can run.
MaxNodesPU: The maximum number of nodes that a job can request.

Batch scripts¶

Calmodulin: Job MPI

#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=1
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load program/program_version

srun binario < input

Calmodulin: Job OpenMP

#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=48
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load program/program_version

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

binary < input

Calmodulin: Job híbrido (MPI+OpenMP)

#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=4
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=12
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load program/program_version

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun binary < input

Calmodulin: Job with 1 GPU

#!/bin/bash
#SBATCH --partition=compute
#SBATCH --job-name=JOB_NAME
#SBATCH --cpus-per-task=1
#SBATCH --gres=gpu:nvidia_a40:1
#SBATCH --mem=200gb
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err

module load program/program_version

srun binary < input

Toolchains¶

A toolchain is a coherent grouping of a series of tools that can be used to: compile, install or for the runtime of an application.

Its definition allows delimiting a work environment at the level of versions, compilers and libraries.

The description of the table indicates which are the components that we would load to the environment if we loaded an associated module to a toolchain:

Toolchain	Description
foss	GCC, OpenMPI, OpenBLAS/LAPACK, ScaLAPACK(/BLACS), FFTW
fosscuda	GCC, OpenMPI, CUDA, OpenBLAS/LAPACK, ScaLAPACK(/BLACS), FFTW
gompi	GCC, OpenMPI
gompic	GCC, OpenMPI, CUDA
gcccuda	GCC, CUDA
intel	icc, ifort, imkl, impi
iimpi	icc, ifort, impi

Foe example, if we loaded foss/2021a:

$ module load foss/2025b

We would see that the follwing modules have been loaded:

[aritzleo@calmodulin ~]$ module list

Currently Loaded Modules:
  1) GCCcore/14.3.0                (H)   7) libxml2/2.14.3-GCCcore-14.3.0      (H)  13) libfabric/2.1.0-GCCcore-14.3.0 (H)  19) FlexiBLAS/3.4.5-GCC-14.3.0     (H)
  2) zlib/1.3.1-GCCcore-14.3.0     (H)   8) libpciaccess/0.18.1-GCCcore-14.3.0 (H)  14) PMIx/5.0.8-GCCcore-14.3.0      (H)  20) FFTW/3.3.10-GCC-14.3.0
  3) binutils/2.44-GCCcore-14.3.0  (H)   9) hwloc/2.12.1-GCCcore-14.3.0        (H)  15) PRRTE/3.0.11-GCCcore-14.3.0    (H)  21) gompi/2025b                    (H)
  4) GCC/14.3.0                         10) OpenSSL/3                          (H)  16) UCC/1.4.4-GCCcore-14.3.0       (H)  22) FFTW.MPI/3.3.10-gompi-2025b    (H)
  5) numactl/2.0.19-GCCcore-14.3.0 (H)  11) libevent/2.1.12-GCCcore-14.3.0     (H)  17) OpenMPI/5.0.8-GCC-14.3.0            23) ScaLAPACK/2.2.2-gompi-2025b-fb (H)
  6) XZ/5.8.1-GCCcore-14.3.0       (H)  12) UCX/1.19.0-GCCcore-14.3.0          (H)  18) OpenBLAS/0.3.30-GCC-14.3.0     (H)  24) foss/2025b

  Where:
   H:  Hidden Module

As we can see, among other dependencies, the environment has been loaded with:

A compiler: GCCcore/14.3.0, GCC/14.3.0
A particular implementatio of MPI: OpenMPI/5.0.8-GCC-14.3.0
Scientific libraries: OpenBLAS/0.3.30-GCC-14.3.0, OpenBLAS/0.3.30-GCC-14.3.0, ScaLAPACK/2.2.2-gompi-2025b-fb, FFTW/3.3.10-gompi-2025b

The other modules are, as a general rule, dependencies of these main modules.

Also note that some toolchains may have another as a dependency. In this case gompi/2025b would be a subtoolchain of foss/2025b.

Compilers¶

Generally, in multipurpose computers with x86_64 architecture, two families of compilers are used:

GNU: the open source compilers of the GNU initiative.
Intel: Intel’s proprietary (and free as of 2021) compilers. They are especially interesting on platforms with Intel processors, since they generate optimized machine code for their own processors.

Here’s a list of how to call these compilers.

Compiler	C	FORTRAN	C++	MPI C	MPI FORTRAN	MPI C++
GNU	gcc	gfortran	g++	mpicc	mpif90	mpicxx
intel	icc	ifort	icpc	mpiicc	mpiifort	mpiicpc

Software¶

GROMACS¶

Version	CPU	GPU
GROMACS/2025.4-foss-2025b-CUDA-12.9.1	✔	✔

Here you can find documentation associated with the GROMACS versions to launch jobs efficiently. It’s interesting given that the 2016 and 2021 versions have different release options.

Versión	Documentación
2025	https://manual.gromacs.org/documentation/2025.4/index.html

Calmodulin: GROMACS with GPU

#!/bin/bash
#SBATCH --job-name=gmx-test
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=12
#SBATCH --gres=gpu:nvidia_a40:1
#SBATCH --gres-flags=enforce-binding
#SBATCH --mem=16G
#SBATCH --time=01:00:00
#SBATCH --output=gmx-%j.out
#SBATCH --error=gmx-%j.err

module purge
module load GROMACS/2025.4-foss-2025b-CUDA-12.9.1

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

echo "Host: $(hostname)"
echo "Job ID: $SLURM_JOB_ID"
echo "CPUs: $SLURM_CPUS_PER_TASK"
echo "GPU: $CUDA_VISIBLE_DEVICES"

gmx mdrun \
    -deffnm md \
    -ntmpi 1 \
    -ntomp $OMP_NUM_THREADS \
    -nb gpu \
    -pme gpu

In the case of GROMACS, what usually works best is choosing a balanced number of processes and threads. Here we show a table with cases in which all 48 cores of the machine (that is, the entire machine) or 24 cores (that is, half machine).

Number of total cores	Processes MPI	Threads per process
48	8 (`--ntasks-per-node=8`)	6 (`--cpus-per-task=6`)
48	6 (`--ntasks-per-node=6`)	8 (`--cpus-per-task=8`)
48	12 (`--ntasks-per-node=12`)	4 (`--cpus-per-task=4`)
48	4 (`--ntasks-per-node=4`)	12 (`--cpus-per-task=12`)
24	6 (`--ntasks-per-node=6`)	4 (`--cpus-per-task=4`)
24	4 (`--ntasks-per-node=4`)	6 (`--cpus-per-task=6`)

As we can see, the product of the number of MPI processes and the number of threads per process is equal to the number of cores that we wish to use.

This is how the script would look for the first of the cases that we show in the table:

It is also convenient to use the environment variables that SLURM makes available to us:

environment variable	Value
SLURM_CPUS_PER_TASK	Threads per process. Equals the amount of `--cpus-per-task=` in the batch script.
SLURM_NTASKS_PER_NODE	MPI processes. Equals the number of `--ntasks-per-node=` in the batch script.

SCHRÖDINGER¶

Schrodinger suite is available. This is a special software that comunicates with SLURM and respects the queue system although it has a particular way of sending jobs. Please read carefully the script below and note that sbatch ... does not have be invoked when sending scripts.

Calmodulin: SCHRÖDINGER w/o gpu

#!/bin/bash

set -e

# ----------------------------------------------------------
# Desmond launcher for Calmodulin
# All jobs are submitted through:
#   Schrodinger Job Server -> Slurm
#
# Usage: (NOTE THERE IS NO sbatch calling)
# SEND_SCHRODINGER.sh JOBNAME input.cms protocol.msj config.cfg
# ----------------------------------------------------------

if [ "$#" -ne 4 ]; then
    echo "Usage:"
    echo "  desmond-slurm JOBNAME input.cms protocol.msj config.cfg"
    exit 1
fi

JOBNAME="$1"
CMS="$2"
MSJ="$3"
CFG="$4"

# Load Schrodinger
if command -v module >/dev/null 2>&1; then
    module load Schrodinger/2026-3
fi

if [ -z "${SCHRODINGER:-}" ]; then
    echo "ERROR: Schrodinger/2026-3 is not loaded."
    exit 1
fi

# Ensure that this user is registered with the central Job Server
if ! "$SCHRODINGER/jsc" cert verify calmodulin:8030 >/dev/null 2>&1; then
    echo
    echo "ERROR: You are not registered with the Calmodulin Job Server."
    echo
    echo "Run once:"
    echo "  \$SCHRODINGER/jsc cert get calmodulin:8030"
    echo
    exit 1
fi

# Check inputs
for f in "$CMS" "$MSJ" "$CFG"; do
    if [ ! -f "$f" ]; then
        echo "ERROR: File not found: $f"
        exit 1
    fi
done

echo "Submitting Desmond through:"
echo "  Main job : cpu:1  -> Slurm"
echo "  MD job   : gpu    -> Slurm/A40"
echo

exec "$SCHRODINGER/utilities/multisim" \
    -JOBNAME "$JOBNAME" \
    -HOST cpu:1 \
    -SUBHOST gpu \
    -m "$MSJ" \
    -c "$CFG" \
    "$CMS"

System manager¶

Aritz Leonardo