Current Version

While only parts of Freesurfer 6.0.0 are parallelized, we have found optimal performance within a single node using 28 cores. A basic overview of a Freesurfer workload can be found here: https://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall

These steps have been modified for use on SeaWulf with a sample batch submission script available in /gpfs/projects/samples/freesurfer.

FreeSurfer Examples

First make a new directory and copy the example Slurm script and data file from the samples folder:

module load freesurfer/6.0.0

cd /gpfs/projects/samples/freesurfer_samples
mkdir -p $HOME/freesurfer_example
cp * $HOME/freesurfer_example && cd $_

Then load the Slurm module and submit the job script to the scheduler:

module load slurm
sbatch freesurfer.slurm

The submission script, freesurfer.slurm, converts the sample file sample-001.mgz to nifti format and performs a full recon-all on the file.

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=28
#SBATCH --time=48:00:00
#SBATCH --job-name freesurfer_example2
#SBATCH -p long-28core

module load freesurfer/6.0.0

cd $HOME/freesurfer_example

export SUBJECTS_DIR=$PWD

mri_convert sample-001.mgz sample-001.nii.gz

recon-all -openmp 28 -i sample-001.nii.gz -s bert -all > out.txt

After completion a directory called bert/mri and bert/surf should have been created containing the results. Along with these directories a file, out.txt, should contain a record of the job.

Visualizing the Results

After completion the results can be visualized through an interactive job and the freeview  command:

module load slurm

salloc -N 1 --ntasks-per-node=28 --time=08:00:00 -p long-28core

ssh -X $SLURM_NODELIST

module load freesurfer/6.0.0

cd $HOME/freesurfer_example

freeview -v \
    bert/mri/T1.mgz \
    bert/mri/wm.mgz \
    bert/mri/brainmask.mgz \
    bert/mri/aseg.mgz:colormap=lut:opacity=0.2 \
    -f \
    bert/surf/lh.white:edgecolor=blue \
    bert/surf/lh.pial:edgecolor=red \
    bert/surf/rh.white:edgecolor=blue \
    bert/surf/rh.pial:edgecolor=red

SUBMIT A TICKET