Last modified: April 19 2017.

hpchelp@iitd.ac.in

Software

For list of available software, please check available modules:
$ module avail
If the required software is not available/listed in modules, users can
  • Install the software in your own account ($HOME) -or-
  • Request installation.

Installing Software in your own account

User can install software in their own accounts. Super user access cannot be provided for any installation. Please check modules before requesting any dependencies.

Requesting installation of software

If multiple users need access to a software, the supervisor/Head/HPC represnetative can request central installation of software. Please send email(s) to hpchelp.

ANSYS FLUENT

Steps for preparing and submitting your job

  1. Case and Data files
  2. Transfer your case and data files to your /home or /scratch directory. If you are generating large data (> 10GB), please use /scratch.
  3. Journal File e.g. journal.jou
  4. A "journal file" is needed to execute fluent commands in the batch mode. e.g.
    rcd case_and_data
    /solve/dual-time-iterate 240000 50
    wcd output
    exit ok
    
    This journal file will read case_and_data.cas and case_and_data.dat. it will runa dual time iteration with 50 iteration per time-step for 240000 time-steps. It will write the output in "output".
  5. PBS submit file. e.g. pbssubmit.sh
  6. #!/bin/bash
    #PBS -N jobname
    #PBS -P department
    #PBS -m bea
    #PBS -M $USER@iitd.ac.in
    #PBS -l select=1:ncpus=20
    #PBS -o stdout_file
    #PBS -e stderr_file
    #PBS -l walltime=168:00:00
    #PBS -l fluent=1
    #PBS -l fluent_hpc=4
    #PBS -l software=ANSYS
    cd $PBS_O_WORKDIR
    module load apps/fluent
    #default version is 15.0.7 
    
    module load compiler/mpi/openmpi/1.10.0/gnu
    
    time -p fluent -g 2ddp -t $NCPUS -i journal.jou -ssh -mpi=openmpi -cnf=$PBS_NODEFILE -pinfiniband &> log
    
    The example submit file will request 20 cpus for 168 hours and run a "2ddp" job over infiniband interconnect using the journal file "journal.jou". It will use 1 base fluent license (fluent=1) and 4 HPC licenses (fluent_hpc=4). The base license allows a 16 process parallel job.
    Following summarizes the fluent command line arguments:
    • 2ddp: Run a 2D simulation in double precision(remove dp to run in single precesion)
    • -g: Run Fluent without the GUI
    • -ssh: Use ssh to login to the available nodes(-rsh is default)
    • -pinfiniband: Use infiniband interconnect
    • -cnf=$PBS_NODEFILE: the list of nodes allocated to the PBS job is also provided to fluent solver.
    • &> : Redirect the Flueent output & error information to "log" file.
    • -mpi=openmpi: specifies the type of MPI (intel,mpich2,openmpi are currently supported on IITD HPC). Please load appropriate mpi module as per specified option.
  7. Job submission
  8. Submit the above job using "qsub pbssubmit.sh"

Checklist

NOTE:

There are some compatibility issues with fluent 17.2 & openmpi (-mpi=openmpi). Please carry out following modifications in PBS job script's command section:
apps/Fluent/17.2/precompiled
time -p fluent -g 2ddp -t $NCPUS -i journal.jou -ssh -mpi=intel -cnf=$PBS_NODEFILE -pinfiniband &> log

MATLAB

Steps for preparing and submitting your job

You can load matlab
module via:
module load apps/matlab
for short jobs/runs you can run matlab gui via
matlab
command. long running jobs should be submitted via batch system a script file, example:
#!/bin/bash
#PBS -N jobname
#PBS -P department
#PBS -m bea
#PBS -M $USER@iitd.ac.in
#PBS -l select=1:ncpus=8
#PBS -o stdout_file
#PBS -e stderr_file
#PBS -l walltime=168:00:00
#PBS -l matlab=1
#PBS -l software=MATLAB
cd $PBS_O_WORKDIR
module load apps/matlab
time -p matlab -nosplash -nodisplay < myprogram.m > matlab.log
now submit the job using qsub command.

GROMACS

Steps for preparing and submitting your job

You can check for available gromacs modules via
 module avail apps/gromacs
------------------------------- /home/soft/modules ---------------------------------------
apps/gromacs/4.6.2         apps/gromacs/4.6.2_plumed  apps/gromacs/4.6.5_plumed  apps/gromacs5.1.1
apps/gromacs/4.6.2_noncuda apps/gromacs/4.6.5         apps/gromacs/5.1.1
You can load gromacs version of your choice via (e. gromacs 4.6.2):
module load apps/gromacs/4.6.2
This loads & sets all the prerequisites for gromacs 4.6.2(cuda). For short test runs you can run (say)
mdrun_mpi
gromacs command, from gpu login nodes. Long running jobs should be submitted via batch system using a script file, example:
#!/bin/bash
#PBS -N jobname
#PBS -P department
#PBS -m bea
#PBS -M $USER@iitd.ac.in
#PBS -l select=4:ncpus=24:mpiprocs=2:ngpus=2
#PBS -o stdout_file
#PBS -e stderr_file
#PBS -l walltime=24:00:00
#PBS -l software=GROMACS
echo "==============================="
echo $PBS_JOBID
cat $PBS_NODEFILE
echo "==============================="
cd $PBS_O_WORKDIR


module load apps/gromacs/4.6.2
time -p mpirun -np $PBS_NTASKS -machinefile $PBS_NODEFILE -genv OMP_NUM_THREADS 12 mdrun_mpi < gromacs specific input files & parameters >
now submit the job using
qsub
command.
This script will request for 4 nodes with 2 gpu cards per node(total 8 gpu cards) & 96 cpu cores (24 core per node).
This script will also launch 8 process in total (2 process per node) on the requested nodes, also each process creates 12 threads (-genv OMP_NUM_THREADS 12) , hence launching 24 threads per node.