PBS Pro v2022.1.3
Portable Batch System (or simply PBS) is the name of computer software that performs job scheduling. Its primary task is to allocate computational tasks, i.e., batch jobs, among the available computing resources.PBS is a distributed workload management system. PBS handles the management and monitoring of the computational workload on a set of one or more computers.
Please note that the login nodes are NOT to be used for long jobs. Any processes running on the login node would be killed by admins. To test, debug and to interactively run any programs, please request for a interactive job.
Basic PBS Commands:
JOB CONTROL- qsub is used to submit an executable script to a batch server.
The script is a shell script which will be executed by a command shell such as
sh or csh. e.g.
qsub -P cc myjobscript
It will return job_ID.hn1.hpc.iitd.ac.in i.e. sequence-number.servername. job_ID is the job identifer which can be further used for any actions involving the job, such as checking job status, modifying the job, tracking the job, or deleting the job. Here, cc is your department identifier, and myjobscript is a textfile conting instructions for the job. some useful optionsqsub -q queue jobscript # submit to specified queue -N jobname # set job name (default: name of jobscript) -l resources=value # request for required resoures # (select,ncpus,ngpus,nmics,walltime,place etc.) -l select=n # request n number of nodes/slots -l ncpus=n # request n number of cpus on a node -l ngpus=n # request n number of gpus per node -l place=scatter # scatter the proceses across nodes -l place=pack # places all the chunks (select) on the same node -l host=node name # run a job on a particular node -P project Name # Specify the name of the project -o Output # for specifying the pbs output file name -e Error # for specifying the pbs error file name -M email # full email address for notification e.g. qsub -N test -lselect=20:ncpus=24 -P cc -o out -e err myjobscript #Will request for 20 node with 24 cpus each under project "cc" with output file name "out" and error file name as "err" e.g. qsub -N test -lselect=2:ncpus=24:ngpus=2 -P cc -o out -e err myjobscript #Will request for 2 node with 24 cpus each with 2 mic cards on each node e.g qsub -N test -lselect=4:ncpus=24:nmics=2 -P cc -o out -e err myjobscript #Will request for 4 node with 24 cpus each with 2 mic cards on each node e.g qsub -N test -lselect=4:ncpus=12 -P cc -l place=scatter myjobscript #Will request for 4 node with 12 cpus each and will scatter the jobs across 4 nodes equally. e.g qsub -N test -lselect=1:ncpus=24:host=cn001 -P cc myjobscript #Will request for 1 node with 24 cpus each but only run on node cn001
NOTE :
Interactive Jobs: If you want to run a job interactively use "-I".Will provide you the shell on the compute nodes to run the job interactively.e.g. qsub -I -P cc -N test -lselect=2:ncpus=24
- qstat is used to view job and job step information for jobs managed by PBS. The default view includes only minimal information.
some useful options qstat -Q # show all queues -Q queue # show status of specified queue -f -Q queue # show full info for specified queue -q # show all queues (alternative format) -q queue # show status of specified queue (alt.) qstat -a # show all jobs -a queue # show all jobs in specified queue -f job_ID # show full info for specified job -n # show all jobs and the nodes they occupy -T # show the estimated starting time of the Job (It is calculated in every six hours.) -x # show recently finished jobs (Job history is preserved for past 24 hours.)
e.g. qstat -ans job_ID # to check the details of the job qstat -u $USER # to check the job submited by the user qstat -T job_ID # shows the estimated starting time of the Job
NOTE:
-Presently there are two queues for users i.e. "standard" which is the default queue, and a low priority queue "low" -Jobs status can be:
"Q": Queued. Job is queued and will start when all resource requirements are met.
"R" : Running. Job is currently running.
"F" : Finished (Accesible via qstat -x)
"H" : Held (Job is held and will NOT run)
Details of the state can be seen viaqstat -awns jobid
orqstat -f jobid
- qdel deletes jobs in the order in which their job identifiers are presented to the command. A job is deleted by sending a Delete Job batch request to the batch server that owns the job. A job that has been deleted is no longer subject to management by batch services.
A batch job may be deleted by its owner, the batch operator, or the batch administrator.
e.g. qdel job_ID # to delete a job
- qalter alters one or more PBS batch jobs. The attributes listed with the options to the qalter command can be modified. If any of the modifications to a job fails, none of the jobs attributes is modified.
e.g. qalter -l ncpus=4 job_ID # change the no. of ncpus for the requsted job qalter -l select=1:ncpus=4:mem=512mb job_ID
NOTE :
-If a job is running, the only resources that can be modified are cput (CPU time) and walltime. -If a job is queued, any resource mentioned in the options to the qalter command can be modified, but requested modifications must fit within the limits set at the server and queue for the amount of each resource allocated for queued jobs. -If a requested modification does not fit within these limits, the modification is rejected. - qhold requests that a server place one or more holds on a job. A job that has a hold is not eligible for execution. Supported holds: USER , OTHER (also known as operator), SYSTEM, and bad password.
A user may place a USER hold upon any job the user owns. An operator, who is a user with operator privilege, may place either a USER hold or an OTHER hold on any job. The batch administrator may place any hold on any job.
The p option can only be set by root or admin user via qhold -h p. The owning user can release with qrls -h p or query by qselect -h p.
e.g. qhold job_ID # place the job on hold
- qrls removes or releases holds which exist on batch jobs.A user may always remove a USER hold on their own jobs, but only privileged users can remove OTHER or SYSTEM holds. An attempt to release a hold for which the user does not have the correct privilege is an error and no holds will be released for that job.
e.g. qrls job_ID # release user hold on the job
NOTE :
-The alloc manager hook sets a "user hold" on the jobse.g. qrls $(qselect -s H -u userid -q low) # users can release their job themselves
- qrerun directs that the specified jobs are to be rerun if possible. To rerun a job is to terminate the session leader of the job and return the job to the queued state in the execution queue in which the job currently resides.
If a job is marked as not rerunable then the rerun request will fail for that job.
e.g. qrerun job_ID # re-run the job
- qrun is used to force a batch server to initiate the execution of a batch job. The job is run regardless of scheduling position or resource requirements.
e.g. qrun job_ID # run the job
NOTE :
-In order to execute qrun, the user must have PBS Operation or Manager privileges. - tracejob utility extracts job status and job events from accounting records, MOM log files, server log files, and scheduler log files. Using it can help identify where, how, a why a job failed. This tool takes a job id as a parameter as well as arguments to specify which logs to search, how far into the past to search, and other conditions.
e.g tracejob -n 10 job_ID #look for the job from last 10 days logs
- Dependancy Jobs
PBS allows you to specify dependencies between two or more jobs. Dependencies are useful for a variety of tasks, such as:
- Specifying the order in which jobs in a set should execute
- Requesting a job run only if an error occurs in another job
- Holding jobs until a particular job starts or completes execution
Dependency Definition after Execute current job after listed jobs have begun afterok Execute current job after listed jobs have terminated without error afternotok Execute current job after listed jobs have terminated with an error afterany Execute current job after listed jobs have terminated for any reason before Listed jobs can be run after current job begins execution beforeok Listed jobs can be run after current job terminates without error beforenotok Listed jobs can be run after current job terminates with an error beforeany Listed jobs can be run after current job terminates for any reason
The construct of a dependency i "-W depend=dependency expression".e.g qsub -W depend=afterok:Job_ID jobscript e.g qsub -W depend=after:Job_ID jobscript e.g qsub -W depend=before:Job_ID jobscript
- qorder
changes the order of the jobs within a queue.The order two jobs is to exchange the job's position.
e.g qorder Job_ID1 Job_ID2 #Will switch the positon of the Jobs.
e.g. qdel -W force job_id # to delete a job forcefully
- Upper time limit: 168 hours (7 days)
- The entire cluster can be subscribed for a maximum of 24 hours
- A fraction of the cluster can be subscribed for a maximum of 24/fraction hours. e.g. half the cluster can be requested for 48 hours, quarter of the cluster can be requested for 96 hours etc.
Examples of resource selection:
- 24 CPU cores per node :
-l select=2:ncpus=24
- 2 GPUs per node :
-l select=2:ngpus=2
- 2 Xeon Phi Per node :
-l select=2:nmics=2
- High memory (505GB):
-l select=1:highmem=1
NOTE: "highmem" flag has been deprecated. Users can use "mem" and "place=excl" flag.
- Old K20 GPU:
-l select=1:ngpus=2:K20GPU=true
-l select=1:mem=30gb
Typical submission script
In a blank text file (e.g. pbsbatch.sh) add the following:
"#PBS" is NOT a comment! #PBS is an instruction to PBS.
#!/bin/sh ### Set the job name (for your reference) #PBS -N Your_job_name ### Set the project name, your department code by default #PBS -P cc ### Request email when job begins and ends #PBS -m bea ### Specify email address to use for notification. #PBS -M $USER@iitd.ac.in #### #PBS -l select=n:ncpus=m ### Specify "wallclock time" required for this job, hhh:mm:ss #PBS -l walltime=01:00:00 #PBS -l software=replace_with_Your_software_name # After job starts, must goto working directory. # $PBS_O_WORKDIR is the directory from where the job is fired. echo "===============================" echo $PBS_JOBID cat $PBS_NODEFILE echo "===============================" cd $PBS_O_WORKDIR #job time -p mpirun -n {n*m} executable #NOTE # The job line is an example : users need to change it to suit their applications # The PBS select statement picks n nodes each having m free processors # OpenMPI needs more options such as $PBS_NODEFILE
submit using:
$ qsub pbsbatch.sh
Interactive job using department (project) cc
2 nodes, 24 procs each:
$ qsub -P cc -I -l select=2:ncpus=24
2 nodes, 24 procs each, 2 mpitasks per node:
$ qsub -P cc -I -l select=2:ncpus=24:mpiprocs=2
2 nodes, 2 gpus each, 2 cpus each (default) for 6 hours:
$ qsub -P cc -I -l select=2:ngpus=2 -l walltime=6:00:00
2 nodes, 2 gpus each, 12 cpus each (explicit):
$ qsub -P cc -l select=2:ngpus=2:ncpus=12