HPC DOWNTIME UPDATES
Dear All, Access to HPC will be tentatively released approximately at 5pm today. The following are the major changes: 1) File system has been changed. HOME: 1.2PB (total); Default home quota is now 100GB, expandable to 2TB as before. SCRATCH: 4.5PB (total); Default scratch quota is now 25TB. This can be increased on a case-to-case basis on approval by the HPC Steering Committee. All active HPC users' HOME data has been transferred. All scratch data of users having less than 5TB of scratch usage has been transferred to the new file system. Rest all users would need to transfer data from the previous scratch file system (/scratchsfa) to the new file system. The "serial" queue is reserved for jobs for transfer of data - the job will utilize "low" queue budget, with limit of 1 job of 4 cpu per user and maximum walltime 168:00:00. In case the data is not fully transferred in one week, the process needs to be repeated. Users can transfer the whole old scratch data to new scratch (limit of 25TB) by following instructions on the HPC website (https://supercomputing.iitd.ac.in).Users can transfer data out of old scratch from klogin1.hpc.iitd.ac.in or klogin2.hpc.iitd.ac.in via rsync/scp/sftp etc. Old scratch is available on /scratchsfa (environment variable $SCRATCHSFA). We recommend using rsync wherever possible. User who have more than 25TB of SCRATCH data need to transfer only the required data to the current file system.
All production nodes in the standard/low/high and other queues except the serial queue only have the new file system. The facility of serial queue would be available till 31st October 2023. Thereafter, old scratch file system will not be available. All users should ensure that all HOME and SCRATCH data has been successfully transferred before 31st October 2023. 2) Jobs : all queued jobs have been put on user hold. Users can release jobs as required using qrlsafter confirming the job working directory and required files. 3) nodes: All Icelake, Skylake are available for use. Most Haswell nodes are also available. School of AI nodes: 3 nodes are available. There is an additional flag "interlink" available to choose the "nvlink" based nodes or the "pci" based node. e.g. select=1:ncpus=2:ngpus=2:interlink=nvlink:centos=amdepyc n.b. node names have been changed to scai0[1-3].
List of users having data more than 5TB or above in SCRATCH
Please click on this link to see the list:Users_List
To migrate data from old storage to new storage, Please copy the PBS script using below command and submit the job with your departmental code
cp /home/apps/skeleton/migration_pbs.sh $HOME
qsub -P [Your departmental code] migration_pbs.sh