Skip to content

KU Community Cluster Operating System Upgrade

Date of Upgrade

Starting at 8:00 AM CST on January 15, 2025 and ending on 8:00 AM CST January 17, 2025, CRC will be upgrading the OS on the KU Community Cluster from CentOS 7 to Rocky 9.

This upgrade will involve updating the Operating System as well as the software packages made available to users. Previously compiled software will most likely not work on the new OS, and will need to be recompiled.

Minor or no changes

  • All user files will remain unchanged
  • Scheduler, partitions, time limits will stay the same
  • Changing node names for easier management on our part. This only effects you if you request a specific node by name or exclude a specific node by name which we don't recommend
  • The majority of software modules previously available will remain available, sometimes with version changes

Signficant changes

  • Anaconda introduced licensing restrictions to their conda repo.
    • We have migrated our conda to miniforge which uses the conda-forge repo by default.
    • We recommend you reinstall any conda environments you have
  • The path /panfs/pfs.local will no longer be available. Replace with /kuhpc.
  • Significant changes to system software have occurred, the Standard Environment modules are now:
    • compiler/gcc/11.4 replaces compiler/gcc/8.3
    • openmpi/5.0 replaces openmpi/4.0
  • If you have software you compiled yourself, it will likely not not run on the new system, and will need to be recompiled
  • PGI compiler is now NVC (compiler/nvc/24). NVIDIA merged them in 2020
  • AMD CPUs and GPUs mixed into sixhour. Need to specify intel/amd or nvidia/amdgpu constraint if you wish to run on specific hardware
  • Older C6320 which have 24 cores will not have Infiniband hooked up. The ConnectX-3 card (56 Gb/s) is not supported by latest NVIDIA IB software.

Test Cluster

We have set up a test cluster to be used before January 15 to test your code and recompile any software you may have. Owners have very graciously allowed us to borrow a variety of nodes with different hardware to have a good mix of nodes to test your jobs on.

You may run your processing on the login nodes if you desire. The 60 minute CPU time limit has been removed for the test cluster.

Access

All current cluster users may access the test cluster at any time. You can use any of the current connecting methods to connect to the test cluster as well.

Login Nodes

  • login1.hpc.crc.ku.edu
  • login2.hpc.crc.ku.edu

Hardware and Features

Features are requested under the --constraint option. Because the cluster is consortium of hardware, attributes allow the user to specify which type of node they wish to use (e.g. ib, edr_ib, intel)

Node Cores Memory GB CPU GPU/SSDs Features
mi226 48 192 AMD EPYC 9454P 48-Core Processor 3 x MI210 64GB avx512_only,avx2,avx512,hdr100_ib,ib,amd,mi210,amdgpu
cre24 28 256 Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz avx2_only,avx2,noib,10gb,intel
sjm21 24 96 Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz avx512_only,avx2,avx512,intel,noib
cho01 48 384 Intel(R) Xeon(R) Gold 6240R CPU @ 2.40GHz avx512_only,avx2,avx512,intel,edr_ib,ib
sho00 40 192 Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz 2 x V100 16GB 1 x A100 40GB avx512_only,avx2,avx512,intel,edr_ib,ib,a100,v100,double,nvidia
jac00 32 128 Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz avx512_only,avx2,avx512,ib,intel
bio12 48 128 Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz 4 x A40 48GB avx512_only,avx2,avx512,intel,noib,a40,single,nvidia
big55 56 256 Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz avx512_only,avx2,avx512,intel,hdr100_ib,ib
tho00 48 256 Intel(R) Xeon(R) Gold 6442Y avx512_only,avx2,avx512,intel,hdr100_ib,ib

Partitions

There are 2 partitions set up for testing. The 30min partition has higher priority than sixhour, meaning jobs, if submitted at the same time, the 30min job will start first.

Partition Time Limit Nodes Restrictions
30min 30 minutes All N/A
sixhour 6 hours All Limited to 100 cores per job

Software

We installed the most used software from the KU Community Cluster. If you do not see the software that you need, please send a software request and make sure to say in the additional information section it is for the Rocky 9.

Anaconda License Restrictions

While Anaconda does state that the imposed licensing restrictions do not effect accredited universities, we are still taking precautions with this OS upgrade to move away from using the Anaconda channel. We suggest all users re-create their conda environments using the new conda/latest module which uses the conda-forge channel.

You should not notice any change in using the conda/latest module from the old one.

  • conda, the command, is the same
  • You can still use the bioconda channel
  • Your previous conda environments will still work with the new miniforge install, but we do recommend to recreate them