Summary

UPDATE: The upgrade to CentOS 7 is complete as of the May annual power downtime and Odyssey 3 is fully operational. All login nodes and compute nodes on the Odyssey cluster are running CentOS 7, have CentOS 7 modules, and the ability to use Singularity containers enabling you to build specialized software for yourself.

Please see our updated documentation, including:

CentOS 7 Transition FAQ
Running Jobs
Singularity on Odyssey 3
Module Search on Portal
Important Notice on NoMachine/NX Use

ORIGINAL INFORMATION: Over the next several months, Research Computing will be making a number of improvements to the Odyssey compute environment. This third generation of high performance computing will include increases in raw compute power, a new CentOS 7-based application stack, and the introduction of the Singularity container system. We are targeting December 2017 for the initial roll out of new hardware, and an early 2018 transition to O3 and CentOS 7. All of these changes are designed to provide a solid foundation for the amazing research being done on Odyssey.

These improvements will impact all Odyssey Slurm compute and login nodes. Virtual machines used for web applications, hybrid nodes, and storage will not be impacted.

Over the next 6 months (12/2017 - 5/2018), you can follow the progress outlined below in the Status section.

Prior to deployment of the new system, RC will host seminars to explain how to use the new application stack and Singularity containers and provide help with migrating existing tools.

Status

• Initial communication, July 2017. Completed.
• Base CentOS 7 image, September 2017. Completed.
  
• New build environment available, October 2017. Completed.
  
• Core compiler and libraries built, February 2018. Completed.
  
• Module migration list published. Completed.
• CentOS 7 re-imaging (May 23, 2018). Completed.
• CentOS 7 modules default, February 2018. Completed.
  
• New Intel Xeon Broadwell cores added, November 2017. Completed.
  
• New queues added. Completed. (details)
• Singlularity containers enabled, March 2018. Completed. Overview
  

Additional Office Hours will be held to help with the transition: 5/24 FAS, 5/29 HSPH, 5/30 FAS, 5/31 FAS, 6/5 HSPH, 6/6 FAS - See our events calendar for times and locations.

 

CentOS7 Test Space

The following information is deprecated now that Odyssey 3 is online. We have provided a test space for CentOS7 for users to test their workflows ahead of the CentOS7 upgrade.  The CentOS7 login hosts are (test host no longer available)  When you get on to those hosts you will want to use the (test cluster closed) cluster which is set up for CentOS7 testing.  We are also migrating labs to CentOS7 ahead of the large scale migration in May.  If you want to migrate your lab now please let us know.

Project details

15,000 new Intel Xeon Broadwell cores

The general compute will be refreshed with nearly 15,000 new Intel Xeon Broadwell cores. Our benchmarking shows that they are 3-4 times faster than the current AMD Abu Dhabi cores. Seriously. Very fast.

Find out more about the new queues and underlying architecture one our Running Jobs page.

CentOS 7 migration

The operating system will be updated from CentOS 6.8 to the latest release of CentOS 7.

There are several advantages to the more modern OS:

• Support to launch modern container at runtime (i.e. Singularity )
• Support for new technologies such as Xeon Phi, GPU's, 3DXpoint Memory, etc.
• Performance and power improvements
• Security enhancements
• Support for newer code bases

Module system

Much of the software on Odyssey is delivered via a LMOD module system that enables software built for CentOS 6.8. In many cases, this software can be migrated to CentOS 7 without any changes. However, some code and libraries, especially those that depend heavily on glibc, can be sensitive to architecture or system packages and will likely need to be recompiled. To support this transition, the modulepaths of the current system will be split into CentOS 6 and CentOS 7 branches with the default modules transitioning to CentOS 7 at a specified time.

"legacy" modules will no longer be available on Odyssey. These modules typically start with prefixes like "hpc/", "math/", or "centos6/". In many cases, these modules are more than 7 years old, based on CentOS 5, and, in the case of MPI-based code, do not work properly with our current environment. The new module system already has several thousand builds of more than 600 applications and so should be able to accommodate any computation required.

We will be consolidating some of the modules during this migration. Particularly for some compiler and MPI-dependent libraries, there are dozens of builds only a handful of which are really used (see NetCDF). We will be using our module load logging data to determine whether applications or libraries are actually used and they will be given priority.

Migration of some legacy applications can be very difficult and time-consuming. RC staff will be available at Office Hours or by appointment to assist where possible.

In addition we will be moving to a new release pattern for modules where updated versions of common modules (such as python, MPI, gcc) will be released on a regular schedule, and older modules will be deprecated. This will help us stay up with the latest software available and reduce the number of redundant builds. More details on this new release pattern will be forthcoming after the upgrade to CentOS 7 is complete.

Check this page for updates

While this project progresses, we will be sending out periodic communications. These communications will be available from this page, so check below for new postings.

 

---