#

ShellShock update issue temporarily affecting Slurm jobs and software modules

Summary

The recent bash shellshock patch introduces a different function definition syntax in exported environments. This may affect batch jobs, where the environment gets defined on a submission host but run on a compute host. The issue arises when the bash versions don't match, such as during a rolling upgrade.

Description

This week we've been patching the Odyssey cluster against a security vulnerability nicknamed shellshock. The issue is multifold:

  • CVE-2014-6271, partially patched on CentOS by bash-4.1.2-15.el6_5.1.x86_64
  • CVE-2014-7169*, patched on CentOS by bash-4.1.2-15.el6_5.2.x86_64

The latter introduces changes to how bash handles exported functions. Specifically, if you define and export a function:

$ myfunction () { echo "hello world"; }
$ export -f myfunction

the corresponding environment variable is now different.

The old form:

myfunction=() { echo "hello world"
}

The new form:

BASH_FUNC_myfunction()=() { echo "hello world"
}

These forms are not compatible, and it causes issues on batch job clusters such as Odyssey if the job submission host has a different version of bash than the node where the job runs.

Specifically, the module command used for loading software on Odyssey is a function, and when using submission hosts that picked up the latest patches, you may see this error in your batch jobs:

/bin/bash: error importing function definition for `BASH_FUNC_module()`

and find that the module command doesn't work. Vice versa, some submission hosts are temporarily behind in version compared to compute nodes, and while you get no error, the module function does not work in that case either.

Workaround

To regain the use of the module function, you can add the following to your batch job script, before any module loads:

source /etc/profile.d/modules.sh

In the meantime, we're bringing all cluster hosts up to the latest bash version, at which point this incompatibility issue will go away.

Errata

  • update 2014-09-30: CVE-2014-7169 was typoed as CVE-2014-7160 before

CC BY-NC 4.0 This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Permissions beyond the scope of this license may be available at Attribution.