LSF: Controlling Job Load with LSF Job Groups

It is possible to control the job load on the cluster using LSF’s job group feature.

[hptc@iliadaccess01 ~]$ bgadd -L 5 /MyJobGroup

Job group  was added.

[hptc@iliadaccess01 ~]$

The -L 5 limits this job group to 5 running or suspended jobs at a time.
Then, when you submit your jobs, associate them with this job group with
the -g directive:

[hptc@iliadaccess01 ~]$ for i in `seq 1 10` ; do bsub -q short_serial -g /MyJobGroup sleep 20; done

Now, even though we’ve submitted 10 jobs and there are cores available to
run them, only five will run at a time:

[hptc@iliadaccess01 ~]$ bjobs | grep "sleep 20"

815620  hptc RUN   short_serial   iliadaccess iliad33     sleep 20   Jul 10

21:38

815621  hptc RUN   short_serial   iliadaccess iliad49     sleep 20   Jul 10

21:38

815622  hptc RUN   short_serial   iliadaccess iliad23     sleep 20   Jul 10

21:38

815623  hptc RUN   short_serial   iliadaccess iliad31     sleep 20   Jul 10

21:38

815624  hptc RUN   short_serial   iliadaccess iliad43     sleep 20   Jul 10

21:38

815625  hptc PEND  short_serial   iliadaccess             sleep 20   Jul 10

21:38

815626  hptc PEND  short_serial   iliadaccess             sleep 20   Jul 10

21:38

815627  hptc PEND  short_serial   iliadaccess             sleep 20   Jul 10

21:38

815628  hptc PEND  short_serial   iliadaccess             sleep 20   Jul 10

21:38

815629  hptc PEND  short_serial   iliadaccess             sleep 20   Jul 10

21:38

[hptc@iliadaccess01 ~]$

If you are doing this to control load on a file system you can monitor the file system’s health by looking at the Storage Ganglia pages.

Job group limits can be modified with the bgmod command. For example, to change the limit on /MyJobGroup to 10 jobs,

[hptc@iliadaccess01 ~]$ bgmod -L 10 /MyJobGroup

Job group /MyJobGroup is modified.

[hptc@iliadaccess01 ~]$