Using the Amazon EC2 Cluster with Odyssey
Amazon supports cluster computing on the Amazon EC2.
This FAQ shows how to use MPI executables compiled on Odyssey on the EC2.
For convenience you should compile your executable statically.
[hptc@iliadaccess04 hello]$ cat Makefile
CC=mpicc
CFLAGS=-g -static
LIBS=-lm
RM=/bin/rm
EXECS = hello
OBJS = hello.o
all: $(EXECS)
$(EXECS): $(OBJS) Makefile
$(CC) $(CFLAGS) -o $(EXECS) $(OBJS) $(LIBS)
$(OBJS): Makefile
.c.o:
$(CC) $(CFLAGS) -o $*.o -c $*.c
clean:
$(RM) -f $(EXECS) $(OBJS) gmon.out *~ *.o *.oo
[hptc@iliadaccess04 hello]$ module load hpc/openmpi-1.4.2_gnu-4.1.2
Loading module hpc/openmpi-1.4.2_gnu-4.1.2.
[hptc@iliadaccess04 hello]$ make
mpicc -g -static -o hello.o -c hello.c
mpicc -g -static -o hello hello.o -lm
/n/sw/openmpi-1.4.2_gnu-4.1.2/lib/libopen-pal.a(dlopen.o): In function `vm_open':
dlopen.c:(.text+0x143): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/liblsf.a(lib.esub.o): In function `runEexec_':
/pcc/build_eng/bldsrv/lsf/7.0.6/linux2.6-glibc2.3-x86_64/ego/base/lib/sdk/../lib.esub.c:624: warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/libbat.a(lsb.conf.o): In function `mygetgrnam':
lsb.conf.c:(.text+0x2f740): warning: Using 'getgrent' in statically linked applications requires at runt
....
Then go on the AWS console and create a cluster using the AMI ami-7ea24a17. Then create an elastic volume for static data. Attach it to one of the instances. Make a file system on it and NFS export it out. The instructions below assumes that you attached the volume as /dev/sdf
[root@ip-10-17-35-122 ~]# fdisk -l /dev/sdf
Disk /dev/sdf: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdf doesn't contain a valid partition table
[root@ip-10-17-35-122 ~]# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):
Using default value 1305
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 83
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@ip-10-17-35-122 ~]# mkfs.ext3 /dev/sdf1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2620595 blocks
131029 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@ip-10-17-35-122 ~]# vi /etc/fstab
[root@ip-10-17-35-122 ~]# mkdir /scratch
[root@ip-10-17-35-122 ~]# mount /scratch
[root@ip-10-17-35-122 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
/dev/sdf1 /scratch ext3 defaults 0 0
[root@ip-10-17-35-122 ~]# cat /etc/exports
/scratch 10.17.41.126(rw,async,no_root_squash)
[root@ip-10-17-35-122 ~]# /etc/init.d/nfs start
Starting NFS services: [ OK ]Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Use the following to find out the internal IP addresses of the other compute nodes:
[root@ip-10-17-41-126 helloworld]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 12:31:39:1B:52:01
inet addr:10.17.41.126 Bcast:10.17.41.127 Mask:255.255.255.192
inet6 addr: fe80::1031:39ff:fe1b:5201/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:48953 errors:0 dropped:0 overruns:0 frame:0
TX packets:26016 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:105183944 (100.3 MiB) TX bytes:99609416 (94.9 MiB)
Also the machine name has the internal IP address in it.
Mount the NFS export on all the other nodes:
[root@ip-10-17-41-126 ~]# cat /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
10.17.35.122:/scratch /scratch nfs defaults 0 0
[root@ip-10-17-41-126 ~]# mount /scratch
Copy libraries you will need over to this shared drive:
[hptc@iliadaccess04 ~]$ scp -r -i ~/.ec2/sndcluster.pem /n/sw/openmpi-1.4.2_gnu-4.1.2 root@ec2-204-236-240-187.compute-1.amazonaws.com:/scratch
example.conf 100% 94 0.1KB/s 00:00
btl-openib-benchmark 100% 622 0.6KB/s 00:00
help-opal-util.txt 100% 1348 1.3KB/s 00:00
help-mca-base.txt 100% 2602 2.5KB/s 00:00
help-mca-param.txt 100% 1574 1.5KB/s 00:00
help-opal-crs-base.txt 100% 793 0.8KB/s 00:00
help-ess-base.txt 100% 1293 1.3KB/s 00:00
ptmalloc2-COPYRIGHT 100% 976 1.0KB/s 00:00
...
[hptc@iliadaccess04 ~]$ scp -r -i ~/.ec2/sndcluster.pem /lsf/7.0/linux2.6-glibc2.3-x86_64/lib root@ec2-204-236-240-187.compute-1.amazonaws.com:/scratch
schmod_aps.so 100% 76KB 75.8KB/s 00:00
cal_jobweight.so 100% 15KB 14.7KB/s 00:00
schmod_limit.so 100% 114KB 114.4KB/s 00:00
liblsf.a 100% 2038KB 2.0MB/s 00:00
...
Make the appropriate links so the executable finds the libraries on the compute nodes:
[root@ip-10-17-35-122 ~]# mkdir -p /n/sw
[root@ip-10-17-35-122 ~]# ln -s /scratch/openmpi-1.4.2_gnu-4.1.2/ /n/sw
[root@ip-10-17-35-122 ~]# mkdir -p /lsf/7.0/linux2.6-glibc2.3-x86_64
[root@ip-10-17-35-122 ~]# ln -s /scratch/lib/ /lsf/7.0/linux2.6-glibc2.3-x86_64/
Make sure you put the appropriate paths in the root’s .bashrc:
[root@ip-10-17-41-126 ~]# cat .bashrc
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
export PATH=$PATH:/n/sw/openmpi-1.4.2_gnu-4.1.2/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/n/sw/openmpi-1.4.2_gnu-4.1.2/lib:/lsf/7.0/linux2.6-glibc2.3-x86_64/lib
Make sure that you can ssh without passwords to all the compute nodes:
[root@ip-10-17-35-122 .ssh]# ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
62:3c:62:3f:5d:41:6c:3b:08:7c:b1:1f:9d:e5:4f:1f root@ip-10-17-35-122
[root@ip-10-17-35-122 .ssh]# cat id_dsa.pub >> authorized_keys
You will need every compute node to have every other compute node’s ssh key.
Create a hostfile:
[root@ip-10-17-41-126 ~]# cat /scratch/helloworld/machines
10.17.41.126 slots=8
10.17.35.122 slots=8
Now you are ready to run the executable:
[root@ip-10-17-41-126 helloworld]# mpirun -np 16 -machinefile /scratch/helloworld/machines /scratch/helloworld/hello
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-41-126: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
ip-10-17-35-122: hello world. n=0
Use scp to copy the results back.
Please note that charges can rack up very quickly. Please use caution when trying this out.