Intel compilers like icc or icl are very useful for any cluster with Intel processors. They’ve been known to produce very efficient numerical code. If you are still a student, you can grab the student Intel Parallel Studio XE Cluster Edition, which includes Fortran and C/C++ for free for a year. Here’s our experience. If you need more information, definitely check out the official Intel Parallel Studio XE Cluster Edition guide.
Dependencies
You should have the GCC C and C++ compilers on your machine. I am using CentOS 7. You will need to install GCC C and C++ compilers on all the machines.
yum install gcc yum install gcc-c++
Getting the Intel compilers and MPI libraries
I’m going to grab the student Intel® Parallel Studio XE Cluster Edition for Linux, which lasts for a year. First thing to do is to join the Intel Developer Zone at the following link:
https://software.intel.com/registration/?lang=en-us
Fill your information and choose an Intel User ID to create. Now, you’ll have an account, but you’ll need to be a student to get the Intel compilers for free at:
https://software.intel.com/en-us/qualify-for-free-software/student
Click on Linux underneath Intel Parallel Studio XE Cluster Edition. Check the items on the next page and fill your e-mail before submitting. After submitting, you’ll receive an e-mail labeled “Thank You for Your Interest in the Intell® Software Development Products.”
The e-mail contains a product serial number that should last a year. The e-mail also contains a DOWNLOAD button that you should click.
After visiting the link, you’ll be brought to Intel® Parallel Studio XE Cluster Edition for Linux*. I prefer the Full Offline Installer Package (3994 MB). If you choose the Full Offline Installer Package, you will need to stay on that link and acquire your license file. In the red text, you’ll see the following sentence:
"If you need to acquire your license file now, for offline installation, please click here to provide your host information and download your license file."
Once you click the here link, you’ll be brought into a Sign In page to download your license file. After signing in, you’ll see your licenses that you can download. Download your license file or e-mail it to yourself. If you download the license, it should be a lic file.
At this point, you should have downloaded two files. parallel_studio_xe_2016_update2.tgz contains the zipped archive of the Intel Parallel Studio XE Cluster Edition, and NCOM….lic is your license.
parallel_studio_xe_2016_update2.tgz NCOM....lic
You should upload these two files to the shared folder of your cluster. My shared folder is /nfs, so I’ll be sending those two files to my /nfs folder.
scp parallel_studio_xe_2016_update2.tgz NCOM...lic [email protected]:/nfs
Now, you can extract the tgz file by running:
ssh [email protected] cd /nfs tar -xvf parallel_studio_xe_2016_update2.tgz
We will put the license file as Licenses in /root.
mv NCOM....lic /root/Licenses
Activation
Now, we will set up the Intel compilers and MPI libraries.
cd parallel_studio_xe_2016_update2 ./install.sh
It should say Initializing, please wait… until a text GUI pops up for installation. Type the number option that installs the installation.
1 2 3 4 5 6 7 8 9 | You will complete the steps below during setup process: Step 1 : Welcome Step 2 : License agreement Step 3 : Activation Step 4 : Intel® Software Improvement Program Step 5 : Options Step 6 : Installation Step 7 : Complete Please type a selection or press "Enter" to accept default choice [1]: |
First, we need to activate. Hit 3 and press Enter.
Step 2 of 7 | License agreement [Press space to continue, 'q' to quit.]
After pressing space a bunch of times, you’ll reach the end of the license.
Type 'accept' to continue or 'decline' to go back to the previous menu:
Type “accept.”
Please type a selection or press "Enter" to accept default choice [1]: Please type your serial number (the format is XXXX-XXXXXXXX):
In another terminal, check the serial number, which will be inside /root/Licenses.
Install
Hit enter and the correct number options for the Intel compilers and libraries to install. You’ll see the installation of Intel MPI Benchmarks, Libraries, C++ Compiler, Fortran Compiler, and more. Using the install.sh script is the sure way to make that all the Intel libraries are installed correctly, but if you really only want specific libraries, then you’ll have to select which ones you want to install inside the rpm/ folder. The full installation may take 15 minutes or more.
Press "Enter" key to continue:
Press "Enter" key to quit:
As for the final step, the paths for Intel may not be set up automatically. I am using CentOS 7 64 bit, so I’ll have to setup the environment for Intel 64 bit. We’ll have to adjust our ~/.bashrc.
vim ~/.bashrc
Add to the end of the file the following:
1 2 3 4 | export PATH=/opt/intel/compilers_and_libraries_2016.2.181/linux/mpi/intel64/bin:$PATH source /opt/intel/impi/5.1.3.181/intel64/bin/mpivars.sh intel64 source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/bin/ifortvars.sh intel64 |
Save and quit. Note: your directories may be slightly different based on the version of Intel Parallel Studio XE Cluster Edition you installed. Adjust those directories accordingly by searching whether the directories match.
source ~/.bashrc
Now, you should be able to access and use the Intel compilers as expected.
RLIMIT_MEMLOCK too small
When you first run your mpirun command with the Intel Parallel Studio XE Cluster Edition, you may receive an error about RLIMIT_MEMLOCK being too small.
mpirun -n 8 -f /nfs/hosts2 ./xhpcg --nx=16 --rt=60
1 2 3 4 5 6 7 8 | [0] DAPL startup: RLIMIT_MEMLOCK too small [1] DAPL startup: RLIMIT_MEMLOCK too small [2] DAPL startup: RLIMIT_MEMLOCK too small [3] DAPL startup: RLIMIT_MEMLOCK too small [4] DAPL startup: RLIMIT_MEMLOCK too small [5] DAPL startup: RLIMIT_MEMLOCK too small [6] DAPL startup: RLIMIT_MEMLOCK too small [7] DAPL startup: RLIMIT_MEMLOCK too small |
The problem is that memory lock is set statically, and it’s too small. For every machine that you want to use MPI, we should set memory lock to unlimited.
ulimit -l unlimited ulimit -l
If the second command says unlimited, we’ve set memory lock to unlimited. Now, we have to make sure that it’s unlimited on every startup instance.
vi /etc/security/limits.conf
Go to the bottom of the file and add the following:
* hard memlock unlimited * soft memlock unlimited
Save and quit. Now, if you run the MPI command again, you should not encounter any problems.
Missing Hydra Files
You may come across an error where you can have missing hydra files. When you run mpirun, you may get:
bash: /usr/local/bin/hydra_pmi_proxy: No such file or directory
How I fixed the problem was: I downloaded MPICH, a different MPI library, and compiled it with these instructions. hydra binaries for MPICH should work with Intel MPI because they’re both the same process manager. I copied MPICH’s hydra binaries to a directory that was also added to the ~/.bashrc PATH.
cp /nfs/mpich2/bin/hydra_persist /nfs/mpich2/bin/hydra_nameserver /nfs/mpich2/bin/hydra_pmi_proxy /usr/local/bin
Then, I added /usr/local/bin to the ~/.bashrc PATH.
vim ~/.bashrc
Add the following line:
export PATH=/usr/local/bin:$PATH
Save the file. And then reload ~/.bashrc.
source ~/.bashrc
Do this for all the nodes where you are missing hydra_pmi_proxy. Afterwards, if you run mpirun again, it should work!