You can use MPI without NFS or a shared file system! We had a situation where we couldn’t find the NFS server or client packages for arm64 for Ubuntu 16.04. We had OpenMPI version 1.10.2 installed on 2 nodes without NFS.

When you use MPI without NFS, you need to ensure that the same version of MPI is installed by every node.

Then, you have to ensure that the same data files, which include the program, hostnames file, and input files, are on every node at the same location relative to that node.

Lastly, you should double-check that you have the same SSH key and have SSHed onto every node from the node that you’re running the program at least once.

Step 1) Ensure that the same version of MPI is installed on every node.

We can check the location of where OpenMPI or any version of MPI is on every node.

which mpicc
 /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpicc

Make sure that this directory is consistent on each node. Now, we should check that the ~/.bashrc sets the same OpenMPI PATH folder on every node.

vi ~/.bashrc

Add the following line somewhere in the file if it is not already there.

export PATH="/opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin:$PATH"

If you just added the above line, you will need to reload your ~/.bashrc. Do this for every node.

source ~/.bashrc

We can double check the version of MPI on every node by running:

mpirun --version
mpirun (Open MPI) 1.10.2
Report bugs to http://www.open-mpi.org/community/help/

Step 2) Same data files on every node.

What I would suggest is to compile the application or program on one node and send the program to all the other nodes because you must have the same exact program on every node in the same location to run mpirun without NFS properly.

For instance, let’s say on node 1 called tegra1-ubuntu, I will compile a basic MPI hello world program.

cd ~
pwd
/ubuntu/home

Now, we use git to download the mpi hello world program.

git clone https://github.com/huyle333/mpi-hello-world
cd mpi-hello-world

We will need to compile the program. My suggestion is to use the full path of mpicc to compile the program. We know the full path of mpicc already since we used which mpicc earlier.

 /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpicc -o mpi_hello_world mpi_hello_world.c

We will send the compiled binary, mpi_hello_world, to the same location as this node to all the other nodes.

scp mpi_hello_world ubuntu@tegra2-ubuntu:~

Now, we will have to create a file that contains the hostnames or IP addresses of the 2 nodes.

cd ~
vi hostnames

Instead of hostnames, you can put the IP addresses.

tegra1-ubuntu
tegra2-ubuntu

We send the hostnames file to the other node.

scp hostnames ubuntu@tegra2-ubuntu:~

Step 3) Make sure that the SSH key is the same on every node.

First, we check if we have an SSH key.

ls ~/.ssh
id_rsa id_rsa.pub known_hosts authorized_keys
cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDJxIA4WSnXiJEWZ16SrRgGKOoIS6Z2sHSZreGKDggf+aJ2unEP5vtnFq07fmKDDxG+nMipTFpzx0bMB5ysXNZaTpnEKmW76BaO7402J/bIf/HsqZBMip39d+swkXkq9NB5yCHSn7+kmzf5PKaL34X8cNLOK6I5IZrqrHj8b10JyhORJ8URxa0VltItsblCvTUrdW5grR0+O8aY3UyzaZXLIwwYBF/vrQnt/bcPSA3j6lW829pUz+XsYOsKeit7aUep+ek0q1F3SYuPUoPe7vwp8+X+TiGBQTbraynZHVEov0ZJwWojw89Xc42qGtAiW1N+NrxkuaNXvJIHpua3ZCUdfJUXLlXfhOpFWZxU7F/C32Rj6x7kz6HJrjXkTaV3UD8puh7J2oVW8sGVOoKk99KPN0bztL//sj8UDVSD8rHxl5FanCHqBICIF+ZBrqcG6v3ElNcAq/KxpVEpypZndYa+FOwXvXJfBMg5IbDzgWXy6WAuK8bI8Iavk5UeRmAOGDvJzXG/30N06lmkQKnZYhtTQ4LY10Y0lbkNSCys7ceimRB3YKbVaoSxdbTiWzhNP2a7XTTmG/b1P022HdEYsZ9+9+iwyXRINmcvT3J+8QSsLryd3u/G5kWVX9iHnFPbEt3TRCZwJLkoQXxN0OTGFveaQpjMsui6Wpu3RKdcKMzY/w== huyle333@bu.edu

We make sure that the contents of id_rsa.pub are somewhere inside the authorized_keys file. authorized_keys will give SSH access to any node as long as id_rsa.pub is inside the file.

cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDJxIA4WSnXiJEWZ16SrRgGKOoIS6Z2sHSZreGKDggf+aJ2unEP5vtnFq07fmKDDxG+nMipTFpzx0bMB5ysXNZaTpnEKmW76BaO7402J/bIf/HsqZBMip39d+swkXkq9NB5yCHSn7+kmzf5PKaL34X8cNLOK6I5IZrqrHj8b10JyhORJ8URxa0VltItsblCvTUrdW5grR0+O8aY3UyzaZXLIwwYBF/vrQnt/bcPSA3j6lW829pUz+XsYOsKeit7aUep+ek0q1F3SYuPUoPe7vwp8+X+TiGBQTbraynZHVEov0ZJwWojw89Xc42qGtAiW1N+NrxkuaNXvJIHpua3ZCUdfJUXLlXfhOpFWZxU7F/C32Rj6x7kz6HJrjXkTaV3UD8puh7J2oVW8sGVOoKk99KPN0bztL//sj8UDVSD8rHxl5FanCHqBICIF+ZBrqcG6v3ElNcAq/KxpVEpypZndYa+FOwXvXJfBMg5IbDzgWXy6WAuK8bI8Iavk5UeRmAOGDvJzXG/30N06lmkQKnZYhtTQ4LY10Y0lbkNSCys7ceimRB3YKbVaoSxdbTiWzhNP2a7XTTmG/b1P022HdEYsZ9+9+iwyXRINmcvT3J+8QSsLryd3u/G5kWVX9iHnFPbEt3TRCZwJLkoQXxN0OTGFveaQpjMsui6Wpu3RKdcKMzY/w== huyle333@bu.edu

Make sure that the 2nd node has the ~/.ssh directory.

ssh ubuntu@tegra2-ubuntu
ls ~/.ssh
known_hosts
exit

Back on node 1, we will send the SSH public and private key and authorized_keys file to node 2.

cd ~/.ssh
scp id_rsa id_rsa.pub authorized_keys ubuntu@tegra2-ubuntu:~/.ssh

Step 4) Run the mpi program with full paths.

To make sure that the mpi program runs properly without NFS, make sure that you run the program with the full path of the MPI binaries and hostnames.

ubuntu@tegra1-ubuntu:~$ /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpirun --hostfile /ubuntu/home/hostnames -n 8 /ubuntu/home/mpi-hello-world/mpi_hello_world
Hello world from processor tegra1-ubuntu, rank 2 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 3 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 1 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 0 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 7 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 5 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 6 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 4 out of 8 processors

You should see that the hello world is processed on both nodes! If you have input files used by your program, make sure that they are also in the same location on both nodes.

Leave a comment if you have any questions.