How to use MPI without NFS

You can use MPI without NFS or a shared file system! We had a situation where we couldn’t find the NFS server or client packages for arm64 for Ubuntu 16.04. We had OpenMPI version 1.10.2 installed on 2 nodes without NFS.

When you use MPI without NFS, you need to ensure that the same version of MPI is installed by every node.

Then, you have to ensure that the same data files, which include the program, hostnames file, and input files, are on every node at the same location relative to that node.

Lastly, you should double-check that you have the same SSH key and have SSHed onto every node from the node that you’re running the program at least once.

Step 1) Ensure that the same version of MPI is installed on every node.

We can check the location of where OpenMPI or any version of MPI is on every node.

which mpicc
 /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpicc

Make sure that this directory is consistent on each node. Now, we should check that the ~/.bashrc sets the same OpenMPI PATH folder on every node.

vi ~/.bashrc

Add the following line somewhere in the file if it is not already there.

export PATH="/opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin:$PATH"

If you just added the above line, you will need to reload your ~/.bashrc. Do this for every node.

source ~/.bashrc

We can double check the version of MPI on every node by running:

mpirun --version
mpirun (Open MPI) 1.10.2
Report bugs to http://www.open-mpi.org/community/help/

Step 2) Same data files on every node.

What I would suggest is to compile the application or program on one node and send the program to all the other nodes because you must have the same exact program on every node in the same location to run mpirun without NFS properly.

For instance, let’s say on node 1 called tegra1-ubuntu, I will compile a basic MPI hello world program.

cd ~
pwd
/ubuntu/home

Now, we use git to download the mpi hello world program.

git clone https://github.com/huyle333/mpi-hello-world
cd mpi-hello-world

We will need to compile the program. My suggestion is to use the full path of mpicc to compile the program. We know the full path of mpicc already since we used which mpicc earlier.

 /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpicc -o mpi_hello_world mpi_hello_world.c

We will send the compiled binary, mpi_hello_world, to the same location as this node to all the other nodes.

scp mpi_hello_world [email protected]:~

Now, we will have to create a file that contains the hostnames or IP addresses of the 2 nodes.

cd ~
vi hostnames

Instead of hostnames, you can put the IP addresses.

tegra1-ubuntu
tegra2-ubuntu

We send the hostnames file to the other node.

scp hostnames [email protected]:~

Step 3) Make sure that the SSH key is the same on every node.

First, we check if we have an SSH key.

ls ~/.ssh
id_rsa id_rsa.pub known_hosts authorized_keys
cat ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDJxIA4WSnXiJEWZ16SrRgGKOoIS6Z2sHSZreGKDggf+aJ2unEP5vtnFq07fmKDDxG+nMipTFpzx0bMB5ysXNZaTpnEKmW76BaO7402J/bIf/HsqZBMip39d+swkXkq9NB5yCHSn7+kmzf5PKaL34X8cNLOK6I5IZrqrHj8b10JyhORJ8URxa0VltItsblCvTUrdW5grR0+O8aY3UyzaZXLIwwYBF/vrQnt/bcPSA3j6lW829pUz+XsYOsKeit7aUep+ek0q1F3SYuPUoPe7vwp8+X+TiGBQTbraynZHVEov0ZJwWojw89Xc42qGtAiW1N+NrxkuaNXvJIHpua3ZCUdfJUXLlXfhOpFWZxU7F/C32Rj6x7kz6HJrjXkTaV3UD8puh7J2oVW8sGVOoKk99KPN0bztL//sj8UDVSD8rHxl5FanCHqBICIF+ZBrqcG6v3ElNcAq/KxpVEpypZndYa+FOwXvXJfBMg5IbDzgWXy6WAuK8bI8Iavk5UeRmAOGDvJzXG/30N06lmkQKnZYhtTQ4LY10Y0lbkNSCys7ceimRB3YKbVaoSxdbTiWzhNP2a7XTTmG/b1P022HdEYsZ9+9+iwyXRINmcvT3J+8QSsLryd3u/G5kWVX9iHnFPbEt3TRCZwJLkoQXxN0OTGFveaQpjMsui6Wpu3RKdcKMzY/w== [email protected]

We make sure that the contents of id_rsa.pub are somewhere inside the authorized_keys file. authorized_keys will give SSH access to any node as long as id_rsa.pub is inside the file.

cat ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDJxIA4WSnXiJEWZ16SrRgGKOoIS6Z2sHSZreGKDggf+aJ2unEP5vtnFq07fmKDDxG+nMipTFpzx0bMB5ysXNZaTpnEKmW76BaO7402J/bIf/HsqZBMip39d+swkXkq9NB5yCHSn7+kmzf5PKaL34X8cNLOK6I5IZrqrHj8b10JyhORJ8URxa0VltItsblCvTUrdW5grR0+O8aY3UyzaZXLIwwYBF/vrQnt/bcPSA3j6lW829pUz+XsYOsKeit7aUep+ek0q1F3SYuPUoPe7vwp8+X+TiGBQTbraynZHVEov0ZJwWojw89Xc42qGtAiW1N+NrxkuaNXvJIHpua3ZCUdfJUXLlXfhOpFWZxU7F/C32Rj6x7kz6HJrjXkTaV3UD8puh7J2oVW8sGVOoKk99KPN0bztL//sj8UDVSD8rHxl5FanCHqBICIF+ZBrqcG6v3ElNcAq/KxpVEpypZndYa+FOwXvXJfBMg5IbDzgWXy6WAuK8bI8Iavk5UeRmAOGDvJzXG/30N06lmkQKnZYhtTQ4LY10Y0lbkNSCys7ceimRB3YKbVaoSxdbTiWzhNP2a7XTTmG/b1P022HdEYsZ9+9+iwyXRINmcvT3J+8QSsLryd3u/G5kWVX9iHnFPbEt3TRCZwJLkoQXxN0OTGFveaQpjMsui6Wpu3RKdcKMzY/w== [email protected]

Make sure that the 2nd node has the ~/.ssh directory.

ssh [email protected]
ls ~/.ssh
known_hosts
exit

Back on node 1, we will send the SSH public and private key and authorized_keys file to node 2.

cd ~/.ssh
scp id_rsa id_rsa.pub authorized_keys [email protected]:~/.ssh

Step 4) Run the mpi program with full paths.

To make sure that the mpi program runs properly without NFS, make sure that you run the program with the full path of the MPI binaries and hostnames.

[email protected]:~$ /opt/arm/openmpi-1.10.2_Cortex-A57_Ubuntu-14.04_aarch64-linux/bin/mpirun --hostfile /ubuntu/home/hostnames -n 8 /ubuntu/home/mpi-hello-world/mpi_hello_world
Hello world from processor tegra1-ubuntu, rank 2 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 3 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 1 out of 8 processors
Hello world from processor tegra1-ubuntu, rank 0 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 7 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 5 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 6 out of 8 processors
Hello world from processor tegra2-ubuntu, rank 4 out of 8 processors

You should see that the hello world is processed on both nodes! If you have input files used by your program, make sure that they are also in the same location on both nodes.

Leave a comment if you have any questions.

How to SSH to Another Computer with Ethernet Cable – Local Area Network

Let’s say that we wanted to connect an Ethernet cable from Ethernet port of one computer to Ethernet port of another computer. I want to be able to SSH into the other machine from my computer given that I know the other machine’s password or have an authenticated SSH key.

ethernet-to-ethernet

You can also SSH to a series of machines if you have a switch with all the other machines connected to that switch through Ethernet cables. The switch is basically a hub point where all the machines can connect their Ethernet cables to and be able to access every other machine on the local area network.

ethernet-cables-to-a-switch

Let’s talk about how to SSH from 1 computer to another computer when the Ethernet cable from port to port of the two computers. One of the computers is my Windows 10 laptop. The other computer is an Ubuntu 16.04 server.

 

Step 1) Start the SSH server on the Ubuntu 16.04 computer.

First, we make sure that we have the OpenSSH server installed. We will require Internet access for this step.

sudo apt-get install openssh-server

Now, we restart the SSH server to be sure that it is working properly.

sudo service ssh restart
ssh stop/waiting
ssh start/running, process 3878

Step 2) Connect Ethernet cable from the Ethernet port of Windows 10 computer to the Ethernet port of the Ubuntu 16.04 computer.

Step 3) Set a static IP address on the Ubuntu 16.04 computer.

If both machines are connected to the Internet on the same router through Wi-Fi, then you will not need to create a static IP address. For more information about setting static IP addresses on Ubuntu, go here.

sudo vi /etc/network/interfaces

Look for an interface that begins with eth. The interface is typically named eth0.

auto eth0
iface eth0 inet static
address 18.125.5.100
netmask 255.255.255.0
gateway 18.125.5.1

Now, you save the file and restart the interface.

sudo ifdown eth0 && sudo ifup eth0

Step 4) Set a static IP address on the Windows 10 computer.

We need to have an IP address on the same IP space as the Ubuntu 16.06 computer. On Windows 10, you will right click the Internet icon on your task bar and click on Open Network and Sharing Center.

change-adapter-settings

On the right side bar, click Change adapter settings. Right click on Ethernet and click Properties. Click on Internet Protocol Version 4. Click Properties.

ethernet-properties-windows-10

Check off Use the following IP address and fill out IP address and Subnet mask. Note that you cannot use an IP address that is already used by the other machine.

internal-network-access-set-local-ip-on-machine

I will set the Windows 10 computer with the IP address 18.125.5.251 and the subnet mask of 255.255.255.0. Default gateway is optional. Use the following DNS server addresses is optional. I set 8.8.8.8 to point to Google’s Public DNS server. I check off Validate settings upon exit and hit OK.

After the validation, your computer should have the new static IP address. If you do not have Wi-Fi, you will not have Internet access at the moment since you set a static IP address that is likely not valid by your DHCP server, but you will be able to SSH into the Ubuntu 16.04 computer.

Step 5) SSH from Windows 10 terminal to the Ubuntu 16.04 computer.

For the Windows 10 terminal, you can use something like Cygwin to SSH into the Ubuntu 16.04 computer. Cygwin is program that allows Windows to use UNIX.  First, we ping the Ubuntu 16.04 machine to see if we have network connectivity.

ping 18.125.5.100

Eureka! Now, we can SSH into the Ubuntu 16.04 computer with its username.

ssh [email protected]

Enter your password, and you will have access to your Ubuntu 16.04 computer from your Windows 10 computer using the local area network and direct Ethernet cable.