I have a Windows 10 laptop, and I am running VirtualBox. One of my virtual machines has Ubuntu 14.04 64 bit operating system. Now, I want to be able to share folders from my Ubuntu to Windows 10 and vice versa. I’m assuming that you have installed the Ubuntu virtual machine already and ready to set up the shared folder.
Step 1) Make a folder to be shared on your Windows 10 host operating system.
I’ll create a folder called shared on my Windows 10 Desktop.
I’ll put a bunch of stuff inside the shared folder to be used by the Ubuntu virtual machine.
Step 2) Make sure that you have an empty optical drive for your virtual machine.
Step 3) Boot your Ubuntu virtual machine. Go to Devices tab and click Insert Guest Additions CD image…
Step 4) Open up a terminal and install the Guest Additions CD.
We make a directory to mount the Guest Additions CD. The Guest Additions CD contains a file system type that we need to use.
sudo mkdir /media/cdrom
Now, we can mount the Guest Additions CD to the /media/cdrom folder.
sudo mount /dev/cdrom /media/cdrom
mount: block device /dev/sr0 is write-protected, mounting read-only
We should install necessary packages if they have not already been installed.
sudo apt-get install make gcc linux-headers-$(uname -r)
Now, we can install and run the Guest Additions CD.
sudo /media/cdrom/VBoxLinuxAdditions.run
Step 5) Click on the Devices tab and click Shared Folder Settings… and integrate the Windows 10 shared folder.
Hit the add shared folder button.
Click on the Folder Path list and find your shared folder on Windows 10. I checked off Read-only, Auto-mount, and Make Permanent. Hit OK.
Step 6) Create a shared folder on Ubuntu virtual machine and mount the shared folder.
You make a directory on the Ubuntu virtual machine that will act as Ubuntu’s shared folder.
mkdir ~/shared-windows10
We mount the shared folder on Windows 10 to the ~/shared-windows10 folder.
sudo mount -t vboxsf shared ~/shared-windows10
shared was the name of the Windows 10 shared folder, and I called the Ubuntu shared folder, ~/shared-windows10, to differentiate between the two.
Step 7) Change directory into the shared folder and see all the shared files.
cd ~/shared-windows10
ls
Huzzah! The shared folder is great for transferring files from Windows 10 to the Ubuntu virtual machine and vice versa.
I set the shared folder to Read-only, but you can uncheck the box in the Shared Folder Settings…
For this blog post, I will be building on top of the Virtual Machine Clustering and NFS Server Setup in part 1. As a result, before you follow the steps of this post, you will want to have a similar setup like I have instructed in part 1:
MPI stands for Message Passing Interface. MPI isn’t your average networking library. It’s optimized for performance, takes the fastest transport for running parallel programs across machines, and is a usable protocol library implemented on nearly all operating systems.
Virtual Machine Information
With the steps from part 1, I have two virtual machines connected together under an internal network with one being the NFS Server and the other being the NFS Client.
Shell
1
2
CentOS-1:10.0.1.2
CentOS-2:10.0.1.3
Setting up SSH Keys
Currently, we can access from machine #1 to machine #2 through SSH. On machine #1,
The problem is that we will always receive a password prompt whenever we SSH across the internal network to the other machine. We want to set up SSH keys to SSH from each machine without passwords.
Let’s make sure that we’re on machine #1. Before we create the SSH key, create the ~/.ssh folder.
You can press Enter to leave the next three prompts as default.
Enter file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter]
Enter passphrase (empty for no passphrase): [Type a passphrase]Enter same passphrase again: [Type passphrase again]
Your identification has been saved in /Users/you/.ssh/id_rsa.
Your public key has been saved in /Users/you/.ssh/id_rsa.pub.
The key fingerprint is:
01:0f:f4:3b:ca:85:d6:17:a1:7d:f0:68:9d:f0:a2:db [email protected]
cd ~/.ssh
We will copy the public key, id_rsa.pub, to authorized_keys to enable this key for access to machine #1.
cp id_rsa.pub authorized_keys
Now, we should send the private key, id_rsa, and public key, id_rsa.pub, from machine #1 to machine #2. We use a command called scp for copying files over machines.
First, we’ll need a library called wget. Wget will allow us the ability to download links to the machine. We will download wget on machine #1.
yum install wget -y
We will now download the source of mpich, which is an implementation of MPI. We download inside the shared folder.
cd /nfs
wget http://www.mpich.org/static/downloads/3.1.4/mpich-3.1.4.tar.gz
After we have downloaded mpich, we will install the C compilers, Fortran compiler, and kernel build tools on machine #1 and machine #2.
yum install gcc gcc-c++ gcc-fortran kernel-devel -y (on both machines)
vi ~/.bashrc
To extract the mpich downloaded tar.gz compressed file, we use the following command. The command will create a mpich-3.1.4 folder with all the contents of the extracted compressed file.
tar -xvf mpich-3.1.4.tar.gz
We will make a directory where all the compiled binaries and libraries of mpich will go.
mkdir /nfs/mpich3
Now, we will configure the settings of mpich for installation.
cd /nfs/mpich-3.1.4
./configure --prefix=/nfs/mpich3
Afterwards, we run a couple of commands for finishing the compiling and installation of mpich.
make
make install
If we cd /nfs/mpich3, we will see folders containing the binaries and libraries of mpich. If we cd /nfs/mpich3/bin, we can see mpi binaries like mpirun.
Currently, we won’t be able to use mpirun from anywhere on the machine. We need to change the ~/.bashrc file on machine #1 and machine #2 to globalize the mpi commands.
On both machines:
vi ~/.bashrc
At the bottom of ~/.bashrc, add the following two lines:
PATH is used for bin folders, and LD_LIBRARY_PATH is used for lib folders. To reload the ~/.bashrc, type the following command on both machines:
source ~/.bashrc
Using MPI binaries: Running MPI
Let’s make a folder where we’ll create space for projects.
mkdir /nfs/projects
cd /nfs/projects
We need to create a hosts file that contains the IP addresses of all the IP’s that we want MPI to run.
vi hosts
10.0.1.2
10.0.1.3
MPI relies on ports for TCP and UDP packet communication. We will need to stop the firewalld for the process to hop between machines.
systemctl stop firewalld
We can test mpirun on a Linux command. The -f flag selects the host file that determines the IP addresses where MPI will decide to run the program. The -n flag determines the number of CPU cores that you want the program to run on. You may have to be on the client machine, in this case, 10.0.1.3, for the command to run without any errors.
mpirun -f hosts -n 4 echo "hello world"
The mpirun command prints the following if you have the machines in each other’s known_hosts records.
Shell
1
2
3
4
hello world
hello world
hello world
hello world
mpirun has run echo “hello world” 4 times because we chose 4 cores. When mpirun is used on a serial (meant for one machine) command, it will tend to run the command as many times as the number of cores selected as separate instances.
top is a useful Linux command to see what processes are running on the machine. If we check machine #2:
top
And if we run the mpirun command on machine #1 again:
mpirun -f hosts -n 4 echo "hello world"
We will see that top on machine #2 will not see any changes. The process does not jump over. Why? The reason is because when you use a serial Linux command like echo or ls or pwd, these commands are built for a single machine use. MPI does not see the need to jump this command over to another machine. MPI tries the fastest transport, which could be running everything on the one machine. MPI has the choice between the IP addresses specified in the hosts file. As a result, echo “hello world” is run only on machine #1.
Press q to quit top.
On the next blog post, we will write a basic MPI C or Python program and use mpi binaries to show you that the process is run across machines on programs designed to run in parallel. If you use mpirun on programs designed to run serially, the processes will likely run only on that machine because MPI will not see the need to make the jump to another machine, and the program in itself will not be programmed to do so.
Here are a series of tutorials for beginners to get started in clustering virtual machines and learn how to parallel program. We will be using VirtualBox and CentOS in this article to simulate a clustered supercomputer.
I have an article on how to install VirtualBox, which is a program that allows you to create and use virtual machines. CentOS is a popular Linux operating. I’m writing these articles to teach the basics on what makes a supercomputer and how to get started with supercomputing and high performance computing.
What makes a supercomputer?
As you may have guessed, supercomputers are computers with high computational capacity compared to your everyday computer. Supercomputers tend to be built as a connected system of smaller computers called nodes. These nodes are connected together through a hub device called a switch where processes can be distributed in parallel via IP or VPI. Less powerful computers/nodes connected to create a supercomputer. With MPI which stands for Message Passing Interface in a distributed system, you can run processes that are passed among the individual nodes and run in parallel to achieve faster computations.
Prequisites
You must first install VirtualBox, which I have written a separate article on. I’ll also leave the link to the VirtualBox download. You will also need to download a Linux operating system ISO for booting the virtual machines with. I will use a minimum ISO of CentOS 7.1 in this example.
Install VirtualBox
Download CentOS 7.1
Create two virtual machines in VirtualBox
We create two virtual machines on VirtualBox. For each of the two virtual machines, go to Settings. Inside Settings, go to Storage. For the second item with a disc icon next to the option, select an ISO such as CentOS 7.1 minimal ISO to boot. Make sure that Live CD/DVD is checked. Press OK.
Start both virtual machines. The setup to boot into the operating system appears for installing CentOS. Enter the operating system installation. Use the default configurations for setting up the operating system. At the second setup window, click Installation Destination and click Done. Finally, click Begin Installation.
Click root password and set a password. Wait until the installation is complete and click Reboot. You can boot into CentOS on both machines. To login, the username is root, and the password is what you set the password as.
Setting up Internet Connection and DHCP
Click the X button on both virtual machines. Power off both virtual machines. Open File Explorer on Windows. Go to C:\Program Files\Oracle\VirtualBox. Hold Shift and Right Click white space on this window. Click Open Command Window Here.
We will create an Internal Network with VirtualBox’s built-in command. It will mimic our DHCP server.
Go to Settings on both virtual machines in VirtualBox. First, we go to Storage. We uncheck Live CD/DVD, and we remove the optical disc that is the CentOS 7 minimal ISO. Next, we go to Network. Click on Adapter 2. Select Internal Network as the type. intnet should be automatically assigned. Click OK. Start both virtual machines.
On both machines, setup the Internet Connection.
nmcli d
enps08
enps03
enps08 will be connected, but enps03 will be disconnected.
vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
Change the last line of this file on both machines to ONBOOT=yes.
Shell
1
ONBOOT=yes
systemctl restart network
Try ping 8.8.8.8. We have access to the Internet. Now, we must get access from one machine to the other machine.
ip addr show
When we check the IP addresses of both machines, we see that:
Shell
1
2
CentOS-1:10.0.1.2
CentOS-2:10.0.1.3
Each of the machines have an IP address within the Internal Network.
On CentOS-1: ping 10.0.1.3
On CentOS-2: ping 10.0.1.2
We have established the ability to ping from one machine to the other machine. We can communicate within the internal network among machines.
Setting up NFS Server: Machine #1
On CentOS-1, we’ll set the machine as the NFS Server. We will need to install a couple of NFS libraries.
We make a folder where the shared folder from Machine #1 will be mounted on Machine #2.
mkdir -p /nfs
We make sure that we can access CentOS-1, the NFS Server. Make sure that the following two commands do not return any errors.
showmount -e 10.0.1.2
rpcinfo -p 10.0.1.2
mount 10.0.1.2:/nfs /nfs
With df -h, we should see that 10.0.1.2:/nfs mount has been created at the bottom. If we create any file inside /nfs, then all the machines connected can see the same file.
df -h
Now, we test that the shared folder actually works.
cd /nfs
touch microwavesam.txt
On CentOS-1, if we cd /nfs, we will see microwavesam.txt is inside the folder.
Making NFS more automatic
When you restart the two virtual machines, the NFS shared folder will not be there. We need to set a more automatic way for the NFS client to look for the NFS folder.
On the Client, we change a file called /etc/fstab.
Whoa… VirtualBox! VirtualBox is a fantastic program for creating virtual instances of your everyday regular computer machines. We normally use the program to pop up virtual Linux distribution machines like CentOS and Ubuntu so that even when we’re using a Windows or Mac machine, we can still have a window that is running Linux.
1) Download VirtualBox
You can find the latest virtual box at the following link:
The set-up is quite simple. Proceed through the directions, prompts, and install.
4) Open VirtualBox
VirtualBox will have any old settings from previous installations of the program.
We will make a New virtual machine by first clicking on the New button at the top left.
5) VirtualBox settings
We will use the default settings for RAM, but make sure that you give enough hard drive memory for your virtual machine.
By default, the hard drive space for a VirtualBox Disk Image (VDI) will be 8 GB. I’ll expand my VDI to 50 GB.
6) After configuring settings, we want to right click our machine and make sure it loads the iso
Right click the virtual machine on the left side and click on Settings (CTRL-S). This will pop up the machineName – Settings.
7) Click on Storage option on the left
Underneath Controller: IDE, click on Empty. Make sure that the check box that says Live CD/DVD is checked.
Click on the Disc icon and click on Choose a virtual CD/DVD disk file. For me, I’ll find where I saved the archlinux.iso that I saved earlier.
Click OK.
8) Allow access to the internet for your virtual machine
Still in the settings window, click on the Network option on the left sidebar. By default, you’ll see Attached to NAT for Adapter 1. Leave this the same. Click on Adapter 2, enable the Adapter and click on the dropdown for Attached to Bridged Adapter. Leave the Name by default. This setting will enable your virtual machine to use the wi-fi or wired connection of your main machine.
9) Click on the machine on the left sidebar and click Start
10) After installing your Linux distribution, you’re all set
You can now use your Linux machine on your other operating system. Arch Linux is installed and ready to go!