Monday, 29 May 2017

Oracle Cloud Compute: Workaround for Docker / XFS issue with Oracle Linux

With the latest Oracle Linux 7.2 UEK4 images on the Oracle Cloud, the following error occurs when running Docker containers.

docker: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/66c96d19f86d1262f2472cda6639dacfde8f13d0bebff9e434b74b9cebdcc32f-init/merged: invalid argument.

This is related to the ftype parameter of the xfs filesystem, which is set at formatting. It can be verified by running

xfs_info /

and the output should look like this:

This is a screenshot from a fresh OL 7.2 UEK4 instance after a full yum update. As you can see, ftype is zero. Unfortunately, that setting is not compatible with the overlayfs from Docker and cannot be changed after formatting.
For more infos on this issue see To solve this, /var/lib/docker must be moved to a different volume with xfs and ftype=1 set or just use a different file system.

This example shows, how to solve this with an EXT4 volume.

On the Compute dashboard, go to the Storage Tab and click on Create Storage Volume.

Set the settings to your like and create the volume.

Klick on the hamburger menu next to your newly created volume and choose Attach to instance.

Choose your Compute instance and attach the volume.

To mount the volume, just follow the steps from the Oracle documentation: Mounting a Storage Volume on a Linux Instance.

First find out the device name via

ls /dev/xvd*

The output should look like this

The first device, the boot volume, is always /dev/xvdb, the next one is /dev/xvdc etc. In my example, the new volume is disk #2, so it is /dev/xvdc.
So the next step is a

fdisk /dev/xvdc

and go through the following steps:

In short, select n for new partition, p for primary, leave 1 as the default partition number and accept both sector numbers (dependent on the size of your volume). Then write the configuration with w.

Next create a file system, eg.

sudo mkfs -t ext4 /dev/xvdc

and the result should look like this

Now we need a mount point, which would be /u01 according to Oracle's styleguides. By default, the OL image already comes with a /u01 directory as mount point, so we can use this.
As we want this to be a permanent mount, add this to /etc/fstab by

sudo vi /etc/fstab

and add the following line

/dev/xvdc /u01 ext4 defaults 0 0

so it should look like this:

Save and restart the instance to check if everything worked. After reboot, check if the file system is mounted, eg. with

mount | grep /u01; ls /u01

and the output should look like this:

Now create a location where to move the directory /var/lib/docker, for example /u01/var/lib. Use any method you like to move the docker directory to the new location. I prefer to use tar, so I have an additional backup, eg.

cd /var/lib
tar cvfz docker.tgz docker
cd /u01/var/lib/
tar xvfz /var/lib/docker.tgz
mv /var/lib/docker /var/lib/docker.bak
cd /var/lib
ln -s /u01/var/lib/docker/ docker

Then set a symlink to your newly created docker lib directory and restart the docker daemon and try it with any image.

Should run now, that's it.