DigitalOcean and kernel inconsistency
August 28, 2013
When I logged into my DigitalOcean VM today I noticed that the advertised kernel on the motd was GNU/Linux 3.5.0-17-generic x86_64, something which came as a bit of a surprise given the fact I’d updated to 3.5.0-39 only a matter of days before. The other slightly concerning fact was that the autoremove function had definitely uninstalled that kernel from my machine at the same time as I’d updated to 3.5.0-39 …
The newer kernels were indeed installed as expected, and as Ubuntu usually does, the autoremove cleanup had removed the older kernels, including the ‘current’ one of 3.5.0-17. Cleanup had also removed the /lib/modules for 3.5.0-17, and consequently there were no modules on the server the loaded kernel - why Digital Ocean does not check for this on boot I have no idea.
Removal of the /lib/modules directory prevents things such as iptables from working (as you’d expect), and consequently other services which depend on iptables (read fail2ban among others) therefore fail.
To test the theory, I started a new Digital Ocean droplet. I updated to the latest packages, and removed the 3.5.0-17 as expected via the autoremove function. Unsurprisingly, iptables is gone:
$ sudo iptables -L
FATAL: Could not load /lib/modules/3.5.0-17-generic/modules.dep: No such file or directory
iptables v1.4.12: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
So, we’ve lost iptables, the kernel directory exists for the new kernel (i.e. /lib/modules/3.5.0-39-generic). We’re on an old kernel, without the modules on the VM … what has gone wrong here?
Having taken a look online, it looks like DigitalOcean provide their kernels via the hypervisor, not on your VM, and consequently, even if you’ve updated the machine and removed the old kernel, the machine will still boot on the one they select, even when the modules have been removed from the machine.
I haven’t come across this behaviour before with VMs I’ve used in the past (Hezner, OVH, Retrosnub), though I am aware that some providers load kernels from the hypervisor - clearly those that do normally ensure an appropriate /lib/modules is imported on boot.
As for DO, to update the kernel on your machine, you can do so via their control panel, select the new kernel via the settings page, and then reboot from the console. This means you get a newer kernel, but still not the most recent one, the most recent for Ubuntu 12.10 I’ve been able to find on their website is 3.5.0-32, rather than the -39 suffix which appears to have gone into general release.
Overall, this is quite concerning that by usual updating processes, kernel modules can be removed, and the boot process does nothing to make sure a consistent kernel environment is loaded. This affects both the security of the system by preventing iptables loading, and potentially other services which rely on kernel modules. In addition, it does not appear the current kernels are always available in line with the Ubuntu update cycle - with only -32 being available, and not the latest -39. I think I’ll have to consider a switch away from Digital Ocean at this time to an alternative provider.