* Home :: Get a random password :: Get a multi-word password :: What is my IP? (v4 | v6)
* My projects

Using Apache CloudStack and KVM with iSCSI, FCoE or any other shared block devices

Tags: howto,cloudstack,kvm,iscsi,fcoe

Date: 20250514

Objective:

Consume shared block storage with CloudStack and KVM.

Introduction:

By far the most common storage for CloudStack has been NFS - it's very stable and used properly is also very fast. This is the #1 recommendation.
However there is now a trend with so many "VMWare refugees" around that all the hardware people have at their disposal is a block storage SAN, either iSCSI, FCoE etc.
So unless you use one of the managed storage platforms which have bothered to create a CloudStack plugin, then you don't have too many options.

So that basically leaves you with 2 choices: a cluster file system like OCFS2 or CLVM (cluster lvm). Alas the implementation used by CloudStack with CLVM is quite old so really there's only one option: a clustered filesystem.


Moving on...

Assumptions:

- your virtualisation platform of choice is CloudStack with KVM
- your storage appliance can only do generic block storage and is already exporting such a volume via iSCSI or similar
- you are possibly coming from VMWare

Recommendations (for now):

- use OCFS2
- use Oracle Linux 9 with UEK kernel
- use per-cluster OCFS2 primary storages and keep your CloudStack clusters 32 hosts max

Limitations and warnings:

- Oracle has capped an OCFS2 cluster to maximum 32 nodes - performance is expected to take a hit beyond that point.
- OCFS2 - a least in the past - is known to introduce some instability here and there with people complaining it can induce a kernel panic in certain situations and so on,
so make sure you test very thoroughly before putting it in production!

- So why Oracle Linux and OCFS2?
The explanation is very practical - first of all OCFS2 is to Oracle what VMFS is to VMWare, a cluster-aware filesystem, among others.
Oracle has a vested interest in OCFS2 peforming as best it can for their own uses (Oracle DBs), so they are the only party afaik that is making sure this works well on their distro.
RedHat has obsoleted it from RHEL in favour of their (slower and more complicated) GFS2 filesystem - which btw has disappeared from EL10! - and Ubuntu pays minimal attention to it.
Make sure you are using their UEK kernel which brings support for it and all sorts of fixes and improvements, as well.

Moving on I'm going to assume you have followed the advice and installed Oracle Linux 9 - although the commands might work well on other EL clones, it is not a guarantee.



I'm not going to cover exporting an iSCSI volume, that will be different from case to case, from one appliance to another.

I'm also not - at least at this time - going to cover setting up multipathing, you'll have to do that by yourself.

Howto:

We'll start with 2 CloudStack KVM hypervisors, let's call them hostA 192.168.122.10 and hostB 192.168.122.11, although in reality you may use proper hostnames for them,
also making sure they are using a network dedicated for storage traffic.

On BOTH hypervisors run the following commands as root to set up OCFS2:


    		# disable the firewalld, we won't need it where we're going
    		systemctl stop firewalld
    		systemctl mask firewalld
    		
    		# install the software
    		dnf -y install ocfs2-tools
    		
    		# add and verify our cluster
    		o2cb add-cluster cloudstack
    		o2cb list-cluster cloudstack
    		
    		# add the 2 hypervisors (nodes) to the cluster
    		o2cb add-node cloudstack hostA --ip 192.168.122.10
    		o2cb add-node cloudstack hostB --ip 192.168.122.11
    		
    		# verify the cluster again, making sure it's displaying correct info
    		o2cb list-cluster
    		
    		# initialise the cluster, making sure to answer the questions correct, 
    		#especially the name of the cluster to starton boot, it should be "cloudstack" in our case
    		/sbin/o2cb.init configure
    		
    		# check the state of the cluster
    		/sbin/o2cb.init status
    		
    		# enable the required services
    		systemctl enable --now o2cb
    		systemctl enable --now ocfs2
    		
    		# tell the system to reboot in 30 seconds should a kernel panic occur and to also trigger a panic in case of an "oops" (a type of non-fatal error)
    		# why? because letting the host hang without rebooting could lead to OCFS2 instability, filesystem corruption, split-brain etc, you really want a quick, clean reset
    		printf 'kernel.panic = 30\nkernel.panic_on_oops = 1' | sudo tee -a /etc/sysctl.d/99-sysctl.conf ; sysctl -p

Now on ONLY a single node create the filesystem. I'll assume your iSCSI or FCoE device is present as /dev/sdb.


    		# instruct mkfs to create a filesystem adequate for storing VM files
    		# I'm assuming you'll want to have a maximum number of nodes (32) using this filesystem, specify it here, you can't easily grow this later
    		# "-T vmstore" basically tells the filesystem to use a block size of 4k and a cluster size of 128k
    		# it would be a good idea to take some time and research best values for your workload here, don't blindly copy/paste from the Internet
    		mkfs.ocfs2 -T vmstore -N 32 -L cloudstack /dev/sdb


Now again on BOTH hypervisors make sure things are still working fine and you can mount the filesystem:



    		# set it up in /etc/fstab so it gets mounted at boot
    		mkdir /mnt/ocfs2-cloudstack
    		echo '/dev/sdb /mnt/ocfs2-cloudstack ocfs2   _netdev,defaults        0 0' | sudo tee -a /etc/fstab
    		
    		# verify it actually can be mounted and do that now
    		systemctl daemon-reload
		mount -a
		df -h /mnt/ocfs2-cloudstack
		/sbin/o2cb.init status
		
		# on both hypervisors create files in the new mount point and make sure you can read them from the other node
		echo "test from `hostname -f`" > /mnt/ocfs2-cloudstack/`hostname -s`
		
		# optional, but recommended, reboot both nodes when you are done, make sure after reboot OCFS2 starts properly and the mount point is usable
		reboot


Now in CloudStack just add a new primary storage pool of Protocol SharedMountPoint and Path /mnt/ocfs2-cloudstack.
Think of adding this primary storage with a Cluster Scope, considering it'll be limited to 32 nodes which may well be also the number of hypervisors in your KVM cluster, up to you.


That's it, good luck!
(c)The Nux(tm) ;-)

No CSS, no Javascript, no some HTML.
No ads, no tracking, no cookies.

Optimised for Dillo browser, great in Lynx, too :)