Kubernetes: Extension of disk in Rook Ceph

In my previous blog, I’ve introduced the storage solution Rook Ceph for Kubernetes. We saw that in our architecture we had 3 workers dedicated for storage using Ceph file system. Each worker had 2 disks of 100 GB and we saw that from the storage point of view one OSD equals one disk. With these 6 disks we had a Ceph cluster with a total capacity of 600 GB. This disk size was fine for our initial tests but we wanted to extend them to 1.5 TB each to be closer to a production-like design.

The pools of our Ceph cluster use the default configuration of 3 replicas for our placement groups (pgs) and permit the loss of 1 replica. The Ceph cluster will still operate with only 2 replicas. By knowing this, we can then control this disk extension by working with one worker at a time. We will both keep our Ceph cluster available to users and avoid the risk of data corruption. Data corruption (or even loss) is what could happen if we were doing this disk extension abruptly without properly preparing our cluster to this operation.

Preparing the Ceph cluster before the disk extension

First let’s have a look at the status of our Ceph cluster:

Then to the configuration of our OSDs in this cluster:

We will detail the disk extension procedure on worker1. The output above shows worker1 disks with the labels osd.0 and osd.3.

The first thing to do is to stop the Ceph operator and delete the 2 OSD deployments related to worker1:

As the operator is stopped, these deployments will not be automatically re-created. We can then continue by moving out osd.0 and osd.3 from this Ceph cluster:

All references to these 2 OSDs have now been removed from this Ceph cluster. We then wait for the cluster to recover from this removal and monitor its status:

You can use the command ceph with -s (same as status) in order to follow this redistribution (use it in combination with the watch command from linux to follow these changes in live). You can notice now that only 4 OSDs are “in” because we removed 2 of them. The cluster is ready when all pgs are in the state active+clean.

Let’s check the OSDs status:

Both disk references have been properly removed from this Ceph cluster. We can now safely proceed with the extension of disks.

Extension of disks

In our environment, worker nodes used for storage are VMware virtual machines. Extending their disks is just a parameter to change. Once done, we can then format and erase each disk as follows:

We can now bring back those disks into our Ceph cluster.

Bringing back those extended disks

This step shows all the power of using Rook for operating Ceph:

You can use the command ceph with -w to monitor live the health of this Ceph cluster. And just like that, by scaling up the operator, the Ceph cluster becomes up and running after a few minutes:

We can see the new size of both disks! All the steps we did to remove those disks have been automatically reverted back. Brilliant!

We just had to repeat the same procedure with the 2 other workers used for storage. We have now a Ceph cluster with 9 TB!