Multi-node install¶
This tutorial will show how to install MicroCeph on three machines, thereby creating a multi-node cluster. For this tutorial, we will utilise physical block devices for storage.
Ensure storage requirements¶
Three OSDs will be required to form a minimal Ceph cluster. This means that, on each of the three machines, one entire disk must be allocated for storage.
The disk subsystem can be inspected with the lsblk command. In this tutorial, the command’s output on each machine looks very similar to what’s shown below. Any output related to possible loopback devices has been suppressed for the purpose of clarity:
lsblk | grep -v loop
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda 252:0 0 40G 0 disk
├─vda1 252:1 0 1M 0 part
└─vda2 252:2 0 40G 0 part /
vdb 252:16 0 20G 0 disk
For the example cluster, each machine will use /dev/vdb for storage.
Prepare the three machines¶
On each of the three machines we will need to:
install the software
disable auto-updates of the software
Below we’ll show these steps explicitly on node-1, which we’ll call the primary node.
Install the most recent stable release of MicroCeph:
sudo snap install microceph
Prevent the software from being auto-updated:
sudo snap refresh --hold microceph
Caution
Allowing the snap to be auto-updated can lead to unintended consequences. In enterprise environments especially, it is better to research the ramifications of software changes before those changes are implemented.
Repeat the above two steps for node-2 and node-3.
Prepare the cluster¶
On node-1 we will now:
initialise the cluster
create registration tokens
Initialise the cluster with the cluster bootstrap command:
sudo microceph cluster bootstrap
Tokens are needed to join the other two nodes to the cluster. Generate these with the cluster add command.
Token for node-2:
sudo microceph cluster add node-2
eyJuYW1lIjoibm9kZS0yIiwic2VjcmV0IjoiYmRjMzZlOWJmNmIzNzhiYzMwY2ZjOWVmMzRjNDM5YzNlZTMzMTlmZDIyZjkxNmJhMTI1MzVkZmZiMjA2MTdhNCIsImZpbmdlcnByaW50IjoiMmU0MmEzYjEwYTg1MDcwYTQ1MDcyODQxZjAyNWY5NGE0OTc4NWU5MGViMzZmZGY0ZDRmODhhOGQyYjQ0MmUyMyIsImpvaW5fYWRkcmVzc2VzIjpbIjEwLjI0Ni4xMTQuMTE6NzQ0MyJdfQ==
Token for node-3:
sudo microceph cluster add node-3
eyJuYW1lIjoibm9kZS0zIiwic2VjcmV0IjoiYTZjYWJjOTZiNDJkYjg0YTRkZTFiY2MzY2VkYTI1M2Y4MTU1ZTNhYjAwYWUyOWY1MDA4ZWQzY2RmOTYzMjBmMiIsImZpbmdlcnByaW50IjoiMmU0MmEzYjEwYTg1MDcwYTQ1MDcyODQxZjAyNWY5NGE0OTc4NWU5MGViMzZmZGY0ZDRmODhhOGQyYjQ0MmUyMyIsImpvaW5fYWRkcmVzc2VzIjpbIjEwLjI0Ni4xMTQuMTE6NzQ0MyJdfQ==
Keep these tokens in a safe place. They’ll be needed in the next step.
Note
Tokens are randomly generated; each one is unique.
Join the non-primary nodes to the cluster¶
The cluster join command is used to join nodes to a cluster.
On node-2, add the machine to the cluster using the token assigned to node-2:
sudo microceph cluster join eyJuYW1lIjoibm9kZS0yIiwic2VjcmV0IjoiYmRjMzZlOWJmNmIzNzhiYzMwY2ZjOWVmMzRjNDM5YzNlZTMzMTlmZDIyZjkxNmJhMTI1MzVkZmZiMjA2MTdhNCIsImZpbmdlcnByaW50IjoiMmU0MmEzYjEwYTg1MDcwYTQ1MDcyODQxZjAyNWY5NGE0OTc4NWU5MGViMzZmZGY0ZDRmODhhOGQyYjQ0MmUyMyIsImpvaW5fYWRkcmVzc2VzIjpbIjEwLjI0Ni4xMTQuMTE6NzQ0MyJdfQ==
On node-3, add the machine to the cluster using the token assigned to node-3:
sudo microceph cluster join eyJuYW1lIjoibm9kZS0zIiwic2VjcmV0IjoiYTZjYWJjOTZiNDJkYjg0YTRkZTFiY2MzY2VkYTI1M2Y4MTU1ZTNhYjAwYWUyOWY1MDA4ZWQzY2RmOTYzMjBmMiIsImZpbmdlcnByaW50IjoiMmU0MmEzYjEwYTg1MDcwYTQ1MDcyODQxZjAyNWY5NGE0OTc4NWU5MGViMzZmZGY0ZDRmODhhOGQyYjQ0MmUyMyIsImpvaW5fYWRkcmVzc2VzIjpbIjEwLjI0Ni4xMTQuMTE6NzQ0MyJdfQ==
Add storage¶
Warning
This step will remove the data found on the target storage disks. Make sure you don’t lose data unintentionally.
On each of the three machines, use the disk add command to add storage:
sudo microceph disk add /dev/vdb --wipe
Adjust the above command per machine according to the storage disks at your disposal. You may also provide multiple disks as space separated arguments.
sudo microceph disk add /dev/vdb /dev/vdc /dev/vdd --wipe
Or use the –all-available flag to enlist all physical devices available on the machine.
sudo microceph disk add --all-available --wipe
Check MicroCeph status¶
On any of the three nodes, the status command can be invoked to check the status of MicroCeph:
sudo microceph status
MicroCeph deployment summary:
- node-01 (10.246.114.11)
Services: mds, mgr, mon, osd
Disks: 1
- node-02 (10.246.114.47)
Services: mds, mgr, mon, osd
Disks: 1
- node-03 (10.246.115.11)
Services: mds, mgr, mon, osd
Disks: 1
Machine hostnames are given along with their IP addresses. The MDS, MGR, MON, and OSD services are running and each node is supplying a single disk, as expected.
Manage the cluster¶
Your Ceph cluster is now deployed and can be managed by following the resources found in the Howto section.
The cluster can also be managed using native Ceph tooling if snap-level commands are not yet available for a desired task:
ceph status
This gives:
cluster:
id: cf16e5a8-26b2-4f9d-92be-dd3ac9602ebf
health: HEALTH_OK
services:
mon: 3 daemons, quorum node-01,node-02,node-03 (age 14m)
mgr: node-01(active, since 43m), standbys: node-02, node-03
osd: 3 osds: 3 up (since 4s), 3 in (since 6s)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 336 MiB used, 60 GiB / 60 GiB avail
pgs: 100.000% pgs unknown
1 unknown