Bare-Metal Ceph as the RBD Backend for OpenStack

OpenStack needs block storage. My first attempt ran Ceph in containers on the three compute nodes, sharing disks with the host OS. It worked, technically — until etcd on top of it fsync’d in 1–2 seconds and Raft quorum timeouts killed a three-node Kubernetes cluster four times in a row.

This post is the replacement: Ceph on dedicated hardware, installed natively, with its own disks and its own 10G path.

The design decision: bare metal, not cephadm

cephadm is container-first by design and is the officially recommended install path. I deliberately did not use it.

With a single host and a lab that I break often, “one less moving part on the storage layer” won. Native it is.

Replication is size=2 with the CRUSH failure domain at OSD level, because a single host cannot survive a host failure anyway — two copies on two different disks protects against the failure mode that actually happens here (a disk dying), and keeps usable capacity at ~50%.

Hardware and topology

Client traffic and replication both ride the 10G path so VM and etcd I/O never crosses the 1G uplink. That single choice is what fixed the etcd latency problem — measured random 4K write went from ~481 IOPS to 4775 IOPS at p50 1.48 ms once boot volumes landed on dedicated disks over 10G.

Pool names were kept identical to the old cluster so existing Cinder volume types (rbd-1, rbd-rep1) kept working after cut-over.

Phase 0 — Preflight

sudo -i
lsb_release -a            # expect Ubuntu 24.04
nproc; free -g            # 48 C / ~125G
lsblk -o NAME,SIZE,MODEL,TRAN

# Ceph requires sane time
systemctl enable --now chrony && chronyc sources -v

# single /etc/hosts line, no 127.0.1.1 entry for the hostname
echo "10.10.10.7 blade-04" >> /etc/hosts

That 127.0.1.1 detail matters: Ubuntu ships it by default, and Ceph’s hostname resolution will pick the loopback address for the monitor, producing a cluster that reports healthy on one node and unreachable from everywhere else.

Phase 1 — Storage network

sudo tee /etc/netplan/99-storage.yaml > /dev/null <<'EOF'
network:
  version: 2
  ethernets:
    ens1:
      mtu: 8950
      dhcp4: false
      addresses: [10.10.70.7/24]
EOF
sudo netplan apply

ip -br addr show ens1          # 10.10.70.7/24
cat /sys/class/net/ens1/mtu    # 8950
ping -M do -s 8922 -c 5 10.10.70.1   # zero loss once the switch port is up

If the switch port is a tagged access port for a VLAN, add a vlans: block with a sub-interface instead of putting the address on ens1 directly. Getting this wrong is silent: the interface comes up, ip addr looks right, and nothing routes.

MTU 8950 was chosen because the switch supports ~8950 max. Verify with a -M do ping at 8922 bytes — if that passes, the whole path agrees on the MTU. If it does not, and you deploy anyway, Ceph will work until the first large recovery operation, then stall.

Phase 2 — Install Ceph

wget -q -O- 'https://download.ceph.com/keys/release.asc' | \
  sudo gpg --dearmor -o /usr/share/keyrings/ceph-release.gpg
echo "deb [signed-by=/usr/share/keyrings/ceph-release.gpg] \
https://download.ceph.com/debian-tentacle/ noble main" | \
  sudo tee /etc/apt/sources.list.d/ceph.list

sudo apt update
sudo apt install -y ceph ceph-mon ceph-mgr ceph-osd
ceph --version        # 20.2.4 tentacle (stable)

Failure #1: the first install silently pulled an older release because /usr/share/keyrings/ceph-release.gpg was 0 bytes — the wget | gpg pipeline had failed and apt ignored the repository without complaint, falling back to the Ubuntu universe package. If ceph --version does not show the release you asked for, check the keyring size before anything else.

Phase 3 — Bootstrap MON + MGR by hand

No cephadm bootstrap, so the monmap is built manually.

FSID=$(uuidgen)
HOST=$(hostname -s)

sudo tee /etc/ceph/ceph.conf > /dev/null <<EOF
[global]
fsid = $FSID
mon initial members = $HOST
mon host = 10.10.70.7
public network = 10.10.70.0/24
cluster network = 10.10.70.0/24
osd pool default size = 2
osd pool default min size = 1
osd crush chooseleaf type = 0
EOF

# keyrings: mon, admin, bootstrap-osd, bootstrap-mgr
sudo -u ceph ceph-authtool --create-keyring /tmp/ceph.mon.keyring \
  --gen-key -n mon. --cap mon 'allow *'
sudo -u ceph ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring \
  --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' \
  --cap mgr 'allow *' --cap mds 'allow *'
sudo -u ceph ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring \
  --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow rwx'
sudo -u ceph ceph-authtool --create-keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring \
  --gen-key -n client.bootstrap-mgr --cap mon 'profile bootstrap-mgr'

sudo -u ceph ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
sudo -u ceph ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
sudo chown -R ceph:ceph /etc/ceph /var/lib/ceph /tmp/ceph.mon.keyring

# monmap + mon mkfs
sudo -u ceph monmaptool --create --add $HOST 10.10.70.7 --fsid $FSID /tmp/monmap
sudo -u ceph mkdir -p /var/lib/ceph/mon/ceph-$HOST
sudo -u ceph ceph-mon --mkfs -i $HOST --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
sudo systemctl enable --now ceph-mon@$HOST

# mgr
sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-$HOST
sudo ceph auth get-or-create mgr.$HOST mon 'allow profile mgr' osd 'allow *' mds 'allow *' \
  -o /var/lib/ceph/mgr/ceph-$HOST/keyring
sudo chown -R ceph:ceph /var/lib/ceph/mgr
sudo systemctl enable --now ceph-mgr@$HOST

sudo ceph -s    # mon quorum + mgr active

Three failures live in this phase, all of them instructive:

Failure #2 — ceph-mon fails with Permission denied. The mon data directory was root-owned. chown -R ceph:ceph /var/lib/ceph/mon before --mkfs.

Failure #3 — ceph -s returns nothing although ceph-mon is running. The monitor had bound the management IP while mon host in ceph.conf pointed at the storage IP. A monitor listening on an address the client cannot reach looks exactly like a dead monitor. Rebuild the monmap with --add <host> <storage-ip> --clobber and re-run --mkfs.

Failure #4 — monmaptool refuses to overwrite an existing map. It needs --clobber. And be careful with the cleanup: I once pasted a rm -rf and a mkdir merged into one line and deleted the fresh mon directory. Run destructive lines one at a time.

Phase 4 — Add the OSDs

Never touch sda (the system disk).

lsblk -o NAME,SIZE,MODEL,TRAN        # /dev/sdb, /dev/sdc

for d in sdb sdc; do
  sudo ceph-volume lvm zap /dev/$d --destroy
  sudo ceph-volume lvm create --bluestore --data /dev/$d
done

sudo ceph -s
sudo ceph osd tree

Failure #5: zap --destroy refused with Device has partitions — the disks still carried linux_raid_member metadata from their previous life in a Harvester md array, and ceph-volume will not destroy an md member it cannot prove is inactive. Confirm with cat /proc/mdstat (no active array), then sgdisk --zap-all /dev/sdb /dev/sdc.

The second OSD failure is an ordering trap: ceph-volume lvm create before Phase 3 completes gives No valid ceph configuration file was loaded. Bootstrap first, OSDs second.

Phase 5 — Pools and OpenStack users

for p in images:16 volumes:32 vms:16 vms-rep1:8 backups:8; do
  name=${p%%:*}; pg=${p##*:}
  ceph osd pool create $name $pg
  ceph osd pool application enable $name rbd
done

ceph osd pool set vms-rep1 size 1
ceph osd pool set vms-rep1 min_size 1

ceph auth get-or-create client.glance mon 'profile rbd' \
  osd 'profile rbd pool=images' mgr 'profile rbd pool=images'
ceph auth get-or-create client.cinder mon 'profile rbd' \
  osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=vms-rep1' \
  mgr 'profile rbd pool=volumes'
ceph auth get-or-create client.cinder-backup mon 'profile rbd' \
  osd 'profile rbd pool=backups' mgr 'profile rbd pool=backups'
ceph auth get-or-create client.nova mon 'profile rbd' \
  osd 'profile rbd pool=vms, profile rbd pool=volumes' mgr 'profile rbd pool=vms'

Failure #6: ceph osd pool set vms-rep1 size 1 → EPERM. Ceph 20 refuses single-copy pools by default. The override is two commands, and the second one is deliberately awkward to type:

ceph config set global mon_allow_pool_size_one true
ceph osd pool set vms-rep1 size 1 --yes-i-really-mean-it

The PG count warning (too many PGs per OSD) on a 2-OSD host is expected and harmless in a lab; the autoscaler will settle it.

Phase 6 — Point Kolla at the new cluster

Keyrings and ceph.conf get staged into the Kolla config tree on the deploy host, then only the RBD consumers are redeployed:

C=~/kolla-ansible/etc/config
mkdir -p $C/glance $C/cinder/cinder-volume $C/cinder/cinder-backup $C/nova

cp ceph.conf                          $C/ceph.conf
cp ceph.client.glance.keyring         $C/glance/
cp ceph.client.cinder.keyring         $C/cinder/
cp ceph.client.cinder.keyring         $C/cinder/cinder-volume/
cp ceph.client.cinder-backup.keyring  $C/cinder/cinder-backup/
cp ceph.client.nova.keyring           $C/nova/
cp ceph.client.cinder.keyring         $C/nova/      # nova-compute reads the volumes pool too

kolla-ansible deploy -i inventory -t glance,cinder,nova --configdir ~/kolla-ansible/etc

Then the guard that has caught more real outages than any other command in this project:

docker exec glance_api  sh -c "wc -c /etc/kolla/glance-api/ceph/ceph.conf"
docker exec cinder_volume sh -c "wc -c /etc/kolla/cinder-volume/ceph/ceph.conf"

If either reports 0, the container cannot talk to Ceph and will fail in a way that looks like an authentication problem. Re-copy ceph.conf into /etc/kolla/<service>/ceph/ on every node and restart the container.

Phase 7 — Verify

ceph -s; ceph osd tree; ceph osd perf

# 4M sequential write — matches the practical ceiling of the public path
rbd bench --io-type write --io-size 4M --io-total 256M --io-threads 4 volumes/bench
rbd rm volumes/bench

# OpenStack side
openstack volume create --image <img-id> --size 10 test-new-ceph --wait
openstack volume show test-new-ceph      # status: available
openstack server create --flavor m1.small --volume test-new-ceph --key-name k8s-key test-boot

Known limits

This is a staging design and it should be described as one. A single host means:

• One host failure takes every volume with it. size=2 protects against one disk failure, not against the node.
• The mon is a single daemon — quorum of one means no automatic failover, and a mon restart is a storage outage.
• Two HDD OSDs cap random IOPS around 300–480 regardless of tuning, because that is the physical seek rate.

Scale-out is straightforward — install the packages on a new node, copy ceph.conf and the bootstrap keys, ceph mon add, then ceph-volume lvm create. The cluster network already exists for replication.

The one thing I would not skip on a rebuild: put the mon on a small SSD, or accept that every mon restart is a full storage outage for the duration.