Ceph orchestrator를 사용하면 손 쉬운 osd 추가/삭제를 진행할 수 있다.
Ref :: https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/
다만 docs에서 설명하는 문서를 보면 osd 추가/삭제는 번거로운것으로 보인다.
osd service만 수정하면 손쉽게 진행할 수 있을 것 같다.
osd 추가
$ vi osd.yml
service_type: osd
service_name: osd
placement:
hosts:
- test-ceph-01
spec:
data_devices:
paths:
- /dev/vda
- /dev/vdb
- /dev/vdc
- /dev/vdd
- /dev/vde
- /dev/vdf
- /dev/vdg
+ - /dev/vdh
filter_logic: AND
objectstore: bluestore
$ sudo ceph orch apply -i osd.yml일정 시간뒤에 자동으로 osd disk로 vdh가 합류하게된다.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
└─sda1 8:1 0 100G 0 part /
sr0 11:0 1 1024M 0 rom
vda 253:0 0 50G 0 disk
└─ceph--b8f83a99--c7d3--41b1--89c3--2a1e2f8faa22-osd--block--3a2dc126--720f--4845--a93e--537b138596ea 252:0 0 50G 0 lvm
vdb 253:16 0 50G 0 disk
└─ceph--800f9575--c4b9--416f--878e--32cc9d8e9f39-osd--block--f744fcae--fe1a--4995--ac13--7d5e1328fc76 252:1 0 50G 0 lvm
vdc 253:32 0 50G 0 disk
└─ceph--a25ae093--5d0b--44bf--992c--71e90e37d1f9-osd--block--43604a92--ed3d--4cda--895e--9a4fc2752d2e 252:2 0 50G 0 lvm
vdd 253:48 0 50G 0 disk
└─ceph--b7583346--3bfb--4c5d--b3a4--5526e4d9efc1-osd--block--1f831518--931f--4710--9a50--436230255eec 252:3 0 50G 0 lvm
vde 253:64 0 50G 0 disk
└─ceph--9141e5a3--2cd1--412f--ba25--0aa5ccf584ab-osd--block--35aed74c--20b7--42c2--bbd4--79764c4de288 252:4 0 50G 0 lvm
vdf 253:80 0 50G 0 disk
└─ceph--617d5492--9ea6--46fd--9749--5b324c4693aa-osd--block--d76453ab--1b21--402c--855d--fcfc9d0496e1 252:5 0 50G 0 lvm
vdg 253:96 0 50G 0 disk
└─ceph--ce784432--e8d1--403b--9c40--57059edc0094-osd--block--b9aa26b6--174f--432b--8a09--94af8babd94d 252:6 0 50G 0 lvm
vdh 253:112 0 50G 0 disk
└─ceph--cf054f6d--b31b--4113--9c07--0cbc9e01727a-osd--block--e867d7ec--8acd--466e--96a3--e641761ead0f 252:7 0 50G 0 lvmosd 수동 삭제
osd 수동삭제는 조금 확인이 필요한 부분이있다.
ceph 용량이라던가, osd id라던가 하는 등 추가보다는 확인을 잘 해야한다.
sudo ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
test-ceph-01 /dev/vda hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdb hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdc hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
r4test-ceph-01 /dev/vdd hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vde hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdf hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdg hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vd1h hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detectedceph 상태
$ sudo ceph -s
cluster:
id: 96e527dc-b6b9-11ef-baef-525400baaf43
health: HEALTH_OK
services:
mon: 1 daemons, quorum test-ceph-01 (age 46h)
mgr: test-ceph-01.fsdjzy(active, since 46h)
osd: 8 osds: 8 up (since 46h), 8 in (since 46h)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
pools: 10 pools, 289 pgs
objects: 4.91k objects, 16 GiB
usage: 38 GiB used, 361 GiB / 400 GiB avail
pgs: 289 active+clean
io:
client: 36 KiB/s wr, 0 op/s rd, 6 op/s wr
[clex@test-ceph-01 ~]$ sudo ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 test-ceph-01 6719M 43.4G 0 3276 0 0 exists,up
1 test-ceph-01 7404M 42.7G 0 3276 0 0 exists,up
2 test-ceph-01 5101M 45.0G 0 0 0 0 exists,up
3 test-ceph-01 5598M 44.5G 0 4096 0 0 exists,up
4 test-ceph-01 5294M 44.8G 16 122k 0 0 exists,up
5 test-ceph-01 3493M 46.5G 0 0 0 0 exists,up
6 test-ceph-01 4786M 45.3G 1 4096 0 0 exists,up
7 test-ceph-01 2785M 45.3G 1 0 0 0 exists,up우선, osd disk의 id를 알아보자.
$ sudo ceph osd metadata -f plain 해당 명령어를 확인해보면, osd metadata를 볼 수 있으며, 각 osd별로 disk의 by-path와 osd id를 확인할 수 있다.
예시로 보면,
$ sudo ceph osd metadata -f plain
[
{
"id": 0,
"arch": "x86_64",
"back_addr": "[v2:10.3.3.175:6804/2637066776,v1:10.3.3.175:6805/2637066776]",
"back_iface": "",
"bluefs": "1",
"bluefs_dedicated_db": "0",
"bluefs_dedicated_wal": "0",
"bluefs_single_shared_device": "1",
"bluestore_bdev_access_mode": "blk",
"bluestore_bdev_block_size": "4096",
"bluestore_bdev_dev_node": "/dev/dm-0",
"bluestore_bdev_devices": "vda",
"bluestore_bdev_driver": "KernelDevice",
"bluestore_bdev_optimal_io_size": "0",
"bluestore_bdev_partition_path": "/dev/dm-0",
"bluestore_bdev_rotational": "1",
"bluestore_bdev_size": "53682896896",
"bluestore_bdev_support_discard": "0",
"bluestore_bdev_type": "hdd",
"bluestore_min_alloc_size": "4096",
"ceph_release": "reef",
"ceph_version": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)",
"ceph_version_short": "18.2.1",
"ceph_version_when_created": "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)",
"container_hostname": "r4test-ceph-01",
"container_image": "10.3.2.170:5000/ceph/ceph@sha256:edd0962107069962a0b73efb38afc9f2577518eb2bf4bfa1981aefac034d357b",
"cpu": "Intel Core Processor (Skylake, IBRS)",
"created_at": "2024-12-10T05:44:42.322990Z",
"default_device_class": "hdd",
"device_ids": "",
"device_paths": "vda=/dev/disk/by-path/pci-0000:00:0a.0",
"devices": "vda",
"distro": "centos",
"distro_description": "CentOS Stream 8",
"distro_version": "8",
"front_addr": "[v2:10.3.3.175:6802/2637066776,v1:10.3.3.175:6803/2637066776]",
"front_iface": "",
"hb_back_addr": "[v2:10.3.3.175:6808/2637066776,v1:10.3.3.175:6809/2637066776]",
"hb_front_addr": "[v2:10.3.3.175:6806/2637066776,v1:10.3.3.175:6807/2637066776]",
"hostname": "r4test-ceph-01",
"journal_rotational": "1",
"kernel_description": "#1 SMP Thu Aug 8 17:47:08 UTC 2024",
"kernel_version": "4.18.0-553.16.1.el8_10.x86_64",
"mem_swap_kb": "0",
"mem_total_kb": "15751292",
"network_numa_unknown_ifaces": "back_iface,front_iface",
"objectstore_numa_unknown_devices": "vda",
"os": "Linux",
"osd_data": "/var/lib/ceph/osd/ceph-0",
"osd_objectstore": "bluestore",
"osdspec_affinity": "None",
"rotational": "1"
},
{
"id": 1,
"arch": "x86_64",
...이런식으로 나온다.
나는 vdh를 지우기위해 이렇게 진행하였다.
## 제거하려는 osd cluster out
$ sudo ceph osd out {osd id}
###예시
$ sudo ceph osd out 7
marked out osd.7.
## 제거하려는 osd crush map 제거
$ sudo ceph osd crush remove osd.{osd id}
###예시
$ sudo ceph osd crush remove osd.7
removed item id 7 name 'osd.7' from crush map
## 제거하려는 osd stop
$ sudo ceph osd stop {osd id}
##예시
$ sudo ceph osd stop 7
stop osd.7.
## osd 제거
$ sudo ceph osd purge {osd id} --yes-i-really-mean-it
###예시
$ sudo ceph osd purge 7 --yes-i-really-mean-it
purged osd.7이제 ceph 상태를 확인해보면 이렇게 나온다.
$ sudo ceph -s
cluster:
id: 2a6a8300-b8f7-11ef-b34a-525400baaf43
health: HEALTH_OK
services:
mon: 1 daemons, quorum test-ceph-01 (age 2h)
mgr: test-ceph-01.edktgu(active, since 2h)
osd: 15 osds: 15 up (since 19s), 15 in (since 44s)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
pools: 9 pools, 257 pgs
objects: 227 objects, 582 KiB
usage: 1.1 GiB used, 749 GiB / 750 GiB avail
pgs: 257 active+clean
$ sudo ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
test-ceph-01 /dev/vda hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdb hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
r4test-ceph-01 /dev/vdc hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdd hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vde hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdf hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
test-ceph-01 /dev/vdg hdd 50.0G No 12m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected