ceph는 osd로 사용하는 disk가 동일한 하드웨어, 동일한 크기 일 때 가장 잘 작동한다.
Ceph는 풀 전체에 걸쳐 동일한 하드웨어를 사용할 때 가장 잘 작동합니다.
크기가 다른 드라이브를 추가한 후 그에 따라 가중치를 조정할 수 있습니다.
하지만 최상의 성능을 위해서는 유형과 크기가 동일한 드라이브로 구성된 CRUSH 계층 구조를 고려해 보세요.
기존 호스트에 크기가 큰 드라이브를 균일하게 추가하는 것이 더 좋습니다.
새 드라이브가 추가될 때마다 작은 드라이브를 교체하면서 점진적으로 추가할 수 있습니다.Docs :: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/
그렇다면 만약에 하드웨어 상 다른 크기의 osd disk가 붙게되면 문제가 발생할텐데...
이 경우엔 어떻게해야할까?
같은 하드웨어가 아니라면 성능저하가 발생할 수 있다.
그렇다면 ceph의 crushmap을 이용해서 pool별로 osd를 지정하면 되지않을까?
대충 계획을 세우면 아래 그림과 같다.
osd1,2,3은 기존 osd... Lun_osd는 신규 osd인데 기존 osd와 크기가 다르다.
우선 구성된 ceph는 이렇게되어있다.
$ sudo ceph -s
cluster:
id: 764245aa-6c1c-11f0-b292-525400d78c1e
health: HEALTH_OK
services:
mon: 3 daemons, quorum storage1,storage3,storage2 (age 3h)
mgr: storage1.kfgbdv(active, since 3h), standbys: storage3.mjvmvg, storage2.gsmjxs
osd: 9 osds: 9 up (since 3h), 9 in (since 3h)
rgw: 3 daemons active (3 hosts, 1 zones)
data:
pools: 10 pools, 289 pgs
objects: 4.75k objects, 15 GiB
usage: 47 GiB used, 8.1 TiB / 8.2 TiB avail
pgs: 289 active+clean
io:
client: 36 KiB/s wr, 0 op/s rd, 4 op/s wr 우선 osd를 추가하자.
나는 ceph 오케스트레이션을 사용하니까 쉽게 추가하자.
$ cat osd.yaml
service_type: osd
service_name: osd.default_osd
placement:
hosts:
...
spec:
data_devices:
paths:
- /dev/sdb
- /dev/sdc
- /dev/sdd
- /dev/sde
filter_logic: AND
objectstore: bluestore
$ vi osd.yaml
$ sudo ceph orch apply -i osd.yaml
Scheduled osd.default_osd update...배포했다면 추가된 osd disk를 확인해보자.
$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 8.38066 root default
-5 2.92375 host 1
1 hdd 0.90948 osd.1 up 1.00000
...
9 hdd 0.19530 osd.9 up 1.00000
...osd 9번이 새로 추가된 osd이다.
그럼 이제 cursh rule을 적용해보자.
우선 CLASS.. 즉 ceph 가 disk를 인식했을 때 CALSS별로 나누는데 나는 VM환경에서 disk를 추가했으니 HDD로 인식되었다.
이걸 LUN이라는 이름으로 다시 정의한다.
우선 CLASS의 HDD를 제거하자.
$ sudo ceph osd crush rm-device-class osd.9
done removing class of osd(s): 9
$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 8.38066 root default
-5 2.92375 host
1 hdd 0.90948 osd.1 up 1.00000
...
9 0.19530 osd.9 up 1.00000 1.00000이제 lun이라는 class명을 넣자
$ sudo ceph osd crush set-device-class lun osd.9
set osd(s) 9 to class 'lun'
$ sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 8.38066 root default
1 hdd 0.90948 osd.1 up 1.00000
9 lun 0.19530 osd.9 up 1.00000 1.00000 그리고 crush rule을 생성하자.
우선 HDD용도의 rule
$ sudo ceph osd crush rule create-replicated hdd_rule default host hdd그리고 lun용도의 rule
$ sudo ceph osd crush rule create-replicated lun_rule default host lun
$ sudo ceph osd crush rule ls
replicated_rule
hdd_rule
lun_rule그럼 위에서 계획했던대로 기존 ceph pool들에 대해선 모두 hdd rule을 적용.
신규 pool을 생성해서 거기엔 lun rule을 적용하자.
hdd rule 적용
$ sudo ceph osd pool set kube crush_rule hdd_rule
set pool 1 crush_rule to hdd_rule
$ sudo ceph osd pool set images crush_rule hdd_rule
set pool 2 crush_rule to hdd_rule
$ sudo ceph osd pool set volumes crush_rule hdd_rule
set pool 3 crush_rule to hdd_rule
$ sudo ceph osd pool set vms crush_rule hdd_rule
set pool 4 crush_rule to hdd_rule
$ sudo ceph osd pool set .mgr crush_rule hdd_rule
set pool 5 crush_rule to hdd_rule
$ sudo ceph osd pool set .rgw.root crush_rule hdd_rule
set pool 6 crush_rule to hdd_rule
$ sudo ceph osd pool set default.rgw.log crush_rule hdd_rule
set pool 7 crush_rule to hdd_rule
$ sudo ceph osd pool set default.rgw.control crush_rule hdd_rule
set pool 8 crush_rule to hdd_rule
$ sudo ceph osd pool set default.rgw.meta crush_rule hdd_rule
set pool 9 crush_rule to hdd_rule
$ sudo ceph osd pool set default.rgw.buckets.index crush_rule hdd_rule
set pool 10 crush_rule to hdd_rule이제 신규 pool을 생성해서 lun rule을 적용하자.
$ sudo ceph osd pool create lun-pool
pool 'lun-pool' created
$ sudo ceph osd pool application enable lun-pool rbd
$ sudo ceph osd pool set lun-pool pg_num 128
set pool 11 pg_num to 128
$ sudo ceph osd pool set lun-pool pgp_num 128
set pool 11 pgp_num to 128
enabled application 'rbd' on pool 'lun-pool'
$ sudo ceph osd pool set lun-pool crush_rule lun_rule
set pool 11 crush_rule to lun_rule신규 ceph pool 이름은 lun-pool, pg는 각 128을 할당
그리고 나는 이 pool을 openstack에서 사용할 것이기에 cinder.j2파일에 lun-pool에 대해서 추가
cinder:
DEFAULT:
rpc_response_timeout: 300
verify_glance_signatures: disabled
glance_ca_certificates_file: /etc/cinder/certs/ca.crt
debug: true
enable_force_upload: true
allowed_direct_url_schemes: cinder
cinder_internal_tenant_project_id: internal_cinder
cinder_internal_tenant_user_id: internal_cinder
+ enabled_backends: "{{ cinder.enabled_backends }},lun-pool"
...
rbd1:
volume_driver: cinder.volume.drivers.rbd.RBDDriver
volume_backend_name: rbd1
rbd_ceph_conf: "/etc/ceph/ceph.conf"
rbd_flatten_volume_from_snapshot: false
rbd_max_clone_depth: 5
rbd_store_chunk_size: 8
rados_connect_timeout: -1
rbd_user: "cinder"
rbd_pool: "volumes"
rbd_secret_uuid: "{{ ceph_secret_uuid }}"
rbd_exclusive_cinder_pool: true
+ lun-pool:
+ volume_driver: cinder.volume.drivers.rbd.RBDDriver
+ volume_backend_name: lun-pool
+ rbd_ceph_conf: "/etc/ceph/ceph.conf"
+ rbd_flatten_volume_from_snapshot: false
+ rbd_max_clone_depth: 5
+ rbd_store_chunk_size: 8
+ rados_connect_timeout: -1
+ rbd_user: "cinder"
+ rbd_pool: "lun-pool"
+ rbd_secret_uuid: "{{ ceph_secret_uuid }}"
+ rbd_exclusive_cinder_pool: trueopenstack에서 본다면?
openstack volume type list
+--------------------------------------+-------------+-----------+
| ID | Name | Is Public |
+--------------------------------------+-------------+-----------+
| 1334144b-b57b-4297-bf7c-04133cd3b48a | lun-pool | True |
| 8f5e09dc-5517-42e9-8e5c-bb3b554ab1ec | rbd1 | False |
| c2a562a6-7667-4d75-a37d-af463522ff0d | __DEFAULT__ | True |
+--------------------------------------+-------------+-----------+volume을 만들어보자.
openstack volume create --type lun-pool --size 100 test-volceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 8.2 TiB 8.1 TiB 48 GiB 48 GiB 0.58
lun 200 GiB 200 GiB 91 MiB 91 MiB 0.04
TOTAL 8.4 TiB 8.3 TiB 48 GiB 48 GiB 0.56
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
kube 1 32 13 GiB 4.67k 39 GiB 0.49 2.6 TiB
images 2 32 20 MiB 9 41 MiB 0 4.0 TiB
volumes 3 32 21 MiB 19 43 MiB 0 4.0 TiB
vms 4 32 90 MiB 39 271 MiB 0 2.6 TiB
.mgr 5 1 769 KiB 2 2.3 MiB 0 2.6 TiB
.rgw.root 6 32 3.9 KiB 8 96 KiB 0 2.6 TiB
default.rgw.log 7 32 3.6 KiB 209 408 KiB 0 2.6 TiB
default.rgw.control 8 32 0 B 8 0 B 0 2.6 TiB
default.rgw.meta 9 32 3.7 KiB 7 79 KiB 0 2.6 TiB
default.rgw.buckets.index 10 32 21 KiB 22 63 KiB 0 2.6 TiB
lun-pool 11 32 3.2 KiB 5 36 KiB 0 63 GiB그럼 lun-pool에 대해서 어떤 pg가 저장되는지보자.
$ sudo ceph pg ls-by-pool lun-pool
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING
11.0 0 0 0 0 0 0 0 0 0 active+clean+remapped 10m 0'0 780:603 [9]p9 [9,6,7]p9 2025-07-29T07:38:11.979783+0000 2025-07-29T07:35:42.794358+0000 1 periodic scrub scheduled @ 2025-07-30T08:20:03.392918+0000
11.1 0 0 0 0 0 0 0 0 0 active+clean+remapped 10m 0'0 780:591 [9]p9 [9,6,7]p9 2025-07-29T07:37:43.976596+0000 2025-07-29T07:35:42.794358+0000 1 periodic scrub scheduled @ 2025-07-30T08:28:40.862471+0000
11.2 0 0 0 0 0 0 0 0 0 active+clean+remapped 10m 0'0 780:591 [9]p9 [9,6,7]p9 2025-07-29T07:38:40.844090+0000 2025-07-29T07:35:42.794358+0000 1 periodic scrub scheduled @ 2025-07-30T17:20:00.430757+0000
11.3 1 0 2 0 0 0 0 1 0 active+clean+remapped 10m 779'1 780:592 [9]p9 [9,6,7]p9 2025-07-29T07:38:39.859058+0000 2025-07-29T07:35:42.794358+0000 1 periodic scrub scheduled @ 2025-07-30T12:47:42.557726+0000
...- up된 osd는 osd.9 번만 사용하고있다.
- acting에는 9,6,7이라는 3개의 osd가 잡혀있는데 이건 replica size가 3인데 lun에 잡힌 osd가 1개여서 replica 복제를 위해서 사용하는 것으로 이렇게되면 lun pool에는 osd9번만 사용하는게 아니니 osd를 추가해야한다.
우선 pg별로 어떤 데이터가 저장되어있는지 확인해보자.
$ sudo rbd ls -p lun-pool --format=json | jq
[
"299b8cfc-970b-4e57-b396-e0d98575c04f"
]저 이미지는 뭐지?
$ sudo rbd info -p lun-pool 299b8cfc-970b-4e57-b396-e0d98575c04f
rbd image '299b8cfc-970b-4e57-b396-e0d98575c04f':
size 100 GiB in 12800 objects
order 23 (8 MiB objects)
snapshot_count: 0
id: 8e1f878d50fc
block_name_prefix: rbd_data.8e1f878d50fc
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 29 17:42:25 2025
access_timestamp: Tue Jul 29 17:42:25 2025
modify_timestamp: Tue Jul 29 17:42:25 2025
아 openstack에서 만든 volume이구나.
그럼 osd가 lun-pool에 할당된 lun class만 사용하도록 하기위해서 replica3을 충족시켜보자.
osd 추가 후 lun class 설정 및 pool할당 완료
$ sudo ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 hdd 0.90948 1.00000 931 GiB 6.8 GiB 6.2 GiB 41 KiB 674 MiB 924 GiB 0.73 0.86 89 up
4 hdd 0.90948 1.00000 931 GiB 7.8 GiB 6.3 GiB 31 KiB 1.5 GiB 924 GiB 0.84 0.98 88 up
7 hdd 0.90948 1.00000 931 GiB 9.2 GiB 7.7 GiB 23 KiB 1.5 GiB 922 GiB 0.99 1.16 91 up
9 lun 0.19530 1.00000 200 GiB 5.1 GiB 5.0 GiB 3 KiB 106 MiB 195 GiB 2.57 3.00 35 up
0 hdd 0.90948 1.00000 931 GiB 7.9 GiB 6.3 GiB 31 KiB 1.5 GiB 923 GiB 0.84 0.99 82 up
3 hdd 0.90948 1.00000 931 GiB 6.5 GiB 5.9 GiB 44 KiB 709 MiB 925 GiB 0.70 0.82 102 up
6 hdd 0.90948 1.00000 931 GiB 10 GiB 9.2 GiB 35 KiB 918 MiB 921 GiB 1.08 1.26 102 up
10 lun 0.19530 1.00000 200 GiB 3.0 GiB 2.9 GiB 0 B 87 MiB 197 GiB 1.52 1.77 40 up
2 hdd 0.90948 1.00000 931 GiB 7.4 GiB 6.8 GiB 42 KiB 616 MiB 924 GiB 0.79 0.93 86 up
5 hdd 0.90948 1.00000 931 GiB 5.6 GiB 4.0 GiB 26 KiB 1.6 GiB 926 GiB 0.61 0.71 90 up
8 hdd 0.90948 1.00000 931 GiB 6.2 GiB 5.0 GiB 20 KiB 1.2 GiB 925 GiB 0.67 0.78 88 up
11 lun 0.19530 1.00000 200 GiB 1.1 GiB 1.1 GiB 0 B 65 MiB 199 GiB 0.57 0.67 5 up
TOTAL 8.8 TiB 77 GiB 66 GiB 302 KiB 10 GiB 8.7 TiB 0.86
MIN/MAX VAR: 0.67/3.00 STDDEV: 0.55pg값을 보면?
$ sudo ceph pg ls-by-pool lun-pool
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING
11.0 18 36 0 0 150994944 0 0 39 0 active+recovery_wait+degraded 67s 798'39 932:10030 [9,11,10]p9 [9,11,10]p9 2025-08-10T17:09:33.515975+0000 2025-08-08T05:15:48.984840+0000 1 periodic scrub scheduled @ 2025-08-12T03:14:52.520304+0000
11.1 17 34 0 0 142606336 0 0 34 0 active+recovery_wait+degraded 67s 840'34 931:33 [10,11,9]p10 [10,11,9]p10 2025-08-10T16:06:50.852548+0000 2025-08-06T22:18:52.952670+0000 1 periodic scrub scheduled @ 2025-08-11T18:41:59.510390+0000
11.2 19 19 0 0 159383552 0 0 38 0 active+recovery_wait+degraded 67s 843'38 931:9720 [9,11,10]p9 [9,11,10]p9 2025-08-09T20:29:17.110181+0000 2025-08-05T23:09:42.897117+0000 1 periodic scrub scheduled @ 2025-08-11T08:17:30.284745+0000
11.3 17 34 0 0 125829120 0 0 37 0 active+recovery_wait+degraded 67s 792'37 931:17 [11,10,9]p11 [11,10,9]p11 2025-08-10T09:40:38.486165+0000 2025-08-08T03:48:25.533377+0000 1 periodic scrub scheduled @ 2025-08-11T19:18:41.539217+0000acting에서 모두 lun-pool에 할당된 osd로 확인된다.
