Ceph ansible을 수행하여 custom된 ceph를 설치할 때 ssh port가 변경되면 문제가 발생하고있다.
다음과 같은 ansible log를 확인할 수 있다.
[user@bjdev-control-01 cloudx]$ ss -nltp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:2222 내 테스트 머신의 포트는 22에서 2222로 변경하고 진행하였다.
이상태로 ceph ansible을 수행하면 다음과 같이 에러가 발생한다.
Friday 07 February 2025 12:49:53 +0900 (0:00:00.080) 0:00:58.622 *******
fatal: [bjdev-control-01 -> bjdev-ceph-01]: FAILED! => {"changed": false, "msg": "file not found: /etc/ceph/ceph.pub"}
NO MORE HOSTS LEFT *******************************************************************************************************************************************************************************************
PLAY RECAP ***************************************************************************************************************************************************************************************************
bjdev-ceph-01 : ok=8 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
bjdev-ceph-02 : ok=8 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
bjdev-ceph-03 : ok=8 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
bjdev-compute-01 : ok=6 changed=1 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
bjdev-control-01 : ok=8 changed=2 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0
bjdev-control-02 : ok=6 changed=1 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
bjdev-control-03 : ok=6 changed=1 unreachable=0 failed=0 skipped=4 rescued=0 ignored=0
Friday 07 February 2025 12:49:55 +0900 (0:00:01.133) 0:00:59.756 *******
===============================================================================
burrito.ceph : Rocky Linux | Install packages on ceph servers ---------------------------------------------------------------------------------------------------------------------------------------- 17.35s
Gathering Facts -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 16.13s
burrito.ceph : Rocky Linux | Install prerequisite packages -------------------------------------------------------------------------------------------------------------------------------------------- 9.01s
burrito.ceph : Rocky Linux | Install packages on ceph clients ----------------------------------------------------------------------------------------------------------------------------------------- 8.50s
burrito.ceph : Bootstrap | check ceph nodes are ready ------------------------------------------------------------------------------------------------------------------------------------------------- 2.37s
burrito.ceph : Bootstrap | set up insecure local registry for podman ---------------------------------------------------------------------------------------------------------------------------------- 2.28s
burrito.ceph : Bootstrap | bootstrap ceph cluster ----------------------------------------------------------------------------------------------------------------------------------------------------- 1.93s
burrito.ceph : sshkey | get ssh public key from the bootstrap node ------------------------------------------------------------------------------------------------------------------------------------ 1.13s
burrito.ceph : Main | include os specific tasks ------------------------------------------------------------------------------------------------------------------------------------------------------- 0.31s
burrito.ceph : Common | include os specific variables ------------------------------------------------------------------------------------------------------------------------------------------------- 0.22s
burrito.ceph : Rocky Linux | Install online packages -------------------------------------------------------------------------------------------------------------------------------------------------- 0.19s
burrito.ceph : Common | include default variables ----------------------------------------------------------------------------------------------------------------------------------------------------- 0.18s
burrito.ceph : Bootstrap | already bootstrapped ------------------------------------------------------------------------------------------------------------------------------------------------------- 0.08sceph.pub 키를 생성하지 못하여 발생하는데 사실 해당 task를 보기 전 이전 task에서 어떻게 수행됐는지 확인이 먼저 필요하다.
우선 ansible로 debug한 내용을 확인해보면 이 전 task인 Bootstrap 에서 실패하는 걸 확인할 수 있다.
TASK [ceph : Bootstrap | bootstrap ceph cluster]
Saturday 08 February 2025 10:09:02 +0900 (0:00:00.844) 0:00:40.894 *****
...
changed: [bjdev-control-01 -> bjdev-ceph-01] => {
"changed": true,
"cmd": [
"cephadm",
"--image",
"***.***.***.***:port/ceph/ceph:v18.2.1",
"bootstrap",
"--allow-overwrite",
"--mon-ip",
"***.***.***.***",
"--skip-dashboard",
"--skip-firewalld",
"--skip-monitoring-stack",
"--ssh-user",
"user"
],
"delta": "0:00:01.094066",
"end": "2025-02-08 10:08:40.893346",
"failed_when_result": false,
"invocation": {
"module_args": {
"_raw_params": "cephadm --image ***.***.***.***:port/ceph/ceph:v18.2.1 bootstrap --allow-overwrite --mon-ip ***.***.***.*** --skip-dashboard --skip-firewalld --skip-monitoring-stack --ssh-user clex",
"_uses_shell": false,
"argv": null,
"chdir": null,
"creates": null,
"executable": null,
"expand_argument_vars": true,
"removes": null,
"stdin": null,
"stdin_add_newline": true,
"strip_empty_ends": true
}
},
"msg": "non-zero return code",
"rc": 1,
"start": "2025-02-08 10:08:39.799280",
"stderr": "Error: \n** Please verify your user's ssh configuration and make sure:\n- User user must have passwordless sudo access\n\n\nERROR: \n** Please verify your user's ssh configuration and make sure:\n- User clex must have passwordless sudo access",
"stderr_lines": [
"Error: ",
"** Please verify your user's ssh configuration and make sure:",
"- User clex must have passwordless sudo access",
"",
"",
"ERROR: ",
"** Please verify your user's ssh configuration and make sure:",
"- User clex must have passwordless sudo access"
],
"stdout": "Verifying ssh connectivity using standard pubkey authentication ...\nAdding key to clex@localhost authorized_keys...\nNon-zero exit code 255 from ssh -o StrictHostKeyChecking=no -i /tmp/ssh_key_3b8ec3c8-e5b9-11ef-be45-525400aed7d8 -o PasswordAuthentication=no clex@bjdev-ceph-01 sudo echo\nssh: stderr ssh: connect to host bjdev-ceph-01 port 22: Connection refused\n\n\n\t***************\n\tCephadm hit an issue during cluster installation. Current cluster files will NOT BE DELETED automatically to change\n\tthis behaviour you can pass the --cleanup-on-failure. To remove this broken cluster manually please run:\n\n\t > cephadm rm-cluster --force --fsid 3b8eab04-e5b9-11ef-be45-525400aed7d8\n\n\tin case of any previous broken installation user must use the rm-cluster command to delete the broken cluster:\n\n\t > cephadm rm-cluster --force --zap-osds --fsid <fsid>\n\n\tfor more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster\n\t***************",
"stdout_lines": [
"Verifying ssh connectivity using standard pubkey authentication ...",
"Adding key to clex@localhost authorized_keys...",
"Non-zero exit code 255 from ssh -o StrictHostKeyChecking=no -i /tmp/ssh_key_3b8ec3c8-e5b9-11ef-be45-525400aed7d8 -o PasswordAuthentication=no clex@bjdev-ceph-01 sudo echo",
"ssh: stderr ssh: connect to host bjdev-ceph-01 port 22: Connection refused",
"",
"",
"\t***************",
"\tCephadm hit an issue during cluster installation. Current cluster files will NOT BE DELETED automatically to change",
"\tthis behaviour you can pass the --cleanup-on-failure. To remove this broken cluster manually please run:",
"",
"\t > cephadm rm-cluster --force --fsid 3b8eab04-e5b9-11ef-be45-525400aed7d8",
"",
"\tin case of any previous broken installation user must use the rm-cluster command to delete the broken cluster:",
"",
"\t > cephadm rm-cluster --force --zap-osds --fsid <fsid>",
"",
"\tfor more information please refer to https://docs.ceph.com/en/latest/cephadm/operations/#purging-a-cluster",
"\t***************"
]
}디버그 로그에서 보면 "ssh: stderr ssh: connect to host bjdev-ceph-01 port 22: Connection refused" 와 같은 로그가 발생한다.
내 ceph node와 22 port로 통신한다.
근데 ansible.cfg에는 remote_port가 2222로 잘 정의되어있다.
이 경우는 cephadm의 문제인데, cephadm으로 구성 시 ssh port를 별도로 지정하는 옵션 자체가 없어서 발생하는 문제이다.
Ref :: https://docs.ceph.com/en/octopus/cephadm/operations/
그러면 playbook에서 몇가지 내용을 추가하자.
$ cat roles/ceph/tasks/bootstrap.yml
---
...
+- name: Create ssh config file.
+ blockinfile:
+ path: "/tmp/.ssh/config"
+ create: yes
+ mode: 0600
+ block: |
+ StrictHostKeyChecking no
+ UserKnownHostsFile /dev/null
+ Port {{ ansible_port }}
+ delegate_to: "{{ groups['mons'][0] }}"
+ when: ansible_port != 22
- name: Bootstrap | bootstrap ceph cluster
ansible.builtin.command: >-
cephadm \
--image {{ quay_image_repo }}/ceph/ceph:{{ ceph_version }} \
bootstrap {{ ceph_bootstrap_params }} \
--allow-overwrite \
--mon-ip {{ hostvars[groups['mons'][0]]['ansible_' + storage_iface_name].ipv4.address }} \
--skip-dashboard \
--skip-firewalld \
--skip-monitoring-stack \
+ {% if ansible_port != 22 %}
+ --ssh-config "/tmp/.ssh/config" \
+ {% endif %}
--ssh-user {{ ansible_user }}
become: true
register: _res
delegate_to: "{{ groups['mons'][0] }}"
run_once: true
failed_when: _res.rc not in [0, 1]
...create ssh config file을 생성하는 task를 추가하고, 해당 task는 22 port가 아니면 ssh/config파일을 생성, 이후 bootstrap시 --ssh-config 로 config파일을 가지고 bootstrap을 할 수 있도록 수정하자.
이후 ceph ansible을 수행하면 다음과 같다.
[user@bjdev-control-01 cloudx]$ sudo ceph -s
cluster:
id: e870fa90-e5da-11ef-b024-525400aed7d8
health: HEALTH_OK
services:
mon: 1 daemons, quorum bjdev-ceph-01 (age 2m)
mgr: bjdev-ceph-01.onzaxn(active, since 117s)
osd: 3 osds: 3 up (since 32s), 3 in (since 68s)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
pools: 9 pools, 9 pgs
objects: 227 objects, 582 KiB
usage: 85 MiB used, 300 GiB / 300 GiB avail
pgs: 9 active+clean잘 설치 된다.
즉 문제는 cephadm의 ssh port 에 대한 옵션이 없어 ssh config를 만들어 그걸로 ssh port통신을 할 수 있도록 변경하는 것이다.
