이전에 rancher desktop으로 ELK를 구동하는 방법에 대해 작성하였다.
Rancher desktop으로 ELK 구동하기
이번에는 Lima를 통해 ELK를 작성하고, lazydocker를 이용해서 Container 로그 및 정보를 보는것 까지 진행한다.
Lima?
Linux VM 를 Mac 환경에 만들어주는 오픈소스. Mac 전용이다 보니 M1환경에서도 정상적으로 동작함으로써 로컬에서 개발용으로 진행할때는, Lima를 통해서 개발하는것이 더욱 좋다고 판단된다.
Lima 구성
Lima를 구성하기 위해서 Karl이라는 분이 작성하신, 사이트를 토대로 구성하였다. 예제를 워낙 잘 작성해 주셔서 초반 예제는 무리 없이 수행됨을 확인하였다. (물론 ELK를 위한 삽질은 여전하다)
참고사이트 [Slicon M1 Mac에서 Lima로 Docker Desktop 대체 하기]
Step 1. lima 설치
이전글과 마찬가지로 Image pull 관련 문제가 생긴다면 docker-credential-helper도 brew install 하자
brew install lima docker docker-composeStep 2. VM 생성
예제에서는 https://raw.githubusercontent.com/lima-vm/lima/master/examples/docker.yaml 위치에 있는 예제파일을 사용하지만, ELK를 구동하기 위한 설정이 부족하거나 기본값이 낮아서 제대로 구동이 되질 않았다. 해서 다음과 같이 직접 추가한 설정파일을 사용한다.
# Example to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine
# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...
# This example requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20221018/ubuntu-22.04-server-cloudimg-amd64.img"
arch: "x86_64"
digest: "sha256:4d8d5b95082ed3551cf06b086b854b99d2025c903d1936c35ef34e173c57a817"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20221018/ubuntu-22.04-server-cloudimg-arm64.img"
arch: "aarch64"
digest: "sha256:f753d6f9cea84e4f35160b77189c422578fbb007e789b7e66d96edd6d8a3fa34"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
arch: "aarch64"
##########################################################################
# CPUs: if you see performance issues, try limiting cpus to 1.
# 🟢 Builtin default: 4
cpus: null
# Memory size
# 🟢 Builtin default: "4GiB"
memory: "6GiB"
# Disk size
# 🟢 Builtin default: "100GiB"
disk: null
##########################################################################
mounts:
- location: "~"
writable: true
- location: "/data"
writable: true
- location: "/tmp/lima"
writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
system: false
user: false
provision:
- mode: system
# This script defines the host.docker.internal hostname when hostResolver is disabled.
# It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
# Names defined in /etc/hosts inside the VM are not resolved inside containers when
# using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
script: |
#!/bin/sh
sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
script: |
#!/bin/bash
set -eux -o pipefail
command -v docker >/dev/null 2>&1 && exit 0
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://get.docker.com | sh
# NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
systemctl disable --now docker
apt-get install -y uidmap dbus-user-session
# Custom Liam - vm.max_map_count set
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
# Custom Liam - uLimit Set
echo "* hard memlock unlimited" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nproc 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536" >> /etc/security/limits.conf
- mode: user
script: |
#!/bin/bash
set -eux -o pipefail
systemctl --user start dbus
dockerd-rootless-setuptool.sh install
docker context use rootless
probes:
- script: |
#!/bin/bash
set -eux -o pipefail
if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
echo >&2 "docker is not installed yet"
exit 1
fi
if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
echo >&2 "rootlesskit (used by rootless docker) is not running"
exit 1
fi
hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
# hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
# resolve inside containers, and not just inside the VM itself.
hosts:
host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
To run `docker` on the host (assumes docker-cli is installed), run the following commands:
------
docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
docker context use lima-{{.Name}}
docker run hello-world
------
limactl start default.yml
? Creating an instance "default" Proceed with the current configuration
INFO[0002] Attempting to download the image from "https://cloud-images.ubuntu.com/releases/22.04/release-20221018/ubuntu-22.04-server-cloudimg-arm64.img" digest="sha256:f753d6f9cea84e4f35160b77189c422578fbb007e789b7e66d96edd6d8a3fa34"
INFO[0002] Using cache "/Users/liam/Library/Caches/lima/download/by-url-sha256/297016616ba62c629d64ff3d19cfe1c9bc89d60944996bd80435ccd5af146bff/data"
INFO[0003] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/liam/.lima/default/serial.log")
INFO[0003] SSH Local Port: 60022
INFO[0003] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0013] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0024] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0024] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0024] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0024] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0024] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0033] [hostagent] The essential requirement 3 of 5 is satisfied
INFO[0033] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf (/etc/fuse3.conf) to contain \"user_allow_other\""
INFO[0036] [hostagent] The essential requirement 4 of 5 is satisfied
INFO[0037] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0037] [hostagent] The essential requirement 5 of 5 is satisfied
INFO[0037] [hostagent] Mounting "/Users/liam" on "/Users/liam"
INFO[0037] [hostagent] Mounting "/data" on "/data"
INFO[0037] [hostagent] :/data: Not a directory
WARN[0068] [hostagent] failed to confirm whether /data [remote] is successfully mounted
INFO[0068] [hostagent] Mounting "/tmp/lima" on "/tmp/lima"
INFO[0068] [hostagent] Waiting for the optional requirement 1 of 1: "user probe 1/1"
INFO[0068] [hostagent] Forwarding "/run/user/501/docker.sock" (guest) to "/Users/liam/.lima/default/sock/docker.sock" (host)
INFO[0068] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/liam/.lima/default/ga.sock" (host)
INFO[0068] [hostagent] Not forwarding TCP 0.0.0.0:22
INFO[0069] [hostagent] Not forwarding TCP 127.0.0.53:53
INFO[0069] [hostagent] Not forwarding TCP [::]:22
INFO[0084] [hostagent] The optional requirement 1 of 1 is satisfied
INFO[0084] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0087] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0087] READY. Run `lima` to open the shell.
INFO[0087] Message from the instance "default":
To run `docker` on the host (assumes docker-cli is installed), run the following commands:
------
docker context create lima-default --docker "host=unix:///Users/liam/.lima/default/sock/docker.sock"
docker context use lima-default
docker run hello-world
------한번 기동이 된다음, stop 이후 다시 start해야 한다.
vm.max_map_count의 값이 변경되질 않는다. (임시적용이 아닌 전체 적용으로 해서 첫 구동에는 적용되질 않음을 확인)
# lima vm 명령어
limactl list
limactl stop [name]
limactl start [name]
limactl remove [name]➜ lima git:(master) ✗ lima
bash: line 1: cd: /data/docker/lima: No such file or directory
liam@lima-default:/Users/liam$ sysctl vm.max_map_count
vm.max_map_count = 262144Step 3. SSH 연결
VM을 생성하였지만, 보면 이런생각이 든다. "어라? 그럼 VM에 직접 들어가서 작업해야하나? 그럼 처음에 brew로 뭐하러 docker를 설치했지?
가상환경의 docker와 로컬호스트를 연결해서 로컬호스트에서도 정상적으로 명령어가 실행될 수 있도록 해야한다.
~/.ssh/config/
limactl show-ssh --format=config default >> ~/.ssh/config
Host lima-default
IdentityFile "/Users/liam/.lima/_config/user"
IdentityFile "/Users/liam/.ssh/podman-machine-default"
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
NoHostAuthenticationForLocalhost yes
GSSAPIAuthentication no
PreferredAuthentications publickey
Compression no
BatchMode yes
IdentitiesOnly yes
Ciphers "^aes128-gcm@openssh.com,aes256-gcm@openssh.com"
User liam
ControlMaster auto
ControlPath "/Users/liam/.lima/default/ssh.sock"
ControlPersist 5m
Hostname 127.0.0.1
Port 60022접속 확인
➜ lima git:(master) ✗ ssh lima-default
Last login: Tue Dec 20 07:54:25 2022 from 192.168.5.2
liam@lima-default:~$export DOCKER_HOST
docker_host에 등록하는 방식이 예제에서는 ssh://lima-default:[port]의 방식으로 되어있지만, 여기서는 README에 작성된 방식으로 한다.
echo "export DOCKER_HOST=$(limactl list default --format 'unix://{{.Dir}}/sock/docker.sock')" >> ~/.zshrc
source ~/.zshrc
➜ ~ docker images
REPOSITORY TAG IMAGE ID CREATED SIZEELK 실행
Docker VM을 실행하기 위한 설정중(default.yml), ulimit와 sysctl 값을 변경하였다. 두값 모두 ES를 구동하기 위한 설정인데 해당 부분을 docker-compose에서 설정할 수 없다.
docker-desktop
VM 설정에 ulimit와 sysctl의 설정이 없어도 ES의 구동이 문제가 없다.
rancher-desktop
VM 설정에 ulimit의 설정이 없어도 ES의 구동이 문제가 없다. 단, sysctl의 설정이 필요한데 yml을 통해서 할 수 없고 직접 수동으로 다른 컨테이너를 올려서 처리해야 한다.
lima
VM 설정에 ulimit와 sysctl의 설정이 필요하다. 직접 설정해야 하지만 애초에 VM자체를 개발자가 직접 띄우는것이라 설정파일에 넣거나 접속해서 설정하면 되니까 가장 편리하고 커스텀이 용이하다.
lima docker-compose.yml
docker-desktop에서 사용한 docker-compose.yml 파일을 그대로 수행하면 다음과 같은 예외가 발생
➜ elasticsearch git:(master) ✗ docker-compose -f ./docker-compose.yml up -d
[+] Running 2/4
⠿ Container es01 Starting 0.8s
⠿ Container es02 Starting 0.8s
⠿ Container logstash Recreated 0.0s
⠿ Container kibana Recreated 0.0s
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error setting rlimit type 8: operation not permitted: unknown이 에러를 구글링 했는데 단 한건도 해당 예외에 대해서 정확히 언급된 문서가 없다. docker run 명령어로 개별 ES는 구동이 되는것을 확인했으니 docker-compose의 문제로 확인하고 설정을 하나씩 제거 하면서 구동을 했더니 ulimit의 설정 문제였다.
해서 위에서 VM단계에서 ulimit의 설정을 변경하고 docker-compose단계에서는 해당 옵션을 제거한다.
lima-docker-compose.yml
# volumes의 위치는 /tmp/lima로 해줘야 한다. VM단계에서 해당 위치를 쓰기 가능 영역으로 설정했기 때문에
# logstash의 volumes는 상대경로로 되어있는데, 이때 로컬영역의 /data가 VM영역에는 존재하지 않아 예외가 발생한다.
# 해서 VM 구동시 /data를 마운트 걸어줘야 한다.
# ( 해당 예제를 보시는 분들은 docker-compose.yml이 있는 위치가 VM영역에도 있는 위치(예를들어 ~.)라면 마운트는 필요하지 않고 쓰기 권한만 필요할 수 있음 )
version: '3.6'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.2
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02
- bootstrap.memory_lock=true
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
volumes:
- data01:/tmp/lima/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:8.5.2
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- discovery.seed_hosts=es01,es02
- cluster.initial_master_nodes=es01,es02
- bootstrap.memory_lock=true
- xpack.security.enabled=false
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
volumes:
- data02:/tmp/lima/elasticsearch/data
networks:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana:8.5.2
container_name: kibana
ports:
- 5601:5601
environment:
- ELASTICSEARCH_HOSTS=["http://es01:9200"]
depends_on:
- es01
networks:
- elastic
logstash:
image: docker.elastic.co/logstash/logstash:8.5.2
container_name: logstash
ports:
- 5000:5000
- 9600:9600
environment:
- node.name=logstash
- xpack.monitoring.enabled=false
- xpack.monitoring.elasticsearch.hosts=["http://es01:9200"]
- "LS_JAVA_OPTS=-Xms1g -Xmx1g"
volumes:
- ./logstash/curl_test.conf:/usr/share/logstash/pipeline/curl_test.conf
depends_on:
- es01
networks:
- elastic
volumes:
data01:
driver: local
data02:
driver: local
networks:
elastic:
driver: bridge실행
docker-compose -f ./lima-docker-compose.yml up -d
[+] Running 36/36
⠿ es01 Pulled 99.9s
⠿ e5c09d26504d Pull complete 9.6s
⠿ 35273a2ff40b Pull complete 10.1s
⠿ a9f83b3255a0 Pull complete 10.1s
⠿ d1e01ba33709 Pull complete 95.6s
⠿ 31972e3819e4 Pull complete 95.7s
⠿ 082e00c0a0c5 Pull complete 95.7s
⠿ e4e7decef637 Pull complete 95.8s
⠿ 7ef6faa6d06a Pull complete 95.8s
⠿ a0c670cb75c6 Pull complete 95.8s
⠿ es02 Pulled 99.9s
⠿ logstash Pulled 100.0s
⠿ 4dd30882b185 Pull complete 23.2s
⠿ b20d02782678 Pull complete 23.3s
⠿ d6faaf4838b9 Pull complete 94.8s
⠿ 1df00f879581 Pull complete 94.9s
⠿ 7de283d3e6ef Pull complete 94.9s
⠿ 14ba978465ab Pull complete 95.0s
⠿ d08f7a054409 Pull complete 95.0s
⠿ a9132a329441 Pull complete 95.1s
⠿ f2d8151cd23a Pull complete 95.1s
⠿ 111b142ba237 Pull complete 95.2s
⠿ kibana Pulled 93.5s
⠿ c0029d061a13 Pull complete 31.1s
⠿ 54a8958e19ad Pull complete 31.4s
⠿ 621cd2799c65 Pull complete 32.1s
⠿ 8066c8726539 Pull complete 34.9s
⠿ 89732bc75041 Pull complete 35.3s
⠿ 2ecee2c5df06 Pull complete 36.1s
⠿ abe383bb6d76 Pull complete 88.3s
⠿ 58d2abfb210f Pull complete 88.4s
⠿ d34448f3b8fd Pull complete 88.5s
⠿ 34596b102ef8 Pull complete 88.5s
⠿ 8aaf9e6ad1f3 Pull complete 88.5s
⠿ 6cd18defa2bb Pull complete 88.6s
⠿ 8a8c5bf0022c Pull complete 88.6s
[+] Running 7/7
⠿ Network elasticsearch_elastic Created 0.0s
⠿ Volume "elasticsearch_data02" Created 0.0s
⠿ Volume "elasticsearch_data01" Created 0.0s
⠿ Container es01 Started 0.5s
⠿ Container es02 Started 0.5s
⠿ Container logstash Started 1.1s
⠿ Container kibana Started
➜ elasticsearch git:(master) ✗ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fcf6be01f8fb docker.elastic.co/kibana/kibana:8.5.2 "/bin/tini -- /usr/l…" 42 seconds ago Up 40 seconds 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp kibana
224625ee7d52 docker.elastic.co/logstash/logstash:8.5.2 "/usr/local/bin/dock…" 42 seconds ago Up 40 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp, 0.0.0.0:9600->9600/tcp, :::9600->9600/tcp, 5044/tcp logstash
842c603b40ce docker.elastic.co/elasticsearch/elasticsearch:8.5.2 "/bin/tini -- /usr/l…" 42 seconds ago Up 40 seconds 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp es01
0cc4d37e1bb3 docker.elastic.co/elasticsearch/elasticsearch:8.5.2 "/bin/tini -- /usr/l…" 42 seconds ago Up 40 seconds 9200/tcp, 9300/tcplazydocker
rancher-desktop을 docker-desktop의 대체제로 생각하였지만, rancher-desktop은 컨테이너에 대한 UI가 존재하지 않아, 모니터링이 힘들다. 단순히 VM을 위해 rancher-desktop을 사용하는(본인이 kubernetes를 사용하지 않는다면) 것은 옳지 않고, VM옵션을 원하는데로 설정할 수가 없어서 lima를 선택했고, 그것에 따른 UI가 필요해서 보니 lazydocker라는 것이 있어서 해당 TUI로 대체한다.
설치
brew install lazydocker실행
정상적으로 docker가 구동되어있다면 (docker.sock을 찾을 수 있다면) 다음 명령어로 바로 구동확인이 가능하다
lazydockerstat
log
container env
container config
container top
option
11년차 검색개발자 입니다. 여러 지식과 함께 실제 서비스를 운영 하면서 발생한 이슈에 대해서 정리하고 공유하고자 합니다.