Monstache를 이용한 MongoDB + Elasticsearch 동기화

dh·2024년 12월 9일

Monstache란?

Monstache는 MongoDB컬렉션들을 Elasticsearch로 지속적으로 인덱싱하는 sync 데몬
MongoDB의 변경 사항을 실시간으로 감지하고, 이를 Elasticsearch로 인덱싱하여 빠르고 효율적인 검색을 지원
동기화를 통해 MongoDB의 유연한 데이터저장과 Elasticsearch의 빠른 검색 기능 사용 가능

전체 구조

elastic-mongo-monstache/
├── config/
│   ├── config.toml		# Monstache 설정용
│   
├── mongodb/			# MongoDB 설정
│   ├── db1/
│   ├── db2/
│   ├── key/
│   ├── Dockerfile
│   ├── replicaSet.js
│   ├── setup.sh
│   ├── docker-compose.yml  #linux환경에서는 필요 없음(MongoDB replicaSet key파일 생성용)
│   
├── .env
│   
├── docker-compose.yml

Elasticsearch, Kibana 구성

.env

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject
MONGO_USER=changeme
MONGO_PASSWORD=changeme


# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=changeme


# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=changeme

MONSTACHE_PASSWORD=changeme

# Version of Elastic products
STACK_VERSION=8.7.1


# Set the cluster name
CLUSTER_NAME=docker-cluster


# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial


# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200


# Port to expose Kibana to the host
KIBANA_PORT=5601


# Increase or decrease based on the available host memory (in bytes)
ES_MEM_LIMIT=1073741824
KB_MEM_LIMIT=1073741824
LS_MEM_LIMIT=1073741824


# SAMPLE Predefined Key only to be used in POC environments
ENCRYPTION_KEY=c34d38b3a14956121ff2170e5030b471551370178f43e5626eec58b04a30fae2

docker-compose.yml(elasticsearch_setup)

version: '3.7'

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

networks:
  monstache-network:
    driver: bridge

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    container_name: elasticsearch_setup
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    networks:
      - monstache-network
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: monstache\n"\
          "    dns:\n"\
          "      - monstache\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "Setting monstahce_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/monstahce_system/_password -d "{\"password\":\"${MONSTACHE_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

Elastic Stack 및 Docker Compose 시작하기를 참고해서 elasticsearch와 kibana를 ssl/tls를 사용하는 방법으로 구현했습니다.

1. CA(Certificate Authority) 생성

    if [ ! -f config/certs/ca.zip ]; then
      echo "Creating CA";
      bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
      unzip config/certs/ca.zip -d config/certs;
    fi;

CA(Certificate Authority) 인증서를 생성하여 config/certs/ca.zip에 저장 후 CA 파일을 config/certs 디렉토리에 압축 해제합니다.

2. 서비스 인증서 생성

instances.yml를 입력으로 해서 ca.crt와 ca.key를 이용해 인증서를 pem형식으로 생성합니다.

config/certs/
├── ca/
│   ├── ca.crt          # CA 인증서
│   ├── ca.key          # CA 개인 키
├── es01/
│   ├── es01.crt        # Elasticsearch 노드 인증서
│   ├── es01.key        # Elasticsearch 노드 개인 키
├── kibana/
│   ├── kibana.crt      # Kibana 인증서
│   ├── kibana.key      # Kibana 개인 키
├── monstache/
│   ├── monstache.crt   # Monstache 인증서
│   ├── monstache.key   # Monstache 개인 키

docker-compose.yml(elasticsearch, kibana)

 es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    container_name: elasticsearch
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: 1g
    networks:
      - monstache-network
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    container_name: kibana
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
    mem_limit: 1g
    networks:
      - monstache-network
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

MongoDB replicaSet 구성

Monstache를 이용해 MongoDB와 Elasticsearch를 동기화 하기 위해서 MongoDB의 삽입, 수정 작업에 대한 기록이 필요합니다.
이를 위해 MongoDB를 replicaSet으로 구성하여 작업에 대한 oplog를 생성하게 됩니다.
Monstache는 이 oplog를 바탕으로 Elasticsearch에 index를 생성해 실시간으로 데이터를 동기화하게 됩니다.

mongodb/
├── db1/
│   
├── db2/
│   
├── key/
│   
├── Dockerfile
│  
├── replicaSet.js
│  
├── setup.sh
│  
├── docker-compose.yml  #linux환경에서는 필요 없음(key파일 생성용)

위와 같은 폴더 구조 생성.
MongoDB replicaSet설정을 하는 docker이미지를 생성하기 위해 Dockerfile 만듭니다.

Dockerfile

FROM mongo

WORKDIR /usr/src
RUN mkdir configs
WORKDIR /usr/src/configs

COPY replicaSet.js .
COPY setup.sh .

RUN chmod +x ./setup.sh

CMD ["./setup.sh"]

setup.sh

mongodb id와 password 본인이 설정

#!/bin/bash               
sleep 10 | echo Sleeping
mongosh mongodb://{ID}:{Password}@mongodb1:27017/admin /usr/src/configs/replicaSet.js

replicaSet.js

rs.initiate({
    _id : "replication",
    members: [
      {_id:0,host : "mongodb1:27017"},
      {_id:1,host : "mongodb2:27017"}
    ]
  } )

rs.conf();

Dockerfile이 있는 경로에서 docker build -t setup-rspl .를 실행하여 이미지를 생성합니다.

key 설정

linux계열에서는 key디렉토리 경로에서 다음과 같이 설정하면 됩니다.

openssl rand -base64 756 > mongodb.key
chmod 400 mongodb.key
chown 999:999 mongodb.key #또는 mongodb:mongodb

window환경에서는 아래와 같이 docker-compose.yml을 구성합니다.

version: "3.8"

services:
  nginx:
    image: nginx:latest
    container_name: nginx
    ports:
      - "3015:80"
    volumes:
      - ./key:/key

docker-compose up -d로 컨테이너 실행 후 key경로에서 위에 명령어 실행하면 mongodb.key생성 완료

docker-compose.yml(MongoDB)

  mongodb1:
    restart: always
    image: mongo:latest
    container_name: mongodb1
    expose:
      - "27017"
    ports:
      - "27017:27017"
    volumes:
      - ./mongodb/db1:/data/db
      - ./mongodb/key:/etc
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_USER}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASSWORD}
    command:
      - mongod
      - "--replSet"
      - "replication"
      - "--keyFile"
      - "/etc/mongodb.key"
      - "--bind_ip_all"
    networks:
      - monstache-network
    user: "mongodb"

  mongodb2:
    restart: always
    image: mongo:latest
    container_name: mongodb2
    expose:
      - "27017"
    ports:
      - "27018:27017"
    volumes:
      - ./mongodb/db2:/data/db
      - ./mongodb/key:/etc
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_USER}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_PASSWORD}
    command:
      - mongod
      - "--replSet"
      - "replication"
      - "--keyFile"
      - "/etc/mongodb.key"
      - "--bind_ip_all"
    depends_on:
      - mongodb1
    networks:
      - monstache-network
    user: "mongodb"
    
  mongosetup:
    image: "setup-rspl:latest"
    container_name: mongosetup
    entrypoint: ["/bin/bash", "/usr/src/configs/setup.sh"]
    depends_on:
      - mongodb1
    networks:
      - monstache-network

Monstache 구성

/elastic-mongo-monstache
├── config/
│   ├── config.toml

config.toml

https://rwynn.github.io/monstache-site/config/#elasticsearch-pem-file

mongo-url = "mongodb://root:root@mongodb1:27017,mongodb2:27017/admin?replicaSet=replication"

elasticsearch-urls = ["https://es01:9200"]
elasticsearch-user="elastic"
elasticsearch-password="엘라스틱서치 Password"
elasticsearch-pem-file = "/usr/share/monstache/config/certs/ca/ca.crt" # TLS요청 보내기 위한 CA인증서

delete-strategy=0	# document삭제시 elasticsearch에도 삭제
direct-read-namespaces = [ "mootd.photo_test", "test.test"] # 기존 구성되어있던 mongo에서 복사할 콜렉션
dropped-databases=false
resume=false
resume-write-unsafe=true
index-as-update=true
index-oplog-time=true
verbose=true

[[script]]  #MongoDB의 Database명.collection명으로 인덱스를 생성하여 Elasticsearch에 저장
script="""
module.exports=function(doc,ns){
    doc._meta_monstache={index: ns};
    return doc;
}
"""

Aws Ec2환경에서 할때 mongodb1, mongodb2로 안돼서 호스트서버의 도메인명으로 했던것 같습니다?

docker-compose.yml(Monastache)

  monstache:
    restart: always
    image: rwynn/monstache:rel6
    container_name: monstache
    command: -f ./config.toml &
    volumes:
      - ./config/config.toml:/config.toml
      - certs:/usr/share/monstache/config/certs
    environment:
      # - MONSTACHE_ES_USER=monstache_system
      # - MONSTACHE_ES_PASS=${MONSTACHE_PASSWORD}
      - MONSTACHE_ES_URLS=https://es01:9200 

    depends_on:
      - es01
      - mongodb1
      - mongodb2
    links:
      - es01
    ports:
      - "8082:8082"
    networks:
      - monstache-network