서버 구성

Kernel 파라미터 설정

$ sudo vi /etc/sysctl.conf
vm.dirty_ratio = 7
vm.dirty_background_ratio = 2
vm.swappiness = 1

net.core.somaxconn = 4096
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_max_syn_backlog = 4096

Docker-compose.yml

$ sudo vi docker-compose.yml
version: "2.4"

services:
  mongodb:
    image: mongo:4.2.5
    restart: always
    network_mode: host
    environment:
      - TZ=Asia/Seoul
    mem_limit: 30g
    volumes:
      - /etc/mongod.conf:/etc/mongod.conf
      - /data/db:/data/db
      - /data/journal:/data/db/journal
      - /data/log:/var/log/mongodb
    entrypoint: ["mongod", "-f", "/etc/mongod.conf"]
    container_name: "mongodb"

    ulimits:
      nproc: 64000
      nofile:
        soft: 64000
        hard: 64000

# Container 실행        
$ docker-compose up -d

Logrotate 설정

# logrotate 파일 생성
$ sudo vi /etc/logrotate.d/mongodb
/data/log/mongod.log{
  su root root
  daily
  maxage 31
  missingok
  dateext
  compress
  delaycompress
  copytruncate
}

# logrotate debug 확인
$ sudo logrotate -d /etc/logrotate.d/mongodb
reading config file /etc/logrotate.d/mongodb
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /data/log/*.log after 1 days (no old logs will be kept)
empty log files are rotated, old logs are removed
considering log /data/log/*.log
  log /data/log/*.log does not exist -- skipping

# logrotate 설정적용
$ sudo logrotate -fv /etc/logrotate.d/mongodb

# crontab(root 계정) 적용
$ sudo mkdir /data/script
$ sudo mv /etc/cron.daily/logrotate /data/script/logrotate.sh
$ sudo su
$ crontab -e
23 30 * * * /data/script/logrotate.sh

# cron 적용을 위한 crond 서비스 재부팅
$ sudo systemctl restart crond

Backup & Resotre

# 기존 추천DB SSH, docker container 접속
$ mongo admin -u user -p 'pwd' --authenticationDatabase admin

# Dump
$ mongodump -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d Reco -o /data/db
$ mongodump -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d Base -o /data/db
$ mongodump -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d Seg -o /data/db

# 신규 추천DB Dump 파일 전송
$ scp -r -i "mongodb.pem" /data/db/Reco ec2-user@"IP":/data/db/dump/
$ scp -r -i "mongodb.pem" /data/db/Base ec2-user@"IP":/data/db/dump/
$ scp -r -i "mongodb.pem" /data/db/Seg ec2-user@"IP":/data/db/dump/

# 신규 추천DB SSH, docker container 접속 후 Restore
$ mongorestore -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d Recommend /data/db/dump/Reco
$ mongorestore -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d RecommendBase /data/db/dump/Base
$ mongorestore -h 127.0.0.1:27017 --authenticationDatabase admin -u user -p 'pwd' -d Segmentation /data/db/dump/Seg

LoadBalancer 설정

Private NLB 리스너

Public NLB 리스너

TargetGroup 설정

DB 연결상태 확인

APM 연동

MongoDB_Exporter 설치

# Exporter 설치
$ mkdir mongodb_exporter
$ wget https://github.com/percona/mongodb_exporter/releases/download/v0.7.1/mongodb_exporter-0.7.1.linux-amd64.tar.gz
$ tar -xvzf mongodb_exporter-0.7.1.linux-amd64.tar.gz
$ sudo mv mongodb_exporter /usr/local/bin/

# Exporter 실행
$ /usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://mongodb_exporter:vQlM1r#CGpvF@localhost:27017 & > /dev/null

Prometheus 설정

# Prometheus Job 변경
$ cd prometheus
$ vi prometheus.yml
...
- job_name: 'mongodb'
  metrics_path:/'metrics'
  static_configs:
... 중략

  # 신규 추천DB로 IP 변경
	- targets: ['IP:9216']
    labels:
      instance: 'Prod Recom'
$ cd

# prometheus 재시작
$ docker-compose down && docker-compose up -d

Dashboard 확인