๐กELK ์คํ์ ์ด์ฉํด์ ํ๋ผ์คํฌ์ ๋ก๊ทธ๋ฅผ ๋จ๊ธฐ์
ELK ๋, Elastic search, Logstash, Kibana์ ์ธ๊ฐ์ง ์คํ์์ค ํ๋ก์ ํธ์ ์ฝ์์ด๋ค
๊ฐ ํ๋ก์ ํธ๊ฐ ์ฐ๋๋์ด ๋ฐ์ดํฐ ์์ง ๋ฐ ๋ถ์ ํด๋ก ์ฌ์ฉ ํ ์ ์๋ค
$ git clone https://github.com/paullee714/ELK-docker-python.git
์๋ณธ ํ๋ก์ ํธ - https://github.com/deviantony/docker-elk
ELK-docker-python
โโโ README.md
โโโ docker-elk
โ โโโ LICENSE
โ โโโ README.md
โ โโโ docker-compose.yml
โ โโโ docker-stack.yml
โ โโโ elasticsearch
โ โโโ extensions
โ โโโ kibana
โ โโโ logstash
โโโ elk-flask
โ โโโ __pycache__
โ โโโ app.py
โ โโโ elk_lib
โ โโโ route
โโโ requirements.txt
โโโ venv
โโโ bin
โโโ lib
โโโ pyvenv.cfg
ํ๋ก์ ํธ ํด๋ ์์ ์๋ docker-elk ๋๋ ํฐ๋ฆฌ๋ก ๋ค์ด๊ฐ docker-compose๋ฅผ ์คํ์์ผ์ค๋ค
$ cd docker-elk
$ docker-compose build && docker-compose up -d
์คํ๊ฒฐ๊ณผ
โ docker-elk git:(develop) docker-compose build && docker-compose up -d
Building elasticsearch
Step 1/2 : ARG ELK_VERSION
Step 2/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
---> f29a1ee41030
Successfully built f29a1ee41030
Successfully tagged docker-elk_elasticsearch:latest
Building logstash
Step 1/2 : ARG ELK_VERSION
Step 2/2 : FROM docker.elastic.co/logstash/logstash:${ELK_VERSION}
---> fa5b3b1e9757
Successfully built fa5b3b1e9757
Successfully tagged docker-elk_logstash:latest
Building kibana
Step 1/2 : ARG ELK_VERSION
Step 2/2 : FROM docker.elastic.co/kibana/kibana:${ELK_VERSION}
---> f70986bc5191
Successfully built f70986bc5191
Successfully tagged docker-elk_kibana:latest
Starting docker-elk_elasticsearch_1 ... done
Starting docker-elk_kibana_1 ... done
Starting docker-elk_logstash_1 ... done
docker ps๋ก ํ์ธ ํด ์ฃผ์
โ docker-elk git:(develop) docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ae318f58a9af docker-elk_logstash "/usr/local/bin/dockโฆ" 2 days ago Up 47 seconds 0.0.0.0:5000->5000/tcp, 0.0.0.0:9600->9600/tcp, 0.0.0.0:5000->5000/udp, 5044/tcp docker-elk_logstash_1
00a032b5c5c4 docker-elk_kibana "/usr/local/bin/dumbโฆ" 2 days ago Up 47 seconds 0.0.0.0:5601->5601/tcp docker-elk_kibana_1
3b62a3ba2e21 docker-elk_elasticsearch "/usr/local/bin/dockโฆ" 2 days ago Up 47 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp docker-elk_elasticsearch_1
docker ps ๋ก ํ์ธ ํด ๋ณด์์ ๋ ๊ฐ๊ฐ์ port๋ ๋ค์๊ณผ ๊ฐ๋ค
Elastic Search : 9200, 9300
Logstash : 5000, 9600
Kibana : 5601
docker-compose.ymlํ์ผ๊ณผ ๊ฐ configํ์ผ์ ํ์ธํด๋ณด์
์๋์ docker-compose.yml ํ์ผ์ ๋ณด๋ฉด, ๊ฐ๊ฐ service๋ค์ configํ์ผ๋ค์ ์ค์ ์ ๊ฐ์ ธ์์ ์ค์ ํ๋ค.
Elastic Search : /elasticsearch/config/elasticsearch.yml
Logstash : /logstash/config/logstash.yml
Kibana : /kibana/config/kibana.yml
docker-elk docker-compose.yml file
# /ELK-docker-python/docker-elk/docker-compose.yml
version: '3.2'
services:
elasticsearch:
build:
context: elasticsearch/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./elasticsearch/config/elasticsearch.yml
target: /usr/share/elasticsearch/config/elasticsearch.yml
read_only: true
- type: volume
source: elasticsearch
target: /usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
ELASTIC_PASSWORD: changeme
# Use single node discovery in order to disable production mode and avoid bootstrap checks
# see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
discovery.type: single-node
networks:
- elk
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./logstash/config/logstash.yml
target: /usr/share/logstash/config/logstash.yml
read_only: true
- type: bind
source: ./logstash/pipeline
target: /usr/share/logstash/pipeline
read_only: true
ports:
- "5000:5000/tcp"
- "5000:5000/udp"
- "9600:9600"
environment:
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
networks:
- elk
depends_on:
- elasticsearch
kibana:
build:
context: kibana/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- type: bind
source: ./kibana/config/kibana.yml
target: /usr/share/kibana/config/kibana.yml
read_only: true
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
volumes:
elasticsearch:
flask์ ๋ก๊ทธ๋ฅผ ๋ฐ๋ ๊ฒ์ ELK์คํ ์ค, Logstash ์ด๋ค.
Logstash โ> Elastic Search โ> Kibana(์กฐํ/๋ถ์)
๊ทธ๋ ๊ธฐ ๋๋ฌธ์ Logstash์ ์ค์ ์ด ์ค์ํ๋ค.
ES์ ์ธ๋ฑ์ค๋ฅผ 'elk-logger' ๋ก ์ค์ ํ๊ณ ๋ก๊ทธ๋ฅผ ๋ชจ์ ๋ด ์๋ค
$ vim /ELK-docker-python/docker-elk/logstash/pipeline/logstash.conf
input {
tcp {
port => 5000
}
}
## Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
password => "changeme"
index => "elk-logger"
}
}
requirements.txt
certifi==2020.4.5.1
click==7.1.2
elasticsearch==7.7.1
Flask==1.1.2
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
python-dotenv==0.13.0
python-json-logger==0.1.11
python-logstash==0.4.6
python3-logstash==0.4.80
urllib3==1.25.9
Werkzeug==1.0.1
requirements.txt๋ฅผ ๋จ๊ฒจ๋์๋ค. ๊ฐ์ํ๊ฒฝ์์ ํจํค์ง๋ฅผ ์ค์นํ๋ฉด flask๋ฐ ์ฌ๋ฌ ๊ธฐํ ๋ชจ๋๋ค์ ์ฌ์ฉ ํ ์ ์๋ค.
import logging, logstash
log_format = logging.Formatter('\n[%(levelname)s|%(name)s|%(filename)s:%(lineno)s] %(asctime)s > %(message)s')
def create_logger(logger_name):
logger = logging.getLogger(logger_name)
if len(logger.handlers) > 0:
return logger # Logger already exists
logger.setLevel(logging.INFO)
**logger.addHandler(logstash.TCPLogstashHandler('localhost', 5000, version=1))**
return logger
logger๋ฅผ ์ค์ ํ๋ ๋ถ๋ถ ์ค, addHandler(logstash.TCPLogstashHandler('localhost',5000,version=1)) ์ด logstash๋ก ํด๋นํ๋ ๋ก๊ทธ๋ฅผ ๋ณด๋ด๊ฒ ๋ค ๋ผ๋ ์๊ธฐ์ด๋ค.
@elk_test.route('/', methods=['GET'])
def elk_test_show():
logger = elk_logger.create_logger('elk-test-logger')
logger.info('hello elk-test-logstash')
return "hello world!"
๋ก๊น ์ ๋จ๊ธฐ๋ ๋ฉ์๋๋ฅผ ๋ง๋ค์ด ๋ก๊ทธ๋ฅผ ์ ๋ฌํ๋ฉด ๋๋ค.
๊ฐ๊ฐ์ configํ์ผ์์ id,pw๋ฅผ ์ค์ ํ ๋๋ก ๋ก๊ทธ์ธ ํ๋ฉด ๋๋ค.
์ค์ ํ์ผ์ ๋ฐ๋ก ๋ฐ๊พธ์ง ์์๋ค๋ฉด, id๋ elastic, pw๋ changeme ์ด๋ค
๋ก๊ทธ์ธ ํ, index pattern๊ณผ ๋ฐ์ดํฐ๊ฐ ์ ๋ค์ด๊ฐ ์๋ ๊ฒ์ ํ์ธ ํ ์ ์๋ค.
$ docker-compose down
์์ ๋ช ๋ น์ด๋ก ํ๋ก์ธ์ค๋ฅผ ์ข ๋ฃ ํด์ค๋ค :D