CPU | MEM | Program | |
---|---|---|---|
우분투 | 2 | 4 | ElasticSearch, Kibana |
우분투 | 1 | 1 | Logstash |
ElasticSearch 설치
apt update
apt install -y openjdk-8-jdk
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install elasticsearch
확인
curl -XGET localhost:9200
systemctl status elasticsearch
# 키는법
systemctl start elasticsearch
Kibana 설치
apt update
apt install -y openjdk-8-jdk
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install kibana
/etc/kibana/kibana.yml 설정
server.host: "키바나가 설치된 컴퓨터의 IP"
설정 저장 후
systemctl restart kibana
웹브라우저로
http://키바나가설치된컴퓨터IP:5601/
Logstash
apt update
apt install -y openjdk-8-jdk
Logstash 설치
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install logstash
설정파일 생성
vi /etc/logstash/conf.d/logstash.conf
input {
tcp {
port => 5000
codec => json { charset => "UTF-8" }
}
}
filter {
mutate {
remove_field => [ "port", "path", "message", "@version", "@timestamp", "host" ]
}
}
output {
elasticsearch {
hosts => ["http://192.168.216.135:9200"]
index => "coffee"
}
}
웹 브라우저 이동
PUT coffee/_mapping
{
"properties" : {
"brand" : { "type" : "text" },
"coffee_geo" : { "type" : "geo_point" }
}
}
pip install python-logstash
pip install logstash
import json
import logging
from logging import StreamHandler
import logstash
host = '192.168.216.135'
test_logger = logging.getLogger('python-logstash-logger')
test_logger.setLevel(logging.INFO)
test_logger.addHandler(logstash.TCPLogstashHandler(host, 5000, version=1))
test_logger.addHandler(StreamHandler())
inout_f = open("starbucks_ediya.csv", 'r', encoding="UTF-8")
inout_lines = inout_f.readlines()
json_f = open("coffee.json", 'w', encoding="UTF-8")
for i in inout_lines:
brand = i.split(',')[0]
lon = i.split(',')[1]
lat = i.split(',')[2]
# print(brand)
data = {
"brand": brand,
"coffee_geo": {"lat": lat, "lon": lon}
}
json_f.write(json.dumps(data, ensure_ascii=False) + "\n")
test_logger.info('커피', extra=data)
inout_f.close()
json_f.close()