kafka 설치

Han Hanju·2021년 6월 24일

다운로드

wget http://apache.mirror.cdnetworks.com/kafka/2.7.1/kafka_2.13-2.7.1.tgz

설정

config/zookeeper.properties 수정

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/opt/kafka_2.13-2.7.1/
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080

server.1=aidw-001:2888:3888
server.2=aidw-002:2888:3888
server.3=aidw-003:2888:3888

config/server.properties 수정
- aidw-002, aidw-003의 경우 id값만 2,3으로 변경.

broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://aidw-001:9092
zookeeper.connect=aidw-001:2181,aidw-002:2181,aidw-003:2181

zookeeper.properties파일 안 dataDir에 설정한 경로에 아래와같이 파일을 만든다.
- aidw-002에는 echo 2> ... 로 aidw-003에는 echo 3 > ...으로 만든다.

root@aidw-001:/opt/kafka_2.13-2.7.1# echo 1 > /opt/kafka_2.13-2.7.1/myid
root@aidw-001:/opt/kafka_2.13-2.7.1# ll
total 52
drwxr-xr-x 7 root root  4096  6월 24 15:33 ./
drwxr-xr-x 9 root root  4096  6월 24 15:17 ../
-rw-r--r-- 1 root root 14535  4월  8 20:29 LICENSE
-rw-r--r-- 1 root root   953  4월  8 20:29 NOTICE
drwxr-xr-x 3 root root  4096  4월  8 20:35 bin/
drwxr-xr-x 2 root root  4096  4월  8 20:35 config/
drwxr-xr-x 2 root root  4096  6월 24 15:17 libs/
drwxr-xr-x 2 root root  4096  4월  8 20:35 licenses/
-rw-r--r-- 1 root root     2  6월 24 15:33 myid
drwxr-xr-x 2 root root  4096  4월  8 20:35 site-docs/
root@aidw-001:/opt/kafka_2.13-2.7.1# vi myid

실행

zookeeper 활성화

hadoop HA를 위해 zookeeper를 설치한경우.


/opt/zookeeper-3.4.10/bin/zkServer.sh start
/opt/kafka_2.13-2.7.1/bin/kafka-server-start.sh config/server.properties

(base) root@aidw-001:/opt/kafka_2.13-2.7.1/bin# jps
205074 Kafka
203808 QuorumPeerMain
205693 Jps

kafka만 설치한경우

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

테스트

토픽생성

(base) root@aidw-001:/opt/kafka_2.13-2.7.1# # bin/kafka-topics.sh --create --zookeeper aidw-001:2181,aidw-002:2181,aidw-003:2181 --replication-factor 3 -partitions 1 --topic hana01
Created topic hana01.

토픽생성 확인

(base) root@aidw-001:/opt/kafka_2.13-2.7.1# bin/kafka-topics.sh --list --zookeeper aidw-001:2181,aidw-002:2181,aidw-003:2181
hana01

producer 메세지 생산

(base) root@aidw-001:/opt/kafka_2.13-2.7.1# bin/kafka-console-producer.sh --broker-list aidw-001:9092,aidw-002:9092,aidw-003:9092 --topic hana01
>hello
>i am han
>good!

메시지 확인

root@aidw-002:/opt/kafka_2.13-2.7.1# bin/kafka-console-consumer.sh --bootstrap-server aidw-001:9092,aidw-002:9092,aidw-003:9092 --topic hana01 --from-beginning
hello
i am han
good!

truble shooting

1
- 해결: config/zookeeper.properties에서 initLimit추가

root@aidw-001:/opt/kafka_2.13-2.7.1# bin/zookeeper-server-start.sh config/zookeeper.properties
[2021-06-24 15:48:12,861] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-06-24 15:48:12,861] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-06-24 15:48:12,865] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-06-24 15:48:12,865] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-06-24 15:48:12,872] ERROR Invalid config, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing config/zookeeper.properties
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:157)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:113)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.lang.IllegalArgumentException: initLimit is not set
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.checkValidity(QuorumPeerConfig.java:732)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:609)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:423)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:153)
        ... 2 more
Invalid config, exiting abnormally

2
- 해결: 기존에 zookeeper를 hadoop을 HA를 위해 설치해놔서 생긴 오류.
- bin/zookeeper-server-start.sh config/zookeeper.properties말고 기존에 설치한 zkServer.sh start로 실행 후 "QuorumPeerMain"를 활성화시켰다
- 이후 bin/kafka-server-start.sh config/server.properties실행시 정상작동

[2021-06-24 16:23:56,106] WARN Cannot open channel to 3 at election address aidw-003/1.209.179.133:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:373)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:436)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2021-06-24 16:23:56,106] WARN Cannot open channel to 2 at election address aidw-002/1.209.179.132:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:373)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:436)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2021-06-24 16:23:56,325] INFO Notification time out: 400 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-06-24 16:23:56,326] WARN Cannot open channel to 3 at election address aidw-003/1.209.179.133:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:373)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:436)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2021-06-24 16:23:56,331] WARN Cannot open channel to 2 at election address aidw-002/1.209.179.132:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)

Reference

https://chulkang.tistory.com/36

Han Hanju

Data Analytics Engineer

이전 포스트

Kaka 개요 1.

다음 포스트

kafka 설치

다운로드

설정

실행

테스트

truble shooting

Reference

Kaka 개요 1.

pytorch graph

0개의 댓글

관련 채용 정보