1. Kafka 설치
요약해서,
Confluent Kafka는 Apache Kafka와 동일하게 사용할 수 있으며,
내부 라이브러리가 잘 정돈된 상태이고,
Default로 다양한 플러그인이 내장되어 있어서 사용하기 용이하다.
1.1 Java 설치
굳이 환경 변수까지 설정하지 않아도 무방하다.
1.2 Confluent Kafka 설치
$ wget https://packages.confluent.io/archive/7.1/confluent-community-7.1.2.tar.gz $ tar -xvf confluent-community-7.1.2.tar.gz
- 사용자가 원하는 버전을 고려해서 설치한다.
1.3 디렉터리 이름 변경
$ mv confluent-community-7.1.2 confluent가독성을 높이고 이후 환경 변수 설정에 용이하게 하기 위함이다.
2. 환경 변수 설정
2.1 환경 변수 설정
$ vi .bashrc
# or
$ sudo nano .bashrc
# 다음 Code를 해당 파일 맨 밑에 추가한다.
export CONFLUENT_HOME=/home/chan-kaf/extools/confluent
export PATH=.:$PATH:$CONFLUENT_HOME/bin2.2 확인
2.2.1 간단 확인
$ echo $CONFLUENT_HOME2.2.2 눈에 띄는 확인
$ kafka-topics
Create, delete, describe, or change a topic.
Option Description
------ -----------
--alter Alter the number of partitions,
replica assignment, and/or
configuration for the topic.
--at-min-isr-partitions if set when describing topics, only
show partitions whose isr count is
equal to the configured minimum.
--bootstrap-server <String: server to REQUIRED: The Kafka server to connect
connect to> to.
--command-config <String: command Property file containing configs to be
config property file> passed to Admin Client. This is used
only with --bootstrap-server option
for describing and altering broker
configs.
--config <String: name=value> A topic configuration override for the
topic being created or altered. The
following is a list of valid
configurations:
cleanup.policy
compression.type
delete.retention.ms
file.delete.delay.ms
flush.messages
flush.ms
follower.replication.throttled.
replicas
index.interval.bytes
leader.replication.throttled.replicas
local.retention.bytes
local.retention.ms
max.compaction.lag.ms
max.message.bytes
message.downconversion.enable
message.format.version
message.timestamp.difference.max.ms
message.timestamp.type
min.cleanable.dirty.ratio
min.compaction.lag.ms
min.insync.replicas
preallocate
remote.storage.enable
retention.bytes
retention.ms
segment.bytes
segment.index.bytes
segment.jitter.ms
segment.ms
unclean.leader.election.enable
See the Kafka documentation for full
details on the topic configs. It is
supported only in combination with --
create if --bootstrap-server option
is used (the kafka-configs CLI
supports altering topic configs with
a --bootstrap-server option).
--create Create a new topic.
--delete Delete a topic
--delete-config <String: name> A topic configuration override to be
removed for an existing topic (see
the list of configurations under the
--config option). Not supported with
the --bootstrap-server option.
--describe List details for the given topics.
--disable-rack-aware Disable rack aware replica assignment
--exclude-internal exclude internal topics when running
list or describe command. The
internal topics will be listed by
default
--help Print usage information.
--if-exists if set when altering or deleting or
describing topics, the action will
only execute if the topic exists.
--if-not-exists if set when creating topics, the
action will only execute if the
topic does not already exist.
--list List all available topics.
--partitions <Integer: # of partitions> The number of partitions for the topic
being created or altered (WARNING:
If partitions are increased for a
topic that has a key, the partition
logic or ordering of the messages
will be affected). If not supplied
for create, defaults to the cluster
default.
--replica-assignment <String: A list of manual partition-to-broker
broker_id_for_part1_replica1 : assignments for the topic being
broker_id_for_part1_replica2 , created or altered.
broker_id_for_part2_replica1 :
broker_id_for_part2_replica2 , ...>
--replication-factor <Integer: The replication factor for each
replication factor> partition in the topic being
created. If not supplied, defaults
to the cluster default.
--topic <String: topic> The topic to create, alter, describe
or delete. It also accepts a regular
expression, except for --create
option. Put topic name in double
quotes and use the '\' prefix to
escape regular expression symbols; e.
g. "test\.topic".
--topic-id <String: topic-id> The topic-id to describe.This is used
only with --bootstrap-server option
for describing topics.
--topics-with-overrides if set when describing topics, only
show topics that have overridden
configs
--unavailable-partitions if set when describing topics, only
show partitions whose leader is not
available
--under-min-isr-partitions if set when describing topics, only
show partitions whose isr count is
less than the configured minimum.
--under-replicated-partitions if set when describing topics, only
show under replicated partitions
--version Display Kafka version.2.3 이점
일일이 패키지 디렉터리로 접근하여 실행하지 않아도
해당 프로그램을 명령어로 동작할 수 있다.
3. 추가 설정 (logs)
기본적으로 Kafka와 Zookeeper의 log 저장 디렉터리는
머신의 On/Off시 데이터를 자동으로 삭제하는 /tmp 디렉터리에 저장된다.
따라서, 이 Log들을 저장할 디렉터리를 임의로 지정하고 참고할 예정이므로,
다음과 같이 설정한다.
3.1 log를 저장할 디렉터리 생성
$ mkdir data
$ cd data
$ mkdir kafka-logs
$ mkdir zookeeper3.2 log를 저장할 디렉터리로 설정 변경
$ cd $CONFLUENT_HOME/etc/kafka
$ vi or sudo nano server.properties
.
.
.
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/home/chan-kaf/data/kafka-logs
.
.
.
$ vi or sudo nano zookeeper.properties
.
.
.
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/home/chan-kaf/data/zookeeper
# the port at which the clients will connect
.
.
.4. Kafka 실행
실행 순서
Zookeeper -> Kafka
그렇지 않으면 오류
Kafka가 자체적으로 메타데이터와 상태 정보를 관리하기보다 주키퍼를 통해 이를 처리하기 때문
4.1 실행하기 전에..
먼저, SSH를 최소 3개 실행한다.
(해당 조건은 daemon 실행을 모른다는 가정이기 때문에,
4.2 Zookeeper 실행
$CONFLUENT_HOME/bin/zookeeper-server-start $CONFLUENT_HOME/etc/kafka/zookeeper.properties4.3 Kafka 실행
$CONFLUENT_HOME/bin/kafka-server-start $CONFLUENT_HOME/etc/kafka/server.properties4.4 Topic 생성
$ kafka-topics --bootstrap-server localhost:9092 --create --topic welcome_topic5. Log 확인
클라우드, 데이터, DevOps 엔지니어 지향 || 글보단 사진 지향