[Kafka] 자주 사용하는 CLI 명령어 정리

서버 시작, 종료

Zookeeper 시작

• Kafka 내장 zookeeper 사용시

kafka/bin/zookeeper-server-start.sh 
{zookeeper_config_path}/zookeeper.properties

• 독립된 Zookeeper 사용시

zookeeper/bin/zkServer.sh start

Kafka 시작

• zookeeper running 상태에서 시작

kafka/bin/kafka-server-start.sh config/server.properties

zookeeper, kafka 종료하기

• kafka server 종료 -> zookeeper 종료

kafka/bin/kafka-server-stop.sh config/server.properties
kafka/bin/zookeeper-server-stop.sh config/zookeeper.properties

broker 정보 조회

Kafka broker 목록 조회

• 내장 Zookeeper 사용시

kafka/bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids

• 독립된 Zookeeper 사용시

zookeeper/bin/zkCli.sh -server localhost:2181 ls /brokers/ids

Kafka broker 상태 조회

• 내장 Zookeeper 사용시

kafka/bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/{broker_id}

• 독립된 Zookeeper 사용시

zookeeper/bin/zkCli.sh -server localhost:2181 get /brokers/ids/{broker_id}

topic, partition 관리

show topic list

kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

create the topic

kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic numtest

describe topic

• 각 topic 별 partition, leader,replication, ISR (In Sync Replica) 정보 출력

kafka/bin/kafka-topics.sh --describe --bootstrap-server localhost:9092

delete the topic

kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic my_topic

topic leader follower check

kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --topic my_topic --describe

topic, partition offset

./bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic test.topic

• --partitions or --topic-partitions 로 partition 지정

./bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic test.topic --partitions 0,1

• --topic-partitions 의 경우 0-8, -5 (0~5), 6- (6~) 등 범위 표현 가능

./bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic-partitions test.topic:0,test.topic2:1-3,test.topic3:5-,test.topic4:-8

Consumer, Consumer group

consuming message

• --from-beginning 옵션은 저장된 모든 메시지를 읽기 때문에, 데이터 양이 많을 경우 주의

kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic numtest

특정 partition message consuming

kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --partition 1 --topic numtest

consumer groups check

kafka/bin/kafka-consumer-groups.sh  --bootstrap-server localhost:9092 --list

consumer status and offset check

• client-id, host, end-offset 과 current-offset, consumer lag 등

kafka/bin/kafka-consumer-groups.sh  --bootstrap-server localhost:9092 --group <group_name> --describe

consumer group delete

kafka/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --delete --group <group_name>

log 조회

server log check

cat kafka/logs/server.log 

connect log check

cat kafka/logs/connect.log