[Kafka] Partition reassign 절차 정리

주의사항

• broker를 추가한다고 partition 이 자동으로 할당되지 않아 reassign 필요
• Partition reassign 절차는 많은 비용이 소모되므로 주의

---

Reassign 후보 생성

partition reassign 할 topic 목록 JSON 으로 작성

• 본 포스트에선 topic_list.json 으로 저장

> cat topic_list.json 
{
 "version":1,
 "topics":[{"topic":"my_topic"}]
}

reassign 후보 생성

> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topic_list.json --broker-list "0,2" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"my_topic","partition":0,"replicas":[2,0],"log_dirs":["any","any"]},{"topic":"my_topic","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"my_topic","partition":2,"replicas":[2,0],"log_dirs":["any","any"]},{"topic":"my_topic","partition":3,"replicas":[0],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"my_topic","partition":1,"replicas":[0,2],"log_dirs":["any","any"]},{"topic":"my_topic","partition":0,"replicas":[2,0],"log_dirs":["any","any"]},{"topic":"my_topic","partition":3,"replicas":[0,2],"log_dirs":["any","any"]},{"topic":"my_topic","partition":2,"replicas":[2,0],"log_dirs":["any","any"]}]}

하단의 Proposed partition reassignment configuration 을 json 파일로 저장

• 본 포스트에선 'proposed_partition.json' 파일로 저장

> cat proposed_partition.json 
{"version":1,"partitions":[{"topic":"my_topic","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"my_topic","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"my_topic","partition":3,"replicas":[0],"log_dirs":["any"]},{"topic":"my_topic","partition":2,"replicas":[2],"log_dirs":["any"]}]}

※ 상기 과정은 생략 후 JSON custom 하여 진행 가능

---

Reassign 실행

reassign 실행

> kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file proposed_partition.json --execute

reassign 진행상황 확인

• ※ reassign 시 사용했던 파일과 동일한 파일이어야함

> kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file proposed_partition.json --verify
Status of partition reassignment: 
Reassignment of partition my_topic-2 is still in progress
Reassignment of partition my_topic-0 is still in progress
Reassignment of partition my_topic-1 completed successfully
Reassignment of partition my_topic-3 completed successfully