Kafka cluster 구성일 시 __consumer_offset topic의 reader, replicas, Isr가 하나로 설정 되어 있을 때 해결방안을 알아보자!

Karim·2021년 11월 3일
kafka장애
5

kafka 운영

목록 보기
6/9
post-thumbnail

1. Version

💬

  • Kafka : 2.6.0

2. 문제원인 및 방안

💬 원인

  • kafka를 single 서버로 구축 한 후,
    cluster 구축 시 그 dir를 그대로 3대의 서버 에 올린다음 myid랑 properties cluster로 설정한 후 실행시켜서 발생한 듯 하다.
    !!!구축자 실수!!!
  • Kafka cluster 구성일 시 특정 서버 1대가 죽으면 consumer에서 message를 polling 못하게 된다!
    !!!cluster 의미가 없어짐!!!

💬 해결방안

  • __consumer_offset topic의 1개로 설정 된 레플리카를 3개로 변경한다.!

3. __consumer_offset가 무엇일까?

💬

  • Kafka는 consumer group이 읽어 간 offset 정보를 정한다.
  • 그 정보는 __consumer_offsets topic에 저장된다.
  • consumer가 kafka에서 받은 message를 처리하고 offset에 commit을 한다.
  • 만약 consumer가 죽으면 위에 저장된 offset부터 다시 읽어 올 수 있다.

4. __consumer_offset describe

💬 비 정상

  • ReplicationFactor, Leader, Replicas, Isr가 한 서버로 몰려있는 것을 확인할 수 있다.
  • 이런 경우 서버1이 죽으면 나머지 서버2,서버3은 message를 구독할 수 없다.!!
    !!!장애!!! 
[karim@kafka_cluster bin]$ ./kafka-topics.sh --describe --zookeeper server1:2181,server2:2181,server3:2181 --topic __consumer_offsets
Topic: __consumer_offsets       PartitionCount: 50      ReplicationFactor: 1    Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 1    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 2    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 3    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 4    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 5    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 6    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 7    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 8    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 9    Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 10   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 11   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 12   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 13   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 14   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 15   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 16   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 17   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 18   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 19   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 20   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 21   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 22   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 23   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 24   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 25   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 26   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 27   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 28   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 29   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 30   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 31   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 32   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 33   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 34   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 35   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 36   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 37   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 38   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 39   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 40   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 41   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 42   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 43   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 44   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 45   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 46   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 47   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 48   Leader: 1       Replicas: 1 Isr: 1
        Topic: __consumer_offsets       Partition: 49   Leader: 1       Replicas: 1 Isr: 1

5. 해결방안

💬 1. kafka 실행파일이 있는 directory 이동

[kaimr@kafka_cluster bin]$ cd .../kafka/bin/

💬 2. __consumer_offset topic replicas 변경

  • 기존 1개로 설정되어 있던것을 3으로 변경하기 위해서 json 파일을 생성 합니다.
vi replicas.json
{"version":1,
"partitions":[
{"topic":"__consumer_offsets","partition":0,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":1,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":2,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":3,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":4,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":5,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":6,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":7,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":8,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":9,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":10,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":11,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":12,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":13,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":14,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":15,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":16,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":17,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":18,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":19,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":20,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":21,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":22,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":23,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":24,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":25,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":26,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":27,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":28,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":29,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":30,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":31,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":32,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":33,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":34,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":35,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":36,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":37,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":38,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":39,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":40,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":41,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":42,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":43,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":44,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":45,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":46,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":47,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":48,"replicas":[1,2,3]},
{"topic":"__consumer_offsets","partition":49,"replicas":[1,2,3]}
]}

💬 3. replicas.json 설정 적용

[karim@kafka_cluster bin]$ ./kafka-reassign-partitions.sh --zookeeper server1:2181,server2:2181,server3:2181 --reassignment-json-file replics.json --execute

💬 4. replicas.json 설정 적용 확인

6. 적용 후__consumer_offset describe

💬 정상

  • ReplicationFactor가 3 Leader, Replicas, Isr가 고르게 분배되어 있는 것을 확인할 수 있다.
[karim@kafka_cluster bin]$ ./kafka-topics.sh --describe --zookeeper server1:2181,server2:2181,server3:2181 --topic __consumer_offsets
Topic: __consumer_offsets       PartitionCount: 50      ReplicationFactor: 3    Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
        Topic: __consumer_offsets       Partition: 0    Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 1    Leader: 2       Replicas: 2,1,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 2    Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 3    Leader: 1       Replicas: 1,2,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 4    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 5    Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 6    Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 7    Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 8    Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 9    Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 10   Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 11   Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 12   Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 13   Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 14   Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 15   Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 16   Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 17   Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 18   Leader: 1       Replicas: 1,3,2 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 19   Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 20   Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 21   Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 22   Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 23   Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 24   Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 25   Leader: 2       Replicas: 2,1,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 26   Leader: 3       Replicas: 3,2,1 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 27   Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 28   Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 29   Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 30   Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 31   Leader: 2       Replicas: 2,1,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 32   Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 33   Leader: 1       Replicas: 1,2,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 34   Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 35   Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 36   Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 37   Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 38   Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 39   Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 40   Leader: 2       Replicas: 2,3,1 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 41   Leader: 3       Replicas: 3,1,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 42   Leader: 1       Replicas: 1,3,2 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 43   Leader: 2       Replicas: 2,1,3 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 44   Leader: 3       Replicas: 3,2,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 45   Leader: 1       Replicas: 1,2,3 Isr: 3,2,1
        Topic: __consumer_offsets       Partition: 46   Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 47   Leader: 3       Replicas: 3,1,2 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 48   Leader: 1       Replicas: 1,3,2 Isr: 2,3,1
        Topic: __consumer_offsets       Partition: 49   Leader: 2       Replicas: 2,1,3 Isr: 3,2,1

📌 여담

  • kafka를 실무에서 사용하는 동안 장애가 생기면 --describe 명령어를 가장 먼저 많이 사용했던거 같다!
  • 가끔 kafka의 데이터가 많으면 Isr이 깨지는 경우도 있다.!
profile
Karim
나도 보기 위해 정리해 놓은 벨로그
이전 포스트

kafka consumer auto.offset.reset 옵션을 알아보자

다음 포스트

kafka 에서 Connection to node -1 (/ip:port) could not be established. Broker may not be available. 해결방안을 알아보자!

2개의 댓글

comment-user-thumbnail
김신영
2021년 11월 5일

안녕하세요 최근에 kafka를 사용해 메신져 기능을 만들어 보려 하는데 topic안에 저장되어있는 메세지 전체를 한번에 불러올 수 있는 방법이 있나요?..

1개의 답글