kafka docker, jmx 포트 충돌

세팅 환경

1. kafka를 Docker로 사용.
2. 프로메테우스와 그라파나 모니터링을 위해 JMX 메트릭 사용.
3. 아래와 같이 JMX port를 9101를 활용하고, jmx_exporter의 Port로 8080을 활용함

services:
  
  broker:
    image: confluentinc/cp-kafka:7.7.0
    hostname: broker
    container_name: broker
    ports:
      - "8080:8080"
      - "9092:9092"
      - "9101:9101"
      - "29093:29093"
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://10.178.0.3:29092,PLAINTEXT_HOST://10.178.0.3:9092'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 3
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_PROCESS_ROLES: 'broker,controller'
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: server1
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@10.178.0.3:29093,2@10.178.0.4:29093, 3@10.178.0.5:29093'
      KAFKA_LISTENERS: 'PLAINTEXT://broker:29092,CONTROLLER://broker:29093,PLAINTEXT_HOST://0.0.0.0:9092'
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
      KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
      # Replace CLUSTER_ID with a unique base64 UUID using "bin/kafka-storage.sh random-uuid" 
      # See https://docs.confluent.io/kafka/operations-tools/kafka-tools.html#kafka-storage-sh
      CLUSTER_ID: 'TeSi-wM2RYORkvQHhvR5NQ'
      KAFKA_OPTS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.19.0.jar=8080:/usr/share/jmx_exporter/kafka-broker.yml
    volumes:
      - kafka:/var/lib/kafka/data
      - ../jmx-exporter:/usr/share/jmx_exporter/
      - ../connect/libs/jmx_prometheus_javaagent-0.19.0.jar:/opt/kafka/libs/jmx_prometheus_javaagent-0.19.0.jar

volumes:
  kafka:

문제

• Docker로 들어가서, kafka /bin 안에 있는 shell 파일들을 실행시키려고 할때, (/bin/kafka-topics 등등) 아래와 같은 에러가 생성 됨.

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.net.BindException: Address already in use
	at java.base/sun.nio.ch.Net.bind0(Native Method)
	at java.base/sun.nio.ch.Net.bind(Net.java:555)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
	at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
	at jdk.httpserver/sun.net.httpserver.ServerImpl.<init>(ServerImpl.java:142)
	at jdk.httpserver/sun.net.httpserver.HttpServerImpl.<init>(HttpServerImpl.java:50)
	at jdk.httpserver/sun.net.httpserver.DefaultHttpServerProvider.createHttpServer(DefaultHttpServerProvider.java:35)
	at jdk.httpserver/com.sun.net.httpserver.HttpServer.create(HttpServer.java:150)
	at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$Builder.build(HTTPServer.java:365)
	at io.prometheus.jmx.common.http.HTTPServerFactory.createHTTPServer(HTTPServerFactory.java:113)
	at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:41)
	... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message Outstanding error when calling method in invokeJavaAgentMainMethod at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 619
*** java.lang.instrument ASSERTION FAILED ***: "success" with message invokeJavaAgentMainMethod failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 459
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x89fb92]  jni_FatalError+0xb2
V  [libjvm.so+0x9ef35d]  JvmtiExport::post_vm_initialized()+0x12d
V  [libjvm.so+0xe64579]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x849
V  [libjvm.so+0x8b5132]  JNI_CreateJavaVM+0x52
C  [libjli.so+0x3b7a]  JavaMain+0x8a
C  [libjli.so+0x84b9]  ThreadJavaMain+0x9

Aborted (core dumped)

해결책

• kafka docker 속에 bin파일속에 들어가 있는 파일들을 실행시키기 위해서는 항상 KAFKA_OPTS를 함께 가지고 와서 실행을 시킴.
• 따라서 KAFKA_OPTS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.19.0.jar=8080:/usr/share/jmx_exporter/kafka-broker.yml 이 명령어가 또 실행되기 때문에, 이미 잡혀있는 port라고 뜨는 것!
• 결론적으로 KAFKA_OPTS를 EXTRA_ARGS 로 변경해서 yml파일을 사용하면 됨.