log4j deubg레벨로 인한 OOM 발생한 케이스

현상

OOM 발생으로 서비스 불가

접근방식

해당 서버는 IBM Java이기 때문에, Javacore가 남았고, 스레드 덤프 내 Current Thread 검색 해 OOM을 발생시킨 스레드 내용을 분석함.

1XMCURTHDINFO  Current thread

NULL           ----------------------

3XMTHREADINFO      "Thread-5706 [TEST-16605]" J9VMThread:0x000001002497F000, j9thread_t:0x0000010019612570, java/lang/Thread:0x070000000D1059C8, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x40DD, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x42B0029, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/StringBuffer.ensureCapacityImpl(StringBuffer.java:338(Compiled Code))
4XESTACKTRACE                at java/lang/StringBuffer.append(StringBuffer.java:204(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/pattern/MessagePatternConverter.format(MessagePatternConverter.java:58(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/PatternLayout.format(PatternLayout.java:561(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/WriterAppender.subAppend(WriterAppender.java:361(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/WriterAppender.append(WriterAppender.java:176(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/AppenderSkeleton.doAppend(AppenderSkeleton.java:286(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/helpers/AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:67(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/Category.callAppenders(Category.java:218(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/Category.forcedLog(Category.java:587(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/Category.log(Category.java:1168(Compiled Code))
4XESTACKTRACE                at org/apache/log4j/Category.log(Category.java:1177(Compiled Code))
4XESTACKTRACE                at org/slf4j/impl/Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:204(Compiled Code))
4XESTACKTRACE                at test/test/test/process/control/NonameCtr.sendPostNonameJSONData(NonameCtr.java(Compiled Code))
                                                                              .
4XESTACKTRACE                at test/test/tset/batch/jobs/testNonamePrcJob$KilledThread.run(testNonamePrcJob.java:347(Compiled Code))

해당 APPL은 Log4j deubg 레벨로 설정 돼 있고 console appender를 사용하고 있었음. 힙 덤프 분석시 LIST는 모두 null(\u0000) 값으로 들어가 있어서 해당 List가 OOM을 만들 었고 json을 받아서 처리하는 APPL 이였는데, json 데이터가 null값으로 크게 받고 해당 LIST를 로그로 생성하면서 OOM이 발생함.

조치

log4j의 console appender를 file appender로 변경하고, debug 레벨을 수정했으며, 문제가 되는 로거는 제거했다.