[확실하지 않음] zeppelin 0.11.1 버그 리포트

유알·2024년 7월 2일
0

일단 확실치 않은 상태에서 버그 리포트 적어봄(나중에 제보할거임)

실행 환경

Ubuntu 24.04 LTS
완전 분산모드 hadoop, yarn
zeppelin 은 다른 서버에서 띄우고 spark yarn cluster 모드로 접속하는 상황

실행 창에 다음과 같이 입력

%spark.ipyspark
print("hel")

zeppelin 0.11.1 버전

문제 상황

zeppelin 쉘 측에 표시된 에러

%spark.ipyspark
print("hel")
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861)
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769)
	at org.apache.zeppelin.scheduler.Job.run(Job.java:186)
	at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135)
	at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
	at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:322)
	at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:333)
	at org.apache.zeppelin.spark.IPySparkInterpreter.open(IPySparkInterpreter.java:57)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
	... 8 more
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:140)
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
	... 12 more
Caused by: scala.reflect.internal.FatalError: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0006/container_1719905050899_0006_01_000001/._spark-scala-2.12-0.11.1.jar
	at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:113)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:101)
	at scala.tools.nsc.util.ClassPath.list(ClassPath.scala:36)
	at scala.tools.nsc.util.ClassPath.list$(ClassPath.scala:36)
	at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:30)
	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:298)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:250)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:258)
	at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:74)
	at scala.tools.nsc.Global.rootMirror(Global.scala:72)
	at scala.tools.nsc.Global.rootMirror(Global.scala:44)
	at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:301)
	at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:301)
	at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1511)
	at scala.tools.nsc.Global$Run.<init>(Global.scala:1213)
	at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:124)
	at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:146)
	at org.apache.zeppelin.spark.SparkScala212Interpreter.createSparkILoop(SparkScala212Interpreter.scala:195)
	at org.apache.zeppelin.spark.AbstractSparkScalaInterpreter.open(AbstractSparkScalaInterpreter.java:116)
	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:124)
	... 13 more
Caused by: java.io.IOException: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0006/container_1719905050899_0006_01_000001/._spark-scala-2.12-0.11.1.jar
	at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:190)
	at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:238)
	at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:235)
	at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:272)
	at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:272)
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76)
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63)
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62)
	at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:58)
	at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:105)
	... 40 more
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:231)
	at java.util.zip.ZipFile.<init>(ZipFile.java:157)
	at java.util.zip.ZipFile.<init>(ZipFile.java:171)
	at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:187)
	... 49 more

yarn 컨테이너 안에서 실행되는 파일의 압축 도중 문제가 생긴것으로 보임

/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0006/container_1719905050899_0006_01_000001/._spark-scala-2.12-0.11.1.jar

해당 컨테이너의 tmp 디렉토리에 접근하여 직접 파일을 확인해봄

._spark-scala-2.12-0.11.1.jar

문제의 이 파일의 특성

  • jar 치고는 너무 크기가 작음
  • jar 의 파일 형식 아닌것 같음(unzip, jar tf 명령어에서 에러 남)

이 파일은 zeppelin 릴리즈의 특정 파일을 복사한 것이라는 것을 알게 됨

$ZEPPELIN_HOME/interpreter/spark/scalar-2.12/

이 위치의 파일을 yarn 컨테이너로 복사한듯

$ jar tvf ./._spark-scala-2.12-0.11.1.jar
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.<init>(ZipFile.java:231)
	at java.util.zip.ZipFile.<init>(ZipFile.java:157)
	at java.util.zip.ZipFile.<init>(ZipFile.java:128)
	at sun.tools.jar.Main.list(Main.java:1115)
	at sun.tools.jar.Main.run(Main.java:293)
	at sun.tools.jar.Main.main(Main.java:1288)

문제가 발생하는 것을 볼 수 있다.

$ cat ./._spark-scala-2.12-0.11.1.jar
Mac OS X        	2q�ATTR��
                                 �
                                  com.apple.provenanceՅ88DH/

뭐 말도 안되지만, cat으로 출력해 보았다. 아마 애플 맥의 메타데이터인것 같다.
아마도 빌드하시는 분이 실수하셔서 맥에서 빌드하면서 실수로 생성한듯 하다.

$ ll interpreter/spark/scala-2.12
total 44
drwxr-xr-x 2 on5949 on5949  4096 Jul  2 08:47 ./
drwxr-xr-x 8 on5949 on5949  4096 Nov 14  2022 ../
-rw-r--r-- 1 on5949 on5949   163 Nov 14  2022 ._spark-scala-2.12-0.11.1.jar
-rw-r--r-- 1 on5949 on5949 32660 Nov 14  2022 spark-scala-2.12-0.11.1.jar

결정적으로 기존 버전의 zeppelin에서는 이 파일이 존재하지 않는다.

연관된 질문

스택 오버플로우에도 이와 관련된 질문이 있다.
https://stackoverflow.com/questions/78672880/interpreterexception-in-apache-zeppelin-v-0-11-1-executing-a-paragraph

한줄 요약

빌드 실수인듯

느낀점

진짜... 이거 때문에 3일 날렸다..

뭐든지 최신버전은 절대 쓰지말자.. 전에 스프링에서도 내가 1커밋 했는데 이로 인해서 뭐가 안된다는 사람들이 엄청 많았었따.

풀 로그

yarn std.out

 INFO [2024-07-02 18:51:30,027] ({main} Logging.scala[logInfo]:60) - Registering signal handler for TERM
 INFO [2024-07-02 18:51:30,057] ({main} Logging.scala[logInfo]:60) - Registering signal handler for HUP
 INFO [2024-07-02 18:51:30,058] ({main} Logging.scala[logInfo]:60) - Registering signal handler for INT
 WARN [2024-07-02 18:51:33,091] ({main} NativeCodeLoader.java[<clinit>]:60) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 INFO [2024-07-02 18:51:33,464] ({main} Logging.scala[logInfo]:60) - ApplicationAttemptId: appattempt_1719905050899_0007_000001
 INFO [2024-07-02 18:51:35,473] ({main} Logging.scala[logInfo]:60) - Starting the user application in a separate Thread
 INFO [2024-07-02 18:51:35,591] ({main} Logging.scala[logInfo]:60) - Waiting for spark context initialization...
 INFO [2024-07-02 18:51:36,166] ({RegisterThread} RemoteInterpreterServer.java[run]:620) - Start registration
 INFO [2024-07-02 18:51:36,233] ({RemoteInterpreterServer-Thread} RemoteInterpreterServer.java[run]:200) - Launching ThriftServer at 192.168.0.10:45787
 INFO [2024-07-02 18:51:37,276] ({RegisterThread} RemoteInterpreterServer.java[run]:634) - Registering interpreter process
 INFO [2024-07-02 18:51:37,331] ({RegisterThread} RemoteInterpreterServer.java[run]:636) - Registered interpreter process
 INFO [2024-07-02 18:51:37,331] ({RegisterThread} RemoteInterpreterServer.java[run]:657) - Registration finished
 WARN [2024-07-02 18:51:38,034] ({pool-7-thread-1} ZeppelinConfiguration.java[<init>]:87) - Failed to load XML configuration, proceeding with a default,for a stacktrace activate the debug log
 INFO [2024-07-02 18:51:38,052] ({pool-7-thread-1} ZeppelinConfiguration.java[create]:137) - Server Host: 127.0.0.1
 INFO [2024-07-02 18:51:38,053] ({pool-7-thread-1} ZeppelinConfiguration.java[create]:141) - Server Port: 8080
 INFO [2024-07-02 18:51:38,053] ({pool-7-thread-1} ZeppelinConfiguration.java[create]:143) - Context Path: /
 INFO [2024-07-02 18:51:38,053] ({pool-7-thread-1} ZeppelinConfiguration.java[create]:144) - Zeppelin Version: 0.11.1
 INFO [2024-07-02 18:51:38,055] ({pool-7-thread-1} RemoteInterpreterServer.java[createLifecycleManager]:293) - Creating interpreter lifecycle manager: org.apache.zeppelin.interpreter.lifecycle.NullLifecycleManager
 INFO [2024-07-02 18:51:38,056] ({pool-7-thread-1} RemoteInterpreterServer.java[init]:236) - Creating RemoteInterpreterEventClient with connection pool size: 100
 INFO [2024-07-02 18:51:38,667] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.SparkInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,679] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.SparkSqlInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,699] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.PySparkInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,724] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.IPySparkInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,735] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.SparkRInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,744] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.SparkIRInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,753] ({pool-7-thread-1} RemoteInterpreterServer.java[createInterpreter]:406) - Instantiate interpreter org.apache.zeppelin.spark.SparkShinyInterpreter, isForceShutdown: false
 INFO [2024-07-02 18:51:38,907] ({pool-7-thread-2} SchedulerFactory.java[<init>]:56) - Scheduler Thread Pool Size: 100
 INFO [2024-07-02 18:51:38,914] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: interpreter_1143129738
 INFO [2024-07-02 18:51:38,928] ({pool-7-thread-2} SchedulerFactory.java[createOrGetParallelScheduler]:88) - Create ParallelScheduler: org.apache.zeppelin.spark.SparkSqlInterpreter1803283727 with maxConcurrency: 10
 INFO [2024-07-02 18:51:38,932] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: interpreter_1371342314
 INFO [2024-07-02 18:51:38,933] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: interpreter_141019945
 INFO [2024-07-02 18:51:38,934] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: org.apache.zeppelin.spark.SparkRInterpreter1665384
 INFO [2024-07-02 18:51:38,935] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: interpreter_873566014
 INFO [2024-07-02 18:51:38,937] ({pool-7-thread-2} SchedulerFactory.java[createOrGetFIFOScheduler]:76) - Create FIFOScheduler: interpreter_1398357413
 INFO [2024-07-02 18:51:39,056] ({FIFOScheduler-interpreter_141019945-Worker-1} AbstractScheduler.java[runJob]:130) - Job paragraph_1719913403215_1110656502 started by scheduler interpreter_141019945
 INFO [2024-07-02 18:51:39,162] ({FIFOScheduler-interpreter_141019945-Worker-1} SparkInterpreter.java[extractScalaVersion]:272) - Using Scala: 2.12
 INFO [2024-07-02 18:51:39,276] ({FIFOScheduler-interpreter_141019945-Worker-1} SparkScala212Interpreter.scala[createSparkILoop]:170) - Scala shell repl output dir: /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/tmp/spark7015610283561317121
 INFO [2024-07-02 18:51:40,066] ({FIFOScheduler-interpreter_141019945-Worker-1} SparkScala212Interpreter.scala[createSparkILoop]:179) - UserJars: file:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/__app__.jar:file:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/spark-scala-2.12-0.11.1.jar:file:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/._spark-scala-2.12-0.11.1.jar:file:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/zeppelin-interpreter-shaded-0.11.1.jar:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/__app__.jar:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/spark-scala-2.12-0.11.1.jar:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/._spark-scala-2.12-0.11.1.jar:/tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/zeppelin-interpreter-shaded-0.11.1.jar
ERROR [2024-07-02 18:51:44,615] ({FIFOScheduler-interpreter_141019945-Worker-1} SparkInterpreter.java[open]:139) - Fail to open SparkInterpreter
scala.reflect.internal.FatalError: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/._spark-scala-2.12-0.11.1.jar
	at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:113) ~[scala-compiler-2.12.18.jar:?]
	at scala.collection.Iterator.foreach(Iterator.scala:943) ~[scala-library-2.12.18.jar:?]
	at scala.collection.Iterator.foreach$(Iterator.scala:943) ~[scala-library-2.12.18.jar:?]
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) ~[scala-library-2.12.18.jar:?]
	at scala.collection.IterableLike.foreach(IterableLike.scala:74) ~[scala-library-2.12.18.jar:?]
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73) ~[scala-library-2.12.18.jar:?]
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56) ~[scala-library-2.12.18.jar:?]
	at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:101) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.util.ClassPath.list(ClassPath.scala:36) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.util.ClassPath.list$(ClassPath.scala:36) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:30) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:298) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:250) ~[scala-compiler-2.12.18.jar:?]
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:258) ~[scala-reflect-2.12.18.jar:?]
	at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:74) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.Global.rootMirror(Global.scala:72) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.Global.rootMirror(Global.scala:44) ~[scala-compiler-2.12.18.jar:?]
	at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:301) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:301) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1511) ~[scala-reflect-2.12.18.jar:?]
	at scala.tools.nsc.Global$Run.<init>(Global.scala:1213) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:124) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:146) ~[scala-compiler-2.12.18.jar:?]
	at org.apache.zeppelin.spark.SparkScala212Interpreter.createSparkILoop(SparkScala212Interpreter.scala:195) ~[spark-scala-2.12-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.spark.AbstractSparkScalaInterpreter.open(AbstractSparkScalaInterpreter.java:116) ~[__app__.jar:0.11.1]
	at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:124) ~[__app__.jar:0.11.1]
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:322) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:333) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.spark.IPySparkInterpreter.open(IPySparkInterpreter.java:57) ~[__app__.jar:0.11.1]
	at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.scheduler.Job.run(Job.java:186) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
Caused by: java.io.IOException: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0007/container_1719905050899_0007_01_000001/._spark-scala-2.12-0.11.1.jar
	at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:190) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:238) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:235) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:272) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:272) ~[scala-reflect-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:58) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:105) ~[scala-compiler-2.12.18.jar:?]
	... 40 more
Caused by: java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method) ~[?:1.8.0_412]
	at java.util.zip.ZipFile.<init>(ZipFile.java:231) ~[?:1.8.0_412]
	at java.util.zip.ZipFile.<init>(ZipFile.java:157) ~[?:1.8.0_412]
	at java.util.zip.ZipFile.<init>(ZipFile.java:171) ~[?:1.8.0_412]
	at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:187) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:238) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:235) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:272) ~[scala-reflect-2.12.18.jar:?]
	at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:272) ~[scala-reflect-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:58) ~[scala-compiler-2.12.18.jar:?]
	at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:105) ~[scala-compiler-2.12.18.jar:?]
	... 40 more
 INFO [2024-07-02 18:51:44,642] ({FIFOScheduler-interpreter_141019945-Worker-1} SparkInterpreter.java[close]:191) - Close SparkInterpreter
 INFO [2024-07-02 18:51:44,643] ({FIFOScheduler-interpreter_141019945-Worker-1} IPySparkInterpreter.java[close]:147) - Close IPySparkInterpreter
 INFO [2024-07-02 18:51:44,676] ({FIFOScheduler-interpreter_141019945-Worker-1} AbstractScheduler.java[runJob]:150) - Job paragraph_1719913403215_1110656502 finished by scheduler interpreter_141019945 with status ERROR
ERROR [2024-07-02 18:53:15,836] ({main} Logging.scala[logError]:97) - Uncaught exception: 
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259) ~[scala-library-2.12.18.jar:?]
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263) ~[scala-library-2.12.18.jar:?]
	at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:48) ~[spark-common-utils_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310) ~[spark-core_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:506) [spark-yarn_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:265) [spark-yarn_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:934) [spark-yarn_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:933) [spark-yarn_2.12-3.5.1.jar:3.5.1]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_412]
	at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_412]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) [hadoop-client-api-3.3.4.jar:?]
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:933) [spark-yarn_2.12-3.5.1.jar:3.5.1]
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) [spark-yarn_2.12-3.5.1.jar:3.5.1]
 INFO [2024-07-02 18:53:15,846] ({main} Logging.scala[logInfo]:60) - Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
	at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:48)
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:506)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:265)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:934)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:933)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:933)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
)
 INFO [2024-07-02 18:53:15,867] ({ShutdownThread} RemoteInterpreterServer.java[run]:674) - Shutting down...
 INFO [2024-07-02 18:53:15,869] ({ShutdownThread} RemoteInterpreterServer.java[run]:675) - Shutdown initialized by ShutdownHook
 INFO [2024-07-02 18:53:15,870] ({ShutdownThread} SchedulerFactory.java[destroy]:61) - Destroy all executors
 INFO [2024-07-02 18:53:15,871] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler interpreter_1371342314
 WARN [2024-07-02 18:53:15,872] ({SchedulerFactory3} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 INFO [2024-07-02 18:53:15,882] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler org.apache.zeppelin.spark.SparkRInterpreter1665384
 INFO [2024-07-02 18:53:15,884] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler interpreter_873566014
 WARN [2024-07-02 18:53:15,885] ({SchedulerFactory5} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 INFO [2024-07-02 18:53:15,889] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler interpreter_1398357413
 INFO [2024-07-02 18:53:15,890] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler org.apache.zeppelin.spark.SparkSqlInterpreter1803283727
 INFO [2024-07-02 18:53:15,891] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler interpreter_141019945
 WARN [2024-07-02 18:53:15,894] ({SchedulerFactory6} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 WARN [2024-07-02 18:53:15,895] ({SchedulerFactory4} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 WARN [2024-07-02 18:53:15,895] ({SchedulerFactory7} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 INFO [2024-07-02 18:53:15,898] ({ShutdownThread} SchedulerFactory.java[destroy]:65) - Stopping Scheduler interpreter_1143129738
 WARN [2024-07-02 18:53:15,901] ({SchedulerFactory1} AbstractScheduler.java[run]:91) - FIFOScheduler is interrupted
 WARN [2024-07-02 18:53:15,903] ({SchedulerFactory2} AbstractScheduler.java[run]:91) - ParallelScheduler is interrupted
 INFO [2024-07-02 18:53:15,909] ({ShutdownThread} RemoteInterpreterServer.java[run]:707) - Unregister interpreter process
 INFO [2024-07-02 18:53:15,913] ({shutdown-hook-0} Logging.scala[logInfo]:60) - Deleting staging directory hdfs://192.168.0.8:9000/user/on5949/.sparkStaging/application_1719905050899_0007
 INFO [2024-07-02 18:53:16,535] ({pool-7-thread-1} RemoteInterpreterServer.java[shutdown]:251) - Unregister interpreter process
ERROR [2024-07-02 18:53:16,541] ({pool-7-thread-1} ProcessFunction.java[process]:47) - Internal error processing shutdown
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_412]
	at java.lang.Runtime.removeShutdownHook(Runtime.java:231) ~[?:1.8.0_412]
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.shutdown(RemoteInterpreterServer.java:259) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1960) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1940) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
 INFO [2024-07-02 18:53:16,545] ({pool-7-thread-2} RemoteInterpreterServer.java[shutdown]:251) - Unregister interpreter process
ERROR [2024-07-02 18:53:16,550] ({pool-7-thread-2} ProcessFunction.java[process]:47) - Internal error processing shutdown
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_412]
	at java.lang.Runtime.removeShutdownHook(Runtime.java:231) ~[?:1.8.0_412]
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.shutdown(RemoteInterpreterServer.java:259) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1960) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1940) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
 INFO [2024-07-02 18:53:16,553] ({pool-7-thread-3} RemoteInterpreterServer.java[shutdown]:251) - Unregister interpreter process
ERROR [2024-07-02 18:53:16,555] ({pool-7-thread-3} ProcessFunction.java[process]:47) - Internal error processing shutdown
java.lang.IllegalStateException: Shutdown in progress
	at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_412]
	at java.lang.Runtime.removeShutdownHook(Runtime.java:231) ~[?:1.8.0_412]
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.shutdown(RemoteInterpreterServer.java:259) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1960) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$shutdown.getResult(RemoteInterpreterService.java:1940) ~[zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at shaded.org.apache.zeppelin.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) [zeppelin-interpreter-shaded-0.11.1.jar:0.11.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
 INFO [2024-07-02 18:53:16,850] ({shutdown-hook-0} Logging.scala[logInfo]:60) - Shutdown hook called
 INFO [2024-07-02 18:53:19,061] ({RemoteInterpreterServer-Thread} RemoteInterpreterServer.java[run]:205) - RemoteInterpreterServer-Thread finished
 INFO [2024-07-02 18:53:19,358] ({ShutdownThread} RemoteInterpreterServer.java[run]:736) - Shutting down
profile
유알
더 좋은 구조를 고민하는 개발자 입니다
이전 포스트

실험 결과 공유 : 라즈베리파이 클러스터에 하둡과 쿠버네티스 동시 구축하기

다음 포스트

소스 코드 까지 까보며 찾은 zeppelin 오류 해결법..

1개의 댓글

comment-user-thumbnail
sda1547
2024년 8월 12일

유알님의 글을 보고 힌트를 얻어 해결한 내용 공유드립니다.
문제되는 ._* 접두사 파일 제거 후 실행하면 정상적으로 실행됩니다.
find $ZEPPELIN_HOME -name "._*" -delete

답글 달기