spark application 배포

deploy mode: client(default)

$ ./sbin/start-master.sh
$ ./sbin/start-worker.sh spark://spark-master-01:7177
$ ./bin/spark-submit --master spark://spark-master-01:7177 --class org.apache.spark.examples.SparkPi /skybluelee/spark3/examples/jars/spark-examples_2.12-3.2.1.jar 3000

spark-submit -> spark application 배포
--class org.apache.spark.examples.SparkPi -> class로 SparkPi 실행
/skybluelee/spark3/examples/jars/spark-examples_2.12-3.2.1.jar 3000 -> 자료 위치

jps
3794 CoarseGrainedExecutorBackend
3668 Worker
3861 Jps
3606 Master
3724 SparkSubmit

SparkSubmit -> spark driver program
CoarseGrainedExecutorBackend -> 할당받은 executor
SparkSubmit, CoarseGrainedExecutorBackend가 쌍으로 작동

SparkSubmit == driver

deploy mode: cluster

$ ./bin/spark-submit --deploy-mode cluster --master spark://spark-master-01:7177 --class org.apache.spark.examples.SparkPi /skybluelee/spark3/examples/jars/spark-examples_2.12-3.2.1.jar 3000

--deploy-mode cluster 가 추가됨

// running
$ jps
3668 Worker
3606 Master
4040 CoarseGrainedExecutorBackend
4074 Jps
3979 DriverWrapper

// complete
$ jps
4179 Jps
3668 Worker
3606 Master

작업 중 executor와 driverwrapper가 추가되고 작업이 끝나면 전부 사라짐

SparkSubmit은 배포만 하고 종료, 실제 driver program은 DriverWrapper

deploy mode: cluster(Core, Memory Config)

$ ./bin/spark-submit --total-executor-cores 2 --executor-cores 1 --executor-memory 512M --driver-memory 512M --deploy-mode cluster --master spark://spark-master-01:7177 --class org.apache.spark.examples.SparkPi /skybluelee/spark3/examples/jars/spark-examples_2.12-3.2.1.jar 3000

executor memory를 기존 1GB에서 512MB로 낮추었고 이는 Running Application(1)의 Application ID에서 확인 가능. 기존에는 1GB의 executor 1개만 사용하였지만 이번에는 512MB의 executor 2개 사용. 최적화가 좋아짐.

// running
$ jps
4401 Jps
4259 DriverWrapper
3668 Worker
4342 CoarseGrainedExecutorBackend
3606 Master
4332 CoarseGrainedExecutorBackend

// complete 
$ jps
3668 Worker
4502 Jps
3606 Master

deploy mode : cluster (Driver Auto Restart)

$ ./bin/spark-submit --supervise --total-executor-cores 2 --executor-cores 1 --executor-memory 512M --driver-memory 512M --deploy-mode cluster --master spark://spark-master-01:7177 --class org.apache.spark.examples.SparkPi /skybluelee/spark3/examples/jars/spark-examples_2.12-3.2.1.jar 9000

자동 재기동을 하기 위해서는 driver가 cluster 내부에 존재해야 함. cluster 바깥에 있다면 사람만이 재기동 시킬 수 있음.
--supervise -> driver가 죽으면 재기동시킴

$ jps
4627 CoarseGrainedExecutorBackend
3668 Worker
4566 DriverWrapper
3606 Master
4633 CoarseGrainedExecutorBackend
4682 Jps

DriverWrapper를 kill 한 후

$ kill -9 4566
$ jps
3668 Worker
3606 Master
5005 Jps
4991 DriverWrapper

Executor가 사라지고 DriverWrapper를 재기동

$ jps
3668 Worker
5076 CoarseGrainedExecutorBackend
3606 Master
5082 CoarseGrainedExecutorBackend
5119 Jps
4991 DriverWrapper

시간이 지나면 Executor도 생성됨

application을 영구적으로 kill

자동 재기동을 하면 kill을 하더라도 계속해서 진행됨
영구적으로 kill하기 위해서는 master Web UI: http://spark-master-01:8180 의 Running Driver를 kill 하거나

$ ./bin/spark-class org.apache.spark.deploy.Client kill <master url> <driver ID>

$ ./bin/spark-class org.apache.spark.deploy.Client kill spark-master-01:7177 driver-20230322133715-0003

driver ID는 master Web UI의 Running Drivers에서 확인 가능