driver, master, worker

master를 local에서 multi-thread 환경으로 실행

$ cd /skybluelee/spark3
$ ./bin/spark-shell --master local[*]

driver web UI: http://spark-master-01:4040
로컬에서 실행하였으므로 executors 존재 X

cluster 분산 환경 실행

master 실행

$ cd /skybluelee/spark3
$ ./sbin/start-master.sh

$ spark@spark-master-01:~$ jps
1840 Master
1984 Jps

master web UI: http://spark-master-01:8180

worker 실행

$ ./sbin/start-worker.sh spark://spark-master-01:7177
starting org.apache.spark.deploy.worker.Worker, logging to /skybluelee/spark3/logs/spark-spark-org.apache.spark.deploy.worker.Worker-1-spark-master-01.out

master port에 맞게 worker 실행

worker web UI: http://spark-master-01:8180
master web UI에서 worker가 하나 추가된 것 확인 가능

spark shell 실행

$ ./bin/spark-shell --master spark://spark-master-01:7177

$ jps
1840 Master
2225 CoarseGrainedExecutorBackend
2151 SparkSubmit
2027 Worker
2284 Jps

master web UI에서 Running Applications (1) 확인

spark shell 종료

$ jps
1840 Master
2356 Jps
2027 Worker

master web UI에서
Running Applications (0)
Completed Applications (1) 확인