📗Spark on Kubernetes Provisioning

🏳️‍🌈 [궁금한점]

• Spark를 쿠버네티스에서 구동할 수 있나

목차

• 설치
  • 헬름 REPO 설정
  • SPARK-ON-KUBERNETES 설치
  • 서비스 어카운트 생성
  • 클러스터 롤 바인딩
  • 상태 확인
  • 오퍼레이터 POD 확인
• 잡 생성
  • SPARK-SUBMIT
  • MYSUBMIT POD 확인

설치

헬름 REPO 설정

helm repo add spark-operator https://kubeflow.github.io/spark-operator

SPARK-ON-KUBERNETES 설치

helm delete myspark -n myspark

helm install myspark spark-operator/spark-operator \
    --namespace myspark \
    --create-namespace \
    --set webhook.enable=true

서비스 어카운트 생성

k delete sa myspark-sa -n myspark
kubectl create serviceaccount myspark-sa -n myspark

클러스터 롤 바인딩

k delete clusterrolebinding myspark-cluster-admin-binding
kubectl create clusterrolebinding myspark-cluster-admin-binding --clusterrole=cluster-admin --serviceaccount=myspark:myspark-sa

상태 확인

vagrant@master:~$ helm status --namespace myspark myspark
NAME: myspark
LAST DEPLOYED: Sun Apr 20 08:42:56 2025
NAMESPACE: myspark
STATUS: deployed
REVISION: 1
TEST SUITE: None

오퍼레이터 POD 확인

vagrant@master:~$ k get pod -n myspark
NAME                                                READY   STATUS    RESTARTS   AGE
myspark-spark-operator-controller-9b884b965-9dwdr   1/1     Running   0          84s
myspark-spark-operator-webhook-6784dd785-sz7p8      1/1     Running   0          84s

잡 생성

SPARK-SUBMIT

spark-submit --master k8s://192.168.56.10:6443 --name mysubmit \
    --deploy-mode cluster \
    --driver-cores 1 \
    --driver-memory 512m \
    --num-executors 1 \
    --executor-cores 1 \
    --executor-memory 512m \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.namespace=myspark \
    --conf spark.kubernetes.container.image=spark:3.5.5 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=myspark-sa \
    local:///opt/spark/examples/jars/spark-examples_2.12-3.5.5.jar

MYSUBMIT POD 확인

vagrant@master:~$ k get po -n myspark
NAME                                                READY   STATUS      RESTARTS   AGE
myspark-spark-operator-controller-9b884b965-9dwdr   1/1     Running     0          7m9s
myspark-spark-operator-webhook-6784dd785-sz7p8      1/1     Running     0          7m9s
mysubmit-b8de529652633e64-driver                    0/1     Completed   0          47s

• 로그 확인

k logs mysubmit-b8de529652633e64-driver -n myspark
...
Pi is roughly 3.138275691378457
...