📗 프로비저닝 - Spark / Docker

🏳️‍🌈 [궁금한점]

• Hive 4.0.1에 연결하여 Spark로 Query를 날리는 방법

---

🔗[목차]

Spark 설치

1️⃣ 준비

hms 설치

HMS 프로비저닝

2️⃣ 설정

도커 네트워크 생성

docker network create mynetwork

3️⃣ 설치

Spark Master 설치

docker run -d \
  --name spark-master \
  -e SPARK_MODE=master \
  -p 8080:8080 \
  -p 7077:7077 \
  --network mynetwork \
  bitnami/spark:3.5.1

Spark Worker 설치

docker run -d \
  --name spark-worker \
  -e SPARK_MODE=worker \
  -e SPARK_MASTER_URL=spark://spark-master:7077 \
  --link spark-master \
  -p 8081:8081 \
  --network mynetwork \
  bitnami/spark:3.5.1

4️⃣ 검증 - 설치 확인

Spark Shell 실행

docker exec -it spark-master spark-shell \
  --master spark://spark-master:7077 \
  --conf spark.hadoop.hive.metastore.uris=thrift://hms:9083 \
  --conf spark.sql.catalogImplementation=hive

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
25/06/01 08:35:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://e4bcb6bf79eb:4040
Spark context available as 'sc' (master = spark://spark-master:7077, app id = app-20250601083515-0000).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.1
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 17.0.12)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

데이터베이스 연결 확인

spark-shell \
  --conf spark.sql.catalog.myhive=hive \
  --conf spark.hadoop.hive.metastore.uris=thrift://localhost:9083

spark-shell \
  --master spark://spark-master:7077 \
  --conf spark.hadoop.hive.metastore.uris=thrift://hms:9083 \
  --conf spark.sql.catalogImplementation=hive

scala> spark.conf.get("spark.sql.catalogImplementation")
res0: String = hive

scala> spark.conf.get("spark.hadoop.hive.metastore.uris")
res1: String = thrift://localhost:9083

scala> spark.sql("SHOW DATABASES").show()
+---------+
|namespace|
+---------+
|  default|
|     mydb|
+---------+