(1)hive 설치하기

설치하기 앞서

필자는 hadoop 3.3.1 / hive 3.1.2로 진행함
hadoop을 아직 설치하지 않은 사람은 https://velog.io/@kidae92/Hadoop-Cluster 참고하길 바란다

설치

install

wget https://dlcdn.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
tar -xvf apache-hive-3.1.2-bin.tar.gz
mv apache-hive-3.1.2-bin /usr/local/hive

Initailzed data store

hdfs안에 폴더 생성 및 권한 부여

/usr/local/hadoop/bin/hadoop fs -mkdir /tmp
/usr/local/hadoop/bin/hadoop fs -mkdir -p /user/hive/warehouse
/usr/local/hadoop/bin/hadoop fs -chmod g+w /tmp
/usr/local/hadoop/bin/hadoop fs -chmod g+w /user/hive/warehouse

환경변수 설정

vi ~/.bashrc

export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:/usr/local/spark/bin
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export PATH=$PATH:$HIVE_HOME/bin

# save
source ~/.bashrc

Install MySQL

# MySQL connector 설치
# 필자는 MySQL 8을 도커로 띄울 예정이다
https://dev.mysql.com/downloads/connector/j/
ln -s /usr/share/java/mysql-connector-java-8.0.29.jar /usr/local/hive/lib/mysql-connector-java.jar  #필자는 mysql 8이기 때문에 버전보고 해야함
docker run -it --name mysql –v {volume_path}:/var/lib/mysql --restart=always –e MYSQL_ROOT_PASSWORD={password} -d -p 3306:3306 mysql:8

Create table for Hive user

mysql> CREATE USER ‘hiveuser’@’%’ IDENTIFIED BY ‘hivepassword’; 
mysql> GRANT all on *.* to ‘hiveuser’@localhost identified by ‘hivepassword’;

# 만약에 
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'identified by {password} 
# 이런 에러가 난다면 아래 명령어

GRANT ALL PRIVILEGES ON *.* TO 'hiveuser'@'%' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON connectdb.* to 'hiveuser'@'%' WITH GRANT OPTION;
flush privileges;

Edit hive environment

cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml
vi $HIVE_HOME/conf/hive-site.xml

<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?useSSL=false&amp;allowPublicKeyRetrieval=true</value>
    <description>JDBC connection string used by Hive Metastore</description>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>JDBC Driver class</description>
  </property>

  <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hiveuser</value>
      <description>Metastore database user name</description>
  </property>

  <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>{password}</value>
      <description>Metastore database password</description>
  </property>

  <property>
      <name>hive.metastore.uris</name>
      <value>thrift://localhost:9084</value>
      <description>Thrift server hostname and port</description>
  </property>
</configuration>

Initialize Hive scheme in Mysql

/usr/local/hive/bin/schematool -dbType mysql –initSchema

결과

혹시 DB이 없다는 에러가 뜨면 컨테이너 접속해서 해당 DB을 만들어주면 된다