zookeeper 설치

Han Hanju·2021년 6월 18일
hadoop
0
post-thumbnail

1) zookeeper 다운로드 및 압축해제

wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz


--2021-06-19 01:41:29--  https://archive.apache.org/dist/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
Resolving archive.apache.org (archive.apache.org)... 138.201.131.134, 2a01:4f8:172:2ec5::2
Connecting to archive.apache.org (archive.apache.org)|138.201.131.134|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35042811 (33M) [application/x-gzip]
Saving to: ‘zookeeper-3.4.10.tar.gz’

zookeeper-3.4.10.tar.gz    100%[========================================>]  33.42M  3.34MB/s    in 11s

2021-06-19 01:41:41 (3.06 MB/s) - ‘zookeeper-3.4.10.tar.gz’ saved [35042811/35042811]

2) conf/zoo_sample.cfg변경

  • zoo.cfg 으로 변경 후 시작/종료 테스트(zoo.cfg파일 변경후 나머지 서버에도 zookeeper를 복사해줘야한다.)

  • conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/opt/zookeeper-3.4.10
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=aidw-001:2888:3888
server.2=aidw-002:2888:3888
server.3=aidw-003:2888:3888
(base) root@aidw-001:/opt/zookeeper-3.4.10# bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
(base) root@aidw-001:/opt/zookeeper-3.4.10# bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
  • 환경변수 추가
export HDFS_ZKFC_USER="root"
export ZOOKEEPER_HOME="/opt/zookeeper-3.4.10"
export HDFS_JOURNALNODE_USER="root"

export PATH=$HADOOP_HOME/sbin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$SQOOP_HOME/bin:$ZOOKEEPER_HOME/bin:
  • myid 추가
    • zoo.cfg안에있는 dataDir=/opt/zookeeper-3.4.10 경로에 각 zookeeper id 명시
    • root@aidw-001:/opt/zookeeper-3.4.10# echo 1 > /opt/kafka_2.13-2.7.1/myid

3) 시작

  • bin/zkServer.sh start로 모든서버 시작
root@aidw-002:/opt/zookeeper-3.4.10# bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
root@aidw-002:/opt/zookeeper-3.4.10# netstat -anp | grep 2181
tcp6       0      0 :::2181
  • zookeeper실행 확인
(base) root@aidw-001:/opt/zookeeper-3.4.10# bin/zkCli.sh -server aidw-001:2181
Connecting to aidw-001:2181
2021-06-19 02:28:14,436 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4                       f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2021-06-19 02:28:14,439 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=aidw-001
2021-06-19 02:28:14,439 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_292
2021-06-19 02:28:14,440 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Private Build
2021-06-19 02:28:14,440 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-8-o                       penjdk-amd64/jre
2021-06-19 02:28:14,440 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper-                       3.4.10/bin/../build/classes:/opt/zookeeper-3.4.10/bin/../build/lib/*.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-log                       4j12-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.10/bin/../lib/netty-3.10.5.F                       inal.jar:/opt/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/opt/z                       ookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/opt/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.10/b                       in/../conf:
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/pac                       kages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/us                       r/lib
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=5.4.0-74-generic
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2021-06-19 02:28:14,441 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/zookeeper-3.4.10
2021-06-19 02:28:14,442 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=aidw-001                       :2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@42110406
Welcome to ZooKeeper!
2021-06-19 02:28:14,457 [myid:] - INFO  [main-SendThread(aidw-001:2181):ClientCnxn$SendThread@1032] - Opening socke                       t connection to server aidw-001/1.209.179.131:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2021-06-19 02:28:14,503 [myid:] - INFO  [main-SendThread(aidw-001:2181):ClientCnxn$SendThread@876] - Socket connect                       ion established to aidw-001/1.209.179.131:2181, initiating session
2021-06-19 02:28:14,513 [myid:] - INFO  [main-SendThread(aidw-001:2181):ClientCnxn$SendThread@1299] - Session estab                       lishment complete on server aidw-001/1.209.179.131:2181, sessionid = 0x17a2024bdf10001, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: aidw-001:2181(CONNECTED) 0] help
ZooKeeper -server host:port cmd args
        stat path [watch]
        set path data [version]
        ls path [watch]
        delquota [-n|-b] path
        ls2 path [watch]
        setAcl path acl
        setquota -n|-b val path
        history
        redo cmdno
        printwatches on|off
        delete path [version]
        sync path
        listquota path
        rmr path
        get path [watch]
        create [-s] [-e] path data acl
        addauth scheme auth
        quit
        getAcl path
        close
        connect host:port
[zk: aidw-001:2181(CONNECTED) 1]

4) 연동 테스트

  • 아래와같이 001서버에서 create한 후 002서버에서 나타나면 성공!
[zk: aidw-001:2181(CONNECTED) 1] create /zk_test my_data
Created /zk_test


[zk: aidw-002:2181(CONNECTED) 1] ls /
[zookeeper, zk_test]

5) 하둡과 연동.

  • hadoop/conf/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
    <!-- <property>
      <name>dfs.replication</name>
      <value>3</value>
    </property> -->
    <property>
      <name>dfs.namenode.name.dir</name>
      <!-- <value>file:/usr/local/hadoop/namenode</value> -->
      <value>file:/dw/hadoop/namenode</value>
    </property>
    <property>
      <name>dfs.datanode.data.dir</name>
      <!-- <value>file:/usr/local/hadoop/datanode</value> -->
      <value>file:/dw/hadoop/datanode</value>
    </property>

    <!--############################################################-->
    <property>
      <name>dfs.journalnode.edits.dir</name>
      <value>/usr/local/hadoop/journalnode</value>
    </property>
    <property>
      <name>dfs.nameservices</name>
      <value>mycluster</value>
    </property>
    <property>
      <name>dfs.ha.namenodes.mycluster</name>
      <value>nn1,nn2,nn3</value>
    </property>
    
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn1</name>
      <value>aidw-001:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn2</name>
      <value>aidw-002:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.mycluster.nn3</name>
      <value>aidw-003:8020</value>
    </property>

    <property>
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>aidw-001:9870</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>aidw-002:9870</value>
    </property>
    <property>
      <name>dfs.namenode.http-address.mycluster.nn3</name>
      <value>aidw-003:9870</value>
    </property>

    <property>
      <name>dfs.namenode.shared.edits.dir</name>
      <value>qjournal://aidw-001:8485;aidw-002:8485;aidw-003:8485/mycluster</value>
    </property>
    <property>
      <name>dfs.client.failover.proxy.provider.mycluster</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>

    <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>
    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/root/.ssh/id_rsa</value>
    </property>

    <property>
      <name>dfs.ha.automatic-failover.enabled</name>
      <value>true</value>
    </property>
    <property>
      <name>ha.zookeeper.quorum</name>
      <value>aidw-001:2181,aidw-002:2181,aidw-003:2181</value>
    </property>
    <!--############################################################-->

</configuration>
  • journalnode을 이용해 cluster들을 연결한다.
    • journalnode이 켜있지 않으면 서버끼리 connecting이 안되므로 주의!

6) 실행순서

  • 우선 zkServer를 각 zookeeper연동된 모든 서버에서 켜 준다.()
root@aidw-001:/usr/local/hadoop/logs# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

root@aidw-002:/dw/hadoop# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

root@aidw-003:/usr/local/hadoop/logs# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
  • 어느 서버가 leader인지 확인해보자
root@aidw-001:/usr/local/hadoop/logs# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: follower

root@aidw-002:/dw/hadoop# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: leader

root@aidw-003:/usr/local/hadoop/logs# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: follower
  • 하둡 실행!
root@aidw-001:/usr/local/hadoop/logs# start-all.sh
Starting namenodes on [aidw-001 aidw-002 aidw-003]
Starting datanodes
Starting journal nodes [aidw-003 aidw-002 aidw-001]
Starting ZK Failover Controllers on NN hosts [aidw-001 aidw-002 aidw-003]
Starting resourcemanager
Starting nodemanagers
  • 잘 실행되었는지 확인
root@aidw-001:/usr/local/hadoop/logs# jps
114496 QuorumPeerMain
114948 DataNode
115447 DFSZKFailoverController
115590 ResourceManager
114790 NameNode
115752 NodeManager
116123 Jps
115199 JournalNode

root@aidw-002:/dw/hadoop# jps
46147 Jps
45673 JournalNode
45818 DFSZKFailoverController
45180 QuorumPeerMain
45534 DataNode
45934 NodeManager

root@aidw-003:/usr/local/hadoop/logs# jps
31221 QuorumPeerMain
31799 DFSZKFailoverController
31655 JournalNode
32088 Jps
31915 NodeManager
31516 DataNode

Trouble shooting

오류 발생 or hdfs 파일 조회 안될때

  • Stop the Hdfs service
  • Start only the journal nodes (as they will need to be made aware of the formatting)
  • On the first namenode (as user hdfs)
    • 아래방법으로 안될시(datanode, namenode, mycluster 삭제 후 아래 다시시작해보기)
    • hadoop namenode -format
    • hdfs namenode -initializeSharedEdits -force (for the journal nodes)
    • hdfs zkfc -formatZK -force (to force zookeeper to reinitialise)
    • restart that first namenode
  • On the second namenode
    • hdfs namenode -bootstrapStandby -force (force synch with first namenode)
  • On every datanode clear the data directory
  • Restart the HDFS service
profile
Han Hanju
Data Analytics Engineer
이전 포스트

Battle of the Transformers: ELECTRA, BERT, RoBERTa, or XLNet

다음 포스트

HRNet pose estimation 논문 리뷰

0개의 댓글

관련 채용 정보