hadoop ecosystem 실행 command

1. 모든서버 재부팅 후 처음 시작하는 경우

1) zk로 연동된 서버들 먼저 실행 (001, 003, 004)

• journalnode 실행
  • hdfs --daemon start journalnode
• ZooKeeper 장애 컨트롤러 (zkfc) 시작
  • zkServer.sh start
  • hdfs --daemon start zkfc (001, 004)
• Kafka 실행
  • nohup /opt/kafka_2.13-2.7.1/bin/kafka-server-start.sh config/server.properties &
• jps 실행결과

(base) root@aidw-001:/opt/kafka_2.13-2.7.1# jps
7394 DFSZKFailoverController
8868 Jps
8422 Kafka
6504 JournalNode
7305 QuorumPeerMain

2) 나머지 datanode 실행(001~010)

• master node에서 실행 (001)
  • start-all.sh
• 결과

(base) root@aidw-001:/usr/local/hadoop/etc/hadoop# start-all.sh
Starting namenodes on [aidw-001 aidw-004]
Starting datanodes
Starting journal nodes [aidw-004 aidw-003 aidw-001]
Starting ZK Failover Controllers on NN hosts [aidw-001 aidw-004]
Starting resourcemanagers on [ aidw-001 aidw-004]
Starting nodemanagers

(base) root@aidw-001:/usr/local/hadoop/etc/hadoop# jps
13584 NodeManager
12561 DataNode
13059 DFSZKFailoverController
14037 WebAppProxyServer
10806 JobHistoryServer
8422 Kafka
13399 ResourceManager
12808 JournalNode
7305 QuorumPeerMain
14077 Jps
12383 NameNode

• stanby server namenode 확인 (002)

(base) root@aidw-004:/opt/kafka_2.13-2.7.1# jps
5361 NodeManager
5235 ResourceManager
4711 NameNode
5511 Jps
4841 DataNode
4186 Kafka
2683 DFSZKFailoverController
2606 QuorumPeerMain
2543 JournalNode
(base) root@aidw-004:/opt/kafka_2.13-2.7.1#

3) job history server 실행 (001)

• mapred --daemon start

4) 하둡 UI

• 전체 개요 및 hdfs
  • http://aidw-001:9870 (active 서버)
  • http://aidw-004:9870 (stanby 서버)
• yarn resource manager
  • http://aidw-001:8088/cluster
• job history
  • http://aidw-001:19888/jobhistory