[Hadoop] 7. Hadoop & Yarn 실행, 테스트

1) < master > Namenode 초기화

---

hdfs namenode -format

---

---

2) < master > Namenode 실행

---

# hdfs namenode 실행
hdfs --daemon start namenode

# node 확인 
jps # NameNode 생성확인

---

---

3) < worker1 > Standby NameNode

---

hdfs namenode -bootstrapStandby

---

---

4) < master > (Hadoop)start-dfs.sh 실행

---

• DFSZKFailoverController: master, worker1
  • master 서버에서 장애가 생기면, worker1 서버가 작동하게 된다.
• NameNode: master, worker1
• DataNode: worker1, worker2, worker3

# master에서만 실행
start-dfs.sh

# 전체 실행
jps

---

---

5) < master, worker1 > "active", "standby" NameNode 확인

---

hdfs haadmin -getServiceState namenode1 
hdfs haadmin -getServiceState namenode2

• <참고> < master, worker1 > "active", "standby" NameNode 강제 변경

# Standby -> Active
hdfs haadmin -transitionToActive namenode1 --forcemanual

# Active -> Standby
hdfs haadmin -transitionToStandby namenode2 --forcemanual

---

---

6) < master > (Yarn)start-yarn.sh 실행

---

• ResourceManager: master
• NodeManager: worker1, worker2, worker3

# master에서 실행
start-yarn.sh

# 전체 실행
jps

---

---

7) < master > JobHistory 실행

---

• JobHistoryServer: master

# master에서 실행
mapred --daemon start historyserver

# 전체 실행
jps

---

---

8) < master > Word Count 예제 테스트

---

# HDFS test 디렉토리 생성
hdfs dfs -mkdir /test
# HDFS LICENSE.txt 파일을 test 디렉토리에 삽입
hdfs dfs -put /usr/local/hadoop/LICENSE.txt /test/

# Word Count 예제 실행
yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount hdfs:///test/LICENSE.txt /test/output

# 결과 확인
hdfs dfs -text /test/output/*

---

---