datanode를 확장하는 방법
java version
-
hadoop3.3.0 ecosystem은 현재 기본적으로 java8기준으로 설치하였지만 spark3.0.2에서 java11을 요구 하므로, java8을 기준으로 version을 맞추되 java11도 설치 해 주어야한다.
-
OpenJDK 설치
sudo apt update
sudo apt install openjdk-8-jdk
sudo apt install openjdk-11-jdk- jaba version을 변경 해 준다.
update-alternatives --config javaSSH 설정
- namenode가 설치될곳에 각각 ssh공개키를 만들어준다.
(base) root@aidw-001:~/.ssh# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:y1tbQQ19ZPlfiTjy5f4qJTSYaHDjHU4OlaoeFXa5fco root@aidw-004
The key's randomart image is:
+---[RSA 3072]----+
| ..o .. .+|
| . * * o.o.|
| = % * o o.o|
| * O B + .o|
| + S = O o|
| o . . E + .|
| . . o . = |
| . o + . |
| . . ..o. |
+----[SHA256]-----+
(base) root@aidw-001:~/.ssh# ll
합계 28K
drwx------ 3 root root 4.0K 6월 22 14:18 ./
drwx------ 44 root root 4.0K 6월 22 14:09 ../
drwxr-xr-x 2 root root 4.0K 6월 22 14:18 back/
-rw------- 1 root root 2.6K 6월 22 14:18 id_rsa
-rw-r--r-- 1 root root 567 6월 22 14:18 id_rsa.pub
-rw-r--r-- 1 root root 4.4K 5월 7 15:57 known_hosts- ssh로 연결할 서버에 공개키를 복사해준다.
- ssh-copy-id 명령어는 대상 서버의 .ssh 디렉터리 안에 있는 authorized_keys 파일에 공개키를 입력하며, 다음과 같은 형식으로 실행한다.
root@aidw-001:~/.ssh# ssh-copy-id -i ~/.ssh/id_rsa.pub root@aidw-004
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@aidw-004's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'root@aidw-004'"
and check to make sure that only the key(s) you wanted were added.
root@aidw-001:~/.ssh# ssh aidw-004
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-74-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
94 updates can be applied immediately.
12 of these updates are standard security updates.
추가 업데이트를 확인하려면 apt list --upgradable 을 실행하세요.
Last login: Tue Jun 22 14:09:22 2021 from 1.209.179.131
(base) root@aidw-004:~#- aidw-004에 authorized_keys가 공개키정보로 포함된걸 확인 할 수 있다.
drwx------ 3 root root 4.0K 6월 22 14:23 ./
drwx------ 44 root root 4.0K 6월 22 14:09 ../
-rw------- 1 root root 567 6월 22 14:23 authorized_keys
drwxr-xr-x 2 root root 4.0K 6월 22 14:18 back/
-rw------- 1 root root 2.6K 6월 22 14:18 id_rsa
-rw-r--r-- 1 root root 567 6월 22 14:18 id_rsa.pub
-rw-r--r-- 1 root root 4.4K 5월 7 15:57 known_hosts
(base) root@aidw-004:~/.ssh#hadoop
- hadoop의 workers, spark의 slaves파일에 추가할 노드를 삽입해준다.
aidw-001
aidw-002
aidw-003
aidw-004
aidw-005
aidw-006
aidw-007
aidw-009
aidw-010- master에 있는 hadoop폴더 이동
scp -r /usr/local/hadoop root@aidw-010:/usr/local- master에 있는 spark폴더 이동
scp -r /opt/spark-3.0.2/ root@aidw-010:/opt/- hadoop 및 spark 재시작!
root@aidw-001:/opt/spark-3.0.2/conf# stop-all.sh
Stopping namenodes on [aidw-001 aidw-002 aidw-003]
Stopping datanodes
Stopping journal nodes [aidw-003 aidw-002 aidw-001]
Stopping ZK Failover Controllers on NN hosts [aidw-001 aidw-002 aidw-003]
Stopping nodemanagers
Stopping resourcemanager
root@aidw-001:/opt/spark-3.0.2/sbin# ./stop-all.sh
root@aidw-001:/opt/spark-3.0.2/sbin# jps
114496 QuorumPeerMain
159079 Jps
root@aidw-001:/opt/spark-3.0.2/sbin#
root@aidw-001:/opt/spark-3.0.2/sbin# start-all.sh
Starting namenodes on [aidw-001 aidw-002 aidw-003]
Starting datanodes
Starting journal nodes [aidw-003 aidw-002 aidw-001]
Starting ZK Failover Controllers on NN hosts [aidw-001 aidw-002 aidw-003]
Starting resourcemanager
Starting nodemanagers
root@aidw-001:/opt/spark-3.0.2/sbin# jps
114496 QuorumPeerMain
160112 ResourceManager
159297 NameNode
160289 NodeManager
159718 JournalNode
160631 Jps
159471 DataNode
159965 DFSZKFailoverController
root@aidw-001:/opt/spark-3.0.2/sbin# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.master.Master-1-aidw-001.out
aidw-007: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-007.out
aidw-002: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-002.out
aidw-001: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-001.out
aidw-004: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-004.out
aidw-010: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-010.out
aidw-006: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-006.out
aidw-003: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-003.out
aidw-005: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-005.out
aidw-009: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-3.0.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-aidw-009.out
root@aidw-001:/opt/spark-3.0.2/sbin# jps
114496 QuorumPeerMain
160112 ResourceManager
159297 NameNode
160289 NodeManager
159718 JournalNode
160676 Master
160917 Jps
159471 DataNode
160847 Worker
159965 DFSZKFailoverController
root@aidw-001:/opt/spark-3.0.2/sbin#
Data Analytics Engineer