1. 도커 컨테이너 격리(생략)
2. 컨테이너 네트워크/IPTables
1. 도커 없이 네트워크 네임스페이스 환경에서 통신 구성
도커없이 컨테이너 만들기 시리즈를 스터디 전에 한번 본 적이 있었지만, "내가 보기엔 어려운 내용이네" 라며 넘어갔었는데... 이번 스터디에서 좋은 기회로 다시 볼 수 있게 되었다.
이 중에서 3편.네트워크 네임스페이스에 대한 내용을 토대로 스터디 및 과제를 하게되어 정리해보려 한다.
1.0 사전 환경구성도
✔ 이미지 출처 : 가시다님 노션
1.1 RED ↔ BLUE 네트워크 네임스페이스 간 통신
다음 내용은 대부분 명령 실습을 정리한 내용으로 부가설명은 별도로 작성하지 않았다.
결론 : 네트워크 네임스페이스 생성 시 호스트 네트워크와 구별된다
우선 ip-link 명령으로 veth(가상 이더넷 디바이스)를 생성하여 peer 해준다.
ip link add veth0 type veth peer name veth1정상적으로 생성되었는지 확인한다.
ip -c addr다음으로 RED, BLUE 네트워크 네임스페이스를 생성한다. 앞으로 실습에서 다음과 같이 네임스페이스를 생성/분리하여 사용할 예정이다.
ip netns add RED
ip netns add BLUE네트워크 네임스페이스 목록을 확인한다.
ip netns list
BLUE (id: 1)
RED (id: 0)앞서 생성한 RED 네임스페이스에 veth0을 할당한다. 그리고 네임스페이스를 확인해보면 veth1가 보이지 않는 것을 확인할 수 있다.
그 이유는 veth0 - vetn1가 peer 되어있었지만 네임스페이스가 변경되며 격리되었기 떄문에 목록에서 보이지 않는다.(맞는 설명인지 모르곘다😅)
ip link set veth0 netns RED
# 네임스페이스 목록 확인
ip netns list
BLUE
RED (id: 0)링크 목록을 확인해보면 veth1의 peer 정보가 변경된 것을 알 수 있다.
다음으로 RED 네임스페이스에 접속해서 ip -c a 명령으로 정보를 확인한다. DOWN 상태인 것을 확인할 수 있고, 호스트에서 보이지 않던 vetn1 확인된다.
ip -c link
...
14: veth0@if13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 9e:76:33:d2:a8:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
# RED 네임스페이스에서 ip 확인
ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth0@if13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 9e:76:33:d2:a8:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0다음으로 veth1 을 BLUE 네트워크 네임스페이스로 옮긴 뒤 네임스페이스에 접속해서 ip -c a 명령으로 정보를 확인한다. (DOWN 상태)
ip link set veth1 netns BLUE
# 링크 확인 -> veth0, veth1 모두 없음
ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:d2:ca:0f:a6:12 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:a5:b3:86 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:0c:37:c7:b2 brd ff:ff:ff:ff:ff:ff
# BLUE 네임스페이스에서 ip 확인
ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
13: veth1@if14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:38:af:ae:1e:3a brd ff:ff:ff:ff:ff:ff link-netns REDRED, BLUE 네임스페이스에 접속해서 veth0, veth1 상태를 활성화 한다.
# RED 네임스페이스
ip netns exec RED ip link set veth0 up
ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth0@if13: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
link/ether 9e:76:33:d2:a8:e4 brd ff:ff:ff:ff:ff:ff link-netns BLUE
# BLUE 네임스페이스
ip netns exec BLUE ip link set veth1 up
ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
13: veth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:38:af:ae:1e:3a brd ff:ff:ff:ff:ff:ff link-netns RED
inet6 fe80::38:afff:feae:1e3a/64 scope link
valid_lft forever preferred_lft foreverveth0, veth1 에 IP 설정 설정해준다. 이때, 아이피 대역은 11.11.11.x/24로 설정하였다.
# RED
ip netns exec RED ip addr add 11.11.11.2/24 dev veth0
ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 9e:76:33:d2:a8:e4 brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet 11.11.11.2/24 scope global veth0
valid_lft forever preferred_lft forever
inet6 fe80::9c76:33ff:fed2:a8e4/64 scope link
valid_lft forever preferred_lft forever
# BLUE
ip netns exec BLUE ip addr add 11.11.11.3/24 dev veth1
ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
13: veth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:38:af:ae:1e:3a brd ff:ff:ff:ff:ff:ff link-netns RED
inet 11.11.11.3/24 scope global veth1
valid_lft forever preferred_lft forever
inet6 fe80::38:afff:feae:1e3a/64 scope link
valid_lft forever preferred_lft forever이제 각 네임스페이스(RED, BLUE)에 IP 설정까지 끝났으니 서로 통신이 되는지 확인해본다.
다음 실습은 터미널 1,2,3을 나눠 진행한다.
실습1
- 터미널1 (RED 11.11.11.2)
tree /var/run/netns
/var/run/netns
├── BLUE
└── RED
0 directories, 2 files
#
nsenter --net=/var/run/netns/RED
root@docker1:~# ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 9e:76:33:d2:a8:e4 brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet 11.11.11.2/24 scope global veth0
valid_lft forever preferred_lft forever
inet6 fe80::9c76:33ff:fed2:a8e4/64 scope link
valid_lft forever preferred_lft forever
# ip neigh 옵션은?
ip -c neigh
ip -c route
filter -S
iptables -t n11.11.11.0/24 dev veth0 proto kernel scope link src 11.11.11.2
iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT- 터미널2 (호스트)
lsns -t net
NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
4026531992 net 117 1 root unassigned /sbin/init
4026532183 net 1 10943 root 0 /run/netns/RED -bash
# 생략
ip -c a
ip -c neigh
10.0.2.2 dev enp0s3 lladdr 52:54:00:12:35:02 REACHABLE
10.0.2.3 dev enp0s3 lladdr 52:54:00:12:35:03 STALE
ip -c route
default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.50.0/24 dev enp0s8 proto kernel scope link src 192.168.50.10
iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN- 터미널3 (BLUE 11.11.11.3)
ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
13: veth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:38:af:ae:1e:3a brd ff:ff:ff:ff:ff:ff link-netns RED
inet 11.11.11.3/24 scope global veth1
valid_lft forever preferred_lft forever
inet6 fe80::38:afff:feae:1e3a/64 scope link
valid_lft forever preferred_lft forever
ip -c neigh
ip -c route
11.11.11.0/24 dev veth1 proto kernel scope link src 11.11.11.3
iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT실습2. ping 통신 확인
실습 후 각 네임스페이스에서 나온 뒤 네임스페이스 삭제
# 터미널1(RED)
exit
# 터미널3(BLUE)
exit
# 터미널2(호스트)
ip netns delete RED
ip netns delete BLUE1.2 RED ← Bridge(br0) → BLUE 간 통신
📌결론 : arp table, route table, iptables 와 호스트의 bridge fdb 를 통하여 통신
이번에는 1.1에서 실습한 veth0 - veth1을 직접 peer 하는 형태가 아닌 별도 브릿지(br0)를 만어서 통신하는 실습을 해본다.
RED, BLUE 네임스페이스를 생성하고 reth0, reth1, beth0, beth1을 위 그림과 같은 구조로 설정한다.
ip netns add RED
ip link add reth0 type veth peer name reth1
ip link set reth0 netns RED
ip netns add BLUE
ip link add beth0 type veth peer name beth1
ip link set beth0 netns BLUE네임스페이스가 잘 생성되었는지 확인하고, 각 네임스페이스에서 정보를 확인한다.
ip netns list
BLUE (id: 1)
RED (id: 0)
ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:d2:ca:0f:a6:12 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:a5:b3:86 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:0c:37:c7:b2 brd ff:ff:ff:ff:ff:ff
15: reth1@if16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 5e:05:a9:4f:65:b0 brd ff:ff:ff:ff:ff:ff link-netns RED
17: beth1@if18: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 4e:12:11:e3:1d:a1 brd ff:ff:ff:ff:ff:ff link-netns BLUE
ip netns exec RED ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: reth0@if15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d2:42:c3:de:1a:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
ip netns exec BLUE ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
18: beth0@if17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:58:c1:c9:8f:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0브리지 목록을 확인해보면 기본 도커를 설치하면 생성되는 docker0 브리지만 확인이 된다.
# 브리지 정보 확인
brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.02420c37c7b2 no본 실습에서는 별도의 br0 브리지를 생성한 뒤 RED, BLUE 네임스페이스의 중간다리 역할로 사용할 예정이다.
# br0 브리지 생성
ip link add br0 type bridge
# 확인
brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.000000000000 no
brctl showmacs br0
port no mac addr is local? ageing timer
brctl showstp br0
br0
bridge id 8000.000000000000
designated root 8000.000000000000
root port 0 path cost 0
max age 20.00 bridge max age 20.00
hello time 2.00 bridge hello time 2.00
forward delay 15.00 bridge forward delay 15.00
ageing time 300.00
hello timer 0.00 tcn timer 0.00
topology change timer 0.00 gc timer 0.00
flags이제 앞서 생성한 reth1 beth1 을 br0에 연결한다.
ip link set reth1 master br0
ip link set beth1 master br0br0에 각 인터페이스가 연결된 것을 확인할 수 있다.
brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.4e1211e31da1 no beth1
reth1showmacs, link를 확인한다.(🤦♂️각 명령어 및 옵션은 좀 더 스터디를 해야할 것 같다.)
brctl showmacs br0
port no mac addr is local? ageing timer
2 4e:12:11:e3:1d:a1 yes 0.00
2 4e:12:11:e3:1d:a1 yes 0.00
1 5e:05:a9:4f:65:b0 yes 0.00
1 5e:05:a9:4f:65:b0 yes 0.00
ip -br -c link
lo UNKNOWN 00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp0s3 UP 02:d2:ca:0f:a6:12 <BROADCAST,MULTICAST,UP,LOWER_UP>
enp0s8 UP 08:00:27:a5:b3:86 <BROADCAST,MULTICAST,UP,LOWER_UP>
docker0 DOWN 02:42:0c:37:c7:b2 <NO-CARRIER,BROADCAST,MULTICAST,UP>
reth1@if16 DOWN 5e:05:a9:4f:65:b0 <BROADCAST,MULTICAST>
beth1@if18 DOWN 4e:12:11:e3:1d:a1 <BROADCAST,MULTICAST>
br0 DOWN 4e:12:11:e3:1d:a1 <BROADCAST,MULTICAST>reth0 beth0 에 IP 설정 및 활성화, br0 활성화한다.
ip netns exec RED ip addr add 11.11.11.2/24 dev reth0
ip netns exec BLUE ip addr add 11.11.11.3/24 dev beth0
ip netns exec RED ip link set reth0 up; ip link set reth1 up
ip netns exec BLUE ip link set beth0 up; ip link set beth1 up
ip link set br0 up
ip -br -c addr
lo UNKNOWN 127.0.0.1/8 ::1/128
enp0s3 UP 10.0.2.15/24 fe80::d2:caff:fe0f:a612/64
enp0s8 UP 192.168.50.10/24 fe80::a00:27ff:fea5:b386/64
docker0 DOWN 172.17.0.1/16
reth1@if16 UP fe80::5c05:a9ff:fe4f:65b0/64
beth1@if18 UP fe80::4c12:11ff:fee3:1da1/64
br0 UP fe80::4c12:11ff:fee3:1da1/64실습1. 네임스페이스별 사전 설정작업
- 터미널1 (RED 11.11.11.2)
nsenter --net=/var/run/netns/RED
# 현재 네임스페이스 확인
ip netns identify $$
RED
ip -c a;echo; ip -c route;echo; ip -c neigh
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: reth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d2:42:c3:de:1a:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.2/24 scope global reth0
valid_lft forever preferred_lft forever
inet6 fe80::d042:c3ff:fede:1abe/64 scope link
valid_lft forever preferred_lft forever
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
- 터미널2 (호스트)
brctl showmacs br0
port no mac addr is local? ageing timer
2 02:58:c1:c9:8f:8d no 65.45
2 4e:12:11:e3:1d:a1 yes 0.00
2 4e:12:11:e3:1d:a1 yes 0.00
1 5e:05:a9:4f:65:b0 yes 0.00
1 5e:05:a9:4f:65:b0 yes 0.00
1 d2:42:c3:de:1a:be no 69.55
# fdb 명령은 뭐지?
bridge fdb show
(생략)
bridge fdb show dev br0
33:33:00:00:00:01 self permanent
01:00:5e:00:00:6a self permanent
33:33:00:00:00:6a self permanent
01:00:5e:00:00:01 self permanent
33:33:ff:e3:1d:a1 self permanent
# iptalbes 확인
iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
iptables -t filter -L -n -v
Chain INPUT (policy ACCEPT 4817 packets, 8293K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 3179 packets, 240K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
- 터미널3 (BLUE 11.11.11.3)
nsenter --net=/var/run/netns/BLUE
ip netns identify $$
BLUE
ip -c a;echo; ip -c route;echo; ip -c neigh
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
18: beth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 02:58:c1:c9:8f:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.3/24 scope global beth0
valid_lft forever preferred_lft forever
inet6 fe80::58:c1ff:fec9:8f8d/64 scope link
valid_lft forever preferred_lft forever
11.11.11.0/24 dev beth0 proto kernel scope link src 11.11.11.3
실습2. ping 통신 전 사전 설정
- 터미널2 (호스트)
# iptables 정보 확인
iptables -t filter -S | grep '\-P'
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
iptables -nvL -t filter
Chain INPUT (policy ACCEPT 4912 packets, 8299K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 3241 packets, 245K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
# Ubuntu 호스트에서 패킷 라우팅 설정 확인 : 커널의 IP Forwarding (routing) 기능 확인 - 0(off), 1(on)
# echo 1 > /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward
1실습3. ping 통신 테스트
- 터미널1 (RED 11.11.11.2) >> ping 왜 실패했을까요?
ping -c 3 11.11.11.3
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
--- 11.11.11.3 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2024ms- 터미널2
# tcpdump로 확인
tcpdump -l -i br0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:10:46.766725 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 13743, seq 1, length 64
16:10:47.767935 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 13743, seq 2, length 64
16:10:48.791201 IP 11.11.11.2 > 11.11.11.3: ICMP echo request, id 13743, seq 3, length 64
# DROP수가 증가하는 것을 확인할 수 있음
watch -d 'iptables -v --numeric --table filter --list FORWARD'
# 이 옵션은 정확히 모르겠음. 별도 스터디 필요
watch -d 'iptables -v --numeric --table filter --list FORWARD;echo;iptables -v --numeric --table filter --list DOCKER-USER;echo;iptables -v --numeric --table filter --list DOCKER-ISOLATION-STAGE-1'- 터미널3
tcpdump -l -i beth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on beth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:10:51.799968 ARP, Request who-has 11.11.11.3 tell 11.11.11.2, length 28
16:10:51.799995 ARP, Reply 11.11.11.3 is-at 02:58:c1:c9:8f:8d (oui Unknown), length 28RED -> BLUE로 Ping 테스트시에 통신이 되지 않는것을 알 수 있다. 특히 이 실습에서 중요한 부분은 BLUE에서는 아무런 응답이 없지만 호스트에서는 ICMP 요청이 들어갔다는 tcpdump를 볼 수 있었고, iptables에서도 DROP된 목록을 확인할 수 있었다.
-
RED -> BLUE로 Ping 쏘는 모습
(사실... 정확히 이해는 되지 않지만 우선 마음으로 받아들이려 한다.😥)
-
호스트 터미널에서 iptables 정보에서 DROP 수가 증가하는 모습
1.3 RED/BLUE → 호스트 & 외부(인터넷) 통신
마지막 실습이다. RED/BLUE에서 호스트 & 외부로 통신할 수 있도록 설정을 해본다.
사실 이게 가장 중요한 포인트인 것 같다. 네트워크 네임스페이스간에 격리가 되어있어도 호스트 or 외부로 통신이 안된다면 아무런 의미가 없을테니깐...
결론 : 호스트에 RED/BLUE와 통신 가능한 IP 설정 및 라우팅 추가, iptables NAT 를 통하여 통신
실습0. 호스트에서 RED 나 BLUE 로 ping 통신 → RED 에서 외부로 통신구성
다음과 같은 흐름으로 통신이 된다고 한다.
실습1. 호스트 -> RED
- 터미널1
nsenter --net=/var/run/netns/RED
tcpdump -i any
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes- 터미널3 (호스트)
exit
tcpdump -i br0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes- 터미널2 (호스트) >> 호스트에서 RED 로 통신이 안되는 이유가 무엇일까요?
ping -c 1 11.11.11.2
ip -c route
default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15
10.0.2.2 dev enp0s3 proto dhcp scope link src 10.0.2.15 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.50.0/24 dev enp0s8 proto kernel scope link src 192.168.50.10
ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:d2:ca:0f:a6:12 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 76606sec preferred_lft 76606sec
inet6 fe80::d2:caff:fe0f:a612/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:a5:b3:86 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.10/24 brd 192.168.50.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fea5:b386/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:0c:37:c7:b2 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
15: reth1@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 5e:05:a9:4f:65:b0 brd ff:ff:ff:ff:ff:ff link-netns RED
inet6 fe80::5c05:a9ff:fe4f:65b0/64 scope link
valid_lft forever preferred_lft forever
17: beth1@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether 4e:12:11:e3:1d:a1 brd ff:ff:ff:ff:ff:ff link-netns BLUE
inet6 fe80::4c12:11ff:fee3:1da1/64 scope link
valid_lft forever preferred_lft forever
19: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 4e:12:11:e3:1d:a1 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4c12:11ff:fee3:1da1/64 scope link
valid_lft forever preferred_lft forever
실습2. RED -> 호스트(호스트에 br0 IP 추가)
- 터미널2 (호스트) >> br0 에 IP 추가(라우팅 정보)
# br0에 호스트 게이트웨이 추가
ip addr add 11.11.11.1/24 dev br0
ping -c 1 11.11.11.2
PING 11.11.11.2 (11.11.11.2) 56(84) bytes of data.
64 bytes from 11.11.11.2: icmp_seq=1 ttl=64 time=0.081 ms
--- 11.11.11.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.081/0.081/0.081/0.000 ms
ping -c 1 11.11.11.3
PING 11.11.11.3 (11.11.11.3) 56(84) bytes of data.
64 bytes from 11.11.11.3: icmp_seq=1 ttl=64 time=0.179 ms
--- 11.11.11.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.179/0.179/0.179/0.000 ms- 터미널1 (RED 11.11.11.2) >> 10.0.2.15(호스트) 와 통신이 안되는 이유는 무엇일까요?
ping -c 1 11.11.11.1
PING 11.11.11.1 (11.11.11.1) 56(84) bytes of data.
64 bytes from 11.11.11.1: icmp_seq=1 ttl=64 time=0.054 ms
--- 11.11.11.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.054/0.054/0.054/0.000 ms
ping -c 1 10.0.2.15
ping: connect: Network is unreachable
ip -c route
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
ip -c a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: reth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d2:42:c3:de:1a:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 11.11.11.2/24 scope global reth0
valid_lft forever preferred_lft forever
inet6 fe80::d042:c3ff:fede:1abe/64 scope link
valid_lft forever preferred_lft forever
실습3. RED -> 외부 통신
- 터미널3 (호스트)
tcpdump -i any icmp -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
16:40:56.117822 IP 11.11.11.2 > 10.0.2.15: ICMP echo request, id 15967, seq 1, length 64
16:40:56.117822 IP 11.11.11.2 > 10.0.2.15: ICMP echo request, id 15967, seq 1, length 64
16:40:56.117980 IP 10.0.2.15 > 11.11.11.2: ICMP echo reply, id 15967, seq 1, length 64
16:40:56.117986 IP 10.0.2.15 > 11.11.11.2: ICMP echo reply, id 15967, seq 1, length 64
16:41:07.550084 IP 11.11.11.2 > 8.8.8.8: ICMP echo request, id 15977, seq 1, length 64
16:41:07.550084 IP 11.11.11.2 > 8.8.8.8: ICMP echo request, id 15977, seq 1, length 64- 터미널1 (RED 11.11.11.2)
ip route add default via 11.11.11.1
# 호스트로 Ping 정상동작
ping -c 1 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.190 ms
--- 10.0.2.15 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.190/0.190/0.190/0.000 ms
# 라우팅 룰 확인
ip -c route
default via 11.11.11.1 dev reth0
11.11.11.0/24 dev reth0 proto kernel scope link src 11.11.11.2
# 외부통신 >> 외부 대역(인터넷)과 통신이 안되는 이유가 무엇일까요?
ping -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
- 터미널2 (호스트)
iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 60 packets, 6915 bytes)
pkts bytes target prot opt in out source destination
8 432 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 33 packets, 4647 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 29 packets, 2392 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 29 packets, 2392 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
## POSTROUTING : 라우팅 Outbound or 포워딩 트래픽에 의해 트리거되는 netfilter hook
## POSTROUTING 에서는 SNAT(Source NAT) 설정
iptables -t nat -A POSTROUTING -s 11.11.11.0/24 -j MASQUERADE
watch -d 'iptables -v --numeric --table nat --list POSTROUTING'
# iptables 옵션 확인 필요
iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 3 packets, 246 bytes)
pkts bytes target prot opt in out source destination
8 432 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 1 packets, 78 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE all -- * * 11.11.11.0/24 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
# conntrack 명령?
conntrack -L --src-nat
conntrack v1.4.5 (conntrack-tools): 0 flow entries have been shown.
이렇게 모든 시나리오 별 실습을 진행해보았다. 아직 중간중간에 이해되지 않는 부분들이 많이 있다. 하지만 처음 스터디 본방(?)때 가시다님이 해주셨던 설명을 들을 때는 정말 어려웠는데.. 막상 실습을 하며 따라해보니 이해가 되는 것 같다.
역시 엔지니어는 백번 듣고 보는 것 보다 손으로 해보는게...😎
사실 가장 마지막 실습이었던 1.3 RED/BLUE → 호스트 & 외부(인터넷) 통신를 할때 쯤 통신에 대한 이해가 되었고 구조가 머릿속으로 그려진 것 같다.
iptables 관련된 규칙은 정말 공부해야할 옵션들이 많은 것 같다...
CentOS7 에서 단순히 Firewalld만 사용하고 iptables를 직접 다뤄보지 않았더니 익숙치가 않아서 그런 것 같은데, netfilter와 함께 스터디를 꾸준하게 해야겠다.
본 글에서 부족한 실습 부연설명은 차차 이어서 채워놓도록 하겠다.
2주차도 한번 열심히 달려보자!🚀
DevOps를 꿈꾸는 엔지니어 입니다.
크! 장문의 포스팅 정리 내용이네요!
도커 컨테이너 격리(특히 네트워크NS)는 쿠버네티스에서도 파드 동작 이해를 위해서 정말 중요한 내용이네요.
잘 보았습니다.