docker网络之bridge

建议阅读本文章之前了解一下文章，本文不作bridge的基本介绍

https://blog.csdn.net/u014027051/article/details/53908878/

http://williamherry.blogspot.com/2012/05/linux.html

https://tonybai.com/2016/01/15/understanding-container-networking-on-single-host/

linux bridge：

• 创建2个netns

ip netns add ns0
ip netns add ns1

• 为每个netns各添加一个网卡，类型为veth

ip link add veth0_ns0 type veth peer name veth_ns0
ip link add veth0_ns1 type veth peer name veth_ns1
ip link set veth0_ns0 netns ns0
ip link set veth0_ns1 netns ns1　　

　　查看netns下的网络，可以看到ns0和ns1分别新增接口veth0_ns0，veth0_ns1

[root@localhost home]# ip netns exec ns0 ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
12: veth0_ns0@if11: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 82:87:07:8f:59:a9 brd ff:ff:ff:ff:ff:ff link-netnsid 0

[root@localhost home]# ip netns exec ns1 ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
14: veth0_ns1@if13: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 9a:14:d8:63:56:45 brd ff:ff:ff:ff:ff:ff link-netnsid 0

host上查看接口信息，通过网卡序号可以看到veth0_ns0(12)和veth_ns0(11)为一对veth，veth0_ns1(14)和veth_ns1(13)为一对veth

[root@localhost home]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:12:5d:af brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:37:84:0b:5f brd ff:ff:ff:ff:ff:fff
11: veth_ns0@if12: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:e3:ef:a8:9c:08 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth_ns1@if14: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether a6:77:5d:48:10:81 brd ff:ff:ff:ff:ff:ff link-netnsid 3

• 为ns0和ns1的接口配置IP并UP

ip netns exec ns0 ip addr add 1.1.1.1/24 dev veth0_ns0
ip netns exec ns0 ip link set dev veth0_ns0 up

ip netns exec ns1 ip addr add 1.1.1.2/24 dev veth0_ns1 
ip netns exec ns1 ip link set dev veth0_ns1 up

查看ns0和ns1的网卡信息：

[root@localhost home]# ip netns exec ns0 ip a
: lo: <LOOPBACK> mtu  qdisc noop state DOWN group default qlen
    link/loopback ::::: brd :::::
: veth0_ns0@if11: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu  qdisc noqueue state LOWERLAYERDOWN group default qlen
    link/ether :::8f::a9 brd ff:ff:ff:ff:ff:ff link-netnsid
    inet 1.1.1.1/ scope global veth0_ns0
       valid_lft forever preferred_lft forever
[root@localhost home]# ip netns exec ns1 ip a
: lo: <LOOPBACK> mtu  qdisc noop state DOWN group default qlen
    link/loopback ::::: brd :::::
: veth0_ns1@if13: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu  qdisc noqueue state LOWERLAYERDOWN group default qlen
    link/ether 9a::d8::: brd ff:ff:ff:ff:ff:ff link-netnsid
    inet 1.1.1.2/ scope global veth0_ns1
       valid_lft forever preferred_lft forever

因为两个ns相互独立，此时ns0 ping ns1是ping不通的

[root@localhost home]# ip netns exec ns0 ping 1.1.1.2
PING 1.1.1.2 (1.1.1.2) () bytes of data.

• 创建linux bridge并添加ns0和ns1的veth pair: veth_ns0和veth_ns1

ip link add br0 type bridge
ip link set dev br0 up

ip link set dev veth_ns0 up
ip link set dev veth_ns1 up

ip link set dev veth_ns0 master br0
ip link set dev veth_ns1 master br0

查看br0信息，可以看到ns0和ns1的pair veth都已经连接到br0

[root@localhost home]# ip a show br0
15: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:77:5d:48:10:81 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::42e:2dff:fe70:43d7/64 scope link
       valid_lft forever preferred_lft forever

[root@localhost home]# ip a show master br0
11: veth_ns0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether c2:e3:ef:a8:9c:08 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::c0e3:efff:fea8:9c08/64 scope link
       valid_lft forever preferred_lft forever
13: veth_ns1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether a6:77:5d:48:10:81 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::a477:5dff:fe48:1081/64 scope link
       valid_lft forever preferred_lft forever

　　ns0 ping ns1，此时可以ping 通

[root@localhost netns]# ip netns exec ns0 ping 1.1.1.2
PING 1.1.1.2 (1.1.1.2) 56(84) bytes of data.
64 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=0.148 ms

当前组网如下

• 如果ping 外网网关(本环境为192.168.80.2)，因为br0上只连接了2个小网(ns0和ns1)，此时是无法连通外网的，将host网卡ens33添加到bridge(本操作会导致远程连接失败，请在测试环境操作)，并删除ens33的网卡地址和相关路由(原因参见https://blog.csdn.net/sld880311/article/details/77840343的“”连通性“一节)

[root@localhost home]# ip link show master br0
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:12:5d:af brd ff:ff:ff:ff:ff:ff
11: veth_ns0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether c2:e3:ef:a8:9c:08 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth_ns1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether a6:77:5d:48:10:81 brd ff:ff:ff:ff:ff:ff link-netnsid 3

可以看到br0上已经存在ens33网卡，但ns0 ping 网关仍然失败，原因是当前br0上连接了2个ns的接口，而host的接口ens33接入br0之后其IP会失效，导致网络不通

[root@localhost netns]# ip netns exec ns0 ping 192.168.80.2 -I veth0_ns0
PING 192.168.80.2 (192.168.80.2) from 1.1.1.1 veth0_ns0: () bytes of data.

　　一个简单的解决办法是给ns01添加一个与网关同网段的IP，并给br0设置与网关同网段的IP，配置如下

[root@localhost netns]# ip netns exec ns0 ip addr add 192.168.80.80/ dev veth0_ns0
[root@localhost netns]# ip netns exec ns1 ip addr add 192.168.80.81/ dev veth0_ns1
[root@localhost netns]# ip addr add 192.168.80.82/ dev br0

这样ns0和ns1都可以ping 通网关，但这样有个问题就是会导致host主机无法与外界相接，并不是一个好的解决方案

[root@localhost netns]# ip netns exec ns0 ping 192.168.80.2 -I veth0_ns0
PING 192.168.80.2 (192.168.80.2) from 192.168.80.80 veth0_ns0: () bytes of data.
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.236 ms
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.239 ms

[root@localhost netns]# ip netns exec ns1 ping 192.168.80.2 -I veth0_ns1
PING 192.168.80.2 (192.168.80.2) from 192.168.80.81 veth0_ns1: () bytes of data.
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.288 ms
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.280 ms

docker bridge：

　　docker的netns在centos下的路径为：/var/run/docker/netns，每创建一个容器就会在该路径下生成一个对应的namespace文件，使用nsenter进入该ns可以看到与容器的网络信息是一样的

首先创建bridge网络，并启动两个docker

[root@localhost home]# docker network create -d bridge --subnet 172.1.1.0/ my_br
[root@localhost home]# docker run -itd --net=my_br --name=centos0 centos /bin/sh
[root@localhost home]# docker run -itd --net=my_br --name=centos1 centos /bin/sh

查看my_br情况如下，centos0 IP为172.1.1.2，centos1 IP为172.1.1.3，为my_br的子网内地址

[root@localhost home]# docker network inspect my_br
[
    {
        "Name": "my_br",
        "Id": "f830aee4b13fa17479f850ea62d570ea61bc1c7d182a88010709a7285193bb64",
        "Created": "2018-10-17T07:23:53.31341481+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.1.1.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "03cda0f3fdd1fc65d198adb832998e11098bcc8c1bb5a8379f9c2ee82a14be07": {
                "Name": "centos1",
                "EndpointID": "d608d888da293967949340c1d946e92a6be06d525bcec611d0f20a6188de01ff",
                "MacAddress": "02:42:ac:01:01:03",
                "IPv4Address": "172.1.1.3/24",
                "IPv6Address": ""
            },
            "c739d26d51b08a36d3402e32fbe83656a7ac1b3f611a6c228f8ec80c84423439": {
                "Name": "centos0",
                "EndpointID": "9b38292d043fba31a5d04076c9d6a333c5beac08aba68dadeb84a5e17fed4dd6",
                "MacAddress": "02:42:ac:01:01:02",
                "IPv4Address": "172.1.1.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

当然容器centos0是可以直接ping 网关的

[root@localhost home]# docker exec centos0 /bin/sh -c "ping 192.168.80.2"
PING 192.168.80.2 (192.168.80.2) () bytes of data.
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.273 ms
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.642 ms

查看centos0 centos1和host网卡信息，可以看到centos0的eth0与host的veth00f659d为veth pair，centos0的eth0与host的veth05377ae为另一veth pair

[root@localhost home]# docker exec centos0 /bin/sh -c "ip link"
: lo: <LOOPBACK,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN mode DEFAULT group default qlen
    link/loopback ::::: brd :::::
: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UP mode DEFAULT group default
    link/ether ::ac::: brd ff:ff:ff:ff:ff:ff link-netnsid 0
[root@localhost home]# docker exec centos1 /bin/sh -c "ip link"
: lo: <LOOPBACK,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN mode DEFAULT group default qlen
    link/loopback ::::: brd :::::
: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UP mode DEFAULT group default
    link/ether ::ac::: brd ff:ff:ff:ff:ff:ff link-netnsid 

[root@localhost home]# ip link
: lo: <LOOPBACK,UP,LOWER_UP> mtu  qdisc noqueue state UNKNOWN mode DEFAULT group default qlen
    link/loopback ::::: brd :::::
: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP mode DEFAULT group default qlen
    link/ether :0c:::5d:af brd ff:ff:ff:ff:ff:ff
: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu  qdisc noqueue state DOWN mode DEFAULT group default qlen
    link/ether ::::2d:4c brd ff:ff:ff:ff:ff:ff
: virbr0-nic: <BROADCAST,MULTICAST> mtu  qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen
    link/ether ::::2d:4c brd ff:ff:ff:ff:ff:ff
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu  qdisc noqueue state DOWN mode DEFAULT group default
    link/ether ::::0b:5f brd ff:ff:ff:ff:ff:ff
: br-f830aee4b13f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UP mode DEFAULT group default
    link/ether ::af::4b:4e brd ff:ff:ff:ff:ff:ff
: veth00f659d@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue master br-f830aee4b13f state UP mode DEFAULT group default
    link/ether 0e:::f8:: brd ff:ff:ff:ff:ff:ff link-netnsid
: veth05377ae@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue master br-f830aee4b13f state UP mode DEFAULT group default
    link/ether aa:ae:fc:5c:dd: brd ff:ff:ff:ff:ff:ff link-netnsid 

　　查看centos0的路由，可以看到默认网关为172.1.1.1，该地址对应的网卡就是名为my_br的网桥

[root@localhost home]# docker exec centos0 /bin/bash -c "ip route"
default via 172.1.1.1 dev eth0
172.1.1.0/ dev eth0 proto kernel scope link src 172.1.1.2

[root@localhost home]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
8678329d58ab        bridge              bridge              local
e8476b504e33        host                host                local
f830aee4b13f        my_br               bridge              local
96a70c1a9516        none                null                local
[root@localhost home]# ip a|grep f830aee4b13f
: br-f830aee4b13f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc noqueue state UP group default
    inet 172.1.1.1/ scope global br-f830aee4b13f

host上的与centos0相关的路由如下，加粗的第一行为对内centos0的路由，加粗的第二行为对外路由

[root@localhost home]# ip route
default via 192.168.80.2 dev ens33 proto dhcp metric
172.1.1.0/ dev br-f830aee4b13f proto kernel scope link src 172.1.1.1 
172.17.0.0/ dev docker0 proto kernel scope link src 172.17.0.1
192.168.80.0/ dev ens33 proto kernel scope link src 192.168.80.128 metric  

同时查看与172.1.1.0相关的iptables，nat表有如下内容，即对源地址为172.1.1.0/24，出接口非网桥接口的报文进行MASQUERADE，将容器发过来的报文SNAT为host网卡地址

Chain POSTROUTING (policy ACCEPT  packets,  bytes)
 pkts bytes target     prot opt in     out     source               destination
          MASQUERADE  all  --  *      !br-f830aee4b13f  172.1.1.0/         0.0.0.0/

总结一下：centos0上ping外网网关(192.168.80.2)的处理流程：icmp报文目的地址为192.168.80.2，由于没有对应的路由，直接走默认路由，报文从容器的eth0发出去，进入到默认网关my_br(172.1.1.1)，网桥my_br根据host的路由将目的地址为192.168.80.2的报文发送到ens33，同时将源地址使用MASQUERADE SNAT为ens33的地址。这就是docker bridge的报文转发流程。

改造自定义的bridge

　　第一部分的网络方案是有缺陷的，它使得host主机的一个网口失效，根据对docker网络的分析，改造其网络如下：

• 首先创建veth pair，其中一个连接到br0

ip link add veth0 type veth peer name veth1
ip link set dev veth0 up
ip link set dev veth1 up
ip link set dev veth1 master br0

• host上添加到达1.1.1.0/24网络的路由，其中1.1.1.3为br0的地址

[root@localhost home]# ip route add 1.1.1.0/ via 1.1.1.3

• ns0内部添加默认网关路由（注意必须是网关路由）

ip netns exec ns0 ip route add default via 1.1.1.3 dev veth0_ns0

• host主机添加对1.1.1.0/24网段的SNAT

iptables -t nat -A POSTROUTING -s 1.1.1.0/ ! -o br0 -j MASQUERADE

这样就构造了一个模仿docker bridge的网络，ns0就可以ping通外部网关了

[root@localhost home]# ip netns exec ns0 ip route
default via 1.1.1.3 dev veth0_ns0
1.1.1.0/ dev veth0_ns0 proto kernel scope link src 1.1.1.1 

[root@localhost home]# ip netns exec ns0 ping 192.168.80.2
PING 192.168.80.2 (192.168.80.2) () bytes of data.
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.439 ms
 bytes from 192.168.80.2: icmp_seq= ttl= time=0.533 ms

那么使用MASQUERADE时，iptables是怎么判断不同ping的返回包呢？此时host和ns0的icmp返回包有相同的地址和协议，且没有端口号。答案是通过 ip_conntrack识别icmp报文中的id字段来判断不同的ping进程，参见ICMP connections，ip_conntrack是实现NAT的基础。(新版本内核使用nf_conntrack)

TIPS：

• 可以使用iptstate查看/proc/net/nf_conntrack的连接状态，如下

参考：

https://blog.csdn.net/sld880311/article/details/77840343

https://docs.docker.com/network/bridge/#differences-between-user-defined-bridges-and-the-default-bridge

http://success.docker.com/article/networking#dockerbridgenetworkdriver

http://vinllen.com/linux-bridgeshe-ji-yu-shi-xian/