DOCKER学习_005:Flannel网络配置
一 简介
- Flannel是一种基于overlay网络的跨主机容器网络解决方案，也就是将TCP数据包封装在另一种网络包里面进行路由转发和通信，
- Flannel是CoreOS开发,专门用于docker多机互联的一个工具,让集群中的不同节点主机创建的容器都具有全集群唯一的虚拟ip地址
- Flannel使用go语言编写
二 Flannel实现原理
2.1原理说明
- Flannel为每个host分配一个subnet，容器从这个subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端口映射即可实现跨主机通信
- 每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，其职责就是从池子中分配subnet
- Flannel使用etcd存放网络配置、已分配 的subnet、host的IP等信息
- Flannel数据包在主机间转发是由backend实现的，目前已经支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多种backend
2.2 Flannel网络结构图
多了一层封装,进行流量劫持
2.3 数据转发流程
- 容器直接使用目标容器的ip访问，默认通过容器内部的eth0发送出去。
- 报文通过veth pair被发送到vethXXX。
- vethXXX是直接连接到虚拟交换机docker0的，报文通过虚拟bridge docker0发送出去。
- 查找路由表，外部容器ip的报文都会转发到flannel0虚拟网卡，这是一个P2P的虚拟网卡，然后报文就被转发到监听在另一端的flanneld。
- flanneld通过etcd维护了各个节点之间的路由表，把原来的报文UDP封装一层，通过配置的iface发送出去。
- 报文通过主机之间的网络找到目标主机。
- 报文继续往上，到传输层，交给监听在8285端口的flanneld程序处理。
- 数据被解包，然后发送给flannel0虚拟网卡。
- 查找路由表，发现对应容器的报文要交给docker0。
- docker0找到连到自己的容器，把报文发送过去。
2.4 实现原理
- Flannel为每个host分配一个 subnet，容器从这个 subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端□映射即可实现跨主机通信
- 每个 subnet都是从一个更大的IP池中划分的，f1 annel会在每个主机上运行—个叫uf1 annelid的 agent，其职责就是从池子中分配 subnet
- Flannel使用etcd存放网络配置、已分配的 subnet、host的IP等信息
- Flannel数据包在主机间转发是由 backend实现的，目前已经攴持UDP、 VXLAN、host-gw AWS VPC和GCE路由等多种 backend
三 Flannel安装配置
3.1 环境准备
| 节点名称 | IP地址 | 安装软件 |
| docker-server1 | 192.168.132.131 | etcd、flannel、docker |
| docker-server2 | 192.168.132.132 | flannel、docker |
删掉节点的所有容器
[root@docker-server1 ~]# docker ps -aq|xargs docker rm
[root@docker-server2 ~]# docker ps -aq|xargs docker rm
3.2 安装etcd
etcd下载地址：https://github.com/coreos/etcd/releases
下载
[root@docker-server1 ~]# wget https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz
[root@docker-server2 ~]# wget https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz
在131上安装etcd和flannel
[root@docker-server1 ~]# tar -xf etcd-v3.3.9-linux-amd64.tar.gz
[root@docker-server1 ~]# cd etcd-v3.3.9-linux-amd64
[root@docker-server1 etcd-v3.3.9-linux-amd64]# ll
total
drwxr-xr-x joy joy Jul Documentation
-rwxr-xr-x joy joy Jul etcd
-rwxr-xr-x joy joy Jul etcdctl
-rw-r--r-- joy joy Jul README-etcdctl.md
-rw-r--r-- joy joy Jul README.md
-rw-r--r-- joy joy Jul READMEv2-etcdctl.md
[root@docker-server1 etcd-v3.3.9-linux-amd64]# cp etcd* /usr/bin/
启动命令：
[root@docker-server1 ~]# etcd -name etcd-131 -data-dir /var/lib/etcd --advertise-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379 --listen-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379
-name:etc取名
-data-dir:定义数据路径
-- ::36.719599 I | etcdmain: Git SHA: fca8add78
-- ::36.719603 I | etcdmain: Go Version: go1.10.3
-- ::36.719606 I | etcdmain: Go OS/Arch: linux/amd64
-- ::36.719613 I | etcdmain: setting maximum number of CPUs to , total number of available CPUs is
-- ::36.720048 I | embed: listening for peers on http://localhost:2380
-- ::36.720124 I | embed: listening for client requests on 127.0.0.1:
-- ::36.720146 I | embed: listening for client requests on 192.168.132.131:
-- ::36.722745 I | etcdserver: name = etcd-
-- ::36.722778 I | etcdserver: data dir = /var/lib/etcd
-- ::36.722783 I | etcdserver: member dir = /var/lib/etcd/member
-- ::36.722787 I | etcdserver: heartbeat = 100ms
-- ::36.722791 I | etcdserver: election = 1000ms
-- ::36.722794 I | etcdserver: snapshot count =
-- ::36.722811 I | etcdserver: advertise client URLs = http://127.0.0.1:2379,http://192.168.132.131:2379
-- ::36.722816 I | etcdserver: initial advertise peer URLs = http://localhost:2380
-- ::36.722823 I | etcdserver: initial cluster = etcd-=http://localhost:2380
-- ::36.725597 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
-- ::36.725645 I | raft: 8e9e05c52164694d became follower at term
-- ::36.725658 I | raft: newRaft 8e9e05c52164694d [peers: [], term: , commit: , applied: , lastindex: , lastterm: ]
-- ::36.725663 I | raft: 8e9e05c52164694d became follower at term
-- ::36.731392 W | auth: simple token is not cryptographically signed
-- ::36.732944 I | etcdserver: starting server... [version: 3.3., cluster version: to_be_decided]
-- ::36.733497 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding ticks (election ticks )
-- ::36.734281 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
-- ::37.635489 I | raft: 8e9e05c52164694d is starting a new election at term
-- ::37.635568 I | raft: 8e9e05c52164694d became candidate at term
-- ::37.635621 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term
-- ::37.635656 I | raft: 8e9e05c52164694d became leader at term
-- ::37.635676 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term
-- ::37.636216 I | etcdserver: setting up the initial cluster version to 3.3
-- ::37.637689 N | etcdserver/membership: set the initial cluster version to 3.3
-- ::37.637846 I | etcdserver/api: enabled capabilities for version 3.3
-- ::37.637990 I | etcdserver: published {Name:etcd- ClientURLs:[http://127.0.0.1:2379 http://192.168.132.131:2379]} to cluster cdf818194e3a8c32
-- ::37.638099 I | embed: ready to serve client requests
-- ::37.638861 I | embed: ready to serve client requests
-- ::37.639056 E | etcdmain: forgot to set Type=notify in systemd service file?
-- ::37.640375 N | embed: serving insecure client requests on 127.0.0.1:, this is strongly discouraged!
-- ::37.640424 N | embed: serving insecure client requests on 192.168.132.131:, this is strongly discouraged!
[root@docker-server1 ~]# ps -ef|grep etcd
root : pts/ :: etcd -name etcd- -data-dir /var/lib/etcd --advertise-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379 --listen-client-urls http://192.168.132.131:2379,http://127.0.0.1:2379
etcdctl 是一个客户端连接工具
[root@docker-server1 ~]# etcdctl member list
8e9e05c52164694d: name=etcd- peerURLs=http://localhost:2380 clientURLs=http://127.0.0.1:2379,http://192.168.132.131:2379 isLeader=true
使用etcdctl连接ectd数据库,检查etcd的连通性
[root@docker-server1 ~]# etcdctl --endpoints http://127.0.0.1:2379 member list
8e9e05c52164694d: name=etcd- peerURLs=http://localhost:2380 clientURLs=http://127.0.0.1:2379,http://192.168.132.131:2379 isLeader=true
查看etcdctl版本
[root@docker-server1 ~]# etcdctl --version
etcdctl version: 3.3.
API version:
[root@docker-server1 ~]# etcdctl --help
NAME:
etcdctl - A simple command line client for etcd. WARNING:
Environment variable ETCDCTL_API is not set; defaults to etcdctl v2.
Set environment variable ETCDCTL_API= to use v3 API or ETCDCTL_API= to use v2 API. USAGE:
etcdctl [global options] command [command options] [arguments...] VERSION:
3.3. COMMANDS:
backup backup an etcd directory
cluster-health check the health of the etcd cluster
mk make a new key with a given value
mkdir make a new directory
rm remove a key or a directory
rmdir removes the key if it is an empty directory or a key-value pair
get retrieve the value of a key
ls retrieve a directory
set set the value of a key
setdir create a new directory or update an existing directory TTL
update update an existing key with a given value
updatedir update an existing directory
watch watch a key for changes
exec-watch watch a key for changes and exec an executable
member member add, remove and list subcommands
user user add, grant and revoke subcommands
role role add, grant and revoke subcommands
auth overall auth controls
help, h Shows a list of commands or help for one command GLOBAL OPTIONS:
--debug output cURL commands which can be used to reproduce the request
--no-sync don't synchronize cluster information before sending request
--output simple, -o simple output response in the given format (simple, `extended` or `json`) (default: "simple")
--discovery-srv value, -D value domain name to query for SRV records describing cluster endpoints
--insecure-discovery accept insecure SRV records describing cluster endpoints
--peers value, -C value DEPRECATED - "--endpoints" should be used instead
--endpoint value DEPRECATED - "--endpoints" should be used instead
--endpoints value a comma-delimited list of machine addresses in the cluster (default: "http://127.0.0.1:2379,http://127.0.0.1:4001")
--cert-file value identify HTTPS client using this SSL certificate file
--key-file value identify HTTPS client using this SSL key file
--ca-file value verify certificates of HTTPS-enabled servers using this CA bundle
--username value, -u value provide username[:password] and prompt if password is not supplied.
--timeout value connection timeout per request (default: 2s)
--total-timeout value timeout for the command execution (except watch) (default: 5s)
--help, -h show help
--version, -v print the version
更新etxdctl版本
[root@docker-server1 ~]# export ETCDCTL_API=3
[root@docker-server1 ~]# etcdctl --help
指令已经发生变化
[root@docker-server1 ~]# etcdctl --help
NAME:
etcdctl - A simple command line client for etcd3. USAGE:
etcdctl VERSION:
3.3. API VERSION:
3.3 COMMANDS:
get Gets the key or a range of keys
put Puts the given key into the store
del Removes the specified key or range of keys [key, range_end)
txn Txn processes all the requests in one transaction
compaction Compacts the event history in etcd
alarm disarm Disarms all alarms
alarm list Lists all alarms
defrag Defragments the storage of the etcd members with given endpoints
endpoint health Checks the healthiness of endpoints specified in `--endpoints` flag
endpoint status Prints out the status of endpoints specified in `--endpoints` flag
endpoint hashkv Prints the KV history hash for each endpoint in --endpoints
move-leader Transfers leadership to another etcd cluster member.
watch Watches events stream on keys or prefixes
version Prints the version of etcdctl
lease grant Creates leases
lease revoke Revokes leases
lease timetolive Get lease information
lease list List all active leases
lease keep-alive Keeps leases alive (renew)
member add Adds a member into the cluster
member remove Removes a member from the cluster
member update Updates a member in the cluster
member list Lists all members in the cluster
snapshot save Stores an etcd node backend snapshot to a given file
snapshot restore Restores an etcd member snapshot to an etcd directory
snapshot status Gets backend snapshot status of a given file
make-mirror Makes a mirror at the destination etcd cluster
migrate Migrates keys in a v2 store to a mvcc store
lock Acquires a named lock
elect Observes and participates in leader election
auth enable Enables authentication
auth disable Disables authentication
user add Adds a new user
user delete Deletes a user
user get Gets detailed information of a user
user list Lists all users
user passwd Changes password of user
user grant-role Grants a role to a user
user revoke-role Revokes a role from a user
role add Adds a new role
role delete Deletes a role
role get Gets detailed information of a role
role list Lists all roles
role grant-permission Grants a key to a role
role revoke-permission Revokes a key from a role
check perf Check the performance of the etcd cluster
help Help about any command OPTIONS:
--cacert="" verify certificates of TLS-enabled secure servers using this CA bundle
--cert="" identify secure client using this TLS certificate file
--command-timeout=5s timeout for short running command (excluding dial timeout)
--debug[=false] enable client-side debug logging
--dial-timeout=2s dial timeout for client connections
-d, --discovery-srv="" domain name to query for SRV records describing cluster endpoints
--endpoints=[127.0.0.1:] gRPC endpoints
--hex[=false] print byte strings as hex encoded strings
--insecure-discovery[=true] accept insecure SRV records describing cluster endpoints
--insecure-skip-tls-verify[=false] skip server certificate verification
--insecure-transport[=true] disable transport security for client connections
--keepalive-time=2s keepalive time for client connections
--keepalive-timeout=6s keepalive timeout for client connections
--key="" identify secure client using this TLS key file
--user="" username[:password] for authentication (prompt if password is not supplied)
-w, --write-out="simple" set the output format (fields, json, protobuf, simple, table)
3.3 安装Flannel
flannel下载地址：https://github.com/coreos/flannel/releases
下载
[root@docker-server1 ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@docker-server2 ~]# wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[root@docker-server1 ~]# tar -xf flannel-v0.11.0-linux-amd64.tar.gz
[root@docker-server1 ~]# cp flanneld /usr/bin/
[root@docker-server1 ~]# cp mk-docker-opts.sh /usr/bin/
2版本etcdctl使用
添加flannel网络配置信息到etcd：是一个键值对
[root@docker-server1 ~]# etcdctl set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'
如果不是本机可以加参数：--endpoints http://IP:2379
{"Network": "10.0.0.0/16", "SubnetLen": , "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
Network：用于指定Flannel地址池
SubnetLen：用于指定分配给单个宿主机的docker0的ip段的子网掩码的长度
SubnetMin：用于指定最小能够分配的ip段
SudbnetMax：用于指定最大能够分配的ip段，在上面的示例中，表示每个宿主机可以分配一个24位掩码长度的子网，可以分配的子网从10.0.1.0/24到10.0.20.0/24，也就意味着在这个网段中，最多只能有20台宿主机
Backend：用于指定数据包以什么方式转发，默认为udp模式，host-gw模式性能最好，但不能跨宿主机网络
[root@docker-server1 ~]# etcdctl get /coreos.com/network/config
{"Network": "10.0.0.0/16", "SubnetLen": , "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
上面的操作时,时etcdctl的API版本是2可以使用,当升级到3版本后会报错
[root@docker-server1 ~]# etcdctl set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'
Error: unknown command "set" for "etcdctl" Did you mean this?
get
put
del
user Run 'etcdctl --help' for usage.
Error: unknown command "set" for "etcdctl" Did you mean this?
get
put
del
user
3版本用法
[root@docker-server1 ~]# etcdctl --endpoints http://127.0.0.1:2379 put /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}'
OK
[root@docker-server1 ~]# etcdctl get /coreos.com/network/config
/coreos.com/network/config
{"Network": "10.0.0.0/16", "SubnetLen": , "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
下面的操作都是使用的etcdctl v2版本
3.4 启动Flannel
[root@docker-server1 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.131 --etcd-prefix=/coreos.com/network &
[root@docker-server1 ~]# I1109 ::01.696813 main.go:] Searching for interface using 192.168.132.131
I1109 ::01.698311 main.go:] Using interface with name ens33 and address 192.168.132.131
I1109 ::01.698335 main.go:] Defaulting external address to interface address (192.168.132.131)
I1109 ::01.698556 main.go:] Created subnet manager: Etcd Local Manager with Previous Subnet: 10.0.4.0/
I1109 ::01.698571 main.go:] Installing signal handlers
I1109 ::01.700808 main.go:] Found network config - Backend type: vxlan
I1109 ::01.700880 vxlan.go:] VXLAN config: VNI= Port= GBP=false DirectRouting=false
I1109 ::01.703889 local_manager.go:] Found lease (10.0.4.0/) for current IP (192.168.132.131), reusing
I1109 ::01.704954 main.go:] Wrote subnet file to /run/flannel/subnet.env
I1109 ::01.704967 main.go:] Running backend.
I1109 ::01.705791 vxlan_network.go:] watching for new subnet leases
I1109 ::01.709710 main.go:] Waiting for 22h59m59.994619649s to renew lease
[root@docker-server1 ~]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP group default qlen
link/ether :0c:::dd: brd ff:ff:ff:ff:ff:ff
inet 192.168.132.131/ brd 192.168.132.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::bcf9:af19:a325:e2c7/ scope link noprefixroute
valid_lft forever preferred_lft forever
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::3e:dd:: brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/ brd 192.168.0.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80:::3eff:fedd:/ scope link
valid_lft forever preferred_lft forever
: br-b1c2d9c1e522: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::cd:4a::4f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/ brd 172.17.255.255 scope global br-b1c2d9c1e522
valid_lft forever preferred_lft forever
inet6 fe80:::cdff:fe4a:254f/ scope link
valid_lft forever preferred_lft forever
: br-ec4a8380b2d3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::c9:ba::ae brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/ brd 172.18.255.255 scope global br-ec4a8380b2d3
valid_lft forever preferred_lft forever
: br-f42e46889a2a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::b4:b6::7f brd ff:ff:ff:ff:ff:ff
inet 172.22.16.1/ brd 172.22.16.255 scope global br-f42e46889a2a
valid_lft forever preferred_lft forever
inet6 fe80:::b4ff:feb6:357f/ scope link
valid_lft forever preferred_lft forever
: flannel.: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default
link/ether 8e:b1:::b0: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.0/ scope global flannel.
valid_lft forever preferred_lft forever
inet6 fe80::8cb1:37ff:fe26:b059/ scope link
[root@docker-server1 ~]# ps -ef |grep flan
root : pts/ :: /usr/bin/flanneld --etcd-endpoints=http://192.168.132.131:2379 --iface=192.168.132.131 --etcd-prefix=/coreos.com/network
可以使用flannel提供的脚本将subnet.env转写成Docker启动参数，创建好的启动参数默认生成在/run/docker_opts.env文件中：
[root@docker-server1 ~]# mk-docker-opts.sh
[root@docker-server1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.0.0.0/
FLANNEL_SUBNET=10.0.8.1/
FLANNEL_MTU=
FLANNEL_IPMASQ=false
[root@docker-server1 ~]# cat /run/docker_opts.env
DOCKER_OPT_BIP="--bip=10.0.4.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_OPTS=" --bip=10.0.4.1/24 --ip-masq=true --mtu=1450"
需要把这个文件加到docker的启动项
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS -H fd:// --containerd=/run/containerd/containerd.sock
启动docker
[root@docker-server1 ~]# systemctl daemon-reload
[root@docker-server1 ~]# systemctl restart docker
报错
[root@docker-server1 ~]# journalctl -xe -u docker
Nov :: docker-server1 dockerd[]: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following
Nov :: docker-server1 systemd[]: docker.service: main process exited, code=exited, status=/FAILURE
删除/etc/docker/daemon.json里的bip配置
[root@docker-server1 ~]# systemctl restart docker
To force a start use "systemctl reset-failed docker.service" followed by "systemctl start docker.service" again.
[root@docker-server1 ~]# systemctl restart docker
[root@docker-server1 ~]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP group default qlen
link/ether :0c:::dd: brd ff:ff:ff:ff:ff:ff
inet 192.168.132.131/ brd 192.168.132.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::bcf9:af19:a325:e2c7/ scope link noprefixroute
valid_lft forever preferred_lft forever
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::3e:dd:: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.1/ brd 10.0.4.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80:::3eff:fedd:/ scope link
valid_lft forever preferred_lft forever
: br-b1c2d9c1e522: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::cd:4a::4f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/ brd 172.17.255.255 scope global br-b1c2d9c1e522
valid_lft forever preferred_lft forever
inet6 fe80:::cdff:fe4a:254f/ scope link
valid_lft forever preferred_lft forever
: br-ec4a8380b2d3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::c9:ba::ae brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/ brd 172.18.255.255 scope global br-ec4a8380b2d3
valid_lft forever preferred_lft forever
: br-f42e46889a2a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::b4:b6::7f brd ff:ff:ff:ff:ff:ff
inet 172.22.16.1/ brd 172.22.16.255 scope global br-f42e46889a2a
valid_lft forever preferred_lft forever
inet6 fe80:::b4ff:feb6:357f/ scope link
valid_lft forever preferred_lft forever
: flannel.: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default
link/ether 8e:b1:::b0: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.0/ scope global flannel.
valid_lft forever preferred_lft forever
inet6 fe80::8cb1:37ff:fe26:b059/ scope link
valid_lft forever preferred_lft forever
docker0的IP是10.0.4.1
[root@docker-server1 ~]# etcdctl ls /coreos.com/network/
/coreos.com/network/subnets
/coreos.com/network/config
可以看到flannel0网卡的地址和etcd存储的地址一样，这样flannel网络配置完成
Flannel启动过程解析：
- 从etcd中获取network的配置信息
- 划分subnet，并在etcd中进行注册
- 将子网信息记录到/run/flannel/subnet.env中
- Flannel必须先于Docker启动
3.5 验证Flannel网络
查看etcd中的数据：
[root@docker-server1 ~]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.4.0-
3.6 配置Docker
Docker安装完成以后，需要修改其启动参数以使其能够使用flannel进行IP分配，以及网络通讯
在Flannel运行之后，会生成一个环境变量文件，包含了当前主机要使用flannel通讯的相关参数，如下：
[root@docker-server1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.0.0.0/
FLANNEL_SUBNET=10.0.4.1/
FLANNEL_MTU=
FLANNEL_IPMASQ=false
docker-server1相同操作配置flannel，不用安装etcd
[root@docker-server2 ~]# tar -xf flannel-v0.11.0-linux-amd64.tar.gz
[root@docker-server2 ~]# mv flanneld /usr/bin/
[root@docker-server2 ~]# mv mk-docker-opts.sh /usr/bin/
[root@docker-server2 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.132 --etcd-prefix=/coreos.com/network &
[root@docker-server2 ~]# mk-docker-opts.sh -c
[root@docker-server2 ~]# cat /run/docker_opts.env
修改docker启动文件，删除daemon.json的bip配置
[root@docker-server2 ~]# systemctl daemon-reload
[root@docker-server2 ~]# systemctl restart docker
[root@docker-server2 ~]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP group default qlen
link/ether :0c:::fd: brd ff:ff:ff:ff:ff:ff
inet 192.168.132.132/ brd 192.168.132.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::6a92:62ba:1b33:c93d/ scope link noprefixroute
valid_lft forever preferred_lft forever
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::::ff: brd ff:ff:ff:ff:ff:ff
inet 10.0.18.1/ brd 10.0.18.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80:::50ff:fe67:ff90/ scope link
valid_lft forever preferred_lft forever
: flannel.: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default
link/ether :b9:::bf: brd ff:ff:ff:ff:ff:ff
inet 10.0.18.0/ scope global flannel.
valid_lft forever preferred_lft forever
inet6 fe80::60b9:76ff:fe44:bf32/ scope link
valid_lft forever preferred_lft forever
[root@docker-server1 ~]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.8.0-
/coreos.com/network/subnets/10.0.18.0-
两台主机的subnet网段
[root@docker-server1 ~]# etcdctl get /coreos.com/network/subnets/10.0.18.0-24
{"PublicIP":"192.168.132.132","BackendType":"vxlan","BackendData":{"VtepMAC":"62:b9:76:44:bf:32"}}
[root@docker-server1 ~]# etcdctl get /coreos.com/network/subnets/10.0.8.0-24
{"PublicIP":"192.168.132.131","BackendType":"vxlan","BackendData":{"VtepMAC":"62:93:c1:0f:b9:b0"}}
3.7 验证容器互通
[root@docker-server1 ~]# docker run -it busybox
/ # ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
: eth0@if125: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu qdisc noqueue
link/ether ::0a::: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.2/ brd 10.0.4.255 scope global eth0
valid_lft forever preferred_lft forever
[root@docker-server2 ~]# docker run -it busybox
/ # ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu qdisc noqueue
link/ether ::0a::: brd ff:ff:ff:ff:ff:ff
inet 10.0.18.2/ brd 10.0.18.255 scope global eth0
valid_lft forever preferred_lft forever
互ping
192.168..131容器
/ # ping 10.0.18.2
PING 10.0.18.2 (10.0.18.2): data bytes
bytes from 10.0.18.2: seq= ttl= time=2.517 ms
bytes from 10.0.18.2: seq= ttl= time=1.058 ms
bytes from 10.0.18.2: seq= ttl= time=1.676 ms
192.168..132容器
PING 10.0.4.2 (10.0.4.2): data bytes
bytes from 10.0.4.2: seq= ttl= time=2.606 ms
bytes from 10.0.4.2: seq= ttl= time=1.727 ms
两主机的容器可以互通
此时的网络数据包流向如图：
3.8 配置backend为host-gw
host-gw bakcend是flannel的另一个backend。与vxlan不同，host-gw不会封装数据包，而是在主机的路由表中创建到其他主机的subnet的路由条目，从而实现容器网络跨主机通信。需要说明的是，host-gw不能跨宿主机网络通信，或者说跨宿主机网络通信需要物理路由支持。性能最好
修改etcd如下：
[root@docker-server1 ~]# etcdctl --endpoints http://127.0.0.1:2379 set /coreos.com/network/config '{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "host-gw"}}'
{"Network": "10.0.0.0/16", "SubnetLen": , "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "host-gw"}
重启flanneld与docker：
root@docker-server1 ~]# kill -9 86780
[root@docker-server1 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.131 --etcd-prefix=/coreos.com/network &
[root@docker-server1 ~]# systemctl restart docker
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::3e:dd:: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.1/ brd 10.0.4.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80:::3eff:fedd:/ scope link
valid_lft forever preferred_lft forever
: flannel.: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default
link/ether 8e:b1:::b0: brd ff:ff:ff:ff:ff:ff
inet 10.0.4.0/ scope global flannel.
valid_lft forever preferred_lft forever
inet6 fe80::8cb1:37ff:fe26:b059/ scope link
valid_lft forever preferred_lft forever
[root@docker-server2 ~]# kill -9 74050
[root@docker-server2 ~]# /usr/bin/flanneld --etcd-endpoints="http://192.168.132.131:2379" --iface=192.168.132.132 --etcd-prefix=/coreos.com/network &
[root@docker-server2 ~]# systemctl restart docker
[root@docker-server2 ~]# ip addr
: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default qlen
link/loopback ::::: brd :::::
inet 127.0.0.1/ scope host lo
valid_lft forever preferred_lft forever
inet6 ::/ scope host
valid_lft forever preferred_lft forever
: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP group default qlen
link/ether :0c:::fd: brd ff:ff:ff:ff:ff:ff
inet 192.168.132.132/ brd 192.168.132.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::6a92:62ba:1b33:c93d/ scope link noprefixroute
valid_lft forever preferred_lft forever
: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu qdisc noqueue state DOWN group default
link/ether ::::ff: brd ff:ff:ff:ff:ff:ff
inet 10.0.18.1/ brd 10.0.18.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80:::50ff:fe67:ff90/ scope link
valid_lft forever preferred_lft forever
: flannel.: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default
link/ether :b9:::bf: brd ff:ff:ff:ff:ff:ff
inet 10.0.18.0/ scope global flannel.
valid_lft forever preferred_lft forever
inet6 fe80::60b9:76ff:fe44:bf32/ scope link
valid_lft forever preferred_lft forever
可以在宿主机上查看到路由条目：
[root@docker-server1 ~]# ip route
default via 192.168.132.2 dev ens33 proto static metric
10.0.0.0/ dev flannel.
10.0.4.0/ dev docker0 proto kernel scope link src 10.0.4.1
10.0.18.0/ via 192.168.132.132 dev ens33
172.17.0.0/ dev br-b1c2d9c1e522 proto kernel scope link src 172.17.0.1
172.18.0.0/ dev br-ec4a8380b2d3 proto kernel scope link src 172.18.0.1
172.22.16.0/ dev br-f42e46889a2a proto kernel scope link src 172.22.16.1
192.168.132.0/ dev ens33 proto kernel scope link src 192.168.132.131 metric
扩展:
calico网络
- bgp转发:相当于host-gw转发,不能跨网段进行转发
- ipip转发:相当于vxlan转发模式
calico优势:
- 同时开启了两种转发模式,但是flannel只能开启其中一种转发模式
- 自动判断,如果宿主机跨网段,就是用ipip的方式转发,如果没有跨网段,就使用bgp方式转发
- 有流量策略管理,控制流量
ovs:openvswitch,openshift使用的网络
博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！
DOCKER学习_005:Flannel网络配置的更多相关文章
- K8s二进制部署单节点 etcd集群，flannel网络配置 ——锥刺股
K8s 二进制部署单节点 master --锥刺股 k8s集群搭建: etcd集群 flannel网络插件 搭建master组件 搭建node组件 1.部署etcd集群 2.Flannel 网络 ...
- docker学习3-虚拟网络模式
一.虚拟机网络模式 在理解docker网络隔离前,先看下之前虚拟机里对网络的处理,VirtualBox中有4中网络连接方式: NAT Bridged Adapter Internal Host-onl ...
- linux的视频学习4（网络配置和rpm)
linux的视频学习: 1.网络配置的三种方式的介绍. 第一种方式: setup 命令--选择network configuration-->配置固定ip(tab键)和自动分配IP(长空格) / ...
- docker 集群 flannel网络构建
先保证集群状态是正常的 集群管理 kubelet 在创建pod 时会先下载一个pause 镜像,这个镜像用于容器基础网络管理非常重要: 每个node 节点都要执行该操作: iptables -P FO ...
- Docker Network Configuration 高级网络配置
Network Configuration TL;DR When Docker starts, it creates a virtual interface named docker0 on the ...
- Docker 使用指南 (三)—— 网络配置
版权声明:本文由田飞雨原创文章,转载请注明出处: 文章原文链接:https://www.qcloud.com/community/article/95 来源:腾云阁 https://www.qclou ...
- FC网络学习笔记02 -网络配置方法
随着新一代飞机的综合化航电系统对通信需求的不断提高,传统的ARINC429.1553B总线的传输速率分别只有100Kbps和1Mbps,其带宽已远远不 论文联盟 http://Www.LWlm.cOm ...
- 红帽学习笔记[RHCE]网络配置与路由转发
目录 网络配置基本的IPV4与IPV6 拓扑图 操作 新加一块网卡 将增加的网卡分别加到两台虚拟机上 在两台虚拟机上配置IPV4与 IPV6 配置域名访问 拓展路由转发 拓扑图 操作 关于网关设置 重 ...
- (转) docker跨主机 macvlan 网络配置
原文链接 https://github.com/alfredhuang211/study-docker-doc/blob/master/docker%E8%B7%A8%E4%B8%BB%E6%9C%B ...
随机推荐
- 51nod 1686 第K大区间【离散化+二分】
题目链接: http://www.51nod.com/onlineJudge/questionCode.html#!problemId=1686 题意: 定义一个区间的值为其众数出现的次数. 现给出n ...
- Android系列之Android 命令行手动编译打包详解
Android 命令行手动编译打包过程图 [详细步骤]: 1使用aapt生成R.java类文件: 例: E:\androidDev\android-sdk-windows2.2\tools> ...
- LeetCode54 Spiral Matrix
题目: Given a matrix of m x n elements (m rows, n columns), return all elements of the matrix in spira ...
- Python--day69--ORM查询的13种方法
ORM查询的13种方法: 必知必会13条 <1> all(): 查询所有结果 <2> filter(**kwargs): 它包含了与所给筛选条件相匹配的对象 <3> ...
- behavior planning——12.example cost funtion -lane change penalty
In the image above, the blue self driving car (bottom left) is trying to get to the goal (gold sta ...
- Getting started with the basics of programming exercises_5
1.编写函数,把由十六进制数字组成的字符串转换为对应的整型值 编写函数htoi(s),把由十六进制数字组成的字符串(包含可选的前缀0x或0X)转换为与之等价的整型值.字符串中允许包含的数字包括:0~9 ...
- 解析P2P金融的业务安全
看了很多乙方同学们写的业务安全,总结下来,其出发点主要是在技术层面风险问题.另外捎带一些业务风险.今天我要谈的是甲方眼里的业务安全问题,甲方和乙方在业务安全的视野上会有一些区别和一些重合.在同一个问题 ...
- js写出你的名字的拼音，判断哪个字母出现的最多
function fn(str) { var obj = {}; for (var i = 0; i < str.length; i++) { if (!obj[str.charAt(i)]) ...
- H3C DCC概念
- KMPnext数组运用、最小循环节问题
http://www.cnblogs.com/jackge/archive/2013/01/05/2846006.html http://www.cnblogs.com/wuyiqi/archive/ ...