记一次bond引起的网络故障

本案中3个关键服务器

物理服务器：192.168.6.63，简称P，(Physical server)

KVM-VM：192.168.6.150，是物理服务器P上的一个KVM虚机，简称VM

NAS：外部NAS服务器，用来做ping/arp测试服务器，简称NAS

物理服务器P的配置：

#uname -a

Linux cz63 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64 GNU/Linux

#cat /etc/network/interfaces

auto lo

iface lo inet loopback

auto enp3s0f0

iface enp3s0f0 inet manual

auto enp3s0f1

iface enp3s0f1 inet manual

auto ens1f0

iface ens1f0 inet manual

auto ens1f1

iface ens1f1 inet manual

auto bond0

iface bond0 inet manual

	bond-slaves enp3s0f0 enp3s0f1

	bond-miimon 100

	bond-mode balance-rr

auto bond1

iface bond1 inet manual

	bond-slaves ens1f0 ens1f1

	bond-miimon 100

	bond-mode balance-rr

auto vmbr0

iface vmbr0 inet static

	address  192.168.6.63

	netmask  255.255.255.0

	gateway  192.168.6.1

	bridge-ports bond0

	bridge-stp off

	bridge-fd 0

auto vmbr1

iface vmbr1 inet static

	address  10.1.1.63

	netmask  255.255.255.0

	bridge-ports bond1

	bridge-stp off

	bridge-fd 0

#brctl show

bridge name	bridge id		STP enabled	interfaces

vmbr0		8000.ac1f6b342094	no		bond0

							tap401000001i0

vmbr1		8000.74a4b500e768	no		bond1

安装KVM-VM，之后VM与外部服务器的连接非常不稳定，90%情况下会出现ping不可达。这里测试了centos、ubuntu、win7都是一样的不稳定。

在VM内执行ping 192.168.6.40（NAS）

结果ping不通。

此时，在P上执行

tcpdump -leni vmbr0 arp

tcpdump -leni tap401000001i0 arp

发现情况如下：

P上

#tcpdump -leni vmbr0 arp | grep 2a:f0:5f:ae:c9:8b

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46

P上

#tcpdump -leni tap401000001i0  arp | grep 2a:f0:5f:ae:c9:8b

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46

2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28

NAS上

# tcpdump -leni vmx0 arp | grep c9:8b

02:03:50.930907 2a:f0:5f:ae:c9:8b > 00:50:56:87:86:b9, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 tell 192.168.6.150, length 46

02:03:50.930923 00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 42: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 28

02:04:02.669823 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:02.670131 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:03.670770 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:03.671059 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:04.672736 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:04.672992 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:06.671878 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:06.672021 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:07.674726 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:07.674773 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:08.676733 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:08.676868 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

02:04:10.673678 2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.32 tell 192.168.6.150, length 46

02:04:10.674026 00:50:56:87:e3:6d > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.32 is-at 00:50:56:87:e3:6d, length 46

vmbr0上发现了2个request包和1个reply包

tap401000001i0上仅发现了request包，而且length不一致，未发现reply包。

NAS上一切正常

看到的现象就是VM在发送icmp之前需要先获得NAS的MAC地址，所以需要先通过ARP广播获取到NAS的MAC地址。

P的vmbr0异常：接收到2个request，1个reply
P的vmbr0异常：接收到reply未发送至tap401000001i0

由于以上2个原因，导致VM无法收到reply，所以icmp不能进行，ping失败。

问题定位在数据链路层的ARP协议上。

在VM上执行arping -c 10 192.168.6.40

现象依据。再次证明以上2个现象。

# brctl show

bridge name	bridge id		STP enabled	interfaces

vmbr0		8000.ac1f6b342094	no		bond0

							tap401000001i0

vmbr1		8000.74a4b500e768	no		bond1

# brctl showstp  vmbr0

vmbr0

 bridge id		8000.ac1f6b342094

 designated root	8000.ac1f6b342094

 root port		   0			path cost		   0

 max age		  20.00			bridge max age		  20.00

 hello time		   2.00			bridge hello time	   2.00

 forward delay		   0.00			bridge forward delay	   0.00

 ageing time		 300.00

 hello timer		   0.00			tcn timer		   0.00

 topology change timer	   0.00			gc timer		   4.18

 flags

bond0 (1)

 port id		8001			state		     forwarding

 designated root	8000.ac1f6b342094	path cost		   4

 designated bridge	8000.ac1f6b342094	message age timer	   0.00

 designated port	8001			forward delay timer	   0.00

 designated cost	   0			hold timer		   0.00

 flags

tap401000001i0 (2)

 port id		8002			state		     forwarding

 designated root	8000.ac1f6b342094	path cost		 100

 designated bridge	8000.ac1f6b342094	message age timer	   0.00

 designated port	8002			forward delay timer	   0.00

 designated cost	   0			hold timer		   0.00

 flags

# brctl showmacs vmbr0

port no	mac addr		is local?	ageing timer

  1	2a:f0:5f:ae:c9:8b	no		   1.13

  1	00:50:56:87:e3:6d	no		   0.00

# iptables -L

Chain INPUT (policy ACCEPT)

target     prot opt source               destination

Chain FORWARD (policy ACCEPT)

target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)

target     prot opt source               destination

# ebtables -L

Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

# ip rule show

0:	from all lookup local

32766:	from all lookup main

32767:	from all lookup default

# ip route show table all

default via 192.168.6.1 dev vmbr0 onlink

10.1.1.0/24 dev vmbr1 proto kernel scope link src 10.1.1.63

192.168.6.0/24 dev vmbr0 proto kernel scope link src 192.168.6.63

broadcast 10.1.1.0 dev vmbr1 table local proto kernel scope link src 10.1.1.63

local 10.1.1.63 dev vmbr1 table local proto kernel scope host src 10.1.1.63

broadcast 10.1.1.255 dev vmbr1 table local proto kernel scope link src 10.1.1.63

broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1

local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1

local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1

broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1

broadcast 192.168.6.0 dev vmbr0 table local proto kernel scope link src 192.168.6.63

local 192.168.6.63 dev vmbr0 table local proto kernel scope host src 192.168.6.63

broadcast 192.168.6.255 dev vmbr0 table local proto kernel scope link src 192.168.6.63

# bridge vlan show

port	vlan ids

bond0	 1 PVID Egress Untagged

vmbr0	 1 PVID Egress Untagged

bond1	 1 PVID Egress Untagged

vmbr1	 1 PVID Egress Untagged

tap401000001i0	 1 PVID Egress Untagged

以上一切正常，未发现输出问题，思路该怎么展开？

google了一下午加一个晚上，各种不解决。

最后看到了linux关于bond的说明：

https://forum.huawei.com/enterprise/zh/thread-282727.html

再看看P的网卡，明明是mode0啊！

这时候再去交换机上一看！哈啊~！

interface GigabitEthernet0/0/5

 port link-type access

 port default vlan 6

#

interface GigabitEthernet0/0/6

 port link-type access

 port default vlan 6

#

interface GigabitEthernet0/0/7

 port link-type access

 port default vlan 6

#

interface GigabitEthernet0/0/8

 port link-type access

 port default vlan 6

感觉好像是这个问题。

然后ifdown ens1f1，问题解决。

VM与所有外部网络通讯正常，无任何故障现象。

通过阅读华为网站的文章，确定了balance-rr需要交换机做修改。

不想劳烦网管，所以自己改成mode6。

一切归于平静。

问题只是临时解决了，但是产生现象的原理还没明白。

2009年自己就详细看了windows teaming linux bond的原理及配置，苦于只是知道OS层面，不知道switch侧应该怎么对应，所以知识点一直有欠缺，2017年、2018年都曾经入坑，可惜哪里有2台物理服务器和物理交换机实践一下呀！
这次总算补齐了OS和switch的配置。

能用mode6就用mode6，再其次就是mode4。

以下给自己看的:

brctl show							#查看bridge信息

brctl showstp  vmbr0				#查看vnbr0的stp信息，是否转发等

brctl showmacs vmbr0 | grep c9:8b	#查看bridge的MAC地址

tcpdump -leni vmbr0 arp				#debug vmbr0网卡的arp包信息

tcpdump -leni vmbr0 icmp			#debug vmbr0网卡的icmp包信息

iptables -A FORWARD -i vmbr0 -o vmbr0 -j ACCEPT

iptables -L		#查看iptables

ebtables -L		#查看数据链路层的table信息

arping -c 10 192.168.1.1			#arp ping

ip rule show

ip route show table local

ip route show table all

bridge monitor			#monitor fdb update

bridge vlan show		#查看bridge的vlan信息

ping -c 1 -I veth1 192.168.3.1		#指定网卡进行ping测试

/proc/sys/net/bridge/bridge-nf-call-iptables	#干嘛的

/proc/sys/net/ipv4/ip_forward					#多网卡下，数据包转发

sysctl -a | grep bridge

net.bridge.bridge-nf-call-arptables中的0或者1代表什么意思？？

echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore		#什么意思？？

echo 8 > /proc/sys/net/ipv4/conf/eth0/arp_announce		#什么意思？？

记一次bond引起的网络故障的更多相关文章

记一次诡异的网络故障排除 - tpc_tw_recycle参数引起的网络故障
一.故障现象我们团队访问腾讯云上部署的测试环境中的Web系统A时,偶尔会出现类似于网络闪断的情况,浏览器卡很久没有反应,最终报Connection Timeout. 不过奇怪的是,当团队中的某个人无 ...
IEEP部署企业级网络工程-网络故障-环路故障
网络故障 1.环路故障概念 1).以太网是一个支持广播的网络, 在没有环路的环境中,广播报文在网络中以泛洪的形式被送达到网络的第一个角落,以保证每个设备都能够接受到它.每台二层设备在接收到广播报文以 ...
React Native 在用户网络故障时自动调取缓存
App往往都有缓存功能,例如常见的新闻类应用,如果你关闭网络,你上次打开App加载的数据还在,只是不能加载新的数据了. 我的博客bougieblog.cn,欢迎前来尬聊. 集中处理请求如果你fetc ...
ZK 网络故障应对法
网络故障可以说是分布式系统天生的宿敌.如果永远不发生网络故障,我们实际上可以设计出高可用强一致的分布式系统.可惜的是不发生网络故障的分布式环境还不存在,ZK 使用过程中也需要小心的应付网络故障. 让我 ...
Linux多网卡绑定（bond）及网络组(team)
Linux多网卡绑定(bond)及网络组(team) 很多时候,由于生产环境业务的特殊需求,我们需要对服务器的物理网卡实施特殊的配置,从而来满足不同业务场景下对服务器网络的特殊性要求.如高并发的网 ...
traceroute排查网络故障 www.qq.com排查网络故障网络不通先ping自己
网络不通先ping自己在ping网关再ping外网再ping别人的ip 背景需求 Linux 因为其强大的网络处理能力,被广泛用于网关(实例链接)和服务器(实例链接).实际工作中,快速排查这些 ...
一次“不负责任”的 K8s 网络故障排查经验分享
作者 | 骆冰利来源 | Erda 公众号某天晚上,客户碰到了这样的问题:K8s 集群一直扩容失败,所有节点都无法正常加入集群.在经过多番折腾无解后,客户将问题反馈到我们这里,希望得到技术支持 ...
linux篇-公司网络故障那些事（路由器变交换机）
首先这次网络故障是断电引起的我给大家画个模型三层的为八口交换机一层的为五口打印机笔记本代表两台无线打印机首先八口的连接了公司采购电脑一台,业务电脑一台,其他电脑三台第二个五口交换的连接财务 ...
记一次linux Docker网络故障排除经历
背景: 之前做了一个项目,需要在容器内访问宿主机提供的Redis 服务(这是一个比较常见的应用场景哈), 常规方案: ① 主机网络(docker run --network=host): 完全应用 ...

随机推荐

浅析toString()和toLocaleString()的区别
翻看<javascript高级程序设计>,对toString()和toLocaleString()两者的区别并没有直白的说明,简单查阅了一下 ecmascript官方文档和 MDN 二者 ...
C# autocomplete
前台代码 <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml"> <head runa ...
asp.net开启多线程异步处理
protected void Page_Load(object sender, EventArgs e) { if (!IsPostBack) { try { Thread categoryThrea ...
Oracle数据库基础SQL语句练习
一.说明第一次使用Oracle,想做一些练习,熟悉一些oracle. 表:使用的是scott用户,默认的表具体表讲解,可以参考该文档:https://www.cnblogs.com/xjcheng ...
（转）python中函数参数中如果带有默认参数list的特殊情况
在python中函数参数中如果带有默认参数list遇到问题先看一段代码 1 2 3 4 5 6 7 8 9 def f(x,l=[]): for i in range(x): ...
vue-----表单与组件
<!DOCTYPE html><html><head> <meta charset="utf-8"> <meta name=& ...
【scarpy】笔记三：实战一
一.前提我们开始爬虫前,基本按照以下步骤来做: 1.爬虫步骤:新建项目,明确爬虫目标,制作爬虫,存储爬虫内容二.实战(已豆瓣为例子) 2.1 创建项目 1.打开pycharm -> 点开te ...
Servlet接口UML图
Servlet.class源码
ruoyi管理系统建立子项目，卡住
这个一定不要勾选,不然依赖加了还是引用不到.
How to detect, enable and disable SMBv1, SMBv2, and SMBv3 in Windows and Windows Server
转自:https://support.microsoft.com/en-us/help/2696547/detect-enable-disable-smbv1-smbv2-smbv3-in-windo ...

记一次bond引起的网络故障

记一次bond引起的网络故障的更多相关文章

随机推荐

热门专题