)
作者张华 发表于2023-07-12版权声明可以任意转载转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明IPVS是Linux内核态的一个L4 LB (KTCPVS, Kernel TCP Virtual Server则是L7 LB), IPVS通过在Netfilter框架中的不同位置注册自己的处理函数来捕获数据包并根据与IPVS相关的信息表对数据包进行处理按照IPVS规则中定义的不同的包转发模式对数据包进行不同的转发处理。IPVS有哪些包转发模式NAT、IP tunneling和Direct Routing。本文要做的实验是关于Direct Routeing (client - LB - backend - client), 即从backend返回的包直接direct routing至client (若仍由LB转发的叫NAT, 不会对传输层的端口做修改叫tunneling).实验./generate-bundle.sh --name ussuri-ovn-2203 --replay --run juju config neutron-api-plugin-ovn dns-servers10.5.0.15 ./tools/vault-unseal-and-authorise.sh ./configure openstack router remove subnet provider-router private_subnet openstack subnet delete private_subnet openstack network delete private source ./novarc network172.16.0.0/24 net_startipcalc $network| awk $1HostMin: {print $2} net_endipcalc $network| awk $1HostMax: {print $2} d${net_start##*.} net_start${net_start%.*}.$((d)) n_idopenstack network create --enable nosec --disable-port-security -c id -f value sn_idopenstack subnet create --allocation-pool start$net_start,end$net_end \ --subnet-range $network --dhcp --ip-version 4 --network $n_id nosec_subnet -c id -f value openstack router add subnet provider-router $sn_id openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey openstack server create --wait --image jammy --flavor m1.small --key-name mykey --nic net-id$n_id nosecvm1 openstack server create --wait --image jammy --flavor m1.small --key-name mykey --nic net-id$n_id nosecvm2 openstack server create --wait --image jammy --flavor m1.small --key-name mykey --nic net-id$n_id nosecvm3 ./tools/float_all.sh IPVS server: nosecvm1172.16.0.223/fa:16:3e:1e:fe:eb real server: nosecvm2172.16.0.75/fa:16:3e:06:8e:ab eth-vip172.16.0.177/b2:65:dc:e1:a8:e5 client server: nosecvm3172.16.0.24/fa:16:3e:5f:46:c0 #on nosecvm1 apt install ipvsadm -y echo 1 /proc/sys/net/ipv4/ip_forward ipvsadm -C ipvsadm -A -t 172.16.0.177:80 -s rr ipvsadm -a -t 172.16.0.177:80 -r 172.16.0.75 -g ipvsadm -l -n #on nosecvm2 sudo apt install nginx net-tools -y VIP172.16.0.177 RIP172.16.0.75 sudo arptables -F sudo arptables -D INPUT -d $VIP -j DROP sudo arptables -D OUTPUT -s $VIP -j mangle --mangle-ip-s $RIP sudo arptables -L -n -v sudo modprobe -v dummy numdummies1 sudo ip addr add $VIP/32 dev dummy0 ip link add eth-vip type dummy ip link set eth-vip up ip a add $VIP/32 dev eth-vip arptables -F arptables -A INPUT -d $VIP -j DROP arptables -A OUTPUT -s $VIP -j DROP arptables -L -n -v #on nosecvm3 curl 172.16.0.177 Then I successfully reproduce the problem. rootnosecvm3:~# curl 172.16.0.177 curl: (7) Failed to connect to 172.16.0.177 port 80 after 3069 ms: No route to host Why was my test result (the comment 2023-08-02 08:49 UTC) random last time? Thats because I didnt run above arptables commands to suppress the ARP reply from the real server. I can reproduce the problem again after running above arptables commands and run arp -d 172.16.0.177 in client server even if I havent created p1 (openstack port create --enable-port-security --network nosec p1).实验要注意一点必须运行上面的arptables命令来防止client直接访问了real server, 正确地应该是client通过IPVS来访问real server.分析TCP回顾如下使用了IPVS DR的TCP拓扑如下( https://bugzilla.redhat.com/show_bug.cgi?id2175037) 上面的drop package只在ovn中出现 在ovs中不出现,“ovn-trace” 与 ovs-appctl ofproto/trace确认没有drop ACK包, 所以ACK应该已经到了LB.所以可能和ovs的conntrack有关但是问题发生时’ovs-appctl dpctl/dump-conntrack’显示是ESTABLISHfdb is not suspect because ACK packets are dropped in br-int on the compute node, and br-int doesn’t have NORMAL rules.这里有一个commit很可疑 - https://github.com/ovn-org/ovn/commit/d17ece7f3706bd1257e181d69705019c0abb7e51conntract学习 - https://blog.motitan.com/2017/09/02/ovn%E5%AD%A6%E4%B9%A0-5-conntrack/其他 - 检查conntrck都disable port SG了不应该有conntrack了确认它while true; do curl 192.168.21.238; sleep 3; done conntrack -L |grep 192.168.21 |grep -v sport22 ovs-appctl dpctl/dump-conntrack |grep 192.168.21 |grep -v sport22 ovs-dpctl dump-conntrack |grep 192.168.21 |grep -v sport22 ovs-dpctl dump-flows |grep 192.168.21 |grep -v sport22其他 - 查看一个port下的流ovn-nbctl lsp-get-ls neutron_port_id ovn-sbctl find Port_Binding logical_portneutron_port_id ovn-sbctl dump-flows datapath_id_from_previous_command or neutron-net-id20230829更新VIP: 192.168.22.222 Client: 192.168.22.219 Frontend: 192.168.22.40 Backend: 192.168.22.192 正常的只有zone7, 10 tcp 6 431998 ESTABLISHED src192.168.22.219 dst192.168.22.192 sport34850 dport80 src192.168.22.192 dst192.168.22.219 sport80 dport34850 [ASSURED] mark0 zone7 use1 tcp 6 431998 ESTABLISHED src192.168.22.219 dst192.168.22.192 sport34850 dport80 src192.168.22.192 dst192.168.22.219 sport80 dport34850 [ASSURED] mark0 zone10 use1 但有问题的有zone7, 10, 8 tcp 6 55 SYN_RECV src192.168.22.219 dst192.168.22.222 sport57756 dport80 src192.168.22.222 dst192.168.22.219 sport80 dport57756 mark0 zone10 use1 tcp 6 115 SYN_SENT src 192.168.22.219 dst192.168.22.222 sport57756 dport80 [UNREPLIED] src192.168.22.222 dst192.168.22.219 sport80 dport57756 mark0 zone8 use1 tcp 6 298 ESTABLISHED src192.168.22.219 dst192.168.22.222 sport57756 dport80 src192.168.22.222 dst192.168.22.219 sport80 dport57756 [ASSURED] mark0 zone7 use1IPVS Route模式路由是非对称的 此时在conntrack就容易引起问题OVN运行conntrack两次(ingress and egress), zone是根据port id来的这样ingress的包被conntrack看成了zonefrontend, 但是egress的包被conntrack看成了zonebackend, 所以不匹配出问题了。从Client到Backend的包先经过了LB VM, 但由于是IPVS Direct Routing模式从backend回来的SYNACK包没有经过LB VM而是直接到了Client. 所以connection tracking将始终显示SYN_SENT状态. Client得到SYNACK之后的回复包又得经过LB VM这时就会被作为inv给drop掉。Ingress: Client - Frontend (IPVS Load Balancer) - Backend (Web Server) Egress: Backend (Web Server) - ClientOVN目前已经优化如针对stateless的SG rule会略过conntrackhttps://github.com/ovn-org/ovn/commit/a0f82efdd9dfd3ef2d9606c1890e353df1097a51这样LB应该disable SG, 因为它是将client来的不同IPs的包发给backend, 我们在openstack中无法使用allowed_address_pairs因为它是来自不同互联网中的随机IP的LB也应该disable conction tracking而目前OVN只会在statelesss模式时disable connection tracking, neutron也是不支持在disable SG还能用 stateless SG的一个work around是Client得到SYNACK之后的回复包又经过LB VM作为inv设置不被drop掉 (sudo ovn-nbctl set NB_Global . options:use_ct_inv_matchfalse), 但这个option是全局性的, 这里有一些讨论 - https://mail.openvswitch.org/pipermail/ovs-dev/2021-March/380859.html又一案例如上图 从物理路由器AGG1上ping 12.1.1.12时 12.1.1.12所在的host nc17上对vhu4630a950-37抓包时能看到arp request, 但VNE虚拟路由器可能用ECMP or HA Active/Standby导致从12.1.1.11 /nc14 (ping reply: (src:mac-12.1.1.11:12.1.1.12, dst:mac-12.1.1.9:12.1.1.9) (NOTE: the source IP is set to 12.1.1.12 rather than 12.1.1.11)上构建ping reply包出去(它应该用PBR这种策略路由算法让ping reply包从nc17出去。ovn是在acl_eval pipline中来处理packages的#the pipline acl_eval on nc17 1, 65532 DROP ct.inv 2, 65532 ALLOW established 3, 65532 ALLOW related 4, User SG rule, eg: tcp tcp.dst 22 #src-port is temporary port from nc14, not 22 5, Default - DROP对称路由(从nc17进又从nc17回的包 走上面第2步的established路径就行。非对称路由(从nc17进(conntrack with zonenc17)又从nc14回的包(conntrack with zonenc14))此时在nc17看来是一个ct.int包 被上面的第1步给drop掉了。用use_ct_inv_matchfalse就在第1步高优先级就被放行了。非对称路由 use_ct_inv_matchfalse stateful fw行不行呢 答案是也不行。因为也不符合第2步65532 ALLOW established的规则也会被DROP掉。所以解决方法只有两种1, use_ct_inv_matchfalse port_security_enablefalse2, use_ct_inv_matchfalse port_security_enabletrue stateless fw下面是如何创建stateless SG (必须是双向显式定义 没有回包自动放行) # 1. 创建 stateless SG关键是 --stateless openstack security group create --stateless vne-stateless-sg # 2. stateless SG 必须双向显式定义规则没有回包自动放行 # 允许来自 12.1.1.0/24 的 ICMP 进入 VNE openstack security group rule create --direction ingress --protocol icmp --remote-ip 12.1.1.0/24 vne-stateless-sg # 允许 VNE 向 12.1.1.0/24 发出 ICMP包括 asymmetric reply openstack security group rule create --direction egress --protocol icmp --remote-ip 12.1.1.0/24 vne-stateless-sg # 根据需要加其他stateless rules # 3. 应用到 VNE 的两个 port openstack port set --no-security-group --security-group vne-stateless-sg port-id 如何用stateless SG: # 第一步开启 port-securityanti-spoofing 恢复 openstack port set --enable-port-security 4d7f6059-bb31-4935-8c97-5fda2b83fab6 # 第二步摘掉原来的 stateful SGvne_SecurityGroup openstack port set --no-security-group 4d7f6059-bb31-4935-8c97-5fda2b83fab6 # 第三步挂上 stateless SG openstack port set --security-group vne-stateless-sg 4d7f6059-bb31-4935-8c97-5fda2b83fab6总结一下就是The logic for processing stateful packages in ovn is generally as shown in the following pseudo-code. Step 1: priority 65532 - DROP ct.inv # drop INVALID packets Step 2: priority 65532 - ALLOW ct.est ct.rpl # allow established packets Step 3: priority 65532 - ALLOW ct.rel # allow related traffic Step 4: User SG rules, eg: tcp tcp.dst 22 # only matches new connections in the defined direction (eg: stateless fw rules) Step 5: Default - DROP # drop everything else Due to special asymmetric design, the conntrack zones for the request package and reply package are different. Consequently, the reply package was dropped by step 1 above. The modification use_ct_inv_matchfalse removes only Step 1, but step 5 remains. Since this kind of asymmetric reply package cant meet the condtion of steps 2,3, or 4, it will be dropped by step 5. For this reason, the valid configurations are: 1, VALID: use_ct_inv_matchfalse port_security_enabledfalse 2, VALID: use_ct_inv_matchfalse port_security_enabledtrue stateless firewall 3, NOT VALID: use_ct_inv_matchfalse port_security_enabledtrue stateful firewall (even removing anti-spoofing via allowed-address-pairs) This is not a bug. The conntrack state for this connection exists only on nc17 (where the request entered), while the reply exits from nc14. Conntrack is per-host and per-zone by design, and this asymmetric cross-host traffic pattern is outside its design scope.20260716 - 一个子网上创建了两个路由 - ecmp_symmetric_reply客户在同一个subnet上创建了两个routers, 且两个routers被调度到了不同节点上导致SYN-ACK回包有时候被从一个节点通过tunnel传到另一个节点后再回给client, 这时client就会丢弃SYN-ACK包。客户说说他有两个client:client 10.97.144.221, 50%的时候有问题收不到SYN-ACK, SYN-ACK会从active router CN1 传到 CN3 (standby router 没有snat-xxx所以无法做SNAT)client 10.65.107.17 100成功无问题总是能收到SYN-ACK两个clients都在这个openstack之外我首先想到的是我以前做过的一个lp bug - https://bugs.launchpad.net/neutron/bug/1732067we have a problem in openstack deployed with random asymmetric routing on the host running amphora VM. Openstack is Ussuri on Focal, OVN is 22.03. Setup Communication path: clients (10.97.144.221, 10.65.107.17) - floating IP 10.96.90.219 (active gateway chassis CN1) - master amphora (10.0.0.177, VM running on host CN2) - further to customer backend mysql VMs Both clients are outside of openstack scope, 10.97.144.221 is a client from another openstack deployed in another zone. Problem In ~50% of time client 10.97.144.221 fails to establish TCP connection. All client SYN packets reach amphora but sometimes SYN-ACK is sent back not over CN1 but over another host CN3. Client 10.65.107.17 works fine, responses always go through CN1. Floating IP has five gateway chassis in a group and compute node CN1 has the highest priority. CN3 is also one of the gateway chassis for that fIP but with a lower priority.客户端 │ │ 1. [SYN] src10.97.144.221:52818 → dst10.96.90.219:3306 ▼ [物理网络] │ ▼ j701lwzr (Active GW, CN2) ←─┐ │ │ │ 2. DNAT: 10.96.90.219 → 10.0.0.177 (VIP) │ 3. 包进入 Amphora VM (tap) │ │ 4. Amphora 回复 [SYN-ACK] src10.0.0.177:3306 → dst10.97.144.221:52818 │ │ ✅ 5. OVN 流表判断本地有 FIP binding → 执行 SNAT: │ src10.0.0.177 → 10.96.90.219 │ │ 6. [SYN-ACK] src10.96.90.219:3306 → dst10.97.144.221:52818 │ ↑ 从 bondB 接口发出带 SNAT ▼ 客户端收到 → 握手成功 ✅客户端 │ │ 1. [SYN] src10.97.144.221:11772 → dst10.96.90.219:3306 ▼ [物理网络] │ ▼ j701lwzr (Active GW, CN2) ←─┐ │ │ │ 2. DNAT: 10.96.90.219 → 10.0.0.177 正常 │ 3. 包进入 Amphora VM (tap) │ │ 4. Amphora 回复 [SYN-ACK] src10.0.0.177:3306 → dst10.97.144.221:11772 │ │ ⚠️ 错误发生OVN 流表误判 │ “该 FIP 的 SNAT 应由 j701lwzx 处理” │ → 不执行本地 SNAT而是封装成 Geneve 隧道包 │ │ 5. Geneve 封装 │ outer: srcj701lwzr(100.98.169.32) → dstj701lwzx(100.98.159.32) │ inner: src10.0.0.177:3306 → dst10.97.144.221:11772 [SYN-ACK] ▼ j701lwzx (备用 GW, CN3) ←─┘ │ │ 6. j701lwzx 收到 Geneve 包解封装 │ → 发现这是“去往外部”的回程流量 │ → 但它的 OVN 流表认为“我不是 FIP 的 owner不负责 SNAT” │ → 直接从 bondB 发出原始包 │ [SYN-ACK] src10.0.0.177:3306 → dst10.97.144.221:11772 │ ❗ 没有 SNAT源 IP 仍是内网 VIP ▼ 客户端收到 src10.0.0.177非 FIP→ 被 TCP 协议栈丢弃 因为客户端发起连接的目标是 10.96.90.219却收到 10.0.0.177 的响应 │ │ 客户端重传 SYN → 最终超时 ❌lp bug 1945306 是说用户在使用standby router做为compute node, 所以导致这上面的VM无法访问外网 因为standby router上是没有 snat-xxx)的 openstack社区说不要使用dvr_snat的结点作为compute - https://review.opendev.org/c/openstack/neutron//801503 charm的patch是 https://bugs.launchpad.net/charm-neutron-openvswitch/bug/1947300但是bug 1947300说的是:dvr_snat的结点不适合做计算节点但这不是ovs而是ovn环境它是推荐做为计算节点的啊.explicitly_egress_direct 这个参数也只是对ovs环境有效对ovn无效啊虽然 OVN 设计上允许 Standby GW 做计算节点但你当前遇到的 Bug #1945306 恰恰说明在 Ussuri OVN 22.03 这个特定版本组合下当 Amphora VM 与 Active GW 同宿主机时流表生成存在缺陷。角色可以共置不等于完全没有要求。计算兼 gateway 节点需要正确配置 external physnet bridge mapping provider bridge 确实连接外部网络 Geneve tunnel IP 双向稳定 BFD 正常 external VLAN、MTU、物理交换机配置一致 有足够 CPU 处理 VM 流量、OVN pipeline、conntrack 和 gateway 流量 所有 gateway candidate 对同一 external network 都具备真实的物理连通性。例如ovs-vsctl get Open_vSwitch . external_ids:ovn-cms-options ovs-vsctl get Open_vSwitch . external_ids:ovn-bridge-mappings ovs-vsctl get Open_vSwitch . external_ids:ovn-encap-ip enable-chassis-as-gw physnet1:br-ex 192.0.2.13如果 CN3 被加入 gateway group却没有与 CN1 等价的 br-ex/physnet 连通性它仍可能在 BFD failover 后成为 active但北南向流量会失败。CN1/CN3 同时运行 nova-compute 本身是 OVN 支持的不应作为回程漂移的根因。更需要核对的是CN3 是否真的拥有 external physnet 连通性 CN2 到 CN1 的 BFD/Geneve 是否间歇 down CN2 与 CN3 是否将不同 chassis 判为 active gateway failover 时入方向为什么仍由物理网络送往 CN1 FIP NAT 是集中式还是分布式。新发现客户的下面的数据提提到了这个mac: fa:16:3e:3d:4e:a9, 它正是另一个router的网关mac1. Short summary A client (10.97.144.221) opens two TCP connections per minute to fIP 10.96.90.219:3306 running on chassis/host j701lwzr fIP is assigned to octavia LB with IP 10.0.0.177 active on master amphora VM with primary IP 10.0.0.240 running on the same host j701lwzr one active haproxy backend server VM is 10.0.0.116:32279 running on host j70151hz around 50% of connections fail because first SYN-ACK response is sent over another host j701lwzx, instead of host j701lwzr 2. Example of successful request Client 10.97.144.221 opens connection on src port 52818. #2 t486.652836 10.97.144.221:52818 → 10.96.90.219:3306 [SYN] #3 t486.654502 10.96.90.219:3306 → 10.97.144.221:52818 [SYN-ACK] ← SNAT applied, correct FIP src #4 t486.660698 10.97.144.221:52818 → 10.96.90.219:3306 [ACK] ← handshake complete SYN received: 10.97.144.221:52818 → 10.0.0.177:3306 [SYN] SYN-ACK sent: 10.0.0.177:3306 → 10.97.144.221:52818 [SYN-ACK] #33251 t486.661334 outer: 100.98.169.32(j701lwzr) → 100.98.129.31(j70151hz) inner: 10.0.0.240:62621 → 10.0.0.116:32279 [SYN] ← haproxy→backend SYN received and answered — full session established SYN-ACK exits j701lwzr-bondB with SNAT applied ( 10.0.0.177 → 10.96.90.219 ). 3. Example of failed request Client 10.97.144.221 opens connection on src port 11772. #1 t486.158774 10.97.144.221:11772 → 10.96.90.219:3306 [SYN] ← arrives on active GW #33 t487.168960 10.97.144.221:11772 → 10.96.90.219:3306 [SYN] ← client retry 1 #34 t489.216802 10.97.144.221:11772 → 10.96.90.219:3306 [SYN] ← client retry 2 ← NO SYN-ACK ever seen here #1 t486.160835 10.97.144.221:11772 → 10.0.0.177:3306 [SYN] ← DNAT on j701lwzr ✓ #2 t486.160915 10.0.0.177:3306 → 10.97.144.221:11772 [SYN-ACK] ← amphora replies #34 t487.169050 (SYN retry received, SYN-ACK repeated — no ACK ever arrives) (j701lwzr geneve — SYN-ACK leaves via tunnel to j701lwzx!) #29304 t486.162354 outer: 100.98.169.32(j701lwzr) → 100.98.159.32(j701lwzx) inner: 10.0.0.177:3306 → 10.97.144.221:11772 [SYN-ACK] #7973 t486.162383 outer: 100.98.169.32(j701lwzr) → 100.98.159.32(j701lwzx) inner: 10.0.0.177:3306 → 10.97.144.221:11772 [SYN-ACK] (j701lwzx sends to wire — no SNAT) #1 t486.163709 src_macfa:16:3e:3d:4e:a9 10.0.0.177:3306 → 10.97.144.221:11772 [SYN-ACK] ← private VIP, NOT FIP! ← No traffic for this connection. Haproxy never opened backend connection而这个mac fa:16:3e:3d:4e:a9来自另一个router:router 6f333dd3-9587-437d-afd2-4b5de6cd6f4e (neutron-4ebeade0-fb54-44b4-b2fd-d75574adbb68) (aka sdp-g-prod-oam_mon-router) port lrp-b3331655-fce8-4f4c-9a46-ba5b89de925e mac: fa:16:3e:a6:46:7c networks: [10.0.0.5/24] port lrp-276be1f9-4c26-4071-9831-d1382a4b17a6 mac: fa:16:3e:3d:4e:a9 networks: [10.96.90.155/26] gateway chassis: [ic-atgraf1-j701lwzr.ic-atgraf1.xxx ic-atgraf1-s4cej834.ic-atgraf1.xxx ic-atgraf1-j701lwzx.ic-atgraf1.xxx ic-atgraf1-s4cfc247.ic-atgraf1.xxx ic-atgraf1-j701lwzn.ic-atgraf1.xxx] router 7085fdbe-93c6-4813-a22e-11ff7cd4b05b (neutron-3a3767a3-0511-40bb-9532-6bc7768a4065) (aka sdp-g-prod-app-router) port lrp-d74f14e6-a8d2-43ee-9fab-70832c945adc mac: fa:16:3e:42:35:55 networks: [10.0.0.6/24] port lrp-47a4767f-d9a6-4d0d-a32d-a9dfdf43df2f mac: fa:16:3e:6e:3a:fb networks: [10.96.90.199/27]所以似乎是两个router(10.96.90.155/26 and 10.96.90.199/27) 挂在同一个subnet上 (10.0.0.0/24), 挂了2个router。我的问题是:在ovn中两个router gw是可以同时分布在不同的网关节点上的.在我们的环境中sdp-g-prod-oam_mon-router 的网关端口被调度到了 j701lwzx而 Octavia LB 所在的 Active Gateway 是 j701lwzr。由于它们共享同一个底层物理网络10.0.0.0/24当 Amphora VM 在 j701lwzr 上发出回程包时OVN 的 ovn-controller 在查询 MAC_Binding 表时发生了混淆误以为目的 MAC fa:16:3e:3d:4e:a9 属于远端的 j701lwzx从而将包通过 Geneve 隧道发给了 j701lwzx若设置juju config neutron-ovn enble-distributed-floating-ipfalse是不是就会让两个router gw部署到priority最大的那一个网关节点上这样就解决了问题呢它控制的是 FIP NAT 执行位置不控制 gateway LRP 调度。Gateway chassis 仍由 ovn_l3_scheduler 按每个 router 独立选择(同一个子网上存在两个 router port 时VM/amphora 必须通过二层目的 MAC明确选择进入哪个 router)。解决思路既然明确了是“两个路由器共享同一个底层物理网段导致 MAC_Binding 冲突”你可以考虑以下几种真正的解决方向方向一隔离底层物理网络最根本的解决方式 如果可能让这两个路由器Octavia LB 路由器和 sdp-g-prod-oam_mon-router使用不同的物理 Provider 网络即不同的 VLAN 或不同的物理网段。只要它们的网关端口不在同一个二层广播域MAC 地址就不会冲突OVN 的 MAC_Binding 也就不会混淆。 方向二调整 Gateway Chassis Group 的调度规避冲突 如果你无法更改物理网络拓扑可以尝试手动干预 OVN 的调度。 你可以使用 ovn-nbctl 命令将 sdp-g-prod-oam_mon-router 的网关端口强制绑定到与 Octavia LB 相同的 Active Gateway Chassis 上。 当两个路由器的网关端口都在 j701lwzr 上时OVN 在本地就能正确处理路由和 SNAT不会再发生跨节点的 MAC_Binding 误判。 方向三升级 OVN 版本 OVN 社区在后续的版本中22.09 及以后针对分布式路由器的 MAC_Binding 刷新机制和跨路由器网关调度做了大量优化。如果条件允许升级 OVN 是解决此类底层逻辑缺陷的长期方案。如果客户坚决要在一个subnet上用两个router, 强制将两个路由器的网关绑定到同一个节点最推荐:# 1. 获取目标物理节点 j701lwzr 的 Gateway_Chassis UUID # (假设你的 chassis name 就是主机名请根据实际情况替换) CHASSIS_UUID$(ovn-sbctl --bare --columns _uuid find Chassis nameic-atgraf1-j701lwzr | tr -d ) # 2. 强制将 lrp-276be1f9... 这个端口的网关绑定到 j701lwzr # 这里的 100 是优先级数字越大优先级越高 ovn-nbctl lrp-set-gateway-chassis lrp-276be1f9-4c26-4071-9831-d1382a4b17a6 $CHASSIS_UUID 100 # 3. 验证绑定结果 ovn-nbctl show | grep -A 5 lrp-276be1f9问了客户为什么用两个router后他说For router sdp-g-prod-router that acts as a default gateway (10.0.0.1) for subnet [10.0.0.0/24](http://10.0.0.0/24), I have noticed that it has the same route twice, but with different nexthop: $ openstack router show 321e6ca8-b53c-4286-84ee-765f990b10ee -c routes -f yaml | grep 10.97.144.220 -A1 - destination: [10.97.144.220/30](http://10.97.144.220/30) nexthop: 10.0.0.5 - destination: [10.97.144.220/30](http://10.97.144.220/30) nexthop: 10.0.0.6 They are these two routers: 1. sdp-g-prod-oam_mon-router, 4ebeade0-fb54-44b4-b2fd-d75574adbb68, [10.0.0.5/24](http://10.0.0.5/24), [10.96.90.155/26](http://10.96.90.155/26) 2. sdp-g-prod-app-router, 3a3767a3-0511-40bb-9532-6bc7768a4065, [10.0.0.6/24](http://10.0.0.6/24), [10.96.90.199/27](http://10.96.90.199/27) I will ask the client to keep only 10.0.0.6 and well see if it helps.作为默认网关的 sdp-g-prod-router 上存在两条完全相同的路由规则目标网段: 10.97.144.220/30下一跳1: 10.0.0.5 (指向 sdp-g-prod-oam_mon-router)下一跳2: 10.0.0.6 (指向 sdp-g-prod-app-router)这是一个典型的等价多路径ECMP路由配置。这意味着当 sdp-g-prod-router 需要转发去往 10.97.144.220/30 网段的流量时它会在两个下一跳之间进行负载均衡。1, 客户端10.97.144.221的流量到达默认网关 sdp-g-prod-router。2, sdp-g-prod-router 需要将流量转发给 Octavia 的 FIP10.96.90.219这个 FIP 所在的网络 10.96.90.199/27 是由 sdp-g-prod-app-router (10.0.0.6) 管理的。3, 但由于 ECMP 的存在sdp-g-prod-router 有 50% 的概率会将流量错误地转发给 sdp-g-prod-oam_mon-router (10.0.0.5)。4, sdp-g-prod-oam_mon-router 收到这个包后发现目标 IP 10.96.90.219 并不在自己的直连网络中于是会丢弃这个包或尝试转发给错误的下一跳导致连接失败。解决方法删除 sdp-g-prod-router 上指向 10.0.0.5 的那条静态路由。访问FIP10.96.90.219的包的唯一下一跳就只有10.0.0.6, 然后将 包交给 OVN 的 Active Gateway Chassis (j701lwzr) 在 j701lwzr 上执行 DNAT10.96.90.219 - 10.0.0.177并通过内部网络送达 Amphora VM.这修复了去程流量的问题 但回程(amphora-client)依然有风险如果此时 Amphora 回复 SYN-ACK由于 sdp-g-prod-oam_mon-router 的网关端口依然可能和 Octavia 的网关端口不在同一个物理节点上OVN 依然有概率把回程包错误地封装进 Geneve 隧道发给 j701lwzx导致未做 SNAT 的裸包源 IP 为 10.0.0.177被发给客户端依然会被丢弃。为了让这条链路 100% 恢复正常你需要按顺序完成这两步第一步客户已准备做删除 sdp-g-prod-router 上指向 10.0.0.5 的静态路由。作用解决去程 ECMP 随机丢包。第二步你需要指导客户做将 sdp-g-prod-oam_mon-router 的网关端口强制绑定到与 Octavia LB 相同的 Active Gateway Chassis 上即 j701lwzr。也可以考虑ecmp_symmetric_reply参数:sudo ovn-nbctl set logical_router ROUTER-UUID options:ecmp_symmetric_replytrue状态记录当去程流量Client - Amphora通过 ECMP 的某一条路径比如下一跳 10.0.0.6进入路由器时OVN 会在本地j701lwzr的连接跟踪表conntrack中记录下这个会话的状态。强制对称当回程流量Amphora - Client产生时OVN 会去查 conntrack 表。一旦发现这是之前那个会话的回程包它会无视 ECMP 的负载均衡规则强制让回程包从去程包进来的那个接口/节点原路返回。Reference[1] https://zhuanlan.zhihu.com/p/627514565?utm_id0
