We deployed a greenfield VCF 9.1 fleet with the new Distributed Transit Gateway + VNA model (the "no edge cluster needed" VPC architecture). Everything works — hosts, overlay, edges, BGP, existing VPCs, SNAT — except the VNA cluster's datapath never comes alive, which breaks creating new VPCs (HTTP 500) and therefore Supervisor/VKS.
After a very long debugging session I can honestly say this is the most thoroughly diagnosed problem I've ever not fixed. The short version:
The smoking gun:
- The VNA's HA/BFD session (fp-eth0, link-local 169.254.x, VLAN encap on transport VLAN 72) stays DOWN forever with
Diag: Concatenated Path Down (from NSXA) and Remote_discr: 0 — zero packets ever exchanged.
pktcap-uw on all four fastpath vNIC switchports, both directions: zero frames. Not even ARP. The appliance simply never transmits on the fastpath.
- Meanwhile the management-plane BFD between the same two appliances is UP (captured it: BFDv1 Multihop, State Up) — it's the session carrying the "concat path down" diag to the peer. So BFD stack, appliances and network are all fine.
- The fp vNICs attach to NSX ports with
attachment.type: PARENT + com.vmware.port.extraConfig.vpc.serviceport = parent. Child VIFs are never created (/api/v1/fabric/vifs shows mgmt + 4 parents, nothing else).
- Appliance accepts its HostConfig with
error_code: 0 (vtep, transport_vlan 72, MTU 9000 all correct), NSXA marks fp-eth0 UP, adds the HA filter… and then datapathd just holds BFD down.
Why I'm sure it's not the network:
We deployed a second, completely fresh VNA cluster — new nodes, static mgmt IPs — and got byte-for-byte identical behavior. Even better: DRS put both new nodes on the same ESXi host, so the HA traffic never needs to touch a NIC, cable or switch — still dead. On top of that we verified/eliminated: trunk/VLAN config, jumbo MTU end-to-end (8972 DF vmkping passes), no ip redirects on the SVIs, NIC firmware (BCM5741x updated past the GENEVE checksum bug), ENS_INTERRUPT host switch mode, SpoofGuard/DFW (default ALLOW + exclusion list), IP block exhaustion, licensing (valid VCF Solution license), a degraded NSX manager cluster (found + fixed mid-hunt, 3/3 STABLE now), VCF Automation inventory sync, host config resync, node redeploys, and the "Activate NSX on DVPGs" step from KB 417250 (was indeed not activated — activating it changed nothing).
Regular NSX edges on the same VDS and fabric have all tunnels UP. Host↔host GENEVE is UP. It's exclusively the VNA serviceport datapath.
Environment: NSX 9.1.0.0200.25524173, ESX/vCenter 9.1.0, SDDC Manager 9.1.0.0400, physical Dell hosts (not nested), VDS ENS_INTERRUPT, Nexus 9K vPC ToR.
My questions:
- Anyone actually running DTGW+VNA on 9.1.0.0200 on physical hosts? Does your
get bfd-sessions show the VLAN-encap session UP, and do you see child VIFs on the serviceports?
- Does anyone know what NSXA's "concatenated path" validation actually checks before the fastpath is allowed to transmit?
- Is there a newer NSX 9.1.x build with a fix? Our depot tops out at 9.1.0.0200.
No support entitlement on this environment (lab/POC), so community wisdom is my best hope. Happy to share the full evidence bundle (API dumps, pcaps, appliance syslogs, host logs). And yes — the irony of the appliance-that-removes-the-need-for-edge-clusters being the only broken thing is not lost on me.