Hi,
I'm running OpenWrt 24.10.0 (flippy build, kernel 6.1.115) on a Radxa E24C board with RK3528 SoC and RTL8367B switch. I have a dual-WAN setup with mwan3 for load balancing, but throughput is capped at ~300 Mbps. I've done extensive testing and identified the root cause. I'm sharing my findings below in case they're useful for the community or if there's a solution I'm missing.
---
## 1. Hardware Setup
- **SoC:** Rockchip RK3528 (4× Cortex-A53 @ 2.0 GHz max)
- **Switch:** Realtek RTL8367B (4× GbE ports)
- **NIC driver:** stmmac (Synopsys DWMAC4/5, `st_gmac` module)
- **RAM:** 1 GB
- **Storage:** NVMe Intel 128 GB
### Switch Port Mapping (confirmed via testing)
| Port | VLAN | Interface | Role |
|------|------|-----------|------|
| 0 | VLAN 3 | eth0.3 → wan2 | WAN2 (second modem, 600 Mbps) |
| 1 | VLAN 1 | br-lan | LAN (PC connected) |
| 2 | VLAN 1 | br-lan | LAN (free) |
| 3 | VLAN 2 | eth0.2 → wan | WAN1 (main modem, 1000 Mbps) |
| 6 | — | CPU (internal) | SoC uplink |
### Current NIC Configuration
```
Driver: st_gmac (Synopsys DWMAC4/5)
RGMII link speed: 1000 Mbps (between stmmac and RTL8367B)
Queues: RX=1, TX=1 (single-queue hardware)
TSO: enabled in dmesg but tx_tso_frames=0 (not actually offloading)
RPS on eth0: 0xd (CPUs 0,2,3) ✓
RPS on eth0.2: 0x0 (disabled) ✗
RPS on eth0.3: 0x0 (disabled) ✗
Ring buffer: 1024 (max)
IRQ 55 (eth0): on CPU0, affinity 0-3
IRQ 56 (eth0): 0 interrupts (unused)
```
---
## 2. Software Configuration
- **mwan3:** Load balancing with weighted policy (5:3 ratio for 1000:600 Mbps links)
- **AdGuard Home:** DNS filtering on port 53
- **Firewall:** iptables (fw3), WAN REJECT policy
- **CPU governor:** schedutil (scales frequency with load)
- **Swap:** zram 512 MB (zstd) + swapfile 4 GB on NVMe
- **TCP:** BBR congestion control, fq_codel qdisc
---
## 3. The Problem
With both WANs active and mwan3 load balancing, throughput is capped at **~300 Mbps** aggregate. The expected combined throughput should be ~1600 Mbps (1000 + 600). Individual links can reach their full speed when tested alone (without mwan3), but with mwan3 active, the aggregate drops to ~300 Mbps.
---
## 4. Root Cause Analysis
### 4.1 Confirmed: Flow Offloading (SFE and xt_FLOWOFFLOAD) Breaks mwan3
I tested two different offloading mechanisms, and **both** caused mwan3 to enter "error" state:
**Test 1 — SFE (Shortcut Forwarding Engine):**
- Module: `kmod-shortcut-fe` + `kmod-shortcut-fe-cm` (compiled into kernel, `CONFIG_SHORTCUT_FE=y`)
- Result: mwan3 went to "error" state, network lost connectivity
- Had to disable SFE and restart mwan3 to restore
**Test 2 — xt_FLOWOFFLOAD (iptables flow offload):**
- Module: `xt_FLOWOFFLOAD` (loaded but inactive, `flow_offloading='0'`)
- Enabled via: `uci set firewall.@defaults[0].flow_offloading='1'` + firewall restart
- Result: mwan3 went to "error" state on both WAN interfaces
- Internet still worked on individual interfaces (ping via eth0.3 succeeded), but mwan3 tracking probes failed
- Had to restore firewall backup and restart mwan3 to restore
### 4.2 Why Flow Offloading Conflicts with mwan3
**mwan3** works by:
Marking packets in the `mangle` PREROUTING chain (`iptables -m mark --set-mark`)
Using `ip rule fwmark` for policy-based routing (selects WAN based on mark)
The routing decision determines the outgoing interface per-connection
**Flow offloading** (both SFE and xt_FLOWOFFLOAD) works by:
Caching the resolved route for established flows (including the output device)
Subsequent packets in the flow bypass conntrack, iptables, and routing — they're fast-pathed directly to the cached device
**The conflict:** Flow offloading bypasses the iptables mangle chain where mwan3 marks packets. Even though the first packet of a flow goes through the full path (mark → route → NAT), the flow offload entry caches the route. When mwan3's tracking probes (ICMP) are sent, they may be affected by the firewall restart or chain reordering, causing the tracking to fail.
Additionally, when the firewall is restarted (required to apply flow offloading changes), mwan3's iptables rules are temporarily removed and re-added. During this window, tracking probes fail and mwan3 marks the interfaces as "error".
### 4.3 Software NAT Bottleneck
Without any offloading, every packet goes through the full path:
conntrack lookup (hash-based, O(1))
iptables mangle (mwan3 mark processing)
Routing decision (policy routing via ip rule)
NAT rewriting (MASQUERADE/SNAT)
Forwarding to outgoing interface
On the RK3528 (Cortex-A53 @ 1.4–2.0 GHz), this path is CPU-bound. The single-queue NIC means the TX path (NAT rewriting) is single-threaded, limiting throughput to ~300 Mbps.
### 4.4 Additional Bottlenecks Found
| Issue | Status | Impact |
|-------|--------|--------|
| VLAN interfaces (eth0.2, eth0.3) have RPS=0 | Confirmed | RX processing not distributed across CPUs for VLAN traffic |
| TSO enabled but tx_tso_frames=0 | Confirmed | TCP segmentation not offloaded to NIC — CPU does all segmentation |
| RGMII link = 1 Gbps | Confirmed | Total aggregate throughput for all ports limited to 1 Gbps |
| CPU at 1416 MHz (not max 2016 MHz) | Confirmed | schedutil scales down — may not ramp up fast enough during burst |
| Single queue (RX=1, TX=1) | Confirmed | No multi-queue parallelism |
---
## 5. What Was NOT Tested (Due to Risk)
- **nftables flowtable** (`CONFIG_NFT_FLOW_OFFLOAD=m`): Likely has the same conflict as xt_FLOWOFFLOAD since both use the nf_flow_table infrastructure
- **Hardware NAT** (`flow_offloading_hw='1'`): RK3528 does not have a hardware packet processing engine (PPE), so this would fall back to software anyway
- **Bonding (LACP/LACP):** Requires ISP-side support, not applicable for two different ISPs
---
## 6. Questions for Support
**Is there a way to make flow offloading work with mwan3?** Specifically, can we configure xt_FLOWOFFLOAD or nftables flowtable to only offload AFTER mwan3 has marked and routed the packet, without interfering with tracking probes?
**Can the stmmac driver be configured for multi-queue?** The hardware reports `RX: 1, TX: 1` max queues. Is there a way to enable more DMA channels on RK3528?
**Why is TSO not working?** TSO is enabled in dmesg (`TSO feature enabled`) but `tx_tso_frames=0`. Is there a driver or firmware issue?
**Is there a way to increase the RGMII link speed?** The current link is 1 Gbps. Some RK3528 variants support RGMII at 2.5 Gbps. Is this configurable via device tree?
**Are there any kernel patches or driver modifications** that could improve single-queue NAT throughput on this hardware?
---
## 7. Current Working Configuration
The system is currently stable with:
- Flow offloading: **disabled** (`flow_offloading='0'`)
- SFE: **disabled** (module loaded but not active)
- mwan3: **active** (load balancing 5:3)
- Throughput: **~300 Mbps** aggregate
- Both WANs functional with failover
---
## 8. Summary
| Metric | Value |
|--------|-------|
| Hardware | RK3528 + RTL8367B |
| OpenWrt | 24.10.0 (flippy), kernel 6.1.115 |
| NIC driver | stmmac (single queue) |
| RGMII speed | 1 Gbps |
| Flow offloading | Confirmed broken with mwan3 (both SFE and xt_FLOWOFFLOAD) |
| Current throughput | ~300 Mbps aggregate |
| Expected throughput | ~1600 Mbps (1000 + 600) |
I appreciate any guidance or suggestions. Thank you for your time.
Best regards,