Nazeem.Me

A blog about technology, football and all the other random stuff in my life

Three Tunnels, One Ceiling

Written by

in

·

Having just found that my router’s VPN was carrying only half my traffic, the obvious next question was how fast the half it did carry actually went. What I expected was a straightforward protocol comparison. What I got was four wrong conclusions in a row, each one caused by a measurement that looked reasonable and wasn’t.

The numbers are below, and so is the method — including the parts of the method that failed, because those turned out to be the useful bit.

What was being compared

Three ways to run the same commercial VPN, plus an untunnelled control:

PathWhere encryption happensCovers
Router OpenVPNuserspace daemon on the router’s ARM CPUany device, per-device policy
Router WireGuardin-kernel on the router’s ARM CPUany device, per-device policy
Client WireGuardin-kernel on the desktop’s Ryzen 7 7800X3Dthat one machine
Directnowhere — control

Method

Every run used the same endpoint, the same file, the same client and the same command. What changed between runs was one routing decision.

  1. One test host, on the fast NIC. A desktop with a 10 GbE interface carrying the IPv4 default route. The LAN behind it measures 9.4 Gbps, so the cabling could not be the limit.
  2. Force the address family. Every command is curl -4. This is not optional — see the failures below.
  3. Prove the path before measuring it. Before each run: check the routing policy on the router, then ask an external service what address it sees. Each of the four paths has a distinct exit address, so the reply is unambiguous proof of which tunnel carried the request.
  4. Single stream, then four in parallel. One stream measures latency-bound behaviour; parallel streams find the ceiling and expose how a path degrades under concurrency.
  5. Sample CPU across the transfer. Read /proc/stat before and after and compute busy time. This is what separates “this hardware is too slow” from “something else is the limit” — the single most valuable measurement here.
  6. Keep an untunnelled control. Without it a tunnel number is meaningless; you cannot tell a slow tunnel from a slow line.
  7. Load two paths at once. If two tunnels running simultaneously carry more in total than either does alone, the constraint is per-tunnel — not the hardware, and not the account.

Results

The control is the whole story. Every tunnel is a rounding error against the untunnelled line — at best you keep 9% of the connection.

Throughput by path Four parallel streams · IPv4 forced · same endpoint · Mbps 0 3000 6000 Mbps OpenVPN 220 Router WG 739 Client WG 532 Direct 8474
The tunnels, on their own scale Single stream vs four parallel · Mbps 0 200 400 600 Mbps OpenVPN 158 220 Client WG 439 532 Router WG 558 739 Router OpenVPN Router WireGuard Client WireGuard pale bar = single stream
PathSingle stream4 parallelShare of directCPU during transfer
Router OpenVPN158 Mbps220 Mbps2.6%61.6% of 4 ARM cores
Router WireGuard558 Mbps739 Mbps8.7%61.0% of 4 ARM cores
Client WireGuard439 Mbps532 Mbps6.3%6.4% of 16 threads
Direct (control)3297 Mbps8474 Mbps100%
WireGuard on the router beats OpenVPN by 3.4×. The client-side tunnel — on a CPU roughly fifty times faster at this cipher — comes third.

How each path behaves under concurrency

Averages hide the more interesting difference. Here is what each of the four parallel streams actually got:

Per-stream throughput, four streams at once Router OpenVPN vs router WireGuard · Mbps per stream 0 50 100 150 200 stream 1 98 215 stream 2 44 157 stream 3 39 183 stream 4 39 185 Router OpenVPN Router WireGuard
StreamRouter OpenVPNRouter WireGuard
198 Mbps215 Mbps
244 Mbps157 Mbps
339 Mbps183 Mbps
439 Mbps185 Mbps
Total220 Mbps739 Mbps
OpenVPN doesn’t share: the first stream takes what it can and the rest starve. WireGuard holds all four within a normal spread. Per-stream figures are rounded, so they sum to slightly more than the measured total.

That shape — one stream dominating while the others collapse — is what a single-threaded encryptor running out of one core looks like.

The result that broke the theory

Client-side WireGuard should have won easily. The desktop benchmarks 38.7 Gbps of ChaCha20-Poly1305 on a single core — the cipher WireGuard actually uses — against a router doing 739 Mbps across four ARM cores. I predicted 3–6 Gbps.

It came third, at 532 Mbps. And it did so using 6.4% of sixteen threads, roughly one core, almost entirely idle.

A component that is 94% idle is not the bottleneck, no matter how slow the result is.

That single number invalidated the whole framing. If encryption were the constraint, the fast machine would have won and its CPU would have been busy. Neither happened. So the limit had to be somewhere neither CPU could see — which is what the last method step was for.

The decisive test: run both tunnels at once

Running simultaneouslyThroughput
Router WireGuard842 Mbps
Client WireGuard446 Mbps
Combined1288 Mbps
More than either tunnel managed alone — and more than the 739 Mbps I had already recorded as the router’s ceiling.

Two tunnels aggregate. So it is not an account-wide cap, and it is not the router’s CPU — that same router had just been called saturated at a number it exceeded here while doing other work. What remains is the provider’s per-connection capacity: roughly 500–850 Mbps per tunnel, varying by which endpoint you land on.

Four wrong answers, and what caused each

Every one of these was measured, written down and believed before the next test overturned it.

What I concludedWhy it was wrong
“Throughput is a wash between the protocols”
Both measured ~1 Gbps.
Tested with curl running on the router. Locally-originated traffic bypasses the hardware flow accelerator, so the router’s own TCP stack capped everything at ~1 Gbps and hid a 3.4× difference.
“The ISP isn’t delivering the plan”
Everything capped at ~1.07 Gbps.
The test hostname resolved AAAA-first. IPv6 was tunnelled and, on that host, pinned to a 1 GbE NIC. The “direct control” was neither direct nor on the fast interface. Forcing -4: 8474 Mbps.
“The router can’t encrypt at line rate”
61% of 4 cores for 739 Mbps.
A CPU percentage averaged across all cores hid that nothing was saturated. The same router later sustained 842 Mbps concurrently with other load.
“A faster CPU will fix it”
Predicted 3–6 Gbps client-side.
Reasoned from a component benchmark instead of measuring the system. The cipher was never the constraint, so 50× the crypto throughput bought nothing.

There is one shape common to all four. Each came from measuring a component — the router’s own stack, one address family, a CPU average, a cipher benchmark — and reporting it as a fact about the system. The fix each time was the same: change one variable, prove the path, and keep a control.

What I actually concluded

Move off OpenVPN. On identical hardware to the identical provider, WireGuard is 3.4× faster and degrades gracefully under load where OpenVPN starves its own streams. This one is a real hardware limit: a single-threaded userspace daemon on a router CPU.

Don’t move the tunnel to the client. It was slower, needs its own config and its own kill switch, and covers exactly one machine. There is no version of this that pays off while the provider is the constraint.

Accept the tunnel tax, or split your traffic. 8.5 Gbps direct against 739 through the best tunnel is not a tuning problem. The only lever that moves it is more tunnels, since they aggregate — and that is a lot of machinery for a household. The honest alternative is per-device policy: tunnel what needs privacy, leave what needs speed on the fast path, and know which is which.

Measure CPU next to throughput, always. It is one extra read of /proc/stat and it is the only cheap way to tell a slow component from a slow supplier. Every wrong answer above would have been caught earlier by taking it seriously.


Test host: Ryzen 7 7800X3D, 10 GbE, IPv4 default route. Router: ASUS RT-BE88U, Asuswrt-Merlin 3006.102.8_4, quad-core ARM. Provider: hide.me, Endpoint: a 1 GB file from a local host, plus independent endpoints for cross-checks. All figures are single measurements taken back to back in one session, not averaged over repeated trials — treat them as ratios rather than precise constants.