My testing setup is, all on the same subnet, ipv4
- Windows Machine with Intel x520
-
- Direct Connect 10Gbps cable
- USW Aggregation switch (10Gbps)
-
- Direct Connect 10Gbps cable
- Synology NAS with Intel x520
-
- SRVIO Connection
- Debian VM
iperf3 Windows To Debian: 6Gbits/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output
Connecting to host 192.168.11.57, port 5201
[ 5] local 192.168.11.132 port 56855 connected to 192.168.11.57 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 817 MBytes 6.79 Gbits/sec
[ 5] 1.01-2.00 sec 806 MBytes 6.82 Gbits/sec
[ 5] 2.00-3.01 sec 822 MBytes 6.85 Gbits/sec
[ 5] 3.01-4.00 sec 805 MBytes 6.81 Gbits/sec
[ 5] 4.00-5.01 sec 818 MBytes 6.81 Gbits/sec
[ 5] 5.01-6.00 sec 806 MBytes 6.82 Gbits/sec
[ 5] 6.00-7.01 sec 821 MBytes 6.83 Gbits/sec
[ 5] 7.01-8.00 sec 805 MBytes 6.80 Gbits/sec
[ 5] 8.00-9.01 sec 820 MBytes 6.82 Gbits/sec
[ 5] 9.01-10.00 sec 809 MBytes 6.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec sender
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec receiver
Server output:
-----------------------------------------------------------
Server listening on 5201 (test #9)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56854
[ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56855
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 806 MBytes 6.76 Gbits/sec
[ 5] 1.00-2.00 sec 812 MBytes 6.82 Gbits/sec
[ 5] 2.00-3.00 sec 816 MBytes 6.85 Gbits/sec
[ 5] 3.00-4.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 4.00-5.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 5.00-6.00 sec 812 MBytes 6.81 Gbits/sec
[ 5] 6.00-7.00 sec 815 MBytes 6.84 Gbits/sec
[ 5] 7.00-8.00 sec 811 MBytes 6.80 Gbits/sec
[ 5] 8.00-9.00 sec 814 MBytes 6.82 Gbits/sec
[ 5] 9.00-10.00 sec 815 MBytes 6.84 Gbits/sec
[ 5] 10.00-10.00 sec 1.12 MBytes 4.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 7.94 GBytes 6.82 Gbits/sec receiver
iperf Done.
iperf3 debian to windows 9.5Gbits/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output -R
Connecting to host 192.168.11.57, port 5201
Reverse mode, remote host 192.168.11.57 is sending
[ 5] local 192.168.11.132 port 56845 connected to 192.168.11.57 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 1.11 GBytes 9.40 Gbits/sec
[ 5] 1.01-2.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 2.00-3.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 3.01-4.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 4.00-5.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 5.01-6.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 6.00-7.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 7.01-8.00 sec 1.09 GBytes 9.47 Gbits/sec
[ 5] 8.00-9.01 sec 1.11 GBytes 9.47 Gbits/sec
[ 5] 9.01-10.00 sec 1.09 GBytes 9.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec receiver
Server output:
-----------------------------------------------------------
Server listening on 5201 (test #7)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56844
[ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56845
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 2.01 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.47 Gbits/sec 0 2.01 MBytes
[ 5] 10.00-10.00 sec 2.50 MBytes 7.72 Gbits/sec 0 2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.46 Gbits/sec 0 sender
iperf Done.
I find that rather curious, something in the windows 10 tcp settings that limit the outgoing throughput, window size maybe?
Debian MTU
ip link show ens3 | grep -i "mtu"
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
Windows MTU
netsh interface ipv4 show subinterfaces
MTU MediaSenseState Bytes In Bytes Out Interface
------ --------------- --------- --------- -------------
1500 1 273580016987 64376522487 Ethernet 4
You must log in or register to comment.
Does 10Gb needs that much cooling?
No, the photo is of a general computer. So most of the cooling is for the CPU and GPU.
SFP modules can get quite hot, especially if they’re pushing high frequencies over copper. So you’ll see some people complaining about their 10 gigabit copper SFP modules overheating. Especially on passively cooled network equipment.
Fiber optics don’t get nearly as hot, and don’t have overheating issues. Direct connect cables also do not get hot.
I’d much rather have all of my networking equipment with fiber optics, then copper. The trade-off is it’s harder to splice your own fiber optic cable and put termination on it, the benefit is the equipment is much much cheaper, More reliable, cooler, goes further distance… A fiber optic SFP module cost about $10 for one, but a copper SFP module is like $100.
Thanks for the clarification ! It wasn’t clear to me what I was looking at first sight. I fought that was some self-hosted/DIY 10Gbs router/switch. Sorry if my question came the wrong way or was a stupid one. I’m just a curious guy !
Thanks for the additional information !!
Use the -P flag to set the number of parallel streams. Probably, TCP windows size is limiting you on a single TCP connection but multiple connections would work fine.
That is a great idea to test!
iperf windows to debian -P4 6Gbit/sec
.\iperf3.exe -c 192.168.11.57 --get-server-output --dont-fragmen -P 4 Connecting to host 192.168.11.57, port 5201 [ 5] local 192.168.11.132 port 56910 connected to 192.168.11.57 port 5201 [ 7] local 192.168.11.132 port 56911 connected to 192.168.11.57 port 5201 [ 9] local 192.168.11.132 port 56912 connected to 192.168.11.57 port 5201 [ 11] local 192.168.11.132 port 56913 connected to 192.168.11.57 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 7] 0.00-1.00 sec 192 MBytes 1.60 Gbits/sec [ 9] 0.00-1.00 sec 192 MBytes 1.60 Gbits/sec [ 11] 0.00-1.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 0.00-1.00 sec 764 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 7] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 9] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [ 11] 1.00-2.01 sec 190 MBytes 1.58 Gbits/sec [SUM] 1.00-2.01 sec 760 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.01-3.00 sec 186 MBytes 1.58 Gbits/sec [ 7] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [ 9] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [ 11] 2.01-3.00 sec 189 MBytes 1.60 Gbits/sec [SUM] 2.01-3.00 sec 754 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 7] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 9] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [ 11] 3.00-4.01 sec 190 MBytes 1.58 Gbits/sec [SUM] 3.00-4.01 sec 761 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 9] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 4.01-5.00 sec 188 MBytes 1.59 Gbits/sec [SUM] 4.01-5.00 sec 754 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 7] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 5.00-6.01 sec 191 MBytes 1.59 Gbits/sec [ 11] 5.00-6.01 sec 189 MBytes 1.58 Gbits/sec [SUM] 5.00-6.01 sec 762 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.01-7.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 6.01-7.00 sec 189 MBytes 1.59 Gbits/sec [ 9] 6.01-7.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 6.01-7.00 sec 189 MBytes 1.60 Gbits/sec [SUM] 6.01-7.00 sec 754 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.01 sec 192 MBytes 1.59 Gbits/sec [ 7] 7.00-8.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 7.00-8.01 sec 192 MBytes 1.59 Gbits/sec [ 11] 7.00-8.01 sec 191 MBytes 1.59 Gbits/sec [SUM] 7.00-8.01 sec 766 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 7] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 9] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [ 11] 8.01-9.00 sec 188 MBytes 1.59 Gbits/sec [SUM] 8.01-9.00 sec 752 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 7] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 9] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [ 11] 9.00-10.01 sec 191 MBytes 1.59 Gbits/sec [SUM] 9.00-10.01 sec 764 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec sender [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 7] 0.00-10.01 sec 1.86 GBytes 1.59 Gbits/sec sender [ 7] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 9] 0.00-10.01 sec 1.86 GBytes 1.59 Gbits/sec sender [ 9] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 11] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec sender [ 11] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [SUM] 0.00-10.01 sec 7.41 GBytes 6.36 Gbits/sec sender [SUM] 0.00-10.01 sec 7.41 GBytes 6.35 Gbits/sec receiver Server output: ----------------------------------------------------------- Server listening on 5201 (test #15) ----------------------------------------------------------- Accepted connection from 192.168.11.132, port 56909 [ 5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56910 [ 8] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56911 [ 10] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56912 [ 12] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56913 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 188 MBytes 1.57 Gbits/sec [ 8] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 0.00-1.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 0.00-1.00 sec 187 MBytes 1.57 Gbits/sec [SUM] 0.00-1.00 sec 754 MBytes 6.32 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 8] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 10] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [ 12] 1.00-2.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 1.00-2.00 sec 754 MBytes 6.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 188 MBytes 1.58 Gbits/sec [ 8] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [ 10] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [ 12] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [SUM] 2.00-3.00 sec 760 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 8] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 10] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [ 12] 3.00-4.00 sec 189 MBytes 1.58 Gbits/sec [SUM] 3.00-4.00 sec 755 MBytes 6.34 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 4.00-5.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 4.00-5.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 5.00-6.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 5.00-6.00 sec 188 MBytes 1.58 Gbits/sec [SUM] 5.00-6.00 sec 758 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 6.00-7.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 6.00-7.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 7.00-8.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 7.00-8.00 sec 760 MBytes 6.38 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 8] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 10] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [ 12] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [SUM] 8.00-9.00 sec 758 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 10] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [ 12] 9.00-10.00 sec 190 MBytes 1.59 Gbits/sec [SUM] 9.00-10.00 sec 759 MBytes 6.37 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 8] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 10] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [ 12] 10.00-10.01 sec 2.50 MBytes 1.51 Gbits/sec [SUM] 10.00-10.01 sec 10.0 MBytes 6.03 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 8] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 10] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [ 12] 0.00-10.01 sec 1.85 GBytes 1.59 Gbits/sec receiver [SUM] 0.00-10.01 sec 7.41 GBytes 6.35 Gbits/sec receiver iperf Done.Though curiously i see the same behavior, 9.5Gbit linux->windows, and 5Gbit windows->linux… i’m perplexed
deleted by creator
Same results with -P but without the “don’t fragment”?
Yes, exactly the same results.
I’m thinking it might be on the linux side? i’m double checking my debian network stack now
5 Gbps is the theoretical PCIe 1x limit, but I haven’t seen those in a while, so probably not that.
I’m… half tempted to ask you if you can reverse the cables in the path lol.
Maybe a pair somewhere isn’t as good as the others and affects data rate in that direction.
Far fetched, but…I like the way you think! This is good engineering
I replaced both direct connect cables with new ones, I used new switch ports.
The asymmetry persists. Linux to windows 9.5GiB/s, Windows to Linux 6.5GiB/s
now we know it probably isn’t the cables!
