[dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

[dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

[Wu, Xiaoban]

Dear All,


I have two Dell PowerEdge R740 servers A and B running with ubuntu 16.04, each one has a Mellanox MCX556A-ECAT NIC installed on the PCIe x16 slot. And, the two NICs are directly connected back to back with a copper cable.


On server A, it runs a RX program which runs a function to process and analyze the received packets. On server B, it runs a packet-gen program which generates and sends packets out. They are both compiled with dpdk-17.11


Now the interesting thing is that on server B, it is kind of smart and considerate enough to automatically adjust its TX throughput according to how fast server A processes the packets. If server A processes the packets faster, then server B sends packets with a higher throughput. Similarly, if server A processes the packets slower, then server B sends packets with a lower throughput. Please note that once the program on server B starts, it is never interrupted by any way.


However, I think the server B should send out the packets with a constant throughput no matter how fast server A processes the packets.


Does anybody else notice this interesting behavior of MLX5 driver? Can anybody help me disable this feature permanently? Thanks very much for your help.


Best wishes,

Xiaoban

Re: [dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

[Stephen Hemminger]

On Thu, 26 Apr 2018 22:26:36 +0000
"Wu, Xiaoban" <Xiaoban_Wu@student.uml.edu> wrote:

> Dear All,
> 
> 
> I have two Dell PowerEdge R740 servers A and B running with ubuntu 16.04, each one has a Mellanox MCX556A-ECAT NIC installed on the PCIe x16 slot. And, the two NICs are directly connected back to back with a copper cable.
> 
> 
> On server A, it runs a RX program which runs a function to process and analyze the received packets. On server B, it runs a packet-gen program which generates and sends packets out. They are both compiled with dpdk-17.11
> 
> 
> Now the interesting thing is that on server B, it is kind of smart and considerate enough to automatically adjust its TX throughput according to how fast server A processes the packets. If server A processes the packets faster, then server B sends packets with a higher throughput. Similarly, if server A processes the packets slower, then server B sends packets with a lower throughput. Please note that once the program on server B starts, it is never interrupted by any way.
> 
> 
> However, I think the server B should send out the packets with a constant throughput no matter how fast server A processes the packets.
> 
> 
> Does anybody else notice this interesting behavior of MLX5 driver? Can anybody help me disable this feature permanently? Thanks very much for your help.
> 
> 
> Best wishes,
> 
> Xiaoban

Turn off Ethernet flow control.  It was a design mistake that NIC vendors always seem to like because it gives 0 loss benchmarks.

Re: [dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

[Wu, Xiaoban]

Hi Stephen,


Thanks very much for your help. Though I don't know if it is a overkill, the following does the job.


/sbin/ethtool -s enp59s0f0 autoneg off
/sbin/ethtool -A enp59s0f0 rx off
/sbin/ethtool -A enp59s0f0 tx off


Best wishes,

Xiaoban

________________________________
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Thursday, April 26, 2018 6:40:55 PM
To: Wu, Xiaoban
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

On Thu, 26 Apr 2018 22:26:36 +0000
"Wu, Xiaoban" <Xiaoban_Wu@student.uml.edu> wrote:

> Dear All,
>
>
> I have two Dell PowerEdge R740 servers A and B running with ubuntu 16.04, each one has a Mellanox MCX556A-ECAT NIC installed on the PCIe x16 slot. And, the two NICs are directly connected back to back with a copper cable.
>
>
> On server A, it runs a RX program which runs a function to process and analyze the received packets. On server B, it runs a packet-gen program which generates and sends packets out. They are both compiled with dpdk-17.11
>
>
> Now the interesting thing is that on server B, it is kind of smart and considerate enough to automatically adjust its TX throughput according to how fast server A processes the packets. If server A processes the packets faster, then server B sends packets with a higher throughput. Similarly, if server A processes the packets slower, then server B sends packets with a lower throughput. Please note that once the program on server B starts, it is never interrupted by any way.
>
>
> However, I think the server B should send out the packets with a constant throughput no matter how fast server A processes the packets.
>
>
> Does anybody else notice this interesting behavior of MLX5 driver? Can anybody help me disable this feature permanently? Thanks very much for your help.
>
>
> Best wishes,
>
> Xiaoban

Turn off Ethernet flow control.  It was a design mistake that NIC vendors always seem to like because it gives 0 loss benchmarks.

Re: [dpdk-users] Interesting considerate but annoying behavior of MLX5 driver

[Adrien Mazarguil]

On Fri, Apr 27, 2018 at 01:29:38AM +0000, Wu, Xiaoban wrote:
> Hi Stephen,
> 
> 
> Thanks very much for your help. Though I don't know if it is a overkill, the following does the job.
> 
> 
> /sbin/ethtool -s enp59s0f0 autoneg off
> /sbin/ethtool -A enp59s0f0 rx off
> /sbin/ethtool -A enp59s0f0 tx off

Note that DPDK applications should rely on rte_eth_dev_flow_ctrl_get/set()
with fc_conf->mode = RTE_FC_NONE to achieve the same behavior in a generic
fashion.

While doing so translates exactly to the above ethtool calls with mlx4 and
mlx5, other PMDs do not typically leave a kernel netdevice around for
external applications to play with.

-- 
Adrien Mazarguil
6WIND