[dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

[dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

[Nitin Katiyar]

Hi,
While testing MTU configuration of physical ports using OVS-DPDK we have found that Fortville and Niantic behaves differently for Tagged packets. Both allows TX of packets with size up to programmed MTU value but in receive direction Fortville drops packets of size equal to configured MTU. Additionally, Fortville does not report any error/drop counter if packets with size more than configured MTU (max frame size) are received. In Niantic we can see error counters getting incremented if packets of size more than MTU are received.

When ports are started, device attribute max_rx_pkt_len is set during device/queue init by application (OVS in our case) and this max_rx_pkt_len is used to program hardware register in device which in turn determines the maximum size of packet/frame that it can receive.
What we have found during testing is that Niantic could receive tagged/untagged packets of size equal to max_rx_pkt_len but Fortville could only receive tagged packets (single tag) up to size <= (max_rx_pkt_len - 4). Datasheet of Niantic mentions that device implicitly accounts for  VLAN tag(s) in addition to Maximum Frame size programmed which is not the case for Fortville. This causes issue with MTU settings and maximum frame size that NIC can receive with tagged and untagged traffic.
We have tested it with OVS-DPDK where it uses device attribute max_rx_pkt_len to set max frame size in accordance with the configured MTU size of port. However, Ixgbe (Niantic) and i40e (Fortville) interpret it differently. I looked at some other PMD drivers and different drivers interpret dev_conf.rxmode.max_rx_pkt_len differently i.e. some adds one or two VLAN, few don't include it and some use this field differently. It creates issue with MTU while running same application on different NICs and PMD drivers need to be fixed to have consistent behavior with MTU/Max Frame Size settings.

Regards,
Nitin Katiyar

Re: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

[Stephen Hemminger]

On Tue, 24 Oct 2017 12:25:38 +0000
Nitin Katiyar <nitin.katiyar@ericsson.com> wrote:

> Hi,
> While testing MTU configuration of physical ports using OVS-DPDK we have found that Fortville and Niantic behaves differently for Tagged packets. Both allows TX of packets with size up to programmed MTU value but in receive direction Fortville drops packets of size equal to configured MTU. Additionally, Fortville does not report any error/drop counter if packets with size more than configured MTU (max frame size) are received. In Niantic we can see error counters getting incremented if packets of size more than MTU are received.
> 
> When ports are started, device attribute max_rx_pkt_len is set during device/queue init by application (OVS in our case) and this max_rx_pkt_len is used to program hardware register in device which in turn determines the maximum size of packet/frame that it can receive.
> What we have found during testing is that Niantic could receive tagged/untagged packets of size equal to max_rx_pkt_len but Fortville could only receive tagged packets (single tag) up to size <= (max_rx_pkt_len - 4). Datasheet of Niantic mentions that device implicitly accounts for  VLAN tag(s) in addition to Maximum Frame size programmed which is not the case for Fortville. This causes issue with MTU settings and maximum frame size that NIC can receive with tagged and untagged traffic.
> We have tested it with OVS-DPDK where it uses device attribute max_rx_pkt_len to set max frame size in accordance with the configured MTU size of port. However, Ixgbe (Niantic) and i40e (Fortville) interpret it differently. I looked at some other PMD drivers and different drivers interpret dev_conf.rxmode.max_rx_pkt_len differently i.e. some adds one or two VLAN, few don't include it and some use this field differently. It creates issue with MTU while running same application on different NICs and PMD drivers need to be fixed to have consistent behavior with MTU/Max Frame Size settings.
> 
> Regards,
> Nitin Katiyar

MTU on most operating systems is advisory to device driver.
The device driver may receive packets up to that MTU or larger.
How much bigger depends on how the hardware receive packet limit is implemented.
Same thing happens in Linux and BSD.

Re: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

[Nitin Katiyar]

Thanks Stephen for reply. I agree that it is advisory for device driver. I am also reporting the issue with PMD dpdk drivers. We see different behavior with 2 NICs which we have tested with OVS-DPDK (i.e. Fortville and Niantic). And by looking at other PMD driver, there could be issue with other NICs also. If you look at *_dev_mtu_set then you can see that some driver adds Vlan tag(s) while determining max_rx_pkt_len (which is MTU + ETH_HDR (14 bytes) + CRC(4 bytes) + VLAN (4 or 8 bytes) and some don't do that. But during *_rx_init() driver uses only max_rx_pkt_len (which does not have VLAN added into it which means MTU + ETH_HDR(14 bytes) + CRC (4bytes)) is used. OVS doesn't use *_dev_mtu_set(). 

So, there is inconsistency with the maximum size tagged packet that different device can receive due to consideration of Vlan tag size in configuring hw register for Max Frame Size. This should be addressed by drivers not the application.

Regards,
Nitin

-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org] 
Sent: Tuesday, October 24, 2017 6:31 PM
To: Nitin Katiyar <nitin.katiyar@ericsson.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

On Tue, 24 Oct 2017 12:25:38 +0000
Nitin Katiyar <nitin.katiyar@ericsson.com> wrote:

> Hi,
> While testing MTU configuration of physical ports using OVS-DPDK we have found that Fortville and Niantic behaves differently for Tagged packets. Both allows TX of packets with size up to programmed MTU value but in receive direction Fortville drops packets of size equal to configured MTU. Additionally, Fortville does not report any error/drop counter if packets with size more than configured MTU (max frame size) are received. In Niantic we can see error counters getting incremented if packets of size more than MTU are received.
> 
> When ports are started, device attribute max_rx_pkt_len is set during device/queue init by application (OVS in our case) and this max_rx_pkt_len is used to program hardware register in device which in turn determines the maximum size of packet/frame that it can receive.
> What we have found during testing is that Niantic could receive tagged/untagged packets of size equal to max_rx_pkt_len but Fortville could only receive tagged packets (single tag) up to size <= (max_rx_pkt_len - 4). Datasheet of Niantic mentions that device implicitly accounts for  VLAN tag(s) in addition to Maximum Frame size programmed which is not the case for Fortville. This causes issue with MTU settings and maximum frame size that NIC can receive with tagged and untagged traffic.
> We have tested it with OVS-DPDK where it uses device attribute max_rx_pkt_len to set max frame size in accordance with the configured MTU size of port. However, Ixgbe (Niantic) and i40e (Fortville) interpret it differently. I looked at some other PMD drivers and different drivers interpret dev_conf.rxmode.max_rx_pkt_len differently i.e. some adds one or two VLAN, few don't include it and some use this field differently. It creates issue with MTU while running same application on different NICs and PMD drivers need to be fixed to have consistent behavior with MTU/Max Frame Size settings.
> 
> Regards,
> Nitin Katiyar

MTU on most operating systems is advisory to device driver.
The device driver may receive packets up to that MTU or larger.
How much bigger depends on how the hardware receive packet limit is implemented.
Same thing happens in Linux and BSD.

Re: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

[Nitin Katiyar]

Hi,
Does anyone have any inputs/suggestion on this issue?

Regards,
Nitin

-----Original Message-----
From: Nitin Katiyar 
Sent: Wednesday, October 25, 2017 11:57 AM
To: 'Stephen Hemminger' <stephen@networkplumber.org>
Cc: dev@dpdk.org
Subject: RE: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

Thanks Stephen for reply. I agree that it is advisory for device driver. I am also reporting the issue with PMD dpdk drivers. We see different behavior with 2 NICs which we have tested with OVS-DPDK (i.e. Fortville and Niantic). And by looking at other PMD driver, there could be issue with other NICs also. If you look at *_dev_mtu_set then you can see that some driver adds Vlan tag(s) while determining max_rx_pkt_len (which is MTU + ETH_HDR (14 bytes) + CRC(4 bytes) + VLAN (4 or 8 bytes) and some don't do that. But during *_rx_init() driver uses only max_rx_pkt_len (which does not have VLAN added into it which means MTU + ETH_HDR(14 bytes) + CRC (4bytes)) is used. OVS doesn't use *_dev_mtu_set(). 

So, there is inconsistency with the maximum size tagged packet that different device can receive due to consideration of Vlan tag size in configuring hw register for Max Frame Size. This should be addressed by drivers not the application.

Regards,
Nitin

-----Original Message-----
From: Stephen Hemminger [mailto:stephen@networkplumber.org] 
Sent: Tuesday, October 24, 2017 6:31 PM
To: Nitin Katiyar <nitin.katiyar@ericsson.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Issue with MTU/max_rx_pkt_len handling by different NICs/PMD drivers

On Tue, 24 Oct 2017 12:25:38 +0000
Nitin Katiyar <nitin.katiyar@ericsson.com> wrote:

> Hi,
> While testing MTU configuration of physical ports using OVS-DPDK we have found that Fortville and Niantic behaves differently for Tagged packets. Both allows TX of packets with size up to programmed MTU value but in receive direction Fortville drops packets of size equal to configured MTU. Additionally, Fortville does not report any error/drop counter if packets with size more than configured MTU (max frame size) are received. In Niantic we can see error counters getting incremented if packets of size more than MTU are received.
> 
> When ports are started, device attribute max_rx_pkt_len is set during device/queue init by application (OVS in our case) and this max_rx_pkt_len is used to program hardware register in device which in turn determines the maximum size of packet/frame that it can receive.
> What we have found during testing is that Niantic could receive tagged/untagged packets of size equal to max_rx_pkt_len but Fortville could only receive tagged packets (single tag) up to size <= (max_rx_pkt_len - 4). Datasheet of Niantic mentions that device implicitly accounts for  VLAN tag(s) in addition to Maximum Frame size programmed which is not the case for Fortville. This causes issue with MTU settings and maximum frame size that NIC can receive with tagged and untagged traffic.
> We have tested it with OVS-DPDK where it uses device attribute max_rx_pkt_len to set max frame size in accordance with the configured MTU size of port. However, Ixgbe (Niantic) and i40e (Fortville) interpret it differently. I looked at some other PMD drivers and different drivers interpret dev_conf.rxmode.max_rx_pkt_len differently i.e. some adds one or two VLAN, few don't include it and some use this field differently. It creates issue with MTU while running same application on different NICs and PMD drivers need to be fixed to have consistent behavior with MTU/Max Frame Size settings.
> 
> Regards,
> Nitin Katiyar

MTU on most operating systems is advisory to device driver.
The device driver may receive packets up to that MTU or larger.
How much bigger depends on how the hardware receive packet limit is implemented.
Same thing happens in Linux and BSD.