Wrong rx queue assignment running DPDK on Azure

Wrong rx queue assignment running DPDK on Azure

[Yang Luan]

[-- Attachment #1: Type: text/plain, Size: 1578 bytes --]

Hello,


Our application uses the net_netvsc driver in DPDK 21.08 on Azure. We found
an issue where RSS doesn’t deliver packets to the correct rx queue. The
instance type we use is Standard_L80s_v3 with MT27800 Family [ConnectX-5
Virtual Function] NICs. The NIC is configured with 40 rx queues and the
RETA table is configured (confirmed by rte_eth_dev_rss_reta_query()) as
below.



  0:       0    1    2    3    4    5    6    7

  8:       8    9   10   11   12   13   14   15

 16:      16   17   18   19   20   21   22   23

24:      24   25   26   27   28   29   30   31

32:      32   33   34   35   36   37   38   39

40:       0    1    2    3    4    5    6    7

48:       8    9   10   11   12   13   14   15

56:      16   17   18   19   20   21   22   23

64:      24   25   26   27   28   29   30   31

72:      32   33   34   35   36   37   38   39

80:       0    1    2    3    4    5    6    7

88:       8    9   10   11   12   13   14   15

96:      16   17   18   19   20   21   22   23

104:      24   25   26   27   28   29   30   31

112:      32   33   34   35   36   37   38   39

120:       0    1    2    3    4    5    6    7



One example is a packet received with rss key 0xEDE25D84 was incorrectly
delivered to rx queue 12. We expect it to be queue 4 as 0xEDE25D84 & 0x7F =
4 assuming seven of the least significant bits (LSBs) are used for indexing
into the RETA table.  Is it a bug or we made a wrong assumption on how the
RETA table is accessed with that device?


Thank you.

Yang

[-- Attachment #2: Type: text/html, Size: 11908 bytes --]

Re: Wrong rx queue assignment running DPDK on Azure

[Stephen Hemminger]

On Tue, 1 Nov 2022 17:25:24 -0700
Yang Luan <luan.penny@gmail.com> wrote:

> Hello,
> 
> 
> Our application uses the net_netvsc driver in DPDK 21.08 on Azure. We found
> an issue where RSS doesn’t deliver packets to the correct rx queue. The
> instance type we use is Standard_L80s_v3 with MT27800 Family [ConnectX-5
> Virtual Function] NICs. The NIC is configured with 40 rx queues and the
> RETA table is configured (confirmed by rte_eth_dev_rss_reta_query()) as
> below.
> 
> 
> 
>   0:       0    1    2    3    4    5    6    7
> 
>   8:       8    9   10   11   12   13   14   15
> 
>  16:      16   17   18   19   20   21   22   23
> 
> 24:      24   25   26   27   28   29   30   31
> 
> 32:      32   33   34   35   36   37   38   39
> 
> 40:       0    1    2    3    4    5    6    7
> 
> 48:       8    9   10   11   12   13   14   15
> 
> 56:      16   17   18   19   20   21   22   23
> 
> 64:      24   25   26   27   28   29   30   31
> 
> 72:      32   33   34   35   36   37   38   39
> 
> 80:       0    1    2    3    4    5    6    7
> 
> 88:       8    9   10   11   12   13   14   15
> 
> 96:      16   17   18   19   20   21   22   23
> 
> 104:      24   25   26   27   28   29   30   31
> 
> 112:      32   33   34   35   36   37   38   39
> 
> 120:       0    1    2    3    4    5    6    7
> 
> 
> 
> One example is a packet received with rss key 0xEDE25D84 was incorrectly
> delivered to rx queue 12. We expect it to be queue 4 as 0xEDE25D84 & 0x7F =
> 4 assuming seven of the least significant bits (LSBs) are used for indexing
> into the RETA table.  Is it a bug or we made a wrong assumption on how the
> RETA table is accessed with that device?
> 
> 
> Thank you.
> 
> Yang

Are you using failsafe PMD or the native netvsc PMD?
Are you programming reta table via DPDK or through other API's?

Re: Wrong rx queue assignment running DPDK on Azure

[Yang Luan]

[-- Attachment #1: Type: text/plain, Size: 2334 bytes --]

We use netvsc PMD (drivers/net/netvsc/).
We don't explicitly configure the RETA table. We configure the device with
40 rx queues (rte_eth_dev_configure) and use rte_eth_dev_rss_reta_query()
to query the RETA table (result posted earlier).

On Fri, Nov 4, 2022 at 9:52 AM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Tue, 1 Nov 2022 17:25:24 -0700
> Yang Luan <luan.penny@gmail.com> wrote:
>
> > Hello,
> >
> >
> > Our application uses the net_netvsc driver in DPDK 21.08 on Azure. We
> found
> > an issue where RSS doesn’t deliver packets to the correct rx queue. The
> > instance type we use is Standard_L80s_v3 with MT27800 Family [ConnectX-5
> > Virtual Function] NICs. The NIC is configured with 40 rx queues and the
> > RETA table is configured (confirmed by rte_eth_dev_rss_reta_query()) as
> > below.
> >
> >
> >
> >   0:       0    1    2    3    4    5    6    7
> >
> >   8:       8    9   10   11   12   13   14   15
> >
> >  16:      16   17   18   19   20   21   22   23
> >
> > 24:      24   25   26   27   28   29   30   31
> >
> > 32:      32   33   34   35   36   37   38   39
> >
> > 40:       0    1    2    3    4    5    6    7
> >
> > 48:       8    9   10   11   12   13   14   15
> >
> > 56:      16   17   18   19   20   21   22   23
> >
> > 64:      24   25   26   27   28   29   30   31
> >
> > 72:      32   33   34   35   36   37   38   39
> >
> > 80:       0    1    2    3    4    5    6    7
> >
> > 88:       8    9   10   11   12   13   14   15
> >
> > 96:      16   17   18   19   20   21   22   23
> >
> > 104:      24   25   26   27   28   29   30   31
> >
> > 112:      32   33   34   35   36   37   38   39
> >
> > 120:       0    1    2    3    4    5    6    7
> >
> >
> >
> > One example is a packet received with rss key 0xEDE25D84 was incorrectly
> > delivered to rx queue 12. We expect it to be queue 4 as 0xEDE25D84 &
> 0x7F =
> > 4 assuming seven of the least significant bits (LSBs) are used for
> indexing
> > into the RETA table.  Is it a bug or we made a wrong assumption on how
> the
> > RETA table is accessed with that device?
> >
> >
> > Thank you.
> >
> > Yang
>
> Are you using failsafe PMD or the native netvsc PMD?
> Are you programming reta table via DPDK or through other API's?
>

[-- Attachment #2: Type: text/html, Size: 3288 bytes --]

Re: Wrong rx queue assignment running DPDK on Azure

[Stephen Hemminger]

On Fri, 4 Nov 2022 10:15:36 -0700
Yang Luan <luan.penny@gmail.com> wrote:

> We use netvsc PMD (drivers/net/netvsc/).
> We don't explicitly configure the RETA table. We configure the device with
> 40 rx queues (rte_eth_dev_configure) and use rte_eth_dev_rss_reta_query()
> to query the RETA table (result posted earlier).

More questions to help someone (not me I don't have free Azure access now);
get a reproduction.

Did you reconfigure the RSS key?
The default for the netvsc PMD should be the same default as Mellanox.


Have you enabled accelerated networking (ie VF). If so then does your DPDK
build support Mellanox.  Probably yes to both.

Is this the first packet, or later packets in the flow?
Are you using any layered encapsulation (like GRE or VXLAN), and/or IP options.

Are you changing RSS options so that is different than default L3/L4?

Are you printing the RSS key in the mbuf?  It might be byte swapped.

Why 21.08? it is not a long term supported version, and therefore does not
receive bugfixes like:

$ git log --oneline v21.08..v20.11.6 -- drivers/net/netvsc/ 
9d474a9565a5 net/netvsc: fix vmbus device reference in multi-process
a61bd9df25dc net/netvsc: fix calculation of checksums based on mbuf flag
0b5a6c7b32c8 fix spelling in comments and strings
e97bb2a91151 net/netvsc: ignore unsupported packet on sync command

Re: Wrong rx queue assignment running DPDK on Azure

[Yang Luan]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

Thanks Stephen.

We reconfigure the RSS key via rte_eth_dev_rss_hash_update(). We just set
it to the same default key value which is below just in case it changes in
the future.

    0x2c, 0xc6, 0x81, 0xd1, 0x5b, 0xdb, 0xf4, 0xf7,
    0xfc, 0xa2, 0x83, 0x19, 0xdb, 0x1a, 0x3e, 0x94,
    0x6b, 0x9e, 0x38, 0xd9, 0x2c, 0x9c, 0x03, 0xd1,
    0xad, 0x99, 0x44, 0xa7, 0xd9, 0x56, 0x3d, 0x59,
    0x06, 0x3c, 0x25, 0xf3, 0xfc, 0x1f, 0xdc, 0x2a,

Yes, we use accelerated network and our DPDK build includes mlx.

It happens on the first packet. However we don't know if subsequent packets
will work as our application stops if the first is delivered incorrectly.

We don't use layered encapsulation. Our packets are standard UDP packets.

No, we use the standard ETH_RSS_NONFRAG_IPV4_UDP.

Yes, we print the rss field in rte_mbuf. It's not byte swapped from what I
can tell.

21.08 has a feature we wanted at the time. We haven't had time to switch to
a LTS version.

On Fri, Nov 4, 2022 at 3:18 PM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Fri, 4 Nov 2022 10:15:36 -0700
> Yang Luan <luan.penny@gmail.com> wrote:
>
> > We use netvsc PMD (drivers/net/netvsc/).
> > We don't explicitly configure the RETA table. We configure the device
> with
> > 40 rx queues (rte_eth_dev_configure) and use rte_eth_dev_rss_reta_query()
> > to query the RETA table (result posted earlier).
>
> More questions to help someone (not me I don't have free Azure access now);
> get a reproduction.
>
> Did you reconfigure the RSS key?
> The default for the netvsc PMD should be the same default as Mellanox.
>
>
> Have you enabled accelerated networking (ie VF). If so then does your DPDK
> build support Mellanox.  Probably yes to both.
>
> Is this the first packet, or later packets in the flow?
> Are you using any layered encapsulation (like GRE or VXLAN), and/or IP
> options.
>
> Are you changing RSS options so that is different than default L3/L4?
>
> Are you printing the RSS key in the mbuf?  It might be byte swapped.
>
> Why 21.08? it is not a long term supported version, and therefore does not
> receive bugfixes like:
>
> $ git log --oneline v21.08..v20.11.6 -- drivers/net/netvsc/
> 9d474a9565a5 net/netvsc: fix vmbus device reference in multi-process
> a61bd9df25dc net/netvsc: fix calculation of checksums based on mbuf flag
> 0b5a6c7b32c8 fix spelling in comments and strings
> e97bb2a91151 net/netvsc: ignore unsupported packet on sync command
>

[-- Attachment #2: Type: text/html, Size: 3396 bytes --]