Re: [dpdk-users] [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC

[Mike DeVico <mdevico@xcom-labs.com>]

I figured it out!!

All I needed to do was change the rss_hf from:

eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP |
 					  ETH_RSS_UDP |
 				 	  ETH_RSS_TCP |
 					  ETH_RSS_SCTP;
to simply:

eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_L2_PAYLOAD

And now I see:

Pool 0: 0 0 0 0 0 0 0 0 
Pool 1: 0 0 16 0 0 0 0 0 
Pool 2: 0 0 0 0 0 0 0 0 
Pool 3: 0 0 0 0 0 0 0 0 
Pool 4: 0 0 0 0 0 0 0 0 
Pool 5: 0 0 0 0 0 0 0 0 
Pool 6: 0 0 0 0 0 0 0 0 
Pool 7: 0 0 0 0 0 0 0 0 
Pool 8: 0 0 0 0 0 0 0 0 
Pool 9: 0 0 0 0 0 0 0 0 
Pool 10: 0 0 0 0 0 0 0 0 
Pool 11: 0 0 0 0 0 0 0 0 
Pool 12: 0 0 0 0 0 0 0 0 
Pool 13: 0 0 0 0 0 0 0 0 
Pool 14: 0 0 0 0 0 0 0 0 
Pool 15: 0 0 0 0 0 0 0 0 
Finished handling signal 1

Which is exactly how it should be!!!

So in summary, we definitely need to enable rss, but we also need to 
set the rss_hf to simply ETH_RSS_L2_PAYLOAD so that it completely 
ignores any L3 fields.

--Mike


On 9/19/19, 6:34 AM, "users on behalf of Mike DeVico" <users-bounces@dpdk.org on behalf of mdevico@xcom-labs.com> wrote:

    [EXTERNAL SENDER]
    
    Hi Xiao,
    
    Thanks for looking into this!
    
    So here’s the situation...
    
    This is a raw Ethernet packet. No IP. This
    exact setup works fine with an 82599ES.
    It looks like the hardware limitation with
    the x710 is the real problem. If we have to
    enable RSS to make it work and RSS requires a valid IP addr/port, then it’s a  catch-22 for us unless there is something we can change in the driver to account for this.
    
    Thanks!
    —Mike
    
    > On Sep 18, 2019, at 7:52 PM, Zhang, Xiao <xiao.zhang@intel.com> wrote:
    >
    > [EXTERNAL SENDER]
    >
    >> -----Original Message-----
    >> From: Mike DeVico [mailto:mdevico@xcom-labs.com]
    >> Sent: Thursday, September 19, 2019 9:23 AM
    >> To: Christensen, ChadX M <chadx.m.christensen@intel.com>; Zhang, Xiao
    >> <xiao.zhang@intel.com>; Thomas Monjalon <thomas@monjalon.net>
    >> Cc: users@dpdk.org; Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z
    >> <qi.z.zhang@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
    >> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Yigit, Ferruh
    >> <ferruh.yigit@intel.com>; Tia Cassett <tiac@xcom-labs.com>; Wu, Jingjing
    >> <jingjing.wu@intel.com>; Wong1, Samuel <samuel.wong1@intel.com>
    >> Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>
    >> As suggested I tried the following:
    >>
    >> I have an Intel FlexRAN FerryBridge broadcasting a packet 1/s which looks like
    >> the following (sudo tcpdump -i p7p1 -xx):
    >>
    >>        0x0000:  ffff ffff ffff 0000 aeae 0000 8100 4001
    >>        0x0010:  0800 0009 0000 0000 0001 8086 3600 010f
    >>        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
    >>        0x0030:  0000 0000 0000 0000 0000 0000
    >
    > There is error in the packets as I checked with wireshark, could you try with normal packets?
    >
    > No issue with following packet as I tried:
    > 0000   ff ff ff ff ff ff 00 40 05 40 ef 24 81 00 40 01
    > 0010   08 00 45 00 00 34 3b 64 40 00 40 06 b7 9b 83 97
    > 0020   20 81 83 97 20 15 04 95 17 70 51 d4 ee 9c 51 a5
    > 0030   5b 36 80 10 7c 70 12 c7 00 00 01 01 08 0a 00 04
    > 0040   f0 d4 01 99 a3 fd
    >
    >>
    >> The first 12 bytes are the dest/src MAC address followed by the 802.1Q Header
    >> (8100 4001) If you crack this, the MS 16 bits are the TPID which is set to 8100 by
    >> the Ferrybridge.
    >> The next 16 bits (0x4001) make up the PCP bits [15:13], the DEI [12] and the VID
    >> [11:0]. So if you crack the 0x4001 this makes the PCP 2 (010b), the DEI 0 and VID
    >> 1 (000000000001b).
    >>
    >> Given this I expect the packets to but placed in Pool 1/Queue 2 (based on VID 1
    >> and PCP 2).
    >> However, when I run:
    >>
    >> ./vmdq_dcb_app -w 0000:05:00.0 -w 0000:05:00.1 -l 1 -- -p 3 --nb-pools 16 --nb-
    >> tcs 8 --enable-rss
    >> EAL: Detected 24 lcore(s)
    >> EAL: Detected 2 NUMA nodes
    >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
    >> EAL: Probing VFIO support...
    >> EAL: PCI device 0000:05:00.0 on NUMA socket 0
    >> EAL:   probe driver: 8086:1572 net_i40e
    >> EAL: PCI device 0000:05:00.1 on NUMA socket 0
    >> EAL:   probe driver: 8086:1572 net_i40e
    >> vmdq queue base: 64 pool base 1
    >> Configured vmdq pool num: 16, each vmdq pool has 8 queues Port 0 modified
    >> RSS hash function based on hardware support,requested:0x3bffc
    >> configured:0x3ef8 Port 0 MAC: e8 ea 6a 27 b5 4d Port 0 vmdq pool 0 set mac
    >> 52:54:00:12:00:00 Port 0 vmdq pool 1 set mac 52:54:00:12:00:01 Port 0 vmdq
    >> pool 2 set mac 52:54:00:12:00:02 Port 0 vmdq pool 3 set mac 52:54:00:12:00:03
    >> Port 0 vmdq pool 4 set mac 52:54:00:12:00:04 Port 0 vmdq pool 5 set mac
    >> 52:54:00:12:00:05 Port 0 vmdq pool 6 set mac 52:54:00:12:00:06 Port 0 vmdq
    >> pool 7 set mac 52:54:00:12:00:07 Port 0 vmdq pool 8 set mac 52:54:00:12:00:08
    >> Port 0 vmdq pool 9 set mac 52:54:00:12:00:09 Port 0 vmdq pool 10 set mac
    >> 52:54:00:12:00:0a Port 0 vmdq pool 11 set mac 52:54:00:12:00:0b Port 0 vmdq
    >> pool 12 set mac 52:54:00:12:00:0c Port 0 vmdq pool 13 set mac
    >> 52:54:00:12:00:0d Port 0 vmdq pool 14 set mac 52:54:00:12:00:0e Port 0 vmdq
    >> pool 15 set mac 52:54:00:12:00:0f vmdq queue base: 64 pool base 1 Configured
    >> vmdq pool num: 16, each vmdq pool has 8 queues Port 1 modified RSS hash
    >> function based on hardware support,requested:0x3bffc configured:0x3ef8 Port
    >> 1 MAC: e8 ea 6a 27 b5 4e Port 1 vmdq pool 0 set mac 52:54:00:12:01:00 Port 1
    >> vmdq pool 1 set mac 52:54:00:12:01:01 Port 1 vmdq pool 2 set mac
    >> 52:54:00:12:01:02 Port 1 vmdq pool 3 set mac 52:54:00:12:01:03 Port 1 vmdq
    >> pool 4 set mac 52:54:00:12:01:04 Port 1 vmdq pool 5 set mac 52:54:00:12:01:05
    >> Port 1 vmdq pool 6 set mac 52:54:00:12:01:06 Port 1 vmdq pool 7 set mac
    >> 52:54:00:12:01:07 Port 1 vmdq pool 8 set mac 52:54:00:12:01:08 Port 1 vmdq
    >> pool 9 set mac 52:54:00:12:01:09 Port 1 vmdq pool 10 set mac
    >> 52:54:00:12:01:0a Port 1 vmdq pool 11 set mac 52:54:00:12:01:0b Port 1 vmdq
    >> pool 12 set mac 52:54:00:12:01:0c Port 1 vmdq pool 13 set mac
    >> 52:54:00:12:01:0d Port 1 vmdq pool 14 set mac 52:54:00:12:01:0e Port 1 vmdq
    >> pool 15 set mac 52:54:00:12:01:0f Core 0(lcore 1) reading queues 64-191
    >>
    >> <SIGHUP>
    >>
    >> Pool 0: 0 0 0 0 0 0 0 0
    >> Pool 1: 119 0 0 0 0 0 0 0
    >> Pool 2: 0 0 0 0 0 0 0 0
    >> Pool 3: 0 0 0 0 0 0 0 0
    >> Pool 4: 0 0 0 0 0 0 0 0
    >> Pool 5: 0 0 0 0 0 0 0 0
    >> Pool 6: 0 0 0 0 0 0 0 0
    >> Pool 7: 0 0 0 0 0 0 0 0
    >> Pool 8: 0 0 0 0 0 0 0 0
    >> Pool 9: 0 0 0 0 0 0 0 0
    >> Pool 10: 0 0 0 0 0 0 0 0
    >> Pool 11: 0 0 0 0 0 0 0 0
    >> Pool 12: 0 0 0 0 0 0 0 0
    >> Pool 13: 0 0 0 0 0 0 0 0
    >> Pool 14: 0 0 0 0 0 0 0 0
    >> Pool 15: 0 0 0 0 0 0 0 0
    >>
    >> Even with --enable-rss, the packets are still being placed in VLAN Pool 1/Queue 0
    >> instead of VLAN Pool 1/Queue 2.
    >>
    >> As I mentioned in my original email, if I use an 82599ES (dual 10G NIC), it all
    >> works as expected.
    >>
    >> What am I missing?
    >> --Mike
    >>
    >> On 9/18/19, 7:54 AM, "Christensen, ChadX M" <chadx.m.christensen@intel.com>
    >> wrote:
    >>
    >>    [EXTERNAL SENDER]
    >>
    >>    Hi Mike,
    >>
    >>    Did that resolve it?
    >>
    >>    Thanks,
    >>
    >>    Chad Christensen | Ecosystem Enablement Manager
    >>    chadx.m.christensen@intel.com | (801) 786-5703
    >>
    >>    -----Original Message-----
    >>    From: Mike DeVico <mdevico@xcom-labs.com>
    >>    Sent: Wednesday, September 18, 2019 8:17 AM
    >>    To: Zhang, Xiao <xiao.zhang@intel.com>; Thomas Monjalon
    >> <thomas@monjalon.net>
    >>    Cc: users@dpdk.org; Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z
    >> <qi.z.zhang@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
    >> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Yigit, Ferruh
    >> <ferruh.yigit@intel.com>; Christensen, ChadX M
    >> <chadx.m.christensen@intel.com>; Tia Cassett <tiac@xcom-labs.com>; Wu,
    >> Jingjing <jingjing.wu@intel.com>; Wong1, Samuel <samuel.wong1@intel.com>
    >>    Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>
    >>    Sure enough, I see it now. I'll give it a try.
    >>
    >>    Thanks!!!
    >>    --Mike
    >>
    >>    On 9/18/19, 12:11 AM, "Zhang, Xiao" <xiao.zhang@intel.com> wrote:
    >>
    >>        [EXTERNAL SENDER]
    >>
    >>> -----Original Message-----
    >>> From: Thomas Monjalon [mailto:thomas@monjalon.net]
    >>> Sent: Wednesday, September 18, 2019 3:03 PM
    >>> To: Zhang, Xiao <xiao.zhang@intel.com>
    >>> Cc: Mike DeVico <mdevico@xcom-labs.com>; users@dpdk.org; Xing,
    >> Beilei
    >>> <beilei.xing@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; Richardson,
    >> Bruce
    >>> <bruce.richardson@intel.com>; Ananyev, Konstantin
    >>> <konstantin.ananyev@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
    >>> Christensen, ChadX M <chadx.m.christensen@intel.com>; Tia Cassett
    >>> <tiac@xcom-labs.com>; Wu, Jingjing <jingjing.wu@intel.com>
    >>> Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>>
    >>> 18/09/2019 09:02, Zhang, Xiao:
    >>>>
    >>>> There is some hardware limitation and need to enable RSS to distribute
    >>> packets for X710.
    >>>
    >>> Is this limitation documented?
    >>
    >>        Yes, it's documented in doc/guides/nics/i40e.rst
    >>
    >>        "DCB works only when RSS is enabled."
    >>
    >>>
    >>
    >>
    >>
    >>
    >