DPDK usage discussions
 help / color / mirror / Atom feed
* Re: [dpdk-users] [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
@ 2019-10-10 21:12 Mike DeVico
  0 siblings, 0 replies; 20+ messages in thread
From: Mike DeVico @ 2019-10-10 21:12 UTC (permalink / raw)
  To: Zhang, Helin, Jeff Weeks, Thomas Monjalon
  Cc: users, Xing, Beilei, Zhang, Qi Z, Richardson, Bruce, Ananyev,
	Konstantin, Yigit, Ferruh, Zhang, Xiao, Wong1, Samuel,
	Kobi Cohen-Arazi

I was also able reproduce the same issue by connecting an actual Ferrybridge to one port (0000:82.0)

and then use the vmdq_dcb app to receive and forward the packet to another port (0000:82.1)

which I have connected via a passive copper cable to a third port (0000:82.2) which I then read

using my rxtest app.



So it looks like this:



Ferrybridge -- > Port 0000:82.0 --> vmdq_dcb --> 0000:82.1 --> passive copper cable directly connected to --> 0000:82.2 --> rxtest app.



Packet received by the vmdq_dcb app (on 0000:82:00.0)



RX Packet at [0x7f47778797c0], len=60

00000000: 02 00 00 00 00 01 E8 EA 6A 27 B8 FA 81 00 40 01 | ........j'....@.

00000010: 08 00 00 09 00 00 00 00 00 01 80 86 36 00 01 0F | ............6...

00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................

00000030: 00 00 00 00 00 00 00 00 00 00 00 00 |  |  |  |  | ............

Forwarding to queue: 74 (Port 0000:82.00.1)



Packet as seen from my simple rxtest app listening on Port 0000:82.00.2.



dump mbuf at 0x7f813eab4e00, iova=dfeab4e80, buf_len=2176

  pkt_len=60, ol_flags=180, nb_segs=1, in_port=0

  segment at 0x7f813eab4e00, data=0x7f813eab4f00, data_len=60

  Dump data at [0x7f813eab4f00], len=60

00000000: 02 00 00 00 00 01 E8 EA 6A 27 B8 FA 81 00 00 01 | ........j'......

00000010: 08 00 00 09 00 00 00 00 00 01 80 86 36 00 01 0F | ............6...

00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................

00000030: 00 00 00 00 00 00 00 00 00 00 00 00 |  |  |  |  | ............



Why does the transmitted packet go out with the PCP/DEI portion of the VLAN tag

zeroed out and what do I need to configure to prevent this from happening?



Thanks!

--Mike





On 10/3/19, 4:56 PM, "users on behalf of Mike DeVico" <users-bounces@dpdk.org on behalf of mdevico@xcom-labs.com> wrote:



    [EXTERNAL SENDER]



    I set up a basic loopback test by connecting two ports of



    the same X710 NIC with a 10G passive copper cable.







    In my case, the two ports map to PCI addresses



    0000:82:00.0 and 0000:82:00.1. 0000:82:00.0 is bound



    to igb_uio and .1 is left bound to the kernel driver (i40e)



    which is mapped to linux eth iface p4p1.







    I then created a simple txtest app (see attached) based on the



    vmdq_dcb example app that sends out a fixed packet once per



    second over a specified port. It also prints out the packet each



    time it sends it.







    I run txtest as follows:







    sudo ./txtest -w 0000:82:00.0 -c 0x3







    TX Packet at [0x7fa407c2ee80], len=60



    00000000: 00 00 AE AE 00 00 E8 EA 6A 27 B8 F9 81 00 40 01 | ........j'....@.



    00000010: 08 00 00 08 00 00 00 00 AA BB CC DD 00 00 00 00 | ................



    00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................



    00000030: 00 00 00 00 00 00 00 00 00 00 00 00 |  |  |  |  | ............







    ...







    Note the byte shown in red. This corresponds to the PCP field in the VLAN header.







    I then run tcpdump on p2p4 (the receiving end of the loopback) as shown:







    sudo tcpdump -xxi p4p2







    19:02:08.124694 IP0



            0x0000:  0000 aeae 0000 e8ea 6a27 b8f9 8100 0001



            0x0010:  0800 0008 0000 0000 aabb ccdd 0000 0000



            0x0020:  0000 0000 0000 0000 0000 0000 0000 0000



            0x0030:  0000 0000 0000 0000 0000 0000



    ...







    Once again, note the PCP byte shown in red. The source of my



    problem would appear to be that the PCP field is somehow getting



    set to 0 even though it’s being passed to rte_eth_tx_burst as 0x40.



    Here’s the section of the code in my main.c.







    lcore_main(void *arg)



    {



        uint32_t lcore_id = rte_lcore_id();







        struct rte_mempool *mbuf_pool = (struct rte_mempool*)arg;







        RTE_LOG(INFO, TXTEST, "tx entering main loop on lcore %u\n", lcore_id);







        static const uint8_t tx_data[] = {



             0x00, 0x00, 0xAE, 0xAE, 0x00, 0x00, 0xE8, 0xEA, 0x6A, 0x27, 0xB8, 0xF9, 0x81, 0x00, 0x40, 0x01,



             0x08, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0xAA, 0xBB, 0xCC, 0xDD, 0x00, 0x00, 0x00, 0x00,



             0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,



             0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00



        };







        struct rte_mbuf* tx_pkt      = rte_pktmbuf_alloc(mbuf_pool);



        uint8_t*         tx_pkt_dptr = (uint8_t *) rte_pktmbuf_append(tx_pkt, sizeof(tx_data));



        memcpy(tx_pkt_dptr,tx_data,sizeof(tx_data));



        const uint8_t* dptr = rte_pktmbuf_mtod(tx_pkt,uint8_t*);







        while (!force_quit) {



            uint32_t nb_rx = rte_eth_tx_burst(0, 0, &tx_pkt, 1);



           if (nb_rx == 1) {



                rte_hexdump(stdout, "TX Packet", dptr,



                            rte_pktmbuf_data_len(tx_pkt));



            }



            sleep(1);



        }



        return 0;



    }







    As can be seen, the packet is fixed and it is dumped immediately



    after being sent, so why is the PCP field going out over the wire



    as a 0?







    For reference, I’m using dpdk version 18.08 and the only config change



    is to set CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM=8 (from the default 4).



    I did this because our actual application requires the RX to be configured with



    16 pools with 8 TCs queues per pool and I wanted to resemble the target



    config as closely as possible.







    Any insight/assistance would be greatly appreciated.







    Thank you in advance.



    --Mike







    On 9/29/19, 7:22 PM, "Zhang, Helin" <helin.zhang@intel.com> wrote:







        [EXTERNAL SENDER]







        Hi Mike







        There you need to try to use the example applications (e.g. testpmd) to reproduce your issue. Then send out the detailed information to the maintainers. Thanks!







        Regards,



        Helin







        -----Original Message-----



        From: Mike DeVico [mailto:mdevico@xcom-labs.com]



        Sent: Thursday, September 26, 2019 9:32 PM



        To: Zhang, Helin <helin.zhang@intel.com>; Jeff Weeks <jweeks@sandvine.com>; Thomas Monjalon <thomas@monjalon.net>



        Cc: users@dpdk.org; Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Zhang, Xiao <xiao.zhang@intel.com>; Wong1, Samuel <samuel.wong1@intel.com>



        Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC







        Hi Helin,







        Yes, and the reason the RX packets were not being queued to the proper queue was due to RSS not being enabled/configured. Once I did this, the RX packets were placed in the proper queue.







        That being said, what I see now is that the TX side seems to have an issue. So the way it works is that the Ferrybridge broadcasts out what's called a Present packet at 1s intervals. Once the host application detects the packet (which it now does) it verifies that the packet is correctly formatted and such and then sends a packet back to the Ferrybridge to tell it to stop sending this packet. However, that TX packet apparently is not going out because I continue to receive Present packets from the Ferrybridge at the 1s interval. What's not clear to me is what queue I should be sending this packet to. I actually tried sending it out all 128 queues, but I still keep receiving the Present packet. What I lack is the ability to actually sniff what's going out over the wire.







        Any ideas how to approach this issue?







        Thanks in advance,



        --Mike







        On 9/26/19, 9:02 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:







            [EXTERNAL SENDER]







            Hi Mike







            Can you check if you are using the right combination of DPDK version and NIC firmware, and kernel driver if you are using?



            You can find the recommended combination from https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides%2Fnics%2Fi40e.html%23recommended-matching-list&amp;data=02%7C01%7Cmdevico%40xcom-labs.com%7Cf2f84f03821d4326691b08d7485d4837%7C30eb8c3e540c4f279dfa86b6e7bfafa8%7C0%7C0%7C637057437871195895&amp;sdata=WOxDFmbaGcLNX1m4heyB5uu6dv5lY10XPcI13wYTxzA%3D&amp;reserved=0. Hopefully that helps!







            Regards,



            Helin







            > -----Original Message-----



            > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Mike DeVico



            > Sent: Friday, September 13, 2019 2:10 AM



            > To: Jeff Weeks; Thomas Monjalon



            > Cc: dev@dpdk.org; Xing, Beilei; Zhang, Qi Z; Richardson, Bruce; Ananyev,



            > Konstantin; Yigit, Ferruh



            > Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC



            >



            > Hi Jeff,



            >



            > Thanks for chiming in...



            >



            > Yeah, In my case I get the packets, but they end up being put in queue 0



            > instead of 2.



            >



            > --Mike



            >



            > From: Jeff Weeks <jweeks@sandvine.com>



            > Date: Thursday, September 12, 2019 at 10:47 AM



            > To: Mike DeVico <mdevico@xcom-labs.com>, Thomas Monjalon



            > <thomas@monjalon.net>



            > Cc: "dev@dpdk.org" <dev@dpdk.org>, Beilei Xing <beilei.xing@intel.com>, Qi



            > Zhang <qi.z.zhang@intel.com>, Bruce Richardson



            > <bruce.richardson@intel.com>, Konstantin Ananyev



            > <konstantin.ananyev@intel.com>, "ferruh.yigit@intel.com"



            > <ferruh.yigit@intel.com>



            > Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC



            >



            > [EXTERNAL SENDER]



            >



            > I don't have much else to add, except that I also see dcb fail on the same NIC:



            >



            >



            >



            >   i40e_dcb_init_configure(): default dcb config fails. err = -53, aq_err = 3.



            >



            >



            >



            > My card doesn't receive any packets, though; not sure if it's related to this, or



            > not.



            >



            >



            >



            > --Jeff



            >



            > ________________________________



            > /dev/jeff_weeks.x2936



            > Sandvine Incorporated



            >



            > ________________________________



            > From: dev <dev-bounces@dpdk.org> on behalf of Mike DeVico



            > <mdevico@xcom-labs.com>



            > Sent: Thursday, September 12, 2019 1:06 PM



            > To: Thomas Monjalon



            > Cc: dev@dpdk.org; Beilei Xing; Qi Zhang; Bruce Richardson; Konstantin



            > Ananyev; ferruh.yigit@intel.com



            > Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC



            >



            > [EXTERNAL]



            >



            > Still no hits...



            >



            > --Mike



            >



            > On 9/9/19, 1:39 PM, "Thomas Monjalon" <thomas@monjalon.net> wrote:



            >



            >     [EXTERNAL SENDER]



            >



            >     Adding i40e maintainers and few more.



            >



            >     07/09/2019 01:11, Mike DeVico:



            >     > Hello,



            >     >



            >     > I am having an issue getting the DCB feature to work with an Intel



            >     > X710 Quad SFP+ NIC.



            >     >



            >     > Here’s my setup:



            >     >



            >     > 1.      DPDK 18.08 built with the following I40E configs:



            >     >



            >     > CONFIG_RTE_LIBRTE_I40E_PMD=y



            >     > CONFIG_RTE_LIBRTE_I40E_DEBUG_RX=n



            >     > CONFIG_RTE_LIBRTE_I40E_DEBUG_TX=n



            >     > CONFIG_RTE_LIBRTE_I40E_DEBUG_TX_FREE=n



            >     > CONFIG_RTE_LIBRTE_I40E_RX_ALLOW_BULK_ALLOC=y



            >     > CONFIG_RTE_LIBRTE_I40E_INC_VECTOR=y



            >     > CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=n



            >     > CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF=64



            >     > CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM=8



            >     >



            >     > 2.      /opt/dpdk-18.08/usertools/dpdk-devbind.py --status-dev net



            >     >



            >     > Network devices using DPDK-compatible driver



            >     > ============================================



            >     > 0000:3b:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio



            > unused=i40e



            >     > 0000:3b:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio



            > unused=i40e



            >     > 0000:3b:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio



            > unused=i40e



            >     > 0000:3b:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio



            > unused=i40e



            >     >



            >     >        Network devices using kernel driver



            >     >        ===================================



            >     >        0000:02:00.0 'I350 Gigabit Network Connection 1521' if=enp2s0f0



            > drv=igb unused=igb_uio *Active*



            >     >        0000:02:00.1 'I350 Gigabit Network Connection 1521' if=enp2s0f1



            > drv=igb unused=igb_uio *Active*



            >     >



            >     >        Other Network devices



            >     >        =====================



            >     >        <none>



            >     >



            >     > 3.      We have a custom FPGA board connected to port 1 of the X710 NIC



            > that’s broadcasting



            >     > a packet tagged with VLAN 1 and PCP 2.



            >     >



            >     > 4.      I use the vmdq_dcb example app and configure the card with 16



            > pools/8 queue each



            >     > as follows:



            >     >        sudo ./vmdq_dcb_app -l 1 -- -p3 --nb-pools 16 --nb-tcs 8 -p 3



            >     >



            >     >



            >     > The apps starts up fine and successfully probes the card as shown below:



            >     >



            >     > sudo ./vmdq_dcb_app -l 1 -- -p3 --nb-pools 16 --nb-tcs 8 -p 3



            >     > EAL: Detected 80 lcore(s)



            >     > EAL: Detected 2 NUMA nodes



            >     > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket



            >     > EAL: Probing VFIO support...



            >     > EAL: PCI device 0000:02:00.0 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1521 net_e1000_igb



            >     > EAL: PCI device 0000:02:00.1 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1521 net_e1000_igb



            >     > EAL: PCI device 0000:3b:00.0 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.1 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.2 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.3 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > vmdq queue base: 64 pool base 1



            >     > Configured vmdq pool num: 16, each vmdq pool has 8 queues



            >     > Port 0 MAC: e8 ea 6a 27 b5 4d



            >     > Port 0 vmdq pool 0 set mac 52:54:00:12:00:00



            >     > Port 0 vmdq pool 1 set mac 52:54:00:12:00:01



            >     > Port 0 vmdq pool 2 set mac 52:54:00:12:00:02



            >     > Port 0 vmdq pool 3 set mac 52:54:00:12:00:03



            >     > Port 0 vmdq pool 4 set mac 52:54:00:12:00:04



            >     > Port 0 vmdq pool 5 set mac 52:54:00:12:00:05



            >     > Port 0 vmdq pool 6 set mac 52:54:00:12:00:06



            >     > Port 0 vmdq pool 7 set mac 52:54:00:12:00:07



            >     > Port 0 vmdq pool 8 set mac 52:54:00:12:00:08



            >     > Port 0 vmdq pool 9 set mac 52:54:00:12:00:09



            >     > Port 0 vmdq pool 10 set mac 52:54:00:12:00:0a



            >     > Port 0 vmdq pool 11 set mac 52:54:00:12:00:0b



            >     > Port 0 vmdq pool 12 set mac 52:54:00:12:00:0c



            >     > Port 0 vmdq pool 13 set mac 52:54:00:12:00:0d



            >     > Port 0 vmdq pool 14 set mac 52:54:00:12:00:0e



            >     > Port 0 vmdq pool 15 set mac 52:54:00:12:00:0f



            >     > vmdq queue base: 64 pool base 1



            >     > Configured vmdq pool num: 16, each vmdq pool has 8 queues



            >     > Port 1 MAC: e8 ea 6a 27 b5 4e



            >     > Port 1 vmdq pool 0 set mac 52:54:00:12:01:00



            >     > Port 1 vmdq pool 1 set mac 52:54:00:12:01:01



            >     > Port 1 vmdq pool 2 set mac 52:54:00:12:01:02



            >     > Port 1 vmdq pool 3 set mac 52:54:00:12:01:03



            >     > Port 1 vmdq pool 4 set mac 52:54:00:12:01:04



            >     > Port 1 vmdq pool 5 set mac 52:54:00:12:01:05



            >     > Port 1 vmdq pool 6 set mac 52:54:00:12:01:06



            >     > Port 1 vmdq pool 7 set mac 52:54:00:12:01:07



            >     > Port 1 vmdq pool 8 set mac 52:54:00:12:01:08



            >     > Port 1 vmdq pool 9 set mac 52:54:00:12:01:09



            >     > Port 1 vmdq pool 10 set mac 52:54:00:12:01:0a



            >     > Port 1 vmdq pool 11 set mac 52:54:00:12:01:0b



            >     > Port 1 vmdq pool 12 set mac 52:54:00:12:01:0c



            >     > Port 1 vmdq pool 13 set mac 52:54:00:12:01:0d



            >     > Port 1 vmdq pool 14 set mac 52:54:00:12:01:0e



            >     > Port 1 vmdq pool 15 set mac 52:54:00:12:01:0f



            >     >



            >     > Skipping disabled port 2



            >     >



            >     > Skipping disabled port 3



            >     > Core 0(lcore 1) reading queues 64-191



            >     >



            >     > However, when I issue the SIGHUP I see that the packets



            >     > are being put into the first queue of Pool 1 as follows:



            >     >



            >     > Pool 0: 0 0 0 0 0 0 0 0



            >     > Pool 1: 10 0 0 0 0 0 0 0



            >     > Pool 2: 0 0 0 0 0 0 0 0



            >     > Pool 3: 0 0 0 0 0 0 0 0



            >     > Pool 4: 0 0 0 0 0 0 0 0



            >     > Pool 5: 0 0 0 0 0 0 0 0



            >     > Pool 6: 0 0 0 0 0 0 0 0



            >     > Pool 7: 0 0 0 0 0 0 0 0



            >     > Pool 8: 0 0 0 0 0 0 0 0



            >     > Pool 9: 0 0 0 0 0 0 0 0



            >     > Pool 10: 0 0 0 0 0 0 0 0



            >     > Pool 11: 0 0 0 0 0 0 0 0



            >     > Pool 12: 0 0 0 0 0 0 0 0



            >     > Pool 13: 0 0 0 0 0 0 0 0



            >     > Pool 14: 0 0 0 0 0 0 0 0



            >     > Pool 15: 0 0 0 0 0 0 0 0



            >     > Finished handling signal 1



            >     >



            >     > Since the packets are being tagged with PCP 2 they should be getting



            >     > mapped to 3rd queue of Pool 1, right?



            >     >



            >     > As a sanity check, I tried the same test using an 82599ES 2 port 10GB NIC



            > and



            >     > the packets show up in the expected queue. (Note, to get it to work I had



            >     > to modify the vmdq_dcb app to set the vmdq pool MACs to all FF’s)



            >     >



            >     > Here’s that setup:



            >     >



            >     > /opt/dpdk-18.08/usertools/dpdk-devbind.py --status-dev net



            >     >



            >     > Network devices using DPDK-compatible driver



            >     > ============================================



            >     > 0000:af:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'



            > drv=igb_uio unused=ixgbe



            >     > 0000:af:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'



            > drv=igb_uio unused=ixgbe



            >     >



            >     > Network devices using kernel driver



            >     > ===================================



            >     > 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=enp2s0f0 drv=igb



            > unused=igb_uio *Active*



            >     > 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=enp2s0f1 drv=igb



            > unused=igb_uio *Active*



            >     > 0000:3b:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'



            > if=enp59s0f0 drv=i40e unused=igb_uio



            >     > 0000:3b:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'



            > if=enp59s0f1 drv=i40e unused=igb_uio



            >     > 0000:3b:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'



            > if=enp59s0f2 drv=i40e unused=igb_uio



            >     > 0000:3b:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'



            > if=enp59s0f3 drv=i40e unused=igb_uio



            >     >



            >     > Other Network devices



            >     > =====================



            >     > <none>



            >     >



            >     > sudo ./vmdq_dcb_app -l 1 -- -p3 --nb-pools 16 --nb-tcs 8 -p 3



            >     > EAL: Detected 80 lcore(s)



            >     > EAL: Detected 2 NUMA nodes



            >     > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket



            >     > EAL: Probing VFIO support...



            >     > EAL: PCI device 0000:02:00.0 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1521 net_e1000_igb



            >     > EAL: PCI device 0000:02:00.1 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1521 net_e1000_igb



            >     > EAL: PCI device 0000:3b:00.0 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.1 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.2 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:3b:00.3 on NUMA socket 0



            >     > EAL:   probe driver: 8086:1572 net_i40e



            >     > EAL: PCI device 0000:af:00.0 on NUMA socket 1



            >     > EAL:   probe driver: 8086:10fb net_ixgbe



            >     > EAL: PCI device 0000:af:00.1 on NUMA socket 1



            >     > EAL:   probe driver: 8086:10fb net_ixgbe



            >     > vmdq queue base: 0 pool base 0



            >     > Port 0 MAC: 00 1b 21 bf 71 24



            >     > Port 0 vmdq pool 0 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 1 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 2 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 3 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 4 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 5 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 6 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 7 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 8 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 9 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 10 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 11 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 12 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 13 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 14 set mac ff:ff:ff:ff:ff:ff



            >     > Port 0 vmdq pool 15 set mac ff:ff:ff:ff:ff:ff



            >     > vmdq queue base: 0 pool base 0



            >     > Port 1 MAC: 00 1b 21 bf 71 26



            >     > Port 1 vmdq pool 0 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 1 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 2 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 3 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 4 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 5 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 6 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 7 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 8 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 9 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 10 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 11 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 12 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 13 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 14 set mac ff:ff:ff:ff:ff:ff



            >     > Port 1 vmdq pool 15 set mac ff:ff:ff:ff:ff:ff



            >     >



            >     > Now when I send the SIGHUP, I see the packets being routed to



            >     > the expected queue:



            >     >



            >     > Pool 0: 0 0 0 0 0 0 0 0



            >     > Pool 1: 0 0 58 0 0 0 0 0



            >     > Pool 2: 0 0 0 0 0 0 0 0



            >     > Pool 3: 0 0 0 0 0 0 0 0



            >     > Pool 4: 0 0 0 0 0 0 0 0



            >     > Pool 5: 0 0 0 0 0 0 0 0



            >     > Pool 6: 0 0 0 0 0 0 0 0



            >     > Pool 7: 0 0 0 0 0 0 0 0



            >     > Pool 8: 0 0 0 0 0 0 0 0



            >     > Pool 9: 0 0 0 0 0 0 0 0



            >     > Pool 10: 0 0 0 0 0 0 0 0



            >     > Pool 11: 0 0 0 0 0 0 0 0



            >     > Pool 12: 0 0 0 0 0 0 0 0



            >     > Pool 13: 0 0 0 0 0 0 0 0



            >     > Pool 14: 0 0 0 0 0 0 0 0



            >     > Pool 15: 0 0 0 0 0 0 0 0



            >     > Finished handling signal 1



            >     >



            >     > What am I missing?



            >     >



            >     > Thankyou in advance,



            >     > --Mike



            >     >



            >     >



            >



            >



            >



            >



            >

















    -------------- next part --------------

    A non-text attachment was scrubbed...

    Name: txtest.tar

    Type: application/x-tar

    Size: 20480 bytes

    Desc: txtest.tar

    URL: <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmails.dpdk.org%2Farchives%2Fusers%2Fattachments%2F20191003%2F5ed5e742%2Fattachment.tar&amp;data=02%7C01%7Cmdevico%40xcom-labs.com%7Cf2f84f03821d4326691b08d7485d4837%7C30eb8c3e540c4f279dfa86b6e7bfafa8%7C0%7C0%7C637057437871195895&amp;sdata=o2NiRH3D2gZ2d7r870YXNTPcKGhm05Twx7lAJP1bsGA%3D&amp;reserved=0>



^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [dpdk-users] [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
@ 2019-09-20 21:57 Mike DeVico
  2019-10-10 21:23 ` Christensen, ChadX M
  0 siblings, 1 reply; 20+ messages in thread
From: Mike DeVico @ 2019-09-20 21:57 UTC (permalink / raw)
  To: Zhang, Xiao
  Cc: Christensen, ChadX M, Thomas Monjalon, users, Xing, Beilei,
	Zhang, Qi Z, Richardson, Bruce, Ananyev, Konstantin, Yigit,
	Ferruh, Tia Cassett, Wu, Jingjing, Wong1,  Samuel

I figured it out!!

All I needed to do was change the rss_hf from:

eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP |
 					  ETH_RSS_UDP |
 				 	  ETH_RSS_TCP |
 					  ETH_RSS_SCTP;
to simply:

eth_conf->rx_adv_conf.rss_conf.rss_hf = ETH_RSS_L2_PAYLOAD

And now I see:

Pool 0: 0 0 0 0 0 0 0 0 
Pool 1: 0 0 16 0 0 0 0 0 
Pool 2: 0 0 0 0 0 0 0 0 
Pool 3: 0 0 0 0 0 0 0 0 
Pool 4: 0 0 0 0 0 0 0 0 
Pool 5: 0 0 0 0 0 0 0 0 
Pool 6: 0 0 0 0 0 0 0 0 
Pool 7: 0 0 0 0 0 0 0 0 
Pool 8: 0 0 0 0 0 0 0 0 
Pool 9: 0 0 0 0 0 0 0 0 
Pool 10: 0 0 0 0 0 0 0 0 
Pool 11: 0 0 0 0 0 0 0 0 
Pool 12: 0 0 0 0 0 0 0 0 
Pool 13: 0 0 0 0 0 0 0 0 
Pool 14: 0 0 0 0 0 0 0 0 
Pool 15: 0 0 0 0 0 0 0 0 
Finished handling signal 1

Which is exactly how it should be!!!

So in summary, we definitely need to enable rss, but we also need to 
set the rss_hf to simply ETH_RSS_L2_PAYLOAD so that it completely 
ignores any L3 fields.

--Mike


On 9/19/19, 6:34 AM, "users on behalf of Mike DeVico" <users-bounces@dpdk.org on behalf of mdevico@xcom-labs.com> wrote:

    [EXTERNAL SENDER]
    
    Hi Xiao,
    
    Thanks for looking into this!
    
    So here’s the situation...
    
    This is a raw Ethernet packet. No IP. This
    exact setup works fine with an 82599ES.
    It looks like the hardware limitation with
    the x710 is the real problem. If we have to
    enable RSS to make it work and RSS requires a valid IP addr/port, then it’s a  catch-22 for us unless there is something we can change in the driver to account for this.
    
    Thanks!
    —Mike
    
    > On Sep 18, 2019, at 7:52 PM, Zhang, Xiao <xiao.zhang@intel.com> wrote:
    >
    > [EXTERNAL SENDER]
    >
    >> -----Original Message-----
    >> From: Mike DeVico [mailto:mdevico@xcom-labs.com]
    >> Sent: Thursday, September 19, 2019 9:23 AM
    >> To: Christensen, ChadX M <chadx.m.christensen@intel.com>; Zhang, Xiao
    >> <xiao.zhang@intel.com>; Thomas Monjalon <thomas@monjalon.net>
    >> Cc: users@dpdk.org; Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z
    >> <qi.z.zhang@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
    >> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Yigit, Ferruh
    >> <ferruh.yigit@intel.com>; Tia Cassett <tiac@xcom-labs.com>; Wu, Jingjing
    >> <jingjing.wu@intel.com>; Wong1, Samuel <samuel.wong1@intel.com>
    >> Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>
    >> As suggested I tried the following:
    >>
    >> I have an Intel FlexRAN FerryBridge broadcasting a packet 1/s which looks like
    >> the following (sudo tcpdump -i p7p1 -xx):
    >>
    >>        0x0000:  ffff ffff ffff 0000 aeae 0000 8100 4001
    >>        0x0010:  0800 0009 0000 0000 0001 8086 3600 010f
    >>        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
    >>        0x0030:  0000 0000 0000 0000 0000 0000
    >
    > There is error in the packets as I checked with wireshark, could you try with normal packets?
    >
    > No issue with following packet as I tried:
    > 0000   ff ff ff ff ff ff 00 40 05 40 ef 24 81 00 40 01
    > 0010   08 00 45 00 00 34 3b 64 40 00 40 06 b7 9b 83 97
    > 0020   20 81 83 97 20 15 04 95 17 70 51 d4 ee 9c 51 a5
    > 0030   5b 36 80 10 7c 70 12 c7 00 00 01 01 08 0a 00 04
    > 0040   f0 d4 01 99 a3 fd
    >
    >>
    >> The first 12 bytes are the dest/src MAC address followed by the 802.1Q Header
    >> (8100 4001) If you crack this, the MS 16 bits are the TPID which is set to 8100 by
    >> the Ferrybridge.
    >> The next 16 bits (0x4001) make up the PCP bits [15:13], the DEI [12] and the VID
    >> [11:0]. So if you crack the 0x4001 this makes the PCP 2 (010b), the DEI 0 and VID
    >> 1 (000000000001b).
    >>
    >> Given this I expect the packets to but placed in Pool 1/Queue 2 (based on VID 1
    >> and PCP 2).
    >> However, when I run:
    >>
    >> ./vmdq_dcb_app -w 0000:05:00.0 -w 0000:05:00.1 -l 1 -- -p 3 --nb-pools 16 --nb-
    >> tcs 8 --enable-rss
    >> EAL: Detected 24 lcore(s)
    >> EAL: Detected 2 NUMA nodes
    >> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
    >> EAL: Probing VFIO support...
    >> EAL: PCI device 0000:05:00.0 on NUMA socket 0
    >> EAL:   probe driver: 8086:1572 net_i40e
    >> EAL: PCI device 0000:05:00.1 on NUMA socket 0
    >> EAL:   probe driver: 8086:1572 net_i40e
    >> vmdq queue base: 64 pool base 1
    >> Configured vmdq pool num: 16, each vmdq pool has 8 queues Port 0 modified
    >> RSS hash function based on hardware support,requested:0x3bffc
    >> configured:0x3ef8 Port 0 MAC: e8 ea 6a 27 b5 4d Port 0 vmdq pool 0 set mac
    >> 52:54:00:12:00:00 Port 0 vmdq pool 1 set mac 52:54:00:12:00:01 Port 0 vmdq
    >> pool 2 set mac 52:54:00:12:00:02 Port 0 vmdq pool 3 set mac 52:54:00:12:00:03
    >> Port 0 vmdq pool 4 set mac 52:54:00:12:00:04 Port 0 vmdq pool 5 set mac
    >> 52:54:00:12:00:05 Port 0 vmdq pool 6 set mac 52:54:00:12:00:06 Port 0 vmdq
    >> pool 7 set mac 52:54:00:12:00:07 Port 0 vmdq pool 8 set mac 52:54:00:12:00:08
    >> Port 0 vmdq pool 9 set mac 52:54:00:12:00:09 Port 0 vmdq pool 10 set mac
    >> 52:54:00:12:00:0a Port 0 vmdq pool 11 set mac 52:54:00:12:00:0b Port 0 vmdq
    >> pool 12 set mac 52:54:00:12:00:0c Port 0 vmdq pool 13 set mac
    >> 52:54:00:12:00:0d Port 0 vmdq pool 14 set mac 52:54:00:12:00:0e Port 0 vmdq
    >> pool 15 set mac 52:54:00:12:00:0f vmdq queue base: 64 pool base 1 Configured
    >> vmdq pool num: 16, each vmdq pool has 8 queues Port 1 modified RSS hash
    >> function based on hardware support,requested:0x3bffc configured:0x3ef8 Port
    >> 1 MAC: e8 ea 6a 27 b5 4e Port 1 vmdq pool 0 set mac 52:54:00:12:01:00 Port 1
    >> vmdq pool 1 set mac 52:54:00:12:01:01 Port 1 vmdq pool 2 set mac
    >> 52:54:00:12:01:02 Port 1 vmdq pool 3 set mac 52:54:00:12:01:03 Port 1 vmdq
    >> pool 4 set mac 52:54:00:12:01:04 Port 1 vmdq pool 5 set mac 52:54:00:12:01:05
    >> Port 1 vmdq pool 6 set mac 52:54:00:12:01:06 Port 1 vmdq pool 7 set mac
    >> 52:54:00:12:01:07 Port 1 vmdq pool 8 set mac 52:54:00:12:01:08 Port 1 vmdq
    >> pool 9 set mac 52:54:00:12:01:09 Port 1 vmdq pool 10 set mac
    >> 52:54:00:12:01:0a Port 1 vmdq pool 11 set mac 52:54:00:12:01:0b Port 1 vmdq
    >> pool 12 set mac 52:54:00:12:01:0c Port 1 vmdq pool 13 set mac
    >> 52:54:00:12:01:0d Port 1 vmdq pool 14 set mac 52:54:00:12:01:0e Port 1 vmdq
    >> pool 15 set mac 52:54:00:12:01:0f Core 0(lcore 1) reading queues 64-191
    >>
    >> <SIGHUP>
    >>
    >> Pool 0: 0 0 0 0 0 0 0 0
    >> Pool 1: 119 0 0 0 0 0 0 0
    >> Pool 2: 0 0 0 0 0 0 0 0
    >> Pool 3: 0 0 0 0 0 0 0 0
    >> Pool 4: 0 0 0 0 0 0 0 0
    >> Pool 5: 0 0 0 0 0 0 0 0
    >> Pool 6: 0 0 0 0 0 0 0 0
    >> Pool 7: 0 0 0 0 0 0 0 0
    >> Pool 8: 0 0 0 0 0 0 0 0
    >> Pool 9: 0 0 0 0 0 0 0 0
    >> Pool 10: 0 0 0 0 0 0 0 0
    >> Pool 11: 0 0 0 0 0 0 0 0
    >> Pool 12: 0 0 0 0 0 0 0 0
    >> Pool 13: 0 0 0 0 0 0 0 0
    >> Pool 14: 0 0 0 0 0 0 0 0
    >> Pool 15: 0 0 0 0 0 0 0 0
    >>
    >> Even with --enable-rss, the packets are still being placed in VLAN Pool 1/Queue 0
    >> instead of VLAN Pool 1/Queue 2.
    >>
    >> As I mentioned in my original email, if I use an 82599ES (dual 10G NIC), it all
    >> works as expected.
    >>
    >> What am I missing?
    >> --Mike
    >>
    >> On 9/18/19, 7:54 AM, "Christensen, ChadX M" <chadx.m.christensen@intel.com>
    >> wrote:
    >>
    >>    [EXTERNAL SENDER]
    >>
    >>    Hi Mike,
    >>
    >>    Did that resolve it?
    >>
    >>    Thanks,
    >>
    >>    Chad Christensen | Ecosystem Enablement Manager
    >>    chadx.m.christensen@intel.com | (801) 786-5703
    >>
    >>    -----Original Message-----
    >>    From: Mike DeVico <mdevico@xcom-labs.com>
    >>    Sent: Wednesday, September 18, 2019 8:17 AM
    >>    To: Zhang, Xiao <xiao.zhang@intel.com>; Thomas Monjalon
    >> <thomas@monjalon.net>
    >>    Cc: users@dpdk.org; Xing, Beilei <beilei.xing@intel.com>; Zhang, Qi Z
    >> <qi.z.zhang@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
    >> Ananyev, Konstantin <konstantin.ananyev@intel.com>; Yigit, Ferruh
    >> <ferruh.yigit@intel.com>; Christensen, ChadX M
    >> <chadx.m.christensen@intel.com>; Tia Cassett <tiac@xcom-labs.com>; Wu,
    >> Jingjing <jingjing.wu@intel.com>; Wong1, Samuel <samuel.wong1@intel.com>
    >>    Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>
    >>    Sure enough, I see it now. I'll give it a try.
    >>
    >>    Thanks!!!
    >>    --Mike
    >>
    >>    On 9/18/19, 12:11 AM, "Zhang, Xiao" <xiao.zhang@intel.com> wrote:
    >>
    >>        [EXTERNAL SENDER]
    >>
    >>> -----Original Message-----
    >>> From: Thomas Monjalon [mailto:thomas@monjalon.net]
    >>> Sent: Wednesday, September 18, 2019 3:03 PM
    >>> To: Zhang, Xiao <xiao.zhang@intel.com>
    >>> Cc: Mike DeVico <mdevico@xcom-labs.com>; users@dpdk.org; Xing,
    >> Beilei
    >>> <beilei.xing@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; Richardson,
    >> Bruce
    >>> <bruce.richardson@intel.com>; Ananyev, Konstantin
    >>> <konstantin.ananyev@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
    >>> Christensen, ChadX M <chadx.m.christensen@intel.com>; Tia Cassett
    >>> <tiac@xcom-labs.com>; Wu, Jingjing <jingjing.wu@intel.com>
    >>> Subject: Re: [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC
    >>>
    >>> 18/09/2019 09:02, Zhang, Xiao:
    >>>>
    >>>> There is some hardware limitation and need to enable RSS to distribute
    >>> packets for X710.
    >>>
    >>> Is this limitation documented?
    >>
    >>        Yes, it's documented in doc/guides/nics/i40e.rst
    >>
    >>        "DCB works only when RSS is enabled."
    >>
    >>>
    >>
    >>
    >>
    >>
    >
    


^ permalink raw reply	[flat|nested] 20+ messages in thread
[parent not found: <834B2FF6-9FC7-43E4-8CA7-67D861FEE70E@xcom-tech.com>]
end of thread, other threads:[~2019-10-11  7:56 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-10 21:12 [dpdk-users] [dpdk-dev] Issue with DCB with X710 Quad 10Gb NIC Mike DeVico
  -- strict thread matches above, loose matches on Subject: below --
2019-09-20 21:57 Mike DeVico
2019-10-10 21:23 ` Christensen, ChadX M
2019-10-10 21:25   ` Mike DeVico
     [not found] <834B2FF6-9FC7-43E4-8CA7-67D861FEE70E@xcom-tech.com>
     [not found] ` <2953945.eKoDkclGR7@xps>
2019-09-17 18:54   ` Mike DeVico
2019-09-18  3:32     ` Zhang, Xiao
2019-09-18  4:20       ` Mike DeVico
2019-09-18  7:02         ` Zhang, Xiao
2019-09-18  7:03           ` Thomas Monjalon
2019-09-18  7:10             ` Zhang, Xiao
2019-09-18 14:17               ` Mike DeVico
2019-09-18 14:53                 ` Christensen, ChadX M
2019-09-18 20:22                   ` Mike DeVico
2019-09-19  1:23                   ` Mike DeVico
2019-09-19  2:52                     ` Zhang, Xiao
2019-09-19 13:34                       ` Mike DeVico
2019-09-19 14:34                         ` Johnson, Brian
     [not found]   ` <0BD0EAA3-BB16-4B09-BF25-4744C0A879A0@xcom-tech.com>
     [not found]     ` <b9318aa4f0a943958171cc6fc53a010f@sandvine.com>
     [not found]       ` <61798E93-724B-4BE6-A03C-63B274E71AD2@xcom-tech.com>
     [not found]         ` <F35DEAC7BCE34641BA9FAC6BCA4A12E71B4ADE0E@SHSMSX103.ccr.corp.intel.com>
2019-09-26 20:31           ` Mike DeVico
2019-09-30  2:21             ` Zhang, Helin
2019-10-03 23:56               ` Mike DeVico

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).