Re: skeleton code failing (Lokesh Chakka)

["Varghese, Vipin" <Vipin.Varghese@amd.com>]

Hi Lokesh,

Based on the DPDK logs, following are some of the observations

1. application is built with shared library mode ` gcc main.c -g `pkg-config --cflags libdpdk --libs libdpdk` which may require to pass shared dpdk PMD library with option `-d` in eal args
2. EAL PCIe probe identified `0000:83:00.0` and `0000:83:00.1`
3. there is warning from application on core NUMA-Socket vs NIC NUMA-Socket for the DPDK ports ` WARNING, port 0 is on remote NUMA node to polling thread. Performance will not be optimal. WARNING, port 1 is on remote NUMA node to polling thread. Performance will not be optimal.`
4. application has started on CPU logical thread 0 and waiting for user to interrupt the same ` Core 0 forwarding packets. [Ctrl+C to quit]`

With these observations the application is running with DPDK ports. 

With respect to queries
1. surprisingly cards are not showing up even in ifconfig - Braodcom NIC card (PF or VF) when bind with DPDK get removed from from kernel `bxnt` and added to `igb_uio|vfio_pci|uio_pci_generic`. This means there is no kernel netlink device attached to the NIC port.
2. dpdk-pmdinfo.py ./a.out But I am not seeing any output - as mentioned above this could be because the application is linked in shared library mode.
3. `gcc main.c `pkg-config --cflags libdpdk --libs libdpdk` Hence it is statically linked code. - this understanding is incorrect as the command to link statically is `pkg-config --static --cflags --libs libdpdk`.
4. Unlike other examples like l2fwd, l3fwd, testpmd there is procedure or call to function to display PMD stats for NIC ports

Hence, I recommend to spent some time https://doc.dpdk.org/guides/sample_app_ug/skeleton.html and then 
1. if the intention is build with static library, use the right command to to built the application.
2. Do not expect Broadcom NIC to make both Kernel netlink and DPDK available at the same time for PCIe id.
3. Use `log-level=pmd,8` to get more PMD related information.
4. Send traffic DPDK ports, and use `dpdk-procinfo` with option `--stats` to get the stats via DPDK secondary app.

Note: It could be NIC related issue you have to reach out Broadcom as NIC vendor for your current use case.

Regards
Vipin Varghese 

> -----Original Message-----
> From: users-request@dpdk.org <users-request@dpdk.org>
> Sent: Friday, August 26, 2022 3:30 PM
> To: users@dpdk.org
> Subject: users Digest, Vol 353, Issue 5
> 
> [CAUTION: External Email]
> 
> Send users mailing list submissions to
>         users@dpdk.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> s.dpdk.org%2Flistinfo%2Fusers&amp;data=05%7C01%7Cvipin.varghese%40a
> md.com%7C3b2eb0cad85d4133795e08da8749bf40%7C3dd8961fe4884e608e1
> 1a82d994e183d%7C0%7C0%7C637971048067638133%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=WhU4xEpP2iZ8T3fp%2BKZjhGr4
> 47C6PUh5kRMAeNKNUyU%3D&amp;reserved=0
> or, via email, send a message with subject or body 'help' to
>         users-request@dpdk.org
> 
> You can reach the person managing the list at
>         users-owner@dpdk.org
> 
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of users digest..."
> 
> 
> Today's Topics:
> 
>    1. DPD-22.07 bind error=-19 (sysman@scaled.cloud)
>    2. Re: skeleton code failing (Lokesh Chakka)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Thu, 25 Aug 2022 13:50:16 +0300
> From: sysman@scaled.cloud
> To: users@dpdk.org
> Subject: DPD-22.07 bind error=-19
> Message-ID: <cdfd3482ad4b4b19ed34fdc21a92c1ea@scaled.cloud>
> Content-Type: text/plain; charset=US-ASCII; format=flowed
> 
> Hello All!
> 
>   Can someone help me to resolved situation:
> 
> ubuntu 20 lts
> dpdk 22-0
> 
> 
> root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py -s
> 
> Network devices using kernel driver
> ===================================
> 0000:05:00.0 'I211 Gigabit Network Connection 1539' if=enp5s0 drv=igb
> unused=vfio-pci,uio_pci_generic *Active*
> 0000:06:00.0 'I211 Gigabit Network Connection 1539' if=enp6s0 drv=igb
> unused=vfio-pci,uio_pci_generic
> 0000:07:00.0 'I211 Gigabit Network Connection 1539' if=enp7s0 drv=igb
> unused=vfio-pci,uio_pci_generic
> 0000:08:00.0 'I211 Gigabit Network Connection 1539' if=enp8s0 drv=igb
> unused=vfio-pci,uio_pci_generic
> 0000:09:00.0 'I211 Gigabit Network Connection 1539' if=enp9s0 drv=igb
> unused=vfio-pci,uio_pci_generic
> 0000:0a:00.0 'I211 Gigabit Network Connection 1539' if=enp10s0 drv=igb
> unused=vfio-pci,uio_pci_generic
> 
> Other Network devices
> =====================
> 0000:01:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> unused=i40e,vfio-pci,uio_pci_generic
> 0000:01:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> unused=i40e,vfio-pci,uio_pci_generic
> 0000:01:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> unused=i40e,vfio-pci,uio_pci_generic
> 0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572'
> unused=i40e,vfio-pci,uio_pci_generic
> 
> 
> When i try to bind:
> 
> 
> root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py -b i40e
> 0000:01:00.0
> Error: bind failed for 0000:01:00.0 - Cannot bind to driver i40e: [Errno 19] No
> such device root@noserve1:/home/user/Works/tootoo2# dpdk-devbind.py
> -b uio_pci_generic 0000:01:00.0
> Error: bind failed for 0000:01:00.0 - Cannot bind to driver
> uio_pci_generic: [Errno 19] No such device
> 
> 
> 
> Thanks in advance!
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 26 Aug 2022 10:05:22 +0530
> From: Lokesh Chakka <lvenkatakumarchakka@gmail.com>
> To: David Marchand <david.marchand@redhat.com>
> Cc: users <users@dpdk.org>
> Subject: Re: skeleton code failing
> Message-ID:
>         <CACh--shYBbmtc0-
> Bp8poVKVMnfsoSpNxwYEpEHiMHkjt8waM5g@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hello,
> 
> As I am not able to get sufficient support from the intel support team or from
> the dpdk forum, the support team from intel asked me to raise a ticket in
> Intel Premium Support. Can someone help me get registered there as it is
> asking for an intel agent.
> 
> Thanks & Regards
> --
> Lokesh Chakka.
> 
> 
> On Wed, Jul 13, 2022 at 12:56 PM Lokesh Chakka <
> lvenkatakumarchakka@gmail.com> wrote:
> 
> > Dear David,
> >
> > following is some more stuff i did
> > ==================================================
> > $ sudo dpdk-devbind.py -b vfio-pci 83:00.0 $ sudo dpdk-devbind.py -b
> > vfio-pci 83:00.1 $ sudo dpdk-devbind.py -b uio_pci_generic 83:00.0
> > 83:00.1
> > Error: Driver 'uio_pci_generic' is not loaded.
> > $ sudo dpdk-devbind.py -b igb_uio 83:00.0 83:00.1
> > Error: Driver 'igb_uio' is not loaded.
> > $ sudo dpdk-devbind.py -b vfio-pci 83:00.0 83:00.1
> > Notice: 0000:83:00.0 already bound to driver vfio-pci, skipping
> > Notice: 0000:83:00.1 already bound to driver vfio-pci, skipping
> > ==================================================
> > ~/Desktop/dpdk_examples/skeleton$ gcc main.c -g `pkg-config --cflags
> > libdpdk --libs libdpdk`
> > lokesh@lokesh-ProLiant-DL385-
> Gen10:~/Desktop/dpdk_examples/skeleton$
> > sudo ./a.out
> > EAL: Detected CPU lcores: 64
> > EAL: Detected NUMA nodes: 4
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: No available 1048576 kB hugepages reported
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:83:00.0
> > (socket 2)
> > EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:83:00.1
> > (socket 2)
> > TELEMETRY: No legacy callbacks, legacy socket not created Port 0 MAC:
> > bc 97 e1 ce 84 f0 Port 1 MAC: bc 97 e1 ce 84 f1
> >
> > WARNING: Too many lcores enabled. Only 1 used.
> > WARNING, port 0 is on remote NUMA node to polling thread.
> > Performance will not be optimal.
> > WARNING, port 1 is on remote NUMA node to polling thread.
> > Performance will not be optimal.
> >
> > Core 0 forwarding packets. [Ctrl+C to quit] ^C
> > ==================================================
> > After a few seconds, I presses ctrl+c
> >
> > surprisingly cards are not showing up even in ifconfig.
> >
> >
> >
> > Thanks & Regards
> > --
> > Lokesh Chakka.
> >
> >
> > On Wed, Jul 13, 2022 at 12:43 PM Lokesh Chakka <
> > lvenkatakumarchakka@gmail.com> wrote:
> >
> >> Dear David,
> >>
> >> =================================================
> >> $ lspci | grep -i broadcom
> >> 83:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57508
> >> NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (rev 11)
> >> 83:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57508
> >> NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb/200Gb Ethernet (rev 11) $
> lspci
> >> -n -s 83:00.0
> >> 83:00.0 0200: 14e4:1750 (rev 11)
> >> =================================================
> >>
> >> I am compiling my code like this :
> >> =================================================
> >> gcc main.c `pkg-config --cflags libdpdk --libs libdpdk`
> >> =================================================
> >>
> >> Hence it is statically linked code.
> >> If I try
> >> $ dpdk-pmdinfo.py ./a.out
> >>
> >> But I am not seeing any output
> >>
> >>
> >>
> >> Thanks & Regards
> >> --
> >> Lokesh Chakka.
> >>
> >>
> >> On Wed, Jul 13, 2022 at 12:22 PM David Marchand <
> >> david.marchand@redhat.com> wrote:
> >>
> >>> On Wed, Jul 13, 2022 at 7:35 AM Lokesh Chakka
> >>> <lvenkatakumarchakka@gmail.com> wrote:
> >>> > Would like to understand if I am missing something. I am new to
> >>> > this
> >>> platform.
> >>> > rte_eth_dev_count_avail is returning zero.
> >>> > OS is Ubuntu 22.04. DPDK is latest version.
> >>> > Cards are being detected by Linux. Ifconfig is showing the cards up.
> >>> LED is also glowing.
> >>>
> >>> Indeed, DPDK provides a userspace driver for some NetXtreme nics
> >>> (which is net/bnxt).
> >>> This userspace driver does not rely on the bnxt Linux kernel driver.
> >>> IOW, this card being detecting and working with the Linux kernel
> >>> does not automatically mean that this nic can work with DPDK.
> >>>
> >>> We need more info on your nic, first.
> >>>
> >>> Can you share the pci id of this nic (like running lspci -n -s
> >>> $pci_address)?
> >>> It should be a 14e4:XXXX.
> >>>
> >>> Then you can check this XXXX against what your dpdk application
> supports.
> >>>
> >>> If it is a statically linked application, you can run:
> >>> $ dpdk-pmdinfo.py /path/to/your/application
> >>>
> >>> Else, if your application is dynamically linked against DPDK driver,
> >>> you can run this command against the net/bnxt driver .so.22 (for
> >>> 21.11 and later releases):
> >>> $ dpdk-pmdinfo.py /path/to/your/dpdk/drivers/librte_net_bnxt.so.22
> >>>
> >>> You should get a list of supported NetXtreme nics, like:
> >>>
> >>> [snipped some other drivers compiled in my application] PMD NAME:
> >>> net_bnxt PMD HW SUPPORT:
> >>>  Broadcom Inc. and subsidiaries (14e4) : BCM5745X NetXtreme-E RDMA
> >>> Virtual Function (1606) (All Subdevices)  Broadcom Inc. and
> >>> subsidiaries (14e4) : BCM5745X NetXtreme-E Ethernet Virtual Function
> >>> (1609) (All Subdevices)  Broadcom Inc. and subsidiaries (14e4) :
> >>> BCM57454 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb Ethernet (1614)
> (All
> >>> Subdevices)  Broadcom Inc. and subsidiaries (14e4) : NetXtreme-E
> >>> RDMA Virtual Function (16c1) (All Subdevices)  Broadcom Inc. and
> >>> subsidiaries (14e4) : NetXtreme-C Ethernet Virtual Function (16cb)
> >>> (All Subdevices) [snipped the rest]
> >>>
> >>> I hope you can find a (XXXX) corresponding to your NIC.
> >>>
> >>>
> >>> --
> >>> David Marchand
> >>>
> >>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail
> s.dpdk.org%2Farchives%2Fusers%2Fattachments%2F20220826%2F09fcb44d
> %2Fattachment-
> 0001.htm&amp;data=05%7C01%7Cvipin.varghese%40amd.com%7C3b2eb0ca
> d85d4133795e08da8749bf40%7C3dd8961fe4884e608e11a82d994e183d%7C0
> %7C0%7C637971048067638133%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%
> 7C%7C%7C&amp;sdata=SfVPC2z%2BqkUkKTCfsY6A7FB02znexfvnKPSyKkk9
> MpA%3D&amp;reserved=0>
> 
> End of users Digest, Vol 353, Issue 5
> *************************************