Re: [dpdk-dev] DPDK PDUMP Issue

[Stephen Hemminger <stephen@networkplumber.org>]

On Tue, 28 Jul 2020 14:51:58 +0000
Dikshant Chitkara <dchitkara@Airspan.com> wrote:

> Hi Team,
> 
> With fix suggest in prev mail thread at testpmd side, PDUMP works with testpmd.
> 
> However when we try to run our own primary app with PDUMP as a secondary process, PDUMP console comes up, however it does not capture any packets.
> 
> Changes made at primary app side:
> 1. PDUMP initialised just after rte eal init:
> 
>   ret_pdump = rte_pdump_init();
>        if (ret_pdump < 0) {
>                printf("rte_pdump_init failed\n");
>        }
>        else
>        {
>                printf("rte_pdump_init success\n");
>        }
> 
> 2. Makefile modified to add all relevant files to be linked:
> 
> LIB_SO     =-L$(DPDK_LIB) -lrte_mbuf -lrte_eal -lnuma -lrte_pmd_pcap -lrte_pdump  -lrte_pmd_i40e -lrte_eal -lrte_ring -lrte_mempool -lrte_cryptodev -lrte_ethdev -lrte_mbuf -lrte_mempool_ring  -lrte_member  -lrte_eventdev  -lrte_bus_vmbus   -lrte_pci  -lrte_bus_pci  -lrte_bus_vdev  -lrte_net  -lrte_distributor  -lrte_reorder  -lrte_kni  -lrte_pipeline  -lrte_table  -lrte_timer  -lrte_hash  -lrte_jobstats  -lrte_lpm  -lrte_power  -lrte_acl  -lrte_meter  -lrte_sch
> 
> Even tried to link final DPDK lib so as well:
> LIB_SO     =-L$(DPDK_LIB) -Wl,--whole-archive -ldpdk -Wl,--no-whole-archive -L/usr/lib/x86_64-linux-gnu/ -fPIC
> 
> It seems both apps are able to connect over mp socket but pdump unable to capture data sent from primary app.
> 
> Can you tell us what else we might be missing here.
> 
> Note: We are using DPDK 19_05 built in shared lib mode.
> 
> 
> Primary app console:
> 
> EAL: Probing VFIO support...
> EAL: PCI device 0000:41:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:41:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:86:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.2 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.3 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:88:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> EAL: PCI device 0000:88:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> rte_pdump_init success
> pool_create successed
> DPDK TX Ethernet device (destination) opened successfully
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> Before send burst
> Packets sent. Count=32
> >>>  
> 
> Pdump console:
> 
> [root@flexran3 x86_64-native-linux-icc]# ./app/dpdk-pdump -d librte_pmd_i40e.so -d librte_pmd_pcap.so -- --pdump 'port=0,queue=*,tx-dev=/home/dchitkara/capture.pcap'
> EAL: Detected 80 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket_45398_5ed780258ea46
> EAL: Probing VFIO support...
> EAL: PCI device 0000:41:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:41:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:86:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.2 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.3 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:88:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> EAL: PCI device 0000:88:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> Port 1 MAC: 02 70 63 61 70 00
>  core (0), capture for (1) tuples
>  - port 0 device ((null)) queue 65535
> ^C
> 
> Signal 2 received, preparing to exit...
> ##### PDUMP DEBUG STATS #####
>  -packets dequeued:                     0
>  -packets transmitted to vdev:          0
>  -packets freed:                        0
> [root@flexran3 x86_64-native-linux-icc]#
> 
> 
> 
> 
> -----Original Message-----
> From: Varghese, Vipin <vipin.varghese@intel.com> 
> Sent: 22 July 2020 09:38
> To: Stephen Hemminger <stephen@networkplumber.org>; Dikshant Chitkara <dchitkara@Airspan.com>
> Cc: users@dpdk.org; dev@dpdk.org; Amir Ilan <ailan@Airspan.com>; Veeresh Patil <vpatil@Airspan.com>
> Subject: RE: [dpdk-dev] DPDK PDUMP Issue
> 
> NOT FROM AIRSPAN - Caution - External from: vipin.varghese@intel.com Snipped
> 
> > Dikshant Chitkara <dchitkara@Airspan.com> wrote:
> >   
> > > Hi Team,
> > >
> > > We are facing an issue w.r.t to using DPDK PDUMP utility and want to 
> > > check if  
> > you can help us in this regard.  
> > >
> > > This issue is specific to DPDK 19.05 version or may  be newer 
> > > versions and  
> > was not found on DPDK 18.08 version.  
> > >
> > > The issue is that when we built DPDK in shared library mode and try 
> > > to run  
> > PDUMP we get following error on PDUMP console:  
> > >
> > > EAL: Failed to hotplug add device
> > > EAL: Error - exiting with code: 1
> > >   Cause: vdev creation failed
> > >
> > > On Debugging above error message, I found that inside 
> > > rte_dev_probe()  
> > function in the file  lib/librte_eal/common/eal_common_dev.c , 
> > req.result = -
> > 14 which is non-zero value and points to enum EFAULT (Bad address) 
> > value, thus leading to above error.  
> > >
> > > However, when I ran PDUMP in DPDK 18.08, I did not observe any such 
> > > issue  
> > and it worked fine.  
> > >
> > > DPDK was built with following flags set in config/common_base file:
> > >
> > > CONFIG_RTE_LIBRTE_PMD_PCAP=y
> > > CONFIG_RTE_LIBRTE_PDUMP=y
> > >
> > > I ran PDUMP with testpmd as primary application and was trying to 
> > > send  
> > some traffic on the Ethernet dpdk port using testpmd and capture it 
> > using pdump.  
> > >
> > > Attached are the logs for both testpmd and pdump in both DPDK19_05 
> > > and  
> > DPDK_18_08 version for your reference.  
> > >
> > > Let us know if you can help in this regard.
> > >
> > > Thanks,
> > > Dikshant
> > >  
> > 
> > Pdump runs as secondary process, what is your primary process.
> > Is the secondary process using the same exact versions of DPDK and 
> > libraries as the primary process?
> > The secondary process needs access to the same devices and drivers as 
> > the primary process to work.  
> 
> Based on the log files shared, I have made assumptions as
> 
> 1. Primary: testpmd
> 2. DPDK version: 19.05
> 3. SHARED LIB in .config: yes
> 4. export LD_LIBRARY_PATH: yes
> 
> Using arguments passed in primary, I faced error ```
> EAL: Error - exiting with code: 1
>   Cause: Creation of mbuf pool for socket 0 failed: Invalid argument
> 
>   Cause: Mempool creation failed: Invalid argument ```
> 
> Hence had to modify with ` -d librte_mempool_ring.so`.  Using arguments passed in secondary, I faced similar error too. 
> 
> With these corrections, I am able to get similar error shared as ```
> EAL: failed to parse device "vdev:net_pcap_tx_0"
> EAL: Failed to hotplug add device on primary
> EAL: Error - exiting with code: 1
>   Cause: vdev creation failed
> ```
> 
> Checking the testpmd Makefile, it is narrowed down to missing PCAP_PMD build factor. I rebuild testpmd with ` LD_FLAGS="-lrte_pmd_pcap" make` and re-ran the same. With this I can get it working properly. 
> 
> Ideally there should had been an entry in Makefile as ``` ifeq ($(CONFIG_RTE_LIBRTE_PMD_PCAP),y) LDLIBS += -lrte_pmd_pcap endif ```
> 
> reference: This is similar to the error and solution provided in ` https://stackoverflow.com/questions/62795017/dpdk-pdump-failed-to-hotplug-add-device/62869287#62869287`
> 
> 

The primary/secondary work is that both processes have to discover the same memory areas
and devices.

The NUMA thing might be related, does your system have multiple cpu sockets.
What does the cpu_layout python script show.