DPDK usage discussions
 help / color / Atom feed
From: Dikshant Chitkara <dchitkara@Airspan.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: "Varghese, Vipin" <vipin.varghese@intel.com>,
	"users@dpdk.org" <users@dpdk.org>, "dev@dpdk.org" <dev@dpdk.org>,
	Amir Ilan <ailan@Airspan.com>, Veeresh Patil <vpatil@Airspan.com>
Subject: Re: [dpdk-users] [dpdk-dev] DPDK PDUMP Issue
Date: Tue, 28 Jul 2020 16:41:38 +0000
Message-ID: <81f0e3064eaf47778a80b9b2849ce862@Airspan.com> (raw)
In-Reply-To: <20200728092337.076df2e3@hermes.lan>

Hi Stephen,
Our system has 2 sockets as seen from below :

[root@flexran3 dchitkara]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                80
On-line CPU(s) list:   0-79
Thread(s) per core:    2
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2

-----Original Message-----
From: Stephen Hemminger <stephen@networkplumber.org> 
Sent: 28 July 2020 21:54
To: Dikshant Chitkara <dchitkara@Airspan.com>
Cc: Varghese, Vipin <vipin.varghese@intel.com>; users@dpdk.org; dev@dpdk.org; Amir Ilan <ailan@Airspan.com>; Veeresh Patil <vpatil@Airspan.com>
Subject: Re: [dpdk-dev] DPDK PDUMP Issue

NOT FROM AIRSPAN - Caution - External from: stephen@networkplumber.org On Tue, 28 Jul 2020 14:51:58 +0000 Dikshant Chitkara <dchitkara@Airspan.com> wrote:

> Hi Team,
> 
> With fix suggest in prev mail thread at testpmd side, PDUMP works with testpmd.
> 
> However when we try to run our own primary app with PDUMP as a secondary process, PDUMP console comes up, however it does not capture any packets.
> 
> Changes made at primary app side:
> 1. PDUMP initialised just after rte eal init:
> 
>   ret_pdump = rte_pdump_init();
>        if (ret_pdump < 0) {
>                printf("rte_pdump_init failed\n");
>        }
>        else
>        {
>                printf("rte_pdump_init success\n");
>        }
> 
> 2. Makefile modified to add all relevant files to be linked:
> 
> LIB_SO     =-L$(DPDK_LIB) -lrte_mbuf -lrte_eal -lnuma -lrte_pmd_pcap -lrte_pdump  -lrte_pmd_i40e -lrte_eal -lrte_ring -lrte_mempool -lrte_cryptodev -lrte_ethdev -lrte_mbuf -lrte_mempool_ring  -lrte_member  -lrte_eventdev  -lrte_bus_vmbus   -lrte_pci  -lrte_bus_pci  -lrte_bus_vdev  -lrte_net  -lrte_distributor  -lrte_reorder  -lrte_kni  -lrte_pipeline  -lrte_table  -lrte_timer  -lrte_hash  -lrte_jobstats  -lrte_lpm  -lrte_power  -lrte_acl  -lrte_meter  -lrte_sch
> 
> Even tried to link final DPDK lib so as well:
> LIB_SO     =-L$(DPDK_LIB) -Wl,--whole-archive -ldpdk -Wl,--no-whole-archive -L/usr/lib/x86_64-linux-gnu/ -fPIC
> 
> It seems both apps are able to connect over mp socket but pdump unable to capture data sent from primary app.
> 
> Can you tell us what else we might be missing here.
> 
> Note: We are using DPDK 19_05 built in shared lib mode.
> 
> 
> Primary app console:
> 
> EAL: Probing VFIO support...
> EAL: PCI device 0000:41:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:41:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:86:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.2 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.3 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:88:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> EAL: PCI device 0000:88:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> rte_pdump_init success
> pool_create successed
> DPDK TX Ethernet device (destination) opened successfully Before send 
> burst Packets sent. Count=32 Before send burst Packets sent. Count=32 
> Before send burst Packets sent. Count=32 Before send burst Packets 
> sent. Count=32 Before send burst Packets sent. Count=32 Before send 
> burst Packets sent. Count=32 Before send burst Packets sent. Count=32
> >>>  
> 
> Pdump console:
> 
> [root@flexran3 x86_64-native-linux-icc]# ./app/dpdk-pdump -d librte_pmd_i40e.so -d librte_pmd_pcap.so -- --pdump 'port=0,queue=*,tx-dev=/home/dchitkara/capture.pcap'
> EAL: Detected 80 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket 
> /var/run/dpdk/rte/mp_socket_45398_5ed780258ea46
> EAL: Probing VFIO support...
> EAL: PCI device 0000:41:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:41:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:37d2 net_i40e
> EAL: PCI device 0000:86:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.2 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:86:00.3 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> EAL: PCI device 0000:88:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> EAL: PCI device 0000:88:00.1 on NUMA socket 1
> EAL:   probe driver: 8086:158b net_i40e
> Port 1 MAC: 02 70 63 61 70 00
>  core (0), capture for (1) tuples
>  - port 0 device ((null)) queue 65535
> ^C
> 
> Signal 2 received, preparing to exit...
> ##### PDUMP DEBUG STATS #####
>  -packets dequeued:                     0
>  -packets transmitted to vdev:          0
>  -packets freed:                        0
> [root@flexran3 x86_64-native-linux-icc]#
> 
> 
> 
> 
> -----Original Message-----
> From: Varghese, Vipin <vipin.varghese@intel.com>
> Sent: 22 July 2020 09:38
> To: Stephen Hemminger <stephen@networkplumber.org>; Dikshant Chitkara 
> <dchitkara@Airspan.com>
> Cc: users@dpdk.org; dev@dpdk.org; Amir Ilan <ailan@Airspan.com>; 
> Veeresh Patil <vpatil@Airspan.com>
> Subject: RE: [dpdk-dev] DPDK PDUMP Issue
> 
> NOT FROM AIRSPAN - Caution - External from: vipin.varghese@intel.com 
> Snipped
> 
> > Dikshant Chitkara <dchitkara@Airspan.com> wrote:
> >   
> > > Hi Team,
> > >
> > > We are facing an issue w.r.t to using DPDK PDUMP utility and want 
> > > to check if
> > you can help us in this regard.  
> > >
> > > This issue is specific to DPDK 19.05 version or may  be newer 
> > > versions and
> > was not found on DPDK 18.08 version.  
> > >
> > > The issue is that when we built DPDK in shared library mode and 
> > > try to run
> > PDUMP we get following error on PDUMP console:  
> > >
> > > EAL: Failed to hotplug add device
> > > EAL: Error - exiting with code: 1
> > >   Cause: vdev creation failed
> > >
> > > On Debugging above error message, I found that inside
> > > rte_dev_probe()
> > function in the file  lib/librte_eal/common/eal_common_dev.c , 
> > req.result = -
> > 14 which is non-zero value and points to enum EFAULT (Bad address) 
> > value, thus leading to above error.
> > >
> > > However, when I ran PDUMP in DPDK 18.08, I did not observe any 
> > > such issue
> > and it worked fine.  
> > >
> > > DPDK was built with following flags set in config/common_base file:
> > >
> > > CONFIG_RTE_LIBRTE_PMD_PCAP=y
> > > CONFIG_RTE_LIBRTE_PDUMP=y
> > >
> > > I ran PDUMP with testpmd as primary application and was trying to 
> > > send
> > some traffic on the Ethernet dpdk port using testpmd and capture it 
> > using pdump.
> > >
> > > Attached are the logs for both testpmd and pdump in both DPDK19_05 
> > > and
> > DPDK_18_08 version for your reference.  
> > >
> > > Let us know if you can help in this regard.
> > >
> > > Thanks,
> > > Dikshant
> > >  
> > 
> > Pdump runs as secondary process, what is your primary process.
> > Is the secondary process using the same exact versions of DPDK and 
> > libraries as the primary process?
> > The secondary process needs access to the same devices and drivers 
> > as the primary process to work.
> 
> Based on the log files shared, I have made assumptions as
> 
> 1. Primary: testpmd
> 2. DPDK version: 19.05
> 3. SHARED LIB in .config: yes
> 4. export LD_LIBRARY_PATH: yes
> 
> Using arguments passed in primary, I faced error ```
> EAL: Error - exiting with code: 1
>   Cause: Creation of mbuf pool for socket 0 failed: Invalid argument
> 
>   Cause: Mempool creation failed: Invalid argument ```
> 
> Hence had to modify with ` -d librte_mempool_ring.so`.  Using arguments passed in secondary, I faced similar error too. 
> 
> With these corrections, I am able to get similar error shared as ```
> EAL: failed to parse device "vdev:net_pcap_tx_0"
> EAL: Failed to hotplug add device on primary
> EAL: Error - exiting with code: 1
>   Cause: vdev creation failed
> ```
> 
> Checking the testpmd Makefile, it is narrowed down to missing PCAP_PMD build factor. I rebuild testpmd with ` LD_FLAGS="-lrte_pmd_pcap" make` and re-ran the same. With this I can get it working properly. 
> 
> Ideally there should had been an entry in Makefile as ``` ifeq 
> ($(CONFIG_RTE_LIBRTE_PMD_PCAP),y) LDLIBS += -lrte_pmd_pcap endif ```
> 
> reference: This is similar to the error and solution provided in ` 
> https://stackoverflow.com/questions/62795017/dpdk-pdump-failed-to-hotp
> lug-add-device/62869287#62869287`
> 
> 

The primary/secondary work is that both processes have to discover the same memory areas and devices.

The NUMA thing might be related, does your system have multiple cpu sockets.
What does the cpu_layout python script show.

  reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-14  8:40 [dpdk-users] " Dikshant Chitkara
2020-07-14 16:29 ` [dpdk-users] [dpdk-dev] " Stephen Hemminger
2020-07-22  4:08   ` Varghese, Vipin
2020-07-28 14:51     ` Dikshant Chitkara
2020-07-28 16:23       ` Stephen Hemminger
2020-07-28 16:41         ` Dikshant Chitkara [this message]
2020-07-28 17:03           ` Stephen Hemminger
2020-07-28 17:08             ` Dikshant Chitkara
2020-07-29  0:59       ` Varghese, Vipin
2020-07-29  9:14         ` Dikshant Chitkara
2020-07-29  9:51           ` Varghese, Vipin
2020-07-29  9:56             ` Varghese, Vipin
2020-07-29 14:08             ` Dikshant Chitkara
2020-07-30  2:35               ` Varghese, Vipin

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=81f0e3064eaf47778a80b9b2849ce862@Airspan.com \
    --to=dchitkara@airspan.com \
    --cc=ailan@Airspan.com \
    --cc=dev@dpdk.org \
    --cc=stephen@networkplumber.org \
    --cc=users@dpdk.org \
    --cc=vipin.varghese@intel.com \
    --cc=vpatil@Airspan.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox