Issue on capturing using DPDK Pdump on secondary interface.

Issue on capturing using DPDK Pdump on secondary interface.

[Navin Srinivas]

[-- Attachment #1.1: Type: text/plain, Size: 1005 bytes --]

Hello,

We are facing a strange issue, we have the following setup for our
application. We wanted to capture packets using dpdk pdump for our
application, and started to notice an issue in capturing on DPDK 21.11
So we went back to DPDK 20.11 and we still saw the issue.

[image: image.png]
Think there is an issue in the pdump library to capture packets on the
secondary interface. But I'm not sure where to look.
APP1 - Primary APP
APP2 - Secondary APP.
Ports and ring inits are done by APP1 for APP2.

We are able to capture packets using the standard example application of
pdump and capture on port 0, but we are not able to capture on port1.

Has anyone come across this issue? Is there an open bug for this?
The multi process is working properly, and we are able to transmit and
receive packets on the secondary interface without any issue. We are facing
issue only on capturing packets on that interface(port1).

Are there any pointers where I can start to look?

Thanks & Regards,
Navin Srinivas

[-- Attachment #1.2: Type: text/html, Size: 1386 bytes --]

[-- Attachment #2: image.png --]
[-- Type: image/png, Size: 6463 bytes --]

Re: Issue on capturing using DPDK Pdump on secondary interface.

[Stephen Hemminger]

On Thu, 25 Aug 2022 11:43:11 +0530
Navin Srinivas <g.navinsrinivas@gmail.com> wrote:

> Hello,
> 
> We are facing a strange issue, we have the following setup for our
> application. We wanted to capture packets using dpdk pdump for our
> application, and started to notice an issue in capturing on DPDK 21.11
> So we went back to DPDK 20.11 and we still saw the issue.
> 
> [image: image.png]
> Think there is an issue in the pdump library to capture packets on the
> secondary interface. But I'm not sure where to look.
> APP1 - Primary APP
> APP2 - Secondary APP.
> Ports and ring inits are done by APP1 for APP2.
> 
> We are able to capture packets using the standard example application of
> pdump and capture on port 0, but we are not able to capture on port1.
> 
> Has anyone come across this issue? Is there an open bug for this?
> The multi process is working properly, and we are able to transmit and
> receive packets on the secondary interface without any issue. We are facing
> issue only on capturing packets on that interface(port1).
> 
> Are there any pointers where I can start to look?
> 
> Thanks & Regards,
> Navin Srinivas

Pdump is a secondary application itself.

Pdump doesn't handle multiple interfaces well, it puts everything
together.  You probably are better off using the new dumpcap which
supports pcapng. Pcapng supports multiple interfaces in single capture.

Re: Issue on capturing using DPDK Pdump on secondary interface

[Varghese, Vipin]

[AMD Official Use Only - General]

Hi Srinivas,

Based on the internal testing with dpdk-pdump I do not find the issue with PDUMP capture with secondary interface. 
I am able capture for both port0 and port1. Following are the steps I followed

```
cd [dpdk parent folder]
cd app/pdump
gcc main.c $(pkg-config --static --libs --cflags libdpdk) -o mypdump

in terminal 1: start dpdk testpmd "dpdk-testpmd -l 15-16  --vdev=net_tap0 --vdev=net_tap1 --no-pci -- -i --forward=txonly". Then execute "start" in interactive terminal
in terminal 2: start pdump application "./mypdump --no-pci -l 12  -- --pdump 'port=0,queue=*,tx-dev=/tmp/0.pcap'"
in terminal 3: start pdump application "./mypdump --no-pci -l 13  -- --pdump 'port=1,queue=*,tx-dev=/tmp/1.pcap'"
```

Stop pdump captures in terminal 2 and 3
Execute on terminal ` tcpdump -nr /tmp/0.pcap`
Execute on terminal ` tcpdump -nr /tmp/1.pcap`

```
# tcpdump -nr /tmp/1.pcap  | more
reading from file /tmp/1.pcap, link-type EN10MB (Ethernet)
23:25:15.682623 IP 198.18.0.1.9 > 198.18.0.2.9: UDP, length 22
23:25:15.682625 IP 198.18.0.1.9 > 198.18.0.2.9: UDP, length 22
```

Hence DPDK PDUMP works with second port. I have also tried the same with Intel X710 nic too.

note: 
1. are you building DPDK application in shared library mode? If yes please ensure for application and dpdk-pdump pass all relevant shared library to init
2. are you passing desired pcie device? Please use dpdk option `-a` to pass the interfaces used in primary application for secondary 
3. have you installed pcap-dev library? DPDK pdump makes use of pcap-dev lib to write as pcap file. Hence the target should have the library.
4. does the DPDK logs gives any hint? Please re-run both primary and pdump with eal option `--log-level=8`

Regards
Vipin Varghese 

> -----Original Message-----
> From: users-request@dpdk.org <users-request@dpdk.org>
> Sent: Saturday, September 17, 2022 3:30 PM
> To: users@dpdk.org
> Subject: users Digest, Vol 356, Issue 9
> 
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
> 
> 
> Send users mailing list submissions to
>         users@dpdk.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> s.dpdk.org%2Flistinfo%2Fusers&amp;data=05%7C01%7Cvipin.varghese%40a
> md.com%7C905f5dbf0db045e968b408da989364df%7C3dd8961fe4884e608e1
> 1a82d994e183d%7C0%7C0%7C637990056071355174%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=l0csfydT0085Uztryx%2Bj1mGOy
> 2DS5DIkCfnbMgkOIx8%3D&amp;reserved=0
> or, via email, send a message with subject or body 'help' to
>         users-request@dpdk.org
> 
> You can reach the person managing the list at
>         users-owner@dpdk.org
> 
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of users digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: l2fwd: Requested device 0000:02:00.0 cannot be used
>       (Ubuntu 22.04, DPDK 22.07, N5095, I225-V) (Stephen Hemminger)
>    2. Re: Issue on capturing using DPDK Pdump on secondary
>       interface. (Stephen Hemminger)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 16 Sep 2022 08:10:59 -0700
> From: Stephen Hemminger <stephen@networkplumber.org>
> To: F32 <feng32@163.com>
> Cc: users@dpdk.org
> Subject: Re: l2fwd: Requested device 0000:02:00.0 cannot be used
>         (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
> Message-ID: <20220916081059.5b9841a1@hermes.local>
> Content-Type: text/plain; charset=US-ASCII
> 
> On Wed, 14 Sep 2022 11:18:11 +0800 (CST)
> F32 <feng32@163.com> wrote:
> 
> > Hi,
> >
> >
> > Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V
> ethernet adapters under Ubuntu Server 22.04.1.
> >
> >
> > $ uname -r
> > 5.15.0-47-generic
> >
> >
> > $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
> > GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G
> hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
> > $ update-grub
> > $ reboot
> >
> >
> > $ dmesg | grep DMAR
> > [    0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL  EDK2
> 00000002      01000013)
> > [    0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-
> 0x787d6087]
> > [    0.029273] DMAR: IOMMU enabled
> > [    0.093267] DMAR: Host address width 39
> > [    0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
> > [    0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap
> 1c0000c40660462 ecap 49e2ff0505e
> > [    0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
> > [    0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap
> d2008c40660462 ecap f050da
> > [    0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
> > [    0.093296] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
> > [    0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
> > [    0.093301] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> > [    0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [    0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
> > [    0.372113] DMAR: No ATSR found
> > [    0.372115] DMAR: No SATC found
> > [    0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
> > [    0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
> > [    0.372122] DMAR: IOMMU feature nwfs inconsistent
> > [    0.372124] DMAR: IOMMU feature pds inconsistent
> > [    0.372125] DMAR: IOMMU feature eafs inconsistent
> > [    0.372127] DMAR: IOMMU feature prs inconsistent
> > [    0.372129] DMAR: IOMMU feature nest inconsistent
> > [    0.372130] DMAR: IOMMU feature mts inconsistent
> > [    0.372132] DMAR: IOMMU feature sc_support inconsistent
> > [    0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
> > [    0.372136] DMAR: dmar0: Using Queued invalidation
> > [    0.372141] DMAR: dmar1: Using Queued invalidation
> > [    0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
> >
> >
> > $ lspci
> > lspci
> > 00:00.0 Host bridge: Intel Corporation Device 4e24
> > 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD
> > Graphics] (rev 01)
> > 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning
> > service
> > 00:08.0 System peripheral: Intel Corporation Device 4e11
> > 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
> > 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
> > 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host
> > Controller (rev 01)
> > 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
> > 00:16.0 Communication controller: Intel Corporation Management Engine
> > Interface (rev 01)
> > 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
> > 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
> > 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
> > 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
> > 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
> > 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
> > 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
> > 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
> > 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev
> > 01)
> > 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
> > 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
> > 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
> > 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
> > 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash)
> > Controller (rev 01)
> > 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> >
> >
> > (after huge page setup)
> > $ python3 dpdk-hugepages.py -s
> > Node Pages Size Total
> > 0    1607  2Mb    3Gb
> >
> >
> > Hugepages mounted on /dev/hugepages /mnt/huge
> >
> >
> > $ ifconfig enp2s0 down
> > $ ifconfig enp3s0 down
> > $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0 $ python3
> > dpdk-devbind.py --bind=vfio-pci 0000:03:00.0 $ python3 dpdk-devbind.py
> > -s
> >
> >
> > Network devices using DPDK-compatible driver
> > ============================================
> > 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
> > 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
> >
> >
> > Network devices using kernel driver
> > ===================================
> > 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc
> > unused=vfio-pci *Active*
> > 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc
> > unused=vfio-pci
> > 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc
> > unused=vfio-pci
> >
> >
> > No 'Baseband' devices detected
> > ==============================
> >
> >
> > No 'Crypto' devices detected
> > ============================
> >
> >
> > No 'DMA' devices detected
> > =========================
> >
> >
> > No 'Eventdev' devices detected
> > ==============================
> >
> >
> > No 'Mempool' devices detected
> > =============================
> >
> >
> > No 'Compress' devices detected
> > ==============================
> >
> >
> > No 'Misc (rawdev)' devices detected
> > ===================================
> >
> >
> > No 'Regex' devices detected
> > ===========================
> >
> >
> > After the above configuration, dpck setup should be done. Then I tried to
> run l2fwd. It generates the following errror messages:
> >
> >
> > $ ./l2fwd
> > EAL: Detected CPU lcores: 4
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0
> > (socket 0)
> > EAL: Releasing PCI mapped resource for 0000:02:00.0
> > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
> > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
> > EAL: Requested device 0000:02:00.0 cannot be used
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0
> > (socket 0)
> > EAL: Releasing PCI mapped resource for 0000:03:00.0
> > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
> > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
> > EAL: Requested device 0000:03:00.0 cannot be used
> > TELEMETRY: No legacy callbacks, legacy socket not created MAC updating
> > enabled
> > EAL: Error - exiting with code: 1
> >   Cause: No Ethernet ports - bye
> >
> >
> > It seems that the I225 adapters are not recognized by DPDK. I can not see
> any detailed reasons why the adapters cannot be used.
> > Any suggestions to fix this issue?
> >
> >
> > Regards,
> > F32
> 
> Check the IOMMU groups on that system.
> Often multiple PCI devices (especially multi-port NICs) share a single IOMMU
> group and VFIO does not allow using these devices unless all subdevices in
> the group are unbound from the kernel driver.
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 16 Sep 2022 09:42:24 -0700
> From: Stephen Hemminger <stephen@networkplumber.org>
> To: Navin Srinivas <g.navinsrinivas@gmail.com>
> Cc: dev@dpdk.org, users <users@dpdk.org>
> Subject: Re: Issue on capturing using DPDK Pdump on secondary
>         interface.
> Message-ID: <20220916094224.4979921b@hermes.local>
> Content-Type: text/plain; charset=US-ASCII
> 
> On Thu, 25 Aug 2022 11:43:11 +0530
> Navin Srinivas <g.navinsrinivas@gmail.com> wrote:
> 
> > Hello,
> >
> > We are facing a strange issue, we have the following setup for our
> > application. We wanted to capture packets using dpdk pdump for our
> > application, and started to notice an issue in capturing on DPDK 21.11
> > So we went back to DPDK 20.11 and we still saw the issue.
> >
> > [image: image.png]
> > Think there is an issue in the pdump library to capture packets on the
> > secondary interface. But I'm not sure where to look.
> > APP1 - Primary APP
> > APP2 - Secondary APP.
> > Ports and ring inits are done by APP1 for APP2.
> >
> > We are able to capture packets using the standard example application
> > of pdump and capture on port 0, but we are not able to capture on port1.
> >
> > Has anyone come across this issue? Is there an open bug for this?
> > The multi process is working properly, and we are able to transmit and
> > receive packets on the secondary interface without any issue. We are
> > facing issue only on capturing packets on that interface(port1).
> >
> > Are there any pointers where I can start to look?
> >
> > Thanks & Regards,
> > Navin Srinivas
> 
> Pdump is a secondary application itself.
> 
> Pdump doesn't handle multiple interfaces well, it puts everything together.
> You probably are better off using the new dumpcap which supports pcapng.
> Pcapng supports multiple interfaces in single capture.
> 
> 
> End of users Digest, Vol 356, Issue 9
> *************************************