Re: Issue on capturing using DPDK Pdump on secondary interface

["Varghese, Vipin" <Vipin.Varghese@amd.com>]

[AMD Official Use Only - General]

Hi Srinivas,

Based on the internal testing with dpdk-pdump I do not find the issue with PDUMP capture with secondary interface. 
I am able capture for both port0 and port1. Following are the steps I followed

```
cd [dpdk parent folder]
cd app/pdump
gcc main.c $(pkg-config --static --libs --cflags libdpdk) -o mypdump

in terminal 1: start dpdk testpmd "dpdk-testpmd -l 15-16  --vdev=net_tap0 --vdev=net_tap1 --no-pci -- -i --forward=txonly". Then execute "start" in interactive terminal
in terminal 2: start pdump application "./mypdump --no-pci -l 12  -- --pdump 'port=0,queue=*,tx-dev=/tmp/0.pcap'"
in terminal 3: start pdump application "./mypdump --no-pci -l 13  -- --pdump 'port=1,queue=*,tx-dev=/tmp/1.pcap'"
```

Stop pdump captures in terminal 2 and 3
Execute on terminal ` tcpdump -nr /tmp/0.pcap`
Execute on terminal ` tcpdump -nr /tmp/1.pcap`

```
# tcpdump -nr /tmp/1.pcap  | more
reading from file /tmp/1.pcap, link-type EN10MB (Ethernet)
23:25:15.682623 IP 198.18.0.1.9 > 198.18.0.2.9: UDP, length 22
23:25:15.682625 IP 198.18.0.1.9 > 198.18.0.2.9: UDP, length 22
```

Hence DPDK PDUMP works with second port. I have also tried the same with Intel X710 nic too.

note: 
1. are you building DPDK application in shared library mode? If yes please ensure for application and dpdk-pdump pass all relevant shared library to init
2. are you passing desired pcie device? Please use dpdk option `-a` to pass the interfaces used in primary application for secondary 
3. have you installed pcap-dev library? DPDK pdump makes use of pcap-dev lib to write as pcap file. Hence the target should have the library.
4. does the DPDK logs gives any hint? Please re-run both primary and pdump with eal option `--log-level=8`

Regards
Vipin Varghese 

> -----Original Message-----
> From: users-request@dpdk.org <users-request@dpdk.org>
> Sent: Saturday, September 17, 2022 3:30 PM
> To: users@dpdk.org
> Subject: users Digest, Vol 356, Issue 9
> 
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
> 
> 
> Send users mailing list submissions to
>         users@dpdk.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> s.dpdk.org%2Flistinfo%2Fusers&amp;data=05%7C01%7Cvipin.varghese%40a
> md.com%7C905f5dbf0db045e968b408da989364df%7C3dd8961fe4884e608e1
> 1a82d994e183d%7C0%7C0%7C637990056071355174%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=l0csfydT0085Uztryx%2Bj1mGOy
> 2DS5DIkCfnbMgkOIx8%3D&amp;reserved=0
> or, via email, send a message with subject or body 'help' to
>         users-request@dpdk.org
> 
> You can reach the person managing the list at
>         users-owner@dpdk.org
> 
> When replying, please edit your Subject line so it is more specific than "Re:
> Contents of users digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: l2fwd: Requested device 0000:02:00.0 cannot be used
>       (Ubuntu 22.04, DPDK 22.07, N5095, I225-V) (Stephen Hemminger)
>    2. Re: Issue on capturing using DPDK Pdump on secondary
>       interface. (Stephen Hemminger)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 16 Sep 2022 08:10:59 -0700
> From: Stephen Hemminger <stephen@networkplumber.org>
> To: F32 <feng32@163.com>
> Cc: users@dpdk.org
> Subject: Re: l2fwd: Requested device 0000:02:00.0 cannot be used
>         (Ubuntu 22.04, DPDK 22.07, N5095, I225-V)
> Message-ID: <20220916081059.5b9841a1@hermes.local>
> Content-Type: text/plain; charset=US-ASCII
> 
> On Wed, 14 Sep 2022 11:18:11 +0800 (CST)
> F32 <feng32@163.com> wrote:
> 
> > Hi,
> >
> >
> > Recently I'm testing dpdk on a mini PC with N5095 CPU and 5 I225-V
> ethernet adapters under Ubuntu Server 22.04.1.
> >
> >
> > $ uname -r
> > 5.15.0-47-generic
> >
> >
> > $ cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
> > GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G
> hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt"
> > $ update-grub
> > $ reboot
> >
> >
> > $ dmesg | grep DMAR
> > [    0.016399] ACPI: DMAR 0x00000000787D6000 000088 (v02 INTEL  EDK2
> 00000002      01000013)
> > [    0.016443] ACPI: Reserving DMAR table memory at [mem 0x787d6000-
> 0x787d6087]
> > [    0.029273] DMAR: IOMMU enabled
> > [    0.093267] DMAR: Host address width 39
> > [    0.093269] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
> > [    0.093276] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap
> 1c0000c40660462 ecap 49e2ff0505e
> > [    0.093281] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
> > [    0.093287] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap
> d2008c40660462 ecap f050da
> > [    0.093291] DMAR: RMRR base: 0x0000007b800000 end: 0x0000007fbfffff
> > [    0.093296] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
> > [    0.093299] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
> > [    0.093301] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> > [    0.095158] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [    0.288306] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
> > [    0.372113] DMAR: No ATSR found
> > [    0.372115] DMAR: No SATC found
> > [    0.372118] DMAR: IOMMU feature fl1gp_support inconsistent
> > [    0.372120] DMAR: IOMMU feature pgsel_inv inconsistent
> > [    0.372122] DMAR: IOMMU feature nwfs inconsistent
> > [    0.372124] DMAR: IOMMU feature pds inconsistent
> > [    0.372125] DMAR: IOMMU feature eafs inconsistent
> > [    0.372127] DMAR: IOMMU feature prs inconsistent
> > [    0.372129] DMAR: IOMMU feature nest inconsistent
> > [    0.372130] DMAR: IOMMU feature mts inconsistent
> > [    0.372132] DMAR: IOMMU feature sc_support inconsistent
> > [    0.372134] DMAR: IOMMU feature dev_iotlb_support inconsistent
> > [    0.372136] DMAR: dmar0: Using Queued invalidation
> > [    0.372141] DMAR: dmar1: Using Queued invalidation
> > [    0.373075] DMAR: Intel(R) Virtualization Technology for Directed I/O
> >
> >
> > $ lspci
> > lspci
> > 00:00.0 Host bridge: Intel Corporation Device 4e24
> > 00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD
> > Graphics] (rev 01)
> > 00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning
> > service
> > 00:08.0 System peripheral: Intel Corporation Device 4e11
> > 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
> > 00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
> > 00:15.0 Serial bus controller: Intel Corporation Serial IO I2C Host
> > Controller (rev 01)
> > 00:15.2 Serial bus controller: Intel Corporation Device 4dea (rev 01)
> > 00:16.0 Communication controller: Intel Corporation Management Engine
> > Interface (rev 01)
> > 00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
> > 00:19.0 Serial bus controller: Intel Corporation Device 4dc5 (rev 01)
> > 00:19.1 Serial bus controller: Intel Corporation Device 4dc6 (rev 01)
> > 00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
> > 00:1c.4 PCI bridge: Intel Corporation Device 4dbc (rev 01)
> > 00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
> > 00:1c.6 PCI bridge: Intel Corporation Device 4dbe (rev 01)
> > 00:1c.7 PCI bridge: Intel Corporation Device 4dbf (rev 01)
> > 00:1e.0 Communication controller: Intel Corporation Device 4da8 (rev
> > 01)
> > 00:1e.3 Serial bus controller: Intel Corporation Device 4dab (rev 01)
> > 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
> > 00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
> > 00:1f.4 SMBus: Intel Corporation JaserLake SMBus (rev 01)
> > 00:1f.5 Serial bus controller: Intel Corporation JaserLake SPI (flash)
> > Controller (rev 01)
> > 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 02:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> > 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> > I225-V (rev 03)
> >
> >
> > (after huge page setup)
> > $ python3 dpdk-hugepages.py -s
> > Node Pages Size Total
> > 0    1607  2Mb    3Gb
> >
> >
> > Hugepages mounted on /dev/hugepages /mnt/huge
> >
> >
> > $ ifconfig enp2s0 down
> > $ ifconfig enp3s0 down
> > $ python3 dpdk-devbind.py --bind=vfio-pci 0000:02:00.0 $ python3
> > dpdk-devbind.py --bind=vfio-pci 0000:03:00.0 $ python3 dpdk-devbind.py
> > -s
> >
> >
> > Network devices using DPDK-compatible driver
> > ============================================
> > 0000:02:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
> > 0000:03:00.0 'Ethernet Controller I225-V 15f3' drv=vfio-pci unused=igc
> >
> >
> > Network devices using kernel driver
> > ===================================
> > 0000:01:00.0 'Ethernet Controller I225-V 15f3' if=enp1s0 drv=igc
> > unused=vfio-pci *Active*
> > 0000:04:00.0 'Ethernet Controller I225-V 15f3' if=enp4s0 drv=igc
> > unused=vfio-pci
> > 0000:05:00.0 'Ethernet Controller I225-V 15f3' if=enp5s0 drv=igc
> > unused=vfio-pci
> >
> >
> > No 'Baseband' devices detected
> > ==============================
> >
> >
> > No 'Crypto' devices detected
> > ============================
> >
> >
> > No 'DMA' devices detected
> > =========================
> >
> >
> > No 'Eventdev' devices detected
> > ==============================
> >
> >
> > No 'Mempool' devices detected
> > =============================
> >
> >
> > No 'Compress' devices detected
> > ==============================
> >
> >
> > No 'Misc (rawdev)' devices detected
> > ===================================
> >
> >
> > No 'Regex' devices detected
> > ===========================
> >
> >
> > After the above configuration, dpck setup should be done. Then I tried to
> run l2fwd. It generates the following errror messages:
> >
> >
> > $ ./l2fwd
> > EAL: Detected CPU lcores: 4
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:02:00.0
> > (socket 0)
> > EAL: Releasing PCI mapped resource for 0000:02:00.0
> > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100800000
> > EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x1100900000
> > EAL: Requested device 0000:02:00.0 cannot be used
> > EAL: Using IOMMU type 1 (Type 1)
> > EAL: Probe PCI driver: net_igc (8086:15f3) device: 0000:03:00.0
> > (socket 0)
> > EAL: Releasing PCI mapped resource for 0000:03:00.0
> > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100904000
> > EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1100a04000
> > EAL: Requested device 0000:03:00.0 cannot be used
> > TELEMETRY: No legacy callbacks, legacy socket not created MAC updating
> > enabled
> > EAL: Error - exiting with code: 1
> >   Cause: No Ethernet ports - bye
> >
> >
> > It seems that the I225 adapters are not recognized by DPDK. I can not see
> any detailed reasons why the adapters cannot be used.
> > Any suggestions to fix this issue?
> >
> >
> > Regards,
> > F32
> 
> Check the IOMMU groups on that system.
> Often multiple PCI devices (especially multi-port NICs) share a single IOMMU
> group and VFIO does not allow using these devices unless all subdevices in
> the group are unbound from the kernel driver.
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 16 Sep 2022 09:42:24 -0700
> From: Stephen Hemminger <stephen@networkplumber.org>
> To: Navin Srinivas <g.navinsrinivas@gmail.com>
> Cc: dev@dpdk.org, users <users@dpdk.org>
> Subject: Re: Issue on capturing using DPDK Pdump on secondary
>         interface.
> Message-ID: <20220916094224.4979921b@hermes.local>
> Content-Type: text/plain; charset=US-ASCII
> 
> On Thu, 25 Aug 2022 11:43:11 +0530
> Navin Srinivas <g.navinsrinivas@gmail.com> wrote:
> 
> > Hello,
> >
> > We are facing a strange issue, we have the following setup for our
> > application. We wanted to capture packets using dpdk pdump for our
> > application, and started to notice an issue in capturing on DPDK 21.11
> > So we went back to DPDK 20.11 and we still saw the issue.
> >
> > [image: image.png]
> > Think there is an issue in the pdump library to capture packets on the
> > secondary interface. But I'm not sure where to look.
> > APP1 - Primary APP
> > APP2 - Secondary APP.
> > Ports and ring inits are done by APP1 for APP2.
> >
> > We are able to capture packets using the standard example application
> > of pdump and capture on port 0, but we are not able to capture on port1.
> >
> > Has anyone come across this issue? Is there an open bug for this?
> > The multi process is working properly, and we are able to transmit and
> > receive packets on the secondary interface without any issue. We are
> > facing issue only on capturing packets on that interface(port1).
> >
> > Are there any pointers where I can start to look?
> >
> > Thanks & Regards,
> > Navin Srinivas
> 
> Pdump is a secondary application itself.
> 
> Pdump doesn't handle multiple interfaces well, it puts everything together.
> You probably are better off using the new dumpcap which supports pcapng.
> Pcapng supports multiple interfaces in single capture.
> 
> 
> End of users Digest, Vol 356, Issue 9
> *************************************