Re: Help Running Example

[Stephen Hemminger <stephen@networkplumber.org>]

On Mon, 7 Aug 2023 12:40:21 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> On Sun, 6 Aug 2023 11:33:43 -0400
> Alan Beadle <ab.beadle@gmail.com> wrote:
> 
> > Hi,
> > 
> > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > 5.15 kernel with the same results.
> > 
> > Here is my NIC info from lspci:
> > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > I219-LM (rev 09)
> > 
> > I built and installed DPDK from source, and applied the following boot
> > flags: "intel_iommu=on iommu=pt"
> > 
> > After booting I did the following as root:
> > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > ifconfig enp0s31f6 down
> > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > 
> > All of this appeared to work.
> > 
> > I tried running the "skeleton" example program and got the following output:
> > sudo ./build/basicfwd
> > EAL: Detected CPU lcores: 16
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > group bound to VFIO or unbound
> > EAL: Requested device 0000:00:1f.6 cannot be used
> > TELEMETRY: No legacy callbacks, legacy socket not created
> > EAL: Error - exiting with code: 1
> >   Cause: Error: number of ports must be even
> > 
> > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > 
> > -Alan  
> 
> IOMMU groups are when multiple PCI devices share the same channel
> in the IOMMU. The group is used to determine what mapping to use when
> device does DMA. Since this is a security thing, devices in same IOMMU
> group can not be shared between kernel and non-kernel usage.
> 
> The IOMMU group is determined by wiring on the motherboard.
> Usually it is things like multiple Ethernet ports sharing the same group.
> But can be much more confused.
> 
> The only option is to unbind all devices in the group before using
> one with DPDK.

More info on IOMMU groups is in kernel documentation:
https://www.kernel.org/doc/html/latest/driver-api/vfio.html

and in this article
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive