Help Running Example

Help Running Example

[Alan Beadle]

Hi,

I need some help getting DPDK working. I am running Ubuntu 20.04 with
a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
5.15 kernel with the same results.

Here is my NIC info from lspci:
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
I219-LM (rev 09)

I built and installed DPDK from source, and applied the following boot
flags: "intel_iommu=on iommu=pt"

After booting I did the following as root:
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
ifconfig enp0s31f6 down
dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6

All of this appeared to work.

I tried running the "skeleton" example program and got the following output:
sudo ./build/basicfwd
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
group bound to VFIO or unbound
EAL: Requested device 0000:00:1f.6 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Error - exiting with code: 1
  Cause: Error: number of ports must be even

I'm not at all familiar with DPDK or VFIO. What might the problem be?

-Alan

Re: Help Running Example

[Stephen Hemminger]

On Sun, 6 Aug 2023 11:33:43 -0400
Alan Beadle <ab.beadle@gmail.com> wrote:

> Hi,
> 
> I need some help getting DPDK working. I am running Ubuntu 20.04 with
> a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> 5.15 kernel with the same results.
> 
> Here is my NIC info from lspci:
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
> 
> I built and installed DPDK from source, and applied the following boot
> flags: "intel_iommu=on iommu=pt"
> 
> After booting I did the following as root:
> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> ifconfig enp0s31f6 down
> dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> 
> All of this appeared to work.
> 
> I tried running the "skeleton" example program and got the following output:
> sudo ./build/basicfwd
> EAL: Detected CPU lcores: 16
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> group bound to VFIO or unbound
> EAL: Requested device 0000:00:1f.6 cannot be used
> TELEMETRY: No legacy callbacks, legacy socket not created
> EAL: Error - exiting with code: 1
>   Cause: Error: number of ports must be even
> 
> I'm not at all familiar with DPDK or VFIO. What might the problem be?
> 
> -Alan

IOMMU groups are when multiple PCI devices share the same channel
in the IOMMU. The group is used to determine what mapping to use when
device does DMA. Since this is a security thing, devices in same IOMMU
group can not be shared between kernel and non-kernel usage.

The IOMMU group is determined by wiring on the motherboard.
Usually it is things like multiple Ethernet ports sharing the same group.
But can be much more confused.

The only option is to unbind all devices in the group before using
one with DPDK.

Re: Help Running Example

[Stephen Hemminger]

On Mon, 7 Aug 2023 12:40:21 -0700
Stephen Hemminger <stephen@networkplumber.org> wrote:

> On Sun, 6 Aug 2023 11:33:43 -0400
> Alan Beadle <ab.beadle@gmail.com> wrote:
> 
> > Hi,
> > 
> > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > 5.15 kernel with the same results.
> > 
> > Here is my NIC info from lspci:
> > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > I219-LM (rev 09)
> > 
> > I built and installed DPDK from source, and applied the following boot
> > flags: "intel_iommu=on iommu=pt"
> > 
> > After booting I did the following as root:
> > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > ifconfig enp0s31f6 down
> > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > 
> > All of this appeared to work.
> > 
> > I tried running the "skeleton" example program and got the following output:
> > sudo ./build/basicfwd
> > EAL: Detected CPU lcores: 16
> > EAL: Detected NUMA nodes: 1
> > EAL: Detected shared linkage of DPDK
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Selected IOVA mode 'VA'
> > EAL: VFIO support initialized
> > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > group bound to VFIO or unbound
> > EAL: Requested device 0000:00:1f.6 cannot be used
> > TELEMETRY: No legacy callbacks, legacy socket not created
> > EAL: Error - exiting with code: 1
> >   Cause: Error: number of ports must be even
> > 
> > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > 
> > -Alan  
> 
> IOMMU groups are when multiple PCI devices share the same channel
> in the IOMMU. The group is used to determine what mapping to use when
> device does DMA. Since this is a security thing, devices in same IOMMU
> group can not be shared between kernel and non-kernel usage.
> 
> The IOMMU group is determined by wiring on the motherboard.
> Usually it is things like multiple Ethernet ports sharing the same group.
> But can be much more confused.
> 
> The only option is to unbind all devices in the group before using
> one with DPDK.

More info on IOMMU groups is in kernel documentation:
https://www.kernel.org/doc/html/latest/driver-api/vfio.html

and in this article
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive

Fwd: Help Running Example

[Fuji Nafiul]

[-- Attachment #1: Type: text/plain, Size: 2607 bytes --]

---------- Forwarded message ---------
From: Fuji Nafiul <nafiul.fuji@gmail.com>
Date: Tue, Aug 8, 2023 at 9:22 AM
Subject: Re: Help Running Example
To: Alan Beadle <ab.beadle@gmail.com>


In the case of the " number of ports must be even " error, it's a simple
one. if you look into the code, you will see that it is checking for the
number of ports and it is giving an error if the port number is not even as
the skeleton app mainly receives packet on 1 port and forward them to a
different port (as far as I remember). So, for the sake of simplicity, if
you have odd numbers of ports or just a single port, don't worry, just
comment out the port number checking and you can even comment out the tx
part ( rte_eth_tx_burst() ) and rather free all received packets. Then run
and monitor the packet counting which will give you a kick start and later
you can try replying to arp requests to expose IP by which you will be able
to establish udp/tcp connection.. I suggest try udp first (I also only
worked with UDP in dpdk) and you will get a lot of help about this from
"bond" app in the official examples.

On Sun, Aug 6, 2023 at 9:34 PM Alan Beadle <ab.beadle@gmail.com> wrote:

> Hi,
>
> I need some help getting DPDK working. I am running Ubuntu 20.04 with
> a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> 5.15 kernel with the same results.
>
> Here is my NIC info from lspci:
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
>
> I built and installed DPDK from source, and applied the following boot
> flags: "intel_iommu=on iommu=pt"
>
> After booting I did the following as root:
> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> ifconfig enp0s31f6 down
> dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
>
> All of this appeared to work.
>
> I tried running the "skeleton" example program and got the following
> output:
> sudo ./build/basicfwd
> EAL: Detected CPU lcores: 16
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> group bound to VFIO or unbound
> EAL: Requested device 0000:00:1f.6 cannot be used
> TELEMETRY: No legacy callbacks, legacy socket not created
> EAL: Error - exiting with code: 1
>   Cause: Error: number of ports must be even
>
> I'm not at all familiar with DPDK or VFIO. What might the problem be?
>
> -Alan
>

[-- Attachment #2: Type: text/html, Size: 3323 bytes --]

Re: Help Running Example

[Fuji Nafiul]

[-- Attachment #1: Type: text/plain, Size: 2409 bytes --]

In the case of the " number of ports must be even " error, it's a simple
one. if you look into the code, you will see that it is checking for the
number of ports and it is giving an error if the port number is not even as
the skeleton app mainly receives packet on 1 port and forward them to a
different port (as far as I remember). So, for the sake of simplicity, if
you have odd numbers of ports or just a single port, don't worry, just
comment out the port number checking and you can even comment out the tx
part ( rte_eth_tx_burst() ) and rather free all received packets. Then run
and monitor the packet counting which will give you a kick start and later
you can try replying to arp requests to expose IP by which you will be able
to establish udp/tcp connection.. I suggest try udp first (I also only
worked with UDP in dpdk) and you will get a lot of help about this from
"bond" app in the official examples.

On Sun, Aug 6, 2023 at 9:34 PM Alan Beadle <ab.beadle@gmail.com> wrote:

> Hi,
>
> I need some help getting DPDK working. I am running Ubuntu 20.04 with
> a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> 5.15 kernel with the same results.
>
> Here is my NIC info from lspci:
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
>
> I built and installed DPDK from source, and applied the following boot
> flags: "intel_iommu=on iommu=pt"
>
> After booting I did the following as root:
> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> ifconfig enp0s31f6 down
> dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
>
> All of this appeared to work.
>
> I tried running the "skeleton" example program and got the following
> output:
> sudo ./build/basicfwd
> EAL: Detected CPU lcores: 16
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: VFIO support initialized
> EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> group bound to VFIO or unbound
> EAL: Requested device 0000:00:1f.6 cannot be used
> TELEMETRY: No legacy callbacks, legacy socket not created
> EAL: Error - exiting with code: 1
>   Cause: Error: number of ports must be even
>
> I'm not at all familiar with DPDK or VFIO. What might the problem be?
>
> -Alan
>

[-- Attachment #2: Type: text/html, Size: 2821 bytes --]

Re: Help Running Example

[Alan Beadle]

Thanks Stephen. It looks like my memory controller is in the same
IOMMU group. I assume this means I won't be able to do this with this
NIC?

-Alan

On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 7 Aug 2023 12:40:21 -0700
> Stephen Hemminger <stephen@networkplumber.org> wrote:
>
> > On Sun, 6 Aug 2023 11:33:43 -0400
> > Alan Beadle <ab.beadle@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > 5.15 kernel with the same results.
> > >
> > > Here is my NIC info from lspci:
> > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > I219-LM (rev 09)
> > >
> > > I built and installed DPDK from source, and applied the following boot
> > > flags: "intel_iommu=on iommu=pt"
> > >
> > > After booting I did the following as root:
> > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > ifconfig enp0s31f6 down
> > > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > >
> > > All of this appeared to work.
> > >
> > > I tried running the "skeleton" example program and got the following output:
> > > sudo ./build/basicfwd
> > > EAL: Detected CPU lcores: 16
> > > EAL: Detected NUMA nodes: 1
> > > EAL: Detected shared linkage of DPDK
> > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > EAL: Selected IOVA mode 'VA'
> > > EAL: VFIO support initialized
> > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > group bound to VFIO or unbound
> > > EAL: Requested device 0000:00:1f.6 cannot be used
> > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > EAL: Error - exiting with code: 1
> > >   Cause: Error: number of ports must be even
> > >
> > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > >
> > > -Alan
> >
> > IOMMU groups are when multiple PCI devices share the same channel
> > in the IOMMU. The group is used to determine what mapping to use when
> > device does DMA. Since this is a security thing, devices in same IOMMU
> > group can not be shared between kernel and non-kernel usage.
> >
> > The IOMMU group is determined by wiring on the motherboard.
> > Usually it is things like multiple Ethernet ports sharing the same group.
> > But can be much more confused.
> >
> > The only option is to unbind all devices in the group before using
> > one with DPDK.
>
> More info on IOMMU groups is in kernel documentation:
> https://www.kernel.org/doc/html/latest/driver-api/vfio.html
>
> and in this article
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive

Re: Help Running Example

[Alan Beadle]

Here is how I checked what other devices are in the same group as the NIC:

I ran this command as root:
dmesg|egrep group|awk '{print $NF" "$0}'|sort -n

Here is an excerpt of the output showing the group that the NIC is in:

10 [   17.029705] pci 0000:00:1f.0: Adding to iommu group 10
10 [   17.029732] pci 0000:00:1f.2: Adding to iommu group 10
10 [   17.029761] pci 0000:00:1f.3: Adding to iommu group 10
10 [   17.029788] pci 0000:00:1f.4: Adding to iommu group 10
10 [   17.029815] pci 0000:00:1f.5: Adding to iommu group 10
10 [   17.029842] pci 0000:00:1f.6: Adding to iommu group 10


And here is an excerpt of the lspci output showing what each of those
devices is:

00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI
Controller (rev 09)
00:1f.2 Memory controller: Intel Corporation C620 Series Chipset
Family Power Management Controller (rev 09)
00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09)
00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series
Chipset Family SPI Controller (rev 09)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
I219-LM (rev 09)

Based on this grouping, it seems like I can't feasibly unbind all of
these, unless I misunderstand something.

-Alan

On Tue, Aug 8, 2023 at 11:25 AM Alan Beadle <ab.beadle@gmail.com> wrote:
>
> Thanks Stephen. It looks like my memory controller is in the same
> IOMMU group. I assume this means I won't be able to do this with this
> NIC?
>
> -Alan
>
> On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 7 Aug 2023 12:40:21 -0700
> > Stephen Hemminger <stephen@networkplumber.org> wrote:
> >
> > > On Sun, 6 Aug 2023 11:33:43 -0400
> > > Alan Beadle <ab.beadle@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > > 5.15 kernel with the same results.
> > > >
> > > > Here is my NIC info from lspci:
> > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > > I219-LM (rev 09)
> > > >
> > > > I built and installed DPDK from source, and applied the following boot
> > > > flags: "intel_iommu=on iommu=pt"
> > > >
> > > > After booting I did the following as root:
> > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > > ifconfig enp0s31f6 down
> > > > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > > >
> > > > All of this appeared to work.
> > > >
> > > > I tried running the "skeleton" example program and got the following output:
> > > > sudo ./build/basicfwd
> > > > EAL: Detected CPU lcores: 16
> > > > EAL: Detected NUMA nodes: 1
> > > > EAL: Detected shared linkage of DPDK
> > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > EAL: Selected IOVA mode 'VA'
> > > > EAL: VFIO support initialized
> > > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > > group bound to VFIO or unbound
> > > > EAL: Requested device 0000:00:1f.6 cannot be used
> > > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > > EAL: Error - exiting with code: 1
> > > >   Cause: Error: number of ports must be even
> > > >
> > > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > > >
> > > > -Alan
> > >
> > > IOMMU groups are when multiple PCI devices share the same channel
> > > in the IOMMU. The group is used to determine what mapping to use when
> > > device does DMA. Since this is a security thing, devices in same IOMMU
> > > group can not be shared between kernel and non-kernel usage.
> > >
> > > The IOMMU group is determined by wiring on the motherboard.
> > > Usually it is things like multiple Ethernet ports sharing the same group.
> > > But can be much more confused.
> > >
> > > The only option is to unbind all devices in the group before using
> > > one with DPDK.
> >
> > More info on IOMMU groups is in kernel documentation:
> > https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> >
> > and in this article
> > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive

Re: Help Running Example

[Stephen Hemminger]

On Tue, 8 Aug 2023 11:31:52 -0400
Alan Beadle <ab.beadle@gmail.com> wrote:

> Here is how I checked what other devices are in the same group as the NIC:
> 
> I ran this command as root:
> dmesg|egrep group|awk '{print $NF" "$0}'|sort -n
> 
> Here is an excerpt of the output showing the group that the NIC is in:
> 
> 10 [   17.029705] pci 0000:00:1f.0: Adding to iommu group 10
> 10 [   17.029732] pci 0000:00:1f.2: Adding to iommu group 10
> 10 [   17.029761] pci 0000:00:1f.3: Adding to iommu group 10
> 10 [   17.029788] pci 0000:00:1f.4: Adding to iommu group 10
> 10 [   17.029815] pci 0000:00:1f.5: Adding to iommu group 10
> 10 [   17.029842] pci 0000:00:1f.6: Adding to iommu group 10
> 
> 
> And here is an excerpt of the lspci output showing what each of those
> devices is:
> 
> 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI
> Controller (rev 09)
> 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset
> Family Power Management Controller (rev 09)
> 00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09)
> 00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
> 00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series
> Chipset Family SPI Controller (rev 09)
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
> 
> Based on this grouping, it seems like I can't feasibly unbind all of
> these, unless I misunderstand something.
> 
> -Alan
> 
> On Tue, Aug 8, 2023 at 11:25 AM Alan Beadle <ab.beadle@gmail.com> wrote:
> >
> > Thanks Stephen. It looks like my memory controller is in the same
> > IOMMU group. I assume this means I won't be able to do this with this
> > NIC?
> >
> > -Alan
> >
> > On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:  
> > >
> > > On Mon, 7 Aug 2023 12:40:21 -0700
> > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >  
> > > > On Sun, 6 Aug 2023 11:33:43 -0400
> > > > Alan Beadle <ab.beadle@gmail.com> wrote:
> > > >  
> > > > > Hi,
> > > > >
> > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > > > 5.15 kernel with the same results.
> > > > >
> > > > > Here is my NIC info from lspci:
> > > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > > > I219-LM (rev 09)
> > > > >
> > > > > I built and installed DPDK from source, and applied the following boot
> > > > > flags: "intel_iommu=on iommu=pt"
> > > > >
> > > > > After booting I did the following as root:
> > > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > > > ifconfig enp0s31f6 down
> > > > > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > > > >
> > > > > All of this appeared to work.
> > > > >
> > > > > I tried running the "skeleton" example program and got the following output:
> > > > > sudo ./build/basicfwd
> > > > > EAL: Detected CPU lcores: 16
> > > > > EAL: Detected NUMA nodes: 1
> > > > > EAL: Detected shared linkage of DPDK
> > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > EAL: Selected IOVA mode 'VA'
> > > > > EAL: VFIO support initialized
> > > > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > > > group bound to VFIO or unbound
> > > > > EAL: Requested device 0000:00:1f.6 cannot be used
> > > > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > > > EAL: Error - exiting with code: 1
> > > > >   Cause: Error: number of ports must be even
> > > > >
> > > > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > > > >
> > > > > -Alan  
> > > >
> > > > IOMMU groups are when multiple PCI devices share the same channel
> > > > in the IOMMU. The group is used to determine what mapping to use when
> > > > device does DMA. Since this is a security thing, devices in same IOMMU
> > > > group can not be shared between kernel and non-kernel usage.
> > > >
> > > > The IOMMU group is determined by wiring on the motherboard.
> > > > Usually it is things like multiple Ethernet ports sharing the same group.
> > > > But can be much more confused.
> > > >
> > > > The only option is to unbind all devices in the group before using
> > > > one with DPDK.  
> > >
> > > More info on IOMMU groups is in kernel documentation:
> > > https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> > >
> > > and in this article
> > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive  


Right you need to find a different system, use a VM or add an external NIC to use DPDK.

Re: Help Running Example

[Alan Beadle]

I'm going to get a dedicated NIC so that I can continue on the other
machine, but in the meantime I discovered that one of my old laptops
places the ethernet NIC in a unique IOMMU group:

02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetLink
BCM57780 Gigabit Ethernet PCIe (rev 01)
1 [    1.726704] pci 0000:00:01.0: Adding to iommu group 1
1 [    1.726892] pci 0000:01:00.0: Adding to iommu group 1
2 [    1.726720] pci 0000:00:02.0: Adding to iommu group 2
3 [    1.726734] pci 0000:00:14.0: Adding to iommu group 3
4 [    1.726751] pci 0000:00:16.0: Adding to iommu group 4
5 [    1.726764] pci 0000:00:1a.0: Adding to iommu group 5
6 [    1.726778] pci 0000:00:1b.0: Adding to iommu group 6
7 [    1.726792] pci 0000:00:1c.0: Adding to iommu group 7
8 [    1.726805] pci 0000:00:1c.1: Adding to iommu group 8
9 [    1.726818] pci 0000:00:1c.2: Adding to iommu group 9
10 [    1.726833] pci 0000:00:1d.0: Adding to iommu group 10
11 [    1.726859] pci 0000:00:1f.0: Adding to iommu group 11
11 [    1.726872] pci 0000:00:1f.2: Adding to iommu group 11
11 [    1.726886] pci 0000:00:1f.3: Adding to iommu group 11
12 [    1.726905] pci 0000:02:00.0: Adding to iommu group 12
13 [    1.726920] pci 0000:03:00.0: Adding to iommu group 13
14 [    1.726933] pci 0000:04:00.0: Adding to iommu group 14

After following the same steps on this machine, and commenting out the
port counting as suggested, I have a new error:

EAL: Detected CPU lcores: 4
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Error - exiting with code: 1
  Cause: Cannot create mbuf pool

Based on my research, it looks like this can be caused by a lack of hugepages.
Here is an excerpt from /proc/meminfo:

HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0

Which seems fine to me. Any idea what might be causing this problem?

Thanks again.

On Mon, Aug 7, 2023 at 11:25 PM Fuji Nafiul <nafiul.fuji@gmail.com> wrote:
>
> In the case of the " number of ports must be even " error, it's a simple one. if you look into the code, you will see that it is checking for the number of ports and it is giving an error if the port number is not even as the skeleton app mainly receives packet on 1 port and forward them to a different port (as far as I remember). So, for the sake of simplicity, if you have odd numbers of ports or just a single port, don't worry, just comment out the port number checking and you can even comment out the tx part ( rte_eth_tx_burst() ) and rather free all received packets. Then run and monitor the packet counting which will give you a kick start and later you can try replying to arp requests to expose IP by which you will be able to establish udp/tcp connection.. I suggest try udp first (I also only worked with UDP in dpdk) and you will get a lot of help about this from "bond" app in the official examples.
>
> On Sun, Aug 6, 2023 at 9:34 PM Alan Beadle <ab.beadle@gmail.com> wrote:
>>
>> Hi,
>>
>> I need some help getting DPDK working. I am running Ubuntu 20.04 with
>> a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
>> 5.15 kernel with the same results.
>>
>> Here is my NIC info from lspci:
>> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
>> I219-LM (rev 09)
>>
>> I built and installed DPDK from source, and applied the following boot
>> flags: "intel_iommu=on iommu=pt"
>>
>> After booting I did the following as root:
>> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>> ifconfig enp0s31f6 down
>> dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
>>
>> All of this appeared to work.
>>
>> I tried running the "skeleton" example program and got the following output:
>> sudo ./build/basicfwd
>> EAL: Detected CPU lcores: 16
>> EAL: Detected NUMA nodes: 1
>> EAL: Detected shared linkage of DPDK
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Selected IOVA mode 'VA'
>> EAL: VFIO support initialized
>> EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
>> group bound to VFIO or unbound
>> EAL: Requested device 0000:00:1f.6 cannot be used
>> TELEMETRY: No legacy callbacks, legacy socket not created
>> EAL: Error - exiting with code: 1
>>   Cause: Error: number of ports must be even
>>
>> I'm not at all familiar with DPDK or VFIO. What might the problem be?
>>
>> -Alan