DPDK usage discussions
 help / color / mirror / Atom feed
From: Stephen Hemminger <stephen@networkplumber.org>
To: Alan Beadle <ab.beadle@gmail.com>
Cc: users@dpdk.org
Subject: Re: Help Running Example
Date: Tue, 8 Aug 2023 09:01:49 -0700	[thread overview]
Message-ID: <20230808090149.52634af3@hermes.local> (raw)
In-Reply-To: <CANTAOdyO73KNUXAfp1=5FrQ2E6k3wPzPavP=URnSApppXu1SKA@mail.gmail.com>

On Tue, 8 Aug 2023 11:31:52 -0400
Alan Beadle <ab.beadle@gmail.com> wrote:

> Here is how I checked what other devices are in the same group as the NIC:
> 
> I ran this command as root:
> dmesg|egrep group|awk '{print $NF" "$0}'|sort -n
> 
> Here is an excerpt of the output showing the group that the NIC is in:
> 
> 10 [   17.029705] pci 0000:00:1f.0: Adding to iommu group 10
> 10 [   17.029732] pci 0000:00:1f.2: Adding to iommu group 10
> 10 [   17.029761] pci 0000:00:1f.3: Adding to iommu group 10
> 10 [   17.029788] pci 0000:00:1f.4: Adding to iommu group 10
> 10 [   17.029815] pci 0000:00:1f.5: Adding to iommu group 10
> 10 [   17.029842] pci 0000:00:1f.6: Adding to iommu group 10
> 
> 
> And here is an excerpt of the lspci output showing what each of those
> devices is:
> 
> 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI
> Controller (rev 09)
> 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset
> Family Power Management Controller (rev 09)
> 00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09)
> 00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09)
> 00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series
> Chipset Family SPI Controller (rev 09)
> 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> I219-LM (rev 09)
> 
> Based on this grouping, it seems like I can't feasibly unbind all of
> these, unless I misunderstand something.
> 
> -Alan
> 
> On Tue, Aug 8, 2023 at 11:25 AM Alan Beadle <ab.beadle@gmail.com> wrote:
> >
> > Thanks Stephen. It looks like my memory controller is in the same
> > IOMMU group. I assume this means I won't be able to do this with this
> > NIC?
> >
> > -Alan
> >
> > On Mon, Aug 7, 2023 at 8:26 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:  
> > >
> > > On Mon, 7 Aug 2023 12:40:21 -0700
> > > Stephen Hemminger <stephen@networkplumber.org> wrote:
> > >  
> > > > On Sun, 6 Aug 2023 11:33:43 -0400
> > > > Alan Beadle <ab.beadle@gmail.com> wrote:
> > > >  
> > > > > Hi,
> > > > >
> > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 with
> > > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu
> > > > > 5.15 kernel with the same results.
> > > > >
> > > > > Here is my NIC info from lspci:
> > > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3)
> > > > > I219-LM (rev 09)
> > > > >
> > > > > I built and installed DPDK from source, and applied the following boot
> > > > > flags: "intel_iommu=on iommu=pt"
> > > > >
> > > > > After booting I did the following as root:
> > > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> > > > > ifconfig enp0s31f6 down
> > > > > dpdk-devbind.py --bind=vfio-pci 0000:00:1f.6
> > > > >
> > > > > All of this appeared to work.
> > > > >
> > > > > I tried running the "skeleton" example program and got the following output:
> > > > > sudo ./build/basicfwd
> > > > > EAL: Detected CPU lcores: 16
> > > > > EAL: Detected NUMA nodes: 1
> > > > > EAL: Detected shared linkage of DPDK
> > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > > > > EAL: Selected IOVA mode 'VA'
> > > > > EAL: VFIO support initialized
> > > > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMMU
> > > > > group bound to VFIO or unbound
> > > > > EAL: Requested device 0000:00:1f.6 cannot be used
> > > > > TELEMETRY: No legacy callbacks, legacy socket not created
> > > > > EAL: Error - exiting with code: 1
> > > > >   Cause: Error: number of ports must be even
> > > > >
> > > > > I'm not at all familiar with DPDK or VFIO. What might the problem be?
> > > > >
> > > > > -Alan  
> > > >
> > > > IOMMU groups are when multiple PCI devices share the same channel
> > > > in the IOMMU. The group is used to determine what mapping to use when
> > > > device does DMA. Since this is a security thing, devices in same IOMMU
> > > > group can not be shared between kernel and non-kernel usage.
> > > >
> > > > The IOMMU group is determined by wiring on the motherboard.
> > > > Usually it is things like multiple Ethernet ports sharing the same group.
> > > > But can be much more confused.
> > > >
> > > > The only option is to unbind all devices in the group before using
> > > > one with DPDK.  
> > >
> > > More info on IOMMU groups is in kernel documentation:
> > > https://www.kernel.org/doc/html/latest/driver-api/vfio.html
> > >
> > > and in this article
> > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-dive  


Right you need to find a different system, use a VM or add an external NIC to use DPDK.

  reply	other threads:[~2023-08-08 16:01 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-06 15:33 Alan Beadle
2023-08-07 19:40 ` Stephen Hemminger
2023-08-08  0:26   ` Stephen Hemminger
2023-08-08 15:25     ` Alan Beadle
2023-08-08 15:31       ` Alan Beadle
2023-08-08 16:01         ` Stephen Hemminger [this message]
     [not found] ` <CA+3hWewvP9xSncfK4vcUz+KH7cQsO7hjXU4=hgDdntHNjfJ2-w@mail.gmail.com>
2023-08-08  3:23   ` Fwd: " Fuji Nafiul
2023-08-08  3:25 ` Fuji Nafiul
2023-08-09 17:35   ` Alan Beadle

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230808090149.52634af3@hermes.local \
    --to=stephen@networkplumber.org \
    --cc=ab.beadle@gmail.com \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).