From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D3BF74300E for ; Tue, 8 Aug 2023 18:01:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5123643247; Tue, 8 Aug 2023 18:01:54 +0200 (CEST) Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by mails.dpdk.org (Postfix) with ESMTP id 6719E40A87 for ; Tue, 8 Aug 2023 18:01:53 +0200 (CEST) Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-56334abe325so3461614a12.3 for ; Tue, 08 Aug 2023 09:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1691510512; x=1692115312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=8YQOS0LrZLdc8tleA3BmoIAwxvfLVLpr3kff5T0r53E=; b=NLbQ2JxBYV9OZrL1TMXUirKVttUChWLqi0DDzFp73UREnGtXwPW1lwEkzwgHdcvQZr lHvtffA969GWh21Sr9evR9XiKDyT9TrSHFNsrZNWdXCvzwIVYoNHXEtM4LO8fsQtWay5 izEjCNUM5x0Ndr+IrWKrmOBVRQlp5QudURusq7LtC9AKG6ZCyiwCdoqSWXzBA9HAIOqN PtqTov35UnlBNlNL+tFEu+tcEfVSrutAp5j2bkDUn/xU33sBYgsUmemwlvphahPM2OXk BVqgcfMMDohNbvsXQ9OBV2fNVZt7a/Q+qSyHKnzpCPwiVb6Py69qNvW/OLUK+0+Ii60k 2U9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691510512; x=1692115312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8YQOS0LrZLdc8tleA3BmoIAwxvfLVLpr3kff5T0r53E=; b=ItAaZJfXnh9Z4jpH3v+58tHd3Bde1gmDuKsghTpi9PHWF+x9RhC5ABeFKige0D6N38 7+2id94QtFhvKQE2na8LjTr2GpNXSInXCo2Yi97/H0mw3X4AcfE2ac/KgMhXoGNZIaZI WhdzWllThVoIxuUQqV/84ntrdLidDs4Qyohi+vzDeHrLsnBWOeJHoS4+rBz928asWkkx 7rF6hKAJFwwA9zvH0H5JMRPRHxjU3wxLrY0dzmJ7np5UA8KesFReRBRwt8ZEkKkCtGH6 j+GNpvmSrELrqXnu3j2k2uoT4gwZXd8uhgdz2P8fc/YKXcAHfLjXI/CJbe9r7+xOCe0L oilA== X-Gm-Message-State: AOJu0YyYEIKO983fG21QXuHkHNwsyZWmRoSQguqEDLGo/Jb6skLhiH7U F1o9eVQmvOOkxuTO1lEz6fN1Sg== X-Google-Smtp-Source: AGHT+IGkT8SYG2IAkctSQvb9tHKwqDiUBfl9J2RwGoZRIEOH3seM9tulOIwc5Yz/YMipHdfaCd8y7A== X-Received: by 2002:a17:90a:5385:b0:263:fbe5:2125 with SMTP id y5-20020a17090a538500b00263fbe52125mr9885620pjh.15.1691510512193; Tue, 08 Aug 2023 09:01:52 -0700 (PDT) Received: from hermes.local (204-195-127-207.wavecable.com. [204.195.127.207]) by smtp.gmail.com with ESMTPSA id h23-20020a17090aa89700b002694fee879csm4790749pjq.36.2023.08.08.09.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 09:01:52 -0700 (PDT) Date: Tue, 8 Aug 2023 09:01:49 -0700 From: Stephen Hemminger To: Alan Beadle Cc: users@dpdk.org Subject: Re: Help Running Example Message-ID: <20230808090149.52634af3@hermes.local> In-Reply-To: References: <20230807124021.325b6d68@hermes.local> <20230807172645.3288f7e4@hermes.local> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Tue, 8 Aug 2023 11:31:52 -0400 Alan Beadle wrote: > Here is how I checked what other devices are in the same group as the NIC: >=20 > I ran this command as root: > dmesg|egrep group|awk '{print $NF" "$0}'|sort -n >=20 > Here is an excerpt of the output showing the group that the NIC is in: >=20 > 10 [ 17.029705] pci 0000:00:1f.0: Adding to iommu group 10 > 10 [ 17.029732] pci 0000:00:1f.2: Adding to iommu group 10 > 10 [ 17.029761] pci 0000:00:1f.3: Adding to iommu group 10 > 10 [ 17.029788] pci 0000:00:1f.4: Adding to iommu group 10 > 10 [ 17.029815] pci 0000:00:1f.5: Adding to iommu group 10 > 10 [ 17.029842] pci 0000:00:1f.6: Adding to iommu group 10 >=20 >=20 > And here is an excerpt of the lspci output showing what each of those > devices is: >=20 > 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI > Controller (rev 09) > 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset > Family Power Management Controller (rev 09) > 00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09) > 00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09) > 00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series > Chipset Family SPI Controller (rev 09) > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) > I219-LM (rev 09) >=20 > Based on this grouping, it seems like I can't feasibly unbind all of > these, unless I misunderstand something. >=20 > -Alan >=20 > On Tue, Aug 8, 2023 at 11:25=E2=80=AFAM Alan Beadle = wrote: > > > > Thanks Stephen. It looks like my memory controller is in the same > > IOMMU group. I assume this means I won't be able to do this with this > > NIC? > > > > -Alan > > > > On Mon, Aug 7, 2023 at 8:26=E2=80=AFPM Stephen Hemminger > > wrote: =20 > > > > > > On Mon, 7 Aug 2023 12:40:21 -0700 > > > Stephen Hemminger wrote: > > > =20 > > > > On Sun, 6 Aug 2023 11:33:43 -0400 > > > > Alan Beadle wrote: > > > > =20 > > > > > Hi, > > > > > > > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 = with > > > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubun= tu > > > > > 5.15 kernel with the same results. > > > > > > > > > > Here is my NIC info from lspci: > > > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connectio= n (3) > > > > > I219-LM (rev 09) > > > > > > > > > > I built and installed DPDK from source, and applied the following= boot > > > > > flags: "intel_iommu=3Don iommu=3Dpt" > > > > > > > > > > After booting I did the following as root: > > > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > > > > ifconfig enp0s31f6 down > > > > > dpdk-devbind.py --bind=3Dvfio-pci 0000:00:1f.6 > > > > > > > > > > All of this appeared to work. > > > > > > > > > > I tried running the "skeleton" example program and got the follow= ing output: > > > > > sudo ./build/basicfwd > > > > > EAL: Detected CPU lcores: 16 > > > > > EAL: Detected NUMA nodes: 1 > > > > > EAL: Detected shared linkage of DPDK > > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > > EAL: Selected IOVA mode 'VA' > > > > > EAL: VFIO support initialized > > > > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IO= MMU > > > > > group bound to VFIO or unbound > > > > > EAL: Requested device 0000:00:1f.6 cannot be used > > > > > TELEMETRY: No legacy callbacks, legacy socket not created > > > > > EAL: Error - exiting with code: 1 > > > > > Cause: Error: number of ports must be even > > > > > > > > > > I'm not at all familiar with DPDK or VFIO. What might the problem= be? > > > > > > > > > > -Alan =20 > > > > > > > > IOMMU groups are when multiple PCI devices share the same channel > > > > in the IOMMU. The group is used to determine what mapping to use wh= en > > > > device does DMA. Since this is a security thing, devices in same IO= MMU > > > > group can not be shared between kernel and non-kernel usage. > > > > > > > > The IOMMU group is determined by wiring on the motherboard. > > > > Usually it is things like multiple Ethernet ports sharing the same = group. > > > > But can be much more confused. > > > > > > > > The only option is to unbind all devices in the group before using > > > > one with DPDK. =20 > > > > > > More info on IOMMU groups is in kernel documentation: > > > https://www.kernel.org/doc/html/latest/driver-api/vfio.html > > > > > > and in this article > > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linu= x/7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep= -dive =20 Right you need to find a different system, use a VM or add an external NIC = to use DPDK.