From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2661A4300D for ; Tue, 8 Aug 2023 17:32:06 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E48DA43247; Tue, 8 Aug 2023 17:32:05 +0200 (CEST) Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) by mails.dpdk.org (Postfix) with ESMTP id D6CDB40A87 for ; Tue, 8 Aug 2023 17:32:04 +0200 (CEST) Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-56d6dfa1f3cso1496337eaf.0 for ; Tue, 08 Aug 2023 08:32:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691508724; x=1692113524; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Am7/v1mClbZ2eXE/X9LzChTROlAchhB4XweklKz2ToQ=; b=pF3jI/5tfTGL2c+uRenAcnqu3b34hNSvJ0gS6AirhRwL7y+N2ZNYRyuuCS2XH9EYeU q1qwBpbI3fOjnuIPdZn1mSMZMvZ0+Qoan2iOUx0+BaRPmsHbcJF9KtE16ig9/OyOHNi0 ZpIZdSGtbfyYMG30/u/aLD7fSIqJ9c+ALO7h5ix0f5ysO0kBbMTS4VIv8JJAIrY9qhPI Q0oK5OSFoCSwYQMO+tbX8m+lP/hfb68dASmKHop6DAhdMmSytkLo3oyCeT3ZhY19KlWh oCsvxuxHBMECfQQlHwPWX/Bf7pcIs3GCyWc3WATFob4u7z/iTjw4NPwfluNuJ2pNuqXK olag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691508724; x=1692113524; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Am7/v1mClbZ2eXE/X9LzChTROlAchhB4XweklKz2ToQ=; b=a1vjbV58In2yYQwc2OBYedtg+7RxFFrg5LfWz/cv5VdciMmAoPnbQWzI/7mZhNPkHw ZtN6ixALf2XRC5JFr8SvsuVi8LtZHWVwOGVCOwwZgO8Fm9/6+XEU/TlhvnPNGFbsD34Q LlmvaYZif0ELJie1pCehDC3VpHMeXPPNJRpWYJ4E85kaXRjL7FU0YgtxBmQep7Huy9wG IRsmqFoDY3KeCeN+2SjYTkAWOuW5OnmqDpM2gi2OV1UiqOwUxEGbRGfwrUHbOYDeATLD 4dHmOfrmAA7+QILMyW9USbAhq1kH/mbJGa78A45kDc5gdTkyA3OSqCwZSW6+Pjtzmxyl gCjA== X-Gm-Message-State: AOJu0YxYyaJZlXtKsXvb9Uz6p33YDmj54Jn2AYIm5qsBphZJqyPaLKXX gY5FooPJxeq88ZcCi5PfNtYwUc5BV2ElBjC3iNs= X-Google-Smtp-Source: AGHT+IHC3sS0+vbhOZV2ax5aw4e2HG89L+KbPtmCu3NhKSdvkVlaNoYOYyeGkC2EHcAZ2sBHu9/pcR4FB83yqyOVXIo= X-Received: by 2002:a4a:9099:0:b0:56d:2229:1ade with SMTP id j25-20020a4a9099000000b0056d22291ademr147696oog.8.1691508723988; Tue, 08 Aug 2023 08:32:03 -0700 (PDT) MIME-Version: 1.0 References: <20230807124021.325b6d68@hermes.local> <20230807172645.3288f7e4@hermes.local> In-Reply-To: From: Alan Beadle Date: Tue, 8 Aug 2023 11:31:52 -0400 Message-ID: Subject: Re: Help Running Example To: Stephen Hemminger Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Here is how I checked what other devices are in the same group as the NIC: I ran this command as root: dmesg|egrep group|awk '{print $NF" "$0}'|sort -n Here is an excerpt of the output showing the group that the NIC is in: 10 [ 17.029705] pci 0000:00:1f.0: Adding to iommu group 10 10 [ 17.029732] pci 0000:00:1f.2: Adding to iommu group 10 10 [ 17.029761] pci 0000:00:1f.3: Adding to iommu group 10 10 [ 17.029788] pci 0000:00:1f.4: Adding to iommu group 10 10 [ 17.029815] pci 0000:00:1f.5: Adding to iommu group 10 10 [ 17.029842] pci 0000:00:1f.6: Adding to iommu group 10 And here is an excerpt of the lspci output showing what each of those devices is: 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI Controller (rev 09) 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset Family Power Management Controller (rev 09) 00:1f.3 Audio device: Intel Corporation Device a1f0 (rev 09) 00:1f.4 SMBus: Intel Corporation C620 Series Chipset Family SMBus (rev 09) 00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series Chipset Family SPI Controller (rev 09) 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (3) I219-LM (rev 09) Based on this grouping, it seems like I can't feasibly unbind all of these, unless I misunderstand something. -Alan On Tue, Aug 8, 2023 at 11:25=E2=80=AFAM Alan Beadle w= rote: > > Thanks Stephen. It looks like my memory controller is in the same > IOMMU group. I assume this means I won't be able to do this with this > NIC? > > -Alan > > On Mon, Aug 7, 2023 at 8:26=E2=80=AFPM Stephen Hemminger > wrote: > > > > On Mon, 7 Aug 2023 12:40:21 -0700 > > Stephen Hemminger wrote: > > > > > On Sun, 6 Aug 2023 11:33:43 -0400 > > > Alan Beadle wrote: > > > > > > > Hi, > > > > > > > > I need some help getting DPDK working. I am running Ubuntu 20.04 wi= th > > > > a modified Linux 5.4 kernel, but I have also tried the stock Ubuntu > > > > 5.15 kernel with the same results. > > > > > > > > Here is my NIC info from lspci: > > > > 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection = (3) > > > > I219-LM (rev 09) > > > > > > > > I built and installed DPDK from source, and applied the following b= oot > > > > flags: "intel_iommu=3Don iommu=3Dpt" > > > > > > > > After booting I did the following as root: > > > > echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > > > ifconfig enp0s31f6 down > > > > dpdk-devbind.py --bind=3Dvfio-pci 0000:00:1f.6 > > > > > > > > All of this appeared to work. > > > > > > > > I tried running the "skeleton" example program and got the followin= g output: > > > > sudo ./build/basicfwd > > > > EAL: Detected CPU lcores: 16 > > > > EAL: Detected NUMA nodes: 1 > > > > EAL: Detected shared linkage of DPDK > > > > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > > > > EAL: Selected IOVA mode 'VA' > > > > EAL: VFIO support initialized > > > > EAL: 0000:00:1f.6 VFIO group is not viable! Not all devices in IOMM= U > > > > group bound to VFIO or unbound > > > > EAL: Requested device 0000:00:1f.6 cannot be used > > > > TELEMETRY: No legacy callbacks, legacy socket not created > > > > EAL: Error - exiting with code: 1 > > > > Cause: Error: number of ports must be even > > > > > > > > I'm not at all familiar with DPDK or VFIO. What might the problem b= e? > > > > > > > > -Alan > > > > > > IOMMU groups are when multiple PCI devices share the same channel > > > in the IOMMU. The group is used to determine what mapping to use when > > > device does DMA. Since this is a security thing, devices in same IOMM= U > > > group can not be shared between kernel and non-kernel usage. > > > > > > The IOMMU group is determined by wiring on the motherboard. > > > Usually it is things like multiple Ethernet ports sharing the same gr= oup. > > > But can be much more confused. > > > > > > The only option is to unbind all devices in the group before using > > > one with DPDK. > > > > More info on IOMMU groups is in kernel documentation: > > https://www.kernel.org/doc/html/latest/driver-api/vfio.html > > > > and in this article > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/= 7/html/virtualization_deployment_and_administration_guide/sect-iommu-deep-d= ive