From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6E95548945 for ; Wed, 15 Oct 2025 19:40:11 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4825B402E0; Wed, 15 Oct 2025 19:40:11 +0200 (CEST) Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by mails.dpdk.org (Postfix) with ESMTP id C943C40273 for ; Wed, 15 Oct 2025 19:40:09 +0200 (CEST) Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-b57d93ae3b0so4172834a12.1 for ; Wed, 15 Oct 2025 10:40:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1760550009; x=1761154809; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=TuhPLi4RvqYvj7ZmxKsFxECmZAx46ceosGi9Wa7hOrk=; b=z4n1Lh7F6R7bWPRenOmn/JGrqi2Kdn29Owh46lVzL7zrB0yQGkD4bHcFz/dUwFISf6 n9L4z28hUJEKdcitbelQ/XCni+Pfko2Ki8gDtuboOPJDqo7igMwCqemwfzjemVhqIXYg c3vz++J/DH7ooKotlTmAu8zxcZbzYLbCjgEoYXXhXQe3mItpmMFBoCLCTUKbrFdQlD8S CW0Rgc9DFZVwDjIr5XLNRizvuvyEGH/oBld521CfOBh6acFIrZLbuYJN2h1tXi+WKjJZ 3xatDXhKUU7lDfX5VUTit3Ib05tQX/6nwIwZO8u6ImbLrqPdSXfH6Qsk0Enq+8S2MGW4 qcaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760550009; x=1761154809; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TuhPLi4RvqYvj7ZmxKsFxECmZAx46ceosGi9Wa7hOrk=; b=tHmwMx9AkNaWhluGLxW81EEsr8cfq+HgncerMG6a1ioy9MRhJHNj5xASZekeRQRIjy Bmtoy36Jw7Ca2/xinmJjiDbZ8MvT7ccIATzEtXw5tnHzVdXVgWaJiao5vBYy02lf1IKk OUijyIFQEPALBPFR8kUjTqR//eXFYJrgkNbzsGeVzZdjWhVjhsdcNpIENS4iDH1miWYa 8+Lg04SYBQtEKR43kFMv/qRhFa7VgjeIDlRS/PbfzZyxT1aA8B1K6aWxYI4WBTWyFwJA fQDz+28Ef3oWHldTk6h+3/xar6hy5JY7QMu+lPcTPU16R0wXJLomWBgTZTPfwIo28jAS AiiQ== X-Gm-Message-State: AOJu0YyLadAVgfOSkpToqTdNJNQHLnpfR9OFkOB9LbXnCr4n/QiaYiUT E3e0QlyQPISk4PJRlPjkKAoYe6QeFx8H5X4Ly9XnGS215DVPMDrTD/nglBQQYMhbgRM= X-Gm-Gg: ASbGncsQSILmiOEG/NtMoMLxmYjAbpdsvV1vBFBx/Wvok1rg2gZsbGBXnlJzPOMTnir GlaRKfQdjvXIn0QBOJ5yTgVRm4rCZ6HoDBSQIH7Ls+up2/FK2NDoD5uxeZrhDKvoEkRVObYGL3I bISR6hyJvuj3IrZahH+tdK/ew0sb6Mc/mkXdVPsyEReLGDO7D53trS8hiPK/53PaYTYToY39FpL jcOrOMP0/Fmva/h7GPbiD6seV0uruepdnnq3ZkfED2oPZJtE0DvA+iiKiYd3qh2fnChzO55uiS+ xH4yiNfFhpKtjmF1u71a+m2C8eGMjWWoRCitny2uBJ+IfQfda0y4ahHC4Sd27RDYa9bFUuxdaTj 2N/PIdVA9lNY8UITTFW8LpOeZgJWAxm3+80pIQ4bJmR981dgHJcrMWuCaMatB7l/3rSZ6X6u+Cn cTYsr/fwVcOj9aJcH4iSrkS7rQzw0l X-Google-Smtp-Source: AGHT+IHocMNYsSaos2yYlb8lphuNsTMmtkEQMOpoVU0w5eAUVs4HgGALrD+YEce5D2NZguApRWGM2g== X-Received: by 2002:a17:903:2a87:b0:25a:24f2:af00 with SMTP id d9443c01a7336-2902721305cmr353091945ad.12.1760550008736; Wed, 15 Oct 2025 10:40:08 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-290993466absm2533005ad.35.2025.10.15.10.40.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Oct 2025 10:40:08 -0700 (PDT) Date: Wed, 15 Oct 2025 10:40:06 -0700 From: Stephen Hemminger To: Oleksandr Nahnybida Cc: users , Vitaliy Ivanov , Taras Bilous Subject: Re: How to use vfio-pci in vmware VM with PCI passthrough? Message-ID: <20251015104006.319fa83a@hermes.local> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Thu, 2 Oct 2025 17:40:07 +0300 Oleksandr Nahnybida wrote: > Hi all, > > We are working on migrating our DPDK application from igb_uio to vfio-pci. > Our target environment is a VMware ESXi host running on an AMD Epyc server > with NICs configured for PCI Passthrough to a guest VM with Debian Bookworm > (Kernel 6.1.0-39-amd64) > > We've encountered a couple of issues. > > Problem 1: > > Initially, attempting to use vfio-pci failed with an error code of -22, and > the /sys/class/iommu/ directory was empty. We discovered the "expose IOMMU > to guest OS" option in VMware and enabled it. > > This led to a new error: > "The virtual machine cannot be powered on because IOMMU virtualization is > not compatible with PCI passthru on AMD platforms" > > We found a workaround by adding amd.iommu.supportsPcip = "TRUE" to the VM's > configuration. The VM now boots, and the IOMMU is visible in the guest. > > However, when we run our DPDK application, it hangs after printing "EAL: > VFIO support initialized", and shortly after, the guest kernel panics with > a soft lockup error, making the system eventually unresponsive. > BUG: soft lockup - CPU#34 stuck for 75s! [kcompactd0:529] > > Problem 2: > > Separately, we've noticed that our IOMMU groups are not ideal. Many groups > contain not only the NICs we need to bind, but also other devices like PCI > bridges. > > IOMMU Group 7: > 0000:00:17.0 - PCI bridge: VMware PCI Express Root Port > 0000:00:17.1 > 0000:00:17.2 > 0000:00:17.3 > 0000:00:17.4 > 0000:00:17.5 > 0000:00:17.6 > 0000:00:17.7 > 0000:13:00.0 - nic > 0000:13:00.1 - nic > 0000:14:00.0 - nic > 0000:14:00.1 - nic > > Questions: > > 1. Is enabling guest IOMMU virtualization in VMware with the > amd.iommu.supportsPcip workaround, the correct approach here? > 2. I see that vfio-pci can be run in unsafe mode, but is there any > benefit to using it over igb_uio in this case? > 1. In my understanding, the hypervisor is using actual hardware IOMMU > to implement PCI pass-through anyway, so what's the point of having it > inside of a guest VM again? > 2. Also, usually enable_unsafe_noiommu_mode is not compiled in, so we > need to recompile vfio separately, and since we already compile igb_uio > anyway, vfio won't be any better in terms of deployment. > 3. There also seems to be an option of using SR-IOV instead of > passthrough, but we haven't explored this option yet. The question here is, > do you still need to "expose iommu" to be able to bind VF to vfio? And > what's the correct workflow here in general? > > Best Regarads, > Oleksandr You may have to use vfio-pci without iommu option.