From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C679A046B for ; Tue, 23 Jul 2019 11:57:41 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3B06F1BFC4; Tue, 23 Jul 2019 11:57:40 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 63D1D1BFA4 for ; Tue, 23 Jul 2019 11:57:38 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Jul 2019 02:57:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,298,1559545200"; d="scan'208";a="174497500" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.83]) ([10.237.220.83]) by orsmga006.jf.intel.com with ESMTP; 23 Jul 2019 02:57:33 -0700 To: Jerin Jacob Kollanukkaran , "Stojaczyk, Dariusz" , Thomas Monjalon , David Marchand Cc: "dev@dpdk.org" References: <1562795329-16652-1-git-send-email-david.marchand@redhat.com> <1563800213-29839-1-git-send-email-david.marchand@redhat.com> <1735440.496QTsyyT4@xps> From: "Burakov, Anatoly" Message-ID: <3397bb38-115b-40e8-c208-006fa5dd7405@intel.com> Date: Tue, 23 Jul 2019 10:57:33 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 23-Jul-19 6:27 AM, Jerin Jacob Kollanukkaran wrote: >> -----Original Message----- >> From: Stojaczyk, Dariusz >> Sent: Tuesday, July 23, 2019 10:24 AM >> To: Jerin Jacob Kollanukkaran ; Thomas Monjalon >> ; David Marchand ; >> Burakov, Anatoly >> Cc: dev@dpdk.org >> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection >> >>> -----Original Message----- >>> From: Jerin Jacob Kollanukkaran [mailto:jerinj@marvell.com] >>> Sent: Tuesday, July 23, 2019 6:19 AM >>> >>>> -----Original Message----- >>>> From: Stojaczyk, Dariusz >>>> Sent: Tuesday, July 23, 2019 9:06 AM >>>> To: Thomas Monjalon ; David Marchand >>>> ; Burakov, Anatoly >>>> ; Jerin Jacob Kollanukkaran >>>> >>>> Cc: dev@dpdk.org >>>> Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode >>>> selection >>>> >>>> This introduces a regression where uio-bound devies are attached to >>>> a >>> DPDK >>>> app at runtime. >>> >>> Just to understand the requirements; >>> # Is this requirement for SPDK? >>> # Is brand new PCI device scanned and attached to DPDK at runtime? >>> # Any specific reason for using uio vs vfio? >> >> Jerin, > > Stojaczyk, > > There reason to choose VA incase if bus detects DC is following: > > - All drivers are expected to work in RTE_IOVA_VA mode, irrespective of > physical address availability. > - By default, the mempool, first asks for IOVA-contiguous memory using > ``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may > affect the application boot time. > - It is easy to enable large amount of IOVA-contiguous memory use-cases > with IOVA in VA mode. > >> >> It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give >> you the steps but it should be reproducible even with testpmd. >> >> The PCI device could have been simply hotplugged to the system after DPDK >> app start. DPDK didn't know about it at initialization, so it picked >> RTE_IOVA_VA and then would fail to attach any UIO-bound device ever >> after: >> >> EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing > > We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work > Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on those devices? > > >> EAL: Driver cannot attach the device (0000:00:09.0) >> EAL: Failed to attach device on primary process >> >> UIO is commonly used on systems without IOMMU- including VMs. > > The latest machines has IOMMU. Which machines you are testing against, > Can we detect the machines without IOMMU and switch to PA? A machine without an IOMMU shouldn't have picked IOVA as VA in the first place. Perhaps this is something we could fix? I'm not sure how to detected that condition though, i don't think there's a mechanism to know that for sure. Some kernels create a "iommu" sysfs directories, but i'm not too sure if they're 1) there for older kernels we support, and 2) always there. On machines with IOMMU, VFIO should be the default, and we should discourage people from using igb_uio. Is there any reason why SPDK is not using VFIO by default? On my machine, "/sys/devices/virtual/iommu" exists when IOMMU is enabled, but doesn't exist if it isn't ("/sys/class/iommu" exists in both cases, but is empty when IOMMU is disabled). Perhaps we could go off that? -- Thanks, Anatoly