From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <Eric.Zhang@windriver.com> Received: from mail5.wrs.com (mail5.windriver.com [192.103.53.11]) by dpdk.org (Postfix) with ESMTP id 958D85681 for <dev@dpdk.org>; Mon, 17 Sep 2018 21:05:41 +0200 (CEST) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail5.wrs.com (8.15.2/8.15.2) with ESMTPS id w8HJ4PA8011043 (version=TLSv1 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 17 Sep 2018 12:04:46 -0700 Received: from [128.224.56.213] (128.224.56.213) by ALA-HCA.corp.ad.wrs.com (147.11.189.50) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 17 Sep 2018 12:04:35 -0700 To: "Stojaczyk, Dariusz" <dariusz.stojaczyk@intel.com>, "Burakov, Anatoly" <anatoly.burakov@intel.com>, Jerin Jacob <jerin.jacob@caviumnetworks.com> CC: santosh <santosh.shukla@caviumnetworks.com>, "hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>, =?UTF-8?Q?Ga=c3=abtan_Rivet?= <gaetan.rivet@6wind.com>, "Richardson, Bruce" <bruce.richardson@intel.com>, "dev@dpdk.org" <dev@dpdk.org>, "Allain.Legacy@windriver.com" <Allain.Legacy@windriver.com>, "Matt.Peters@windriver.com" <Matt.Peters@windriver.com> References: <1535558289-10336-1-git-send-email-eric.zhang@windriver.com> <e96cdef8-3247-9f75-9ed0-6b3234d861ce@intel.com> <20180830094323.37xkgud4fz3mflbg@bidouze.vm.6wind.com> <0f19de06-fccd-eff2-b33e-71d49a005dbb@nxp.com> <a8b41461-9888-b88f-7b3b-1c5a4b225e37@caviumnetworks.com> <9ae6e3dd-5f33-d874-9a58-730435b58de3@windriver.com> <20180906073449.GA20576@jerin> <1781101b-ceed-ac80-5564-cc01232cb29f@intel.com> <29ab4eb7-8af5-f66d-5359-dcfd7fbb2ba5@windriver.com> <2dab0826-ee74-89fe-c012-ad9dd8e5f0cc@windriver.com> <FBE7E039FA50BF47A673AD0BD3CD56A8461E7DB4@HASMSX105.ger.corp.intel.com> From: Eric Zhang <eric.zhang@windriver.com> Message-ID: <e90d4e56-c7ec-d243-3620-0b5e24339df6@windriver.com> Date: Mon, 17 Sep 2018 15:04:33 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <FBE7E039FA50BF47A673AD0BD3CD56A8461E7DB4@HASMSX105.ger.corp.intel.com> Content-Language: en-US X-Originating-IP: [128.224.56.213] Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Mon, 17 Sep 2018 19:05:48 -0000 Hi All, Considered that most people commented that eal option "--iova-mode" is a better solution that gives user a chance not to use bus iova auto detection scheme and want to override for some reason, I will generate 2nd version patch which will use eal option instead of compilation configuration. Eric On 09/17/2018 04:32 AM, Stojaczyk, Dariusz wrote: > Hi, > > A little bit of self-advertising: > I recently pushed patches that will make DPDK default to RTE_IOVA_VA when physical addresses were not explicitly requested and are not available, e.g. when running as a non-privileged user. It shouldn't cause any conflicts with the changes you're proposing here, but any review is welcome. > > pci/linux: use RTE_IOVA_VA whenever possible http://patches.dpdk.org/patch/44392/ > [v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available http://patches.dpdk.org/patch/44420/ > > As for the --iova-mode=<pa/va>, I agree it could help DPDK use cases where most drivers or devices are hotplugged/hotattached at runtime - e.g. SPDK, it could certainly make use of such param. > > D. > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Eric Zhang >> Sent: Tuesday, September 11, 2018 7:22 PM >> To: Burakov, Anatoly <anatoly.burakov@intel.com>; Jerin Jacob >> <jerin.jacob@caviumnetworks.com> >> Cc: santosh <santosh.shukla@caviumnetworks.com>; >> hemant.agrawal@nxp.com; Gaëtan Rivet <gaetan.rivet@6wind.com>; >> Richardson, Bruce <bruce.richardson@intel.com>; dev@dpdk.org; >> Allain.Legacy@windriver.com; Matt.Peters@windriver.com >> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical >> >> >> >> On 09/07/2018 04:13 PM, Eric Zhang wrote: >>> >>> On 09/07/2018 05:26 AM, Burakov, Anatoly wrote: >>>> On 06-Sep-18 8:34 AM, Jerin Jacob wrote: >>>>> -----Original Message----- >>>>>> Date: Tue, 4 Sep 2018 23:40:36 -0400 >>>>>> From: Eric Zhang <eric.zhang@windriver.com> >>>>>> To: santosh <santosh.shukla@caviumnetworks.com>, >>>>>> hemant.agrawal@nxp.com, >>>>>> Gaëtan Rivet <gaetan.rivet@6wind.com>, "Burakov, Anatoly" >>>>>> <anatoly.burakov@intel.com> >>>>>> CC: bruce.richardson@intel.com, dev@dpdk.org, >>>>>> Allain.Legacy@windriver.com, >>>>>> Matt.Peters@windriver.com >>>>>> Subject: Re: [dpdk-dev] [PATCH] eal: force IOVA mode to physical >>>>>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 >>>>>> Thunderbird/52.9.1 >>>>>> >>>>>> On 08/30/2018 08:59 AM, santosh wrote: >>>>>>> On Thursday 30 August 2018 05:43 PM, Hemant wrote: >>>>>>>> External Email >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> On 8/30/2018 3:13 PM, Gaëtan Rivet wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> On Thu, Aug 30, 2018 at 10:09:04AM +0100, Burakov, Anatoly wrote: >>>>>>>>>> On 29-Aug-18 4:58 PM, eric zhang wrote: >>>>>>>>>>> This patch adds a configuration option to force the IOVA mode >>>>>>>>>>> to physical address (PA). There exists virtual devices that >>>>>>>>>>> are not directly attached to the PCI bus, and therefore the >>>>>>>>>>> auto detection of the IOVA mode based on probing the PCI bus >>>>>>>>>>> and IOMMU configuration may not report the required >> addressing >>>>>>>>>>> mode. Having the configuration option permits the mode to be >>>>>>>>>>> explicitly configured in this scenario. >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: eric zhang <eric.zhang@windriver.com> >>>>>>>>>>> --- >>>>>>>>>> Defining this at compile-time seems like an overkill. Wouldn't >>>>>>>>>> it be better to just add an EAL command-line option to force >>>>>>>>>> IOVA mode to a particular value? >>>>>>>> That is a good suggestion. >>>>>>>>>> -- >>>>>>>>>> Thanks, >>>>>>>>>> Anatoly >>>>>>>>> What is the bus of these devices and why not implement >>>>>>>>> get_iommu_class in it? >>>>>>>> There are cases, where you are using dpdk libraries with external >>>>>>>> libraries and you need to change the default behavior DPDK lib to >>>>>>>> use physical address instead of virtual address. >>>>>>>> Providing an option to user will help. >>>>>>>> >>>>>>>> >>>>>>> More appropriate solution could be: >>>>>>> * Either fix it at bus layer .. i.e.. get_iommu_class().. >>>>>>> * Or introduce something like [1] --iova-mode=<pa/va> param. >>>>>>> >>>>>>> Former is better solution than latter if autodetection is a key >>>>>>> criteria. >>>>>>> Thanks. >>>>>>> >>>>>>> [1] http://patchwork.dpdk.org/patch/25192/ >>>>>>> >>>>>> It's not generic which couldn't be fixed at bus layer. >>>>>> So what's the preference of EAL option or compile time solution? >>>>>> Adding --iova-mode as patch [1] will overrivde auto-detection >>>>>> rte_bus_get_iommu_class() >>>>>> make it no use; compile time solution will align with upstream and >>>>>> keep new atuodetection solution in #ifndef. >>>>> If it is for vdev devices, why not introduce something like >>>>> RTE_PCI_DRV_IOVA_AS_VA and let vdev device describe its personality. >>>>> And based on the devices(flags) on vdev bus, >>>>> rte_bus_get_iommu_class() of vdev can decide the mode just like PCI >> bus. >>>> That seems like a better option to me, +1. As far as i know, at the >>>> moment if there are no devices attached at all, or if there are only >>>> vdev devices attached, DPDK will default to IOVA as PA mode for no >>>> good reason; such a change would certainly fix this. >>> Thanks for the suggestions however our virtual device doesn't run dpdk >>> vdev code so we can't use the flag. >>> Notice that in eal.c there is one workaround that force iova to be PA >>> per virtual device is not directly attached to pci. That case is >>> checking kni module. Ours is a similar case that virtual device not >>> attach pci directly. >>> So we have to turn to force iova to PA either 1. compilation option 2. >>> eal option. Which one should be the preference by taking into >>> consideration that align with upstream? >>> >>> Thanks >>> >> Any comments? >>>>>> Thanks >>>>>> Eric >>>>>> >>>>