From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BD01AA0A0E; Mon, 10 May 2021 13:20:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A4C140140; Mon, 10 May 2021 13:20:27 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id B37E04003E for ; Mon, 10 May 2021 13:20:24 +0200 (CEST) IronPort-SDR: YB1GNroy3N9OtdH5xyB8lq1cqPqCBIgOjPY6ae418I19dvrZXCAoNCoZo2DbPVUOuCYxmVmd5r gS7HZh4nCwbg== X-IronPort-AV: E=McAfee;i="6200,9189,9979"; a="220116386" X-IronPort-AV: E=Sophos;i="5.82,287,1613462400"; d="scan'208";a="220116386" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2021 04:20:22 -0700 IronPort-SDR: +auizMp/0DJpc6oAkBBZu1HVDWRSfrH05H1NwLW8BYPRIJPNFUvwAOHkABsTITlKHplQ/pyFMu 2vnjFylR66jg== X-IronPort-AV: E=Sophos;i="5.82,287,1613462400"; d="scan'208";a="436100957" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.17.68]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 10 May 2021 04:20:19 -0700 Date: Mon, 10 May 2021 12:20:15 +0100 From: Bruce Richardson To: Thomas Monjalon Cc: "Burakov, Anatoly" , dev@dpdk.org, Harry van Haaren , david.marchand@redhat.com Message-ID: References: <20210506150908.797110-1-bruce.richardson@intel.com> <18d300a4-f8f2-06b9-894e-91cd899bf933@intel.com> <2168873.xxFOWYceCV@thomas> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2168873.xxFOWYceCV@thomas> Subject: Re: [dpdk-dev] [PATCH] raw/ioat: fix bus requiring virtual addressing when no devs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, May 10, 2021 at 12:18:50PM +0200, Thomas Monjalon wrote: > 06/05/2021 17:34, Bruce Richardson: > > On Thu, May 06, 2021 at 04:27:03PM +0100, Burakov, Anatoly wrote: > > > On 06-May-21 4:09 PM, Bruce Richardson wrote: > > > > If after a bus scan, there are no devices using a particular bus, then > > > > that bus should not be taken into account when deciding whether DPDK > > > > should be run in VA or PA addressing mode. This becomes an issue when > > > > the DSA bus driver code is used on a system without an IOMMU. The PCI > > > > bus correctly reports that it only works in PA mode, while the DSA bus - > > > > also correctly - reports that it works only in VA mode. The difference > > > > is that there will be no devices found in a scan for the DSA bus, since > > > > the kernel driver can only present those to userspace in the presence of > > > > an IOMMU. > > > > > > > > While we could change DSA instance to always report that it does not > > > > care about the addressing mode, this would imply that it could be used > > > > with DPDK in PA mode which is not the case. Therefore, this patch > > > > changes the driver to report DC (don't care) in the case where no > > > > devices are present, and VA otherwise. > > > > > > > > NOTE: this addressing mode use of VA-only applies only in the case of > > > > using DSA through the idxd kernel driver. The use of DSA though vfio-pci > > > > is unaffected and works as with other PCI devices. > > > > > > > > Fixes: b7aaf417f936 ("raw/ioat: add bus driver for device scanning automatically") > > > > > > > > Reported-by: Harry van Haaren > > > > Signed-off-by: Bruce Richardson > > > > --- > > > > > > This would be a good opportunity to start a discussion on a "proper" fix for > > > this, because while this hack is acceptable being so close to release, IMO > > > we need a more general solution. Maybe a bus API reporting number of scanned > > > devices would be nice to have, as EAL will then be able to ignore the > > > opinion of those buses who don't have any devices. > > > > > > The "number of devices" is a bit of a fluid concept, as number of scanned > > > devices may not be the same as number of *probed* devices, and there's also > > > hotplug. However, i think that we can ignore all that, because we care about > > > initialization only - any other issues can be addressed by forcing > > > --iova-mode by the user, as we cannot really do anything about hotplug or > > > devices that fail to probe. > > > > > > Thoughts? > > > > > Perhaps rather than just "number of devices" we should have buses report > > whether they are relevant or not. For example, there are a number of > > SoC-specific bus drivers in DPDK, which are presumably only relevant on > > particular SoCs. Similarly here for the DSA bus, the presence of the > > devices is a useful proxy for relevance - especially since it doesn't > > support hotplug - but if we had a way to report relevance/non-relevance DSA > > could just report itself as not-relevant in case where there is no IOMMU > > enabled on the system, or when the "idxd" kernel driver is not present, for > > example, and then have EAL skip even asking about PA/VA support > > I think what you are describing is the same as reporting "don't care". > Why do you consider it as a hack? > It's not the same as don't-care, because the bus just won't work with iova-as-pa, it must have iova-as-va. Therefore to respond as anything other than VA here would be incorrect. The issue is that if one bus must have PA and another must have VA, then obviously both cannot be in use at the same time, and we have no way of sanely resolving that conflict, or specifying which to ignore. The reason this patch is a hack, is that this dsa bus driver is taking it on itself to basic say "ignore me" (by returning DC rather than VA), when no devices are found. Maybe it's the case that busses are supposed to behave that way, but it's not clear from the doxygen comments. I would have thought that we should have a way for a bus to signal back to EAL that the whole bus is not relevant, and then can be skipped for querying PA/VA and for probe etc. /Bruce