From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 48696A04DD; Thu, 22 Oct 2020 11:48:29 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8AE0CA96F; Thu, 22 Oct 2020 11:44:49 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 3700EA96E for ; Thu, 22 Oct 2020 11:44:47 +0200 (CEST) IronPort-SDR: urx9W3TrgYkjddXkWS1TiAd6pkHWgnX1G13jM7XI6S8rQ+HsS7sP40ekWvGsEtJbJ/T4uH+gJG sHFnQVYU9B7w== X-IronPort-AV: E=McAfee;i="6000,8403,9781"; a="167597772" X-IronPort-AV: E=Sophos;i="5.77,404,1596524400"; d="scan'208";a="167597772" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2020 02:44:45 -0700 IronPort-SDR: 0wzG+6empc9m/3I8AE6xtjyaOy1HmwC/6BwUV7h8OEOtrMEMS+oQvsMlKU/HpaMAOFXhvJr4aK tmmv4GoTVucw== X-IronPort-AV: E=Sophos;i="5.77,404,1596524400"; d="scan'208";a="533895758" Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.213.248.224]) ([10.213.248.224]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2020 02:44:43 -0700 To: =?UTF-8?B?6LCi5Y2O5LyfKOatpOaXtuatpOWIu++8iQ==?= , Maxime Coquelin Cc: dev@dpdk.org, anatoly.burakov@intel.com, david.marchand@redhat.com, grive@u256.net, zhihong.wang@intel.com, chenbo.xia@intel.com References: <68ecd941-9c56-4de7-fae2-2ad15bdfd81a@alibaba-inc.com> <1602578499-13975-1-git-send-email-huawei.xhw@alibaba-inc.com> <1602578499-13975-2-git-send-email-huawei.xhw@alibaba-inc.com> <2298e6a6-2ca8-0f68-6742-1b7d5dde97a7@intel.com> <5d26db1a-6970-6e3a-c2e5-8dc540922028@alibaba-inc.com> From: Ferruh Yigit Message-ID: Date: Thu, 22 Oct 2020 10:44:39 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH v4] pci: support both PIO and MMIO BAR for legacy virtio on x86 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/22/2020 10:15 AM, 谢华伟(此时此刻） wrote: > > On 2020/10/22 1:24, Ferruh Yigit wrote: >> On 10/21/2020 1:32 PM, 谢华伟(此时此刻） wrote: >>> >>> On 2020/10/21 19:49, Ferruh Yigit wrote: >>>> On 10/13/2020 9:41 AM, 谢华伟(此时此刻） wrote: >>>>> From: "huawei.xhw" >>>>> >>>>> Legacy virtio-pci only supports PIO BAR resource. As we need to create lots of >>>>> virtio devices and PIO resource on x86 is very limited, we expose MMIO BAR. >>>>> >>>>> Kernel supports both PIO  and MMIO BAR for legacy virtio-pci device. We >>>>> handles >>>>> different type of BAR in the similar way. >>>>> >>>>> In previous implementation, with igb_uio we get PIO address from igb_uio >>>>> sysfs entry; with uio_pci_generic, we get PIO address from >>>>> /proc/ioports. >>>>> For PIO/MMIO RW, there is different path for different drivers and arch. >>>>> For VFIO, PIO/MMIO RW is through syscall, which has big performance >>>>> issue. >>>>> On X86, it assumes only PIO is supported. >>>>> >>>>> All of the above is too much twisted. >>>>> This patch unifies the way to get both PIO and MMIO address for different >>>>> driver >>>>> and arch, all from standard resource attr under pci sysfs. >>>>> >>>> >>>> As mentined above this patch does multiple things. >>>> >>>> The main target is, as far as I understand, you have a legacy virtio device >>>> which supports "memory-mapped I/O" and "port-mapped I/O", but virtio logic >>>> forces legacy devices to use the PIO but you want to be able to use the MMIO >>>> with this device. >>> yes. >>>> >>>> The solution below is adding MMIO support in the PIO funciton, and >>>> distinguish MMIO or PIO based on their address check. >>> Yes, kernel does this in the similar way. >>>> >>>> >>>> Instead of this, can't this be resolved in the virtio side, like if the >>>> legacy device supports MMIO (detect this somehow) use the MMIO istead of >>>> hacking PIO mapping to support MMIO? >>> >>> Get your concern. >>> >>> 1> >>> >>> If we move, I think we should move all those PCI codes into virtio side, not >>> just the mmio part. >>> >>> Without my patch, those PCI codes are virtio-pci device specific, not generic. >>> >>> With this patch, those pci ioport map/rw code could also be used for other >>> devices if they support both PIO and MMIO. >>> >> >> I was not suggesting moving any code into virtio, but within 'vtpci_init()' >> what happens when "hw->modern = 1;" is set? >> And if this is set for your device, will it work without change? > > Yes, this will only affect legacy_device, which uses legacy_ops to access port io. > > If is is modern_device, port access will go through modern_ops. > > We only change the implementation in legacy_ops. > I am saying something else. When a device is marked as "hw->modern = 1;", it will use MMIO, right? If, somehow, your device marked as "hw->modern = 1;", will that path work as expected for your device? > >> >>> Every option is ok. Hope i make myself clear. >>> >>> 2>  I don't think this is hacking. for rte_pci_ioport_map/read/write, if >>> ioport could be both PIO and MMIO, then everything is reasonable. >>> >>> Take how kernel does port map for example: >>> >>>      vp_dev->ioaddr = pci_iomap(pci_dev, 0, 0); >>> >>> Here io doesn't mean PIO only. It could also be MMIO. Kernel then uses >>> ioread/write to access PIO/MMIO port. >>> >>> Actually we are pretty much the same in the interface. >>> >>> I think this patch extends rather then hacks the ioport interface to support >>> MMIO. >>> >>>> >>>> I have other concerns, specially mergin VFIO mapping too, but lets clarify >>>> above first. >>> >>> vfio doesn't affect other driver but only virtio. >>> >> >> Why it doesn't affect other drivers, can't there be other driver using PIO? > > Currently only virtio-pci uses PIO, and only virtio PMD uses these port > map/read/write functions. > > I don't foresee in future any new device uses PIO. > I see but technically there can be other users. > /huawei > >> >>> igb_uio, uio_pci_generic and vfio-pci all uses the same way to map/rw ioport. >>> >> >> For vfio, code changes 'pci_vfio_ioport_read()' to the direct address read, >> first I don't know if this is always safe, and my question why there is a >> syscall introduced at first place if you can read from address directly? > > Original vfio way works, but we don't need that syscall. Under whatever driver, > we could use the simple way as in this patch. > If vfio works, you have already a solution, that is good. But I see you are not happy with its performance. > /huawei > >> >> Is your device works as expected when vfio-pci kernel module used? Since it is >> not suffering from PIO limitation, right? > > Certainly i tested vfio module. Firstly, i didn't intend to fix vfio performance > issue, but i heard that igb_uio will be removed. > Yes, it will be removed in the long run. > /huawei > >> >> >> And I wonder if the patch can be done as three patches to simply it, as: >> 1) Combine 'RTE_PCI_KDRV_IGB_UIO' & 'RTE_PCI_KDRV_UIO_GENERIC' (remove >> pci_ioport_map) >> 2) Update 'pci_uio_ioport_map()' to add memory map support (and update >> read/write functions according) >> 3) Combine vfio & uio >> > Got it. It makes sense to split, but i think this patch is already simple enough. > The patch is doing many things in one patch, I think it is better to separate logically separate issues, although they are simple. > Let me check. > > /huawei > >>>> >>>> Thanks, >>>> ferruh >>>> >>>> >>>> >>>>> We distinguish PIO and MMIO by their address like how kernel does. It is >>>>> ugly but works. >>>>> >>>>> Signed-off-by: huawei.xhw >> >> <...>