From: "Rafał Kozik" <rk@semihalf.com>
To: Bruce Richardson <bruce.richardson@intel.com>
Cc: dev@dpdk.org, "Marcin Wojtas" <mw@semihalf.com>,
"Michał Krawczyk" <mk@semihalf.com>,
"Tzalik, Guy" <gtzalik@amazon.com>,
evgenys@amazon.com, "Matushevsky, Alexander" <matua@amazon.com>,
"Chauskin, Igor" <igorch@amazon.com>,
"Ferruh Yigit" <ferruh.yigit@intel.com>
Subject: Re: [dpdk-dev] [PATCH 0/4] support for write combining
Date: Mon, 16 Apr 2018 13:36:13 +0200 [thread overview]
Message-ID: <CAMG3L492f=4XV+q__sZmGDv5TP=yho-hNu80VRdnwmyLh7q+zw@mail.gmail.com> (raw)
In-Reply-To: <20180411144251.GA33124@bricha3-MOBL.ger.corp.intel.com>
Hello Bruce,
thank you for your advices.
> 1. Why not always have igb_uio support write-combining since it can be
> controlled thereafter via userspace mapping one file or another?
I added parameter to the igb_uio because currently it perform ioremap
and fails if it return NULL.
But performing ioremap makes it impossible to use WC, so I remove it.
ENA driver work well after this change, but I cannot test it on all
drivers and all platforms.
It seems to me that making it configurable prevents form spoiling
other drivers that could use internal_addr returned by ioremap.
> 2. Why not always map both resource and resource_wc files at the PCI level,
> and make them available via different pointers to the driver? Then the
> driver can choose at the per-access level which it wants to use. For
> example, for init of a device, a driver may do all register access via
> uncachable memory, and only use the write-combined support for
> performance-critical parts. [I have a draft patch lying around here
> somewhere that does something similar to that.]
I tried to implement this idea but without good results. I get mapping
with or without WC depending on mapping order.
As I was trying to find solution I come across with this paper:
https://www.kernel.org/doc/ols/2008/ols2008v2-pages-135-144.pdf
In section 5.3 and 5.4 it is discussing access to PCI resources.
According to it:
A request to uncached access can fail if there is already
an existing write-combine mapping for that region. A
request for write-combine access can succeed with un-
cached mapping instead, in the case of already existing
uncached mapping for this region.
We cannot use WC all the time, because it not guaranteed writing order.
On this basis I suppose that better option is to map each resource
only once depending on parameter provided by PMD.
> One last question - if using vfio-pci kernel module, do the resource_wc
> files present the bars as write-combined memory type, or are they
> uncachable?
I tried to use VFIO to map WC memory, but without success.
Best regards,
Rafal Kozik
2018-04-11 16:42 GMT+02:00 Bruce Richardson <bruce.richardson@intel.com>:
> On Wed, Apr 11, 2018 at 04:07:13PM +0200, Rafal Kozik wrote:
>> Support for write combining.
>>
>> Rafal Kozik (4):
>> igb_uio: add wc option
>> bus/pci: reference driver structure
>> eal: enable WC during resources mapping
>> net/ena: enable WC
>>
>> drivers/bus/pci/linux/pci_uio.c | 39 ++++++++++++++++++++++++++++-----------
>> drivers/bus/pci/pci_common.c | 13 ++++++++-----
>> drivers/bus/pci/rte_bus_pci.h | 2 ++
>> drivers/net/ena/ena_ethdev.c | 3 ++-
>> kernel/linux/igb_uio/igb_uio.c | 17 ++++++++++++++---
>> 5 files changed, 54 insertions(+), 20 deletions(-)
>>
> Couple of thoughts on this set.
>
> You add an option to the kernel module to allow wc to be supported on a
> device, but when we go to do PCI mapping, we either map the regular
> resource file or the _wc one. Therefore:
>
> 1. Why not always have igb_uio support write-combining since it can be
> controlled thereafter via userspace mapping one file or another?
>
> 2. Why not always map both resource and resource_wc files at the PCI level,
> and make them available via different pointers to the driver? Then the
> driver can choose at the per-access level which it wants to use. For
> example, for init of a device, a driver may do all register access via
> uncachable memory, and only use the write-combined support for
> performance-critical parts. [I have a draft patch lying around here
> somewhere that does something similar to that.]
>
> One last question - if using vfio-pci kernel module, do the resource_wc
> files present the bars as write-combined memory type, or are they
> uncachable?
>
> Regards,
> /Bruce
next prev parent reply other threads:[~2018-04-16 11:36 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-11 14:07 Rafal Kozik
2018-04-11 14:07 ` [dpdk-dev] [PATCH 1/4] igb_uio: add wc option Rafal Kozik
2018-06-27 16:34 ` Ferruh Yigit
2018-06-28 13:08 ` Rafał Kozik
2018-06-28 13:15 ` [dpdk-dev] [PATCH v2 0/4] support for write combining Rafal Kozik
2018-06-28 13:15 ` [dpdk-dev] [PATCH v2 1/4] igb_uio: add wc option Rafal Kozik
2018-06-28 14:32 ` Ferruh Yigit
2018-06-29 8:35 ` Rafał Kozik
2018-06-29 10:08 ` Ferruh Yigit
2018-06-29 10:24 ` [dpdk-dev] [PATCH v3 0/4] support for write combining Rafal Kozik
2018-06-29 10:24 ` [dpdk-dev] [PATCH v3 1/4] igb_uio: add wc option Rafal Kozik
2018-06-29 10:53 ` Ferruh Yigit
2018-06-29 12:17 ` [dpdk-dev] [PATCH v4 0/4] Support for write combining Rafal Kozik
2018-06-29 12:17 ` [dpdk-dev] [PATCH v4 1/4] igb_uio: add wc option Rafal Kozik
2018-06-29 13:11 ` Rafał Kozik
2018-06-29 13:20 ` Ferruh Yigit
2018-06-29 13:40 ` Ferruh Yigit
2018-06-29 12:17 ` [dpdk-dev] [PATCH v4 2/4] bus/pci: reference driver structure Rafal Kozik
2018-06-29 12:17 ` [dpdk-dev] [PATCH v4 3/4] eal: enable WC during resources mapping Rafal Kozik
2018-06-29 12:17 ` [dpdk-dev] [PATCH v4 4/4] net/ena: enable WC Rafal Kozik
2018-06-29 13:54 ` [dpdk-dev] [PATCH v5 0/4] Support for write combining Rafal Kozik
2018-06-29 13:54 ` [dpdk-dev] [PATCH v5 1/4] igb_uio: add wc option Rafal Kozik
2018-06-29 13:54 ` [dpdk-dev] [PATCH v5 2/4] bus/pci: reference driver structure Rafal Kozik
2018-06-29 13:54 ` [dpdk-dev] [PATCH v5 3/4] eal: enable WC during resources mapping Rafal Kozik
2018-06-29 13:54 ` [dpdk-dev] [PATCH v5 4/4] net/ena: enable WC Rafal Kozik
2018-06-29 14:26 ` [dpdk-dev] [PATCH v5 0/4] Support for write combining Ferruh Yigit
2018-06-29 22:13 ` Thomas Monjalon
2018-06-29 10:24 ` [dpdk-dev] [PATCH v3 2/4] bus/pci: reference driver structure Rafal Kozik
2018-06-29 10:24 ` [dpdk-dev] [PATCH v3 3/4] eal: enable WC during resources mapping Rafal Kozik
2018-06-29 10:24 ` [dpdk-dev] [PATCH v3 4/4] net/ena: enable WC Rafal Kozik
2018-06-28 13:15 ` [dpdk-dev] [PATCH v2 2/4] bus/pci: reference driver structure Rafal Kozik
2018-06-28 13:15 ` [dpdk-dev] [PATCH v2 3/4] eal: enable WC during resources mapping Rafal Kozik
2018-06-28 14:50 ` Ferruh Yigit
2018-06-29 8:58 ` Rafał Kozik
2018-06-29 9:05 ` Ferruh Yigit
2018-06-29 10:28 ` Rafał Kozik
2018-06-29 10:37 ` Ferruh Yigit
2018-06-28 13:15 ` [dpdk-dev] [PATCH v2 4/4] net/ena: enable WC Rafal Kozik
2018-07-02 6:52 ` Michał Krawczyk
2018-04-11 14:07 ` [dpdk-dev] [PATCH 2/4] bus/pci: reference driver structure Rafal Kozik
2018-06-27 16:36 ` Ferruh Yigit
2018-06-28 13:05 ` Rafał Kozik
2018-04-11 14:07 ` [dpdk-dev] [PATCH 3/4] eal: enable WC during resources mapping Rafal Kozik
2018-06-27 16:41 ` Ferruh Yigit
2018-06-28 13:06 ` Rafał Kozik
2018-04-11 14:07 ` [dpdk-dev] [PATCH 4/4] net/ena: enable WC Rafal Kozik
2018-06-27 16:11 ` Thomas Monjalon
2018-06-28 13:04 ` Rafał Kozik
2018-04-11 14:42 ` [dpdk-dev] [PATCH 0/4] support for write combining Bruce Richardson
2018-04-16 11:36 ` Rafał Kozik [this message]
2018-04-27 8:27 ` Rafał Kozik
2018-04-27 11:30 ` Bruce Richardson
2018-04-30 8:07 ` Rafał Kozik
2018-04-30 8:58 ` Bruce Richardson
2018-06-11 9:32 Rafał Kozik
2018-06-26 7:27 ` Rafał Kozik
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMG3L492f=4XV+q__sZmGDv5TP=yho-hNu80VRdnwmyLh7q+zw@mail.gmail.com' \
--to=rk@semihalf.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=evgenys@amazon.com \
--cc=ferruh.yigit@intel.com \
--cc=gtzalik@amazon.com \
--cc=igorch@amazon.com \
--cc=matua@amazon.com \
--cc=mk@semihalf.com \
--cc=mw@semihalf.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).