Re: [dpdk-dev] [PATCH 0/4] support for write combining

["Rafał Kozik" <rk@semihalf.com>]

Hello Bruce,

thank you for your advices.

> 1. Why not always have igb_uio support write-combining since it can be
> controlled thereafter via userspace mapping one file or another?

I added parameter to the igb_uio because currently it perform ioremap
and fails if it return NULL.
But performing ioremap makes it impossible to use WC, so I remove it.
ENA driver work well after this change, but I cannot test it on all
drivers and all platforms.
It seems to me that making it configurable prevents form spoiling
other drivers that could use internal_addr returned by ioremap.

> 2. Why not always map both resource and resource_wc files at the PCI level,
> and make them available via different pointers to the driver? Then the
> driver can choose at the per-access level which it wants to use. For
> example, for init of a device, a driver may do all register access via
> uncachable memory, and only use the write-combined support for
> performance-critical parts. [I have a draft patch lying around here
> somewhere that does something similar to that.]

I tried to implement this idea but without good results. I get mapping
with or without WC depending on mapping order.
As I was trying to find solution I come across with this paper:
https://www.kernel.org/doc/ols/2008/ols2008v2-pages-135-144.pdf
In section 5.3 and 5.4 it is discussing access to PCI resources.
According to it:

A request to uncached access can fail if there is already
an existing write-combine mapping for that region. A
request for write-combine access can succeed with un-
cached mapping instead, in the case of already existing
uncached mapping for this region.

We cannot use WC all the time, because it not guaranteed writing order.
On this basis I suppose that better option is to map each resource
only once depending on parameter provided by PMD.

> One last question - if using vfio-pci kernel module, do the resource_wc
> files present the bars as write-combined memory type, or are they
> uncachable?

I tried to use VFIO to map WC memory, but without success.

Best regards,
Rafal Kozik

2018-04-11 16:42 GMT+02:00 Bruce Richardson <bruce.richardson@intel.com>:
> On Wed, Apr 11, 2018 at 04:07:13PM +0200, Rafal Kozik wrote:
>> Support for write combining.
>>
>> Rafal Kozik (4):
>>   igb_uio: add wc option
>>   bus/pci: reference driver structure
>>   eal: enable WC during resources mapping
>>   net/ena: enable WC
>>
>>  drivers/bus/pci/linux/pci_uio.c | 39 ++++++++++++++++++++++++++++-----------
>>  drivers/bus/pci/pci_common.c    | 13 ++++++++-----
>>  drivers/bus/pci/rte_bus_pci.h   |  2 ++
>>  drivers/net/ena/ena_ethdev.c    |  3 ++-
>>  kernel/linux/igb_uio/igb_uio.c  | 17 ++++++++++++++---
>>  5 files changed, 54 insertions(+), 20 deletions(-)
>>
> Couple of thoughts on this set.
>
> You add an option to the kernel module to allow wc to be supported on a
> device, but when we go to do PCI mapping, we either map the regular
> resource file or the _wc one. Therefore:
>
> 1. Why not always have igb_uio support write-combining since it can be
> controlled thereafter via userspace mapping one file or another?
>
> 2. Why not always map both resource and resource_wc files at the PCI level,
> and make them available via different pointers to the driver? Then the
> driver can choose at the per-access level which it wants to use. For
> example, for init of a device, a driver may do all register access via
> uncachable memory, and only use the write-combined support for
> performance-critical parts. [I have a draft patch lying around here
> somewhere that does something similar to that.]
>
> One last question - if using vfio-pci kernel module, do the resource_wc
> files present the bars as write-combined memory type, or are they
> uncachable?
>
> Regards,
> /Bruce