From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-f68.google.com (mail-lf0-f68.google.com [209.85.215.68]) by dpdk.org (Postfix) with ESMTP id 893301B020 for ; Mon, 16 Apr 2018 13:36:14 +0200 (CEST) Received: by mail-lf0-f68.google.com with SMTP id g203-v6so21527039lfg.11 for ; Mon, 16 Apr 2018 04:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ywbMUfHZQUj1+Q9BTJ+R/dojKjEFGGT7gOnCz0wtoV4=; b=LgcDl4c27Yon0l/5Ruk3Y20H1ODotcZ1xxAiMbCsBu058ZU1pez5Owd0boseTFo74F LYgROemQY6QZ+uBPgrbmqej3rHpkwWLLRcCxgY1EE2xRcm+XvL0I76o2VUMzql8zOuVH WMGxsOcTr/gnCk55XzY4NPFgir7y4qh7g20R/q6kb2e7xIZ1pdfEIOzrx9aMtzMV6POx eWlCze3i4CEbscUXmronBU1SSx9+nEeII9BKKL2n21dKASekrRj/F13umb1IgBOwQVSa qgsx1mhVoEZSS90hWfbzH4utkTgYeqHanu1aeRybnURtgv8K//AeW3xZd4lAuk86JEDO cMzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ywbMUfHZQUj1+Q9BTJ+R/dojKjEFGGT7gOnCz0wtoV4=; b=Jwpur/ZquALM/ONYADjMJwQTiO+j5/JlISk+VcmyBSx+AI2QZZd6Pa3DW7hDHSkAQ7 M9YEImTIzH2Lg0O18/9X1P6Mwib28RgDRF+FNFcxlR4kkGvhu3BinSegg/wscf+ehG1C 9nO20F2wFYlVlMDSFnmVGZuS8rnt022hm2YAr9YRtJBZn5AXsevrmI8eWKxLYnSt0k9Q aKAj4WAx1rOncdfV0K0sQJhaebapNOEfBXsfW6Ip+D9z4eBfCVezD43K79QrIpXHE96X +eJaFeTtiCa5i0DZaQZH5K7Xfi3vkpLfIVTUYOlEPzDuhpa210eJkai3EeL+ANBMT42P jtWQ== X-Gm-Message-State: ALQs6tBraLWGtINADjitRZ5WaMj5Rd/DaYsiOdNuiSz4sNWXuVcYzFOV WUpVO4YuBKxqhSjbG9U2mthug2pZZRb+327ayrgenA== X-Google-Smtp-Source: AIpwx4+1aQBonzvQkWDTzhocc7jAx1G6YpC24HcUiwmbd70266vFWHwFyfMQTX2Pq5kBzNVZwg7M5YFEc4T4Ba+3JVM= X-Received: by 2002:a19:5c05:: with SMTP id q5-v6mr9684641lfb.14.1523878574024; Mon, 16 Apr 2018 04:36:14 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:5709:0:0:0:0:0 with HTTP; Mon, 16 Apr 2018 04:36:13 -0700 (PDT) In-Reply-To: <20180411144251.GA33124@bricha3-MOBL.ger.corp.intel.com> References: <1523455637-31719-1-git-send-email-rk@semihalf.com> <20180411144251.GA33124@bricha3-MOBL.ger.corp.intel.com> From: =?UTF-8?B?UmFmYcWCIEtvemlr?= Date: Mon, 16 Apr 2018 13:36:13 +0200 Message-ID: To: Bruce Richardson Cc: dev@dpdk.org, Marcin Wojtas , =?UTF-8?Q?Micha=C5=82_Krawczyk?= , "Tzalik, Guy" , evgenys@amazon.com, "Matushevsky, Alexander" , "Chauskin, Igor" , Ferruh Yigit Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH 0/4] support for write combining X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Apr 2018 11:36:14 -0000 Hello Bruce, thank you for your advices. > 1. Why not always have igb_uio support write-combining since it can be > controlled thereafter via userspace mapping one file or another? I added parameter to the igb_uio because currently it perform ioremap and fails if it return NULL. But performing ioremap makes it impossible to use WC, so I remove it. ENA driver work well after this change, but I cannot test it on all drivers and all platforms. It seems to me that making it configurable prevents form spoiling other drivers that could use internal_addr returned by ioremap. > 2. Why not always map both resource and resource_wc files at the PCI level, > and make them available via different pointers to the driver? Then the > driver can choose at the per-access level which it wants to use. For > example, for init of a device, a driver may do all register access via > uncachable memory, and only use the write-combined support for > performance-critical parts. [I have a draft patch lying around here > somewhere that does something similar to that.] I tried to implement this idea but without good results. I get mapping with or without WC depending on mapping order. As I was trying to find solution I come across with this paper: https://www.kernel.org/doc/ols/2008/ols2008v2-pages-135-144.pdf In section 5.3 and 5.4 it is discussing access to PCI resources. According to it: A request to uncached access can fail if there is already an existing write-combine mapping for that region. A request for write-combine access can succeed with un- cached mapping instead, in the case of already existing uncached mapping for this region. We cannot use WC all the time, because it not guaranteed writing order. On this basis I suppose that better option is to map each resource only once depending on parameter provided by PMD. > One last question - if using vfio-pci kernel module, do the resource_wc > files present the bars as write-combined memory type, or are they > uncachable? I tried to use VFIO to map WC memory, but without success. Best regards, Rafal Kozik 2018-04-11 16:42 GMT+02:00 Bruce Richardson : > On Wed, Apr 11, 2018 at 04:07:13PM +0200, Rafal Kozik wrote: >> Support for write combining. >> >> Rafal Kozik (4): >> igb_uio: add wc option >> bus/pci: reference driver structure >> eal: enable WC during resources mapping >> net/ena: enable WC >> >> drivers/bus/pci/linux/pci_uio.c | 39 ++++++++++++++++++++++++++++----------- >> drivers/bus/pci/pci_common.c | 13 ++++++++----- >> drivers/bus/pci/rte_bus_pci.h | 2 ++ >> drivers/net/ena/ena_ethdev.c | 3 ++- >> kernel/linux/igb_uio/igb_uio.c | 17 ++++++++++++++--- >> 5 files changed, 54 insertions(+), 20 deletions(-) >> > Couple of thoughts on this set. > > You add an option to the kernel module to allow wc to be supported on a > device, but when we go to do PCI mapping, we either map the regular > resource file or the _wc one. Therefore: > > 1. Why not always have igb_uio support write-combining since it can be > controlled thereafter via userspace mapping one file or another? > > 2. Why not always map both resource and resource_wc files at the PCI level, > and make them available via different pointers to the driver? Then the > driver can choose at the per-access level which it wants to use. For > example, for init of a device, a driver may do all register access via > uncachable memory, and only use the write-combined support for > performance-critical parts. [I have a draft patch lying around here > somewhere that does something similar to that.] > > One last question - if using vfio-pci kernel module, do the resource_wc > files present the bars as write-combined memory type, or are they > uncachable? > > Regards, > /Bruce