From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by inbox.dpdk.org (Postfix) with ESMTP id C6BC2A04C2;
Fri, 15 Nov 2019 14:40:54 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 89B142C18;
Fri, 15 Nov 2019 14:40:54 +0100 (CET)
Received: from mail-io1-f68.google.com (mail-io1-f68.google.com
[209.85.166.68]) by dpdk.org (Postfix) with ESMTP id 2A8F82C08
for <dev@dpdk.org>; Fri, 15 Nov 2019 14:40:53 +0100 (CET)
Received: by mail-io1-f68.google.com with SMTP id v17so10424225iol.12
for <dev@dpdk.org>; Fri, 15 Nov 2019 05:40:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=WFZbf/etmCmkecOBNCOQN9GpsXLRimG+ObZvF3Wt+lI=;
b=pZBnFu1ih7hjkFM1PVUowV9GFWFQlu433aQ1U4toW7SxXesw4urJwm537AlYs7ioNv
lNoTeUtHeKSF7wfFTGmWlKRBTSAiTIYQLjra8asyfwrK0MatyHmL9PNHEs4xjl7fLqFZ
Rmi61iV7M/iQkCC/f0XBWUO28EFWGuZr+rznVdmigjyJ5qSPU9O0bNxYm+1GtZzyT/rJ
oHwMjYrJEIt1MxpqCjUA8fcAW2OZEo5LpI7NpgME8A/YqZe/R/JmiRCjVhsN8iCXdTuf
xUxeztp+4BaAITfSEZ8AoMNk2uUJ2UsVq1E1BCltbX7ZVFJ3I48k0cC/GDzEg94SYk8f
67ag==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=WFZbf/etmCmkecOBNCOQN9GpsXLRimG+ObZvF3Wt+lI=;
b=mpnVBMcdZ2nLlIl4670AOcZjhfLZjYFpZyH97tka6ofyi1+R910YV96XxJpQDdax2C
JwV5yJwVuLidDpJAUxA+YoJC2UqJc/KcXTrp6o0JHyKRDWa5okV1J42gpLaW4pOCwLVu
hrer2du3XCbK6JkROk2tshf0p/OfOjRI0afD2Y0Vq/z6Z9B/HlKfB8qJlaVshkE1MEEh
Mvn0B0+3FNtRUPBtVwf+9yrOzH8rOa2yXzaf9Ehlrf7F47eAKGgRe36NrYoLTwQ6PxC5
UFbaXQEDN0P1Nl7oEw/2IOH7ScbMPyWlVK3x80K5UlfXgEE9JFQKOHtAQzgNjmTE/cDJ
8xGg==
X-Gm-Message-State: APjAAAXmPOo9G8OH9iWpAfh82uSuAhIKGYJnHlsCFZTDLkE3esCtd6Ze
loHzXBdyhJUwxhQ0bUXdHv3ZQYIJYvtR3cb2BUE=
X-Google-Smtp-Source: APXvYqx7LY2YD9Lf2gn63gFt9sq8F3j1RRrxyojP3TPCaQwvEZj41YywL3syA79HqyoBz5LKlUgdYX/TTcY4TImRKeE=
X-Received: by 2002:a5d:8056:: with SMTP id b22mr678726ior.60.1573825252090;
Fri, 15 Nov 2019 05:40:52 -0800 (PST)
MIME-Version: 1.0
References: <20191105110416.8955-1-vattunuru@marvell.com>
<20191115111807.20935-1-vattunuru@marvell.com>
<20191115111807.20935-3-vattunuru@marvell.com>
<CAJFAV8y=euK9XzzqYZn1O-MVBgreFE4EL-WAWkHtJTq3mN4VXw@mail.gmail.com>
In-Reply-To: <CAJFAV8y=euK9XzzqYZn1O-MVBgreFE4EL-WAWkHtJTq3mN4VXw@mail.gmail.com>
From: Jerin Jacob <jerinjacobk@gmail.com>
Date: Fri, 15 Nov 2019 19:10:34 +0530
Message-ID: <CALBAE1MM3Q6SgRxRJuCb8Jjzg-NcyMWmoBQcvjq+_Dtx1+gyXA@mail.gmail.com>
To: David Marchand <david.marchand@redhat.com>
Cc: Vamsi Attunuru <vattunuru@marvell.com>, "Yigit,
Ferruh" <ferruh.yigit@intel.com>, dev <dev@dpdk.org>,
Thomas Monjalon <thomas@monjalon.net>,
Jerin Jacob Kollanukkaran <jerinj@marvell.com>,
Kiran Kumar Kokkilagadda <kirankumark@marvell.com>,
Olivier Matz <olivier.matz@6wind.com>,
"Burakov, Anatoly" <anatoly.burakov@intel.com>,
Andrew Rybchenko <arybchenko@solarflare.com>,
Stephen Hemminger <stephen@networkplumber.org>
Content-Type: text/plain; charset="UTF-8"
Subject: Re: [dpdk-dev] [PATCH v13 2/2] kni: support IOVA mode
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
On Fri, Nov 15, 2019 at 6:29 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> I can't see an interest in splitting this patch from the kmod update.
> Ferruh, what do you think?
>
>
> On Fri, Nov 15, 2019 at 12:19 PM <vattunuru@marvell.com> wrote:
> >
> > From: Vamsi Attunuru <vattunuru@marvell.com>
> >
> > Current KNI implementation only operates in IOVA_PA mode
> > patch adds required functionality to enable KNI in
> > IOVA_VA mode.
> >
> > KNI loopback mode tests will have performance impact in
> > this mode due to IOVA to KVA address translations.
> > However, In KNI real world use cases, the performace
>
> performance
>
> > impact will be based on Linux kernel stack and scheduler
> > latencies. Performance varies based on the KNI use case.
> > If bus iommu scheme is IOVA_DC and KNI module is loaded,
> > DPDK chooses IOVA as PA as existing behaviour.
> >
> > During KNI creation, app's iova_mode details are passed to
> > the KNI kernel module, accordingly kernel module translates
> > PA/IOVA addresses to KVA and vice-versa.
> >
> > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> > Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
> > Suggested-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > ---
> > doc/guides/prog_guide/kernel_nic_interface.rst | 15 +++++++++++++++
> > doc/guides/rel_notes/release_19_11.rst | 15 ++++++++++++++-
> > lib/librte_eal/linux/eal/eal.c | 23 ++++++++++++++++-------
> > lib/librte_kni/rte_kni.c | 6 ++++++
> > 4 files changed, 51 insertions(+), 8 deletions(-)
> >
>
> [snip]
>
> > diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> > index 9e2d50c..53ca84b 100644
> > --- a/lib/librte_eal/linux/eal/eal.c
> > +++ b/lib/librte_eal/linux/eal/eal.c
> > @@ -1086,14 +1086,23 @@ rte_eal_init(int argc, char **argv)
> > }
> > }
> > #ifdef RTE_LIBRTE_KNI
> > - /* Workaround for KNI which requires physical address to work */
> > - if (iova_mode == RTE_IOVA_VA &&
> > - rte_eal_check_module("rte_kni") == 1) {
> > - if (phys_addrs) {
> > + if (rte_eal_check_module("rte_kni") == 1) {
> > +#if KERNEL_VERSION(4, 6, 0) > LINUX_VERSION_CODE
> > + if (iova_mode == RTE_IOVA_VA) {
> > iova_mode = RTE_IOVA_PA;
> > - RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because KNI module is loaded\n");
> > - } else {
> > - RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
> > + RTE_LOG(WARNING, EAL, "Forcing IOVA as 'PA' because "
> > + "Kernel version supports only 'PA' mode for KNI module\n");
> > + }
> > +#endif
> > + if (rte_bus_get_iommu_class() == RTE_IOVA_DC)
> > + iova_mode = RTE_IOVA_PA;
>
> If physical addresses are unavailable, this code forces PA anyway.
>
>
> > +
> > + if (iova_mode == RTE_IOVA_PA) {
> > + if (phys_addrs && is_iommu_enabled())
> > + RTE_LOG(WARNING, EAL, "Forced IOVA as 'PA' because KNI module is loaded\n");
> > +
> > + if (!phys_addrs)
> > + RTE_LOG(DEBUG, EAL, "KNI can not work since physical addresses are unavailable\n");
> > }
>
> Checking physical addresses availability after, and having a log is not enough.
>
>
> So far, KNI could not work with IOVA as VA.
> Your patchset adds support for IOVA as VA if kernel is >= 4.6.
> Repeating my proposal (as far as eal.c is concerned) of just changing:
We need achive the following.
IOVA as PA has performance implication on KNI case. So we need to
make IOVA as PA when KNI module is loaded.
Your suggestion makes IOVA as PA when bus return IOVA as VA due the
fact the KNI check is latter. We need to move that up.
How about the following
[master]dell[dpdk.org] $ git diff
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 9e2d50cfb..085fde767 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1064,6 +1064,21 @@ rte_eal_init(int argc, char **argv)
/* autodetect the IOVA mapping mode */
enum rte_iova_mode iova_mode = rte_bus_get_iommu_class();
+#if defined(RTE_LIBRTE_KNI) && KERNEL_VERSION(4, 6, 0) > LINUX_VERSION_CODE
+ /* IOVA as PA gives better performance for KNI. Choose IOVA as
+ * PA when bus returns RTE_IOVA_DC and KNI module is present.
+ */
+ if (iova_mode == RTE_IOVA_DC &&
+ rte_eal_check_module("rte_kni") == 1) {
+ if (phys_addrs) {
+ iova_mode = RTE_IOVA_PA;
+ RTE_LOG(WARNING, EAL, "Forcing IOVA as
'PA' because KNI module is loaded\n");
+ } else {
+ RTE_LOG(DEBUG, EAL, "KNI can not work
since physical addresses are unavailable\n");
+ }
+ }
+#endif
+
if (iova_mode == RTE_IOVA_DC) {
RTE_LOG(DEBUG, EAL, "Buses did not request a
specific IOVA mode.\n");
@@ -1085,18 +1100,6 @@ rte_eal_init(int argc, char **argv)
RTE_LOG(DEBUG, EAL, "IOMMU is not
available, selecting IOVA as PA mode.\n");
}
}
-#ifdef RTE_LIBRTE_KNI
- /* Workaround for KNI which requires physical address to work */
- if (iova_mode == RTE_IOVA_VA &&
- rte_eal_check_module("rte_kni") == 1) {
- if (phys_addrs) {
- iova_mode = RTE_IOVA_PA;
- RTE_LOG(WARNING, EAL, "Forcing IOVA as
'PA' because KNI module is loaded\n");
- } else {
- RTE_LOG(DEBUG, EAL, "KNI can not work
since physical addresses are unavailable\n");
- }
- }
-#endif
rte_eal_get_configuration()->iova_mode = iova_mode;
} else {
rte_eal_get_configuration()->iova_mode =
[master]dell[dpdk.org] $
>
> @@ -1085,7 +1085,7 @@ rte_eal_init(int argc, char **argv)
> RTE_LOG(DEBUG, EAL, "IOMMU is not
> available, selecting IOVA as PA mode.\n");
> }
> }
> -#ifdef RTE_LIBRTE_KNI
> +#if defined(RTE_LIBRTE_KNI) && KERNEL_VERSION(4, 6, 0) > LINUX_VERSION_CODE
> /* Workaround for KNI which requires physical address to work */
> if (iova_mode == RTE_IOVA_VA &&
>
>
>
> --
> David Marchand
>