From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 96830433B3; Fri, 24 Nov 2023 07:30:08 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6443442F69; Fri, 24 Nov 2023 07:30:08 +0100 (CET) Received: from smtp-relay-internal-1.canonical.com (smtp-relay-internal-1.canonical.com [185.125.188.123]) by mails.dpdk.org (Postfix) with ESMTP id 834D742F31 for ; Fri, 24 Nov 2023 07:30:07 +0100 (CET) Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 4B7683FFFD for ; Fri, 24 Nov 2023 06:30:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1700807407; bh=K2Meo4Iuk0YNTCyDuqDvZz05pb20ZyrAqj5+SEi1zGg=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=Ya4nDO0O4vf73Nl1+AHyTUH3NzOJUwRpbIv4Eo/MtMCla4xGSxULBEjs6WH079pfQ w2LPO7x/zfjLujpb7SkG+cxGjmeTEnS5i4ZcKb0Tdn8KAWr+JoaPq9eoU3Msbhzdmm ea74Zu+HUoM0l9sBheVPxwc5XN7uYEARu6QX72lukcisf2AnnPyfY7/bmk25lsxy+x rqSM4f1BP4OEgdUfrVxj/6dUQoKTHD3f0Mbh+CZ929wg1obceOGO7QBBIWT7jMN3V9 7Kg6KDshhGwCyCXYKwbQstSFeacemk9Kg/0JtSA5xslWN8/+rftRYo64II4NDDbVmA dhRI08zy9l9QQ== Received: by mail-pg1-f200.google.com with SMTP id 41be03b00d2f7-5be3799791fso1885248a12.3 for ; Thu, 23 Nov 2023 22:30:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700807405; x=1701412205; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=K2Meo4Iuk0YNTCyDuqDvZz05pb20ZyrAqj5+SEi1zGg=; b=faU5ciw+HwBtnRAxA29TUUhW3PAa3XE46/C47XO+AqNHGDFRMphgy6F0aT6O8SPEGU RwveLk7oBP3GCFXVi/pDHR/GwT7UyvNQ0BH3wBUzFIKm93wUwjTYAk/e9dgXzUUIwof0 RHB/jgMDHHVeLNEjQmfINtVL4g0jDEe8U2WG4dO810N+cT1hpirqY0mWAb+LJRAZcPs/ wqiM4KPUGJKkIxVN8KPB8BB5mefha7Fq7AaWVhA7Ny8dHPpHnlYz7dsrXKavRK+vR1zz 7mK3i/53X6ySDwgioAoCn+jtqDD7CmJC3ee+YD5AcnPSdRR5vfXRnvUw7ZEi3hx43fx3 b9XQ== X-Gm-Message-State: AOJu0YzCtrxiO/l4DZ8Kq9SJa0FPkO+xhGl7KSEp+qCNY9+B4XhTTzvM 3I0fyWeUL5HtEsDvXtkE8E48fFEN3xsqIwKjJw4NOE9eDLesfGeCo1TyOFpvK2MkZ8qrs6JZ6mx iWQbtenb+wt/FCQJUTUK14+o+bltJ7VqJcvsMTfRKvpCD X-Received: by 2002:a17:90b:4b0f:b0:285:3444:94d2 with SMTP id lx15-20020a17090b4b0f00b00285344494d2mr1951257pjb.9.1700807405544; Thu, 23 Nov 2023 22:30:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IFwvgioJzk5ZP+no1YedGCgNDdU4TS/B3ma0MYBkl5McLNJn5BlmW+Y+JZwGUEhtC0cN5TaJKgvAm8s1msmM1E= X-Received: by 2002:a17:90b:4b0f:b0:285:3444:94d2 with SMTP id lx15-20020a17090b4b0f00b00285344494d2mr1951232pjb.9.1700807405039; Thu, 23 Nov 2023 22:30:05 -0800 (PST) MIME-Version: 1.0 References: <20231123155149.306556-1-christian.ehrhardt@canonical.com> <20231123155149.306556-2-christian.ehrhardt@canonical.com> In-Reply-To: <20231123155149.306556-2-christian.ehrhardt@canonical.com> From: Christian Ehrhardt Date: Fri, 24 Nov 2023 07:29:38 +0100 Message-ID: Subject: Re: [PATCH 1/1] eal/linux: force iova-mode va without pa available To: dev Cc: Luca Boccassi , David Wilder Content-Type: multipart/alternative; boundary="000000000000358477060ae01511" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000358477060ae01511 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Nov 23, 2023 at 4:52=E2=80=AFPM = wrote: > From: David Wilder > > When using --no-huge option physical address are not guaranteed > to be persistent. > > This change effectively makes "--no-huge" the same as > "--no-huge --iova-mode=3Dva". > > When --no-huge is used (or any other condition making physical > addresses unavailable) setting --iova-mode=3Dpa will have no effect. > > Signed-off-by: Christian Ehrhardt > --- > doc/guides/prog_guide/env_abstraction_layer.rst | 9 ++++++--- > lib/eal/linux/eal.c | 14 ++++++++------ > 2 files changed, 14 insertions(+), 9 deletions(-) > > diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst > b/doc/guides/prog_guide/env_abstraction_layer.rst > index 6debf54efb..20c7355e0f 100644 > --- a/doc/guides/prog_guide/env_abstraction_layer.rst > +++ b/doc/guides/prog_guide/env_abstraction_layer.rst > @@ -559,9 +559,12 @@ IOVA Mode is selected by considering what the curren= t > usable Devices on the > system require and/or support. > > On FreeBSD, RTE_IOVA_PA is always the default. On Linux, the IOVA mode i= s > -detected based on a 2-step heuristic detailed below. > +detected based on a heuristic detailed below. > > -For the first step, EAL asks each bus its requirement in terms of IOVA > mode > +For the first step, if no Physical Addresses are available RTE_IOVA_VA i= s > +selected. > + > +Then EAL asks each bus its requirement in terms of IOVA mode > and decides on a preferred IOVA mode. > > - if all buses report RTE_IOVA_PA, then the preferred IOVA mode is > RTE_IOVA_PA, > @@ -575,7 +578,7 @@ and decides on a preferred IOVA mode. > If the buses have expressed no preference on which IOVA mode to pick, > then a > default is selected using the following logic: > > -- if physical addresses are not available, RTE_IOVA_VA mode is used > +- if enable_iova_as_pa was not set at build RTE_IOVA_VA mode is used > - if /sys/kernel/iommu_groups is not empty, RTE_IOVA_VA mode is used > - otherwise, RTE_IOVA_PA mode is used > > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c > index 57da058cec..7d0eedef57 100644 > --- a/lib/eal/linux/eal.c > +++ b/lib/eal/linux/eal.c > @@ -1067,6 +1067,14 @@ rte_eal_init(int argc, char **argv) > > phys_addrs =3D rte_eal_using_phys_addrs() !=3D 0; > > + if (!phys_addrs) { > + /* if we have no access to physical addresses, > + * pick IOVA as VA mode. > + */ > + iova_mode =3D RTE_IOVA_VA; > ^^ this won't work, thanks for trying to rush it :-/ I'll fix and test it before v2 But since it wasn't my patch initially I'm happy if anyone else wants to take over ... > + RTE_LOG(INFO, EAL, "Physical addresses are unavailable, > selecting IOVA as VA mode.\n"); > + } > + > /* if no EAL option "--iova-mode=3D", use bus IOVA scheme = */ > if (internal_conf->iova_mode =3D=3D RTE_IOVA_DC) { > /* autodetect the IOVA mapping mode */ > @@ -1078,12 +1086,6 @@ rte_eal_init(int argc, char **argv) > if (!RTE_IOVA_IN_MBUF) { > iova_mode =3D RTE_IOVA_VA; > RTE_LOG(DEBUG, EAL, "IOVA as VA mode is > forced by build option.\n"); > - } else if (!phys_addrs) { > - /* if we have no access to physical > addresses, > - * pick IOVA as VA mode. > - */ > - iova_mode =3D RTE_IOVA_VA; > - RTE_LOG(DEBUG, EAL, "Physical addresses > are unavailable, selecting IOVA as VA mode.\n"); > } else if (is_iommu_enabled()) { > /* we have an IOMMU, pick IOVA as VA mode > */ > iova_mode =3D RTE_IOVA_VA; > -- > 2.34.1 > > --=20 Christian Ehrhardt Director of Engineering, Ubuntu Server Canonical Ltd --000000000000358477060ae01511 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Nov 23, 2023 at 4:52=E2=80=AF= PM <christian.ehrhar= dt@canonical.com> wrote:
From: David Wilder <dwilder@us.ibm.com>

When using --no-huge option physical address are not guaranteed
to be persistent.

This change effectively makes "--no-huge" the same as
"--no-huge --iova-mode=3Dva".

When --no-huge is used (or any other condition making physical
addresses unavailable) setting --iova-mode=3Dpa will have no effect.

Signed-off-by: Christian Ehrhardt <christian.ehrhardt@canonical.com> ---
=C2=A0doc/guides/prog_guide/env_abstraction_layer.rst |=C2=A0 9 ++++++--- =C2=A0lib/eal/linux/eal.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 14 ++++++++------<= br> =C2=A02 files changed, 14 insertions(+), 9 deletions(-)

diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/p= rog_guide/env_abstraction_layer.rst
index 6debf54efb..20c7355e0f 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -559,9 +559,12 @@ IOVA Mode is selected by considering what the current = usable Devices on the
=C2=A0system require and/or support.

=C2=A0On FreeBSD, RTE_IOVA_PA is always the default. On Linux, the IOVA mod= e is
-detected based on a 2-step heuristic detailed below.
+detected based on a heuristic detailed below.

-For the first step, EAL asks each bus its requirement in terms of IOVA mod= e
+For the first step, if no Physical Addresses are available RTE_IOVA_VA is<= br> +selected.
+
+Then EAL asks each bus its requirement in terms of IOVA mode
=C2=A0and decides on a preferred IOVA mode.

=C2=A0- if all buses report RTE_IOVA_PA, then the preferred IOVA mode is RT= E_IOVA_PA,
@@ -575,7 +578,7 @@ and decides on a preferred IOVA mode.
=C2=A0If the buses have expressed no preference on which IOVA mode to pick,= then a
=C2=A0default is selected using the following logic:

-- if physical addresses are not available, RTE_IOVA_VA mode is used
+- if enable_iova_as_pa was not set at build RTE_IOVA_VA mode is used
=C2=A0- if /sys/kernel/iommu_groups is not empty, RTE_IOVA_VA mode is used<= br> =C2=A0- otherwise, RTE_IOVA_PA mode is used

diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index 57da058cec..7d0eedef57 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -1067,6 +1067,14 @@ rte_eal_init(int argc, char **argv)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 phys_addrs =3D rte_eal_using_phys_addrs() !=3D = 0;

+=C2=A0 =C2=A0 =C2=A0 =C2=A0if (!phys_addrs) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* if we have no ac= cess to physical addresses,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 * pick IOVA as VA = mode.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 */
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0iova_mode =3D RTE_I= OVA_VA;

^^ this won't work, thanks = for trying to rush it :-/
I'll fix and test it before v2
But since it wasn't my patch initially I'm happy if anyone el= se wants to take over ...
=C2=A0
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RTE_LOG(INFO, EAL, = "Physical addresses are unavailable, selecting IOVA as VA mode.\n"= ;);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0}
+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 /* if no EAL option "--iova-mode=3D<pa|= va>", use bus IOVA scheme */
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (internal_conf->iova_mode =3D=3D RTE_IOVA= _DC) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* autodetect the I= OVA mapping mode */
@@ -1078,12 +1086,6 @@ rte_eal_init(int argc, char **argv)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 if (!RTE_IOVA_IN_MBUF) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 iova_mode =3D RTE_IOVA_VA;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 RTE_LOG(DEBUG, EAL, "IOVA as VA= mode is forced by build option.\n");
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0} else if (!phys_addrs) {
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* if we have no access to physical a= ddresses,
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 * pick IOVA as VA mode.
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 */
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0iova_mode =3D RTE_IOVA_VA;
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RTE_LOG(DEBUG, EAL, "Physical ad= dresses are unavailable, selecting IOVA as VA mode.\n");
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 } else if (is_iommu_enabled()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* we have an IOMMU, pick IOVA as VA= mode */
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 iova_mode =3D RTE_IOVA_VA;
--
2.34.1



--
Christian Ehrhardt
Di= rector of Engineering, Ubuntu Server
Canonical Ltd
--000000000000358477060ae01511--