From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 67A81433F4;
	Tue, 28 Nov 2023 15:40:05 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 002A542DF2;
	Tue, 28 Nov 2023 15:40:04 +0100 (CET)
Received: from smtp-relay-internal-0.canonical.com
 (smtp-relay-internal-0.canonical.com [185.125.188.122])
 by mails.dpdk.org (Postfix) with ESMTP id 4AAF042DDB
 for <dev@dpdk.org>; Tue, 28 Nov 2023 15:40:03 +0100 (CET)
Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com
 [209.85.216.70])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
 (No client certificate requested)
 by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 038E43F1C7
 for <dev@dpdk.org>; Tue, 28 Nov 2023 14:40:01 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com;
 s=20210705; t=1701182401;
 bh=8+IVgmX67uDM0bDMw9yQqA01W8bdaOt8tLmTD1didVA=;
 h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject:
 To:Cc:Content-Type;
 b=Lv68sub6xA3xVOhYxZQZG3evnWdnVAu/2wgB6JbBuv/HY4ADKQvbArlEDZBJ3XBa+
 TT2Wj7FN22qLJ4Nx+LfHgDlYruavacNFnHw9/PaP0FgqW4zDD5HY8gfdg/gLR6Eti6
 6/k0N+ZrvUdmscD1TqTt2J1BPwAV7GbIJ2TSlNZfcKKESRgWjgjPP2DSxR74kdNirL
 /BLKWfv0IY0cifbg5CKz9izB/+JItw57lIRHHK98gyRTqSQwoAORAhh3mtUzegR2K9
 ZHzW3Yr1XEfCAMSK8rwFcoz3BKio/w15CwY+RqCEEXK3qhEbkXpDI2honRNcriE98a
 +aLLgz3PIK8CQ==
Received: by mail-pj1-f70.google.com with SMTP id
 98e67ed59e1d1-2855e4715e4so8107880a91.0
 for <dev@dpdk.org>; Tue, 28 Nov 2023 06:40:00 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1701182395; x=1701787195;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=8+IVgmX67uDM0bDMw9yQqA01W8bdaOt8tLmTD1didVA=;
 b=r84mvGjDzX8PiKaaAraZHXqDpDLBbkPDI3H5qN5JGfC4Dlc8J6j2OanlMo9rZMcHXe
 Wq14sRzjCfhksd8fTjPyoShDEQ/kiWna7wV9ZBKm/7sZVQXaZO5OOZzUkleyVJuAsbqd
 3A2lzSbeLuiUkqwrvPq5slMptoFOlroZNgYdoq2YELv+UHYXkEYAE4607dB0t3+4kaao
 7yNc3Mq/umiMy776AeQJvmF7gEM4k6jVo/I5wqj5pYH8GhdjwHZfYgr4G4i5nLIUh1gL
 UDKGrq2JHSnhW20WHVCIFGh1ouQvFDngu8pvZzvImq3QOC6ehmZ3p7ia0E4SVoZQhkxU
 AVuQ==
X-Gm-Message-State: AOJu0YxmQZCnmBbFTjipS/UG41fAWli/DzWxf65GfgQU/rGyqp3ZCwFG
 WJBnHMSoM2nRHxjjy94Pdi9AGA/g1aog4ZG6nzKvPu6Fn2/1ZLDaZlA8WD9KVF2HnxIqE6m8F4s
 lyHvISckivb/v4CpSFD1Gzf9t4tkioyO8ImKrrHV0ftyu
X-Received: by 2002:a17:90a:1997:b0:280:5e66:a1e2 with SMTP id
 23-20020a17090a199700b002805e66a1e2mr15481496pji.22.1701182394920; 
 Tue, 28 Nov 2023 06:39:54 -0800 (PST)
X-Google-Smtp-Source: AGHT+IExydoB94jNvquYKK+omUNwWPPNqJ1r/88r5S4XWeAngEJBN+AI358T9T9olLDJftpBsMZDXdvdMlVdPotko/k=
X-Received: by 2002:a17:90a:1997:b0:280:5e66:a1e2 with SMTP id
 23-20020a17090a199700b002805e66a1e2mr15481475pji.22.1701182394623; Tue, 28
 Nov 2023 06:39:54 -0800 (PST)
MIME-Version: 1.0
References: <20231124100904.388453-1-christian.ehrhardt@canonical.com>
 <20231124132941.4fe0f846@sovereign>
In-Reply-To: <20231124132941.4fe0f846@sovereign>
From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
Date: Tue, 28 Nov 2023 15:39:28 +0100
Message-ID: <CAATJJ0+NNbar_5_10q1Zf-RwHZ2swJj1dKbnb6dkF80XamT-SQ@mail.gmail.com>
Subject: Re: [PATCH v2] eal/linux: force iova-mode va without pa available
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Cc: dev <dev@dpdk.org>, Luca Boccassi <bluca@debian.org>,
 David Wilder <dwilder@us.ibm.com>
Content-Type: multipart/alternative; boundary="000000000000548b7e060b37644d"
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

--000000000000548b7e060b37644d
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Nov 24, 2023 at 11:29=E2=80=AFAM Dmitry Kozlyuk <dmitry.kozliuk@gma=
il.com>
wrote:

> 2023-11-24 11:09 (UTC+0100), christian.ehrhardt@canonical.com:
> [...]
> > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
> > index 57da058cec..2f1fce3c54 100644
> > --- a/lib/eal/linux/eal.c
> > +++ b/lib/eal/linux/eal.c
> > @@ -1067,6 +1067,16 @@ rte_eal_init(int argc, char **argv)
> >
> >       phys_addrs =3D rte_eal_using_phys_addrs() !=3D 0;
> >
> > +     if (!phys_addrs) {
> > +             /* if we have no access to physical addresses, pick IOVA
> as VA mode. */
> > +             if (internal_conf->iova_mode =3D=3D RTE_IOVA_PA)
> > +                     RTE_LOG(WARNING, EAL, "WARNING: --iova-mode=3Dpa,
> but Physical addresses are unavailable, selecting IOVA as VA mode.\n");
>
> If an impossible combination of options is requested,
> initialization should fail instead.
>

You are absolutely right Dmitry.
In fact I was only trying to rebase an old patch that we used to carry.
But the more I look at and think about it the less I like the approach.

A production setup has an admin that should do this consciously.
What we actually should change is not the behavior of EAL, but just the
test automation to work on no-huge ppc64.
That is simpler and has much less impact and therefore probably is the
right solution.

Consider this patch here withdrawn.
I'll submit the fix to the tests in a few seconds.


> > +             else
> > +                     RTE_LOG(DEBUG, EAL, "Physical addresses are
> unavailable, selecting IOVA as VA mode.\n");
> > +             internal_conf->iova_mode =3D RTE_IOVA_VA;
> > +             rte_eal_get_configuration()->iova_mode =3D
> internal_conf->iova_mode;
> > +     }
> > +
> >       /* if no EAL option "--iova-mode=3D<pa|va>", use bus IOVA scheme =
*/
> >       if (internal_conf->iova_mode =3D=3D RTE_IOVA_DC) {
> >               /* autodetect the IOVA mapping mode */
>
> What do you think about keeping the existing code structure:
>
> if (--iova-mode not specified) {
>         iova_mode =3D VA if !phys_addrs or !RTE_IOVA_IN_MBUF (with logs)
>         if (iova_mode =3D=3D DC) {
>                 // autodetect from bus requirements and IOMMU (with logs)
>         }
>         rte_eal_get_configuration()->iova_mode =3D iova_mode;
> } else {
>         rte_eal_get_configuration()->iova_mode =3D
>                 internal_conf->iova_mode;
> }
> // verify rte_eal_get_configuration()->iova_mode
>
> Note: the logic should be consistent across OS when possible.
>
>

--=20
Christian Ehrhardt
Director of Engineering, Ubuntu Server
Canonical Ltd

--000000000000548b7e060b37644d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Fri, Nov 24, 2023 at 11:29=E2=80=
=AFAM Dmitry Kozlyuk &lt;<a href=3D"mailto:dmitry.kozliuk@gmail.com">dmitry=
.kozliuk@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote=
" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);=
padding-left:1ex">2023-11-24 11:09 (UTC+0100), <a href=3D"mailto:christian.=
ehrhardt@canonical.com" target=3D"_blank">christian.ehrhardt@canonical.com<=
/a>:<br>
[...]<br>
&gt; diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c<br>
&gt; index 57da058cec..2f1fce3c54 100644<br>
&gt; --- a/lib/eal/linux/eal.c<br>
&gt; +++ b/lib/eal/linux/eal.c<br>
&gt; @@ -1067,6 +1067,16 @@ rte_eal_init(int argc, char **argv)<br>
&gt;=C2=A0 <br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0phys_addrs =3D rte_eal_using_phys_addrs() !=
=3D 0;<br>
&gt;=C2=A0 <br>
&gt; +=C2=A0 =C2=A0 =C2=A0if (!phys_addrs) {<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* if we have no acce=
ss to physical addresses, pick IOVA as VA mode. */<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (internal_conf-&gt=
;iova_mode =3D=3D RTE_IOVA_PA)<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0RTE_LOG(WARNING, EAL, &quot;WARNING: --iova-mode=3Dpa, but Physical =
addresses are unavailable, selecting IOVA as VA mode.\n&quot;);<br>
<br>
If an impossible combination of options is requested,<br>
initialization should fail instead.<br></blockquote><div><br></div><div>You=
 are absolutely right Dmitry.</div><div>In fact I was only trying to rebase=
 an old patch that we used to carry.</div><div>But the more I look=C2=A0at =
and think about it the less I like the approach.</div><div><br></div><div>A=
 production setup has an admin that should do this consciously.</div><div>W=
hat we actually should change is not the behavior of EAL, but just the test=
 automation to work on no-huge ppc64.</div><div>That is simpler and has muc=
h less impact and therefore probably is the right solution.</div><div><br><=
/div><div>Consider this patch here withdrawn.</div><div>I&#39;ll submit the=
 fix to the tests in a few seconds.</div><div>=C2=A0</div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0else<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0RTE_LOG(DEBUG, EAL, &quot;Physical addresses are unavailable, select=
ing IOVA as VA mode.\n&quot;);<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0internal_conf-&gt;iov=
a_mode =3D RTE_IOVA_VA;<br>
&gt; +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte_eal_get_configura=
tion()-&gt;iova_mode =3D internal_conf-&gt;iova_mode;<br>
&gt; +=C2=A0 =C2=A0 =C2=A0}<br>
&gt; +<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0/* if no EAL option &quot;--iova-mode=3D&lt;=
pa|va&gt;&quot;, use bus IOVA scheme */<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0if (internal_conf-&gt;iova_mode =3D=3D RTE_I=
OVA_DC) {<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* autodetect th=
e IOVA mapping mode */<br>
<br>
What do you think about keeping the existing code structure:<br>
<br>
if (--iova-mode not specified) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 iova_mode =3D VA if !phys_addrs or !RTE_IOVA_IN=
_MBUF (with logs)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (iova_mode =3D=3D DC) {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // autodetect from =
bus requirements and IOMMU (with logs)<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rte_eal_get_configuration()-&gt;iova_mode =3D i=
ova_mode;<br>
} else {<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rte_eal_get_configuration()-&gt;iova_mode =3D<b=
r>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 internal_conf-&gt;i=
ova_mode;<br>
}<br>
// verify rte_eal_get_configuration()-&gt;iova_mode<br>
<br>
Note: the logic should be consistent across OS when possible.<br>
<br>
</blockquote></div><br clear=3D"all"><div><br></div><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d=
iv dir=3D"ltr">Christian Ehrhardt<br><span style=3D"color:rgb(34,34,34)">Di=
rector of Engineering</span>, Ubuntu Server<br>Canonical Ltd</div></div></d=
iv>

--000000000000548b7e060b37644d--