From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 67A81433F4; Tue, 28 Nov 2023 15:40:05 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 002A542DF2; Tue, 28 Nov 2023 15:40:04 +0100 (CET) Received: from smtp-relay-internal-0.canonical.com (smtp-relay-internal-0.canonical.com [185.125.188.122]) by mails.dpdk.org (Postfix) with ESMTP id 4AAF042DDB for <dev@dpdk.org>; Tue, 28 Nov 2023 15:40:03 +0100 (CET) Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 038E43F1C7 for <dev@dpdk.org>; Tue, 28 Nov 2023 14:40:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1701182401; bh=8+IVgmX67uDM0bDMw9yQqA01W8bdaOt8tLmTD1didVA=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=Lv68sub6xA3xVOhYxZQZG3evnWdnVAu/2wgB6JbBuv/HY4ADKQvbArlEDZBJ3XBa+ TT2Wj7FN22qLJ4Nx+LfHgDlYruavacNFnHw9/PaP0FgqW4zDD5HY8gfdg/gLR6Eti6 6/k0N+ZrvUdmscD1TqTt2J1BPwAV7GbIJ2TSlNZfcKKESRgWjgjPP2DSxR74kdNirL /BLKWfv0IY0cifbg5CKz9izB/+JItw57lIRHHK98gyRTqSQwoAORAhh3mtUzegR2K9 ZHzW3Yr1XEfCAMSK8rwFcoz3BKio/w15CwY+RqCEEXK3qhEbkXpDI2honRNcriE98a +aLLgz3PIK8CQ== Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-2855e4715e4so8107880a91.0 for <dev@dpdk.org>; Tue, 28 Nov 2023 06:40:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701182395; x=1701787195; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8+IVgmX67uDM0bDMw9yQqA01W8bdaOt8tLmTD1didVA=; b=r84mvGjDzX8PiKaaAraZHXqDpDLBbkPDI3H5qN5JGfC4Dlc8J6j2OanlMo9rZMcHXe Wq14sRzjCfhksd8fTjPyoShDEQ/kiWna7wV9ZBKm/7sZVQXaZO5OOZzUkleyVJuAsbqd 3A2lzSbeLuiUkqwrvPq5slMptoFOlroZNgYdoq2YELv+UHYXkEYAE4607dB0t3+4kaao 7yNc3Mq/umiMy776AeQJvmF7gEM4k6jVo/I5wqj5pYH8GhdjwHZfYgr4G4i5nLIUh1gL UDKGrq2JHSnhW20WHVCIFGh1ouQvFDngu8pvZzvImq3QOC6ehmZ3p7ia0E4SVoZQhkxU AVuQ== X-Gm-Message-State: AOJu0YxmQZCnmBbFTjipS/UG41fAWli/DzWxf65GfgQU/rGyqp3ZCwFG WJBnHMSoM2nRHxjjy94Pdi9AGA/g1aog4ZG6nzKvPu6Fn2/1ZLDaZlA8WD9KVF2HnxIqE6m8F4s lyHvISckivb/v4CpSFD1Gzf9t4tkioyO8ImKrrHV0ftyu X-Received: by 2002:a17:90a:1997:b0:280:5e66:a1e2 with SMTP id 23-20020a17090a199700b002805e66a1e2mr15481496pji.22.1701182394920; Tue, 28 Nov 2023 06:39:54 -0800 (PST) X-Google-Smtp-Source: AGHT+IExydoB94jNvquYKK+omUNwWPPNqJ1r/88r5S4XWeAngEJBN+AI358T9T9olLDJftpBsMZDXdvdMlVdPotko/k= X-Received: by 2002:a17:90a:1997:b0:280:5e66:a1e2 with SMTP id 23-20020a17090a199700b002805e66a1e2mr15481475pji.22.1701182394623; Tue, 28 Nov 2023 06:39:54 -0800 (PST) MIME-Version: 1.0 References: <20231124100904.388453-1-christian.ehrhardt@canonical.com> <20231124132941.4fe0f846@sovereign> In-Reply-To: <20231124132941.4fe0f846@sovereign> From: Christian Ehrhardt <christian.ehrhardt@canonical.com> Date: Tue, 28 Nov 2023 15:39:28 +0100 Message-ID: <CAATJJ0+NNbar_5_10q1Zf-RwHZ2swJj1dKbnb6dkF80XamT-SQ@mail.gmail.com> Subject: Re: [PATCH v2] eal/linux: force iova-mode va without pa available To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> Cc: dev <dev@dpdk.org>, Luca Boccassi <bluca@debian.org>, David Wilder <dwilder@us.ibm.com> Content-Type: multipart/alternative; boundary="000000000000548b7e060b37644d" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org --000000000000548b7e060b37644d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Nov 24, 2023 at 11:29=E2=80=AFAM Dmitry Kozlyuk <dmitry.kozliuk@gma= il.com> wrote: > 2023-11-24 11:09 (UTC+0100), christian.ehrhardt@canonical.com: > [...] > > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c > > index 57da058cec..2f1fce3c54 100644 > > --- a/lib/eal/linux/eal.c > > +++ b/lib/eal/linux/eal.c > > @@ -1067,6 +1067,16 @@ rte_eal_init(int argc, char **argv) > > > > phys_addrs =3D rte_eal_using_phys_addrs() !=3D 0; > > > > + if (!phys_addrs) { > > + /* if we have no access to physical addresses, pick IOVA > as VA mode. */ > > + if (internal_conf->iova_mode =3D=3D RTE_IOVA_PA) > > + RTE_LOG(WARNING, EAL, "WARNING: --iova-mode=3Dpa, > but Physical addresses are unavailable, selecting IOVA as VA mode.\n"); > > If an impossible combination of options is requested, > initialization should fail instead. > You are absolutely right Dmitry. In fact I was only trying to rebase an old patch that we used to carry. But the more I look at and think about it the less I like the approach. A production setup has an admin that should do this consciously. What we actually should change is not the behavior of EAL, but just the test automation to work on no-huge ppc64. That is simpler and has much less impact and therefore probably is the right solution. Consider this patch here withdrawn. I'll submit the fix to the tests in a few seconds. > > + else > > + RTE_LOG(DEBUG, EAL, "Physical addresses are > unavailable, selecting IOVA as VA mode.\n"); > > + internal_conf->iova_mode =3D RTE_IOVA_VA; > > + rte_eal_get_configuration()->iova_mode =3D > internal_conf->iova_mode; > > + } > > + > > /* if no EAL option "--iova-mode=3D<pa|va>", use bus IOVA scheme = */ > > if (internal_conf->iova_mode =3D=3D RTE_IOVA_DC) { > > /* autodetect the IOVA mapping mode */ > > What do you think about keeping the existing code structure: > > if (--iova-mode not specified) { > iova_mode =3D VA if !phys_addrs or !RTE_IOVA_IN_MBUF (with logs) > if (iova_mode =3D=3D DC) { > // autodetect from bus requirements and IOMMU (with logs) > } > rte_eal_get_configuration()->iova_mode =3D iova_mode; > } else { > rte_eal_get_configuration()->iova_mode =3D > internal_conf->iova_mode; > } > // verify rte_eal_get_configuration()->iova_mode > > Note: the logic should be consistent across OS when possible. > > --=20 Christian Ehrhardt Director of Engineering, Ubuntu Server Canonical Ltd --000000000000548b7e060b37644d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Fri, Nov 24, 2023 at 11:29=E2=80= =AFAM Dmitry Kozlyuk <<a href=3D"mailto:dmitry.kozliuk@gmail.com">dmitry= .kozliuk@gmail.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote= " style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);= padding-left:1ex">2023-11-24 11:09 (UTC+0100), <a href=3D"mailto:christian.= ehrhardt@canonical.com" target=3D"_blank">christian.ehrhardt@canonical.com<= /a>:<br> [...]<br> > diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c<br> > index 57da058cec..2f1fce3c54 100644<br> > --- a/lib/eal/linux/eal.c<br> > +++ b/lib/eal/linux/eal.c<br> > @@ -1067,6 +1067,16 @@ rte_eal_init(int argc, char **argv)<br> >=C2=A0 <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0phys_addrs =3D rte_eal_using_phys_addrs() != =3D 0;<br> >=C2=A0 <br> > +=C2=A0 =C2=A0 =C2=A0if (!phys_addrs) {<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* if we have no acce= ss to physical addresses, pick IOVA as VA mode. */<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (internal_conf->= ;iova_mode =3D=3D RTE_IOVA_PA)<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0RTE_LOG(WARNING, EAL, "WARNING: --iova-mode=3Dpa, but Physical = addresses are unavailable, selecting IOVA as VA mode.\n");<br> <br> If an impossible combination of options is requested,<br> initialization should fail instead.<br></blockquote><div><br></div><div>You= are absolutely right Dmitry.</div><div>In fact I was only trying to rebase= an old patch that we used to carry.</div><div>But the more I look=C2=A0at = and think about it the less I like the approach.</div><div><br></div><div>A= production setup has an admin that should do this consciously.</div><div>W= hat we actually should change is not the behavior of EAL, but just the test= automation to work on no-huge ppc64.</div><div>That is simpler and has muc= h less impact and therefore probably is the right solution.</div><div><br><= /div><div>Consider this patch here withdrawn.</div><div>I'll submit the= fix to the tests in a few seconds.</div><div>=C2=A0</div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg= b(204,204,204);padding-left:1ex">> +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0else<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0RTE_LOG(DEBUG, EAL, "Physical addresses are unavailable, select= ing IOVA as VA mode.\n");<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0internal_conf->iov= a_mode =3D RTE_IOVA_VA;<br> > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rte_eal_get_configura= tion()->iova_mode =3D internal_conf->iova_mode;<br> > +=C2=A0 =C2=A0 =C2=A0}<br> > +<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0/* if no EAL option "--iova-mode=3D<= pa|va>", use bus IOVA scheme */<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0if (internal_conf->iova_mode =3D=3D RTE_I= OVA_DC) {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* autodetect th= e IOVA mapping mode */<br> <br> What do you think about keeping the existing code structure:<br> <br> if (--iova-mode not specified) {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 iova_mode =3D VA if !phys_addrs or !RTE_IOVA_IN= _MBUF (with logs)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (iova_mode =3D=3D DC) {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // autodetect from = bus requirements and IOMMU (with logs)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 rte_eal_get_configuration()->iova_mode =3D i= ova_mode;<br> } else {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 rte_eal_get_configuration()->iova_mode =3D<b= r> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 internal_conf->i= ova_mode;<br> }<br> // verify rte_eal_get_configuration()->iova_mode<br> <br> Note: the logic should be consistent across OS when possible.<br> <br> </blockquote></div><br clear=3D"all"><div><br></div><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d= iv dir=3D"ltr">Christian Ehrhardt<br><span style=3D"color:rgb(34,34,34)">Di= rector of Engineering</span>, Ubuntu Server<br>Canonical Ltd</div></div></d= iv> --000000000000548b7e060b37644d--