From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5935443BF7; Wed, 28 Feb 2024 16:33:33 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2BEAA410D5; Wed, 28 Feb 2024 16:33:33 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 37A6040DF6 for ; Wed, 28 Feb 2024 16:33:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1709134411; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OpBCf2izkvZYlefV2zml5ifuyqFo3mLOhYNlNp/a/uQ=; b=e3v8zL2029tuqB4qhIE5ZeBf7OKaLv9hN5D9oidUhDnOVoAw5QRWbzGB6BNtXyxbu1PsIA 8krtiSrs7IN0Bh9xircVOWrssfDW5v1Tgyy9w1Nc7c0M7MDBIPGuvRRxwEv4dReaNw2xxw lw5RfQdA5aSrUhwkfTT6OJofwzvSymQ= Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-331-kqpFpS33MtmPJQJzUOS0lw-1; Wed, 28 Feb 2024 10:33:28 -0500 X-MC-Unique: kqpFpS33MtmPJQJzUOS0lw-1 Received: by mail-lj1-f200.google.com with SMTP id 38308e7fff4ca-2d25a02f48fso46448881fa.3 for ; Wed, 28 Feb 2024 07:33:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709134407; x=1709739207; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OpBCf2izkvZYlefV2zml5ifuyqFo3mLOhYNlNp/a/uQ=; b=ihsXQJKW9KZ7kq3gx0C6Kg2YKHWwIHA0JdRt1QN4Z48eqV99j6X3NVY4RkR+k10XVb +BPc/GnE4Tq6YMK4WALxuciQNnk6OS8hYgRM/StlHaoyE/W+U5YY3N9aYFbtQ+RIb+jd pIgheDi5Gsyqk+AJFFK3pR37A6z4Hqk1sVXIAU+mD2fTyiJ9W3+vDj2CWiRpg/EVLX7L LALMHCNrQnNvESHi9RZa+yJZrveAJSiHwTCTCzg/prkFqsLJ9dLlgLv5V2Jm8X4TBU2Q J0xVERIcrNg0VxgfSIDUMBO5WLYAkCOD1PTHjpZL5zcE5dJmRCDyJxFzYM1qodWNw/xt Nm6A== X-Forwarded-Encrypted: i=1; AJvYcCUfdDXZ38SbStvm/72yOWGqyy2BFjmEIZyTv9VvtSb9KKGkxk3q11JYi2CirolYrRoXSDLULpjbydTc3pg= X-Gm-Message-State: AOJu0YwZOAVfKfV/oznfjCAa5q3bsWALssIVTALnLt3/t9Hpubb7Wjjd cMCYfDgiR0AuSH8yvs9fmFrL05DUvA1Eb4Pmzo/Zv5skgJP2+dSQdzG4AMQqfqb4fRib6/w4rDc hg0NiaBo3ukd6Jg3VYbS37/KMYXkAuKQqirGQ0jyLARaITKkbib/hU6JzR6sC+5VUzy2E/pwxXP Bcd/iSSj3b5NpGgUA= X-Received: by 2002:a2e:a58f:0:b0:2d2:4fa4:f4b2 with SMTP id m15-20020a2ea58f000000b002d24fa4f4b2mr10031607ljp.28.1709134406883; Wed, 28 Feb 2024 07:33:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IEXvKZ1lHMNTfWbbnDyu+y/Sazv6m5y0ZGamvMLx+ytUSdwM/R1CX+Ybrw8pl6S77AQStJdUMrgvZ+06kYjIj4= X-Received: by 2002:a2e:a58f:0:b0:2d2:4fa4:f4b2 with SMTP id m15-20020a2ea58f000000b002d24fa4f4b2mr10031569ljp.28.1709134406498; Wed, 28 Feb 2024 07:33:26 -0800 (PST) MIME-Version: 1.0 References: <1706657173-26166-1-git-send-email-roretzla@linux.microsoft.com> <1709012499-12813-1-git-send-email-roretzla@linux.microsoft.com> <1709012499-12813-21-git-send-email-roretzla@linux.microsoft.com> <98CBD80474FA8B44BF855DF32C47DC35E9F283@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35E9F283@smartserver.smartshare.dk> From: David Marchand Date: Wed, 28 Feb 2024 16:33:14 +0100 Message-ID: Subject: Re: [PATCH v6 20/23] mbuf: remove and stop using rte marker fields To: =?UTF-8?Q?Morten_Br=C3=B8rup?= Cc: Tyler Retzlaff , dev@dpdk.org, Ajit Khaparde , Andrew Boyer , Andrew Rybchenko , Bruce Richardson , Chenbo Xia , Chengwen Feng , Dariusz Sosnowski , David Christensen , Hyong Youb Kim , Jerin Jacob , Jie Hai , Jingjing Wu , John Daley , Kevin Laatz , Kiran Kumar K , Konstantin Ananyev , Maciej Czekaj , Matan Azrad , Maxime Coquelin , Nithin Dabilpuram , Ori Kam , Ruifeng Wang , Satha Rao , Somnath Kotur , Suanming Mou , Sunil Kumar Kori , Viacheslav Ovsiienko , Yisen Zhuang , Yuying Zhang , Thomas Monjalon X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, Feb 28, 2024 at 4:01=E2=80=AFPM Morten Br=C3=B8rup wrote: > > > From: David Marchand [mailto:david.marchand@redhat.com] > > Sent: Wednesday, 28 February 2024 15.19 > > > > On Tue, Feb 27, 2024 at 6:44=E2=80=AFAM Tyler Retzlaff > > wrote: > > > > > > RTE_MARKER typedefs are a GCC extension unsupported by MSVC. Remove > > > RTE_MARKER fields from rte_mbuf struct. > > > > > > Maintain alignment of fields after removed cacheline1 marker by placi= ng > > > C11 alignas(RTE_CACHE_LINE_MIN_SIZE). > > > > > > Update implementation of rte_mbuf_prefetch_part1() and > > > rte_mbuf_prefetch_part2() inline functions calculate pointer for > > > prefetch of cachline0 and cachline1 without using removed markers. > > > > > > Update static_assert of rte_mbuf struct fields to reference data_off = and > > > packet_type fields that occupy the original offsets of the marker > > > fields. > > > > > > Signed-off-by: Tyler Retzlaff > > > --- > > > doc/guides/rel_notes/release_24_03.rst | 9 ++++++++ > > > lib/mbuf/rte_mbuf.h | 4 ++-- > > > lib/mbuf/rte_mbuf_core.h | 39 +++++++++++++-----------= ------- > > --- > > > 3 files changed, 26 insertions(+), 26 deletions(-) > > > > > > diff --git a/doc/guides/rel_notes/release_24_03.rst > > b/doc/guides/rel_notes/release_24_03.rst > > > index 879bb49..67750f2 100644 > > > --- a/doc/guides/rel_notes/release_24_03.rst > > > +++ b/doc/guides/rel_notes/release_24_03.rst > > > @@ -156,6 +156,15 @@ Removed Items > > > The application reserved statically defined logtypes > > ``RTE_LOGTYPE_USER1..RTE_LOGTYPE_USER8`` > > > are still defined. > > > > > > +* mbuf: ``RTE_MARKER`` fields ``cacheline0`` ``cacheline1`` > > > + ``rx_descriptor_fields1`` and ``RTE_MARKER64`` field ``rearm_data`= ` > > > + have been removed from ``struct rte_mbuf``. > > > + Prefetch of ``cacheline0`` and ``cacheline1`` may be achieved thro= ugh > > > + ``rte_mbuf_prefetch_part1()`` and ``rte_mbuf_prefetch_part2()`` in= line > > > + functions respectively. > > > + Access to ``rearm_data`` and ``rx_descriptor_fields1`` should be > > > + through new inline functions ``rte_mbuf_rearm_data()`` and > > > + ``rte_mbuf_rx_descriptor_fields1()`` respectively. > > > > > > API Changes > > > ----------- > > > diff --git a/lib/mbuf/rte_mbuf.h b/lib/mbuf/rte_mbuf.h > > > index aa7495b..61cda20 100644 > > > --- a/lib/mbuf/rte_mbuf.h > > > +++ b/lib/mbuf/rte_mbuf.h > > > @@ -108,7 +108,7 @@ > > > static inline void > > > rte_mbuf_prefetch_part1(struct rte_mbuf *m) > > > { > > > - rte_prefetch0(&m->cacheline0); > > > + rte_prefetch0(m); > > > } > > > > > > /** > > > @@ -126,7 +126,7 @@ > > > rte_mbuf_prefetch_part2(struct rte_mbuf *m) > > > { > > > #if RTE_CACHE_LINE_SIZE =3D=3D 64 > > > - rte_prefetch0(&m->cacheline1); > > > + rte_prefetch0(RTE_PTR_ADD(m, RTE_CACHE_LINE_MIN_SIZE)); > > > #else > > > RTE_SET_USED(m); > > > #endif > > > diff --git a/lib/mbuf/rte_mbuf_core.h b/lib/mbuf/rte_mbuf_core.h > > > index 36551c2..4e06f15 100644 > > > --- a/lib/mbuf/rte_mbuf_core.h > > > +++ b/lib/mbuf/rte_mbuf_core.h > > > @@ -18,6 +18,7 @@ > > > > > > #include > > > #include > > > +#include > > > #include > > > > > > #include > > > @@ -467,8 +468,6 @@ enum { > > > * The generic rte_mbuf, containing a packet mbuf. > > > */ > > > struct rte_mbuf { > > > - RTE_MARKER cacheline0; > > > - > > > void *buf_addr; /**< Virtual address of segment buf= fer. */ > > > #if RTE_IOVA_IN_MBUF > > > /** > > > @@ -495,7 +494,6 @@ struct rte_mbuf { > > > * To obtain a pointer to rearm_data use the rte_mbuf_rearm_d= ata() > > > * accessor instead of directly referencing through the data_= off > > field. > > > */ > > > - RTE_MARKER64 rearm_data; > > > uint16_t data_off; > > > > One subtile change of removing the marker is that fields may not be > > aligned as before. > > > > #if RTE_IOVA_IN_MBUF > > /** > > * Physical address of segment buffer. > > * This field is undefined if the build is configured to use on= ly > > * virtual address as IOVA (i.e. RTE_IOVA_IN_MBUF is 0). > > * Force alignment to 8-bytes, so as to ensure we have the exac= t > > * same mbuf cacheline0 layout for 32-bit and 64-bit. This make= s > > * working on vector drivers easier. > > */ > > rte_iova_t buf_iova __rte_aligned(sizeof(rte_iova_t)); > > #else > > /** > > * Next segment of scattered packet. > > * This field is valid when physical address field is undefined= . > > * Otherwise next pointer in the second cache line will be used= . > > */ > > struct rte_mbuf *next; > > #endif > > > > When building ! RTE_IOVA_IN_MBUF on a 32 bits arch, the next pointer > > is not force aligned to 64bits. > > Which has a cascade effect on data_off alignement. > > > > In file included from ../lib/mbuf/rte_mbuf_core.h:19, > > from ../lib/mbuf/rte_mbuf.h:42, > > from ../lib/mbuf/rte_mbuf_dyn.c:18: > > ../lib/mbuf/rte_mbuf_core.h:676:1: error: static assertion failed: "dat= a_off" > > 676 | static_assert(!(offsetof(struct rte_mbuf, data_off) !=3D > > | ^~~~~~~~~~~~~ > > > > > > I hope reviewers pay attention to the alignment changes when removing > > those markers. > > This is not trivial to catch in the CI. > > Good catch, David. > > I wonder about the reason for 64 bit aligning the rearm_data group of fie= lds? Perhaps it's there for (64 bit arch) vector instruction purposes? > > Regardless, it's an ABI break, so padding or an alignment attribute must = be added to avoid ABI breakage. If there is no valid reason for the 64 bit = alignment, it could be noted that the padding (or alignment attribute) is t= here for 32 bit arch ABI compatibility reasons only. > > Please note that only RTE_MARKER64 is affected by this. The other marker = types have arch bit-width (or smaller) alignment, i.e. RTE_MARKER is 8 byte= aligned on 64 bit arch and 4 byte aligned on 32 bit arch. Well, strictly speaking other RTE_MARKER users *may* be affected, depending on the alignement for the following fields. For example, I think removing the rxq_fastpath_data_end RTE_MARKER in struct nicvf_rxq (https://git.dpdk.org/dpdk/tree/drivers/net/thunderx/nicvf_struct.h#n72) impacts rx_drop_en alignment and subsequent fields. Now, in practice and focusing only on what this series touches, either the markers were coupled with an explicit alignment constraint (__rte_cache*_aligned) which is preserved by the series, or the alignement constraint is stronger than that of the marker. So there is probably only this ABI breakage I reported. --=20 David Marchand