From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 45AE943D4C; Mon, 25 Mar 2024 23:56:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B9DE840270; Mon, 25 Mar 2024 23:56:19 +0100 (CET) Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by mails.dpdk.org (Postfix) with ESMTP id B83594026C for ; Mon, 25 Mar 2024 23:56:18 +0100 (CET) Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e8f765146fso3693385b3a.0 for ; Mon, 25 Mar 2024 15:56:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=damore-org.20230601.gappssmtp.com; s=20230601; t=1711407378; x=1712012178; darn=dpdk.org; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=q6lN85IDktWsFXkrj7EntAcwZ4tgs3rETNrc/GzjZUA=; b=F0MsYDd2HvArMmgIYoBeRkaqMHAgL6PfYxAn4LeZjW+nJ/5HanrXJ18WSAgoF/WaKH GQOjO1gk9znD7T88oWvoclrpc9fkfaStQDh6hbTJfmsGyKLzaau99v0cR5jHNqwRICE1 GRdL9KVfOVw/AARI//QeT1Aj+LBEL+6gwIIwxMMUiKTk3y+O63YwrZPoMGL8AE6zDxYg t1Fb7DUV4ze5bTymWarPVatuLwspLMk0juJGnQ8a2KFzZMcIykUFpDvR5S5V1Bh3ZZQf fy4mFJ7RR+uf8Cw0Y0pHSBFjSRNI3n0tVlVAhRBQadCRJPcBs4mICIfJUQmkkJdPYS7U zrDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711407378; x=1712012178; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q6lN85IDktWsFXkrj7EntAcwZ4tgs3rETNrc/GzjZUA=; b=iY2/6971G/3yotNN4cqkuWyn+n5Mpe1tnQcU4JBoiBSE8i3BOo2eNd1Kz2lIAVFRhz 4TBGEz4uA2Y3zEYRgVW81Td3/XKRgJ9Z8JyoiYVYGGCXgeITXrataJyQU7WJnT5Opm1+ TBMur+Ios0YKWAjkklogxE/51VUBJCUzZa1Y/F8k4B6J9uyFNaEJVQ3ODDEjRfVjlWZe AyM9qv6Kb53sH7kfg60T/dOS6VE/12ueemq+EKliUkb5HUQLMi/js8Sl6yZ45N3fFVP5 tGc1w/v+beaBoRcXBBKk6Kdl78vmPIcN1cwEnG+zaBiJpamfxrS0WbgGCmK3HApCHBRe GOAQ== X-Gm-Message-State: AOJu0YwbAQF2YTyNaDS7BUrQbmObVnrtm5pxT1T6VpMUEh/7UYs9PC81 pdGdvjFsznGamJ29xxeV+pe+MHLCPOo1BvkHZEMJ/+opvkhAjEAnmFGsPgVZTFk= X-Google-Smtp-Source: AGHT+IFZYwCJuxPzh0S97Ku0pH57N9BaAaies/ot56yM6XUjXfUyo2c75BJpmrJye3X5XHuQs62LVw== X-Received: by 2002:a05:6a20:d491:b0:1a3:7de2:12b6 with SMTP id im17-20020a056a20d49100b001a37de212b6mr1289525pzb.26.1711407377627; Mon, 25 Mar 2024 15:56:17 -0700 (PDT) Received: from [10.41.1.110] ([216.205.115.244]) by smtp.gmail.com with ESMTPSA id fa4-20020a056a002d0400b006e53cc789c3sm4677281pfb.107.2024.03.25.15.56.16 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Mar 2024 15:56:16 -0700 (PDT) Date: Mon, 25 Mar 2024 15:56:09 -0700 From: Garrett D'Amore To: Bruce Richardson , Stephen Hemminger Cc: "=?utf-8?Q?dev=40dpdk.org?=" , Parthakumar Roy Message-ID: In-Reply-To: <20240325102030.46913a06@hermes.local> References: <20240325102030.46913a06@hermes.local> Subject: Re: meson option to customize RTE_PKTMBUF_HEADROOM patch X-Readdle-Message-ID: d426a895-ddc6-465d-9c8d-d713167ae2b0@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="6602010e_64661436_118d0" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --6602010e_64661436_118d0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline So we need (for reasons that I don't want to get to into in too much deta= il) that our UDP payload headers are at a specific offset in the packet. This was not a problem as long as we only used IPv4.=C2=A0=C2=A0(We have = configured 40 bytes of headroom, which is more than any of our PMDs need = by a hefty margin.) Now that we're extending to support IPv6, we need to reduce that headroom= by 20 bytes, to preserve our UDP payload offset. This has big ramifications for how we fragment our own upper layer messag= es, and it has been determined that updating the PMDs to allow us to chan= ge the headroom for this use case (on a per port basis, as we will have s= ome ports on IPv4 and others on IPv6) is the least effort, but a large ma= rgin.=C2=A0=C2=A0(Well, copying the frames via memcpy would be less devel= opment effort, but would be a performance catastrophe.) =46or transmit side we don't need this, as we can simply adjust the packe= t as needed.=C2=A0=C2=A0But for the receive side, we are kind of stuck, a= s the PMDs rely on the hard coded RTE=5FPKTMBU=46=5FHEADROOM to program r= eceive locations. As far as header splitting, that would indeed be a much much nicer soluti= on. I haven't looked in the latest code to see if header splitting is even an= option -- the version of the DPDK I'm working with is a little older (20= .11) -- we have to update but we have other local changes and so updating= is one of the things that we still have to do. At any rate, the version I did look at doesn't seem to support header spl= its on any device other than =46M10K.=C2=A0=C2=A0That's not terrifically = interesting for us.=C2=A0=C2=A0We use Mellanox, E810 (ICE), bnxt, cloud N= ICs (all of them really -- ENA, virtio-net, etc.)=C2=A0 =C2=A0We also hav= e a fair amount of ixgbe and i40e on client systems in the field. We also, unfortunately, have an older DPDK 18 with Mellanox contributions= for IPoverIB.... though I'm not sure we will try to support IPv6 there.=C2= =A0=C2=A0(We are working towards replacing that part of stack with UCX.) Unless header splitting will work on all of this (excepting the IPoIB pie= ce), then it's not something we can really use. On Mar 25, 2024 at 10:20=E2=80=AFAM -0700, Stephen Hemminger , wrote: > On Mon, 25 Mar 2024 10:01:52 +0000 > Bruce Richardson wrote: > > > On Sat, Mar 23, 2024 at 01:51:25PM -0700, Garrett D'Amore wrote: > > > > So we right now (at WEKA) have a somewhat older version of DPDK t= hat we > > > > have customized heavily, and I am going to to need to to make the= > > > > headroom *dynamic* (passed in at run time, and per port.) > > > > We have this requirement because we need payload to be at a speci= fic > > > > offset, but have to deal with different header lengths for IPv4 a= nd now > > > > IPv6. > > > > My reason for pointing this out, is that I would dearly like if w= e > > > > could collaborate on this -- this change is going to touch pretty= much > > > > every PMD (we don't need it on all of them as we only support a s= ubset > > > > of PMDs, but its still a significant set.) > > > > I'm not sure if anyone else has considered such a need -- this > > > > particular message caught my eye as I'm looking specifically in t= his > > > > area right now. > > > > > > Hi > > > > thanks for reaching out. Can you clarify a little more as to the need= for > > this requirement=3F Can you not just set the headroom value to the ma= x needed > > value for any port and use that=3F Is there an issue with having blan= k space > > at the start of a buffer=3F > > > > Thanks, > > /Bruce > > If you have to make such a deep change across all PMD's then maybe > it is not the best solution. What about being able to do some form of b= uffer > chaining or pullup. --6602010e_64661436_118d0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
So we need (for reasons that I don't want to get to= into in too much detail) that our UDP payload headers are at a specific = offset in the packet.

This was not a problem as long as we only used IPv4.&=23160;&=23160;(We h= ave configured 40 bytes of headroom, which is more than any of our PMDs n= eed by a hefty margin.)

Now that we're extending to support IPv6, we need to reduce that headroom= by 20 bytes, to preserve our UDP payload offset.

This has big ramifications for how we fragment our own upper layer messag= es, and it has been determined that updating the PMDs to allow us to chan= ge the headroom for this use case (on a per port basis, as we will have s= ome ports on IPv4 and others on IPv6) is the least effort, but a large ma= rgin.&=23160;&=23160;(Well, copying the frames via memcpy would be less d= evelopment effort, but would be a performance catastrophe.)

=46or transmit side we don't need this, as we can simply adjust the packe= t as needed.&=23160;&=23160;But for the receive side, we are kind of stuc= k, as the PMDs rely on the hard coded RTE=5FPKTMBU=46=5FHEADROOM to progr= am receive locations.

As far as header splitting, that would indeed be a much much nicer soluti= on.

I haven't looked in the latest code to see if header splitting is even an= option -- the version of the DPDK I'm working with is a little older (20= .11) -- we have to update but we have other local changes and so updating= is one of the things that we still have to do.

At any rate, the version I did look at doesn't seem to support header spl= its on any device other than =46M10K.&=23160;&=23160;That's not terrifica= lly interesting for us.&=23160;&=23160;We use Mellanox, E810 (ICE), bnxt,= cloud NICs (all of them really -- ENA, virtio-net, etc.)&=23160; &=23160= ;We also have a fair amount of ixgbe and i40e on client systems in the fi= eld.

We also, unfortunately, have an older DPDK 18 with Mellanox contributions= for IPoverIB.... though I'm not sure we will try to support IPv6 there.&= =23160;&=23160;(We are working towards replacing that part of stack with = UCX.)

Unless header splitting will work on all of this (excepting the IPoIB pie= ce), then it's not something we can really use.
On Mar 25, 2024 at 10:20=E2=80=AFAM= -0700, Stephen Hemminger <stephen=40networkplumber.org>, wrote:
On Mon, 25 Mar 2024 10:01:52 +0000
Bruce Richardson <bruce.richardson=40intel.com> wrote:

On Sat, Mar 23, 2024 at 01:51:25PM -0700, Garrett D'Amore wrote= :
> So we right now (at WEKA) have a somewhat older version of= DPDK that we
> have customized heavily, and I am going to to need to to make the > headroom *dynamic* (passed in at run time, and per port.)
> We have this requirement because we need payload to be at a specific=
> offset, but have to deal with different header lengths for IPv4 and = now
> IPv6.
> My reason for pointing this out, is that I would dearly like if we > could collaborate on this -- this change is going to touch pretty mu= ch
> every PMD (we don't need it on all of them as we only support a subs= et
> of PMDs, but its still a significant set.)
> I'm not sure if anyone else has considered such a need -- this
= > particular message caught my eye as I'm looking specifically in this=
> area right now.
>
Hi

thanks for reaching out. Can you clarify a little more as to the need for=
this requirement=3F Can you not just set the headroom value to the max ne= eded
value for any port and use that=3F Is there an issue with having blank sp= ace
at the start of a buffer=3F

Thanks,
/Bruce

If you have to make such a deep change across all PMD's then maybe
it is not the best solution. What about being able to do some form of buf= fer
chaining or pullup.
--6602010e_64661436_118d0--