DPDK patches and discussions
 help / color / mirror / Atom feed
From: Morten Brørup <mb@smartsharesystems.com>
To: "Bruce Richardson" <bruce.richardson@intel.com>
Cc: "Thomas Monjalon" <thomas@monjalon.net>, <dev@dpdk.org>,
	<techboard@dpdk.org>,
	"Ajit Khaparde" <ajit.khaparde@broadcom.com>,
	"Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	"Andrew Rybchenko" <andrew.rybchenko@oktetlabs.ru>,
	"Yigit, Ferruh" <ferruh.yigit@intel.com>,
	<david.marchand@redhat.com>, <olivier.matz@6wind.com>,
	<jerinj@marvell.com>, <viacheslavo@nvidia.com>,
	<honnappa.nagarahalli@arm.com>, <maxime.coquelin@redhat.com>,
	<stephen@networkplumber.org>, <hemant.agrawal@nxp.com>,
	"Matan Azrad" <matan@nvidia.com>,
	"Shahaf Shuler" <shahafs@nvidia.com>
Subject: Re: [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotterfirst half
Date: Tue, 3 Nov 2020 15:03:50 +0100
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35C613DD@smartserver.smartshare.dk> (raw)
In-Reply-To: <20201103135025.GD1144@bricha3-MOBL.ger.corp.intel.com>

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Tuesday, November 3, 2020 2:50 PM
> 
> On Tue, Nov 03, 2020 at 02:46:17PM +0100, Morten Brørup wrote:
> > > From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> > > Sent: Tuesday, November 3, 2020 1:26 PM
> > >
> > > On Tue, Nov 03, 2020 at 01:10:05PM +0100, Morten Brørup wrote:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > Sent: Monday, November 2, 2020 4:58 PM
> > > > >
> > > > > +Cc techboard
> > > > >
> > > > > We need benchmark numbers in order to take a decision.
> > > > > Please all, prepare some arguments and numbers so we can
> discuss
> > > > > the mbuf layout in the next techboard meeting.
> > > >
> > > > I propose that the techboard considers this from two angels:
> > > >
> > > > 1. Long term goals and their relative priority. I.e. what can be
> > > > achieved with wide-ranging modifications, requiring yet another
> ABI
> > > > break and due notices.
> > > >
> > > > 2. Short term goals, i.e. what can be achieved for this release.
> > > >
> > > >
> > > > My suggestions follow...
> > > >
> > > > 1. Regarding long term goals:
> > > >
> > > > I have argued that simple forwarding of non-segmented packets
> using
> > > > only the first mbuf cache line can be achieved by making three
> > > > modifications:
> > > >
> > > > a) Move m->tx_offload to the first cache line.
> > > > b) Use an 8 bit pktmbuf mempool index in the first cache line,
> > > >    instead of the 64 bit m->pool pointer in the second cache
> line.
> > > > c) Do not access m->next when we know that it is NULL.
> > > >    We can use m->nb_segs == 1 or some other invariant as the
> gate.
> > > >    It can be implemented by adding an m->next accessor function:
> > > >    struct rte_mbuf * rte_mbuf_next(struct rte_mbuf * m)
> > > >    {
> > > >        return m->nb_segs == 1 ? NULL : m->next;
> > > >    }
> > > >
> > > > Regarding the priority of this goal, I guess that simple
> forwarding
> > > > of non-segmented packets is probably the path taken by the
> majority
> > > > of packets handled by DPDK.
> > > >
> > > >
> > > > An alternative goal could be:
> > > > Do not touch the second cache line during RX.
> > > > A comment in the mbuf structure says so, but it is not true
> anymore.
> > > >
> > >
> > > The comment should be true for non-scattered RX, I believe.
> >
> > You are correct.
> >
> > My suggestion was unclear: Extend this remark to include segmented
> packets.
> >
> > This could be a priority if the techboard considers RX segmented
> packets more important than my suggestion for single cache line
> forwarding of non-segmented packets.
> >
> >
> > > I'm not aware of any use of second cacheline for the fast-path RXs
> for many drivers.
> > > Am I missing something that has changed recently here?
> >
> > Check out eth_igb_recv_pkts() in the E1000 driver: rxm->next = NULL;
> > Or pmd_rx_burst() in the TAP driver: new_tail->next = seg->next;
> >
> > Perhaps the documentation should describe best practices for
> implementing RX and TX functions in drivers, including
> allocating/freeing mbufs. Or an example dummy Ethernet driver could do
> it.
> >
> 
> Yes, perhaps I should be clearer about the "fast-path", because I was
> thinking of the optimized RX/TX paths for those nics at 10G and above.
> Probably the documentation should indeed have an update clarifying
> things a
> bit, since using the first cacheline only possible but not mandatory
> for
> simple RX.

I sometimes look at the source code of the simple drivers for reference, as they are easier to understand than the advanced vector drivers.
I suppose new PMD developers also would. :-)

Anyway, it is probably a good idea to add a clarifying note to the documentation, thus reflecting reality.

Just make sure that it says that the second cache line is supposed to be untouched by RX of high performance drivers, so application developers still consider it cold.


  reply	other threads:[~2020-11-03 14:03 UTC|newest]

Thread overview: 170+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-29  9:27 [dpdk-dev] [PATCH 00/15] remove mbuf timestamp Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 01/15] eventdev: remove software Rx timestamp Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 02/15] mbuf: add Rx timestamp dynamic flag Thomas Monjalon
2020-10-29  9:58   ` Andrew Rybchenko
2020-10-29 18:19     ` Ajit Khaparde
2020-10-29  9:27 ` [dpdk-dev] [PATCH 03/15] ethdev: register mbuf field and flags for timestamp Thomas Monjalon
2020-10-29 10:08   ` Andrew Rybchenko
2020-10-29 10:12     ` Thomas Monjalon
2020-10-29 10:33       ` Andrew Rybchenko
2020-10-29 10:46         ` Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 04/15] latency: switch timestamp to dynamic mbuf field Thomas Monjalon
2020-10-29 10:13   ` Andrew Rybchenko
2020-10-29 10:40     ` Thomas Monjalon
2020-10-29 14:20   ` Pattan, Reshma
2020-10-29 16:15     ` Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 05/15] net/ark: " Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 06/15] net/dpaa2: " Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 07/15] net/mlx5: fix dynamic mbuf offset lookup check Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 08/15] net/mlx5: switch timestamp to dynamic mbuf field Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 09/15] net/nfb: " Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 10/15] net/octeontx2: " Thomas Monjalon
2020-10-29 11:02   ` Andrew Rybchenko
2020-10-29 11:34     ` Thomas Monjalon
2020-10-29 11:37       ` Andrew Rybchenko
2020-10-29 11:52     ` Slava Ovsiienko
2020-10-30 12:41       ` Jerin Jacob
2020-11-01 16:12         ` Thomas Monjalon
2020-11-01 20:00         ` Andrew Rybchenko
2020-10-29  9:27 ` [dpdk-dev] [PATCH 11/15] net/pcap: " Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 12/15] app/testpmd: " Thomas Monjalon
2020-10-29 10:20   ` Andrew Rybchenko
2020-10-29 10:43     ` Thomas Monjalon
2020-10-29 10:52       ` Andrew Rybchenko
2020-10-29  9:27 ` [dpdk-dev] [PATCH 13/15] examples/rxtx_callbacks: switch timestamp to dynamic field Thomas Monjalon
2020-10-29 10:21   ` Andrew Rybchenko
2020-10-29 10:44     ` Thomas Monjalon
2020-10-29  9:27 ` [dpdk-dev] [PATCH 14/15] mbuf: remove deprecated timestamp field Thomas Monjalon
2020-10-29 10:23   ` Andrew Rybchenko
2020-10-29 18:18     ` Ajit Khaparde
2020-10-29 14:48   ` Kinsella, Ray
2020-10-29  9:27 ` [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half Thomas Monjalon
2020-10-29 10:50   ` Andrew Rybchenko
2020-10-29 10:56     ` Thomas Monjalon
2020-10-29 14:15       ` Ananyev, Konstantin
2020-10-29 18:45         ` Ajit Khaparde
2020-10-31 18:20           ` [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotterfirst half Morten Brørup
2020-10-31 20:40             ` Thomas Monjalon
2020-11-01  9:12               ` Morten Brørup
2020-11-01 16:21                 ` Thomas Monjalon
2020-11-01 16:38                 ` Thomas Monjalon
2020-11-01 20:59                   ` Morten Brørup
2020-11-02 15:58                     ` Thomas Monjalon
2020-11-03 12:10                       ` Morten Brørup
2020-11-03 12:25                         ` Bruce Richardson
2020-11-03 13:46                           ` Morten Brørup
2020-11-03 13:50                             ` Bruce Richardson
2020-11-03 14:03                               ` Morten Brørup [this message]
2020-11-03 14:02                         ` Slava Ovsiienko
2020-11-03 15:03                           ` Morten Brørup
2020-11-04 15:00                             ` Olivier Matz
2020-11-05  0:25                               ` Ananyev, Konstantin
2020-11-05  9:04                                 ` Morten Brørup
2020-11-05  9:35                                 ` Morten Brørup
2020-11-05 10:29                                   ` Bruce Richardson
2020-10-29 14:42   ` [dpdk-dev] [PATCH 15/15] mbuf: move pool pointer in hotter first half Kinsella, Ray
2020-11-01 18:06 ` [dpdk-dev] [PATCH v2 00/14] remove mbuf timestamp Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 01/14] eventdev: remove software Rx timestamp Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 02/14] mbuf: add Rx timestamp dynamic flag Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 03/14] ethdev: register mbuf field and flags for timestamp Thomas Monjalon
2020-11-02 15:39     ` Olivier Matz
2020-11-02 16:52       ` Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 04/14] latency: switch timestamp to dynamic mbuf field Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 05/14] net/ark: " Thomas Monjalon
2020-11-02 15:32     ` Olivier Matz
2020-11-02 16:10       ` Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 06/14] net/dpaa2: " Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 07/14] net/mlx5: fix dynamic mbuf offset lookup check Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 08/14] net/mlx5: switch timestamp to dynamic mbuf field Thomas Monjalon
2020-11-02  5:08     ` Ruifeng Wang
2020-11-02 23:20     ` David Christensen
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 09/14] net/nfb: " Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 10/14] net/octeontx2: " Thomas Monjalon
2020-11-01 18:28     ` Jerin Jacob
2020-11-02  9:38       ` Thomas Monjalon
2020-11-02 11:01         ` Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 11/14] net/pcap: " Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 12/14] app/testpmd: " Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 13/14] examples/rxtx_callbacks: switch timestamp to dynamic field Thomas Monjalon
2020-11-01 18:06   ` [dpdk-dev] [PATCH v2 14/14] mbuf: remove deprecated timestamp field Thomas Monjalon
2020-11-02 15:41     ` Olivier Matz
2020-11-02 15:47       ` David Marchand
2020-11-02 15:49         ` Thomas Monjalon
2020-11-01 18:08   ` [dpdk-dev] [PATCH v2 00/14] remove mbuf timestamp Thomas Monjalon
2020-11-03  0:13 ` [dpdk-dev] [PATCH v3 00/16] " Thomas Monjalon
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 01/16] eventdev: remove software Rx timestamp Thomas Monjalon
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 02/16] mbuf: add Rx timestamp flag and helpers Thomas Monjalon
2020-11-03  9:33     ` Olivier Matz
2020-11-03  9:59       ` Thomas Monjalon
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 03/16] latency: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 04/16] net/ark: " Thomas Monjalon
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 05/16] net/dpaa2: " Thomas Monjalon
2020-11-03  9:18     ` Hemant Agrawal
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 06/16] net/mlx5: fix dynamic mbuf offset lookup check Thomas Monjalon
2020-11-03  8:12     ` Slava Ovsiienko
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 07/16] net/mlx5: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03  8:12     ` Slava Ovsiienko
2020-11-03  0:13   ` [dpdk-dev] [PATCH v3 08/16] net/nfb: " Thomas Monjalon
2020-11-03 10:20     ` Olivier Matz
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 09/16] net/octeontx2: " Thomas Monjalon
2020-11-03 10:52     ` Harman Kalra
2020-11-03 11:22       ` Thomas Monjalon
2020-11-03 12:21         ` Thomas Monjalon
2020-11-03 14:23           ` [dpdk-dev] [EXT] " Harman Kalra
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 10/16] net/pcap: " Thomas Monjalon
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 11/16] app/testpmd: " Thomas Monjalon
2020-11-03 10:23     ` Olivier Matz
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 12/16] examples/rxtx_callbacks: switch timestamp to dynamic field Thomas Monjalon
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 13/16] ethdev: add doxygen comment for Rx timestamp API Thomas Monjalon
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 14/16] mbuf: remove deprecated timestamp field Thomas Monjalon
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 15/16] mbuf: add Tx timestamp registration helper Thomas Monjalon
2020-11-03  0:14   ` [dpdk-dev] [PATCH v3 16/16] ethdev: include mbuf registration in Tx timestamp API Thomas Monjalon
2020-11-03  7:54     ` Slava Ovsiienko
2020-11-03  9:00   ` [dpdk-dev] [PATCH v3 00/16] remove mbuf timestamp David Marchand
2020-11-03 12:21 ` [dpdk-dev] [PATCH v4 " Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 01/16] eventdev: remove software Rx timestamp Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 02/16] mbuf: add Rx timestamp flag and helpers Thomas Monjalon
2020-11-03 12:34     ` Andrew Rybchenko
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 03/16] latency: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 04/16] net/ark: " Thomas Monjalon
2020-11-03 12:37     ` Andrew Rybchenko
2020-11-03 13:08       ` Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 05/16] net/dpaa2: " Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 06/16] net/mlx5: fix dynamic mbuf offset lookup check Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 07/16] net/mlx5: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 08/16] net/nfb: " Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 09/16] net/octeontx2: " Thomas Monjalon
2020-11-03 12:21   ` [dpdk-dev] [PATCH v4 10/16] net/pcap: " Thomas Monjalon
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 11/16] app/testpmd: " Thomas Monjalon
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 12/16] examples/rxtx_callbacks: switch timestamp to dynamic field Thomas Monjalon
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 13/16] ethdev: add doxygen comment for Rx timestamp API Thomas Monjalon
2020-11-03 12:40     ` Andrew Rybchenko
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 14/16] mbuf: remove deprecated timestamp field Thomas Monjalon
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 15/16] mbuf: add Tx timestamp registration helper Thomas Monjalon
2020-11-03 12:42     ` Andrew Rybchenko
2020-11-03 12:22   ` [dpdk-dev] [PATCH v4 16/16] ethdev: include mbuf registration in Tx timestamp API Thomas Monjalon
2020-11-03 12:45     ` Andrew Rybchenko
2020-11-03 14:09 ` [dpdk-dev] [PATCH v5 00/16] remove mbuf timestamp Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 01/16] eventdev: remove software Rx timestamp Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 02/16] mbuf: add Rx timestamp flag and helpers Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 03/16] latency: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 04/16] net/ark: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 05/16] net/dpaa2: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 06/16] net/mlx5: fix dynamic mbuf offset lookup check Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 07/16] net/mlx5: switch Rx timestamp to dynamic mbuf field Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 08/16] net/nfb: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 09/16] net/octeontx2: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 10/16] net/pcap: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 11/16] app/testpmd: " Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 12/16] examples/rxtx_callbacks: switch timestamp to dynamic field Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 13/16] ethdev: add doxygen comment for Rx timestamp API Thomas Monjalon
2020-11-03 19:07     ` Ajit Khaparde
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 14/16] mbuf: remove deprecated timestamp field Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 15/16] mbuf: add Tx timestamp registration helper Thomas Monjalon
2020-11-03 14:09   ` [dpdk-dev] [PATCH v5 16/16] ethdev: include mbuf registration in Tx timestamp API Thomas Monjalon
2020-11-03 14:17   ` [dpdk-dev] [PATCH v5 00/16] remove mbuf timestamp Olivier Matz
2020-11-03 14:44     ` Thomas Monjalon
2020-11-03 16:08   ` Stephen Hemminger
2020-11-03 16:20     ` Thomas Monjalon
2020-11-03 17:42       ` Stephen Hemminger
2020-11-03 17:55         ` Thomas Monjalon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=98CBD80474FA8B44BF855DF32C47DC35C613DD@smartserver.smartshare.dk \
    --to=mb@smartsharesystems.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=andrew.rybchenko@oktetlabs.ru \
    --cc=bruce.richardson@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=jerinj@marvell.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=matan@nvidia.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=olivier.matz@6wind.com \
    --cc=shahafs@nvidia.com \
    --cc=stephen@networkplumber.org \
    --cc=techboard@dpdk.org \
    --cc=thomas@monjalon.net \
    --cc=viacheslavo@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git