From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: Jerin Jacob <jerinjacobk@gmail.com>
Cc: dpdk-dev <dev@dpdk.org>, Thomas Monjalon <thomas@monjalon.net>,
"Yigit, Ferruh" <ferruh.yigit@intel.com>,
Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
"Yang, Qiming" <qiming.yang@intel.com>,
"Zhang, Qi Z" <qi.z.zhang@intel.com>,
"Xing, Beilei" <beilei.xing@intel.com>,
"techboard@dpdk.org" <techboard@dpdk.org>
Subject: Re: [dpdk-dev] [RFC 0/7] hide eth dev related structures
Date: Tue, 14 Sep 2021 13:33:50 +0000 [thread overview]
Message-ID: <DM6PR11MB4491DAA82C445BA0278C0CD09ADA9@DM6PR11MB4491.namprd11.prod.outlook.com> (raw)
In-Reply-To: <CALBAE1P+gCEJgxBqCXW7fjFEcBcHVFWTMAmsG_mvozA78FJYTQ@mail.gmail.com>
Hi Jerin,
> > NOTE: This is just an RFC to start further discussion and collect the feedback.
> > Due to significant amount of work, changes required are applied only to two
> > PMDs so far: net/i40e and net/ice.
> > So to build it you'll need to add:
> > -Denable_drivers='common/*,mempool/*,net/ice,net/i40e'
> > to your config options.
>
> >
> > That approach was selected to avoid(/minimize) possible performance losses.
> >
> > So far I done only limited amount functional and performance testing.
> > Didn't spot any functional problems, and performance numbers
> > remains the same before and after the patch on my box (testpmd, macswap fwd).
>
>
> Based on testing on octeonxt2. We see some regression in testpmd and
> bit on l3fwd too.
>
> Without patch: 73.5mpps/core in testpmd iofwd
> With out patch: 72 5mpps/core in testpmd iofwd
>
> Based on my understanding it is due to additional indirection.
From your patch below, it looks like not actually additional indirection,
but extra memory dereference - func and dev pointers are now stored
at different places. Plus the fact that now we dereference rte_eth_devices[]
data inside PMD function. Which probably prevents compiler and CPU to load
rte_eth_devices[port_id].data and rte_eth_devices[port_id]. pre_tx_burst_cbs[queue_id]
in advance before calling actual RX/TX function.
About your approach: I don’t mind to add extra opaque 'void *data' pointer,
but would prefer not to expose callback invocations code into inline function.
Main reason for that - I think it still need to be reworked to allow adding/removing
callbacks without stopping the device. Something similar to what was done for cryptodev
callbacks. To be able to do that in future without another ABI breakage callbacks related part
needs to be kept internal.
Though what we probably can do: add two dynamic arrays of opaque pointers to rte_eth_burst_api.
One for rx/tx queue data pointers, second for rx/tx callback pointers.
To be more specific, something like:
typedef uint16_t (*rte_eth_rx_burst_t)( void *rxq, struct rte_mbuf **rx_pkts, uint16_t nb_pkts, void *cbs);
typedef uint16_t (*rte_eth_tx_burst_t)(void *txq, struct rte_mbuf **tx_pkts, uint16_t nb_pkts, void *cbs);
....
struct rte_eth_burst_api {
rte_eth_rx_burst_t rx_pkt_burst;
/**< PMD receive function. */
rte_eth_tx_burst_t tx_pkt_burst;
/**< PMD transmit function. */
rte_eth_tx_prep_t tx_pkt_prepare;
/**< PMD transmit prepare function. */
rte_eth_rx_queue_count_t rx_queue_count;
/**< Get the number of used RX descriptors. */
rte_eth_rx_descriptor_status_t rx_descriptor_status;
/**< Check the status of a Rx descriptor. */
rte_eth_tx_descriptor_status_t tx_descriptor_status;
/**< Check the status of a Tx descriptor. */
struct {
void **queue_data; /* point to rte_eth_devices[port_id].data-> rx_queues */
void **cbs; /* points to rte_eth_devices[port_id].post_rx_burst_cbs */
} rx_data, tx_data;
} __rte_cache_aligned;
static inline uint16_t
rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id,
struct rte_mbuf **rx_pkts, const uint16_t nb_pkts)
{
struct rte_eth_burst_api *p;
if (port_id >= RTE_MAX_ETHPORTS || queue_id >= RTE_MAX_QUEUES_PER_PORT)
return 0;
p = &rte_eth_burst_api[port_id];
return p->rx_pkt_burst(p->rx_data.queue_data[queue_id], rx_pkts, nb_pkts, p->rx_data.cbs[queue_id]);
}
Same for TX.
If that looks ok to everyone, I'll try to prepare next version based on that.
In theory that should avoid extra dereference problem and even reduce indirection.
As a drawback data->rxq/txq should always be allocated for RTE_MAX_QUEUES_PER_PORT entries,
but I presume that’s not a big deal.
As a side question - is there any reason why rte_ethdev_trace_rx_burst() is invoked at very last point,
while rte_ethdev_trace_tx_burst() after CBs but before actual tx_pkt_burst()?
It would make things simpler if tracng would always be done either on entrance or exit of rx/tx_burst.
>
> My suggestion to fix the problem by:
> Removing the additional `data` redirection and pull callback function
> pointers back
> and keep rest as opaque as done in the existing patch like [1]
>
> I don't believe this has any real implication on future ABI stability
> as we will not be adding
> any new item in rte_eth_fp in any way as new features can be added in slowpath
> rte_eth_dev as mentioned in the patch.
>
> [2] is the patch of doing the same as I don't see any performance
> regression after [2].
>
>
> [1]
> - struct rte_eth_burst_api {
> - struct rte_eth_fp {
> + void *data;
> rte_eth_rx_burst_t rx_pkt_burst;
> /**< PMD receive function. */
> rte_eth_tx_burst_t tx_pkt_burst;
> @@ -85,8 +100,19 @@ struct rte_eth_burst_api {
> /**< Check the status of a Rx descriptor. */
> rte_eth_tx_descriptor_status_t tx_descriptor_status;
> /**< Check the status of a Tx descriptor. */
> + /**
> + * User-supplied functions called from rx_burst to post-process
> + * received packets before passing them to the user
> + */
> + struct rte_eth_rxtx_callback
> + *post_rx_burst_cbs[RTE_MAX_QUEUES_PER_PORT];
> + /**
> + * User-supplied functions called from tx_burst to pre-process
> + * received packets before passing them to the driver for transmission.
> + */
> + struct rte_eth_rxtx_callback *pre_tx_burst_cbs[RTE_MAX_QUEUES_PER_PORT];
> uintptr_t reserved[2];
> -} __rte_cache_min_aligned;
> +} __rte_cache_aligned;
>
> [2]
> https://pastebin.com/CuqkrCW4
next prev parent reply other threads:[~2021-09-14 13:33 UTC|newest]
Thread overview: 112+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-20 16:28 Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 1/7] eth: move ethdev 'burst' API into separate structure Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 2/7] eth: make drivers to use new API for Rx Konstantin Ananyev
2021-09-06 18:41 ` Ferruh Yigit
2021-09-14 14:28 ` Ananyev, Konstantin
2021-08-20 16:28 ` [dpdk-dev] [RFC 3/7] eth: make drivers to use new API for Tx Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 4/7] eth: make drivers to use new API for Tx prepare Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 5/7] eth: make drivers to use new API to obtain descriptor status Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 6/7] eth: make drivers to use new API for Rx queue count Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 7/7] eth: hide eth dev related structures Konstantin Ananyev
2021-08-26 12:37 ` [dpdk-dev] [RFC 0/7] " Jerin Jacob
2021-09-06 18:09 ` Ferruh Yigit
2021-09-14 13:33 ` Ananyev, Konstantin [this message]
2021-09-15 9:45 ` Jerin Jacob
2021-09-22 15:08 ` Ananyev, Konstantin
2021-09-27 16:14 ` Jerin Jacob
2021-09-28 9:37 ` Ananyev, Konstantin
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 0/5] " Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 1/5] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 2/5] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-09-23 5:51 ` Wang, Haiyue
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 3/5] ethdev: copy ethdev 'burst' API into separate structure Konstantin Ananyev
2021-09-23 5:58 ` Wang, Haiyue
2021-09-27 18:01 ` Jerin Jacob
2021-09-28 9:42 ` Ananyev, Konstantin
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 4/5] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 5/5] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 0/7] " Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-01 16:48 ` Ferruh Yigit
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-01 16:46 ` Ferruh Yigit
2021-10-01 17:40 ` Ananyev, Konstantin
2021-10-04 8:46 ` Ferruh Yigit
2021-10-04 9:20 ` Ananyev, Konstantin
2021-10-04 10:13 ` Ferruh Yigit
2021-10-04 11:17 ` Ananyev, Konstantin
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 16:53 ` Ferruh Yigit
2021-10-01 17:04 ` Ferruh Yigit
2021-10-01 17:02 ` [dpdk-dev] [PATCH v3 0/7] " Ferruh Yigit
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 " Konstantin Ananyev
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-05 12:09 ` Thomas Monjalon
2021-10-05 16:45 ` Ananyev, Konstantin
2021-10-05 16:49 ` Thomas Monjalon
2021-10-05 12:21 ` Thomas Monjalon
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-05 13:09 ` Thomas Monjalon
2021-10-05 16:41 ` Ananyev, Konstantin
2021-10-05 16:48 ` Thomas Monjalon
2021-10-05 17:04 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-05 9:54 ` David Marchand
2021-10-05 10:13 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-05 13:13 ` Thomas Monjalon
2021-10-05 16:35 ` Ananyev, Konstantin
2021-10-05 16:45 ` Thomas Monjalon
2021-10-05 17:12 ` Ananyev, Konstantin
2021-10-05 17:41 ` Thomas Monjalon
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-05 13:14 ` Thomas Monjalon
2021-10-05 16:21 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-05 10:04 ` David Marchand
2021-10-05 10:43 ` Ferruh Yigit
2021-10-05 11:37 ` David Marchand
2021-10-05 15:57 ` Ananyev, Konstantin
2021-10-05 13:24 ` Thomas Monjalon
2021-10-05 16:19 ` Ananyev, Konstantin
2021-10-05 16:25 ` Thomas Monjalon
2021-10-06 16:42 ` [dpdk-dev] [PATCH v4 0/7] " Ali Alnubani
2021-10-06 17:26 ` Ali Alnubani
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 " Konstantin Ananyev
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 1/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-11 9:20 ` Andrew Rybchenko
2021-10-11 16:25 ` Ananyev, Konstantin
2021-10-11 17:15 ` Andrew Rybchenko
2021-10-11 23:06 ` Ananyev, Konstantin
2021-10-12 5:47 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 3/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-11 8:06 ` Andrew Rybchenko
2021-10-12 17:59 ` Hyong Youb Kim (hyonkim)
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 4/7] ethdev: copy fast-path API into separate structure Konstantin Ananyev
2021-10-09 12:05 ` fengchengwen
2021-10-11 1:18 ` fengchengwen
2021-10-11 8:39 ` Andrew Rybchenko
2021-10-11 15:24 ` Ananyev, Konstantin
2021-10-11 8:35 ` Andrew Rybchenko
2021-10-11 15:15 ` Ananyev, Konstantin
2021-10-11 8:25 ` Andrew Rybchenko
2021-10-11 16:52 ` Ananyev, Konstantin
2021-10-11 17:22 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 5/7] ethdev: make fast-path functions to use new flat array Konstantin Ananyev
2021-10-11 9:02 ` Andrew Rybchenko
2021-10-11 15:47 ` Ananyev, Konstantin
2021-10-11 17:03 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 6/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-11 9:09 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-11 9:20 ` Andrew Rybchenko
2021-10-11 15:54 ` Ananyev, Konstantin
2021-10-11 17:04 ` Andrew Rybchenko
2021-10-08 18:13 ` [dpdk-dev] [PATCH v5 0/7] " Slava Ovsiienko
2021-10-11 9:22 ` Andrew Rybchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM6PR11MB4491DAA82C445BA0278C0CD09ADA9@DM6PR11MB4491.namprd11.prod.outlook.com \
--to=konstantin.ananyev@intel.com \
--cc=andrew.rybchenko@oktetlabs.ru \
--cc=beilei.xing@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=jerinjacobk@gmail.com \
--cc=qi.z.zhang@intel.com \
--cc=qiming.yang@intel.com \
--cc=techboard@dpdk.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).