DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: "Yigit, Ferruh" <ferruh.yigit@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	"Richardson, Bruce" <bruce.richardson@intel.com>
Cc: "Morten Brørup" <mb@smartsharesystems.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	"olivier.matz@6wind.com" <olivier.matz@6wind.com>,
	"andrew.rybchenko@oktetlabs.ru" <andrew.rybchenko@oktetlabs.ru>,
	"honnappa.nagarahalli@arm.com" <honnappa.nagarahalli@arm.com>,
	"jerinj@marvell.com" <jerinj@marvell.com>,
	"gakhil@marvell.com" <gakhil@marvell.com>
Subject: Re: [dpdk-dev] [PATCH] parray: introduce internal API for dynamic arrays
Date: Mon, 21 Jun 2021 15:37:27 +0000	[thread overview]
Message-ID: <DM6PR11MB4491D4F7D4668A319E3BD5F19A0A9@DM6PR11MB4491.namprd11.prod.outlook.com> (raw)
In-Reply-To: <4e63beea-3121-2aaf-f6d6-dc8807c4088d@intel.com>


> 
> On 6/21/2021 3:42 PM, Ananyev, Konstantin wrote:
> >
> >>>>>>>> One more thought here - if we are talking about rte_ethdev[] in particular, I think  we can:
> >>>>>>>> 1. move public function pointers (rx_pkt_burst(), etc.) from rte_ethdev into a separate flat array.
> >>>>>>>> We can keep it public to still use inline functions for 'fast' calls rte_eth_rx_burst(), etc. to avoid
> >>>>>>>> any regressions.
> >>>>>>>> That could still be flat array with max_size specified at application startup.
> >>>>>>>> 2. Hide rest of rte_ethdev struct in .c.
> >>>>>>>> That will allow us to change the struct itself and the whole rte_ethdev[] table in a way we like
> >>>>>>>> (flat array, vector, hash, linked list) without ABI/API breakages.
> >>>>>>>>
> >>>>>>>> Yes, it would require all PMDs to change prototype for pkt_rx_burst() function
> >>>>>>>> (to accept port_id, queue_id instead of queue pointer), but the change is mechanical one.
> >>>>>>>> Probably some macro can be provided to simplify it.
> >>>>>>>>
> >>>>>>>
> >>>>>>> We are already planning some tasks for ABI stability for v21.11, I think
> >>>>>>> splitting 'struct rte_eth_dev' can be part of that task, it enables hiding more
> >>>>>>> internal data.
> >>>>>>
> >>>>>> Ok, sounds good.
> >>>>>>
> >>>>>>>
> >>>>>>>> The only significant complication I can foresee with implementing that approach -
> >>>>>>>> we'll need a an array of 'fast' function pointers per queue, not per device as we have now
> >>>>>>>> (to avoid extra indirection for callback implementation).
> >>>>>>>> Though as a bonus we'll have ability to use different RX/TX funcions per queue.
> >>>>>>>>
> >>>>>>>
> >>>>>>> What do you think split Rx/Tx callback into its own struct too?
> >>>>>>>
> >>>>>>> Overall 'rte_eth_dev' can be split into three as:
> >>>>>>> 1. rte_eth_dev
> >>>>>>> 2. rte_eth_dev_burst
> >>>>>>> 3. rte_eth_dev_cb
> >>>>>>>
> >>>>>>> And we can hide 1 from applications even with the inline functions.
> >>>>>>
> >>>>>> As discussed off-line, I think:
> >>>>>> it is possible.
> >>>>>> My absolute preference would be to have just 1/2 (with CB hidden).
> >>>>>
> >>>>> How can we hide the callbacks since they are used by inline burst functions.
> >>>>
> >>>> I probably I owe a better explanation to what I meant in first mail.
> >>>> Otherwise it sounds confusing.
> >>>> I'll try to write a more detailed one in next few days.
> >>>
> >>> Actually I gave it another thought over weekend, and might be we can
> >>> hide rte_eth_dev_cb even in a simpler way. I'd use eth_rx_burst() as
> >>> an example, but the same principle applies to other 'fast' functions.
> >>>
> >>>  1. Needed changes for PMDs rx_pkt_burst():
> >>>     a) change function prototype to accept 'uint16_t port_id' and 'uint16_t queue_id',
> >>>          instead of current 'void *'.
> >>>     b) Each PMD rx_pkt_burst() will have to call rte_eth_rx_epilog() function at return.
> >>>          This  inline function will do all CB calls for that queue.
> >>>
> >>> To be more specific, let say we have some PMD: xyz with RX function:
> >>>
> >>> uint16_t
> >>> xyz_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
> >>> {
> >>>      struct xyz_rx_queue *rxq = rx_queue;
> >>>      uint16_t nb_rx = 0;
> >>>
> >>>      /* do actual stuff here */
> >>>     ....
> >>>     return nb_rx;
> >>> }
> >>>
> >>> It will be transformed to:
> >>>
> >>> uint16_t
> >>> xyz_recv_pkts(uint16_t port_id, uint16_t queue_id, struct rte_mbuf **rx_pkts, uint16_t nb_pkts)
> >>> {
> >>>          struct xyz_rx_queue *rxq;
> >>>          uint16_t nb_rx;
> >>>
> >>>          rxq = _rte_eth_rx_prolog(port_id, queue_id);
> >>>          if (rxq == NULL)
> >>>              return 0;
> >>>          nb_rx = _xyz_real_recv_pkts(rxq, rx_pkts, nb_pkts);
> >>>          return _rte_eth_rx_epilog(port_id, queue_id, rx_pkts, nb_pkts);
> >>> }
> >>>
> >>> And somewhere in ethdev_private.h:
> >>>
> >>> static inline void *
> >>> _rte_eth_rx_prolog(uint16_t port_id, uint16_t queue_id);
> >>> {
> >>>    struct rte_eth_dev *dev = &rte_eth_devices[port_id];
> >>>
> >>> #ifdef RTE_ETHDEV_DEBUG_RX
> >>>         RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, NULL);
> >>>         RTE_FUNC_PTR_OR_ERR_RET(*dev->rx_pkt_burst, NULL);
> >>>
> >>>         if (queue_id >= dev->data->nb_rx_queues) {
> >>>                 RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id);
> >>>                 return NULL;
> >>>         }
> >>> #endif
> >>>   return dev->data->rx_queues[queue_id];
> >>> }
> >>>
> >>> static inline uint16_t
> >>> _rte_eth_rx_epilog(uint16_t port_id, uint16_t queue_id, struct rte_mbuf **rx_pkts, const uint16_t nb_pkts);
> >>> {
> >>>     struct rte_eth_dev *dev = &rte_eth_devices[port_id];
> >>>
> >>> #ifdef RTE_ETHDEV_RXTX_CALLBACKS
> >>>         struct rte_eth_rxtx_callback *cb;
> >>>
> >>>         /* __ATOMIC_RELEASE memory order was used when the
> >>>          * call back was inserted into the list.
> >>>          * Since there is a clear dependency between loading
> >>>          * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is
> >>>          * not required.
> >>>          */
> >>>         cb = __atomic_load_n(&dev->post_rx_burst_cbs[queue_id],
> >>>                                 __ATOMIC_RELAXED);
> >>>
> >>>         if (unlikely(cb != NULL)) {
> >>>                 do {
> >>>                         nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx,
> >>>                                                 nb_pkts, cb->param);
> >>>                         cb = cb->next;
> >>>                 } while (cb != NULL);
> >>>         }
> >>> #endif
> >>>
> >>>         rte_ethdev_trace_rx_burst(port_id, queue_id, (void **)rx_pkts, nb_rx);
> >>>         return nb_rx;
> >>>  }
> >>>
> >>> Now, as you said above, in rte_ethdev.h we will keep only a flat array
> >>> with pointers to 'fast' functions:
> >>> struct {
> >>>      eth_rx_burst_t             rx_pkt_burst
> >>>       eth_tx_burst_t             tx_pkt_burst;
> >>>       eth_tx_prep_t              tx_pkt_prepare;
> >>>      .....
> >>> } rte_eth_dev_burst[];
> >>>
> >>> And rte_eth_rx_burst() will look like:
> >>>
> >>> static inline uint16_t
> >>> rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id,
> >>>                  struct rte_mbuf **rx_pkts, const uint16_t nb_pkts)
> >>> {
> >>>     if (port_id >= RTE_MAX_ETHPORTS)
> >>>         return 0;
> >>>    return rte_eth_dev_burst[port_id](port_id, queue_id, rx_pkts, nb_pkts);
> >>> }
> >>>
> >>> Yes, it will require changes in *all* PMDs, but as I said before the changes will be a mechanic ones.
> >>>
> >>
> >> I did not like the idea to push to calling Rx/TX callbacks responsibility to the
> >> drivers, I think it should be in the ethdev layer.
> >
> > Well, I'd say it is an ethdev layer function that has to be called by PMD 😊
> >
> >>
> >> What about making 'rte_eth_rx_epilog' an API and call from 'rte_eth_rx_burst()',
> >> which will add another function call for Rx/Tx callback but shouldn't affect the
> >> Rx/Tx burst.
> >
> > But then we either need to expose call-back information to the user or pay the penalty
> > for extra function call, correct?
> >
> 
> Right. As a middle ground, we can keep Rx/Tx burst functions as inline, but have
> the Rx/Tx callback part of it as function, so get the hit only for callbacks.

To avoid the  hit we need to expose CB data to the user.
At least number of call-backs currently installed for each queue. 


  reply	other threads:[~2021-06-21 15:37 UTC|newest]

Thread overview: 61+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-14 10:58 Thomas Monjalon
2021-06-14 12:22 ` Morten Brørup
2021-06-14 13:15   ` Bruce Richardson
2021-06-14 13:32     ` Thomas Monjalon
2021-06-14 14:59       ` Ananyev, Konstantin
2021-06-14 15:48         ` Jerin Jacob
2021-06-15  6:52           ` Thomas Monjalon
2021-06-15  8:00             ` Jerin Jacob
2021-06-15  9:18               ` Thomas Monjalon
2021-06-15  9:33             ` Ananyev, Konstantin
2021-06-15  9:50               ` Thomas Monjalon
2021-06-15 10:08                 ` Ananyev, Konstantin
2021-06-15 14:02                   ` Thomas Monjalon
2021-06-15 14:37                     ` Honnappa Nagarahalli
2021-06-14 15:54         ` Ananyev, Konstantin
2021-06-17 13:08           ` Ferruh Yigit
2021-06-17 14:58             ` Ananyev, Konstantin
2021-06-17 15:17               ` Morten Brørup
2021-06-17 16:12                 ` Ferruh Yigit
2021-06-17 16:55                   ` Morten Brørup
2021-06-18 10:21                     ` Ferruh Yigit
2021-06-17 17:05                   ` Ananyev, Konstantin
2021-06-18  9:14                     ` Morten Brørup
2021-06-18 10:47                       ` Ferruh Yigit
2021-06-18 11:16                         ` Morten Brørup
2021-06-18 10:28                     ` Ferruh Yigit
2021-06-17 15:44               ` Ferruh Yigit
2021-06-18 10:41                 ` Ananyev, Konstantin
2021-06-18 10:49                   ` Ferruh Yigit
2021-06-21 11:06                   ` Ananyev, Konstantin
2021-06-21 12:10                     ` Morten Brørup
2021-06-21 12:30                       ` Ananyev, Konstantin
2021-06-21 13:28                         ` Morten Brørup
     [not found]                           ` <DM6PR11MB4491D4F6FAFDD6E8EEC2A78F9A099@DM6PR11MB4491.namprd11.prod.outlook .com>
2021-06-22  8:33                           ` Ananyev, Konstantin
2021-06-22 10:01                             ` Morten Brørup
2021-06-22 12:13                               ` Ananyev, Konstantin
2021-06-22 13:18                                 ` Morten Brørup
2021-06-21 14:10                         ` Ferruh Yigit
2021-06-21 14:38                           ` Ananyev, Konstantin
2021-06-21 15:56                             ` Ferruh Yigit
2021-06-21 18:17                               ` Ananyev, Konstantin
2021-06-21 14:05                     ` Ferruh Yigit
2021-06-21 14:42                       ` Ananyev, Konstantin
2021-06-21 15:32                         ` Ferruh Yigit
2021-06-21 15:37                           ` Ananyev, Konstantin [this message]
2021-06-14 15:48       ` Morten Brørup
2021-06-15  6:48         ` Thomas Monjalon
2021-06-15  7:53           ` Morten Brørup
2021-06-15  8:44             ` Bruce Richardson
2021-06-15  9:28               ` Thomas Monjalon
2021-06-16  9:42           ` Jerin Jacob
2021-06-16 11:27             ` Morten Brørup
2021-06-16 12:00               ` Jerin Jacob
2021-06-16 13:02               ` Bruce Richardson
2021-06-16 15:01                 ` Morten Brørup
2021-06-16 17:40                   ` Bruce Richardson
2021-06-16 12:22             ` Burakov, Anatoly
2021-06-16 12:59               ` Jerin Jacob
2021-06-16 22:58                 ` Dmitry Kozlyuk
2021-06-14 13:28   ` Thomas Monjalon
2021-06-16 11:11 ` Burakov, Anatoly

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM6PR11MB4491D4F7D4668A319E3BD5F19A0A9@DM6PR11MB4491.namprd11.prod.outlook.com \
    --to=konstantin.ananyev@intel.com \
    --cc=andrew.rybchenko@oktetlabs.ru \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=gakhil@marvell.com \
    --cc=honnappa.nagarahalli@arm.com \
    --cc=jerinj@marvell.com \
    --cc=mb@smartsharesystems.com \
    --cc=olivier.matz@6wind.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).