Re: [dpdk-dev] [PATCH v5 0/7] hide eth dev related structures

[Slava Ovsiienko <viacheslavo@nvidia.com>]

Hi,

I've reviewed the series, and it looks good to me.
I see we did not introduce new indirect referencing on the datapath
(just replaced rte_eth_devices[] being hidden with the new rte_eth_fp_ops[].)
My only concern - we'll get two places where pointers to the PMDs routines are stored,
and it means potential unsynchro between them, but I do not see the actual scenario for that.
For example, mlx5 PMD proposes multiple tx/rx_burst routines, and actual routine
selection happens on dev_start(), but then it is not supposed to be changed till dev_stop().
The internal PMD checks like this:
"if (dev->rx_pkt_burst == mlx5_rx_burst)"
supposed to continue working OK as well.

Hence, for series:
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

With best regards,
Slava

> -----Original Message-----
> From: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Sent: Thursday, October 7, 2021 14:28
> To: dev@dpdk.org
> Cc: xiaoyun.li@intel.com; anoobj@marvell.com; jerinj@marvell.com;
> ndabilpuram@marvell.com; adwivedi@marvell.com;
> shepard.siegel@atomicrules.com; ed.czeck@atomicrules.com;
> john.miller@atomicrules.com; irusskikh@marvell.com;
> ajit.khaparde@broadcom.com; somnath.kotur@broadcom.com;
> rahul.lakkireddy@chelsio.com; hemant.agrawal@nxp.com;
> sachin.saxena@oss.nxp.com; haiyue.wang@intel.com; johndale@cisco.com;
> hyonkim@cisco.com; qi.z.zhang@intel.com; xiao.w.wang@intel.com;
> humin29@huawei.com; yisen.zhuang@huawei.com; oulijun@huawei.com;
> beilei.xing@intel.com; jingjing.wu@intel.com; qiming.yang@intel.com; Matan
> Azrad <matan@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>;
> sthemmin@microsoft.com; NBU-Contact-longli <longli@microsoft.com>;
> heinrich.kuhn@corigine.com; kirankumark@marvell.com;
> andrew.rybchenko@oktetlabs.ru; mczekaj@marvell.com;
> jiawenwu@trustnetic.com; jianwang@trustnetic.com;
> maxime.coquelin@redhat.com; chenbo.xia@intel.com; NBU-Contact-Thomas
> Monjalon <thomas@monjalon.net>; ferruh.yigit@intel.com; mdr@ashroe.eu;
> jay.jayatheerthan@intel.com; Konstantin Ananyev
> <konstantin.ananyev@intel.com>
> Subject: [PATCH v5 0/7] hide eth dev related structures
> 
> v5 changes:
> - Fix spelling (Thomas/David)
> - Rename internal helper functions (David)
> - Reorder patches and update commit messages (Thomas)
> - Update comments (Thomas)
> - Changed layout in rte_eth_fp_ops, to group functions and
>    related data based on their functionality:
>    first 64B line for Rx, second one for Tx.
>    Didn't observe any real performance difference comparing to
>    original layout. Though decided to keep a new one, as it seems
>    a bit more plausible.
> 
> v4 changes:
>  - Fix secondary process attach (Pavan)
>  - Fix build failure (Ferruh)
>  - Update lib/ethdev/verion.map (Ferruh)
>    Note that moving newly added symbols from EXPERIMENTAL to DPDK_22
>    section makes checkpatch.sh to complain.
> 
> v3 changes:
>  - Changes in public struct naming (Jerin/Haiyue)
>  - Split patches
>  - Update docs
>  - Shamelessly included Andrew's patch:
>    https://patches.dpdk.org/project/dpdk/patch/20210928154856.1015020-
> 1-andrew.rybchenko@oktetlabs.ru/
>    into these series.
>    I have to do similar thing here, so decided to avoid duplicated effort.
> 
> The aim of these patch series is to make rte_ethdev core data structures
> (rte_eth_dev, rte_eth_dev_data, rte_eth_rxtx_callback, etc.) internal to DPDK
> and not visible to the user.
> That should allow future possible changes to core ethdev related structures to
> be transparent to the user and help to improve ABI/API stability.
> Note that current ethdev API is preserved, but it is a formal ABI break.
> 
> The work is based on previous discussions at:
> https://www.mail-archive.com/dev@dpdk.org/msg211405.html
> https://www.mail-archive.com/dev@dpdk.org/msg216685.html
> and consists of the following main points:
> 1. Copy public 'fast' function pointers (rx_pkt_burst(), etc.) and
>    related data pointer from rte_eth_dev into a separate flat array.
>    We keep it public to still be able to use inline functions for these
>    'fast' calls (like rte_eth_rx_burst(), etc.) to avoid/minimize slowdown.
>    Note that apart from function pointers itself, each element of this
>    flat array also contains two opaque pointers for each ethdev:
>    1) a pointer to an array of internal queue data pointers
>    2)  points to array of queue callback data pointers.
>    Note that exposing this extra information allows us to avoid extra
>    changes inside PMD level, plus should help to avoid possible
>    performance degradation.
> 2. Change implementation of 'fast' inline ethdev functions
>    (rte_eth_rx_burst(), etc.) to use new public flat array.
>    While it is an ABI breakage, this change is intended to be transparent
>    for both users (no changes in user app is required) and PMD developers
>    (no changes in PMD is required).
>    One extra note - with new implementation RX/TX callback invocation
>    will cost one extra function call with this changes. That might cause
>    some slowdown for code-path with RX/TX callbacks heavily involved.
>    Hope such trade-off is acceptable for the community.
> 3. Move rte_eth_dev, rte_eth_dev_data, rte_eth_rxtx_callback and related
>    things into internal header: <ethdev_driver.h>.
> 
> That approach was selected to:
>   - Avoid(/minimize) possible performance losses.
>   - Minimize required changes inside PMDs.
> 
> Performance testing results (ICX 2.0GHz, E810 (ice)):
>  - testpmd macswap fwd mode, plus
>    a) no RX/TX callbacks:
>       no actual slowdown observed
>    b) bpf-load rx 0 0 JM ./dpdk.org/examples/bpf/t3.o:
>       ~2% slowdown
>  - l3fwd: no actual slowdown observed
> 
> Would like to thank everyone who already reviewed and tested previous
> versions of these series. All other interested parties please don't be shy and
> provide your feedback.
> 
> Andrew Rybchenko (1):
>   ethdev: remove legacy Rx descriptor done API
> 
> Konstantin Ananyev (6):
>   ethdev: allocate max space for internal queue array
>   ethdev: change input parameters for rx_queue_count
>   ethdev: copy fast-path API into separate structure
>   ethdev: make fast-path functions to use new flat array
>   ethdev: add API to retrieve multiple ethernet addresses
>   ethdev: hide eth dev related structures
> 
>  app/test-pmd/config.c                         |  23 +-
>  doc/guides/nics/features.rst                  |   6 +-
>  doc/guides/rel_notes/deprecation.rst          |   5 -
>  doc/guides/rel_notes/release_21_11.rst        |  21 ++
>  drivers/common/octeontx2/otx2_sec_idev.c      |   2 +-
>  drivers/crypto/octeontx2/otx2_cryptodev_ops.c |   2 +-
>  drivers/net/ark/ark_ethdev_rx.c               |   4 +-
>  drivers/net/ark/ark_ethdev_rx.h               |   3 +-
>  drivers/net/atlantic/atl_ethdev.h             |   2 +-
>  drivers/net/atlantic/atl_rxtx.c               |   9 +-
>  drivers/net/bnxt/bnxt_ethdev.c                |   8 +-
>  drivers/net/cxgbe/base/adapter.h              |   2 +-
>  drivers/net/dpaa/dpaa_ethdev.c                |   9 +-
>  drivers/net/dpaa2/dpaa2_ethdev.c              |   9 +-
>  drivers/net/dpaa2/dpaa2_ptp.c                 |   2 +-
>  drivers/net/e1000/e1000_ethdev.h              |  10 +-
>  drivers/net/e1000/em_ethdev.c                 |   1 -
>  drivers/net/e1000/em_rxtx.c                   |  21 +-
>  drivers/net/e1000/igb_ethdev.c                |   2 -
>  drivers/net/e1000/igb_rxtx.c                  |  21 +-
>  drivers/net/enic/enic_ethdev.c                |  12 +-
>  drivers/net/fm10k/fm10k.h                     |   5 +-
>  drivers/net/fm10k/fm10k_ethdev.c              |   1 -
>  drivers/net/fm10k/fm10k_rxtx.c                |  29 +-
>  drivers/net/hns3/hns3_rxtx.c                  |   7 +-
>  drivers/net/hns3/hns3_rxtx.h                  |   2 +-
>  drivers/net/i40e/i40e_ethdev.c                |   1 -
>  drivers/net/i40e/i40e_rxtx.c                  |  30 +-
>  drivers/net/i40e/i40e_rxtx.h                  |   4 +-
>  drivers/net/iavf/iavf_rxtx.c                  |   4 +-
>  drivers/net/iavf/iavf_rxtx.h                  |   2 +-
>  drivers/net/ice/ice_rxtx.c                    |   4 +-
>  drivers/net/ice/ice_rxtx.h                    |   2 +-
>  drivers/net/igc/igc_ethdev.c                  |   1 -
>  drivers/net/igc/igc_txrx.c                    |  23 +-
>  drivers/net/igc/igc_txrx.h                    |   5 +-
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   2 -
>  drivers/net/ixgbe/ixgbe_ethdev.h              |   5 +-
>  drivers/net/ixgbe/ixgbe_rxtx.c                |  22 +-
>  drivers/net/mlx5/mlx5_rx.c                    |  26 +-
>  drivers/net/mlx5/mlx5_rx.h                    |   2 +-
>  drivers/net/netvsc/hn_rxtx.c                  |   4 +-
>  drivers/net/netvsc/hn_var.h                   |   3 +-
>  drivers/net/nfp/nfp_rxtx.c                    |   4 +-
>  drivers/net/nfp/nfp_rxtx.h                    |   3 +-
>  drivers/net/octeontx2/otx2_ethdev.c           |   1 -
>  drivers/net/octeontx2/otx2_ethdev.h           |   3 +-
>  drivers/net/octeontx2/otx2_ethdev_ops.c       |  20 +-
>  drivers/net/sfc/sfc_ethdev.c                  |  29 +-
>  drivers/net/thunderx/nicvf_ethdev.c           |   3 +-
>  drivers/net/thunderx/nicvf_rxtx.c             |   4 +-
>  drivers/net/thunderx/nicvf_rxtx.h             |   2 +-
>  drivers/net/txgbe/txgbe_ethdev.h              |   3 +-
>  drivers/net/txgbe/txgbe_rxtx.c                |   4 +-
>  drivers/net/vhost/rte_eth_vhost.c             |   4 +-
>  drivers/net/virtio/virtio_ethdev.c            |   1 -
>  lib/ethdev/ethdev_driver.h                    | 148 +++++++++
>  lib/ethdev/ethdev_private.c                   |  83 +++++
>  lib/ethdev/ethdev_private.h                   |   7 +
>  lib/ethdev/rte_ethdev.c                       |  89 ++++--
>  lib/ethdev/rte_ethdev.h                       | 288 ++++++++++++------
>  lib/ethdev/rte_ethdev_core.h                  | 171 +++--------
>  lib/ethdev/version.map                        |   8 +-
>  lib/eventdev/rte_event_eth_rx_adapter.c       |   2 +-
>  lib/eventdev/rte_event_eth_tx_adapter.c       |   2 +-
>  lib/eventdev/rte_eventdev.c                   |   2 +-
>  lib/metrics/rte_metrics_telemetry.c           |   2 +-
>  67 files changed, 677 insertions(+), 564 deletions(-)
> 
> --
> 2.26.3