Re: [dpdk-dev] [PATCH v6 0/6] hide eth dev related structures

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 10/18/2021 5:04 PM, Ali Alnubani wrote:
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Ferruh Yigit
>> Sent: Wednesday, October 13, 2021 11:16 PM
>> To: Konstantin Ananyev <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> jerinj@marvell.com; Ajit Khaparde <ajit.khaparde@broadcom.com>; Raslan
>> Darawsheh <rasland@nvidia.com>; Andrew Rybchenko
>> <andrew.rybchenko@oktetlabs.ru>; Qi Zhang <qi.z.zhang@intel.com>;
>> Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>> Cc: xiaoyun.li@intel.com; anoobj@marvell.com; jerinj@marvell.com;
>> ndabilpuram@marvell.com; adwivedi@marvell.com;
>> shepard.siegel@atomicrules.com; ed.czeck@atomicrules.com;
>> john.miller@atomicrules.com; irusskikh@marvell.com;
>> ajit.khaparde@broadcom.com; somnath.kotur@broadcom.com;
>> rahul.lakkireddy@chelsio.com; hemant.agrawal@nxp.com;
>> sachin.saxena@oss.nxp.com; haiyue.wang@intel.com; johndale@cisco.com;
>> hyonkim@cisco.com; qi.z.zhang@intel.com; xiao.w.wang@intel.com;
>> humin29@huawei.com; yisen.zhuang@huawei.com; oulijun@huawei.com;
>> beilei.xing@intel.com; jingjing.wu@intel.com; qiming.yang@intel.com;
>> Matan Azrad <matan@nvidia.com>; Slava Ovsiienko
>> <viacheslavo@nvidia.com>; sthemmin@microsoft.com; NBU-Contact-longli
>> <longli@microsoft.com>; heinrich.kuhn@corigine.com;
>> kirankumark@marvell.com; andrew.rybchenko@oktetlabs.ru;
>> mczekaj@marvell.com; jiawenwu@trustnetic.com;
>> jianwang@trustnetic.com; maxime.coquelin@redhat.com;
>> chenbo.xia@intel.com; NBU-Contact-Thomas Monjalon
>> <thomas@monjalon.net>; mdr@ashroe.eu; jay.jayatheerthan@intel.com
>> Subject: Re: [dpdk-dev] [PATCH v6 0/6] hide eth dev related structures
>>
>> On 10/13/2021 2:36 PM, Konstantin Ananyev wrote:
>>> v6 changes:
>>> - Update comments (Andrew)
>>> - Move callback related variables under corresponding ifdefs (Andrew)
>>> - Few nits in rte_eth_macaddrs_get (Andrew)
>>> - Rebased on top of next-net tree
>>>
>>> v5 changes:
>>> - Fix spelling (Thomas/David)
>>> - Rename internal helper functions (David)
>>> - Reorder patches and update commit messages (Thomas)
>>> - Update comments (Thomas)
>>> - Changed layout in rte_eth_fp_ops, to group functions and
>>>      related data based on their functionality:
>>>      first 64B line for Rx, second one for Tx.
>>>      Didn't observe any real performance difference comparing to
>>>      original layout. Though decided to keep a new one, as it seems
>>>      a bit more plausible.
>>>
>>> v4 changes:
>>>    - Fix secondary process attach (Pavan)
>>>    - Fix build failure (Ferruh)
>>>    - Update lib/ethdev/verion.map (Ferruh)
>>>      Note that moving newly added symbols from EXPERIMENTAL to DPDK_22
>>>      section makes checkpatch.sh to complain.
>>>
>>> v3 changes:
>>>    - Changes in public struct naming (Jerin/Haiyue)
>>>    - Split patches
>>>    - Update docs
>>>    - Shamelessly included Andrew's patch:
>>>      https://patches.dpdk.org/project/dpdk/patch/20210928154856.1015020-
>> 1-andrew.rybchenko@oktetlabs.ru/
>>>      into these series.
>>>      I have to do similar thing here, so decided to avoid duplicated effort.
>>>
>>> The aim of these patch series is to make rte_ethdev core data structures
>>> (rte_eth_dev, rte_eth_dev_data, rte_eth_rxtx_callback, etc.) internal to
>>> DPDK and not visible to the user.
>>> That should allow future possible changes to core ethdev related structures
>>> to be transparent to the user and help to improve ABI/API stability.
>>> Note that current ethdev API is preserved, but it is a formal ABI break.
>>>
>>> The work is based on previous discussions at:
>>> https://www.mail-archive.com/dev@dpdk.org/msg211405.html
>>> https://www.mail-archive.com/dev@dpdk.org/msg216685.html
>>> and consists of the following main points:
>>> 1. Copy public 'fast' function pointers (rx_pkt_burst(), etc.) and
>>>      related data pointer from rte_eth_dev into a separate flat array.
>>>      We keep it public to still be able to use inline functions for these
>>>      'fast' calls (like rte_eth_rx_burst(), etc.) to avoid/minimize slowdown.
>>>      Note that apart from function pointers itself, each element of this
>>>      flat array also contains two opaque pointers for each ethdev:
>>>      1) a pointer to an array of internal queue data pointers
>>>      2)  points to array of queue callback data pointers.
>>>      Note that exposing this extra information allows us to avoid extra
>>>      changes inside PMD level, plus should help to avoid possible
>>>      performance degradation.
>>> 2. Change implementation of 'fast' inline ethdev functions
>>>      (rte_eth_rx_burst(), etc.) to use new public flat array.
>>>      While it is an ABI breakage, this change is intended to be transparent
>>>      for both users (no changes in user app is required) and PMD developers
>>>      (no changes in PMD is required).
>>>      One extra note - with new implementation RX/TX callback invocation
>>>      will cost one extra function call with this changes. That might cause
>>>      some slowdown for code-path with RX/TX callbacks heavily involved.
>>>      Hope such trade-off is acceptable for the community.
>>> 3. Move rte_eth_dev, rte_eth_dev_data, rte_eth_rxtx_callback and
>> related
>>>      things into internal header: <ethdev_driver.h>.
>>>
>>> That approach was selected to:
>>>     - Avoid(/minimize) possible performance losses.
>>>     - Minimize required changes inside PMDs.
>>>
>>> Performance testing results (ICX 2.0GHz, E810 (ice)):
>>>    - testpmd macswap fwd mode, plus
>>>      a) no RX/TX callbacks:
>>>         no actual slowdown observed
>>>      b) bpf-load rx 0 0 JM ./dpdk.org/examples/bpf/t3.o:
>>>         ~2% slowdown
>>>    - l3fwd: no actual slowdown observed
>>>
>>> Would like to thank everyone who already reviewed and tested previous
>>> versions of these series. All other interested parties please don't be shy
>>> and provide your feedback.
>>>
>>> Konstantin Ananyev (6):
>>>     ethdev: allocate max space for internal queue array
>>>     ethdev: change input parameters for rx_queue_count
>>>     ethdev: copy fast-path API into separate structure
>>>     ethdev: make fast-path functions to use new flat array
>>>     ethdev: add API to retrieve multiple ethernet addresses
>>>     ethdev: hide eth dev related structures
>>>
>>
>> For series,
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>
>> No performance regression detected on my testing.
>>
>> I am merging the series to next-net now which helps testing,
>> but before merging to main repo it will be good to get more
>> ack and test results (I can squash new tags later).
>>
>> @Jerin, @Ajit, @Raslan, @Andrew, @Qi, @Honnappa,
>> Can you please test this set for any possible regression?
>>
>> Series applied to dpdk-next-net/main, thanks.
>>
> 
> Tested (on dpdk-next-net/main) single and multi-core packet forwarding performance with testpmd on both ConnectX-5 and ConnectX-6 Dx. I didn't see any noticeable regressions.
> 

Thanks!

At this stage I am putting set to pull request for main repo.
Last day for anyone who wants to test the set.