Re: [dpdk-dev] [RFC PATCH 0/3] ethdev: few changes in rte_ethdev layer

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Fri, 1 Dec 2017 14:55:12 +0000
> From: Konstantin Ananyev <konstantin.ananyev@intel.com>
> To: dev@dpdk.org, dev@dpdk.org
> CC: Konstantin Ananyev <konstantin.ananyev@intel.com>
> Subject: [dpdk-dev] [RFC PATCH 0/3] ethdev: few changes in rte_ethdev layer
> X-Mailer: git-send-email 1.7.0.7
> 

Hi Konstantin,

> The series introduces 2 main changes:
> 
> 1.Introduce a separate data structure (rte_eth_queue_local)
> to store local to given process (i.e. no-shareable) information
> for each configured rx/tx queue.
> Memory for that structure is allocated/freed dynamically during
> rte_eth_dev_configure().
> Reserve a space for queue specific (rx|tx)_pkt_burst(),
> tx_pkt_prepare() function pointers inside that structure.
> Move rx/tx callback related information inside that structure.
> That introduces a change in current behavior: all callbacks for
> un-configured queues will be automatically removed.
> Also as size of struct rte_eth_dev changes that patch is an ABI breakage,
> so deprecation notice for 18.05 is filled.
> Further suggestions how to introduce the same functionality
> without ABI breakage are welcome.

Are we doing this to enable, Tx/Rx queue specific burst functions to
invoke based on new TX/RX queue specific offloads framework.
Meaning, if a offload configured for the given queue,
driver can select a different function pointer for Rx/Tx burst functions.

> 
> 2. Make it safe to remove rx/tx callback at runtime.
> Right now it is not possible for the application to figure out
> when it is safe to free removed callback handle and
> associated with it resources(unless the queue is stopped).
> That's probably not a big problem if all callbacks are static
> hange through whole application lifetime)
> and/or application doesn't allocate any resources for the callback handler.
> Though if callbacks have to be added/removed dynamically and
> callback handler would require more resources to operate properly -
> then it might become an issue.
> So patch #2 fixes that problem - now as soon as
> rte_eth_remove_(rx|tx)_callback() completes successfully, application
> can safely free all associated with the removed callback resources.
> 
> Performance impact:
> If application doesn't use RX/TX callbacks, then the tests I run didn't
> reveal any performance degradation.
> Though if application do use RX/TX callbacks - patch #2 does introduce
> some slowdown.
>  
> To be more specific here, on BDW (E5-2699 v4) 2.2GHz, 4x10Gb (X520-4)
> with http://dpdk.org/dev/patchwork/patch/31864/ patch installed I got:
> 1) testpmd ... --latencystats=1 - slowdown < 1%
> 2) examples//l3fwd ... --parse-ptype - - slowdown < 1%
> 3) examples/rxtx_callbacks - slowdown ~8%
> All that in terms of packet throughput (Mpps).

We tried on an arm64 machine; We got following result on host and guest
using l3fwd.


Note:
+ve number means "Performance regression with patches"
-ve number means "Performance improvement with patches"

Relative performance difference in percentage:

% difference on host (2 x 40G)	
pkt_sz  1c      2c      4c      8c
64	1.41	0.18	0.51	0.29
72	0.50	0.53	0.09	0.19
128	0.31	0.31	0.42	0.00
256	-0.44	-0.44	0.00	0.00
512	1.94	1.94	0.01	0.01
1024	0.00	0.01	0.00	0.01
1518	0.02	0.01	0.02	0.02

% difference on guest (2 x 40G)	
pkt_sz  1c      2c      4c      8c
64      5.78	2.30	-2.45	-0.01
72	1.13	1.13	1.31	-4.29
128	1.36	1.36	1.54	-0.82
256	2.02	2.03	-0.01	-0.35
512	4.05	4.04	0.00	-0.46
1024	0.39	0.38	0.00	-0.74
1518	0.00	0.00	-0.05	-4.20

I think, it is because we added more code under 
the RTE_ETHDEV_RXTX_CALLBACKS and it is enabled by
default.

I think, the impact will vary based on 
architecture and underneath i-cache, d-cache and IPC.

I am just thinking, How we can avoid such impact?
How about,
# Remove RTE_ETHDEV_RXTX_CALLBACKS option
# Hook Rx/TX callbacks under TX/RX offload flags
# ethdev layer gives helper function to invoke
callbacks to driver. But, don't invoke from common code
# When application needs the callback support, it
configured the given RX/TX queue offloads
# If the Rx/TX callback configure by the application
then driver calls the helper functions exposed by
the common code to invoke RX callbacks.

Jerin

> 
> Ability to safely remove callbacks at runtime implies
> some sort of synchronization.
> Even I tried to make it as light as possible,
> probably some slowdown is unavoidable.
> Of course instead of introducing these changes at rte_ethdev layer
> similar technique could be applied on individual callback basis.
> In that case it would be up to callback writer/installer to decide
> does he/she need a removable at runtime callback or not.
> Though in that case, each installed callback might introduce extra
> synchronization overhead and slowdown.
> 
> Konstantin Ananyev (3):
>   ethdev: introduce eth_queue_local
>   ethdev: make it safe to remove rx/tx callback at runtime
>   doc: ethdev ABI change deprecation notice
> 
>  doc/guides/rel_notes/deprecation.rst |   5 +
>  lib/librte_ether/rte_ethdev.c        | 390 ++++++++++++++++++++++-------------
>  lib/librte_ether/rte_ethdev.h        | 174 ++++++++++++----
>  3 files changed, 387 insertions(+), 182 deletions(-)
> 
> -- 
> 2.13.5
>