Re: [dpdk-dev] [PATCH v2 3/3] ethdev: improve flow mark Rx offload deprecation notice

[Jerin Jacob <jerinjacobk@gmail.com>]

On Mon, Dec 2, 2019 at 5:27 PM Andrew Rybchenko
<arybchenko@solarflare.com> wrote:
>
> On 12/2/19 2:09 PM, Jerin Jacob wrote:
> > On Mon, Dec 2, 2019 at 6:16 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 02/12/2019 05:21, Jerin Jacob:
> >>> On Mon, Nov 25, 2019 at 8:39 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>>>
> >>>> 25/11/2019 11:44, Jerin Jacob:
> >>>>> On Sun, Nov 24, 2019 at 3:12 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>>>>>
> >>>>>> 23/11/2019 10:42, Jerin Jacob:
> >>>>>>> On Sat, Nov 23, 2019 at 3:58 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>>>>>>> 22/11/2019 12:53, Andrew Rybchenko:
> >>>>>>>>> On 11/22/19 2:15 PM, Thomas Monjalon wrote:
> >>>>>>>>>> 22/11/2019 11:12, Andrew Rybchenko:
> >>>>>>>>>>> On 11/22/19 1:01 AM, Thomas Monjalon wrote:
> >>>>>>>>>>>> 19/11/2019 13:12, Andrew Rybchenko:
> >>>>>>>>>>>>> The deprecation notice is required since it adds more requirements
> >>>>>>>>>>>>> when RTE flow mark and flag actions may be used and require
> >>>>>>>>>>>>> changes in applications.
> >>>>>>>>>>>> I am still not sure what is the best solution here.
> >>>>>>>>>>>> I continued to think about it in this thread:
> >>>>>>>>>>>>   http://mails.dpdk.org/archives/dev/2019-November/151960.html
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think we cannot require any application change until 20.11
> >>>>>>>>>>>> in order to keep API (and behaviour) compatibility.
> >>>>>>>>>>> Expected, but still very disappointing.
> >>>>>>>>>>>
> >>>>>>>>>>> The feature is implemented by Pavan (@ Marvell), supported by me,
> >>>>>>>>>>> used by Qi (@ Intel), looks better than alternatives from application
> >>>>>>>>>>> developer point of view [1] and finally postponed for 1 year without really
> >>>>>>>>>>> strong motivation.
> >>>>>>>>>>
> >>>>>>>>>> I see different valuable point of views. This is enough motivation.
> >>>>>>>>>
> >>>>>>>>> It looks like I miss it in previous discussion, I would be thankful if
> >>>>>>>>> you give me links to read or hints how to find.
> >>>>>>>>
> >>>>>>>> http://mails.dpdk.org/archives/dev/2019-November/150793.html
> >>>>>>>>
> >>>>>>>>> Introducing new types of controls would make configuration more and
> >>>>>>>>> more complex. I think that many different types of control would
> >>>>>>>>> over-complicate it. May be it is unavoidable, but it should be clear
> >>>>>>>>> why the problem cannot be solved using existing types of controls
> >>>>>>>>> (e.g. offloads).
> >>>>>>>>
> >>>>>>>> The offload control is used as an effective configuration for now.
> >>>>>>>> The features which are configured with DEV_RX_OFFLOAD_*
> >>>>>>>> do not need any other API to be used.
> >>>>>>>> Extending DEV_RX_OFFLOAD_* bits for enabling features which
> >>>>>>>> must be configured via other API anyway, is possible.
> >>>>>>>> The real problem is that features in DEV_RX_OFFLOAD_* are supposed
> >>>>>>>> to be disabled by default. If we add some opt-in features here,
> >>>>>>>> we cannot enable them by default for API compatibility and do the
> >>>>>>>> right thing by default.
> >>>>>>>>
> >>>>>>>> Choosing DEV_RX_OFFLOAD_* bits or rte_eth_dev_opt* functions is a detail.
> >>>>>>>> The real decision is to change the API for using all these features.
> >>>>>>>> Can we keep all features available by default (opt-out)?
> >>>>>>>
> >>>>>>> IMO, *rte_eth_dev_opt* has following problems
> >>>>>>>
> >>>>>>> 1) It is not multi-process friendly. If we are changing the Rx/Tx
> >>>>>>> function pointer, based on
> >>>>>>> the selected offload, then, using *rte_eth_dev_opt* scheme won't
> >>>>>>> really work(if the new API
> >>>>>>> called after the secondary process launch)
> >>>>>>
> >>>>>> Yes it must be used before launching the secondary process.
> >>>>>> It is the same as DEV_RX_OFFLOAD_* config.
> >>>>>
> >>>>> Yes. rte_eth_dev_opt_* has another dimension to enable and disable as API.
> >>>>> So, we need to document, opt-in -> start() -> opt-out case won't work
> >>>>> in multi process
> >>>>> case.
> >>>>>
> >>>>>>
> >>>>>>> 2) If we are taking rte_eth_dev_opt path then by default feature has
> >>>>>>> to be enabled to
> >>>>>>> not break the functional ABI. That scheme won't scale if as when we
> >>>>>>> keep adding the new features.
> >>>>>>> It is always easy for the application to define "what it wants" vs
> >>>>>>> "what it does not want"
> >>>>>>
> >>>>>> Yes, opt-in may look more natural than opt-out.
> >>>>>> But opt-in makes the default more complex, and changes the API.
> >>>>>>
> >>>>>>> 3) Defining the device state after the reconfigure operation.
> >>>>>>>
> >>>>>>> IMO, if any operation is connected to fastpath it is better to use
> >>>>>>> DEV_RX_OFFLOAD_ like
> >>>>>>> this feature where enable or disable PMDs from updating
> >>>>>>> ``rte_mbuf::hash::fdir`` so that if possible
> >>>>>>> we can use different Rx function pointer if possible(Hence it can work
> >>>>>>> with the multi-process case case)
> >>>>>>
> >>>>>> I reply to 2 and 3 together.
> >>>>>>
> >>>>>> We decided that offloads must be disabled by default.
> >>>>>> This is what we have in 19.11:
> >>>>>>         - Some offloads are enabled before start with DEV_?X_OFFLOAD_*
> >>>>>>         - Some offloads are enabled with functions at any time
> >>>>>>
> >>>>>> For the second type of offloads, you want to know, before start,
> >>>>>> whether it will be used or not.
> >>>>>> If adding the second type of offloads (like rte_flow ones)
> >>>>>> to DEV_?X_OFFLOAD_*, it means it must be enabled 2 times:
> >>>>>>         - before start with offload bits
> >>>>>>         - later with more precise functions
> >>>>>>
> >>>>>> I would like to avoid changing the default behaviour,
> >>>>>> which is to enable an offload only one time.
> >>>>>> That's why I think this second category of offloads should
> >>>>>> offer opt-out (global disabling), so it will continue
> >>>>>> to work by default if they are configured.
> >>>>>>
> >>>>>> I hope you understand the difference between the two categories.
> >>>>>
> >>>>> I understand the difference. The only point of "difference in opinion" is
> >>>>> the default behavior of the feature/offload. If it is in RX_OFFLOAD scheme then
> >>>>> by default it is disabled. opt_* scheme makes this new feature/offload
> >>>>> enabled default to avoid changing the default behavior.
> >>>>
> >>>> OK, this is where we disagree.
> >>>> I am for keeping what we agreed this year: all offloads are disabled by default.
> >>>> But I am against the need for double enablement.
> >>>> The offloads which are enabled with a specific function should not need
> >>>> to be also enabled (opt-in) before start.
> >>>
> >>> OK.
> >>>
> >>>>
> >>>>> It is good to avoid functional ABI change. But bad as,
> >>>>> 1) New API starts bloating the ethdev API.
> >>>>
> >>>> In general, I want to clean-up the ethdev API during next year.
> >>>>
> >>>>> 2) It is diffcult for application guys to figure out what are features need to
> >>>>> be disabled to performance as he/she does not know, for the given release,
> >>>>> the enabled features.
> >>>>
> >>>> Yes this is a good point.
> >>>>
> >>>>> Item (1) is a trade-off between elegance vs ABI compatibility. No
> >>>>> strong opinion on this.
> >>>>>
> >>>>> To fix the item (2), Can we get have an API in ethdev to get enabled
> >>>>> features so that
> >>>>> the application can probe and disable if required?
> >>>>
> >>>> We can think about something like that.
> >>>> Note that there is also a need to better advertise all capabilities.
> >>>>
> >>>>> For example, rte_eth_dev_set_ptypes() comes in same category, By default,
> >>>>> ptype parsing is enabled. I think, we can have a general interface to
> >>>>> "probe" the by default enabled features
> >>>>> and disable it if required. Not scattered API for each feature.
> >>>>
> >>>> This is an issue. The packet type parsing should be disabled by default.
> >>>
> >>> IMO, It makes sense to disable by default.
> >>>
> >>> Isn't conflicting? One thread, we are saying for in order to make,
> >>> existing application work without breaking ABI, Default should be
> >>> enabled.
> >>>
> >>> Thoughts?
> >>
> >> Every offloads should be disabled by default.
> >> This is a good reason to break the behaviour in 20.11.
> >
> > Ack.
>
> Yes, I agree as well, but in general we already have an
> exception MBUF_FAST_FREE which is just a nice wrap for
> enabled by default support for mbufs from different
> mempools and support for mbuf reference counters.
> I don't suggest to change it. Just want to highlight
> that we already have exceptions (nicely wrapped).
> That's why I would not touch packet type parsing
> "offload". Packet type parsing is not just on/off and
> adding on/off in addition to existing API looks overkill.
> Yes, it is one more exception, but nicely wrapped as well.

I am all for making offloads disabled by default.

>
> >>> And what would be DEFAULT behavior for the mbuf MARK updation feature?
> >>> (That's where this thread started).
> >>
> >> As all other features, mark is disabled by default.
> >> Using a rte_flow rule, it can be enabled.
> >> No need to pre-enable it.
> >
> > Ok.
>
> But it returns us to the point where we started [1]:
>
> The problem:
> ~~~~~~~~~~~~
> PMD wants to know before port start if application wants to
> to use flow MARK/FLAG in the future. It is required because:
>
> 1. HW may be configured in a different way to reserve resources
>    for MARK/FLAG delivery.
>
> 2. Datapath implementation choice may depend on it (e.g. vPMD
>    is faster, but does not support MARK).
>
> opt-in/opt-out solution has drawbacks mentioned above.
> Also I'm not sure if opt-in/opt-out is per-queue or per-port.
> (Offloads may be naturally per-queue and it is a big advantage).
>
> IMHO feature which should be opt-out is almost equivalent to
> offload enabled by default. It has the same negative properties
> as enabled by default offloads.
>
> Am I missing something again?
>
> From my point of view I see no problem in necessity to say
> in advance (before device start) that application would like
> to use some features at run time.

I agree with your problem definition and solution as offload.

I think, our constraint is, we can not change functional ABI behavior
for the next year. i.e The existing application should work for the
next year without
changing the code.

I think, it all boiling down to adhere to that constraint or not for
this specific case.

Once that is decided, we can wrap it in offload flags vs opt scheme
(by default enabled scheme).

>
> Yes, all features which may be controlled at run-time are
> headache for optimizations (VLAN offloads).
>
> [1]
> http://inbox.dpdk.org/dev/f170105b-9c60-1b04-cb18-52e0951ddcdb@solarflare.com/