Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type update as an offload

[Ori Kam <orika@mellanox.com>]

Yes missed somehow 6.
One more general comment, what happened until now? The Mark is 
already implemented and working more then 2 years.

> -----Original Message-----
> From: Andrew Rybchenko <arybchenko@solarflare.com>
> Sent: Friday, November 8, 2019 2:20 PM
> To: Ori Kam <orika@mellanox.com>; Zhang, Qi Z <qi.z.zhang@intel.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; pbhagavatula@marvell.com; Yigit, Ferruh
> <ferruh.yigit@intel.com>; jerinj@marvell.com; Mcnamara, John
> <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; Adrien Mazarguil
> <adrien.mazarguil@6wind.com>; david.marchand@redhat.com;
> ktraynor@redhat.com; Olivier Matz <olivier.matz@6wind.com>
> Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type update as
> an offload
> 
> On 11/8/19 3:12 PM, Ori Kam wrote:
> >
> > Hi Andrew,
> >
> > Thanks very much for the summery, I think this makes the discussion much
> better.
> > (we even got some new folks 😊)
> >
> > I vote for B and C,
> >
> > Please read my comments below.
> >
> > Ori
> >
> >> -----Original Message-----
> >> From: dev <dev-bounces@dpdk.org> On Behalf Of Zhang, Qi Z
> >> Sent: Friday, November 8, 2019 1:41 PM
> >> To: Thomas Monjalon <thomas@monjalon.net>; Andrew Rybchenko
> >> <arybchenko@solarflare.com>
> >> Cc: Ori Kam <orika@mellanox.com>; dev@dpdk.org;
> >> pbhagavatula@marvell.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> >> jerinj@marvell.com; Mcnamara, John <john.mcnamara@intel.com>;
> >> Kovacevic, Marko <marko.kovacevic@intel.com>; Adrien Mazarguil
> >> <adrien.mazarguil@6wind.com>; david.marchand@redhat.com;
> >> ktraynor@redhat.com; Olivier Matz <olivier.matz@6wind.com>
> >> Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type update
> as
> >> an offload
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: dev <dev-bounces@dpdk.org> On Behalf Of Thomas Monjalon
> >>> Sent: Friday, November 8, 2019 7:04 PM
> >>> To: Andrew Rybchenko <arybchenko@solarflare.com>
> >>> Cc: Ori Kam <orika@mellanox.com>; dev@dpdk.org;
> >>> pbhagavatula@marvell.com; Yigit, Ferruh <ferruh.yigit@intel.com>;
> >>> jerinj@marvell.com; Mcnamara, John <john.mcnamara@intel.com>;
> >>> Kovacevic, Marko <marko.kovacevic@intel.com>; Adrien Mazarguil
> >>> <adrien.mazarguil@6wind.com>; david.marchand@redhat.com;
> >>> ktraynor@redhat.com; Olivier Matz <olivier.matz@6wind.com>
> >>> Subject: Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type
> update as
> >> an
> >>> offload
> >>>
> >>> 08/11/2019 11:42, Andrew Rybchenko:
> >>>> On 11/8/19 1:28 PM, Thomas Monjalon wrote:
> >>>>> 08/11/2019 09:35, Andrew Rybchenko:
> >>>>>> The problem:
> >>>>>> ~~~~~~~~~~~~
> >>>>>> PMD wants to know before port start if application wants to to use
> >>>>>> flow MARK/FLAG in the future. It is required because:
> >>>>>>
> >>>>>> 1. HW may be configured in a different way to reserve resources
> >>>>>>    for MARK/FLAG delivery
> >>>>>>
> >>>>>> 2. Datapath implementation choice may depend on it (e.g. vPMD
> >>>>>>    is faster, but does not support MARK)
> >>>>>
> >>>>> Thank you for the clear problem statement.
> >>>>> I agree with it. This is a real design issue.
> >>>>>
> >>>>>
> >>>>>> Discussed solutions:
> >>>>>> ~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>> May be it is not 100% clear since below are alternatives.
> >>>>
> >>>>>> A. Explicit Rx offload suggested by the patch.
> >>>>>>
> >>>>>> B. Implicit by validation of a flow rule with MARK/FLAG actions used.
> >>>>>>
> >
> > I don’t agree that this method is implicit it is quit explicit, do we support it in
> PMD
> > with the use case or not. Sound very explicit. You can argue that the PMD
> uses
> > validation flow to understand that it need to support MARK, but from
> every perspective
> > it is more correct . what does it mean if the application enabled mark and
> never download a flow?
> 
> It is implicit from PMD point of view, since there is no command
> to enable it. The decision to enable is the result of seen
> attempt to validate mark. If few rules are validated, but an
> application decides to have it disabled finally, how to achieve it?
> Last validation attempt wins?
> Other points are covered by (6) below.
> 
> >>>>>> C. Use dynamic field/flag (i.e. application registers dynamic field
> >>>>>>    and/or flag and PMD uses lookup to solve the problem) plus part
> >>>>>>    of (B) to discover if a feature is supported.
> >>>>>
> >>>>> The dynamic field should be registered via a new API function named
> >>>>> '<feature>_init'.
> >>>>> It means the application must explicit request the feature.
> >>>>> I agree this is the way to go.
> >>>>
> >>>> If I understand your statement correctly, but (C) is not ideal since
> >>>> it looks global. If registered dynamic field of mbuf and is flag that
> >>>> the feature should be enabled, it is a flag to all ports/devices.
> >>>>
> >>>>>> All solutions require changes in applications which use these
> >>>>>> features. There is a deprecation notice in place which advertises
> >>>>>> DEV_RX_OFFLOAD_FLOW_MARK addition, but may be it is OK to
> >>>>>> substitute it with solution (B) or (C). Solution (C) requires
> >>>>>> changes since it should be combined with (B) in order to understand
> >>>>>> if the feature is supported.
> >>>>>
> >>>>> I don't understand.
> >>>>> Application request and PMD support are two different things.
> >>>>> PMD support must be via rte_flow validation on a case by case
> anyway.
> >>>>
> >>>> I mean that application wants to understand if the feature is
> >>>> supported. Then, it wants to enable it. In the case of (B), if I
> >>>> understand the solution correctly, there is no explicit way to enable,
> >>>> PMD just detects it because of discovery is done (that's what I mean
> >>>> by "implicit" and it is a drawback from my point of view, but still
> >>>> could be considered). (C) solves the problem of (B).
> >>>>
> >>>>>> Advantages and drawbacks of solutions:
> >>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>> 1. The main drawback of (A) is a "duplication" since we already
> >>>>>>    have a way to request flow MARK using rte_flow API.
> >>>>>>    I don't fully agree that it is a duplication, but I agree
> >>>>>>    that it sounds like duplication and complicates a bit flow
> >>>>>>    MARK usage by applications. (B) complicates it as well.
> >>>>>>
> >>>>>> 2. One more drawback of the solution (A) is the necessity of
> >>>>>>    similar solution for META and it eats one more offload bit.
> >>>>>>    Yes, that's true and I think it is not a problem.
> >>>>>>    It would make it easier for applications to find out if
> >>>>>>    either MARK or META is supported.
> >>>>>>
> >>>>>> 3. The main advantage of the solution (A) is simplicity.
> >>>>>>    It is simple for application to understand if it supported.
> >>>>>>    It is simple in PMD to understand that it is required.
> >>>>>>    It is simple to disable it - just reconfigure.
> >>>>>>    Also it is easier to document it - just mention that
> >>>>>>    the offload should be supported and enabled.
> >>>>>>
> >
> > This is not true, for example the PMD support Mark only in Nic mode
> > While the application needs E-Switch then the PMD reports incorrectly.
> > Or for example if the mark is supported only as only action, and the
> application needs
> > decap and mark. this means that the user will never be able to use the
> mark
> > so the PMD uses slower Rx function without the mark ever being used.
> 
> It is covered by (6) below.
> 
> >>>>>> 4. The main advantage of the solution (B) is no "duplication".
> >>>>>>    I agree that it is valid argument. Solving the problem
> >>>>>>    without extra entities is always nice, but unfortunately
> >>>>>>    it is too complex in this case.
> >>>>>>
> >
> > You forgot a one more very important reason, the fact that the mark
> > may be supported only on some flows. So if the user needs mark with
> decap
> > and this combination is not supported the user will never we able to use
> mark
> > while using mark Rx function. So you get slow data path with no ability to
> use it.
> 
> Again, see (6) below.
> 
> >>>>>> 5. The main drawback of the solution (B) is the complexity.
> >>>>>>    It is necessary to choose a flow rule which should be used
> >>>>>>    as a criteria. It could be hardware dependent.
> >>>>>>    Complex logic is require in PMD if it wants to address the
> >>>>>>    problem and control MARK delivery based on validated flow
> >>>>>>    rules. It adds dependency between start/stop processing and
> >>>>>>    flow rules validation code.
> >>>>>>    It is pretty complicated to document it.
> >>>>>>
> >>>>>> 6. Useless enabling of the offload in the case of solution (A)
> >>>>>>    if really used flow rules do not support MARK looks like
> >>>>>>    drawback as well, but easily mitigated by a combination
> >>>>>>    with solution (B) and only required if the application wants
> >>>>>>    to dive in the level of optimization and complexity and
> >>>>>>    makes sense if application knows required flow rules in
> >>>>>>    advance. So, it is not a problem in this case.
> >>>>>>
> >>>>>> 7. Solution (C) has drawbacks of the solution (B) for
> >>>>>>    applications to understand if these features are supported,
> >>>>>>    but no drawbacks in PMD, since explicit criteria is used to
> >>>>>>    enable/disable (dynamic field/flag lookup).
> >>>>>>
> >>>>>> 8. Solution (C) is nice since it avoids "duplication".
> >>>>>>
> >>>>>> 9. The main drawback of the solution (C) is asymmetry.
> >>>>>>    As it was discussed in the case of RX_TIMESTAMP
> >>>>>>    (if I remember it correctly):
> >>>>>>     - PMD advertises RX_TIMESTAMP offload capability
> >>>>>>     - application enables the offload
> >>>>>>     - PMD registers dynamic field for timestamp
> >>>>>>    Solution (C):
> >>>>>>      - PMD advertises nothing
> >>>>>>      - application uses solution (B) to understand if
> >>>>>>        these features are supported
> >>>>>>      - application registers dynamic field/flag
> >>>>>>      - PMD does lookup and solve the problem
> >>>>>>    The asymmetry could be partially mitigated if RX_TIMESTAMP
> >>>>>>    solution is changed to require an application to register
> >>>>>>    dynamic fields and PMD to do lookup if the offload is
> >>>>>>    enabled. So, the only difference will be in no offload
> >>>>>>    in the case of flow MARK/FLAG and usage of complex logic
> >>>>>>    to understand if it is supported or no.
> >>>>>>    May be it would be really good since it will allow to
> >>>>>>    have dynamic fields registered before mempool population.
> >>>>>>
> >>>>>> 10. Common drawback of solutions (B) and (C) is no granularity.
> >>>>>>     Solution (A) may be per queue while (B) and (C) cannot be
> >>>>>>     per queue. Moreover (C) looks global - for all devices.
> >>>>>>     It could be really painful.
> >>>>>>
> >>>>>> (C) is nice, but I still vote for simplicity and granularity of
> >>>>>> (A).
> >>>>>
> >>>>> I vote for clear separation of application needs and PMD support, by
> >>>>> using the method C (dynamic fields).
> >>>>> I agree timestamp must use the same path.
> >>>>> I agree it's complicate because we don't know in advance whether a
> >>>>> flow rule will be accepted, but that's the reality, config is complex.
> >>>>
> >>>> Do you think that global nature of the (C) is acceptable?
> >>>
> >>> That's a good question.
> >>> Maybe the feature request should be per port.
> >>> In this case, we are back to solution A with a flag per port?
> >>>
> >>> Note that A and C will not guarantee that the offload will be possible.
> >>> We need B (flow rule validation) anyway.
> >>
> >> I may not understand how solution B can works well for all the cases.
> >>
> >
> > That is the point of solution B it always work, it is a bit harder to use maybe,
> > but you can check and decide everything.
> >
> >> A rte_flow rule can be issued after dev_start, which means the rx_burst
> >> function is already selected at that time,
> >> so does that mean the driver need to switch from a non- mark offload
> aware
> >> path to a mark offload aware path without stop device? or it has to reject
> the
> >> flow?
> >> The question is if we have 2 data path, one support some offload , one
> not
> >> but more fast, which one should be selected during dev_start? Isn't
> Offload
> >> widely used to solve this problem?
> >>
> >> I think the option A solve all the problems, option C might also works, but
> A is
> >> looks much straightforward for me.
> >>
> >
> > Solution A my result in selecting incorrect datapath  please see my answer
> above.
> >
> > As you can guess I'm in favor of B and C (C need B)
> >
> >> Regards
> >> Qi
> >>
> >>
> >>>
> >>> It seems A, B, C are not alternatives but all required as pieces of a
> puzzle...
> >>>
> >