Re: [dpdk-dev] [PATCH 1/2] ethdev: add flow action type update as an offload

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 7/3/2020 3:34 PM, Ferruh Yigit wrote:
> On 11/19/2019 11:09 AM, Thomas Monjalon wrote:
>> 19/11/2019 11:59, Andrew Rybchenko:
>>> On 11/19/19 12:50 PM, Thomas Monjalon wrote:
>>>> 19/11/2019 10:24, Andrew Rybchenko:
>>>>> On 11/8/19 4:30 PM, Thomas Monjalon wrote:
>>>>>> 08/11/2019 14:27, Andrew Rybchenko:
>>>>>>> On 11/8/19 4:17 PM, Thomas Monjalon wrote:
>>>>>>>> 08/11/2019 13:00, Andrew Rybchenko:
>>>>>>>>> On 11/8/19 2:03 PM, Thomas Monjalon wrote:
>>>>>>>>>> 08/11/2019 11:42, Andrew Rybchenko:
>>>>>>>>>>> On 11/8/19 1:28 PM, Thomas Monjalon wrote:
>>>>>>>>>>>> 08/11/2019 09:35, Andrew Rybchenko:
>>>>>>>>>>>>> The problem:
>>>>>>>>>>>>> ~~~~~~~~~~~~
>>>>>>>>>>>>> PMD wants to know before port start if application wants to
>>>>>>>>>>>>> to use flow MARK/FLAG in the future. It is required because:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. HW may be configured in a different way to reserve resources
>>>>>>>>>>>>>     for MARK/FLAG delivery
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Datapath implementation choice may depend on it (e.g. vPMD
>>>>>>>>>>>>>     is faster, but does not support MARK)
>>>>>>>>>>>> Thank you for the clear problem statement.
>>>>>>>>>>>> I agree with it. This is a real design issue.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Discussed solutions:
>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>> May be it is not 100% clear since below are alternatives.
>>>>>>>>>>>
>>>>>>>>>>>>> A. Explicit Rx offload suggested by the patch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> B. Implicit by validation of a flow rule with MARK/FLAG actions used.
>>>>>>>>>>>>>
>>>>>>>>>>>>> C. Use dynamic field/flag (i.e. application registers dynamic field
>>>>>>>>>>>>>     and/or flag and PMD uses lookup to solve the problem) plus part
>>>>>>>>>>>>>     of (B) to discover if a feature is supported.
>>>>>>>>>>>> The dynamic field should be registered via a new API function
>>>>>>>>>>>> named '<feature>_init'.
>>>>>>>>>>>> It means the application must explicit request the feature.
>>>>>>>>>>>> I agree this is the way to go.
>>>>>>>>>>> If I understand your statement correctly, but (C) is not ideal since it
>>>>>>>>>>> looks global. If registered dynamic field of mbuf and is flag that
>>>>>>>>>>> the feature should be enabled, it is a flag to all ports/devices.
>>>>>>>>>>>
>>>>>>>>>>>>> All solutions require changes in applications which use these
>>>>>>>>>>>>> features. There is a deprecation notice in place which advertises
>>>>>>>>>>>>> DEV_RX_OFFLOAD_FLOW_MARK addition, but may be it is OK to substitute
>>>>>>>>>>>>> it with solution (B) or (C). Solution (C) requires changes since
>>>>>>>>>>>>> it should be combined with (B) in order to understand if
>>>>>>>>>>>>> the feature is supported.
>>>>>>>>>>>> I don't understand.
>>>>>>>>>>>> Application request and PMD support are two different things.
>>>>>>>>>>>> PMD support must be via rte_flow validation on a case by case anyway.
>>>>>>>>>>> I mean that application wants to understand if the feature is
>>>>>>>>>>> supported. Then, it wants to enable it. In the case of (B),
>>>>>>>>>>> if I understand the solution correctly, there is no explicit
>>>>>>>>>>> way to enable, PMD just detects it because of discovery is done
>>>>>>>>>>> (that's what I mean by "implicit" and it is a drawback from my
>>>>>>>>>>> point of view, but still could be considered). (C) solves the
>>>>>>>>>>> problem of (B).
>>>>>>>>>>>
>>>>>>>>>>>>> Advantages and drawbacks of solutions:
>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>> 1. The main drawback of (A) is a "duplication" since we already
>>>>>>>>>>>>>     have a way to request flow MARK using rte_flow API.
>>>>>>>>>>>>>     I don't fully agree that it is a duplication, but I agree
>>>>>>>>>>>>>     that it sounds like duplication and complicates a bit flow
>>>>>>>>>>>>>     MARK usage by applications. (B) complicates it as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. One more drawback of the solution (A) is the necessity of
>>>>>>>>>>>>>     similar solution for META and it eats one more offload bit.
>>>>>>>>>>>>>     Yes, that's true and I think it is not a problem.
>>>>>>>>>>>>>     It would make it easier for applications to find out if
>>>>>>>>>>>>>     either MARK or META is supported.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. The main advantage of the solution (A) is simplicity.
>>>>>>>>>>>>>     It is simple for application to understand if it supported.
>>>>>>>>>>>>>     It is simple in PMD to understand that it is required.
>>>>>>>>>>>>>     It is simple to disable it - just reconfigure.
>>>>>>>>>>>>>     Also it is easier to document it - just mention that
>>>>>>>>>>>>>     the offload should be supported and enabled.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. The main advantage of the solution (B) is no "duplication".
>>>>>>>>>>>>>     I agree that it is valid argument. Solving the problem
>>>>>>>>>>>>>     without extra entities is always nice, but unfortunately
>>>>>>>>>>>>>     it is too complex in this case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 5. The main drawback of the solution (B) is the complexity.
>>>>>>>>>>>>>     It is necessary to choose a flow rule which should be used
>>>>>>>>>>>>>     as a criteria. It could be hardware dependent.
>>>>>>>>>>>>>     Complex logic is require in PMD if it wants to address the
>>>>>>>>>>>>>     problem and control MARK delivery based on validated flow
>>>>>>>>>>>>>     rules. It adds dependency between start/stop processing and
>>>>>>>>>>>>>     flow rules validation code.
>>>>>>>>>>>>>     It is pretty complicated to document it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 6. Useless enabling of the offload in the case of solution (A)
>>>>>>>>>>>>>     if really used flow rules do not support MARK looks like
>>>>>>>>>>>>>     drawback as well, but easily mitigated by a combination
>>>>>>>>>>>>>     with solution (B) and only required if the application wants
>>>>>>>>>>>>>     to dive in the level of optimization and complexity and
>>>>>>>>>>>>>     makes sense if application knows required flow rules in
>>>>>>>>>>>>>     advance. So, it is not a problem in this case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 7. Solution (C) has drawbacks of the solution (B) for
>>>>>>>>>>>>>     applications to understand if these features are supported,
>>>>>>>>>>>>>     but no drawbacks in PMD, since explicit criteria is used to
>>>>>>>>>>>>>     enable/disable (dynamic field/flag lookup).
>>>>>>>>>>>>>
>>>>>>>>>>>>> 8. Solution (C) is nice since it avoids "duplication".
>>>>>>>>>>>>>
>>>>>>>>>>>>> 9. The main drawback of the solution (C) is asymmetry.
>>>>>>>>>>>>>     As it was discussed in the case of RX_TIMESTAMP
>>>>>>>>>>>>>     (if I remember it correctly):
>>>>>>>>>>>>>      - PMD advertises RX_TIMESTAMP offload capability
>>>>>>>>>>>>>      - application enables the offload
>>>>>>>>>>>>>      - PMD registers dynamic field for timestamp
>>>>>>>>>>>>>     Solution (C):
>>>>>>>>>>>>>       - PMD advertises nothing
>>>>>>>>>>>>>       - application uses solution (B) to understand if
>>>>>>>>>>>>>         these features are supported
>>>>>>>>>>>>>       - application registers dynamic field/flag
>>>>>>>>>>>>>       - PMD does lookup and solve the problem
>>>>>>>>>>>>>     The asymmetry could be partially mitigated if RX_TIMESTAMP
>>>>>>>>>>>>>     solution is changed to require an application to register
>>>>>>>>>>>>>     dynamic fields and PMD to do lookup if the offload is
>>>>>>>>>>>>>     enabled. So, the only difference will be in no offload
>>>>>>>>>>>>>     in the case of flow MARK/FLAG and usage of complex logic
>>>>>>>>>>>>>     to understand if it is supported or no.
>>>>>>>>>>>>>     May be it would be really good since it will allow to
>>>>>>>>>>>>>     have dynamic fields registered before mempool population.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 10. Common drawback of solutions (B) and (C) is no granularity.
>>>>>>>>>>>>>      Solution (A) may be per queue while (B) and (C) cannot be
>>>>>>>>>>>>>      per queue. Moreover (C) looks global - for all devices.
>>>>>>>>>>>>>      It could be really painful.
>>>>>>>>>>>>>
>>>>>>>>>>>>> (C) is nice, but I still vote for simplicity and
>>>>>>>>>>>>> granularity of (A).
>>>>>>>>>>>> I vote for clear separation of application needs and PMD support,
>>>>>>>>>>>> by using the method C (dynamic fields).
>>>>>>>>>>>> I agree timestamp must use the same path.
>>>>>>>>>>>> I agree it's complicate because we don't know in advance whether
>>>>>>>>>>>> a flow rule will be accepted, but that's the reality, config is complex.
>>>>>>>>>>> Do you think that global nature of the (C) is acceptable?
>>>>>>>>>> That's a good question.
>>>>>>>>>> Maybe the feature request should be per port.
>>>>>>>>>> In this case, we are back to solution A with a flag per port?
>>>>>>>>> Offloads are natively per-queue as well, so (A) keeps the choice
>>>>>>>>> between per-port vs per-queue to PMDs as usual.
>>>>>>>>>
>>>>>>>>>> Note that A and C will not guarantee that the offload will be possible.
>>>>>>>>> Yes, definitely.
>>>>>>>>>
>>>>>>>>>> We need B (flow rule validation) anyway.
>>>>>>>>> Strictly speaking (B) (checking flow rules before device
>>>>>>>>> startup) is required if an application can predict flow
>>>>>>>>> rules and wants to ensure that MARK offload will be usable.
>>>>>>>>> Otherwise, it may be skipped.
>>>>>>>> No no, I mean flow rule validation MUST be used anyway
>>>>>>>> during the runtime before applying a rule.
>>>>>>>> I agree it is hard to predict. I speak only about real rules.
>>>>>>> OK, I see. Of course, flow rule validation is required at runtime.
>>>>>>> I was rather concentrated on the stated problem solutions.
>>>>>>>
>>>>>>>>>> It seems A, B, C are not alternatives but all required
>>>>>>>>>> as pieces of a puzzle...
>>>>>>>>> Unfortunately true in the most complex case.
>>>>>>>>> Right now it will be A with B if required as explained above.
>>>>>>>>> C will come a bit later when the field migrates to dynamic.
>>>>>>>>>
>>>>>>>>> May be it is even better if application registers dynamic
>>>>>>>>> fields before an attempt to enable offload to be sure that
>>>>>>>>> it will not fail because of impossibility to register
>>>>>>>>> dynamic field (lack of space). I'm not sure, but it is not
>>>>>>>>> not that important.
>>>>>>>> Yes of course, lack of mbuf space is another reason for
>>>>>>>> disabling the feature.
>>>>>>>>
>>>>>>>>> If we finally go way A, should we add offloads for META back?
>>>>>>>>> I guess separate Rx and Tx are required.
>>>>>>>> I would prefer to add it as dynamic flags.
>>>>>>>> Why rushing on a very temporary solution while it is not a new issue?
>>>>>>> Basically it means that we go just (B)+(C) in the case of META.
>>>>>>> I have no strong opinion but thought that it could be better to
>>>>>>> align the solution. Of course, we can wait with it. As I understand
>>>>>>> META is an experimental feature.
>>>>>> Yes it is experimental and I think it is too late to align now.
>>>>>>
>>>>>> Anyway, we will probably to discuss again these offloads TAG/MARK/META,
>>>>>> as requested by several people.
>>>>>>
>>>>> The series implements (A) to help to solve the problem described above.
>>>>> What is the fate of the series in v19.11 in accordance with the
>>>>> discussion?
>>>> I am against adding anything related to a feature union'ed in mbuf.
>>>> The feature must move to dynamic field first.
>>>>
>>>> In addition, such capability is very weak.
>>>> I am not sure it is a good idea to have some weak capabilities,
>>>> meaning a feature could be available but not in all cases.
>>>> I think we should discuss more generally how we want to handle
>>>> the rte_flow capabilities conveniently and reliably.
>>>
>>> It is really unexpected outcome from the above discussion.
>>
>> I'm sorry, I thought I was clear in my request to switch to dynamic first.
>>
>>
>>> It is just possibility to deliver and handle marks on datapath and
>>> request to have it. It says almost nothing about rte_flow rules
>>> supported etc. I'll be happy to take part in the discussion.
>>>
>>>> So regarding 19.11, as this feature is not new, it can wait 20.02.
>>>
>>> OK, it is not critical for me, so I don't mind, however, I've seen
>>> patches which try to use it [1] except net/octeontx2 in the second
>>> patch of the series.
>>>
>>> [1] https://patches.dpdk.org/patch/62415/
>>
> 
> Sorry, I have to resurrect this old (long) discussion because the patches are
> still active in the patchwork [1] and the deprecation notice is still there [2].
> 
> Andrew has a good summary in the thread [3], after a year nothing seems changed.
> 
> 
> Pavan, Thomas, Andrew, Ori,
> 
> What is our plan with this series, lets try to have a conclusion.
> 
> 
> 
> 
> [1]
> https://patches.dpdk.org/user/todo/dpdk/?series=7076
> 
> [2]
> http://lxr.dpdk.org/dpdk/v20.05/source/doc/guides/rel_notes/deprecation.rst#L88
> 
> [3]
> http://inbox.dpdk.org/dev/f170105b-9c60-1b04-cb18-52e0951ddcdb@solarflare.com/
> 
> 

I re-read the thread, will try to have a little movement while we are in the new 
release cycle, if there is no update I am planning to reject the patches.

There seems two problems:

P1) Application will keep trying to program NIC for MARK action for each flow, 
since application doesn't know if next one will succeed or not.
If only there would be a way to find out that NIC/PMD doesn't support the MARK 
action at all, this could save application to keep trying.

P2) PMD can make better internal choices if it gets more hint from application 
about MARK action may be used or not.
Application at least may say it won't use the MARK flow action at all.


This patch uses offload flags infrastructure to solve above two problems, 
solution (A) in Andrew's summary.

Although it may solve the issues, there are questions/concerns around using this 
additional flag to control flow API, I also agree it may be confusing in the 
design level although practically using flags can be simple.
And this is not generic solution, what happen with META action question is 
already hanging on in the thread, more flags? How many more can we add?

And also there is option an to use dynamic mbuf flags to detect the capability, 
solution (C) in Andrew's summary, again it may solve the problem but it looks 
again a workaround to solve same flow API design restriction, and this one is 
not as simple as (A).

Overall the discussion seems going on circles without an agreed on decision.



What about trying to solve this with flow API return values,

If a flow rule is not supported at all by the NIC/PMD, it may return 
'-ENO_WAY_JOSE', and application knows it can't be used at all, this may solve 
the (P1) above.

And if a flow rule can be supported for the given pattern, but it is not 
supported right now because current configuration or resourcing restrictions 
doesn't allow creating rule, a special error type can be returned with a 
descriptive error log for application to response:
-ECONFLICT, "Can't enable rule A when rule B is enabled"
-EDATAPATH, "Can't enable this rule when vector datapath is used"
-ERESOURCE, "Can't enable more than 3 rules"
This may solve the (P2) partially.

I am not sure about second part, but at least first part shouldn't be too hard 
to implement, and it is a generic solution, what do you think?