Re: [dpdk-dev] [PATCH 2/5] ethdev: add capability to keep shared objects on restart

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 10/18/2021 12:13 PM, Dmitry Kozlyuk wrote:
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> Sent: 18 октября 2021 г. 11:42
>> To: Dmitry Kozlyuk <dkozlyuk@nvidia.com>; dev@dpdk.org; Andrew Rybchenko
>> <andrew.rybchenko@oktetlabs.ru>; Ori Kam <orika@nvidia.com>; Raslan
>> Darawsheh <rasland@nvidia.com>; NBU-Contact-Thomas Monjalon
>> <thomas@monjalon.net>
>> Cc: Qi Zhang <qi.z.zhang@intel.com>; jerinj@marvell.com; Maxime Coquelin
>> <maxime.coquelin@redhat.com>
>> Subject: Re: [PATCH 2/5] ethdev: add capability to keep shared objects on
>> restart
>>
>> External email: Use caution opening links or attachments
>>
>>
>> On 10/16/2021 9:32 PM, Dmitry Kozlyuk wrote:
>>>> -----Original Message-----
>>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> Sent: 15 октября 2021 г. 19:27
>>>> To: Dmitry Kozlyuk <dkozlyuk@nvidia.com>; dev@dpdk.org; Andrew
>> Rybchenko
>>>> <andrew.rybchenko@oktetlabs.ru>; Ori Kam <orika@nvidia.com>; Raslan
>>>> Darawsheh <rasland@nvidia.com>
>>>> Cc: NBU-Contact-Thomas Monjalon <thomas@monjalon.net>; Qi Zhang
>>>> <qi.z.zhang@intel.com>; jerinj@marvell.com; Maxime Coquelin
>>>> <maxime.coquelin@redhat.com>
>>>> Subject: Re: [PATCH 2/5] ethdev: add capability to keep shared objects
>> on
>>>> restart
>>>>
>>>> External email: Use caution opening links or attachments
>>>>
>>>>
>>>> On 10/15/2021 1:35 PM, Dmitry Kozlyuk wrote:
>>>>>> -----Original Message-----
>>>>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>>> [...]
>>>>>>> Introducing UNKNOWN state seems wrong to me.
>>>>>>> What should an application do when it is reported?
>>>>>>> Now there's just no way to learn how the PMD behaves,
>>>>>>> but if it provides a response, it can't be "I don't know what I do".
>>>>>>>
>>>>>>
>>>>>> I agree 'unknown' state is not ideal, but my intentions is prevent
>>>>>> drivers that not implemented this new feature report wrong
>> capability.
>>>>>>
>>>>>> Without capability, application already doesn't know how underlying
>>>>>> PMD behaves, so this is by default 'unknown' state.
>>>>>> I suggest keeping that state until driver explicitly updates its
>> state
>>>>>> to the correct value.
>>>>>
>>>>> My concern is that when all the drivers are changed to report a proper
>>>>> capability, UNKNOWN remains in the API meaning "there's a bug in
>> DPDK".
>>>>>
>>>>
>>>> When all drivers are changed, of course we can remove the 'unknown'
>> flag.
>>>>
>>>>> Instead of UNKNOWN response we can declare that rte_flow_flush()
>>>>> must be called unless the application wants to keep the rules
>>>>> and has made sure it's possible, or the behavior is undefined.
>>>>> (Can be viewed as "UNKNOWN by default", but is simpler.)
>>>>> This way neither UNKNOWN state is needed,
>>>>> nor the bit saying the flow rules are flushed.
>>>>> Here is why, let's consider KEEP and FLUSH combinations:
>>>>>
>>>>> (1) FLUSH=0, KEEP=0 is equivalent to UNKNOWN, i.e. the application
>>>>>                        must explicitly flush the rules itself
>>>>>                        in order to get deterministic behavior.
>>>>> (2) FLUSH=1, KEEP=0 means PMD flushes all rules on the device stop.
>>>>> (3) FLUSH=0, KEEP=1 means PMD can keep at least some rules,
>>>>>                        exact support must be checked with
>>>> rte_flow_create()
>>>>>                        when the device is stopped.
>>>>> (4) FLUSH=1, KEEP=1 is forbidden.
>>>>>
>>>>
>>>> What is 'FLUSH' here? Are you proposing a new capability?
>>>>
>>>>> If the application doesn't need the PMD to keep flow rules,
>>>>> it can as well flush them always before the device stop
>>>>> regardless of whether the driver does it automatically or not.
>>>>> It's even simpler and probably as efficient. Testpmd does this.
>>>>> If the application wants to take advantage of rule-keeping ability,
>>>>> it just tests the KEEP bit. If it is unset that's the previous case,
>>>>> application should call rte_flow_flush() before the device stop to be
>>>> sure.
>>>>> Otherwise, the application can test capability to keep flow rule kinds
>>>>> it is interested in (see my reply to Andrew).
>>>>>
>>>>
>>>> Overall this is an optimization, application can workaround without
>> this
>>>> capability.
>>>>
>>>> If driver doesn't set KEEP capability, it is not clear what does it
>>>> mean, driver doesn't keep rules or driver is not updated yet.
>>>> I suggest to update comment to clarify the meaning of the missing KEEP
>>>> flag.
>>>>
>>>> And unless we have two explicit status flags application can never be
>>>> sure that driver doesn't keep rules after stop. I am don't know if
>>>> application wants to know this.
>>>>
>>>> Other concern is how PMD maintainers will know that there is something
>>>> to update here, I am sure many driver maintainers won't even be aware
>> of
>>>> this, your patch doesn't even cc them. Your approach feels like you are
>>>> thinking only single PMD and ignore rest.
>>>>
>>>> My intention was to have a way to follow drivers that is not updated,
>>>> by marking them with UNKNOWN flag. But this also doesn't work with new
>>>> drivers, they may forget setting capability.
>>>>
>>>>
>>>> What about following:
>>>> 1) Clarify KEEP flag meaning:
>>>> having KEEP: flow rules are kept after stop
>>>> missing KEEP: unknown behavior
>>>>
>>>> 2) Mark all PMDs with useless flag:
>>>> dev_capa &= ~KEEP
>>>> Maintainer can remove or update this later, and we can easily track it.
>>>
>>> Item 1) is almost what I did in v2. The difference (or clarification) is
>> that
>>> if the bit is set, it doesn't mean that all rules are kept.
>>> It allows the PMD to not support keeping some kinds of rules.
>>> Please see the doc update about how the kind is defined
>>> and how the application can test what is unsupported.
>>>
>>> This complication is needed so that if a PMD cannot keep some exotic
>> kind of rules,
>>> it is not forced to remove the capability completely,
>>> blocking optimizations even if the application doesn't use problematic
>> rule kinds.
>>> It makes the capability future-proof.
>>>
>>> The second flag (FLUSH) would not be of much help.
>>> Consider it is not set, but the PMD can keep some kinds of rules.
>>> The application still needs to test all the kinds it needs.
>>> But it needs to do the same if the KEEP bit is set.
>>> Only if it is set the application can skip the tests and
>> rte_flow_flush(),
>>> but these optimizations are small compared to keeping the rules itself.
>>>
>>> Item 2) needs not to be done, because the absence of the bit is *the*
>> useless value:
>>> it means the unspecified same behavior as it is presently.
>>> It is worth noting that currently any application that relies on the PMD
>>> to keep or flush the rules is non-portable, because PMD is allowed to do
>> anything.
>>> To get a reliable behavior application must explicitly clear the rules.
>>>
>>> Regarding you concern about maintainers forgetting to update PMDs,
>>> I think there are better task-tracking tools then constants in the code
>>> (the authors of golang's context.TODO may disagree :)
>>>
>>
>> Hi Dmitry,
>>
>> This is a valid concern, and adding code to the drivers proved that it
>> works.
>>
>> There are multiple authors updating the ethdev layer and expecting PMD
>> maintainers will do required changes. For your case you are updating the
>> PMD
>> you concern but how other PMD maintainers will even be aware that there is
>> something to change in their PMD?
>> By your change you are putting some responsibility to other maintainers,
>> without even cc'ing them. And it is for sure they are not reading all
>> emails
>> in the mail list, they can't.
>>
>> Task-tracking is an option, it the past I tried to upstream some todo doc
>> for PMDs. But I can see the additional maintenance cost to trace features
>> from a central point, comparing the distributing it to PMDS (adding code
>> to PMDs).
>>
>> I think best method is whoever doing the ethdev layer do the relevant
>> change
>> in the PMDs, but has the obvious problem that not able to know enough
>> about
>> the PMDs to update them.
>>
>> We have used the following option, and it worked in the past:
>> - When an ethdev feature require change in PMDs, ehtdev supports both new
>>     and old method
>> - PMDs set a flag by default to request old method, so there is no update
>>     in the PMD default behavior
>> - When PMD does the required changes, it removes the flag
>> - This lets me (or other maintainer), to trace the update status and ping
>>     relevant maintainers
>> - When all PMDs updated, ethdev support for old method removed
>> - This method allows PMD maintainers do the change on their own time
> 
> Hi Ferruh,
> 
> Thanks for sharing the experience.
> You suggest updating PMDs with an explicit reset of this bit,
> despite that it will be zero anyway, to attract maintainers' attention.

ack, but please with a brief comment to clarify intention.

>  From user's perspective it will be all the same: KEEP bit reset,
> not a special value saying the PMD is not updated
> that we would need to deprecate and remove later.

ack, only it needs to be clear for application that PMD not advertising
KEEP flag means behavior is undefined, it does NOT mean PMD flush rules.
Which you already said updated like this in v2, but I am just stressing it.

> If this understanding is correct, then for sure I can add a patch
> updating the relevant PMDs.
>