Re: [EXT] Re: [PATCH v5 2/2] app/testpmd: add command to process Rx metadata negotiation

[Thomas Monjalon <thomas@monjalon.net>]

OK we are progressing on this topic :)

02/02/2023 11:48, Ivan Malov:
> I apologise, there was a typo in the previous mail: "PMD does
> not need this API". Should read as "TESTPMD does not need it".

testpmd needs all ethdev API,
because its purpose is to test the whole all ethdev API.

Maybe the use of this function is misplaced in testpmd.
It should be a specific command.
By the way, what is the driver default if negotiation is not done?


> On Thu, 2 Feb 2023, Ivan Malov wrote:
> > On Thu, 2 Feb 2023, Ferruh Yigit wrote:
> >
> >> On 2/2/2023 8:50 AM, Ivan Malov wrote:
> >>> On Thu, 2 Feb 2023, Ferruh Yigit wrote:
> >>> 
> >>>> On 2/1/2023 3:22 PM, Jerin Jacob wrote:
> >>>>> On Wed, Feb 1, 2023 at 8:20 PM Ferruh Yigit <ferruh.yigit@amd.com>
> >>>>> wrote:
> >>>>>> 
> >>>>>> On 2/1/2023 1:48 PM, Jerin Jacob wrote:
> >>>>>>> On Wed, Feb 1, 2023 at 5:06 PM Ferruh Yigit <ferruh.yigit@amd.com>
> >>>>>>> wrote:
> >>>>>>>> 
> >>>>>>>> On 2/1/2023 11:15 AM, Jerin Jacob wrote:
> >>>>>>>>> On Wed, Feb 1, 2023 at 4:35 PM Thomas Monjalon
> >>>>>>>>> <thomas@monjalon.net> wrote:
> >>>>>>>>>> 
> >>>>>>>>>> 01/02/2023 11:58, Andrew Rybchenko:
> >>>>>>>>>>> On 2/1/23 13:48, Jerin Jacob wrote:
> >>>>>>>>>>>> On Wed, Feb 1, 2023 at 2:59 PM Andrew Rybchenko
> >>>>>>>>>>>> <andrew.rybchenko@oktetlabs.ru> wrote:
> >>>>>>>>>>>>> Frankly speaking I don't understand why default value is so
> >>>>>>>>>>>>> important if we have a way to change it. Reasons should be
> >>>>>>>>>>>>> really strong to change existing defaults.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> The only reason is, typically testpmd will be used performance
> >>>>>>>>>>>> benchmarking as an industry standard. It is difficult to
> >>>>>>>>>>>> tell/educate
> >>>>>>>>>>>> the QA or customers
> >>>>>>>>>>>> that, "BTW if you need to get better performance add more flag to
> >>>>>>>>>>>> testpmd command line".
> >>>>>>>>>> 
> >>>>>>>>>> I disagree.
> >>>>>>>>>> When you do performance benchmark, you tune settings accordingly.
> >>>>>>>>> 
> >>>>>>>>> IMO, We tune the system resources like queue depth not the disabling
> >>>>>>>>> features for raw performance.
> >>>>>>>>> queue depth etc people know to tune so it is obvious. What is not
> >>>>>>>>> obvious is, testpmd only
> >>>>>>>>> negotiated some features by default.I am not using that feature,
> >>>>>>>>> hence
> >>>>>>>>> I need to explicitly
> >>>>>>>>> disable it.
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>>> When 'rte_eth_rx_metadata_negotiate()' API is NOT used at all, and I
> >>>>>>>> believe that is the case for almost all applications since API is a
> >>>>>>>> relatively new one, PMD default behavior should be to enable Rx
> >>>>>>>> metadata
> >>>>>>>> flow rules, in case user requests them later.
> >>>>>>>> 
> >>>>>>>> So, enabling all in application is same with not calling the API
> >>>>>>>> at all.
> >>>>>>>> 
> >>>>>>>> In this perspective, disabling Rx metadata is additional
> >>>>>>>> optimization/tuning that application can do if it is sure that Rx
> >>>>>>>> metadata flow rules won't be used at all.
> >>>>>>>> And API is more meaningful when it is used to disable Rx metadata.
> >>>>>>>> 
> >>>>>>>> I think it is reasonable to enable all Rx metadata by default in
> >>>>>>>> testpmd
> >>>>>>>> with a capability to disable it when wanted.
> >>>>>>>> 
> >>>>>>>> OR
> >>>>>>>> 
> >>>>>>>> May be we don't call 'rte_eth_rx_metadata_negotiate()' API by
> >>>>>>>> default in
> >>>>>>>> testpmd, it is only called when it is requested explicitly from user,
> >>>>>>>> enable or disable.
> >>>>>>> 
> >>>>>>> Second option looks good to me.
> >>>>>>> When
> >>>>>>> 1) user request for action which is needed negotiate(),
> >>>>>>> AND
> >>>>>>> 2) rte_eth_rx_metadata_negotiate() != ENOSUP
> >>>>>>> then, testpmd print a warning that need to enable
> >>>>>>> rte_eth_rx_metadata_negotiate().
> >>>>>>> 
> >>>>>> 
> >>>>>> We are not suggesting same thing.
> >>>>>> 
> >>>>>> What you described above assumes PMD disabled Rx metadata flow rule
> >>>>>> support by default, and it needs to be enabled explicitly by
> >>>>>> 'rte_eth_rx_metadata_negotiate()' API. This API becomes mandatory for
> >>>>>> functionality.
> >>>>>> 
> >>>>>> As far as I understand PMD wants to disable this flow rule by default
> >>>>>> because of performance concerns. But this creates inconsistency between
> >>>>>> PMDs, because rest of them will enable this flow rule by default (if it
> >>>>>> is supported) and be ready to use it when proper flow rule created.
> >>>>>> 
> >>>>>> With this approach some PMDs will need
> >>>>>> 'rte_eth_rx_metadata_negotiate()'
> >>>>>> to enable Rx metadata flow rules, some won't. This can be confusing for
> >>>>>> applications that *some* PMDs require double enabling with specific API
> >>>>>> call.
> >>>>>> 
> >>>>>> 
> >>>>>> Instead what I was trying to suggest is reverse,
> >>>>>> all PMDs enable the Rx metadata flow rule by default, and don't require
> >>>>>> double enabling.
> >>>>>> But if application knows that it won't use Rx metadata flow rule, it
> >>>>>> can
> >>>>>> disable it to optimize the performance.
> >>>>>> This makes 'rte_eth_rx_metadata_negotiate()' functionally optional, and
> >>>>>> for testpmd context it can be called via a command on demand by user
> >>>>>> for
> >>>>>> optimization purpose.
> >>>>> 
> >>>>> This won't solve concern I have outlined earlier[1].
> >>>>> 
> >>>> 
> >>>> Yes, it won't.
> >>>> 
> >>>>> I think, The part of the problem there is no enough adaption of
> >>>>> rte_eth_rx_metadata_negotiate(),
> >>>>> 
> >>>>> The view is total different from PMD maintainer PoV vs testpmd
> >>>>> application PoV.
> >>>>> 
> >>>> 
> >>>> Agree,
> >>>> and I assume it is different for user application too, which may
> >>>> prioritize consistency and portability.
> >>>> 
> >>>> Overall, I am not fan of the 'rte_eth_rx_metadata_negotiate()' API, I
> >>>> think it is confusing.
> >>> 
> >>> Forgive me, in which way is it confusing?
> >>> 
> >> 
> >> All other flow rules enabled by creating flow rule, but this one
> >
> > Hold on.. Did you just say "flow rules"? But this API is *not* about
> > flow rules. I suggest you please re-read description of the API.
> > It reads: "Negotiate the NIC's ability to deliver specific
> > kinds of metadata to the PMD". Nothing about flows there.
> >
> > Why is it decoupled from flow library this way? Because there is
> > a drastic difference between generating and plumbing metadata
> > INSIDE the NIC's flow engine, on the one hand, and delivering
> > these data from the NIC to the host driver, on the other.
> >
> > Let me explain. Say, one creates the following flow rules:
> >
> > (a) flow create 0 transfer group 0 pattern [A] / end \
> >    actions mark id XXX jump group 1 / end
> > (b) flow create 0 transfer group 0 pattern [B] / end \
> >    actions mark id YYY jump group 1 / end
> >
> > (c) flow create 0 transfer group 1 pattern mark id is XXX / end \
> >    actions represented_port ethdev_id 1 / end
> > (d) flow create 0 transfer group 1 pattern mark id is YYY / end \
> >    actions drop / end
> >
> > In this example, metadata of type "MARK" is used to partition
> > flow group (table) 1 into multiple lookup sections. So the
> > mark value is GENERATED by the flow engine and then it is
> > CONSUMED by this very flow engine. The application may
> > NOT necessarily want to receive the mark with mbufs...
> >
> > And it is only when the application wants these metadata
> > DELIVERED to it that it has to call the negotiate API.
> >
> > The short of it, nothing prevents the driver from accepting
> > flow create requests that leverage some meta items/actions.
> > Drivers do not need the negotiate API to *configure flow*.
> > They only need this API in order to let the application
> > choose whether metadata will be DELIVERED (!) or not.
> >
> >> requires an API to enable it, so I believe it is inconsistent in that way.
> >
> > Please see above. Everything is consistent as *flow library*
> > and *negotiate metadata delivery* API are totally decoupled.
> >
> >> 
> >> From application perspective, assume that it doesn't know NIC details,
> >> should it call this API or not? Without API call should application
> >> assume Rx metadata flow rules are enabled or disabled?
> >
> > Frankly, applications like testpmd need never call this API.
> > Simply because seeing, for example, MARK values in mbufs is
> > useless to it. This API is needed by other applications.
> > For example, OvS has partial MARK+RSS offload. It adds
> > flows that distribute packets across multiple queues
> > and set some MARK values for them. OvS is interested
> > in getting this values with mbufs since they affect
> > further lookups in software... So, OvS, knowing it
> > wants these metadata DELIVERED (!), should invoke
> > this metadata negotiate API to ensure that.
> >
> >> 
> >> 
> >> As I understand intention is to get hint from application if it will
> >> require Rx metadata flow rules so that PMD can optimize better, but if
> >
> > No, nothing about flow rules. Just delivery of metadata with mbufs.
> >
> >> PMD doesn't enable Rx metadata flow rules when this API call is missing
> >
> > Again, PMD shall not make decisions on whether to enable or disable
> > support for some FLOW primitives based on interactions via this API.
> > This API exists solely to let PMD configure delivery of metadata,
> > i.e. not the way it is generated in the first instance.
> >
> >> than it becomes a mandatory API to configure the device. But I think it
> >> should be optional for optimization.
> >> 
> >> Also if application not sure to use this flow rule or not, it will by
> >> default enable all in any case, this will reduce the benefit. As done in
> >> testpmd.
> >
> > Please see above. PMD does not need this API, I take it.
> >
> >> 
> >> 
> >> API works in both ways, it request to enable some Rx metadata flow rule,
> >> but based on what PMD returns application can know what device supports,
> >> this also inconsistent with how other flow rules work, we don't have API
> >> to get capability but detect them via flow create/validate.
> >> Can there be a case API returns a flow rule negotiated, but it fails
> >> when tried to create the flow rule, isn't this confusing for application
> >> 
> >> 
> >> I think if we continue this approach there can be multiple enable and
> >> capability learning APIs for various flow rules or flow rule groups, and
> >> this can make flow API much more harder to use for applications.
> >
> > See my explanations above. This API is not about flows. Period.
> >
> >> 
> >> 
> >> 
> >>>> 
> >>>>> Just to avoid back and forth. We will call off this patch and remove
> >>>>> rte_eth_rx_metadata_negotiate()
> >>>>> PMD callback from cnxk driver. Keep it as old behavior, so we don't
> >>>>> need to care
> >>>>> about rte_eth_rx_metadata_negotiate().
> >>>>> 
> >>>> 
> >>>> When you remove 'rx_metadata_negotiate' callback, what will be the PMD
> >>>> behavior? I assume PMD will do the required preparations as if all Rx
> >>>> metadata is enabled.
> >>>> And what is the performance impact, is removing callback improve the
> >>>> performance?
> >>>> 
> >>>> 
> >>>>> [1]
> >>>>> The only reason is, typically testpmd will be used performance
> >>>>> benchmarking as an industry standard. It is difficult to tell/educate
> >>>>> the QA or customers
> >>>>> that, "BTW if you need to get better performance add more flag to
> >>>>> testpmd command line".
> >>>>> To make that worst, only some PMD needs to give the additional
> >>>>> parameter to get better number.
> >>>>> And also, testpmd usage will be treated as application modeling.
> >>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>>> 
> >>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>>>> To make that worst, only some PMD needs to give the additional
> >>>>>>>>>>>> parameter to get better number.
> >>>>>>>>>>>> And also, testpmd usage will be treated as application modeling.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Since this feature only used on sfc and cnxk driver, What is the
> >>>>>>>>>>>> situation with sfc driver?
> >>>>>>>>>>>> Keeping it as negotiated and not use the feature, will impact
> >>>>>>>>>>>> the per
> >>>>>>>>>>>> core performance of sfc or
> >>>>>>>>>>>> is it just PCI bandwidth thing which really dont show any
> >>>>>>>>>>>> difference in testpmd?
> >>>>>>>>>>> 
> >>>>>>>>>>> Yes, sfc could run faster if no Rx metadata are negotiated. So,
> >>>>>>>>>>> it is better to negotiate nothing by default. But it is always
> >>>>>>>>>>> painful to change defaults. You need to explain that now you
> >>>>>>>>>>> need to negotiate Rx metadata to use mark, flag and tunnel
> >>>>>>>>>>> offloads.
> >>>>>>>>>>> Yes, it will be required on sfc and cnxk only.
> >>>>>>>>>>> As an sfc maintainer I don't mind to change testpmd defaults.
> >>>>>>>>>> 
> >>>>>>>>>> If we change testpmd defaults to "do nothing",
> >>>>>>>>>> then we should disable MBUF_FAST_FREE as well.
> >>>>>>>>> 
> >>>>>>>>> if you see MBUF_FAST_FREE, it does nothing. Actually,
> >>>>>>>>> !MBUF_FAST_FREE is doing more work.
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>> 
> >>>>>> 
> >>>> 
> >>>> 
> >> 
> >> 
> >
>