Re: [PATCH] ethdev: add template table resize API

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 1/30/2024 12:46 PM, Etelson, Gregory wrote:
> Hello Ferruh,
> 
>>
>> If a multi-threaded application can add new and updated old
>> simultaneously, this should be done via monolithic API, like:
>> {
>>  lock
>>    resize
>>  unlock
>>  for each flow
>>    lock
>>    update
>>    unlock
>> }
>>
> 
> The flow template API was designed for performance.
> Application that implements the flow template API expects high
> flows insertions, updates, and removals rates.
> Locks are necessary for the table resize API.
> 
> During the monolithic resize, application has no control over
> PMD locks. Even if resize and each update operations are
> relatively fast, application should expect table lock collisions
> in rules insertions, deletions and updates for the entire
> resize-and-update-and-update operation.
> 
> With the separate resize API, lock collisions are expected
> during the resize phase only.
> After table resize completed, all flow operations will obtain
> a lock without collision.
> Also, application does not have to update all flows at once.
> Updates can be executed in batches scheduled by application.
> Another use case: application can increase a table,
> offload all new flows and run updates while hardware handles
> network traffic according to the new flows scheme.
> 
>> Perhaps questions is, is there a usecase that user does the resize but
>> doesn't want to update the old flows?
> 
> Please see below.
> 
>>
>>> The rte_flow_template_table_resize_complete was added for PMDs that
>>> cannot differentiate flows created before and after table resize.
>>>
>>
>> Can you please elaborate this?
>>
>> Does it mean old flows and new flows require different handling and PMD
>> doesn't know how to differentiate old and new flows?
>> If so how update() converts old flows, there must be a way for driver to
>> differentiate them for update() to work.
>>
>> Also if resize_complete() NOT called at all, does this mean PMD can't
>> handle any flows anymore as it can't differentiate old and new ones?
>>
> 
> Table resize API do not have any effect on running flows.
> PMD uses the same procedure to create flows before and after table resize.
> All flows instantiated from the same type before and after table resize.
> 
> Flow update that follows table resize manages PMD flow object location.
> In MLX5 PMD, flow update moves an object that references a flow
> from old table to a new table.
> After all flows were moved to a new table, PMD has no need for the old
> table and it can be released.
> 
> Since flow update manages PMD memory only,
> application can ignore the update operation
> if it does not care about effective memory management.
> 
> PMD can release the old flow table after all flows it referenced were
> moved to a new table only.
> Event that notifies PMD about empty old table can be ether internal
> or external.
> Internal event assumes PMD ability to track flows in a table.
> External event in form of application call is more general approach.
> 
> Application must call resize_complete after it moved all flows
> to the new table. That call notifies PMD that it safe to release
> resources related to old table.
> If application did not update flows it must not call resize_complte.
> 
> Application can create new flows after table resize
> regardless if it managed PMD memory with update and
> resize_complete calls or not.
>

Thanks for clarification.

So, by design, driver will keep the old table when it is resized.
- Can this have a performance impact, like when rules
updated/removed/inserted driver will need to look more tables?
- Or can this cause additional capacity complexity, like total number of
rules will be sum of rules in all tables, but new rules only can be
added to latest table, so number of rules can be more than size of
latest table.
- Or user can add more flows after resize() and this may not leave
enough room to update old rules to new table, what is expected behavior
for this case?
- Or if user did not updated rules at all after resize(), after each
rule deletion driver won't need to check if old table emptied and needs
to be freed?
- Or can user call resize() API multiple times, causing driver to
maintain multiple tables? How much memory overhead this may bring?


'rte_flow_async_update_resized()' API is called per flow, won't this
force application to trace which flows are created in new table and
which are in old table, so pushing additional work to application.

Or what will happen if update() fails in the middle of update, should
user retry, should PMD restore back the moved rules?



I understood the logic behind the dividing responsibility to multiple
APIs, and it makes sense, but it brings above complexities, and more
work to application.
Can it be possible to have monolithic API but only resize() part of it
is blocking and update() part and later remove table part done
asynchronously?


I will also put more comment on the patch based on latest understanding.