Re: [PATCH v4 0/4] Introduce Topology NUMA grouping for lcores

["Mattias Rönnblom" <hofors@lysator.liu.se>]

On 2025-03-04 11:08, Morten Brørup wrote:
>> From: Varghese, Vipin [mailto:Vipin.Varghese@amd.com]
>> Sent: Monday, 3 March 2025 10.06
>>
>> [Public]
>>
>> Hi Morten,
>>
>> snipped
>>
>>>
>>>> From: Thomas Monjalon [mailto:thomas@monjalon.net]
>>>> Sent: Thursday, 13 February 2025 09.34
>>>>
>>>> 13/02/2025 04:09, Varghese, Vipin:
>>>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>>>
>>>>> Adding Thomas and Ajit to the loop.
>>>>>
>>>>> Hi Ajit, we have been using the patch series for identifying the
>>>> topology and getting l3 cache id for populating the steering tag
>> for
>>>> Device Specific Model & MSI-x driven af-xdp for the experimental
>> STAG
>>>> firmware on Thor.
>>>
>>> Excellent. A real life example use case helps the review process a
>> lot!
>>
>> Steering tag is one of the examples or uses, as shared in the current
>> patch series  we make use of these for other examples too.
>> Eventdev, pkt-distributor and graph nodes are also in works to exploit
>> L2|L3 cache local coherency too.
>>
>>>
>>>>>
>>>>> Hence current use of topology library helps in 1. workload
>> placement
>>>>> in same Cache or IO domain 2. populating id for MSIx or Device
>>>>> specific model for steering tags.
>>>>>
>>>>> Thomas and Ajith can we get some help to get this mainline too?
>>>>
>>>> Yes, sorry the review discussions did not start.
>>>> It has been forgotten.
>>>
>>> I think the topology/domain API in the EAL should be co-designed with
>> the steering
>>> tag API in the ethdev library, so the design can be
>> reviewed/discussed in its entirety.
>>
>> As shared in the discussion, we have been exploring simplified approach
>> for steering tags, namely
>>
>> 1. pci-dev args (crude way)
>> 2. flow api for RX (experimental way)
>>
>> Based on the platform (in case of AMD EPYC, these are translated to `L3
>> id + 1`)
>>
>> We do agree rte_ethdev library can use topology API. Current topology
>> API are designed to be made independent from steering tags, as other
>> examples do make use of the same.
>>
>>>
>>> To help the review discussion, please consider describing the
>> following:
>>> Which APIs are for slow path, and which are for fast path?
>>> Which APIs are "must have", i.e. core to making it work at all, and
>> which APIs are
>>> "nice to have", i.e. support APIs to ease use of the new features?
>>
>> Yes, will try to do the same in updated version. For Slow and Fast path
>> API I might need some help, as I was under the impression current
>> behavior is same rte_lcore (invoked at setup and before remote launch).
>> But will check again.
> 
> Probably they are all used for configuration only, and thus all slow path; but if there are any fast path APIs, they should be highlighted as such.
> 

Preferably, software work schedulers like DSW should be able to read 
topology information during run-time/steady-state operation. If topology 
APIs are slow or non-MT-safe, they will need to build up their own data 
structures for such information (which is not crazy idea, but leads to 
duplication).

I didn't follow the hwloc discussions, so I may lack some context for 
this discussion.

>>
>>>
>>> I haven't looked at the hwloc library's API; but I guess these new
>> EAL functions are
>>> closely related. Is it a thin wrapper around the hwloc library, or is
>> it very different?
>> This is very thin wrapper on top of hwloc library only. But with DPDK
>> RTE_MAX_LCORE & RTE_NUMA boundary check and population.
> 
> OK. The hwloc library is portable across Linux, BSD and Windows, which is great!
> 
> Please also describe the benefits of using this DPDK library, compared to directly using the hwloc library.
>