Re: [RFC 0/2] introduce LLC aware functions

[Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>]

> On Sep 11, 2024, at 10:55 AM, Mattias Rönnblom <hofors@lysator.liu.se> wrote:
> 
> On 2024-09-11 05:26, Varghese, Vipin wrote:
>> [AMD Official Use Only - AMD Internal Distribution Only]
>> <snipped>
>>> 
>>> On 2024-09-09 16:22, Varghese, Vipin wrote:
>>>> [AMD Official Use Only - AMD Internal Distribution Only]
>>>> 
>>>> <snipped>
>>>> 
>>>>>> <snipped>
>>>>>> 
>>>>>> Thank you Mattias for the comments and question, please let me try
>>>>>> to explain the same below
>>>>>> 
>>>>>>> We shouldn't have a separate CPU/cache hierarchy API instead?
>>>>>> 
>>>>>> Based on the intention to bring in CPU lcores which share same L3
>>>>>> (for better cache hits and less noisy neighbor) current API focuses
>>>>>> on using
>>>>>> 
>>>>>> Last Level Cache. But if the suggestion is `there are SoC where L2
>>>>>> cache are also shared, and the new API should be provisioned`, I am
>>>>>> also
>>>>>> 
>>>>>> comfortable with the thought.
>>>>>> 
>>>>> 
>>>>> Rather than some AMD special case API hacked into <rte_lcore.h>, I
>>>>> think we are better off with no DPDK API at all for this kind of functionality.
>>>> 
>>>> Hi Mattias, as shared in the earlier email thread, this is not a AMD special
>>> case at all. Let me try to explain this one more time. One of techniques used to
>>> increase cores cost effective way to go for tiles of compute complexes.
>>>> This introduces a bunch of cores in sharing same Last Level Cache (namely
>>> L2, L3 or even L4) depending upon cache topology architecture.
>>>> 
>>>> The API suggested in RFC is to help end users to selectively use cores under
>>> same Last Level Cache Hierarchy as advertised by OS (irrespective of the BIOS
>>> settings used). This is useful in both bare-metal and container environment.
>>>> 
>>> 
>>> I'm pretty familiar with AMD CPUs and the use of tiles (including the
>>> challenges these kinds of non-uniformities pose for work scheduling).
>>> 
>>> To maximize performance, caring about core<->LLC relationship may well not
>>> be enough, and more HT/core/cache/memory topology information is
>>> required. That's what I meant by special case. A proper API should allow
>>> access to information about which lcores are SMT siblings, cores on the same
>>> L2, and cores on the same L3, to name a few things. Probably you want to fit
>>> NUMA into the same API as well, although that is available already in
>>> <rte_lcore.h>.
>> Thank you Mattias for the information, as shared by in the reply with Anatoly we want expose a new API `rte_get_next_lcore_ex` which intakes a extra argument `u32 flags`.
>> The flags can be RTE_GET_LCORE_L1 (SMT), RTE_GET_LCORE_L2, RTE_GET_LCORE_L3, RTE_GET_LCORE_BOOST_ENABLED, RTE_GET_LCORE_BOOST_DISABLED.
> 
> Wouldn't using that API be pretty awkward to use?
> 
> I mean, what you have is a topology, with nodes of different types and with different properties, and you want to present it to the user.
> 
> In a sense, it's similar to XCM and DOM versus SAX. The above is SAX-style, and what I have in mind is something DOM-like.
> 
> What use case do you have in mind? What's on top of my list is a scenario where a DPDK app gets a bunch of cores (e.g., -l <cores>) and tries to figure out how best make use of them. It's not going to "skip" (ignore, leave unused) SMT siblings, or skip non-boosted cores, it would just try to be clever in regards to which cores to use for what purpose.
> 
>> This is AMD EPYC SoC agnostic and trying to address for all generic cases.
>> Please do let us know if we (Ferruh & myself) can sync up via call?
> 
> Sure, I can do that.
> 
Can this be opened to the rest of the community? This is a common problem that needs to be solved for multiple architectures. I would be interested in attending.

>>> 
>>> One can have a look at how scheduling domains work in the Linux kernel.
>>> They model this kind of thing.
>>> 
>>>> As shared in response for cover letter +1 to expand it to more than
>>>> just LLC cores. We have also confirmed the same to
>>>> https://patchwork.dpdk.org/project/dpdk/cover/20240827151014.201-
>>> 1-vip
>>>> in.varghese@amd.com/
>>>> 
>>>>> 
>>>>> A DPDK CPU/memory hierarchy topology API very much makes sense, but
>>>>> it should be reasonably generic and complete from the start.
>>>>> 
>>>>>>> 
>>>>>>> Could potentially be built on the 'hwloc' library.
>>>>>> 
>>>>>> There are 3 reason on AMD SoC we did not explore this path, reasons
>>>>>> are
>>>>>> 
>>>>>> 1. depending n hwloc version and kernel version certain SoC
>>>>>> hierarchies are not available
>>>>>> 
>>>>>> 2. CPU NUMA and IO (memory & PCIe) NUMA are independent on AMD
>>>>> Epyc Soc.
>>>>>> 
>>>>>> 3. adds the extra dependency layer of library layer to be made
>>>>>> available to work.
>>>>>> 
>>>>>> 
>>>>>> hence we have tried to use Linux Documented generic layer of `sysfs
>>>>>> CPU cache`.
>>>>>> 
>>>>>> I will try to explore more on hwloc and check if other libraries
>>>>>> within DPDK leverages the same.
>>>>>> 
>>>>>>> 
>>>>>>> I much agree cache/core topology may be of interest of the
>>>>>>> application (or a work scheduler, like a DPDK event device), but
>>>>>>> it's not limited to LLC. It may well be worthwhile to care about
>>>>>>> which cores shares L2 cache, for example. Not sure the
>>>>>>> RTE_LCORE_FOREACH_*
>>>>> approach scales.
>>>>>> 
>>>>>> yes, totally understand as some SoC, multiple lcores shares same L2 cache.
>>>>>> 
>>>>>> 
>>>>>> Can we rework the API to be rte_get_cache_<function> where user
>>>>>> argument is desired lcore index.
>>>>>> 
>>>>>> 1. index-1: SMT threads
>>>>>> 
>>>>>> 2. index-2: threads sharing same L2 cache
>>>>>> 
>>>>>> 3. index-3: threads sharing same L3 cache
>>>>>> 
>>>>>> 4. index-MAX: identify the threads sharing last level cache.
>>>>>> 
>>>>>>> 
>>>>>>>> < Function: Purpose >
>>>>>>>> ---------------------
>>>>>>>>    - rte_get_llc_first_lcores: Retrieves all the first lcores in
>>>>>>>> the shared LLC.
>>>>>>>>    - rte_get_llc_lcore: Retrieves all lcores that share the LLC.
>>>>>>>>    - rte_get_llc_n_lcore: Retrieves the first n or skips the first
>>>>>>>> n lcores in the shared LLC.
>>>>>>>> 
>>>>>>>> < MACRO: Purpose >
>>>>>>>> ------------------
>>>>>>>> RTE_LCORE_FOREACH_LLC_FIRST: iterates through all first lcore from
>>>>>>>> each LLC.
>>>>>>>> RTE_LCORE_FOREACH_LLC_FIRST_WORKER: iterates through all first
>>>>>>>> worker lcore from each LLC.
>>>>>>>> RTE_LCORE_FOREACH_LLC_WORKER: iterates lcores from LLC based on
>>>>> hint
>>>>>>>> (lcore id).
>>>>>>>> RTE_LCORE_FOREACH_LLC_SKIP_FIRST_WORKER: iterates lcores from
>>> LLC
>>>>>>>> while skipping first worker.
>>>>>>>> RTE_LCORE_FOREACH_LLC_FIRST_N_WORKER: iterates through `n`
>>> lcores
>>>>>>>> from each LLC.
>>>>>>>> RTE_LCORE_FOREACH_LLC_SKIP_N_WORKER: skip first `n` lcores, then
>>>>>>>> iterates through reaming lcores in each LLC.
>>>>>>>> 
>>>>>> While the MACRO are simple wrapper invoking appropriate API. can
>>>>>> this be worked out in this fashion?
>>>>>> 
>>>>>> <snipped>