RE: [RFC 0/2] introduce LLC aware functions

["Varghese, Vipin" <Vipin.Varghese@amd.com>]

[Public]

Snipped

> >
> >
> >    Based on the discussions we agreed on sharing version-2 FRC for
> >    extending API as `rte_get_next_lcore_extnd` with extra argument as
> >    `flags`.
> >
> >    As per my ideation, for the API ` rte_get_next_sibling_core`, the above
> >    API can easily with flag ` RTE_GET_LCORE_L1 (SMT)`. Is this right
> >    understanding?
> >
> >    We can easily have simple MACROs like `RTE_LCORE_FOREACH_L1` which
> >    allows to iterate SMT sibling threads.
> >
> >
>
> This seems like a lot of new macro and API additions! I'd really like to cut that
> back and simplify the amount of new things we are adding to DPDK for this.
I disagree Bruce, as per the new conversation with Anatoly and you it has been shared the new API are
```
1. rte_get_next_lcore_exntd
2. rte_get_next_n_lcore_exntd
```

While I mentioned custom Macro can augment based on typical flag usage similar to ` RTE_LCORE_FOREACH and RTE_LCORE_FOREACH_WORKER` as
```
RTE_LCORE_FOREACH_FLAG
RTE_LCORE_FOREACH_WORKER_FLAG

Or

RTE_LCORE_FOREACH_LLC
RTE_LCORE_FOREACH_WORKER_LLC
```

Please note I have not even shared version-2 of RFC yet.

> I tend to agree with others that external libs would be better for apps that really want to deal with all this.
I have covered why this is not a good idea for Mattias query.

>
> >
> >    >
> >
> >    > Looking logically, I'm not sure about the BOOST_ENABLED and BOOST_DISABLED flags you propose
> >    The idea for the BOOST_ENABLED & BOOST_DISABLED is based on DPDK power library which allows to enable boost.
> >    Allow user to select lcores where BOOST is enabled|disabled using MACRO or API.
May be there is confusion, so let me try to be explicit here. The intention of any `rte_get_next_lcore##`  is fetch lcores.
Hence with new proposed API `rte_get_next_lcore_exntd` with `flag set for Boost` is to fetch lcores where boost is enabled.
There is no intention to enable or disable boost on lcore with `get` API.

> >
> >
> >
> >    - in a system with multiple possible
> >
> >    > standard and boost frequencies what would those correspond to?
> >
> >    I now understand the confusion, apologies for mixing the AMD EPYC SoC
> >    boost with Intel Turbo.
> >
> >
> >
> >    Thank you for pointing out, we will use the terminology `
> >    RTE_GET_LCORE_TURBO`.
> >
> >
>
> That still doesn't clarify it for me. If you start mixing in power management related functions in with topology ones things will turn into a real headache.
Can you please tell me what is not clarified. DPDK lcores as of today has no notion of Cache, Numa, Power, Turbo or any DPDK supported features.
The initial API introduced were to expose lcore sharing the same Last Level Cache. Based on interaction with Anatoly, extending this to support multiple features turned out to be possibility.
Hence, we said we can share v2 for RFC based on this idea.

But if the claim is not to put TURBO I am also ok for this. Let only keep cache and NUMA-IO domain.

> What does boost or turbo correspond to? Is it for cores that have the feature enabled - whether or not it's currently in use - or is it for finding cores that are
> currently boosted?  Do we need additions for cores that are boosted by 100Mhz vs say 300Mhz. What about cores that are in lower frequencies for
> power-saving. Do we add macros for finding those?
Why are we talking about feq-up and freq-down? This was not even discussed in this RFC patch at all.

> >
> >    What's also
> >
> >    > missing is a define for getting actual NUMA siblings i.e. those
> >    sharing common memory but not an L3 or anything else.
> >
> >    This can be extended into `rte_get_next_lcore_extnd` with flag `
> >    RTE_GET_LCORE_NUMA`. This will allow to grab all lcores under the same
> >    sub-memory NUMA as shared by LCORE.
> >
> >    If SMT sibling is enabled and DPDK Lcore mask covers the sibling
> >    threads, then ` RTE_GET_LCORE_NUMA` get all lcore and sibling threads
> >    under same memory NUMA of lcore shared.
> >
> >
>
> Yes. That can work. But it means we are basing the implementation on a fixed idea of what topologies there are or can exist.
> My suggestion below is just to ignore the whole idea of L1 vs L2 vs NUMA - just give the app a way to find it's nearest nodes.
Bruce, for different vendor SoC, the implementation of architecture is different. Let me share what I know
1. using L1, we can fetch SMT threads
2. using L2 we can get certain SoC on Arm, Intel and power PC which is like efficient cores
3. using L3 we can get certain SoC like AMD, AF64x and others which follow chiplet or tile split L3 domain.

>
> After all, the app doesn't want to know the topology just for the sake of knowing it - it wants it to ensure best placement of work on cores! To that end, it just needs to know what cores are near to each other and what are far away.
Exactly, that is why we want to minimize new libraries and limit to format of existing API `rte_get_next_lcore`. The end user need to deploy another library or external library then map to DPDK lcore mapping to identify what is where.
So as end user I prefer simple API which get my work done.

>
> >
> >    >
> >
> >    > My suggestion would be to have the function take just an integer-type
> >    e.g.
> >
> >    > uint16_t parameter which defines the memory/cache hierarchy level to
> >    use, 0
> >
> >    > being lowest, 1 next, and so on. Different systems may have different
> >    numbers
> >
> >    > of cache levels so lets just make it a zero-based index of levels,
> >    rather than
> >
> >    > giving explicit defines (except for memory which should probably
> >    always be
> >
> >    > last). The zero-level will be for "closest neighbour"
> >
> >    Good idea, we did prototype this internally. But issue it will keep on
> >    adding the number of API into lcore library.
> >
> >    To keep the API count less, we are using lcore id as hint to sub-NUMA.
> >
>
> I'm unclear about this keeping the API count down - you are proposing a lot of APIs and macros up above.
No, I am not. I have shared based on the last discussion with Anatoly we will end up with 2 API in lcore only. Explained in the above response

> My suggestion is basically to add two APIs and no macros: one API to get the max number of topology-nearness levels, and a
> second API to get the next sibling a given nearness level from
> 0(nearest)..N(furthest). If we want, we can also add a FOREACH macro too.
>
> Overall, though, as I say above, let's focus on the problem the app actually
> wants these APIs for, not how we think we should solve it. Apps don't want to
> know the topology for knowledge sake, they want to use that knowledge to
> improve performance by pinning tasks to cores. What is the minimum that we
> need to provide to enable the app to do that? For example, if there are no
> lcores that share an L1, then from an app topology viewpoint that L1 level may
> as well not exist, because it provides us no details on how to place our work.
I have shared above why we need vendor agnostic L1, L2, L3 and sub-NUMA-IO.

Snipped