RE: [PATCH v4 0/4] Introduce Topology NUMA grouping for lcores

["Varghese, Vipin" <Vipin.Varghese@amd.com>]

[AMD Official Use Only - AMD Internal Distribution Only]

Adding Thomas and Ajit to the loop.

Hi Ajit, we have been using the patch series for identifying the topology and getting l3 cache id for populating the steering tag for Device Specific Model & MSI-x driven af-xdp for the experimental STAG firmware on Thor.

Hence current use of topology library helps in
1. workload placement in same Cache or IO domain
2. populating id for MSIx or Device specific model for steering tags.

Thomas and Ajith can we get some help to get this mainline too?



> -----Original Message-----
> From: Vipin Varghese <vipin.varghese@amd.com>
> Sent: Tuesday, November 5, 2024 3:59 PM
> To: dev@dpdk.org; roretzla@linux.microsoft.com; bruce.richardson@intel.com;
> john.mcnamara@intel.com; dmitry.kozliuk@gmail.com
> Cc: pbhagavatula@marvell.com; jerinj@marvell.com; ruifeng.wang@arm.com;
> mattias.ronnblom@ericsson.com; anatoly.burakov@intel.com;
> stephen@networkplumber.org; Yigit, Ferruh <Ferruh.Yigit@amd.com>;
> honnappa.nagarahalli@arm.com; wathsala.vithanage@arm.com;
> konstantin.ananyev@huawei.com; mb@smartsharesystems.com
> Subject: [PATCH v4 0/4] Introduce Topology NUMA grouping for lcores
>
> Caution: This message originated from an External Source. Use proper caution
> when opening attachments, clicking links, or responding.
>
>
> This patch introduces improvements for NUMA topology awareness in relation to
> DPDK logical cores. The goal is to expose API which allows users to select optimal
> logical cores for any application. These logical cores can be selected from various
> NUMA domains like CPU and I/O.
>
> Change Summary:
>  - Introduces the concept of NUMA domain partitioning based on CPU and
>    I/O topology.
>  - Adds support for grouping DPDK logical cores within the same Cache
>    and I/O domain for improved locality.
>  - Implements topology detection and core grouping logic that
>    distinguishes between the following NUMA configurations:
>     * CPU topology & I/O topology (e.g., AMD SoC EPYC, Intel Xeon SPR)
>     * CPU+I/O topology (e.g., Ampere One with SLC, Intel Xeon SPR with SNC)
>  - Enhances performance by minimizing lcore dispersion across tiles|compute
>    package with different L2/L3 cache or IO domains.
>
> Reason:
>  - Applications using DPDK libraries relies on consistent memory access.
>  - Lcores being closer to same NUMA domain as IO.
>  - Lcores sharing same cache.
>
> Latency is minimized by using lcores that share the same NUMA topology.
> Memory access is optimized by utilizing cores within the same NUMA domain or
> tile. Cache coherence is preserved within the same shared cache domain, reducing
> the remote access from tile|compute package via snooping (local hit in either L2 or
> L3 within same NUMA domain).
>
> Library dependency: hwloc
>
> Topology Flags:
> ---------------
>  - RTE_LCORE_DOMAIN_L1: to group cores sharing same L1 cache
>  - RTE_LCORE_DOMAIN_SMT: same as RTE_LCORE_DOMAIN_L1
>  - RTE_LCORE_DOMAIN_L2: group cores sharing same L2 cache
>  - RTE_LCORE_DOMAIN_L3: group cores sharing same L3 cache
>  - RTE_LCORE_DOMAIN_L4: group cores sharing same L4 cache
>  - RTE_LCORE_DOMAIN_IO: group cores sharing same IO
>
> < Function: Purpose >
> ---------------------
>  - rte_get_domain_count: get domain count based on Topology Flag
>  - rte_lcore_count_from_domain: get valid lcores count under each domain
>  - rte_get_lcore_in_domain: valid lcore id based on index
>  - rte_lcore_cpuset_in_domain: return valid cpuset based on index
>  - rte_lcore_is_main_in_domain: return true|false if main lcore is present
>  - rte_get_next_lcore_from_domain: next valid lcore within domain
>  - rte_get_next_lcore_from_next_domain: next valid lcore from next domain
>
> Note:
>  1. Topology is NUMA grouping.
>  2. Domain is various sub-groups within a specific Topology.
>
> Topology example: L1, L2, L3, L4, IO
> Domian example: IO-A, IO-B
>
> < MACRO: Purpose >
> ------------------
>  - RTE_LCORE_FOREACH_DOMAIN: iterate lcores from all domains
>  - RTE_LCORE_FOREACH_WORKER_DOMAIN: iterate worker lcores from all
> domains
>  - RTE_LCORE_FORN_NEXT_DOMAIN: iterate domain select n'th lcore
>  - RTE_LCORE_FORN_WORKER_NEXT_DOMAIN: iterate domain for worker n'th
> lcore.
>
> Future work (after merge):
> --------------------------
>  - dma-perf per IO NUMA
>  - eventdev per L3 NUMA
>  - pipeline per SMT|L3 NUMA
>  - distributor per L3 for Port-Queue
>  - l2fwd-power per SMT
>  - testpmd option for IO NUMA per port
>
> Platform tested on:
> -------------------
>  - INTEL(R) XEON(R) PLATINUM 8562Y+ (support IO numa 1 & 2)
>  - AMD EPYC 8534P (supports IO numa 1 & 2)
>  - AMD EPYC 9554 (supports IO numa 1, 2, 4)
>
> Logs:
> -----
> 1. INTEL(R) XEON(R) PLATINUM 8562Y+:
>  - SNC=1
>         Domain (IO): at index (0) there are 48 core, with (0) at index 0
>  - SNC=2
>         Domain (IO): at index (0) there are 24 core, with (0) at index 0
>         Domain (IO): at index (1) there are 24 core, with (12) at index 0
>
> 2. AMD EPYC 8534P:
>  - NPS=1:
>         Domain (IO): at index (0) there are 128 core, with (0) at index 0
>  - NPS=2:
>         Domain (IO): at index (0) there are 64 core, with (0) at index 0
>         Domain (IO): at index (1) there are 64 core, with (32) at index 0
>
> Signed-off-by: Vipin Varghese <vipin.varghese@amd.com>
>
> Vipin Varghese (4):
>   eal/lcore: add topology based functions
>   test/lcore: enable tests for topology
>   doc: add topology grouping details
>   examples: update with lcore topology API
>
>  app/test/test_lcores.c                        | 528 +++++++++++++
>  config/meson.build                            |  18 +
>  .../prog_guide/env_abstraction_layer.rst      |  22 +
>  examples/helloworld/main.c                    | 154 +++-
>  examples/l2fwd/main.c                         |  56 +-
>  examples/skeleton/basicfwd.c                  |  22 +
>  lib/eal/common/eal_common_lcore.c             | 714 ++++++++++++++++++
>  lib/eal/common/eal_private.h                  |  58 ++
>  lib/eal/freebsd/eal.c                         |  10 +
>  lib/eal/include/rte_lcore.h                   | 209 +++++
>  lib/eal/linux/eal.c                           |  11 +
>  lib/eal/meson.build                           |   4 +
>  lib/eal/version.map                           |  11 +
>  lib/eal/windows/eal.c                         |  12 +
>  14 files changed, 1819 insertions(+), 10 deletions(-)
>
> --
> 2.34.1