RE: [RFC v3 1/3] eal/lcore: add topology based functions

["Morten Brørup" <mb@smartsharesystems.com>]

> > > > > > Does the API need to be prepared for L4 cache?
> > > > > > https://www.anandtech.com/show/16924/did-ibm-just-preview-
> the-
> > > future
> > > > > > -
> > > > > of-caches
> > > > > Thank you for the pointer, yes initial patch was considering L4
> > > cache
> > > > > too. But I was not able to get hold of system or get someone to
> > > test
> > > > > this with L4.
> > > > > Hence removed the L4 instance from dpdk_topology structure.
> > > > >
> > > > > We can introduce into v4. Can we get someone from IBM to
> confirm
> > > the
> > > > > same?
> > > >
> > > > If any of the CPU folks think L4 cache might become relevant in
> the
> > > foreseeable
> > > > future, it should be added to the API without testing at this
> time.
> > >
> > > I remember 3 L4 cache scenario
> > >  1. from IBM power-9 variant suggested in 2020-2021 in hot chips
> 2.
> > > from Intel
> > >   a. Haswell|Broadwell series with eDram as L4
> > >   b. future product (at least in desktop) with L4 cache.
> > >
> > > > Adding it now would prevent API breakage whenever L4 cache
> actually
> > > becomes relevant.
> > > > Otherwise please don't support for it - considering it would be
> dead
> > > and untested
> > > > code.
> > >
> > > I can add the same to the DPDK topology probe in v4, on AMD EPYC v-
> > > cache is treated as extended L3 and not L4. Hence will not be able
> to
> > > test on AMD EPYC.
> >
> > I recall from the Cache Stashing community call... There is some ACPI
> function to
> > get the (opaque) "location IDs" of various parts in the system, to be
> used for setting
> > the Cache Stashing hints.
> > Is there only one "ACPI location ID" (I don't know the correct name)
> shared by the
> > L3 cache and the v-cache in AMD EPYC, or do they have each their own?
> 
> At least on AMD EPYC, the stashing ID updated to either MSI-X table or
> Device Specific Mode is core-id.

Are you saying that on AMD EPYC only L2 caches have a Stashing ID, so no other CPU caches can be stashed into?
If yes, then it's a non-issue for Cache Stashing, since it doesn't need to care about L3 cache or v-cache.

> 
> 
> > If they are not exposed as one ID, but two separate IDs, the Topology
> API might
> > need to reflect this, so it can be used in the Cache Stashing API.
> 
> I have different view on the same and had shared this with Ajit
> (Broadcom) and others. To my understanding, use of rte_ethdev API used
> for caching hints should be inline to rte_lcore. Depending upon the
> platform (ARM's specific implementation, the lcore gets translated to
> L2 or L3 cache ID within the PMD.

The rte_ethdev API for cache stashing provides a higher level of abstraction, yes.

But the layer below it - the Stashing API used by the PMDs to obtain Stashing ID from "location ID" - could use the "location ID" structure type defined by the Topology library's lower layer.

> 
> Note:  the current patch introduces of Topology aware grouping, which
> helps to run application better or tiles or chiplets sharing same L2|L3
> or IO domain.

Both libraries (Topology and Cache Stashing) need to have detailed information about the hardware, although they use the information for two different purposes.

Maybe they could share a common lower layer for the system topology, perhaps just a few header files. Or maybe the Cache Stashing library should depend on the Topology library as its "lower layer" to provide the hardware information it needs.

I'm not saying that it must be so. I'm only saying that I suppose these two libraries have a lot in common, and they could try to take advantage of this, to provide a more uniform API at their lower layers.