RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Jerin Jacob [mailto:jerinjacobk@gmail.com]
> Sent: Thursday, 24 October 2024 07.49
> 
> On Mon, Oct 21, 2024 at 7:23 AM Wathsala Vithanage
> <wathsala.vithanage@arm.com> wrote:
> >
> > Extend the ethdev library to enable the stashing of different data
> > objects, such as the ones listed below, into CPU caches directly
> > from the NIC.
> >
> > - Rx/Tx queue descriptors
> > - Rx packets
> > - Packet headers
> > - packet payloads
> > - Data of a packet at an offset from the start of the packet
> >
> > The APIs are designed in a hardware/vendor agnostic manner such that
> > supporting PMDs could use any capabilities available in the
> underlying
> > hardware for fine-grained stashing of data objects into a CPU cache
> > (e.g., Steering Tags int PCIe TLP Processing Hints).
> >
> > The API provides an interface to query the availability of stashing
> > capabilities, i.e., platform/NIC support, stashable object types,
> etc,
> > via the rte_eth_dev_stashing_capabilities_get interface.
> >
> > The function pair rte_eth_dev_stashing_rx_config_set and
> > rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU,
> > cache level, and data object types) on the Rx and Tx queues.
> >
> > PMDs that support stashing must register their implementations with
> the
> > following eth_dev_ops callbacks, which are invoked by the ethdev
> > functions listed above.
> >
> > - stashing_capabilities_get
> > - stashing_rx_hints_set
> > - stashing_tx_hints_set
> >
> > Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > Reviewed-by: Dhruv Tripathi <dhruv.tripathi@arm.com>
> >
> 
> > +
> > +/** Queue type is RX. */
> > +#define RTE_ETH_DEV_RX_QUEUE           0
> > +/** Queue type is TX. */
> > +#define RTE_ETH_DEV_TX_QUEUE           1
> > +
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this structure may change, or be removed,
> without prior notice
> > + *
> > + * A structure used for configuring the cache stashing hints.

This structure can only describe one stashing hint.
Please use singular, not plural, in its description.

> > + */
> > +struct rte_eth_stashing_config {
> > +       /** ID of the Processor/Container the stashing hints are
> > +        *  applied to
> > +        */
> > +       uint16_t        lcore_id;

The common type used for lcore_id is "unsigned int", ref. e.g. rte_lcore_id() return value.
Alternatively uint32_t, ref. LCORE_ID_ANY.

> > +       /** Set if the target is a CPU containeri.lcore_id will be
> > +        * used to derive container ID
> > +        */
> > +       uint16_t        container : 1;
> > +       uint16_t        padding : 7;
> > +       /** Cache level of the CPU specified by the cpu_id the
> > +        *  stashing hints are applied to
> > +        */
> > +       uint16_t        cache_level : 8;
> > +       /** Object types the configuration is applied to
> > +        */
> > +       uint16_t        objects;
> > +       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
> > +        *  in objects
> > +        */
> > +       off_t           offset;

off_t is for files, ptrdiff_t is for memory.

> > +};
> > +
> > +/**@{@name Stashable Rx/Tx queue object types supported by the
> ethernet device
> > + *@see rte_eth_dev_stashing_capabilities_get
> > + *@see rte_eth_dev_stashing_rx_config_set
> > + *@see rte_eth_dev_stashing_tx_config_set
> > + */
> > +
> > +/**
> > + * Apply stashing hint to data at a given offset from the start of a
> > + * received packet.
> > + */
> > +#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
> > +
> > +/** Apply stashing hint to an rx descriptor. */
> > +#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
> > +
> > +/** Apply stashing hint to a header of a received packet. */
> > +#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004
> > +
> > +/** Apply stashing hint to a payload of a received packet. */
> > +#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
> > +
> > +#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
> > +/**@}*/
> > +
> > +#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)
> \
> > +       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
> > +
> 
> 
> I think, at one of point of time, we need to extend this to other
> device class like(cryptodev etc)
> where the data needs to move over bus. In that context, all the above
> symbols better to be in
> EAL and the device class subsystem(example ethdev) gives PMD callback.

+1

When generalizing this, perhaps "header" and "payload" should be renamed to "device-specific".

For ethdevs, the typical meaning of "device-specific" would be splitting at some header (as suggested by the "header" and "payload" enum values).

Furthermore, for ethdevs, using a "device-specific" would allow the device to split at some other point, controlled through other ethdev APIs.
E.g. the split point could be controlled by rte_flow; this would allow rte_flow to put entire packets in L2 cache for some packet types, and only the packet header in L2 cache for some other packet types. (Someone at the conference call suggested combining Steering Tags with rte_flow - this might be a way of doing it.)