RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API

[Wathsala Wathawana Vithanage <wathsala.vithanage@arm.com>]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Thursday, October 24, 2024 2:00 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>; Wathsala Wathawana Vithanage
> <wathsala.vithanage@arm.com>
> Cc: thomas@monjalon.net; Ferruh Yigit <ferruh.yigit@amd.com>; Andrew
> Rybchenko <andrew.rybchenko@oktetlabs.ru>; dev@dpdk.org; nd
> <nd@arm.com>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> Dhruv Tripathi <Dhruv.Tripathi@arm.com>
> Subject: RE: [RFC v3 2/2] ethdev: introduce the cache stashing hints API
> 
> > From: Jerin Jacob [mailto:jerinjacobk@gmail.com]
> > Sent: Thursday, 24 October 2024 07.49
> >
> > On Mon, Oct 21, 2024 at 7:23 AM Wathsala Vithanage
> > <wathsala.vithanage@arm.com> wrote:
> > >
> > > Extend the ethdev library to enable the stashing of different data
> > > objects, such as the ones listed below, into CPU caches directly
> > > from the NIC.
> > >
> > > - Rx/Tx queue descriptors
> > > - Rx packets
> > > - Packet headers
> > > - packet payloads
> > > - Data of a packet at an offset from the start of the packet
> > >
> > > The APIs are designed in a hardware/vendor agnostic manner such that
> > > supporting PMDs could use any capabilities available in the
> > underlying
> > > hardware for fine-grained stashing of data objects into a CPU cache
> > > (e.g., Steering Tags int PCIe TLP Processing Hints).
> > >
> > > The API provides an interface to query the availability of stashing
> > > capabilities, i.e., platform/NIC support, stashable object types,
> > etc,
> > > via the rte_eth_dev_stashing_capabilities_get interface.
> > >
> > > The function pair rte_eth_dev_stashing_rx_config_set and
> > > rte_eth_dev_stashing_tx_config_set sets the stashing hint (the CPU,
> > > cache level, and data object types) on the Rx and Tx queues.
> > >
> > > PMDs that support stashing must register their implementations with
> > the
> > > following eth_dev_ops callbacks, which are invoked by the ethdev
> > > functions listed above.
> > >
> > > - stashing_capabilities_get
> > > - stashing_rx_hints_set
> > > - stashing_tx_hints_set
> > >
> > > Signed-off-by: Wathsala Vithanage <wathsala.vithanage@arm.com>
> > > Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
> > > Reviewed-by: Dhruv Tripathi <dhruv.tripathi@arm.com>
> > >
> >
> > > +
> > > +/** Queue type is RX. */
> > > +#define RTE_ETH_DEV_RX_QUEUE           0
> > > +/** Queue type is TX. */
> > > +#define RTE_ETH_DEV_TX_QUEUE           1
> > > +
> > > +
> > > +/**
> > > + * @warning
> > > + * @b EXPERIMENTAL: this structure may change, or be removed,
> > without prior notice
> > > + *
> > > + * A structure used for configuring the cache stashing hints.
> 
> This structure can only describe one stashing hint.
> Please use singular, not plural, in its description.
> 
> > > + */
> > > +struct rte_eth_stashing_config {
> > > +       /** ID of the Processor/Container the stashing hints are
> > > +        *  applied to
> > > +        */
> > > +       uint16_t        lcore_id;
> 
> The common type used for lcore_id is "unsigned int", ref. e.g. rte_lcore_id()
> return value.
> Alternatively uint32_t, ref. LCORE_ID_ANY.
> 
+1

> > > +       /** Set if the target is a CPU containeri.lcore_id will be
> > > +        * used to derive container ID
> > > +        */
> > > +       uint16_t        container : 1;
> > > +       uint16_t        padding : 7;
> > > +       /** Cache level of the CPU specified by the cpu_id the
> > > +        *  stashing hints are applied to
> > > +        */
> > > +       uint16_t        cache_level : 8;
> > > +       /** Object types the configuration is applied to
> > > +        */
> > > +       uint16_t        objects;
> > > +       /** The offset if RTE_ETH_DEV_STASH_OBJECT_OFFSET bit is set
> > > +        *  in objects
> > > +        */
> > > +       off_t           offset;
> 
> off_t is for files, ptrdiff_t is for memory.
> 

+1

> > > +};
> > > +
> > > +/**@{@name Stashable Rx/Tx queue object types supported by the
> > ethernet device
> > > + *@see rte_eth_dev_stashing_capabilities_get
> > > + *@see rte_eth_dev_stashing_rx_config_set
> > > + *@see rte_eth_dev_stashing_tx_config_set
> > > + */
> > > +
> > > +/**
> > > + * Apply stashing hint to data at a given offset from the start of
> > > +a
> > > + * received packet.
> > > + */
> > > +#define RTE_ETH_DEV_STASH_OBJECT_OFFSET                0x0001
> > > +
> > > +/** Apply stashing hint to an rx descriptor. */
> > > +#define RTE_ETH_DEV_STASH_OBJECT_DESC          0x0002
> > > +
> > > +/** Apply stashing hint to a header of a received packet. */
> > > +#define RTE_ETH_DEV_STASH_OBJECT_HEADER                0x0004
> > > +
> > > +/** Apply stashing hint to a payload of a received packet. */
> > > +#define RTE_ETH_DEV_STASH_OBJECT_PAYLOAD       0x0008
> > > +
> > > +#define __RTE_ETH_DEV_STASH_OBJECT_MASK                0x000f
> > > +/**@}*/
> > > +
> > > +#define RTE_ETH_DEV_STASH_OBJECTS_VALID(t)
> > \
> > > +       ((!((t) & (~__RTE_ETH_DEV_STASH_OBJECT_MASK))) && (t))
> > > +
> >
> >
> > I think, at one of point of time, we need to extend this to other
> > device class like(cryptodev etc) where the data needs to move over
> > bus. In that context, all the above symbols better to be in EAL and
> > the device class subsystem(example ethdev) gives PMD callback.
> 
> +1
> 
> When generalizing this, perhaps "header" and "payload" should be renamed
> to "device-specific".
> 
> For ethdevs, the typical meaning of "device-specific" would be splitting at
> some header (as suggested by the "header" and "payload" enum values).
> 
> Furthermore, for ethdevs, using a "device-specific" would allow the device to
> split at some other point, controlled through other ethdev APIs.
> E.g. the split point could be controlled by rte_flow; this would allow rte_flow
> to put entire packets in L2 cache for some packet types, and only the packet
> header in L2 cache for some other packet types. (Someone at the conference
> call suggested combining Steering Tags with rte_flow - this might be a way of
> doing it.)

+1