[RFC] dmadev: add QoS capability

[RFC] dmadev: add QoS capability

[Vamsi Attunuru]

Some DMA controllers support QoS at HW command queue level to
differentiate the performance on different HW queues based on
the priority configured. Patch adds required fields in dmadev
structures to get hardware supported priority levels and the
provision to configure the priority from the applications.

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---
 lib/dmadev/rte_dmadev.c | 10 ++++++++++
 lib/dmadev/rte_dmadev.h | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/lib/dmadev/rte_dmadev.c b/lib/dmadev/rte_dmadev.c
index 845727210f..9ff62efcb4 100644
--- a/lib/dmadev/rte_dmadev.c
+++ b/lib/dmadev/rte_dmadev.c
@@ -491,6 +491,16 @@ rte_dma_configure(int16_t dev_id, const struct rte_dma_conf *dev_conf)
 			"Device %d configure too many vchans", dev_id);
 		return -EINVAL;
 	}
+	if (dev_conf->priority &&
+	    !(dev_info.dev_capa & RTE_DMA_CAPA_QOS)) {
+		RTE_DMA_LOG(ERR, "Device %d don't support QoS", dev_id);
+		return -EINVAL;
+	}
+	if (dev_conf->priority >= dev_info.nb_priorities) {
+		RTE_DMA_LOG(ERR,
+			"Device %d configure invalid priority", dev_id);
+		return -EINVAL;
+	}
 	if (dev_conf->enable_silent &&
 	    !(dev_info.dev_capa & RTE_DMA_CAPA_SILENT)) {
 		RTE_DMA_LOG(ERR, "Device %d don't support silent", dev_id);
diff --git a/lib/dmadev/rte_dmadev.h b/lib/dmadev/rte_dmadev.h
index 5474a5281d..08db8ead0a 100644
--- a/lib/dmadev/rte_dmadev.h
+++ b/lib/dmadev/rte_dmadev.h
@@ -268,6 +268,16 @@ int16_t rte_dma_next_dev(int16_t start_dev_id);
 #define RTE_DMA_CAPA_OPS_COPY_SG	RTE_BIT64(33)
 /** Support fill operation. */
 #define RTE_DMA_CAPA_OPS_FILL		RTE_BIT64(34)
+/** Support QoS at DMA HW channel level
+ *
+ * If device supports QoS then application could configure priority to the
+ * DMA HW channel using 'priority' field in struct rte_dma_conf. Number of
+ * supported prioirty levels will be known from 'nb_priorities' field in
+ * struct rte_dma_info.
+ * DMA devices which support QoS at HW channel level can advertise this
+ * capability.
+ */
+#define RTE_DMA_CAPA_QOS		RTE_BIT64(35)
 /**@}*/
 
 /**
@@ -297,6 +307,8 @@ struct rte_dma_info {
 	int16_t numa_node;
 	/** Number of virtual DMA channel configured. */
 	uint16_t nb_vchans;
+	/** Number of priority levels supported by DMA HW channel. */
+	uint16_t nb_priorities;
 };
 
 /**
@@ -332,6 +344,13 @@ struct rte_dma_conf {
 	 * @see RTE_DMA_CAPA_SILENT
 	 */
 	bool enable_silent;
+	/* The prioirty of the DMA HW channel.
+	 * This value cannot be greater than or equal to the field 'nb_priorities'
+	 * of struct rte_dma_info which get from rte_dma_info_get().
+	 * Among the values between '0' and 'nb_priorities - 1', lowest value
+	 * indicates higher priority and vice-versa.
+	 */
+	uint16_t priority;
 };
 
 /**
-- 
2.25.1

RE: [RFC] dmadev: add QoS capability

[Morten Brørup]

> From: Vamsi Attunuru [mailto:vattunuru@marvell.com]
> Sent: Monday, 29 July 2024 13.56
> 
> Some DMA controllers support QoS at HW command queue level to
> differentiate the performance on different HW queues based on
> the priority configured. Patch adds required fields in dmadev
> structures to get hardware supported priority levels and the
> provision to configure the priority from the applications.

Do we foresee anything more advanced than Strict Priority scheduling for DMA anytime in the future?

If not, then consider calling this new capability Prioritization (CAPA_PRIO) instead of Quality Of Service (CAPA_QOS). Then we don't need to add and describe QoS parameters for a more advanced QoS scheduling algorithm (e.g. the "weight" for weighted fair queueing).

> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
>  lib/dmadev/rte_dmadev.c | 10 ++++++++++
>  lib/dmadev/rte_dmadev.h | 19 +++++++++++++++++++
>  2 files changed, 29 insertions(+)
> 
> diff --git a/lib/dmadev/rte_dmadev.c b/lib/dmadev/rte_dmadev.c
> index 845727210f..9ff62efcb4 100644
> --- a/lib/dmadev/rte_dmadev.c
> +++ b/lib/dmadev/rte_dmadev.c
> @@ -491,6 +491,16 @@ rte_dma_configure(int16_t dev_id, const struct
> rte_dma_conf *dev_conf)
>  			"Device %d configure too many vchans", dev_id);
>  		return -EINVAL;
>  	}
> +	if (dev_conf->priority &&

-	if (dev_conf->priority &&
+ 	if (dev_conf->priority != 0 &&

> +	    !(dev_info.dev_capa & RTE_DMA_CAPA_QOS)) {
> +		RTE_DMA_LOG(ERR, "Device %d don't support QoS", dev_id);
> +		return -EINVAL;
> +	}
> +	if (dev_conf->priority >= dev_info.nb_priorities) {
> +		RTE_DMA_LOG(ERR,
> +			"Device %d configure invalid priority", dev_id);
> +		return -EINVAL;
> +	}
>  	if (dev_conf->enable_silent &&
>  	    !(dev_info.dev_capa & RTE_DMA_CAPA_SILENT)) {
>  		RTE_DMA_LOG(ERR, "Device %d don't support silent", dev_id);
> diff --git a/lib/dmadev/rte_dmadev.h b/lib/dmadev/rte_dmadev.h
> index 5474a5281d..08db8ead0a 100644
> --- a/lib/dmadev/rte_dmadev.h
> +++ b/lib/dmadev/rte_dmadev.h
> @@ -268,6 +268,16 @@ int16_t rte_dma_next_dev(int16_t start_dev_id);
>  #define RTE_DMA_CAPA_OPS_COPY_SG	RTE_BIT64(33)
>  /** Support fill operation. */
>  #define RTE_DMA_CAPA_OPS_FILL		RTE_BIT64(34)
> +/** Support QoS at DMA HW channel level
> + *
> + * If device supports QoS then application could configure priority to the
> + * DMA HW channel using 'priority' field in struct rte_dma_conf. Number of
> + * supported prioirty levels will be known from 'nb_priorities' field in
> + * struct rte_dma_info.
> + * DMA devices which support QoS at HW channel level can advertise this
> + * capability.
> + */
> +#define RTE_DMA_CAPA_QOS		RTE_BIT64(35)
>  /**@}*/
> 
>  /**
> @@ -297,6 +307,8 @@ struct rte_dma_info {
>  	int16_t numa_node;
>  	/** Number of virtual DMA channel configured. */
>  	uint16_t nb_vchans;
> +	/** Number of priority levels supported by DMA HW channel. */
> +	uint16_t nb_priorities;

This value must be 0 or > 1, never 1. Please mention in the comment, e.g.:

/** Number of priority levels, if supported by DMA HW channel. 0 otherwise. */

Please add a test case to verify that the DMA device reports nb_priorities > 1 if it has CAPA_PRIO, and 0 if not.
Alternatively:
Assuming we don't foresee anything more advanced than Strict Priority...
Remove the CAPA_PRIO capability flag. Reading nb_priorities should suffice.

>  };
> 
>  /**
> @@ -332,6 +344,13 @@ struct rte_dma_conf {
>  	 * @see RTE_DMA_CAPA_SILENT
>  	 */
>  	bool enable_silent;
> +	/* The prioirty of the DMA HW channel.
> +	 * This value cannot be greater than or equal to the field
> 'nb_priorities'
> +	 * of struct rte_dma_info which get from rte_dma_info_get().
> +	 * Among the values between '0' and 'nb_priorities - 1', lowest value
> +	 * indicates higher priority and vice-versa.
> +	 */
> +	uint16_t priority;
>  };
> 
>  /**
> --
> 2.25.1

Re: [RFC] dmadev: add QoS capability

[Bruce Richardson]

On Mon, Jul 29, 2024 at 03:14:55PM +0200, Morten Brørup wrote:
> > From: Vamsi Attunuru [mailto:vattunuru@marvell.com]
> > Sent: Monday, 29 July 2024 13.56
> > 
> > Some DMA controllers support QoS at HW command queue level to
> > differentiate the performance on different HW queues based on
> > the priority configured. Patch adds required fields in dmadev
> > structures to get hardware supported priority levels and the
> > provision to configure the priority from the applications.
> 
> Do we foresee anything more advanced than Strict Priority scheduling for DMA anytime in the future?
> 
> If not, then consider calling this new capability Prioritization (CAPA_PRIO) instead of Quality Of Service (CAPA_QOS). Then we don't need to add and describe QoS parameters for a more advanced QoS scheduling algorithm (e.g. the "weight" for weighted fair queueing).
> 

There could be more than just regular prioritization settings involved, so
I think it's best to leave some options open. Even with just a
"prioritization" setting, it could be used as a weighting vs strict priority.
Question is whether in such a case - of a single-value number for high vs
low priority - it's better to explicitly separate out a weight priority vs
a strict priority, or give a simpler interface by allowing just a single
number value.

/Bruce

RE: [RFC] dmadev: add QoS capability

[Morten Brørup]

> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
> Sent: Monday, 29 July 2024 15.27
> 
> On Mon, Jul 29, 2024 at 03:14:55PM +0200, Morten Brørup wrote:
> > > From: Vamsi Attunuru [mailto:vattunuru@marvell.com]
> > > Sent: Monday, 29 July 2024 13.56
> > >
> > > Some DMA controllers support QoS at HW command queue level to
> > > differentiate the performance on different HW queues based on
> > > the priority configured. Patch adds required fields in dmadev
> > > structures to get hardware supported priority levels and the
> > > provision to configure the priority from the applications.
> >
> > Do we foresee anything more advanced than Strict Priority scheduling for DMA
> anytime in the future?
> >
> > If not, then consider calling this new capability Prioritization (CAPA_PRIO)
> instead of Quality Of Service (CAPA_QOS). Then we don't need to add and
> describe QoS parameters for a more advanced QoS scheduling algorithm (e.g. the
> "weight" for weighted fair queueing).
> >
> 
> There could be more than just regular prioritization settings involved, so
> I think it's best to leave some options open. Even with just a
> "prioritization" setting, it could be used as a weighting vs strict priority.
> Question is whether in such a case - of a single-value number for high vs
> low priority - it's better to explicitly separate out a weight priority vs
> a strict priority, or give a simpler interface by allowing just a single
> number value.

If we leave some options open, we need to define the API for them.
Let's not go overboard with this, but stay within what could be realistic for a DMA engine.

Remember, the API needs to be cross-platform, so simply replacing the "Priority" parameter with a "QoS Class ID" also requires adding configuration parameters to map each QoS Class ID to a generically defined behavior (e.g. priority, weight).

@Vamsi, how many priority levels does your DMA hardware support?

RE: [RFC] dmadev: add QoS capability

[Vamsi Krishna Attunuru]

>-----Original Message-----
>From: Morten Brørup <mb@smartsharesystems.com>
>Sent: Monday, July 29, 2024 7:23 PM
>To: Bruce Richardson <bruce.richardson@intel.com>
>Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>;
>fengchengwen@huawei.com; dev@dpdk.org; kevin.laatz@intel.com; Jerin
>Jacob <jerinj@marvell.com>; Anoob Joseph <anoobj@marvell.com>
>Subject: [EXTERNAL] RE: [RFC] dmadev: add QoS capability
>
>> From: Bruce Richardson [mailto: bruce. richardson@ intel. com] > Sent:
>> Monday, 29 July 2024 15. 27 > > On Mon, Jul 29, 2024 at 03: 14: 55PM
>> +0200, Morten Brørup wrote: > > > From: Vamsi Attunuru [mailto: 
>> vattunuru@ marvell. com]
>
>> From: Bruce Richardson [mailto:bruce.richardson@intel.com]
>> Sent: Monday, 29 July 2024 15.27
>>
>> On Mon, Jul 29, 2024 at 03:14:55PM +0200, Morten Brørup wrote:
>> > > From: Vamsi Attunuru [mailto:vattunuru@marvell.com]
>> > > Sent: Monday, 29 July 2024 13.56
>> > >
>> > > Some DMA controllers support QoS at HW command queue level to
>> > > differentiate the performance on different HW queues based on the
>> > > priority configured. Patch adds required fields in dmadev
>> > > structures to get hardware supported priority levels and the
>> > > provision to configure the priority from the applications.
>> >
>> > Do we foresee anything more advanced than Strict Priority scheduling
>> > for DMA
>> anytime in the future?
>> >
>> > If not, then consider calling this new capability Prioritization
>> > (CAPA_PRIO)
>> instead of Quality Of Service (CAPA_QOS). Then we don't need to add
>> and describe QoS parameters for a more advanced QoS scheduling
>> algorithm (e.g. the "weight" for weighted fair queueing).
>> >
>>
>> There could be more than just regular prioritization settings
>> involved, so I think it's best to leave some options open. Even with
>> just a "prioritization" setting, it could be used as a weighting vs strict priority.
>> Question is whether in such a case - of a single-value number for high
>> vs low priority - it's better to explicitly separate out a weight
>> priority vs a strict priority, or give a simpler interface by allowing
>> just a single number value.
>
>If we leave some options open, we need to define the API for them.
>Let's not go overboard with this, but stay within what could be realistic for a
>DMA engine.
>
>Remember, the API needs to be cross-platform, so simply replacing the
>"Priority" parameter with a "QoS Class ID" also requires adding configuration
>parameters to map each QoS Class ID to a generically defined behavior (e.g.
>priority, weight).
>
>@Vamsi, how many priority levels does your DMA hardware support?

Hi Morten & Bruce, thanks for the comments.

Our DMA HW supports 4 priorities. Yes, as Bruce pointed, we can have options
open to support both(weighted & strict). Let me get back on this that could be
sensible for DMA kind of hardware.

Regards
Vamsi