RE: [PATCH v2 1/3] eventdev/eth_rx: add API to burst add queues to Rx adapter

[Shijith Thotton <sthotton@marvell.com>]

>>
>> This patch introduces a new API, rte_event_eth_rx_adapter_queues_add(),
>> to allow bulk addition of multiple Rx queues in the eventdev Rx adapter.
>>
>> The existing rte_event_eth_rx_adapter_queue_add() API supports adding
>> multiple queues by specifying rx_queue_id = -1, but it lacks the ability to apply
>> specific configurations to each of the added queues.
>>
>> A new internal PMD operation, eventdev_eth_rx_adapter_queues_add_t, has
>> been introduced to enable this functionality. It takes an array of receive queue
>> IDs along with their corresponding queue configurations.
>>
>> Signed-off-by: Shijith Thotton <sthotton@marvell.com>
>> ---
>>  .../eventdev/event_ethernet_rx_adapter.rst    |  60 +++++--
>>  lib/eventdev/eventdev_pmd.h                   |  34 ++++
>>  lib/eventdev/eventdev_trace.h                 |  14 ++
>>  lib/eventdev/eventdev_trace_points.c          |   3 +
>>  lib/eventdev/rte_event_eth_rx_adapter.c       | 155 ++++++++++++++++++
>>  lib/eventdev/rte_event_eth_rx_adapter.h       |  33 ++++
>>  lib/eventdev/version.map                      |   3 +
>>  7 files changed, 292 insertions(+), 10 deletions(-)
>>
>> diff --git a/doc/guides/prog_guide/eventdev/event_ethernet_rx_adapter.rst
>> b/doc/guides/prog_guide/eventdev/event_ethernet_rx_adapter.rst
>> index 2e68cca798..bae46cc7d7 100644
>> --- a/doc/guides/prog_guide/eventdev/event_ethernet_rx_adapter.rst
>> +++ b/doc/guides/prog_guide/eventdev/event_ethernet_rx_adapter.rst
>> @@ -96,16 +96,23 @@ when the adapter is created using the above-
>> mentioned APIs.
>>  Adding Rx Queues to the Adapter Instance
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> -Ethdev Rx queues are added to the instance using the -
>> ``rte_event_eth_rx_adapter_queue_add()`` function. Configuration for the Rx
>-
>> queue is passed in using a ``struct rte_event_eth_rx_adapter_queue_conf``
>> -parameter. Event information for packets from this Rx queue is encoded in
>the
>> -``ev`` field of ``struct rte_event_eth_rx_adapter_queue_conf``. The -
>> servicing_weight member of the struct
>> rte_event_eth_rx_adapter_queue_conf
>> -is the relative polling frequency of the Rx queue and is applicable when the -
>> adapter uses a service core function. The applications can configure queue -
>> event buffer size in ``struct
>> rte_event_eth_rx_adapter_queue_conf::event_buf_size``
>> -parameter.
>> +Ethdev Rx queues can be added to the instance using either the
>> +``rte_event_eth_rx_adapter_queue_add()`` function or
>> +``rte_event_eth_rx_adapter_queues_add()``. The former is used to add a
>> +single Rx queue at a time, while the latter allows adding multiple Rx
>> +queues in a single call.
>> +
>> +Single Queue Addition
>> +^^^^^^^^^^^^^^^^^^^^^
>> +
>> +The ``rte_event_eth_rx_adapter_queue_add()`` API allows adding a single
>> +Rx queue to the adapter instance. Configuration for the Rx queue is
>> +passed using a ``struct rte_event_eth_rx_adapter_queue_conf``
>> +parameter. Event information for packets from this Rx queue is encoded
>> +in the ``ev`` field of this struct. The ``servicing_weight`` member of
>> +the struct determines the relative polling frequency of the Rx queue
>> +and is applicable when the adapter uses a service core function.
>> +Applications can also configure the queue event buffer size using the
>> ``event_buf_size`` parameter in ``struct
>> rte_event_eth_rx_adapter_queue_conf``.
>>
>>  .. code-block:: c
>>
>> @@ -122,6 +129,39 @@ parameter.
>>                                                  eth_dev_id,
>>                                                  0, &queue_config);
>>
>> +Bulk Queue Addition
>> +^^^^^^^^^^^^^^^^^^^
>> +
>> +The ``rte_event_eth_rx_adapter_queues_add()`` API allows the addition
>> +of multiple Rx queues in a single call. While
>> +``rte_event_eth_rx_adapter_queue_add()`` supports adding multiple
>> +queues by specifying ``rx_queue_id = -1``, it does not allow applying
>> +specific configurations to each queue individually.  The
>> +``rte_event_eth_rx_adapter_queues_add()`` API accepts an array of
>> +receive queue IDs along with their corresponding configurations,
>> +enabling control over each Rx queue's settings.
>> +
>> +.. code-block:: c
>> +
>> +        struct rte_event_eth_rx_adapter_queue_conf
>> queue_config[nb_rx_queues];
>> +        int rx_queue_id[nb_rx_queues];
>> +
>> +        for (int i = 0; i < nb_rx_queues; i++) {
>> +            rx_queue_id[i] = i;
>> +            queue_config[i].rx_queue_flags = 0;
>> +            queue_config[i].ev.queue_id = i;
>> +            queue_config[i].ev.sched_type = RTE_SCHED_TYPE_ATOMIC;
>> +            queue_config[i].ev.priority = 0;
>> +            queue_config[i].servicing_weight = 1;
>> +            queue_config[i].event_buf_size = 1024;
>> +        }
>> +
>> +        err = rte_event_eth_rx_adapter_queues_add(id,
>> +                                                 eth_dev_id,
>> +                                                 rx_queue_id,
>> +                                                 queue_config,
>> +                                                 nb_rx_queues);
>> +
>>  Querying Adapter Capabilities
>>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> diff --git a/lib/eventdev/eventdev_pmd.h b/lib/eventdev/eventdev_pmd.h
>> index 36148f8d86..ad13ba5b03 100644
>> --- a/lib/eventdev/eventdev_pmd.h
>> +++ b/lib/eventdev/eventdev_pmd.h
>> @@ -25,6 +25,7 @@
>>  #include <rte_mbuf_dyn.h>
>>
>>  #include "event_timer_adapter_pmd.h"
>> +#include "rte_event_eth_rx_adapter.h"
>>  #include "rte_eventdev.h"
>>
>>  #ifdef __cplusplus
>> @@ -708,6 +709,37 @@ typedef int
>> (*eventdev_eth_rx_adapter_queue_add_t)(
>>  		int32_t rx_queue_id,
>>  		const struct rte_event_eth_rx_adapter_queue_conf
>> *queue_conf);
>>
>> +/**
>> + * Add ethernet Rx queues to event device in burst. This callback is
>> +invoked if
>> + * the caps returned from rte_eventdev_eth_rx_adapter_caps_get(,
>> +eth_port_id)
>> + * has RTE_EVENT_ETH_RX_ADAPTER_CAP_INTERNAL_PORT set.
>> + *
>> + * @param dev
>> + *   Event device pointer
>> + *
>> + * @param eth_dev
>> + *   Ethernet device pointer
>> + *
>> + * @param rx_queue_id
>> + *   Ethernet device receive queue index array
>> + *
>> + * @param queue_conf
>> + *   Additional configuration structure array
>> + *
>> + * @param nb_rx_queues
>> + *   Number of ethernet device receive queues
>> + *
>> + * @return
>> + *   - 0: Success, ethernet receive queues added successfully.
>> + *   - <0: Error code returned by the driver function.
>> + */
>> +typedef int (*eventdev_eth_rx_adapter_queues_add_t)(
>> +		const struct rte_eventdev *dev,
>> +		const struct rte_eth_dev *eth_dev,
>> +		int32_t rx_queue_id[],
>> +		const struct rte_event_eth_rx_adapter_queue_conf
>> queue_conf[],
>> +		uint16_t nb_rx_queues);
>> +
>>  /**
>>   * Delete ethernet Rx queues from event device. This callback is invoked if
>>   * the caps returned from eventdev_eth_rx_adapter_caps_get(, eth_port_id)
>> @@ -1578,6 +1610,8 @@ struct eventdev_ops {
>>  	/**< Get ethernet Rx adapter capabilities */
>>  	eventdev_eth_rx_adapter_queue_add_t eth_rx_adapter_queue_add;
>>  	/**< Add Rx queues to ethernet Rx adapter */
>> +	eventdev_eth_rx_adapter_queues_add_t
>> eth_rx_adapter_queues_add;
>> +	/**< Add Rx queues to ethernet Rx adapter in burst */
>>  	eventdev_eth_rx_adapter_queue_del_t eth_rx_adapter_queue_del;
>>  	/**< Delete Rx queues from ethernet Rx adapter */
>>  	eventdev_eth_rx_adapter_queue_conf_get_t
>> eth_rx_adapter_queue_conf_get; diff --git a/lib/eventdev/eventdev_trace.h
>> b/lib/eventdev/eventdev_trace.h index 8ff8841729..6b334d8bd1 100644
>> --- a/lib/eventdev/eventdev_trace.h
>> +++ b/lib/eventdev/eventdev_trace.h
>> @@ -159,6 +159,20 @@ RTE_TRACE_POINT(
>>  	rte_trace_point_emit_int(rc);
>>  )
>>
>> +RTE_TRACE_POINT(
>> +	rte_eventdev_trace_eth_rx_adapter_queues_add,
>> +	RTE_TRACE_POINT_ARGS(uint8_t adptr_id, uint16_t eth_dev_id,
>> +		uint16_t nb_rx_queues, void *rx_queue_id,
>> +		const void *queue_conf,
>> +		int rc),
>> +	rte_trace_point_emit_u8(adptr_id);
>> +	rte_trace_point_emit_u16(eth_dev_id);
>> +	rte_trace_point_emit_u16(nb_rx_queues);
>> +	rte_trace_point_emit_ptr(rx_queue_id);
>> +	rte_trace_point_emit_ptr(queue_conf);
>> +	rte_trace_point_emit_int(rc);
>> +)
>> +
>>  RTE_TRACE_POINT(
>>  	rte_eventdev_trace_eth_rx_adapter_queue_del,
>>  	RTE_TRACE_POINT_ARGS(uint8_t adptr_id, uint16_t eth_dev_id, diff -
>> -git a/lib/eventdev/eventdev_trace_points.c
>> b/lib/eventdev/eventdev_trace_points.c
>> index e7af1591f7..8caf6353a1 100644
>> --- a/lib/eventdev/eventdev_trace_points.c
>> +++ b/lib/eventdev/eventdev_trace_points.c
>> @@ -65,6 +65,9 @@
>> RTE_TRACE_POINT_REGISTER(rte_eventdev_trace_eth_rx_adapter_free,
>>
>> RTE_TRACE_POINT_REGISTER(rte_eventdev_trace_eth_rx_adapter_queue_ad
>> d,
>>  	lib.eventdev.rx.adapter.queue.add)
>>
>> +RTE_TRACE_POINT_REGISTER(rte_eventdev_trace_eth_rx_adapter_queues_
>> add,
>> +	lib.eventdev.rx.adapter.queues.add)
>> +
>>
>>
>RTE_TRACE_POINT_REGISTER(rte_eventdev_trace_eth_rx_adapter_queue_del
>> ,
>>  	lib.eventdev.rx.adapter.queue.del)
>>
>> diff --git a/lib/eventdev/rte_event_eth_rx_adapter.c
>> b/lib/eventdev/rte_event_eth_rx_adapter.c
>> index 39674c4604..87bb64bcd5 100644
>> --- a/lib/eventdev/rte_event_eth_rx_adapter.c
>> +++ b/lib/eventdev/rte_event_eth_rx_adapter.c
>> @@ -2793,6 +2793,161 @@ rte_event_eth_rx_adapter_queue_add(uint8_t
>> id,
>>  	return 0;
>>  }
>>
>> +int
>> +rte_event_eth_rx_adapter_queues_add(uint8_t id, uint16_t eth_dev_id,
>> int32_t rx_queue_id[],
>> +				    const struct
>> rte_event_eth_rx_adapter_queue_conf queue_conf[],
>> +				    uint16_t nb_rx_queues)
>> +{
>> +	struct rte_event_eth_rx_adapter_vector_limits limits;
>> +	struct event_eth_rx_adapter *rx_adapter;
>> +	struct eth_device_info *dev_info;
>> +	struct rte_eventdev *dev;
>> +	uint32_t cap, i;
>> +	int ret;
>> +
>> +	if (rxa_memzone_lookup())
>> +		return -ENOMEM;
>> +
>> +	RTE_EVENT_ETH_RX_ADAPTER_ID_VALID_OR_ERR_RET(id, -EINVAL);
>> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(eth_dev_id, -EINVAL);
>> +
>> +	rx_adapter = rxa_id_to_adapter(id);
>> +	if ((rx_adapter == NULL) || (queue_conf == NULL))
>> +		return -EINVAL;
>> +
>> +	if (nb_rx_queues && rx_queue_id == NULL)
>> +		return -EINVAL;
>> +
>> +	if (nb_rx_queues > rte_eth_devices[eth_dev_id].data->nb_rx_queues)
>> {
>> +		RTE_EDEV_LOG_ERR("Invalid number of rx queues %" PRIu16,
>> nb_rx_queues);
>> +		return -EINVAL;
>> +	}
>> +
>> +	ret = rte_event_eth_rx_adapter_caps_get(rx_adapter->eventdev_id,
>> eth_dev_id, &cap);
>> +	if (ret) {
>> +		RTE_EDEV_LOG_ERR("Failed to get adapter caps edev %"
>> PRIu8 "eth port %" PRIu16, id,
>> +				 eth_dev_id);
>> +		return ret;
>> +	}
>> +
>> +	if ((cap & RTE_EVENT_ETH_RX_ADAPTER_CAP_MULTI_EVENTQ) == 0
>> && nb_rx_queues) {
>> +		RTE_EDEV_LOG_ERR("Rx queues can only be connected to
>> single "
>> +				 "event queue, eth port: %" PRIu16 " adapter
>> id: %" PRIu8,
>> +				 eth_dev_id, id);
>> +		return -EINVAL;
>> +	}
>> +
>
>If the adapter does not support en queueing to multiple event queues, why are
>multiple rx queues not supported?
>All the rx_queues queue_conf can be pointing to the same single event queue
>(queue_config[i].ev.queue_id).
>Maybe all the "ev.queue_id"s need to be validated here to be the same?

Okay. I will add the necessary logic to check queue_id. If the adapter does not
support queuing to multiple event queues and the configuration uses multiple
event queues, an error will be returned.

>
>> +	for (i = 0; i < (nb_rx_queues ? nb_rx_queues : 1); i++) {
>> +		const struct rte_event_eth_rx_adapter_queue_conf *conf;
>> +
>> +		conf = &queue_conf[i];
>> +		if ((cap &
>> RTE_EVENT_ETH_RX_ADAPTER_CAP_OVERRIDE_FLOW_ID) == 0 &&
>> +		    (conf->rx_queue_flags &
>> RTE_EVENT_ETH_RX_ADAPTER_QUEUE_FLOW_ID_VALID)) {
>> +			RTE_EDEV_LOG_ERR("Flow ID override is not
>> supported in queue_conf[%" PRIu32
>> +					 "], eth port: %" PRIu16 " adapter id:
>> %" PRIu8,
>> +					 i, eth_dev_id, id);
>> +			return -EINVAL;
>> +		}
>> +
>> +		if (conf->rx_queue_flags &
>> RTE_EVENT_ETH_RX_ADAPTER_QUEUE_EVENT_VECTOR) {
>> +			if ((cap &
>> RTE_EVENT_ETH_RX_ADAPTER_CAP_EVENT_VECTOR) == 0) {
>> +				RTE_EDEV_LOG_ERR(
>> +					"Event vectorization is unsupported in
>> queue_conf[%" PRIu32
>> +					"], eth port: %" PRIu16 " adapter id:
>> %" PRIu8,
>> +					i, eth_dev_id, id);
>> +				return -EINVAL;
>> +			}
>> +
>> +			ret =
>> rte_event_eth_rx_adapter_vector_limits_get(rx_adapter->eventdev_id,
>> +
>> eth_dev_id, &limits);
>> +			if (ret < 0) {
>> +				RTE_EDEV_LOG_ERR("Failed to get event
>> device vector limits,"
>> +						 " eth port: %" PRIu16 "
>> adapter id: %" PRIu8,
>> +						 eth_dev_id, id);
>> +				return -EINVAL;
>> +			}
>> +
>> +			if (conf->vector_sz < limits.min_sz || conf->vector_sz
>> > limits.max_sz ||
>> +			    conf->vector_timeout_ns < limits.min_timeout_ns
>> ||
>> +			    conf->vector_timeout_ns > limits.max_timeout_ns
>> ||
>> +			    conf->vector_mp == NULL) {
>> +				RTE_EDEV_LOG_ERR(
>> +					"Invalid event vector configuration in
>> queue_conf[%" PRIu32
>> +					"], eth port: %" PRIu16 " adapter id:
>> %" PRIu8,
>> +					i, eth_dev_id, id);
>> +				return -EINVAL;
>> +			}
>> +
>> +			if (conf->vector_mp->elt_size < (sizeof(struct
>> rte_event_vector) +
>> +							 (sizeof(uintptr_t) *
>> conf->vector_sz))) {
>> +				RTE_EDEV_LOG_ERR(
>> +					"Invalid event vector configuration in
>> queue_conf[%" PRIu32
>> +					"], eth port: %" PRIu16 " adapter id:
>> %" PRIu8,
>> +					i, eth_dev_id, id);
>> +				return -EINVAL;
>> +			}
>> +		}
>> +
>> +		if ((rx_adapter->use_queue_event_buf && conf-
>> >event_buf_size == 0) ||
>> +		    (!rx_adapter->use_queue_event_buf && conf-
>> >event_buf_size != 0)) {
>> +			RTE_EDEV_LOG_ERR("Invalid Event buffer size in
>> queue_conf[%" PRIu32 "]", i);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	dev = &rte_eventdevs[rx_adapter->eventdev_id];
>> +	dev_info = &rx_adapter->eth_devices[eth_dev_id];
>> +
>> +	if (cap & RTE_EVENT_ETH_RX_ADAPTER_CAP_INTERNAL_PORT) {
>> +		if (*dev->dev_ops->eth_rx_adapter_queues_add == NULL)
>> +			return -ENOTSUP;
>> +
>> +		if (dev_info->rx_queue == NULL) {
>> +			dev_info->rx_queue =
>> +				rte_zmalloc_socket(rx_adapter->mem_name,
>> +						   dev_info->dev->data-
>> >nb_rx_queues *
>> +							   sizeof(struct
>> eth_rx_queue_info),
>> +						   0, rx_adapter->socket_id);
>> +			if (dev_info->rx_queue == NULL)
>> +				return -ENOMEM;
>> +		}
>> +
>> +		ret = (*dev->dev_ops->eth_rx_adapter_queues_add)(
>> +			dev, &rte_eth_devices[eth_dev_id], rx_queue_id,
>> queue_conf, nb_rx_queues);
>> +		if (ret == 0) {
>> +			dev_info->internal_event_port = 1;
>> +
>> +			if (nb_rx_queues == 0)
>> +				rxa_update_queue(rx_adapter, dev_info, -1,
>> 1);
>> +
>> +			for (i = 0; i < nb_rx_queues; i++)
>> +				rxa_update_queue(rx_adapter, dev_info,
>> rx_queue_id[i], 1);
>> +		}
>> +	} else {
>> +		rte_spinlock_lock(&rx_adapter->rx_lock);
>> +		dev_info->internal_event_port = 0;
>> +		ret = rxa_init_service(rx_adapter, id);
>> +		if (ret == 0) {
>> +			uint32_t service_id = rx_adapter->service_id;
>> +
>> +			if (nb_rx_queues == 0)
>> +				ret = rxa_sw_add(rx_adapter, eth_dev_id, -1,
>> &queue_conf[0]);
>> +
>> +			for (i = 0; i < nb_rx_queues; i++)
>> +				ret = rxa_sw_add(rx_adapter, eth_dev_id,
>> rx_queue_id[i],
>> +						 &queue_conf[i]);
>> +
>> +			rte_service_component_runstate_set(service_id,
>> +
>> rxa_sw_adapter_queue_count(rx_adapter));
>> +		}
>> +		rte_spinlock_unlock(&rx_adapter->rx_lock);
>> +	}
>> +
>> +	rte_eventdev_trace_eth_rx_adapter_queues_add(id, eth_dev_id,
>> nb_rx_queues, rx_queue_id,
>> +						     queue_conf, ret);
>> +	return ret;
>> +}
>> +
>>  static int
>>  rxa_sw_vector_limits(struct rte_event_eth_rx_adapter_vector_limits *limits)
>> { diff --git a/lib/eventdev/rte_event_eth_rx_adapter.h
>> b/lib/eventdev/rte_event_eth_rx_adapter.h
>> index 9237e198a7..758e1c5f56 100644
>> --- a/lib/eventdev/rte_event_eth_rx_adapter.h
>> +++ b/lib/eventdev/rte_event_eth_rx_adapter.h
>> @@ -553,6 +553,39 @@ int rte_event_eth_rx_adapter_queue_add(uint8_t
>> id,
>>  			int32_t rx_queue_id,
>>  			const struct rte_event_eth_rx_adapter_queue_conf
>> *conf);
>>
>> +/**
>> + * Add multiple receive queues to an event adapter.
>> + *
>> + * @param id
>> + *  Adapter identifier.
>> + *
>> + * @param eth_dev_id
>> + *  Port identifier of Ethernet device.
>> + *
>> + * @param rx_queue_id
>> + *  Array of Ethernet device receive queue indices.
>> + *  If nb_rx_queues is 0, then rx_queue_id is ignored.
>> + *
>> + * @param conf
>> + *  Array of additional configuration structures of type
>> + *  *rte_event_eth_rx_adapter_queue_conf*. conf[i] is used for
>> rx_queue_id[i].
>> + *  If nb_rx_queues is 0, then conf[0] is used for all Rx queues.
>> + *
>> + * @param nb_rx_queues
>> + *  Number of receive queues to add.
>> + *  If nb_rx_queues is 0, then all Rx queues configured for
>> + *  the device are added with the same configuration in conf[0].
>> + * @see RTE_EVENT_ETH_RX_ADAPTER_CAP_MULTI_EVENTQ
>> + *
>> + * @return
>> + *  - 0: Success, Receive queues added correctly.
>> + *  - <0: Error code on failure.
>> + */
>> +__rte_experimental
>> +int rte_event_eth_rx_adapter_queues_add(uint8_t id, uint16_t eth_dev_id,
>> int32_t rx_queue_id[],
>> +					const struct
>> rte_event_eth_rx_adapter_queue_conf conf[],
>> +					uint16_t nb_rx_queues);
>> +
>>  /**
>>   * Delete receive queue from an event adapter.
>>   *
>> diff --git a/lib/eventdev/version.map b/lib/eventdev/version.map index
>> 42a5867aba..44687255cb 100644
>> --- a/lib/eventdev/version.map
>> +++ b/lib/eventdev/version.map
>> @@ -153,6 +153,9 @@ EXPERIMENTAL {
>>  	__rte_eventdev_trace_port_preschedule_modify;
>>  	rte_event_port_preschedule;
>>  	__rte_eventdev_trace_port_preschedule;
>> +
>> +	# added in 25.03
>> +	rte_event_eth_rx_adapter_queues_add;
>>  };
>>
>>  INTERNAL {
>> --
>> 2.25.1