Re: [dpdk-dev] [RFC] eventdev: add event adapter for ethernet Rx queues

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Tue, 9 May 2017 15:38:46 -0500
> From: Gage Eads <gage.eads@intel.com>
> To: dev@dpdk.org
> CC: nikhil.rao@intel.com, jerin.jacob@caviumnetworks.com,
>  thomas@monjalon.net, bruce.richardson@intel.com,
>  harry.van.haaren@intel.com, hemant.agrawal@nxp.com, nipun.gupta@nxp.com,
>  narender.vangati@intel.com
> Subject: [RFC] eventdev: add event adapter for ethernet Rx queues
> X-Mailer: git-send-email 2.7.4
> 
> From: Nikhil Rao <nikhil.rao@intel.com>

Hi Nikhil and Gage,

Thanks for the RFC. A few questions and comments below.
Looks like SW has more constraints on event producer side, after we
finalize on this RFC(I guess only a few minor changes are only required).
I will align other[1] RFC based on _your_ RFC as we need to
converge on name space and we can't duplicate configs like struct
rte_event_dev_producer_conf etc

[1]
http://dpdk.org/ml/archives/dev/2017-May/065341.html

> 
> Eventdev-based networking applications require a component to dequeue
> packets from NIC Rx queues and inject them into eventdev queues[1]. While
> some platforms (e.g. Cavium Octeontx) do this operation in hardware, other
> platforms use software.
> 
> This RFC introduces an ethernet Rx event adapter and a proposed header
> file. This adapter service dequeues packets from ethernet devices and
> enqueues them to event devices.
> 
> The adapter is designed to work with the EAL service core proposal[2]. If
> an application determines that the adapter is required, it can register and
> launch it on a service core. Alternatively, this adapter can serve as a
> template for applications to design customer ethernet Rx event adapters
> better suited to their needs.
> 
> The adapter can service multiple ethernet devices and queues. Each queue is
> configured with a servicing weight to control the relative frequency with
> which the adapter polls the queue, and the event fields to use when
> constructing packet events. The adapter has two modes for programming an
> event's flow ID: use a static per-queue user-specified value or use the RSS
> hash.
> 
> A detailed description of the adapter is contained in the header's
> comments.
> 
> [1] http://dpdk.org/ml/archives/dev/2017-May/065341.html
> [2] http://dpdk.org/ml/archives/dev/2017-May/065207.html
> 
> Signed-off-by: Nikhil Rao <nikhil.rao@intel.com>
> Signed-off-by: Gage Eads <gage.eads@intel.com>
> ---
> +
> +/**
> + * @file
> + *
> + * RTE Ethernet Rx Adapter for Eventdev
> + *
> + * An eventdev-based packet processing application enqueues/dequeues mbufs
> + * to/from the event device. The ethernet Rx event adapter's role is to transfer
> + * mbufs from the ethernet receive queues managed by DPDK to an event device.
> + * The application uses the adapter APIs to configure the packet flow between
> + * the ethernet devices and event devices. The adapter is designed to work with
> + * the EAL service cores. The adapter's work can be parallelized by dividing the
> + * NIC Rx queues among multiple adapter services that run in parallel.
> + *
> + * Before using the adapter, the application needs to enumerate and configure
> + * the ethernet devices that it wishes to use. This is typically done using the
> + * following DPDK ethdev functions:
> + *  - rte_eth_dev_configure()
> + *  - rte_eth_tx_queue_setup()
> + *  - rte_eth_rx_queue_setup()
> + *  - rte_eth_dev_start()
> + *
> + * The application also configures an event device and creates event ports
> + * to interface with the event device. In addition to the event ports used by
> + * its packet processing functions, the application creates an event port
> + * to be used by this adapter.
> + *
> + * The ethernet Rx event adapter's functions are:
> + *  - rte_eth_rx_event_adapter_create()
> + *  - rte_eth_rx_event_adapter_free()
> + *  - rte_eth_rx_event_adapter_dev_add()
> + *  - rte_eth_rx_event_adapter_dev_del()
> + *  - rte_eth_rx_event_adapter_queue_add()
> + *  - rte_eth_rx_event_adapter_run()
> + *  - rte_eth_rx_event_adapter_stats_get()
> + *  - rte_eth_rx_event_adapter_stats_reset()
> + *
> + * The applicaton creates an event to ethernet adapter using
> + * rte_eth_rx_event_adapter_create(). The event device and event port
> + * identifiers are passed to this function. Next, the application adds ethernet
> + * devices to this adapter using rte_eth_rx_event_adapter_dev_add().
> + *
> + * The adapter needs to know which ethernet rx queues to poll for mbufs as well
> + * as event device parameters such as the event queue identifier, event
> + * priority and scheduling type that the adapter should use when constructing
> + * events. The rte_eth_rx_event_adapter_queue_add() function is provided for
> + * this purpose.
> + *
> + * At the time of adding an ethernet device receive queue, the application can
> + * also specify a static event flow id and set the
> + * RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID bit of the rx_queue_flags
> + * member of the rte_eth_rx_event_adapter_queue_config structure. If the
> + * RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID isn't set, the flow id is
> + * assigned the value of the RSS hash. The adapter generates the RSS hash if it
> + * hasn't been already computed by the NIC, based on source and destination
> + * IPv4/6 addresses, using the rte_softrss_be() routine included in the DPDK.
> + *
> + * The servicing weight parameter in the rte_eth_rx_event_adapter_queue_config
> + * intended to provide application control of the polling frequency of ethernet
> + * device receive queues, for example, the application may want to poll higher
> + * priority queues with a higher frequency but at the same time not starve
> + * lower priority queues completely. If this parameter is zero and the receive
> + * interrupt is enabled when configuring the device, the receive queue is
> + * interrupt driven; else, the queue is assigned a servicing weight of one.

Looks good.

> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#include <stdint.h>
> +#include <rte_mbuf.h>
> +#include <rte_eventdev.h>
> +
> +/* struct rte_eth_rx_event_adapter_queue_config flags definitions */
> +#define RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID	0x1
> +/*< This flag indicates the flow identifier is valid */
> +
> +struct rte_eth_rx_event_adapter_config {

Since this code is going to be at lib/librte_eventdev, We must start all
public symbols and file name with rte_event_*.

example:
May be this structure can be changed as rte_event_eth_rx_adapter_config


> +	uint8_t event_dev_id;
> +	/**< Event device identifier */
> +	uint8_t rx_event_port_id;
> +	/**< Event port identifier, the adapter enqueues mbuf events to this
> +	 * port
> +	 */
> +};
> +
> +struct rte_eth_rx_event_adapter_queue_config {
> +	uint32_t rx_queue_flags;
> +	 /**< Flags for handling received packets */

Better to add references with @see
example:
	@see RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID

> +	uint16_t servicing_weight;
> +	/**< Relative polling frequency of ethernet receive queue, if this
> +	 * is set to zero, the Rx queue is interrupt driven
> +	 */
> +	struct rte_event ev;
> +	/**<
> +	 *  The values from the following event fields will be used when
> +	 *  enqueuing mbuf events:
> +	 *   - event_queue_id: Targeted event queue ID for received packets.
> +	 *   - event_priority: Event priority of packets from this Rx queue in
> +	 *                     the event queue relative to other events.
> +	 *   - sched_type: Scheduling type for packets from this Rx queue.
> +	 *   - flow_id: If the RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID bit
> +	 *		is set in rx_queue_flags, this flow_id is used for all
> +	 *		packets received from this queue. Otherwise the flow ID
> +	 *		is set to the RSS hash.

This scheme is good. I was duplicating the elements in "struct
rte_event_dev_producer_conf"

IMO, We need to set ev.event_type == RTE_EVENT_TYPE_ETHDEV implicitly in
library.
You can mention that here as a info.

> +	 */
> +};
> +
> +struct rte_eth_rx_event_adapter_run_args {
> +	uint8_t id;
> +	/**< Adapter identifier */
> +	unsigned int max_nb_rx;
> +	/**< The adapter can return early if it has processed at least
> +	 * max_nb_rx mbufs. This isn't treated as a requirement; batching may
> +	 * cause the adapter to process more than max_nb_rx mbufs.
> +	 */
> +};
> +
> +struct rte_eth_rx_event_adapter_stats {
> +	uint64_t rx_poll_count;
> +	/**< Receive queue poll count across both polled and interrupt mode
> +	 * queues
> +	 */
> +	uint64_t rx_packets;
> +	/**< Received packet count */
> +	uint64_t rx_enq_fail;
> +	/**< Eventdev enqueue failed count */
> +	uint64_t rx_enq_retry;
> +	/**< Eventdev enqueue retry count */
> +};
> +
> +/**
> + * Create a new ethernet Rx event adapter with the specified identifier.
> + *
> + * @param adapter_id
> + *   Event adapter identifier.
> + * @param config
> + *   Event adapter config parameters.
> + * @return
> + *   - 0: Success
> + *   - <0: Error code on failure
> + */
> +int rte_eth_rx_event_adapter_create(
> +	uint8_t id,
> +	const struct rte_eth_rx_event_adapter_config *config);
> +

One adapter creates one service function. right?
It is good to mention the mapping.It is missing in the doc.

> +/**
> + * Free an event adapter
> + *
> + * @param id
> + *   Adapter identifier.
> + * @return
> + *   - 0: Success
> + *   - <0: Error code on failure
> + */
> +int rte_eth_rx_event_adapter_free(uint8_t id);
> +
> +/**
> + * Add eth device to the event adapter
> + *
> + * @param id
> + *   Adapter identifier.
> + * @param eth_dev_id
> + *  Port identifier of the Ethernet device.
> + * @return
> + *   - 0: Success
> + *   - <0: Error code on failure
> + */
> +int rte_eth_rx_event_adapter_dev_add(uint8_t id, uint8_t eth_dev_id);

rte_eth_event_rx_queue_add() also have eth_dev_id.What is the
significance of eth_dev_id here. Looks like eth_dev_id is a duplicate info.

if it is duplicate or it can be avoided then I propose to reduce the number
of APIs for easiness of application programming(i.e removing rte_eth_rx_event_adapter_dev_add,
rte_eth_rx_event_adapter_dev_del)

You can also mention the following for better clarify. If following is
true.If not, What do you think about, co-existence of poll and event mode?

The rte_eth_rx_burst() result is undefined if application invokes on
bounded ethdev_port and rx_queue_id.

> +
> +/**
> + * Delete eth device from an event adapter
> + *
> + * @param id
> + *   Adapter identifier.
> + * @param eth_dev_id
> + *  Port identifier of the Ethernet device.
> + * @return
> + *   - 0: Success
> + *   - <0: Error code on failure
> + */
> +int rte_eth_rx_event_adapter_dev_del(uint8_t id, uint8_t eth_dev_id);
> +
> +/**
> + * Add receive queue to event adapter
> + *
> + * @param id
> + *   Adapter identifier.
> + * @param eth_dev_id
> + *  Port identifier of Ethernet device.
> + * @param rx_queue_id
> + *  Ethernet device receive queue index.
> + * @param config
> + *  Additonal configuration structure.
> + * @return
> + *  - 0: Success, Receive queue added correctly.
> + *  - <0: Error code on failure.
> + */
> +int rte_eth_event_rx_queue_add(
> +	uint8_t id,
> +	uint8_t eth_dev_id,
> +	uint16_t rx_queue_id,

How about changing it as int32_t rx_queue_id and -1 to denote all Rx
queues configured for given eth_dev_id are added. This will avoid the
case where application needs to call this API one by one when application
interested in all the queues.

> +	const struct rte_eth_rx_event_adapter_queue_config *config);
> +

Don't we need rte_eth_event_rx_queue_del() for tear down?