From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id DD8217CE2 for ; Mon, 24 Jul 2017 12:32:40 +0200 (CEST) Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 24 Jul 2017 03:32:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.40,406,1496127600"; d="scan'208";a="111266843" Received: from irsmsx103.ger.corp.intel.com ([163.33.3.157]) by orsmga004.jf.intel.com with ESMTP; 24 Jul 2017 03:32:38 -0700 Received: from irsmsx102.ger.corp.intel.com ([169.254.2.211]) by IRSMSX103.ger.corp.intel.com ([169.254.3.9]) with mapi id 14.03.0319.002; Mon, 24 Jul 2017 11:32:36 +0100 From: "Van Haaren, Harry" To: Nipun Gupta , "Rao, Nikhil" , "jerin.jacob@caviumnetworks.com" CC: "Eads, Gage" , "dev@dpdk.org" , "thomas@monjalon.net" , "Richardson, Bruce" , Hemant Agrawal , "Vangati, Narender" , "Gujjar, Abhinandan S" Thread-Topic: [PATCH 1/2] eventdev: add event adapter for ethernet Rx queues Thread-Index: AQHS9loYW0A+jeziw0SDbe1hI3ujH6JizMgAgAAULXA= Date: Mon, 24 Jul 2017 10:32:36 +0000 Message-ID: References: <29140c16-909a-1b9a-7391-481f900bd13c@intel.com> <1499377952-5306-1-git-send-email-nikhil.rao@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMmFmNTRlZDAtZTIzMS00NDNhLWE3MzYtYWFiNmU3NTNkMGUzIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlwvV0UyWWJtRVJzNlR1TW5qVkk5QzFMcVpKZDhwZFhRa0VtWnp5azFCR1QwPSJ9 x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 10.0.102.7 dlp-reaction: no-action x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH 1/2] eventdev: add event adapter for ethernet Rx queues X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jul 2017 10:32:41 -0000 > From: Nipun Gupta [mailto:nipun.gupta@nxp.com] > Sent: Monday, July 24, 2017 11:11 AM > To: Rao, Nikhil ; jerin.jacob@caviumnetworks.com > Cc: Eads, Gage ; dev@dpdk.org; thomas@monjalon.net; = Richardson, Bruce > ; Van Haaren, Harry ; Hemant > Agrawal ; Vangati, Narender ; Gujjar, > Abhinandan S > Subject: RE: [PATCH 1/2] eventdev: add event adapter for ethernet Rx queu= es >=20 > Hi Nikhil/Edas, Hi All, I'll reply on the service-cores topic, and leave the config to those more i= nvolved and familiar with them than I :) > > -----Original Message----- > > From: Nikhil Rao [mailto:nikhil.rao@intel.com] > > Sent: Friday, July 07, 2017 3:23 > > To: jerin.jacob@caviumnetworks.com > > Cc: gage.eads@intel.com; dev@dpdk.org; thomas@monjalon.net; > > bruce.richardson@intel.com; harry.van.haaren@intel.com; Hemant Agrawal > > ; Nipun Gupta ; > > narender.vangati@intel.com; Nikhil Rao ; Abhinand= an > > Gujjar > > Subject: [PATCH 1/2] eventdev: add event adapter for ethernet Rx queues > > > > Eventdev-based networking applications require a component to dequeue > > packets from NIC Rx queues and inject them into eventdev queues[1]. Whi= le > > some platforms (e.g. Cavium Octeontx) do this operation in hardware, ot= her > > platforms use software. > > > > This patchset introduces an ethernet Rx event adapter that dequeues pac= kets > > from ethernet devices and enqueues them to event devices. It is based o= n > > a previous RFC[2]. > > > > The adapter is designed to work with the EAL service core[3]. If > > an application determines that the adapter is required, it can register= and > > launch it on a service core. Alternatively, this adapter can serve as a > > template for applications to design customer ethernet Rx event adapters > > better suited to their needs. > > > > The adapter can service multiple ethernet devices and queues. Each queu= e is > > configured with a servicing weight to control the relative frequency wi= th > > which the adapter polls the queue, and the event fields to use when > > constructing packet events. The adapter has two modes for programming a= n > > event's flow ID: use a static per-queue user-specified value or use the= RSS > > hash. > > > > A detailed description of the adapter is contained in the header's > > comments. > > > > [1] http://dpdk.org/ml/archives/dev/2017-May/065341.html > > [2] http://dpdk.org/ml/archives/dev/2017-May/065539.html > > [3] http://dpdk.org/ml/archives/dev/2017-July/069782.html > > > > Signed-off-by: Nikhil Rao > > Signed-off-by: Gage Eads > > Signed-off-by: Abhinandan Gujjar > > --- > > > > v2: > > Thanks Jerin for review - below is a list of changes you > > suggested. > > > > - all public symbols are started with rte_event_. > > - Add Doxygen reference with @see. > > - Mention setting of ev.event_type. > > - Mention adapter to service function mapping. > > - Remove rte_eth_rx_event_adapter_dev_add/del(). > > - Change rx_queuee_id to int32_t and use -1 to denote all Rx queues. > > - Add rte_eth_event_rx_queue_del(). > > > > Other changes > > - Remove adapter's run function (rte_event_eth_rx_adapter_run()) from > > the public interface. The adapter internally uses it to create a > > service. > > - Add a blocked cycle count to stats. Further description is contained > > in the header. > > - Minor struct renames rte_event_eth_rx_adapter_config -> .._conf > > --- > > lib/librte_eventdev/rte_event_eth_rx_adapter.h | 301 +++++++++ > > lib/librte_eventdev/rte_event_eth_rx_adapter.c | 825 > > +++++++++++++++++++++++++ > > lib/librte_eventdev/rte_eventdev_version.map | 13 + > > lib/Makefile | 2 +- > > lib/librte_eventdev/Makefile | 2 + > > 5 files changed, 1142 insertions(+), 1 deletion(-) > > create mode 100644 lib/librte_eventdev/rte_event_eth_rx_adapter.h > > create mode 100644 lib/librte_eventdev/rte_event_eth_rx_adapter.c > > > > diff --git a/lib/librte_eventdev/rte_event_eth_rx_adapter.h > > b/lib/librte_eventdev/rte_event_eth_rx_adapter.h > > new file mode 100644 > > index 000000000..5ccd0bd24 > > --- /dev/null > > +++ b/lib/librte_eventdev/rte_event_eth_rx_adapter.h > > @@ -0,0 +1,301 @@ > > +/* > > + * Copyright(c) 2017 Intel Corporation. All rights reserved. > > + * All rights reserved. > > + * > > + * Redistribution and use in source and binary forms, with or withou= t > > + * modification, are permitted provided that the following condition= s > > + * are met: > > + * > > + * * Redistributions of source code must retain the above copyrigh= t > > + * notice, this list of conditions and the following disclaimer. > > + * * Redistributions in binary form must reproduce the above copyr= ight > > + * notice, this list of conditions and the following disclaimer = in > > + * the documentation and/or other materials provided with the > > + * distribution. > > + * * Neither the name of Intel Corporation nor the names of its > > + * contributors may be used to endorse or promote products deriv= ed > > + * from this software without specific prior written permission. > > + * > > + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND > > CONTRIBUTORS > > + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > > + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND > > FITNESS FOR > > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE > > COPYRIGHT > > + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, > > INCIDENTAL, > > + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT > > NOT > > + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS > > OF USE, > > + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND > > ON ANY > > + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR > > TORT > > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF > > THE USE > > + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH > > DAMAGE. > > + */ > > + > > +#ifndef _RTE_EVENT_ETH_RX_ADAPTER_ > > +#define _RTE_EVENT_ETH_RX_ADAPTER_ > > + > > +/** > > + * @file > > + * > > + * RTE Event Ethernet Rx Adapter > > + * > > + * An eventdev-based packet processing application enqueues/dequeues m= bufs > > + * to/from the event device. The ethernet Rx event adapter's role is t= o transfer > > + * mbufs from the ethernet receive queues managed by DPDK to an event > > device. > > + * The application uses the adapter APIs to configure the packet flow = between > > + * the ethernet devices and event devices. The adapter is designed to = work with > > + * the EAL service cores. The adapter's work can be parallelized by di= viding the > > + * NIC Rx queues among multiple adapter services that run in parallel. >=20 > First of all, apologies for commenting late on this. I am going through t= his patchset and > have some concerns over this. > It is mentioned that "The adapter is designed to work with * the EAL serv= ice cores" > Does this mean that the adapter cannot function without service core? > In the case where Ethdev HW is capable of injecting the packet to compati= ble HW eventdev > driver, there is no service > required. It seems this patchset does not take care of this use-case or m= aybe I am missing > something here? The "service cores" is a method of abstracting lcore requirements. In short= , with a SW PMD, a service core is required to perform the event scheduling= . With the HW use case as you describe, no core is required. So in the HW c= ase, no service cores are allocated in DPDK, and as a result, all your core= s are available to the application. If I understand correctly, the generic "Event Ethernet RX Adapter" will reg= ister a service which needs to be run (by a service core) in order to perfo= rm SW polling of the NIC, and enqueue to the eventdev. In order to support the hardware use-case, no service is registered and no = service lcores required. The HW is allowed to perform the Ethdev -> Eventde= v enqueue. Hope that makes sense! -Harry > > + * > > + * Before using the adapter, the application needs to enumerate and co= nfigure > > + * the ethernet devices that it wishes to use. This is typically done = using the > > + * following DPDK ethdev functions: > > + * - rte_eth_dev_configure() > > + * - rte_eth_tx_queue_setup() > > + * - rte_eth_rx_queue_setup() > > + * - rte_eth_dev_start() > > + * > > + * The application also configures an event device and creates event p= orts > > + * to interface with the event device. In addition to the event ports = used by > > + * its packet processing functions, the application creates an event p= ort > > + * to be used by this adapter. > > + * > > + * The ethernet Rx event adapter's functions are: > > + * - rte_event_eth_rx_adapter_create() > > + * - rte_event_eth_rx_adapter_free() > > + * - rte_event_eth_rx_adapter_queue_add() > > + * - rte_event_eth_rx_adapter_queue_del() > > + * - rte_event_eth_rx_adapter_stats_get() > > + * - rte_event_eth_rx_adapter_stats_reset() > > + * > > + * The applicaton creates an event to ethernet adapter using > > + * rte_event_eth_rx_adapter_create(). The event device and event port > > + * identifiers are passed to this function. > > + * > > + * The adapter needs to know which ethernet rx queues to poll for mbuf= s as > > well > > + * as event device parameters such as the event queue identifier, even= t > > + * priority and scheduling type that the adapter should use when const= ructing > > + * events. The rte_event_eth_rx_adapter_queue_add() function is provid= ed for > > + * this purpose. > > + * > > + * At the time of adding an ethernet device receive queue, the applica= tion can > > + * also specify a static event flow id and set the > > + * RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID bit of the > > rx_queue_flags > > + * member of the rte_event_eth_rx_adapter_queue_conf structure. If the > > + * RTE_ETH_RX_EVENT_ADAPTER_QUEUE_FLOW_ID_VALID isn't set, the flow > > id is > > + * assigned the value of the RSS hash. The adapter generates the RSS h= ash if it > > + * hasn't been already computed by the NIC, based on source and destin= ation > > + * IPv4/6 addresses, using the rte_softrss_be() routine included in th= e DPDK. > > + * > > + * The servicing weight parameter in the > > rte_event_eth_rx_adapter_queue_conf > > + * intended to provide application control of the polling frequency of= ethernet > > + * device receive queues, for example, the application may want to pol= l higher > > + * priority queues with a higher frequency but at the same time not st= arve > > + * lower priority queues completely. If this parameter is zero and the= receive > > + * interrupt is enabled when configuring the device, the receive queue= is > > + * interrupt driven; else, the queue is assigned a servicing weight of= one. > > + * > > + * Note: Interrupt driven receive queues are currentely unimplemented. > > + */ > > + > > +#ifdef __cplusplus > > +extern "C" { > > +#endif > > + > > +#include > > +#include > > + > > +#include "rte_eventdev.h" > > + > > + > > +#define RTE_MAX_EVENT_ETH_RX_ADAPTER_INSTANCE 32 > > + > > +/* struct rte_event_eth_rx_adapter_queue_conf flags definitions */ > > +#define RTE_EVENT_ETH_RX_ADAPTER_QUEUE_FLOW_ID_VALID 0x1 > > +/**< This flag indicates the flow identifier is valid > > + * @see rte_event_eth_rx_adapter_queue_conf > > + */ >=20 > >=20 > > + > > +static int > > +_rte_event_eth_rx_adapter_queue_add(struct rte_event_eth_rx_adapter > > *rx_adapter, > > + struct eth_device_info *dev_info, > > + uint16_t rx_queue_id, > > + const struct > > rte_event_eth_rx_adapter_queue_conf *conf) > > +{ > > + int ret; > > + struct eth_rx_queue_info *queue_info; > > + const struct rte_event *ev; > > + > > + if (rx_queue_id >=3D dev_info->dev->data->nb_rx_queues) > > + return -EINVAL; > > + > > + queue_info =3D &dev_info->rx_queue[rx_queue_id]; > > + if (queue_info->queue_enabled) > > + return -EEXIST; > > + > > + ev =3D &conf->ev; > > + memset(queue_info, 0, sizeof(*queue_info)); > > + queue_info->event_queue_id =3D ev->queue_id; > > + queue_info->sched_type =3D ev->sched_type; > > + queue_info->priority =3D ev->priority; > > + queue_info->wt =3D conf->servicing_weight; > > + > > + if (queue_info->wt =3D=3D 0) { > > + struct rte_eth_dev_data *data =3D dev_info->dev->data; > > + > > + /* If Rx interrupts are disabled set wt =3D 1 */ > > + queue_info->wt =3D !data->dev_conf.intr_conf.rxq; > > + } > > + > > + if (conf-> > > + rx_queue_flags & > > RTE_EVENT_ETH_RX_ADAPTER_QUEUE_FLOW_ID_VALID) { > > + queue_info->flow_id =3D ev->flow_id; > > + queue_info->flow_id_mask =3D ~0; > > + } > > + > > + queue_info->queue_enabled =3D true; > > + rx_adapter->num_rx_polled++; > > + ret =3D eth_poll_wrr_calc(rx_adapter); > > + if (ret) { > > + rx_adapter->num_rx_polled--; > > + queue_info->queue_enabled =3D false; > > + return ret; > > + } > > + > > + return 0; > > +} > > + > > +int > > +rte_event_eth_rx_adapter_queue_add(uint8_t id, > > + uint8_t eth_dev_id, > > + int32_t rx_queue_id, > > + const struct > > rte_event_eth_rx_adapter_queue_conf *conf) > > +{ > > + int ret =3D 0; > > + struct rte_event_eth_rx_adapter *rx_adapter; > > + struct eth_device_info *dev_info; > > + uint32_t i, j; > > + > > + rx_adapter =3D rte_event_eth_rx_adapter[id]; > > + if (!valid_id(id) || !rte_event_eth_rx_adapter || !rx_adapter || > > + eth_dev_id >=3D rte_eth_dev_count() || !conf) > > + return -EINVAL; > > + > > + rte_spinlock_lock(&rx_adapter->rx_lock); > > + > > + dev_info =3D &rx_adapter->eth_devices[eth_dev_id]; > > + if (rx_queue_id =3D=3D -1) { > > + for (i =3D 0; i < dev_info->dev->data->nb_rx_queues; i++) { > > + ret =3D > > + _rte_event_eth_rx_adapter_queue_add(rx_adapter, > > dev_info, i, > > + conf); > > + if (ret) { > > + for (j =3D 0; j < i; j++) > > + > > _rte_event_eth_rx_adapter_queue_del(rx_adapter, > > + > > dev_info, > > + j); > > + } > > + } > > + > > + } else { > > + ret =3D _rte_event_eth_rx_adapter_queue_add(rx_adapter, > > dev_info, > > + > > (uint16_t)rx_queue_id, > > + conf); > > + } > > + > > + rte_spinlock_unlock(&rx_adapter->rx_lock); > > + > > + return ret; > > +} >=20 > Looking at the rte_event_eth_rx_adapter_queue_add & event_eth_rx_adapter_= service_func > it seems that this indeed will not fit with the cases where ethdev is cap= able of enqueing > packets > to the eventdev (as was mentioned in Jerin's first RFC). >=20 > In case of hardware based eventdev and queues, these function should also= invoke > respective PMD > HW configs. e.g. In queue case - rte_eventdev and rte_ethdev - both PMDs = at hw level shall > be configured. >=20 > A typical eventdev hardware will require queues of eth devices will be co= nfigured to > directly attach to > eventdev in the hardware. >=20 > Mapping it to NXP eventdev, enabling this functionality requires some con= figuration where > dev private > information of both the devices (event dev and eth dev) is required at th= e same time, > and the final configuration is provided via eth device to H/W. So, this r= equire inter > device communication in DPDK. >=20 > Jerin, >=20 > I have an impression that Cavium hardware has H/W capability to inject pa= ckets from > Ethernet > devices to event devices? If yes, how do you plan to support it? >=20 > Thanks, > Nipun