From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 1F4FC68F7 for ; Sat, 22 Apr 2017 00:31:55 +0200 (CEST) Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 21 Apr 2017 15:31:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.37,231,1488873600"; d="scan'208";a="92795442" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga006.fm.intel.com with ESMTP; 21 Apr 2017 15:31:54 -0700 Received: from fmsmsx154.amr.corp.intel.com (10.18.116.70) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.319.2; Fri, 21 Apr 2017 15:31:54 -0700 Received: from fmsmsx108.amr.corp.intel.com ([169.254.9.101]) by FMSMSX154.amr.corp.intel.com ([169.254.6.222]) with mapi id 14.03.0319.002; Fri, 21 Apr 2017 15:31:53 -0700 From: "Eads, Gage" To: Jerin Jacob , "dev@dpdk.org" CC: "Richardson, Bruce" , "Van Haaren, Harry" , "hemant.agrawal@nxp.com" , "nipun.gupta@nxp.com" , "Vangati, Narender" Thread-Topic: [RFC] [dpdk-dev] [PATCH] eventdev: abstract ethdev HW capability to inject packets to eventdev Thread-Index: AQHSuEcIkyNfYnH+IE2yHBu3NwyDhaHQULOA Date: Fri, 21 Apr 2017 22:31:52 +0000 Message-ID: <9184057F7FC11744A2107296B6B8EB1E01EA280D@FMSMSX108.amr.corp.intel.com> References: <20170418132307.1888-1-jerin.jacob@caviumnetworks.com> In-Reply-To: <20170418132307.1888-1-jerin.jacob@caviumnetworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.1.200.106] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [RFC] [PATCH] eventdev: abstract ethdev HW capability to inject packets to eventdev X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Apr 2017 22:31:56 -0000 Hi Jerin, Thanks for getting this ball rolling, and I agree that we need a solution t= hat covers the three cases you described. We've also been thinking about an= environment where devices (NIC Rx (or even Tx), crypto, or a timer "device= " that uses librte_timer to inject events) can plug in eventdev -- whether = through a direct connection to the event scheduler (case #3) or using softw= are to bridge the gap -- such that application software can have a consiste= nt view of device interfacing on different platforms. Some initial thoughts on your proposal: 1. I imagine that deploying these service functions at the granularity of a= core can be excessive on devices with few (<=3D 8) cores. For example, if = the crypto traffic rate is low then a cryptodev service function could be c= o-scheduled with other service functions and/or application work. I think w= e'll need a more flexible deployment of these service functions. 2. Knowing which device type a service function is for would be useful -- w= ithout it, it's not possible to assign the function to the NUMA node on whi= ch the device is located. 3. Placing the service core logic in the PMDs is nice in terms of applicati= on ease-of-use, but it forces PMD to write one-size-fits-all service core f= unctions, where, for example, the application's control of the NIC Rx funct= ionality is limited to the options that struct rte_event_queue_producer_con= f exports. An application may want customized service core behavior such as= : prioritized polling of Rx queues, using Rx queue interrupts for low traff= ic rate queues, or (for "closed system" eventdevs) control over whether/whe= n a service core drops events (and a way to notify applications of event dr= ops). For such cases, I think the appropriate solution is allow application= s to plug in their own service core functions (when hardware support isn't = present). Some of these thoughts are reflected in the eventdev_pipeline app[1] that H= arry submitted earlier today, like flexible service function deployment. In= that app, the user supplies a device coremask that can pin a service funct= ion to a core, multiplex multiple functions on the core, or even affinitize= the service function to multiple cores (using cmpset-based exclusion to en= sure it's executed by one lcore at a time). In thinking about this, Narende= r and I have envisioned something like a framework for eventdev application= s in which these service functions can be registered and (in a similar mann= er to eventdev_pipeline's service functions) executed. Thanks, Gage [1] http://dpdk.org/ml/archives/dev/2017-April/064511.html > -----Original Message----- > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > Sent: Tuesday, April 18, 2017 8:23 AM > To: dev@dpdk.org > Cc: Richardson, Bruce ; Van Haaren, Harry > ; hemant.agrawal@nxp.com; Eads, Gage > ; nipun.gupta@nxp.com; Jerin Jacob > > Subject: [RFC] [dpdk-dev] [PATCH] eventdev: abstract ethdev HW capabilit= y to > inject packets to eventdev > =20 > Some Ethdev Hardware is capable of injecting the events(Ethernet packets= ) to > eventdev without the need for dedicated service cores on Rx path. > Since eventdev API is device capability agnostic, we need to address thr= ee > combinations of ethdev and eventdev PMD drivers. > =20 > 1) Ethdev HW is not capable of injecting the packets and SW eventdev dri= ver(All > existing ethdev PMD + drivers/event/sw PMD combination) > 2) Ethdev HW is not capable of injecting the packets and not compatible = HW > eventdev driver(All existing ethdev PMD + driver/event/octeontx PMD > combination)=09 > 3) Ethdev HW is capable of injecting the packet to compatible HW eventde= v > driver. > =20 > This RFC attempts to abstract such capability disparity and have unified= way to > get the functionality in the application. > =20 > Detailed comments are added in the header file. > =20 > Example API usage: > =20 > - rte_eth_dev_configure(port,..); > - rte_eth_rx_queue_setup(port,..); > - rte_eth_dev_start(port,..); > =20 > - rte_event_dev_configure(dev_id,..); > =20 > struct rte_event_queue_producer_conf ethdev_conf =3D { > .event_type =3D RTE_EVENT_TYPE_ETHDEV; > .sched_type =3D RTE_SCHED_TYPE_ATOMIC; > .priority =3D RTE_EVENT_DEV_PRIORITY_LOWEST; > .ethdev.ethdev_port =3D port; > .ethdev.rx_queue_id =3D -1; > }; > =20 > struct rte_event_queue_conf =3D conf { > nb_producers =3D 1; > producers =3D ðdev_conf; > }; > =20 > - rte_event_queue_setup(dev_id, &conf,..); > - rte_event_port_setup(dev_id,..); > =20 > lcore_function_t *fns[RTE_MAX_LCORE]; > nb_serivice_cores =3D rte_event_dev_start(dev_id, fns); nscores =3D 0; > =20 > RTE_LCORE_FOREACH_SLAVE(lcore_id) { > if (nscores < nb_serivice_cores) { > rte_eal_remote_launch(fns[nscores], NULL, lcore_id); > nscores++; > } else { > rte_eal_remote_launch(normal_workers, NULL, lcore_id); > } > } > =20 > Another possible option is to move the lcore launch in the PMD and exten= d > enum rte_rmt_call_master_t to add SKIP_RUNNING to avoid application > browsing the service cores. > =20 > Comments? > =20 > Signed-off-by: Jerin Jacob > --- > lib/librte_eventdev/rte_eventdev.h | 87 > +++++++++++++++++++++++++++++++++++++- > 1 file changed, 85 insertions(+), 2 deletions(-) > =20 > diff --git a/lib/librte_eventdev/rte_eventdev.h > b/lib/librte_eventdev/rte_eventdev.h > index b8ed6ef08..b601b9ecd 100644 > --- a/lib/librte_eventdev/rte_eventdev.h > +++ b/lib/librte_eventdev/rte_eventdev.h > @@ -517,6 +517,64 @@ rte_event_dev_configure(uint8_t dev_id, > * @see rte_event_port_setup(), rte_event_port_link() > */ > =20 > +/** Event queue producers configuration structure. > + * The events are injected to event device through *enqueue* operation > +with > + * op =3D=3D RTE_EVENT_OP_NEW by event producers in the system. If the > +event > + * producers is an Ethernet device then eventdev PMD may operate in > +conjunction > + * with ethdev PMD to injects the events(Ethernet packets) to eventdev. > + * The event injection can happen in HW or SW or the combination of > +these two > + * based on the HW capabilities of target eventdev and ethdev PMDs. > + * If the eventdev PMD needs additional threads to inject the events, > + * a set of callbacks will be provided in rte_event_dev_start(). > + * Application must invoke each callback on each lcores to meet the > +required > + * functionality. > + * > + * @see rte_event_dev_start() > + * > + */ > +struct rte_event_queue_producer_conf { > + uint32_t event_type:4; > + /**< Event type to classify the event source. > + * @see RTE_EVENT_TYPE_ETHDEV, (RTE_EVENT_TYPE_*) > + */ > + uint8_t sched_type:2; > + /**< Scheduler synchronization type (RTE_SCHED_TYPE_*) > + * associated with flow id on a given event queue for the enqueue > + * operation. > + */ > + uint8_t priority; > + /**< Event priority relative to other events in the > + * event queue. The requested priority should in the > + * range of [RTE_EVENT_DEV_PRIORITY_HIGHEST, > + * RTE_EVENT_DEV_PRIORITY_LOWEST]. > + * The implementation shall normalize the requested > + * priority to supported priority value. > + * Valid when the device has > + * RTE_EVENT_DEV_CAP_EVENT_QOS capability. > + */ > + union { > + struct rte_event_ethdev_producer { > + uint16_t ethdev_port; > + /**< The port identifier of the Ethernet device */ > + int32_t rx_queue_id; > + /**< The index of the receive queue from which to > + * retrieve the input packets and inject to eventdev. > + * The value -1 denotes all the Rx queues configured > + * for the given ethdev_port are selected for retrieving > + * the input packets and then injecting the > + * events/packets to eventdev. > + * The rte_eth_rx_burst() result is undefined > + * if application invokes on bounded ethdev_port and > + * rx_queue_id. > + */ > + } ethdev; /* RTE_EVENT_TYPE_ETHDEV */ > + /**< Valid when event_type =3D=3D RTE_EVENT_TYPE_ETHDEV. > + * Implementation may use mbuff's rss->hash value as > + * flow_id for the enqueue operation. > + */ > + }; > +}; > + > /** Event queue configuration structure */ struct rte_event_queue_conf= { > uint32_t nb_atomic_flows; > @@ -545,6 +603,17 @@ struct rte_event_queue_conf { > * event device supported priority value. > * Valid when the device has RTE_EVENT_DEV_CAP_QUEUE_QOS > capability > */ > + uint16_t nb_producers; > + /**< The number of producers to inject the events with operation as > + * RTE_EVENT_OP_NEW to this event queue. > + * > + * @see rte_event_queue_producer_conf RTE_EVENT_OP_NEW > + */ > + struct rte_event_queue_producer_conf *producers; > + /**< Points to an array of *nb_producers* objects of type > + * *rte_event_queue_producer_conf* structure which contain > + * event queue producers configuration information. > + */ > }; > =20 > /** > @@ -590,7 +659,14 @@ rte_event_queue_default_conf_get(uint8_t dev_id, > uint8_t queue_id, > * @return > * - 0: Success, event queue correctly set up. > * - <0: event queue configuration failed > + * - -EDQUOT: Quota exceeded(Application tried to configure the same > producer > + * on more than one event queue) > + * - -EOPNOTSUPP: Implementation is not capable of pulling the events= from > + * the specific producer queue. On this error, application may try to > + * reconfigure the event queue with rx_queue_id as -1 in > + * struct rte_event_queue_producer_conf. > */ > + > int > rte_event_queue_setup(uint8_t dev_id, uint8_t queue_id, > const struct rte_event_queue_conf *queue_conf); @@ - > 755,13 +831,20 @@ rte_event_port_count(uint8_t dev_id); > * > * @param dev_id > * Event device identifier > + * @param[out] fns > + * Block of memory to insert callback pointers into. Application must= launch > + * these callbacks on available lcores using rte_eal_remote_launch() = or > + * equivalent. > + * The caller has to allocate *RTE_MAX_LCORE * sizeof(void\*)* bytes = to > + * store the callback pointers. > * @return > - * - 0: Success, device started. > + * - >=3D 0: Success, device started. > + * - Positive value: Number of callback pointers filled into the fns = array. > * - -ESTALE : Not all ports of the device are configured > * - -ENOLINK: Not all queues are linked, which could lead to deadloc= k. > */ > int > -rte_event_dev_start(uint8_t dev_id); > +rte_event_dev_start(uint8_t dev_id, lcore_function_t *fns[]); > =20 > /** > * Stop an event device. The device can be restarted with a call to > -- > 2.12.2