From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CBF79A0A0A; Wed, 24 Mar 2021 06:39:19 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 67B58140D17; Wed, 24 Mar 2021 06:39:19 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id C8C994067B for ; Wed, 24 Mar 2021 06:39:17 +0100 (CET) IronPort-SDR: 16O59w1E//rtsKFe1xIPrZ9K+VuXD6KU9v0XwyRXt8I7qAWL4RPYUCAd7eEDa6F/b3MtQoJetZ OQIbZVdA0VnQ== X-IronPort-AV: E=McAfee;i="6000,8403,9932"; a="251992873" X-IronPort-AV: E=Sophos;i="5.81,272,1610438400"; d="scan'208";a="251992873" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Mar 2021 22:39:16 -0700 IronPort-SDR: 3UWE4jeItjifdqS/sS65dT03giH39bmYChX3HHMd0sleDpsCMkth1p6iFSSlwwdWzmV+Ywa416 y1/bdx/j+9qw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,272,1610438400"; d="scan'208";a="435860242" Received: from fmsmsx606.amr.corp.intel.com ([10.18.126.86]) by fmsmga004.fm.intel.com with ESMTP; 23 Mar 2021 22:39:16 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx606.amr.corp.intel.com (10.18.126.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Tue, 23 Mar 2021 22:39:15 -0700 Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Tue, 23 Mar 2021 22:39:15 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2 via Frontend Transport; Tue, 23 Mar 2021 22:39:15 -0700 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (104.47.59.177) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2106.2; Tue, 23 Mar 2021 22:39:15 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NadX3PWIzKJnLOrfbPJSdI+1+/1H7Ig+VYd4svubQ0WcoNLTwxUfvyPxR3c6ySFPNhR9ErPs7Va8BcZf+FEcCV/oXaZKKfw32K2zIV8ieyB1sXHOcV46vpiuLEiivjqqgYCBpsd/CTZCaUIy441K4jEeJ/4oKJXWh7KiDCi9mz2uwEEf+eTUqTHomxTNH3bxXMjKzxpWjTBu3sms2sid63H0anUMA/uaQSryK/UDsq//l9Q4aTP6zXmI+yWiojFe5oIR9emtbsH/K7jOQP4w9ykpanOVqguAOL9BJXalVi4WugcW4Ko6QoAgnTu9v5a1i+aFF09Jz135e9QbHio9HA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LstVkTkb3iOX9rm/Y5HXhYkPNqD/BuFCfeZm4jUwGBY=; b=ZDYryZUAAA0VgC7u0s8AU3EajE+X8wB8u+XpUnDUIgf3A42AQZcVPfjG1t1ZiEC3LYh5kofR47x7AGs9JZ42eyKrgV5zyjVLKsZ3kqke9kbQobVz1OtnjrQnUagigp3JFXtvLNjLp7KaCQ3WZuSVQYqLFIkrxhfovfzdBcJ8/n1oLPlAQIoa0DJkBL+okj083OLpDVBbf2kfIpOHmo+4cYwj6uDzknDjtFqIzd8L0N0lsANg+o/barrnHYZStq6DP4VFtMi/HX2l8Om44fFb0nGOpuciB/e3dM0u6vIOS29K3CkOSPghPl1F2aGfhlvcCCj3Opg83HD5DWFppdi/vA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LstVkTkb3iOX9rm/Y5HXhYkPNqD/BuFCfeZm4jUwGBY=; b=iwFTPQzacA351nChFw61Q5CgvdSdGT4thdxjyWZP8lHAYjkugmtbmuB8PiuhmUtXDj1hyYUyTN5sJ5EBHbOsLmjCRgyRCXus+x+hZs8SEqZpNcocxDCiYO1/LCJxZsaBg1a098vgyRUEu5bO2R+li9cAu2e13Sg03QRLj/f9ttg= Received: from SN6PR11MB3117.namprd11.prod.outlook.com (2603:10b6:805:d7::32) by SA0PR11MB4688.namprd11.prod.outlook.com (2603:10b6:806:72::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.25; Wed, 24 Mar 2021 05:39:13 +0000 Received: from SN6PR11MB3117.namprd11.prod.outlook.com ([fe80::4579:2bc0:3dc8:8b37]) by SN6PR11MB3117.namprd11.prod.outlook.com ([fe80::4579:2bc0:3dc8:8b37%5]) with mapi id 15.20.3955.027; Wed, 24 Mar 2021 05:39:13 +0000 From: "Jayatheerthan, Jay" To: "pbhagavatula@marvell.com" , "jerinj@marvell.com" , "Carrillo, Erik G" , "Gujjar, Abhinandan S" , "McDaniel, Timothy" , "hemant.agrawal@nxp.com" , "Van Haaren, Harry" , mattias.ronnblom , "Ma, Liang J" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v5 0/8] Introduce event vectorization Thread-Index: AQHXIGtlTTjEqPmdtkaDA0dMIm2z0qqSm8jg Date: Wed, 24 Mar 2021 05:39:13 +0000 Message-ID: References: <20210319205718.1436-1-pbhagavatula@marvell.com> <20210324050525.4489-1-pbhagavatula@marvell.com> In-Reply-To: <20210324050525.4489-1-pbhagavatula@marvell.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [122.167.113.74] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 043620d0-3c5e-4d3e-0daf-08d8ee87281f x-ms-traffictypediagnostic: SA0PR11MB4688: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3826; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: jRRFF6pMjDv8bCAydDIi/SMW3e0vwPOcHdWZTKzbrgmtzzLoYPVTbGti3T4V56+mpYQ1Pp9uKWnU9FniJhMKnc3IRuX8vS052HBcaGp3lStqfFl0ovLghRA1EyWFFcMS+w/0cQCwjMIKmg1oCSnfRJxxSA8vdKQXjnIHzUh68P0BteF/pj7kxViFnC1ASJRnBA6vwjQa0HVXwx9+JVAbQazhIPj5leP6RsyaF7Eolc7oig6sX8Q+wgLwGO2Ihl+Uj2T4Xhvsu8Nu/lLGA8FBl+hCAnEJOZkaOMCJntre7Ea3gFdusQm9o1I4cMeGN9/CvYYEuEffYl1Sud/oEZ89GFIxxKQBgQqwwAw8QaKcAwiFCyVoJCt1uEzSCihK7aaKYOP2RXwoRkWrtF1WBlhDp3IuDTC5rbqdybiJf08hAeM3cdVBJs9jrsofPKpp2E6gqWp0NvxiIl4vjn0JG3WT5/ImWfVT1iMZCsFx4wP4a/nwoPwckY6m49WcPzkvcee0xa6q6vCSDGCKeBewedVYe/miodOegR/C7ypdZfh1crNH9J4sueMQ6eDalFAl8DoZtuFp6Q3eCVuPcVshulpn6fkjy8a4Oy6DG6QidDBSvoYHAEXzFcmP9m07HLFRq/QeIUlk1u5tQpScLaDrwTbttEnSUDpKR5k8GAtOAlAXGf8= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3117.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(396003)(136003)(376002)(346002)(366004)(316002)(478600001)(110136005)(71200400001)(6506007)(7696005)(8676002)(8936002)(83380400001)(921005)(6636002)(53546011)(4326008)(66946007)(86362001)(64756008)(9686003)(38100700001)(66476007)(186003)(52536014)(66556008)(2906002)(33656002)(76116006)(66446008)(26005)(55016002)(5660300002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?eXtXY2txrK+9X0anVBkz3UFB8d/I6q+6RZwPE5XpkMZzhIlVzXt6Gv5NMQ7/?= =?us-ascii?Q?48E3tJHwyzk9S3ZYjfq91XzsyQl6Dz1Pl8ee52n29ZnxR94nRg4jeXczcDOH?= =?us-ascii?Q?QwKh8+bbuFkiJf920AnXgC+dmd39fZLi73HCING+lPJsz7QtNnqWwy61oCfU?= =?us-ascii?Q?/isi3/pLQpeV2f7cy91bR7CTMUBLNEaKsh5/1N1fcdQvgNoth0yPOEYJkufz?= =?us-ascii?Q?mMPZ/gYDo3eaWDAJ4yOVPSleWuMIvwdWlonODHq+xYyLA2dGXGExP6WlFKTN?= =?us-ascii?Q?M5kLEJfoJeI+4x2m9BjmqmZhkBRdOSzHpO28U9Tj2GGpzKNW/+fNTQoPHhhd?= =?us-ascii?Q?UdT9wX8mswVmmdH7NlBTKW65tl6TIkzSu5l8i+kI9WqmZlKi5/KmIPWEtEtT?= =?us-ascii?Q?PhQuzilxc5F/28Y1nU9RCRq2hyE7lWffnWM8+vMTHbb+78OznVFOHekM9Kvn?= =?us-ascii?Q?Hn2Hp+lD9QlZvhB+/p9CbrGodRQv+ahr83cXc9nXFtOpMIZYlGwEJ4aR2qh9?= =?us-ascii?Q?3qGnSZZ9YIQ2pv8PPpWXn9ndcpe+ynYY8lqGrGlUpOqG0Vtn8YGVIgL7eULR?= =?us-ascii?Q?UzLC3y6KqHC3+khcgKoUDg/bA+WpYcog8Q9fZNn/A8xkHS8BCGbM29Bc/zfc?= =?us-ascii?Q?e7SpMPvD7+7DeLgvTwG7HY0bEW+E+cuiAVsGYlNcuHwT6mPc9M/zifasujLA?= =?us-ascii?Q?PmC37wKQtAEUTQo5PcAe/NCjAZHd9fmbpaUo60Vv7H2Lj0wGUdAUvQ9b4dTj?= =?us-ascii?Q?zxWMlk/h887UI4prTWydIRVb2QIDSbycmZj2EbGm6yZ6S56/G9d5T3AdgKNh?= =?us-ascii?Q?FlKZZYqUH05XVwT1CjAQEk6pBEeSIxJzceZ8DN4YECyO54uPX32e11591r9Z?= =?us-ascii?Q?MnUtI1Rf6Tc9qTvdlTREBAgIlcCkUHj06TzT2mGUJdmWgXvVEgUuAqHNhuYR?= =?us-ascii?Q?HTnTF7j7clDQaxEt7X2HVY9xPZY0xsOSDat98qIHa/uQMcsAsbPNxrh1vLqT?= =?us-ascii?Q?fWKo1rbR8KcGwnwgVDJKQswgd4Cvyqj+AfSrX09+alPf4V2tbd2yLAiobSv5?= =?us-ascii?Q?aMZwVhMwOO6UurSB0f7qIRVeQKNiuaqBUpBVPbmbgOOXorKQndlclBSqMBLY?= =?us-ascii?Q?yZxjE/NutKDCG8WEDxYq3RgONQxaCx1aYFYzldxMzhpCicq5+93CU+rA6Uhi?= =?us-ascii?Q?ewF6pkp/mNnanxfsGImG4Hp+XxaunXQLlnoaH8xEpMUICYrqqITqMPQuU71Y?= =?us-ascii?Q?hLwPQmgpqtIb8QsUaV3/dusA+KiJMvfTxjbxAAWiAq/VBJY4L1XTkFznYzcy?= =?us-ascii?Q?HfL5B4zUF1LzhU9ezSFrZTwO?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3117.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 043620d0-3c5e-4d3e-0daf-08d8ee87281f X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Mar 2021 05:39:13.1286 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 9gkxQt0RYCSAwo+TtpoxmdbH7ZdkFaHpLxdf24KCldSze6zBojYyg62cRs+PJTCNFysi/cRhRgChLL5MYorJ1NMLsotFx+kbsJPa+LUcM/4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4688 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v5 0/8] Introduce event vectorization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: pbhagavatula@marvell.com > Sent: Wednesday, March 24, 2021 10:35 AM > To: jerinj@marvell.com; Jayatheerthan, Jay ;= Carrillo, Erik G ; Gujjar, Abhinandan > S ; McDaniel, Timothy ; hemant.agrawal@nxp.com; Van Haaren, Harry > ; mattias.ronnblom ; Ma, Liang J > Cc: dev@dpdk.org; Pavan Nikhilesh > Subject: [dpdk-dev] [PATCH v5 0/8] Introduce event vectorization >=20 > From: Pavan Nikhilesh >=20 > In traditional event programming model, events are identified by a > flow-id and a uintptr_t. The flow-id uniquely identifies a given event > and determines the order of scheduling based on schedule type, the > uintptr_t holds a single object. >=20 > Event devices also support burst mode with configurable dequeue depth, > i.e. each dequeue call would return multiple events and each event > might be at a different stage of the pipeline. > Having a burst of events belonging to different stages in a dequeue > burst is not only difficult to vectorize but also increases the scheduler > overhead and application overhead of pipelining events further. > Using event vectors we see a performance gain of ~628% as shown in [1]. This is very impressive performance boost. Thanks so much for putting this = patchset together! Just curious, was any performance measurement done for e= xisting applications (non-vector)? >=20 > By introducing event vectorization, each event will be capable of holding > multiple uintptr_t of the same flow thereby allowing applications > to vectorize their pipeline and reduce the complexity of pipelining > events across multiple stages. This also reduces the complexity of handli= ng > enqueue and dequeue on an event device. >=20 > Since event devices are transparent to the events they are scheduling > so the event producers such as eth_rx_adapter, crypto_adapter , etc.. > are responsible for vectorizing the buffers of the same flow into a singl= e > event. >=20 > The series also breaks ABI in the patch [8/8] which is targetted to the > v21.11 release. >=20 > The dpdk-test-eventdev application has been updated with options to test > multiple vector sizes and timeouts. >=20 > [1] > As for performance improvement, with a ARM Cortex-A72 equivalent processe= r, > software event device (--vdev=3Devent_sw0), single worker core, single st= age > and using one service core for Rx adapter, Tx adapter, Scheduling. >=20 > Without event vectorization: > ./build/app/dpdk-test-eventdev -l 7-23 -s 0x700 --vdev=3D"event_sw0" = -- > --prod_type_ethdev --nb_pkts=3D0 --verbose 2 --test=3Dpipeline_q= ueue > --stlist=3Da --wlcores=3D20 > Port[0] using Rx adapter[0] configured > Port[0] using Tx adapter[0] Configured > 4.728 mpps avg 4.728 mpps Is this number before the patchset? If so, it would help put similar number= with the patchset but not using vectorization feature. >=20 > With event vectorization: > ./build/app/dpdk-test-eventdev -l 7-23 -s 0x700 --vdev=3D"event_sw0" = -- > --prod_type_ethdev --nb_pkts=3D0 --verbose 2 --test=3Dpipeline_qu= eue > --stlist=3Da --wlcores=3D20 --enable_vector --nb_eth_queues 1 > --vector_size 256 > Port[0] using Rx adapter[0] configured > Port[0] using Tx adapter[0] Configured > 34.383 mpps avg 34.383 mpps >=20 > Having dedicated service cores for each Rx queues and tweaking the vector= , > dequeue burst size would further improve performance. >=20 > API usage is shown below: >=20 > Configuration: >=20 > struct rte_event_eth_rx_adapter_event_vector_config vec_conf; >=20 > vector_pool =3D rte_event_vector_pool_create("vector_pool", > nb_elem, 0, vector_size, socket_id); >=20 > rte_event_eth_rx_adapter_create(id, event_id, &adptr_conf); > rte_event_eth_rx_adapter_queue_add(id, eth_id, -1, &queue_conf); > if (cap & RTE_EVENT_ETH_RX_ADAPTER_CAP_EVENT_VECTOR) { > vec_conf.vector_sz =3D vector_size; > vec_conf.vector_timeout_ns =3D vector_tmo_nsec; > vec_conf.vector_mp =3D vector_pool; > rte_event_eth_rx_adapter_queue_event_vector_config(id, > eth_id, -1, &vec_conf); > } >=20 > Fastpath: >=20 > num =3D rte_event_dequeue_burst(event_id, port_id, &ev, 1, 0); > if (!num) > continue; >=20 > if (ev.event_type & RTE_EVENT_TYPE_VECTOR) { > switch (ev.event_type) { > case RTE_EVENT_TYPE_ETHDEV_VECTOR: > case RTE_EVENT_TYPE_ETH_RX_ADAPTER_VECTOR: > struct rte_mbuf **mbufs; >=20 > mbufs =3D ev.vector_ev->mbufs; > for (i =3D 0; i < ev.vector_ev->nb_elem; i++) > //Process mbufs. > break; > case ... > } > } > ... >=20 > v5 Changes: > - Make `rte_event_vector_pool_create non-inline` to ease ABI stability.(R= ay) > - Move `rte_event_eth_rx_adapter_queue_event_vector_config` and > `rte_event_eth_rx_adapter_vector_limits_get` implementation to the patc= h > where they are initially defined.(Ray) > - Multiple gramatical and style fixes.(Jerin) > - Add missing release notes.(Jerin) >=20 > v4 Changes: > - Fix missing event vector structure in event structure.(Jay) >=20 > v3 Changes: > - Fix unintended formatting changes. >=20 > v2 Changes: > - Multiple gramatical and style fixes.(Jerin) > - Add parameter to define vector size in power of 2. (Jerin) > - Redo patch series w/o breaking ABI till the last patch.(David) > - Add deprication notice to announce ABI break in 21.11.(David) > - Add vector limits validation to app/test-eventdev. >=20 > Pavan Nikhilesh (8): > eventdev: introduce event vector capability > eventdev: introduce event vector Rx capability > eventdev: introduce event vector Tx capability > eventdev: add Rx adapter event vector support > eventdev: add Tx adapter event vector support > app/eventdev: add event vector mode in pipeline test > doc: announce event Rx adapter config changes > eventdev: simplify Rx adapter event vector config >=20 > app/test-eventdev/evt_common.h | 4 + > app/test-eventdev/evt_options.c | 52 +++ > app/test-eventdev/evt_options.h | 4 + > app/test-eventdev/test_pipeline_atq.c | 310 +++++++++++++++-- > app/test-eventdev/test_pipeline_common.c | 105 +++++- > app/test-eventdev/test_pipeline_common.h | 18 + > app/test-eventdev/test_pipeline_queue.c | 320 ++++++++++++++++-- > .../prog_guide/event_ethernet_rx_adapter.rst | 38 +++ > .../prog_guide/event_ethernet_tx_adapter.rst | 12 + > doc/guides/prog_guide/eventdev.rst | 36 +- > doc/guides/rel_notes/deprecation.rst | 9 + > doc/guides/rel_notes/release_21_05.rst | 8 + > doc/guides/tools/testeventdev.rst | 45 ++- > lib/librte_eventdev/eventdev_pmd.h | 31 +- > .../rte_event_eth_rx_adapter.c | 305 ++++++++++++++++- > .../rte_event_eth_rx_adapter.h | 78 +++++ > .../rte_event_eth_tx_adapter.c | 66 +++- > lib/librte_eventdev/rte_eventdev.c | 53 ++- > lib/librte_eventdev/rte_eventdev.h | 113 ++++++- > lib/librte_eventdev/version.map | 4 + > 20 files changed, 1524 insertions(+), 87 deletions(-) >=20 > -- > 2.17.1 Just a heads up. v5 patchset doesn't apply cleanly on HEAD (5f0849c1155849d= fdbf950c91c52cdf9cd301f59). Although, it applies cleanly on app/eventdev: f= ix timeout accuracy (c33d48387dc8ccf1b432820f6e0cd4992ab486df).