From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 4675EA04DB; Mon, 28 Sep 2020 18:03:03 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1B9B31D612; Mon, 28 Sep 2020 18:03:02 +0200 (CEST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2088.outbound.protection.outlook.com [40.107.22.88]) by dpdk.org (Postfix) with ESMTP id 28F841C190 for ; Mon, 28 Sep 2020 18:02:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SzMPqxwB6LJgbORSfB9tZIZg301mgZink+jTtK5HCZk=; b=H3UtsZKgNaVQMeYRDw60SeTVhLk/kM40OTTUk+RaOiIffRMnj3oNLt1UXAgYfKiA3JbNdl4b9Q6NJ6bB9/rYKiyJVtZRdL4/9L7KDeRFs1GHHMrV7V70mC9b13RdcYGtOCS2XEbtMWiSTgDWmRHoR6e7mubMl8udLtqsSYYBWxs= Received: from AM6P191CA0074.EURP191.PROD.OUTLOOK.COM (2603:10a6:209:8a::15) by AM6PR08MB4984.eurprd08.prod.outlook.com (2603:10a6:20b:ee::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3412.22; Mon, 28 Sep 2020 16:02:55 +0000 Received: from VE1EUR03FT005.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:8a:cafe::1e) by AM6P191CA0074.outlook.office365.com (2603:10a6:209:8a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3412.20 via Frontend Transport; Mon, 28 Sep 2020 16:02:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT005.mail.protection.outlook.com (10.152.18.172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3412.21 via Frontend Transport; Mon, 28 Sep 2020 16:02:54 +0000 Received: ("Tessian outbound 7161e0c2a082:v64"); Mon, 28 Sep 2020 16:02:54 +0000 X-CR-MTA-TID: 64aa7808 Received: from da60ac396e9c.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id C924C1D6-CFC2-4F99-9775-0D58A8993D6E.1; Mon, 28 Sep 2020 16:02:49 +0000 Received: from EUR03-AM5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id da60ac396e9c.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 28 Sep 2020 16:02:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fea0IH9wcjA4TXCkK0+Nu/5WhpL6DYDIO9Rr9vIzD8eXtn8FVM6SM8QqZszIxfz6EeDhXgfA5iY0xEznkL6QLMDH2+/8SbDWUzM2BtY9TKL/e/MVdGAK9vllfcdD13rT2Onc71QuUKufoD14a+cDMIs/bXfLtkVb7C1wHlTms5Lyy5LKPGJwavm5tDWr5839errM8kBMt9Acn6qUxqARkXtuiv9NqrTNyXpHtPddkRyQZQ7NtzihwAo0oOBh24Gwr3PLD7ngqBV0Svm4IGwVJi0uLPaRV5QpQblYjmLilwRo5vRCFmZ0xAyPavFDYxh6yUVbRv+vG48ib5Vd1A8S1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SzMPqxwB6LJgbORSfB9tZIZg301mgZink+jTtK5HCZk=; b=B2KCR+1bNnVlB2HNN+NQsCP+FrR0DND8NpyG+KM/rbNLlFCD5duwn05fHhYJenqhj65Qpi3SHcoRacZWPhRhRHFZUjNp+X8YCRDmVwv6+wt1hpskgVJfUhStIGkGHzPgNiWg9yDFgOnZ3lDPfBpDQdUbuWbT0d9Ngg9xvKK3ByDXp791AjJQcfRlDVUezx7sXmhyMgAG4VZbNsal4XVuTQ8ISHgnnDxeLDq3l50tjGIqJGW4cx5bO0uqj3UgkjwwnG31kHMOfg9nd1/qUESjeWBFECvrQF3oYMjk3oTEXZseG810OoeapppI1W/ciRHIUtJpsjGsp1GdI0aIjyNB2g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SzMPqxwB6LJgbORSfB9tZIZg301mgZink+jTtK5HCZk=; b=H3UtsZKgNaVQMeYRDw60SeTVhLk/kM40OTTUk+RaOiIffRMnj3oNLt1UXAgYfKiA3JbNdl4b9Q6NJ6bB9/rYKiyJVtZRdL4/9L7KDeRFs1GHHMrV7V70mC9b13RdcYGtOCS2XEbtMWiSTgDWmRHoR6e7mubMl8udLtqsSYYBWxs= Received: from AM8PR08MB5810.eurprd08.prod.outlook.com (2603:10a6:20b:1d2::20) by AM0PR08MB3028.eurprd08.prod.outlook.com (2603:10a6:208:5f::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3412.28; Mon, 28 Sep 2020 16:02:48 +0000 Received: from AM8PR08MB5810.eurprd08.prod.outlook.com ([fe80::14fa:b919:69b1:d65b]) by AM8PR08MB5810.eurprd08.prod.outlook.com ([fe80::14fa:b919:69b1:d65b%7]) with mapi id 15.20.3412.029; Mon, 28 Sep 2020 16:02:48 +0000 From: Honnappa Nagarahalli To: Bruce Richardson , "Ananyev, Konstantin" CC: "Nicolau, Radu" , "Van Haaren, Harry" , "dev@dpdk.org" , "jerinj@marvell.com" , nd , Honnappa Nagarahalli , nd Thread-Topic: [dpdk-dev] [PATCH v1] event/sw: performance improvements Thread-Index: AQHWhc4yw0MoIn/BrE2rarxyqK1mbql2JUUwgADKunCAARLdgIAAAmvQgAE8ezKABQG/gA== Date: Mon, 28 Sep 2020 16:02:48 +0000 Message-ID: References: <20200908105211.10066-1-radu.nicolau@intel.com> <46118f3466274596a663d7d44abb680a@intel.com> <20200925102805.GD923@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: <20200925102805.GD923@bricha3-MOBL.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: E9B1114D0781774481FFB83C3CF98303.0 x-checkrecipientchecked: true Authentication-Results-Original: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.112.90.121] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 38dd1207-0025-4f0d-89b1-08d863c7f5f8 x-ms-traffictypediagnostic: AM0PR08MB3028:|AM6PR08MB4984: x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: JvHNQT9v7flagoc/i0KaZ99gXjNOukFRafLiawm64tMYRdhcGVweVpS3TpXd3J+lD1k7ELyqd4i1Kohchb/ds4Tab4Cx77nGGuVYRGSSgugn1hTPMy3voMK4x0lK+jjhxYBxVWh+F7hw3dHajc4kmBrELtrOKLdti72HyKAuB4IOYxb/BQ/c90QbYmilCWrUA2QsdSrP9PQh/0YxKMvEKZJmnehb08uGMPI5FnoFJ18Kfgn2WS6Ht91b06rszOCvfrDx69Yv+Un71uhlB3zksIfnLwpAUCKAbuUQxY6UzUuKMdpTkClL8eQexeoKwBaS0E2jUwLaC0+yb7j9Yedv9ZfxWWioV3uhG0M3nIRuGp/W2blYhuhJk3Zs3/oT0FHv X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM8PR08MB5810.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39850400004)(136003)(396003)(376002)(366004)(346002)(5660300002)(2906002)(76116006)(316002)(110136005)(54906003)(26005)(52536014)(71200400001)(186003)(478600001)(9686003)(8936002)(55016002)(83380400001)(66446008)(64756008)(8676002)(66556008)(66946007)(66476007)(86362001)(6506007)(7696005)(33656002)(4326008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: ++fQIqsJEW3tqTgdg5XvJxL+qjJfLWANq6pZ29m+2RNjHMzrGimelTfH/ckpk+vQgDc6YifwBTi8hZydsaELRsmZZdxSjQZG115Lqb9wG2BaK6In+0j4QPm+i5s5SG5DB0zOzKvW1MzXmu5G8W00R3LHMQuaz5zFjECO4sakhV3Xvo24Qax0ZpF5EruiffygXqM6msAkBn3auD+i86Pu82kK11Do0UzgdbVKXyt7WC/9ytIjQldLYihPF/F3yZzAowbL5lzszeV+OIjjeOKVaSZhAKIf+VEUF81lVBCwO3EPMmmkKSHDDMf6bxB6SqAgyMJ4JJtJ6We4M5zPTWQQKex9jR0cKJjUwxfPVQq2Q+rkO98kMe8igJa8G27xN7sn+p7oQwKE37R2mtd0YIzW/B25VCrjVofb8jaqGQJ7345+aMXYEfQR5gv6ej2s6DLLiFzeU5nlpMca3cJZPjcrXkvFpC1L3qDK+UIMbIodRGR9l32PrNybx5M/Fali8VX5P4Qk6ld86qN00011EZAGS5MSD33mG1WE34KzQlsbIdt69p5lgXPLod4s3AwwI63uHEZQumckFEhy/av3LSAgDgYkjkOcefhQOWGY5P8swd8myZf/CmNlwPXFAaNwCEmL7wGXKUqyRLTQbtd4uQs0Og== Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3028 Original-Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT005.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: ed38cbb5-11ba-40da-c88c-08d863c7f1e9 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ydKpUblc9lGPEW57ihBIsTzDDk62WVVHRquo/WeCmvPSBVeuaHb42awEtwRFMbRD+OxZScDFTx2NN6PjXQXTpgz0e3UpV0pNNds5YiaKDvhy1dlmK6eGgR4jJULeLUgDO4i+yUYaNQZKrXhdWje3DsjkuPlqSpF6L+o3GhuiabtBcXdXFAGyi7w1zl7aQN50brrWg34jQD+CcRAp4InuYFLz6qC4eBWDwZqHHS+PrrFs/1zUuD+oxFSmRy8ZoqkNOurThiOkC7+AngX5o5inGimrcaVGeL5nIYjT7H32ZAUngQaJZd2naMtbOh2byegrxCGy1Y0FQpgEM1tNcnSWWb5bCeMZPqIS5j6aZ4YsLC8nLmCFpxoVJruxflWXEm9y8oLMYLzThcKE6nNRM1jxJx4D8wESGmbdhftjU58LaWpgChRm74ozct25oT3ZttCM X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(346002)(376002)(396003)(39850400004)(136003)(46966005)(336012)(47076004)(81166007)(70586007)(5660300002)(70206006)(36906005)(186003)(110136005)(4326008)(316002)(54906003)(9686003)(7696005)(83380400001)(6506007)(55016002)(2906002)(8676002)(8936002)(82310400003)(356005)(82740400003)(33656002)(478600001)(26005)(86362001)(52536014); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2020 16:02:54.8153 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 38dd1207-0025-4f0d-89b1-08d863c7f5f8 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT005.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4984 Subject: Re: [dpdk-dev] [PATCH v1] event/sw: performance improvements X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > Add minimum burst throughout the scheduler pipeline and a flush counter= . > > Replace ring API calls with local single threaded implementation where > > possible. > > > > Signed-off-by: Radu Nicolau mailto:radu.nicolau@intel.com > > > > Thanks for the patch, a few comments inline. > > > > --- > > drivers/event/sw/sw_evdev.h=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD | 11 +++- > > drivers/event/sw/sw_evdev_scheduler.c | 83 > > +++++++++++++++++++++++---- > > 2 files changed, 81 insertions(+), 13 deletions(-) > > > > diff --git a/drivers/event/sw/sw_evdev.h b/drivers/event/sw/sw_evdev.h > > index 7c77b2495..95e51065f 100644 > > --- a/drivers/event/sw/sw_evdev.h > > +++ b/drivers/event/sw/sw_evdev.h > > @@ -29,7 +29,13 @@ > > /* report dequeue burst sizes in buckets */=EF=BF=BD #define > > SW_DEQ_STAT_BUCKET_SHIFT 2 > > /* how many packets pulled from port by sched */ -#define > > SCHED_DEQUEUE_BURST_SIZE 32 > > +#define SCHED_DEQUEUE_BURST_SIZE 64 > > + > > +#define SCHED_MIN_BURST_SIZE 8 > > +#define SCHED_NO_ENQ_CYCLE_FLUSH 256 > > +/* set SCHED_DEQUEUE_BURST_SIZE to 64 or 128 when setting this to 1*/ > > +#define SCHED_REFILL_ONCE_PER_CALL 1 > > > > Is it possible to make the above #define a runtime option? > > Eg, --vdev event_sw,refill_iter=3D1 > > > > That would allow packaged versions of DPDK to be usable in both modes. > > > > + > > > > #define SW_PORT_HIST_LIST (MAX_SW_PROD_Q_DEPTH) /* size of our > > history list */=EF=BF=BD #define NUM_SAMPLES 64 /* how many data points= use > > for average stats */ @@ -214,6 +220,9 @@ struct sw_evdev { =EF=BF=BD= =EF=BF=BD=EF=BF=BD > > uint32_t xstats_count_mode_port; =EF=BF=BD=EF=BF=BD=EF=BF=BD uint32_t > > xstats_count_mode_queue; > > > > +=EF=BF=BD=EF=BF=BD=EF=BF=BD uint16_t sched_flush_count; =EF=BF=BD=EF= =BF=BD=EF=BF=BD uint16_t > > +sched_min_burst; > > + > > =EF=BF=BD=EF=BF=BD=EF=BF=BD /* Contains all ports - load balanced and = directed */ > > =EF=BF=BD=EF=BF=BD=EF=BF=BD struct sw_port ports[SW_PORTS_MAX] __rte_ca= che_aligned; > > > > diff --git a/drivers/event/sw/sw_evdev_scheduler.c > > b/drivers/event/sw/sw_evdev_scheduler.c > > index cff747da8..ca6d1caff 100644 > > --- a/drivers/event/sw/sw_evdev_scheduler.c > > +++ b/drivers/event/sw/sw_evdev_scheduler.c > > @@ -26,6 +26,29 @@ > > /* use cheap bit mixing, we only need to lose a few bits */=EF=BF=BD > > #define > > SW_HASH_FLOWID(f) (((f) ^ (f >> 10)) & FLOWID_MASK) > > > > + > > +/* single object enq and deq for non MT ring */ static > > +__rte_always_inline void sw_nonmt_ring_dequeue(struct rte_ring *r, > > +void **obj) { =EF=BF=BD=EF=BF=BD=EF=BF=BD if ((r->prod.tail - r->cons.= tail) < 1) > > +=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD return; =EF=BF=BD=EF=BF=BD=EF=BF=BD void **r= ing =3D > > +(void **)&r[1]; =EF=BF=BD=EF=BF=BD=EF=BF=BD *obj =3D ring[r->cons.tail= & r->mask]; > > +=EF=BF=BD=EF=BF=BD=EF=BF=BD r->cons.tail++; } static __rte_always_inli= ne int > > +sw_nonmt_ring_enqueue(struct rte_ring *r, void *obj) { =EF=BF=BD=EF=BF= =BD=EF=BF=BD if > > +((r->capacity + r->cons.tail - r->prod.tail) < 1) > > +=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD return 0; =EF=BF=BD=EF=BF=BD=EF=BF=BD void *= *ring =3D > > +(void **)&r[1]; =EF=BF=BD=EF=BF=BD=EF=BF=BD ring[r->prod.tail & r->mas= k] =3D obj; > > +=EF=BF=BD=EF=BF=BD=EF=BF=BD r->prod.tail++; =EF=BF=BD=EF=BF=BD=EF=BF= =BD return 1; > > + > > Why not make these APIs part of the rte_ring library? You could further > optimize them by keeping the indices on the same cacheline. > > I'm not sure there is any need for non thread-safe rings outside this > particular case. > > [Honnappa] I think if we add the APIs, we will find the use cases. > > But, more than that, I understand that rte_ring structure is exposed to= the > application. The reason for doing that is the inline functions that rte_r= ing > provides. IMO, we should still maintain modularity and should not use the > internals of the rte_ring structure outside of the library. > > > > +1 to that. > > > > BTW, is there any real perf benefit from such micor-optimisation? >=20 > I'd tend to view these as use-case specific, and I'm not sure we should c= lutter > up the ring library with yet more functions, especially since they can't = be > mixed with the existing enqueue/dequeue functions, since they don't use > the head pointers. IMO, the ring library is pretty organized with the recent addition of HTS/R= TS modes. This can be one of the modes and should allow us to use the exist= ing functions (though additional functions are required as well). The other concern I have is, this implementation can be further optimized b= y using a single cache line for the pointers. It uses 2 cache lines just be= cause of the layout of the rte_ring structure. There was a question earlier about the performance improvements of this pat= ch? Are there any % performance improvements that can be shared? It is also possible to change the above functions to use the head/tail poin= ters from producer or the consumer cache line alone to check for perf diffe= rences.