From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00043.outbound.protection.outlook.com [40.107.0.43]) by dpdk.org (Postfix) with ESMTP id 7CA5254AE for ; Fri, 12 Apr 2019 11:25:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jefYo2GoGk2LcRi6ET+Eufol8GUCyljxgixgO6KwMVI=; b=cS1zY49Na4pBvjiupzd5GiHoWYMwnA5VKa0WMdjMZdaTwdIbzK62BCqiglNb2XiT9CWwI01sOaJ5agwiqzLXVgK89fZA4ZN+043M/2w1m/vFgiN3pYKePcqe8ZoLXIpXWAQ2/Nx28gH62BAUVKXgyryDVwn3aS9VqYKh+hrWe3o= Received: from VI1PR04MB4688.eurprd04.prod.outlook.com (20.177.56.80) by VI1PR04MB4845.eurprd04.prod.outlook.com (20.177.49.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1792.14; Fri, 12 Apr 2019 09:25:19 +0000 Received: from VI1PR04MB4688.eurprd04.prod.outlook.com ([fe80::48ee:dfc2:13c2:2f96]) by VI1PR04MB4688.eurprd04.prod.outlook.com ([fe80::48ee:dfc2:13c2:2f96%5]) with mapi id 15.20.1792.016; Fri, 12 Apr 2019 09:25:19 +0000 From: Shreyansh Jain To: "Ananyev, Konstantin" , "Ruifeng Wang (Arm Technology China)" , "dev@dpdk.org" CC: nd Thread-Topic: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer pool per port Thread-Index: AQHU7dG8s4/pFia32kOC0kLQ4avLvaYx/4KAgAY+UtA= Date: Fri, 12 Apr 2019 09:24:49 +0000 Deferred-Delivery: Fri, 12 Apr 2019 09:23:49 +0000 Message-ID: References: <20190103112932.4415-1-shreyansh.jain@nxp.com> <2601191342CEEE43887BDE71AB9772580148A942C6@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB9772580148A942C6@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shreyansh.jain@nxp.com; x-originating-ip: [92.120.1.69] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0fc68c3a-fc8e-4e55-6035-08d6bf28c7d0 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(4618075)(2017052603328)(7193020); SRVR:VI1PR04MB4845; x-ms-traffictypediagnostic: VI1PR04MB4845: x-microsoft-antispam-prvs: x-forefront-prvs: 0005B05917 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(366004)(376002)(346002)(396003)(13464003)(189003)(199004)(81156014)(53936002)(86362001)(6246003)(476003)(71190400001)(99286004)(6116002)(3846002)(105586002)(6666004)(52536014)(256004)(81166006)(14444005)(55016002)(5660300002)(11346002)(9686003)(44832011)(446003)(71200400001)(106356001)(6436002)(74316002)(110136005)(68736007)(33656002)(8936002)(486006)(14454004)(186003)(2906002)(26005)(316002)(4326008)(97736004)(66066001)(2501003)(229853002)(6506007)(102836004)(7736002)(76176011)(25786009)(305945005)(478600001)(53546011)(7696005); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR04MB4845; H:VI1PR04MB4688.eurprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: ghfwJTgaQv15rE5oiaJWU1M2cCNiv22OV2/SDSKCAwVNKevOFwQdBvTcSroBUj1TyaFV0Z1QTJ4GMtPzBULgXrp7DeKMN42O5eyKdhT1EnB79iXFsqhAB3c+wHDuFZuX66EsfvhlCr4gR1jcvuwfsqwCpl4GMC5QoZ11hutVjHOEf/yArdqS9j+oXYCYUzeOFTZIfhrPqDxly2FxmeseKweFT0vPD+ADbcwJO092R/nEewXr9IfmsPIC4ZCUwil88jPCGHPb2KPhVYTOxOPsnn/iJdMibt7cTyUa8LlRMv4YIUgsM9bsmToghH8IzTOrb/fzxxOktWR1zkfBRY052mcNZWatgRtehZd2DiqY6FG6iPgG3gG7yooC87Y+ePN+sjyYY8cLWBhBcpcWjurfyhBrazxVCLakX+i8DkLZiB4= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0fc68c3a-fc8e-4e55-6035-08d6bf28c7d0 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2019 09:25:18.8951 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB4845 Subject: Re: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer pool per port X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 09:25:20 -0000 Hi Konstantin, Ruifeng, > -----Original Message----- > From: Ananyev, Konstantin > Sent: Monday, April 8, 2019 3:00 PM > To: Ruifeng Wang (Arm Technology China) ; > Shreyansh Jain ; dev@dpdk.org > Cc: nd > Subject: RE: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer > pool per port >=20 >=20 >=20 > > > > Hi Shreyansh, > > > > I tried this patch on MacchiatoBin + 82599 NIC. > > Compared with global-pool mode, per-port-pool mode showed slightly > lower performance in single core test. >=20 > That was my thought too - for the case when queues from multiple ports > are handled by the same core > it probably would only slowdown things. Thanks for your comments. This is applicable for cases where separate cores can handle separate ports= - each with their pools. (somehow I felt that message in commit was adequa= te - I can rephrase if that is misleading) In case there is enough number of cores available for datapath, such segreg= ation can result in better performance - possibly because of drop in pool a= nd cache conflicts. At least on some of NXP SoC, this resulted in over 15% improvement. And, in other cases it didn't lead to any drop/negative-impact. > Wonder what is the use case for the patch and what is the performance > gain you observed? For hardware backed pools, hardware access and exclusion are expensive. By = segregating pool/port/lcores it is possible to attain a conflict free path.= This is the use-case this patch targets.=20 And anyways, this is an optional feature. > Konstantin >=20 > > In dual core test, both modes had nearly same performance. OK > > > > My setup only has two ports which is limited. > > Just want to know the per-port-pool mode has more performance gain > when many ports are bound to different cores? Yes, though not necessarily *many* - in my case, I had 4 ports and even the= n about ~10% improvement was directly visible. I increased the port count a= nd I was able to touch about ~15%. I did pin each port to a separate core, = though. But again, important point is that without this feature enabled, I didn't s= ee any drop in performance. Did you observe any drop? > > > > Used commands: > > sudo ./examples/l3fwd/build/l3fwd -c 0x4 -w 0000:01:00.0 -w > 0000:01:00.1 -- -P -p 3 --config=3D'(0,0,2),(1,0,2)' --per-port-pool > > sudo ./examples/l3fwd/build/l3fwd -c 0xc -w 0000:01:00.0 -w > 0000:01:00.1 -- -P -p 3 --config=3D'(0,0,2),(1,0,3)' --per-port-pool > > > > Regards, > > /Ruifeng > > [...] From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id F22C9A0096 for ; Fri, 12 Apr 2019 11:25:23 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3B62258C4; Fri, 12 Apr 2019 11:25:22 +0200 (CEST) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00043.outbound.protection.outlook.com [40.107.0.43]) by dpdk.org (Postfix) with ESMTP id 7CA5254AE for ; Fri, 12 Apr 2019 11:25:20 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jefYo2GoGk2LcRi6ET+Eufol8GUCyljxgixgO6KwMVI=; b=cS1zY49Na4pBvjiupzd5GiHoWYMwnA5VKa0WMdjMZdaTwdIbzK62BCqiglNb2XiT9CWwI01sOaJ5agwiqzLXVgK89fZA4ZN+043M/2w1m/vFgiN3pYKePcqe8ZoLXIpXWAQ2/Nx28gH62BAUVKXgyryDVwn3aS9VqYKh+hrWe3o= Received: from VI1PR04MB4688.eurprd04.prod.outlook.com (20.177.56.80) by VI1PR04MB4845.eurprd04.prod.outlook.com (20.177.49.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1792.14; Fri, 12 Apr 2019 09:25:19 +0000 Received: from VI1PR04MB4688.eurprd04.prod.outlook.com ([fe80::48ee:dfc2:13c2:2f96]) by VI1PR04MB4688.eurprd04.prod.outlook.com ([fe80::48ee:dfc2:13c2:2f96%5]) with mapi id 15.20.1792.016; Fri, 12 Apr 2019 09:25:19 +0000 From: Shreyansh Jain To: "Ananyev, Konstantin" , "Ruifeng Wang (Arm Technology China)" , "dev@dpdk.org" CC: nd Thread-Topic: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer pool per port Thread-Index: AQHU7dG8s4/pFia32kOC0kLQ4avLvaYx/4KAgAY+UtA= Date: Fri, 12 Apr 2019 09:24:49 +0000 Deferred-Delivery: Fri, 12 Apr 2019 09:23:49 +0000 Message-ID: References: <20190103112932.4415-1-shreyansh.jain@nxp.com> <2601191342CEEE43887BDE71AB9772580148A942C6@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB9772580148A942C6@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shreyansh.jain@nxp.com; x-originating-ip: [92.120.1.69] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0fc68c3a-fc8e-4e55-6035-08d6bf28c7d0 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(4618075)(2017052603328)(7193020); SRVR:VI1PR04MB4845; x-ms-traffictypediagnostic: VI1PR04MB4845: x-microsoft-antispam-prvs: x-forefront-prvs: 0005B05917 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(366004)(376002)(346002)(396003)(13464003)(189003)(199004)(81156014)(53936002)(86362001)(6246003)(476003)(71190400001)(99286004)(6116002)(3846002)(105586002)(6666004)(52536014)(256004)(81166006)(14444005)(55016002)(5660300002)(11346002)(9686003)(44832011)(446003)(71200400001)(106356001)(6436002)(74316002)(110136005)(68736007)(33656002)(8936002)(486006)(14454004)(186003)(2906002)(26005)(316002)(4326008)(97736004)(66066001)(2501003)(229853002)(6506007)(102836004)(7736002)(76176011)(25786009)(305945005)(478600001)(53546011)(7696005); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR04MB4845; H:VI1PR04MB4688.eurprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: ghfwJTgaQv15rE5oiaJWU1M2cCNiv22OV2/SDSKCAwVNKevOFwQdBvTcSroBUj1TyaFV0Z1QTJ4GMtPzBULgXrp7DeKMN42O5eyKdhT1EnB79iXFsqhAB3c+wHDuFZuX66EsfvhlCr4gR1jcvuwfsqwCpl4GMC5QoZ11hutVjHOEf/yArdqS9j+oXYCYUzeOFTZIfhrPqDxly2FxmeseKweFT0vPD+ADbcwJO092R/nEewXr9IfmsPIC4ZCUwil88jPCGHPb2KPhVYTOxOPsnn/iJdMibt7cTyUa8LlRMv4YIUgsM9bsmToghH8IzTOrb/fzxxOktWR1zkfBRY052mcNZWatgRtehZd2DiqY6FG6iPgG3gG7yooC87Y+ePN+sjyYY8cLWBhBcpcWjurfyhBrazxVCLakX+i8DkLZiB4= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0fc68c3a-fc8e-4e55-6035-08d6bf28c7d0 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2019 09:25:18.8951 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB4845 Subject: Re: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer pool per port X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190412092449.68xSgLJTLXSNJPVQjYVK_HAgzXPBq8swah0qKBDCcok@z> Hi Konstantin, Ruifeng, > -----Original Message----- > From: Ananyev, Konstantin > Sent: Monday, April 8, 2019 3:00 PM > To: Ruifeng Wang (Arm Technology China) ; > Shreyansh Jain ; dev@dpdk.org > Cc: nd > Subject: RE: [dpdk-dev] [PATCH] examples/l3fwd: support separate buffer > pool per port >=20 >=20 >=20 > > > > Hi Shreyansh, > > > > I tried this patch on MacchiatoBin + 82599 NIC. > > Compared with global-pool mode, per-port-pool mode showed slightly > lower performance in single core test. >=20 > That was my thought too - for the case when queues from multiple ports > are handled by the same core > it probably would only slowdown things. Thanks for your comments. This is applicable for cases where separate cores can handle separate ports= - each with their pools. (somehow I felt that message in commit was adequa= te - I can rephrase if that is misleading) In case there is enough number of cores available for datapath, such segreg= ation can result in better performance - possibly because of drop in pool a= nd cache conflicts. At least on some of NXP SoC, this resulted in over 15% improvement. And, in other cases it didn't lead to any drop/negative-impact. > Wonder what is the use case for the patch and what is the performance > gain you observed? For hardware backed pools, hardware access and exclusion are expensive. By = segregating pool/port/lcores it is possible to attain a conflict free path.= This is the use-case this patch targets.=20 And anyways, this is an optional feature. > Konstantin >=20 > > In dual core test, both modes had nearly same performance. OK > > > > My setup only has two ports which is limited. > > Just want to know the per-port-pool mode has more performance gain > when many ports are bound to different cores? Yes, though not necessarily *many* - in my case, I had 4 ports and even the= n about ~10% improvement was directly visible. I increased the port count a= nd I was able to touch about ~15%. I did pin each port to a separate core, = though. But again, important point is that without this feature enabled, I didn't s= ee any drop in performance. Did you observe any drop? > > > > Used commands: > > sudo ./examples/l3fwd/build/l3fwd -c 0x4 -w 0000:01:00.0 -w > 0000:01:00.1 -- -P -p 3 --config=3D'(0,0,2),(1,0,2)' --per-port-pool > > sudo ./examples/l3fwd/build/l3fwd -c 0xc -w 0000:01:00.0 -w > 0000:01:00.1 -- -P -p 3 --config=3D'(0,0,2),(1,0,3)' --per-port-pool > > > > Regards, > > /Ruifeng > > [...]