From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10058.outbound.protection.outlook.com [40.107.1.58]) by dpdk.org (Postfix) with ESMTP id 48D7A293B; Thu, 23 Aug 2018 09:28:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pAkmGPaG/BWNI+vfxOZfMDMfFShm/KZjV3fL8lJ/HqY=; b=DiSKuef+gGl7G+ksvD5/UPnOxHRmQRzMkENc7bgTnIQGI2tulQ14U0zhHKBTmkrBwOoU8R/snFr843RIlaJOBPSnALd14skoq/jkzM39AA7KHcHn8+RkNUsH1aWgFSKCgo6UDa5lG8OqovxnfPpBxjybiL5MvBOOh3xl/5M8eHQ= Received: from AM0PR0502MB4019.eurprd05.prod.outlook.com (52.133.41.11) by AM0PR0502MB3953.eurprd05.prod.outlook.com (52.133.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1059.23; Thu, 23 Aug 2018 07:28:04 +0000 Received: from AM0PR0502MB4019.eurprd05.prod.outlook.com ([fe80::88fa:4498:85a5:9e5a]) by AM0PR0502MB4019.eurprd05.prod.outlook.com ([fe80::88fa:4498:85a5:9e5a%3]) with mapi id 15.20.1080.015; Thu, 23 Aug 2018 07:28:04 +0000 From: Matan Azrad To: Eric Kinzie CC: Luca Boccassi , Chas Williams <3chas3@gmail.com>, "dev@dpdk.org" , Declan Doherty , Chas Williams , "stable@dpdk.org" Thread-Topic: [dpdk-dev] [dpdk-stable] [PATCH v4] net/bonding: per-slave intermediate rx ring Thread-Index: AQHUOj+hWaQ4YaZC0UCW7VMjN5ENnaTM70pg Date: Thu, 23 Aug 2018 07:28:03 +0000 Message-ID: References: <20180816125202.15980-1-bluca@debian.org> <20180816133208.26566-1-bluca@debian.org> <1534933159.5764.107.camel@debian.org> <20180822174316.GA29821@roosta> In-Reply-To: <20180822174316.GA29821@roosta> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM0PR0502MB3953; 6:CbZdlnkF+dYG8cpIhL1nm3/d2/djyMmsJTJyjkVARoGDxcNjuaEd2AmLevwsMRU1pkOmf+78zoDVnlTp1pUNn2rTFhyRfeQLKEBHStpWFM6TCQ3RNE6V/FK7a3mnkebpVNRjLWTzSi7qnW8LLJfjaNi3FokeqWl6tF7a47xrhTnNn5HU4zru/I7bGk6NWM4uIxG2/ONblo0IWGXsLO8nLY57LDUs+dsNT0SBII2Fw0bVwhr4aVzT3nrQvDLVc02pmsAXbMHSnLCk9R4oUeCf7AtBeu/Puw9ItZtA/rb7c+iz9ZHbC0OlyArc98TMCsJsaeMdNtWecHjixJngVxZqM3MQj846FenaqKqdNgyz88MRgg6IKgcJHRhtKRzPMcMjuxUnUTeXa4FK21CUHyYlFdeHoyiXgUcrh3aYHbEIXOGhsM2CRoKsOlEXGrV0DqmzevAIAD8Ise4rMXYmt4AcDQ==; 5:1WsV/VXstL7EWplbL2sqx5pMa0y+iM4qG5Ygb2P7L+58NsYv8dW+0gq1vfLaq6XUeB1xW7Pem5XIFPF7RtCx4CwkG5jvSlLidV6hCi9Qw1FGWGDHaJwJkBuk6eaWtU8EEXb6f/oU2p3846JnjlquI8w2CeAu/pummhtVo+DcLpg=; 7:0g/pBCF9Orbzw+9ZWx+NRsqLlf9FcJSLwuyr8sz8DtjYjnnxL15rKh/wBryM6YoSzDG4Z39C6tUPlmurkZwfVwARDSU2W9BgT1Nybnx0/x16VkYX0p7xIhuvgaa5TMTQu00BzwSSF93ufiy/TmHx6Qb9IaKFyLViFqkreJCVjJYWF2xqkP/RGvFmbyE0G/U1z+ZwZaLcxl8ESf1t+G38TzItN3lT9I29dF5qOxr5qolazgLgqUwQgxEnbpQXDw+5 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 1a96a1cf-8feb-4d61-0500-08d608c9f6b7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989137)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3953; x-ms-traffictypediagnostic: AM0PR0502MB3953: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(20558992708506)(278428928389397); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(10201501046)(3231311)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699016); SRVR:AM0PR0502MB3953; BCL:0; PCL:0; RULEID:; SRVR:AM0PR0502MB3953; x-forefront-prvs: 0773BB46AC x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(346002)(136003)(39860400002)(366004)(396003)(199004)(189003)(3846002)(105586002)(2900100001)(106356001)(229853002)(33656002)(316002)(97736004)(5250100002)(1411001)(55016002)(6436002)(54906003)(99286004)(7696005)(76176011)(7736002)(5660300001)(6116002)(9686003)(6916009)(53936002)(66066001)(446003)(74316002)(6246003)(486006)(2906002)(476003)(39060400002)(11346002)(478600001)(186003)(4326008)(53546011)(8936002)(68736007)(6506007)(26005)(14454004)(102836004)(86362001)(93886005)(25786009)(305945005)(81166006)(8676002)(81156014)(256004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3953; H:AM0PR0502MB4019.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: hRS5TvMsNK52mqib0r9CRDnhuFIyi5UqLP4nvy493ODOEtcgfnYZi5pHljDh17BT4IkmSUj1RVSqqNFeHUJXbq5upt53KvldQU+I6TuecIV0roLiqNHWi2qCfxiXtW+3WokBUdmhHF5dfmRN9FinBZOB2DjWwBvRIYTHVKPPCA77lp92Dj2S6KF0recgZUHu70nehOSzfNAGS7BVkTAMeBzdnVY43KopNq3n5Rd6Ytp7jrwEh/T7Fs9OwlkSK2a11/QV5zirlEfjT1k7dN1g2edSQuFPDEIA+jIxoPG43O9NRbHNl+NdQIYpsSK338drieHZWlR1PwcmF3QutoC/lIVe3H6E8NYQAJUrtQuOEfM= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1a96a1cf-8feb-4d61-0500-08d608c9f6b7 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Aug 2018 07:28:03.8423 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3953 Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v4] net/bonding: per-slave intermediate rx ring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2018 07:28:05 -0000 Hi From: Eric Kinzie > On Wed Aug 22 11:42:37 +0000 2018, Matan Azrad wrote: > > Hi Luca > > > > From: Luca Boccassi > > > On Wed, 2018-08-22 at 07:09 +0000, Matan Azrad wrote: > > > > Hi Chas > > > > > > > > From: Chas Williams > > > > > On Tue, Aug 21, 2018 at 11:43 AM Matan Azrad > > > > > wrote: > > > > > Hi Chas > > > > > > > > > > From: Chas Williams > > > > > > On Tue, Aug 21, 2018 at 6:56 AM Matan Azrad > > > > > > > > > > > > wrote: > > > > > > Hi > > > > > > > > > > > > From: Chas Williams > > > > > > > This will need to be implemented for some of the other RX > > > > > > > burst methods at some point for other modes to see this > > > > > > > performance improvement (with the exception of active-backup)= . > > > > > > > > > > > > Yes, I think it should be done at least to > > > > > > bond_ethdev_rx_burst_8023ad_fast_queue (should be easy) for > now. > > > > > > > > > > > > There is some duplicated code between the various RX paths. > > > > > > I would like to eliminate that as much as possible, so I was > > > > > > going to give that some thought first. > > > > > > > > > > There is no reason to stay this function as is while its twin is > > > > > changed. > > > > > > > > > > Unfortunately, this is all the patch I have at this time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 16, 2018 at 9:32 AM Luca Boccassi > > > > > > > wrote: > > > > > > > > > > > > > > > During bond 802.3ad receive, a burst of packets is fetched > > > > > > > > from each slave into a local array and appended to > > > > > > > > per-slave ring buffer. > > > > > > > > Packets are taken from the head of the ring buffer and > > > > > > > > returned to the caller.=A0 The number of mbufs provided to > > > > > > > > each slave is sufficient to meet the requirements of the > > > > > > > > ixgbe vector receive. > > > > > > > > > > > > Luca, > > > > > > > > > > > > Can you explain these requirements of ixgbe? > > > > > > > > > > > > The ixgbe (and some other Intel PMDs) have vectorized RX > > > > > > routines that are more efficient (if not faster) taking > > > > > > advantage of some advanced CPU instructions.=A0 I think you nee= d > > > > > > to be receiving at least 32 packets or more. > > > > > > > > > > So, why to do it in bond which is a generic driver for all the > > > > > vendors PMDs, If for ixgbe and other Intel nics it is better you > > > > > can force those PMDs to receive always 32 packets and to manage > > > > > a ring by themselves. > > > > > > > > > > The drawback of the ring is some additional latency on the > > > > > receive path. > > > > > In testing, the additional latency hasn't been an issue for bondi= ng. > > > > > > > > When bonding does processing slower it may be a bottleneck for the > > > > packet processing for some application. > > > > > > > > > The bonding PMD has a fair bit of overhead associated with the > > > > > RX and TX path calculations.=A0 Most applications can just arrang= e > > > > > to call the RX path with a sufficiently large receive.=A0 Bonding > > > > > can't do this. > > > > > > > > I didn't talk on application I talked on the slave PMDs, The slave > > > > PMD can manage a ring by itself if it helps for its own performance= . > > > > The bonding should not be oriented to specific PMDs. > > > > > > The issue though is that the performance problem is not with the > > > individual PMDs - it's with bonding. There were no reports regarding > > > the individual PMDs. > > > This comes from reports from customers from real world production > > > deployments - the issue of bonding being too slow was raised multiple > times. > > > This patch addresses those issues, again in production deployments, > > > where it's been used for years, to users and customers satisfaction. > > > > From Chas I understood that using burst of 32 helps for some slave PMDs > performance which makes sense. > > I can't understand how the extra copy phases improves the bonding itsel= f > performance: > > > > You added 2 copy phases in the bonding RX function: > > 1. Get packets from the slave to a local array. > > 2. Copy packet pointers from a local array to the ring array. > > 3. Copy packet pointers from the ring array to the application array. > > > > Each packet arriving to the application must pass the above 3 phases(in= a > specific call or in previous calls). > > > > Without this patch we have only - > > Get packets from the slave to the application array. > > > > Can you explain how the extra copies improves the bonding performance? > > > > Looks like it improves the slaves PMDs and because of that the bonding > PMD performance becomes better. >=20 > I'm not sure that adding more buffer management to the vector PMDs will > improve the drivers' performance; it's just that calling the rx function = in such > a way that it returns no data wastes time. Sorry, I don't fully understand what you said here, please rephrase. > The bonding driver is already an exercise in buffer management so adding= this layer of indirection here makes > sense in my opinion, as does hiding the details of the consituent interfa= ces where possible. Can you explain how this new buffer management with the extra pointer copie= s improves the bonding itself performance? Looks really strange to me. > > > So I'd like to share this improvement rather than keeping it private > > > - because I'm nice that way :-P > > > > > > > > > Did you check for other vendor PMDs? It may hurt performance > > > > > > there.. > > > > > > > > > > > > I don't know, but I suspect probably not.=A0 For the most part > > > > > > you are typically reading almost up to the vector requirement. > > > > > > But if one slave has just a single packet, then you can't > > > > > > vectorize on the next slave. > > > > > > > > > > > > > > > > I don't think that the ring overhead is better for PMDs which > > > > > are not using the vectorized instructions. > > > > > > > > > > The non-vectorized PMDs are usually quite slow.=A0 The additional > > > > > overhead doesn't make a difference in their performance. > > > > > > > > We should not do things worse than they are. > > > > > > There were no reports that this made things worse. The feedback from > > > production was that it made things better. > > > > Yes, It may be good for specific slaves drivers but hurt another > > slaves drivers, So maybe it should stay private to specific costumers u= sing > specific nics. > > > > Again, I can understand how this patch improves performance of some > > PMDs therefore I think the bonding is not the place to add it but maybe > some PMDs.