From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30065.outbound.protection.outlook.com [40.107.3.65]) by dpdk.org (Postfix) with ESMTP id 486771BA72 for ; Thu, 26 Oct 2017 12:31:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Wxcp2DCn4Uqju74ea2aCwO5xGUCcsYgZ99rezB1lkxU=; b=eUZybM//2FMlGeKRzPcLtTVCEQzH9tzWolMMp0LqpFGqOc+u4PIhmk1iVWcl1L7FWuWRmo2QxAT6Qf4IM7eESAvGF4SJ4t3TC/FcfU9sDu10VEE/+ILAfKKMto/IZOXfcHEjqvGIGdDVGcnFjoK6ZK/0l48M0KOdE/4kOsSoVlI= Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) by HE1PR05MB1916.eurprd05.prod.outlook.com (10.166.87.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Thu, 26 Oct 2017 10:31:06 +0000 Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::c524:908c:b99c:3f4b]) by HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::c524:908c:b99c:3f4b%13]) with mapi id 15.20.0156.007; Thu, 26 Oct 2017 10:31:06 +0000 From: Matan Azrad To: =?iso-8859-1?Q?N=E9lio_Laranjeiro?= , "Ophir Munk" CC: Adrien Mazarguil , "dev@dpdk.org" , Thomas Monjalon , Olga Shern , Mordechay Haimovsky Thread-Topic: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions Thread-Index: AQHTTApr4imkSYrit0WXMr28NXPk2aLzBmWAgABxKwCAALwaAIABuoog Date: Thu, 26 Oct 2017 10:31:06 +0000 Message-ID: References: <1508752838-30408-1-git-send-email-ophirmu@mellanox.com> <1508768520-4810-1-git-send-email-ophirmu@mellanox.com> <1508768520-4810-5-git-send-email-ophirmu@mellanox.com> <20171024135149.fyg4nzcbygo2amtz@laranjeiro-vm> <20171025075006.znxl7mezy4pfyzsj@laranjeiro-vm> In-Reply-To: <20171025075006.znxl7mezy4pfyzsj@laranjeiro-vm> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR05MB1916; 6:M5B+FtsdXyDKi1JOnRpWH2UKejwULJF8hZ541wY6LYIm6jR+3Gtf8VE/CbWnWZbKS2Y7hyqlLhby9Blv3/bEdTabkX+8jWeDj3pi/oxXQjiQzQs9yDG9MYICbJ7FwegYp0+y0tJgF2+OUjTaVQZ4x6zZkF3cV8snO6cr0LlyUaCOAR1433AJogC4/rmJS6qp3+3G4ArxQhZMlE1RMuGgpmw69A9fuDWYlewgba7De5S3rbSX//hGnCFBJa/+5uYi4Y6t4JzMvlTIqFyxfhrOLoQFLPOAhzO+bTrWCh5DxnpLbxwL0rY7KNYYEpzDr9VJvfWI6B0jJvMRGjcV9mFZVUQigjjNqI7PDj+fm8BC8TY=; 5:aD5Cgwg7U6xCokj0qTjwmxhSoh7mTOTacWbqXVLwFomLmC1YX8xB4pBMY/bNb3CqHW3qwVmDb19uB+rYNYjVFdY0Sh+hlv4jMwoln3qmF9aOsP4WqNWG2r5Wpah6Gmcw23ZB9S/tGm6jwW8wY7jJfA6qipjYbf2dgY1fwPsqzts=; 24:JFSeBJG0Wl4+w28MpHJolIdPjRGb7R2Ef6FaCGmYJc6g5hb6WeCbntfcEdeO22vDa/b9USfvLrDIh7jiKdkA5Umq0h97nqnvjvggRQ10Tkc=; 7:wGBL7TxIsG+GyeoGWC0B8FMtpiCfMuZm+hIT4BkCiObTrbxvo0jP7H9ShyOehIq44+rH3Wk7R60plM9NPPwp0i//FZW5ST4lZsaTaXV4yzILBZb8RZj1zGKOA0LwbD1exPnhV/P0tuXCqclKM43ctFWPX6XADiTkUKWEcEeTHSNyr6bF+wcGZSur/v8df/UUPNCFHIHh5aMrW1B1xG977ov/Z9LnABEhA+OSbpntuq9Q1eevtroDDzqW0x6o3OjN x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 93875c2c-f78c-4ec9-3fab-08d51c5caa9f x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4627075)(201703031133081)(201702281549075)(48565401081)(2017052603199); SRVR:HE1PR05MB1916; x-ms-traffictypediagnostic: HE1PR05MB1916: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(100000703101)(100105400095)(3002001)(10201501046)(3231020)(6055026)(6041248)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123560025)(20161123562025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR05MB1916; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR05MB1916; x-forefront-prvs: 04724A515E x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(39860400002)(43544003)(24454002)(189002)(199003)(13464003)(93886005)(110136005)(54906003)(5660300001)(99286003)(55016002)(9686003)(2900100001)(6436002)(316002)(6506006)(53936002)(107886003)(229853002)(7696004)(3280700002)(2906002)(6246003)(4326008)(6636002)(2950100002)(3660700001)(66066001)(53546010)(305945005)(14454004)(7736002)(97736004)(74316002)(8936002)(3846002)(6116002)(102836003)(33656002)(81156014)(81166006)(8676002)(478600001)(5250100002)(25786009)(68736007)(54356999)(50986999)(76176999)(101416001)(189998001)(106356001)(105586002)(86362001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR05MB1916; H:HE1PR0502MB3659.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 93875c2c-f78c-4ec9-3fab-08d51c5caa9f X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Oct 2017 10:31:06.7456 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR05MB1916 Subject: Re: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 10:31:08 -0000 Hi Nelio I think the memory barrier discussion is not relevant for this patch (if it= will be relevant I will create new one). Please see my comments inline. Regarding this specific patch, I didn't see any comment from you, Are you a= gree with it?=20 =20 > -----Original Message----- > From: N=E9lio Laranjeiro [mailto:nelio.laranjeiro@6wind.com] > Sent: Wednesday, October 25, 2017 10:50 AM > To: Ophir Munk > Cc: Adrien Mazarguil ; dev@dpdk.org; > Thomas Monjalon ; Olga Shern > ; Matan Azrad > Subject: Re: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions >=20 > On Tue, Oct 24, 2017 at 08:36:52PM +0000, Ophir Munk wrote: > > Hi, > > > > On Tuesday, October 24, 2017 4:52 PM, N=E9lio Laranjeiro wrote: > > > > > > On Mon, Oct 23, 2017 at 02:21:57PM +0000, Ophir Munk wrote: > > > > From: Matan Azrad > > > > > > > > Merge tx_burst and mlx4_post_send functions to prevent double > > > > asking about WQ remain space. > > > > > > > > This should improve performance. > > > > > > > > Signed-off-by: Matan Azrad > > > > --- > > > > drivers/net/mlx4/mlx4_rxtx.c | 353 > > > > +++++++++++++++++++++---------------------- > > > > 1 file changed, 170 insertions(+), 183 deletions(-) > > > > > > What are the real expectation you have on the remaining patches of > > > the series? > > > > > > According to the comment of this commit log "This should improve > > > performance" there are too many barriers at each packet/segment > > > level to improve something. > > > > > > The point is, mlx4_burst_tx() should write all the WQE without any > > > barrier as it is processing a burst of packets (whereas Verbs > > > functions which may only process a single packet). > > > > > The lonely barrier which should be present is the one to ensure that > > > all the host memory is flushed before triggering the Tx doorbell. > > > > > > > There is a known ConnectX-3 HW limitation: the first 4 bytes of every > > TXWBB (64 bytes chunks) should be > > written in a reversed order (from last TXWBB to first TXWBB). >=20 > This means the first WQE filled by the burst function is the doorbell. > In such situation, the first four bytes of it can be written before > leaving the burst function and after a write memory barrier. >=20 > Until this first WQE is not complete, the NIC won't start processing the > packets. Memory barriers per packets becomes useless. I think this is not true, Since mlx4 HW can prefetch advanced TXbbs if thei= r first 4 bytes are valid in spite of the first WQE is still not valid (please read t= he spec). >=20 > It gives something like: >=20 > uint32_t tx_bb_db =3D 0; > void *first_wqe =3D NULL; >=20 > /* > * Prepare all Packets by writing the WQEs without the 4 first bytes of > * the first WQE. > */ > for () { > if (!wqe) { > first_wqe =3D wqe; > tx_bb_db =3D foo; > } > } > /* Leaving. */ > rte_wmb(); > *(uin32_t*)wqe =3D tx_bb_db; > return n; > I will take care to check if we can do 2 loops: Write all last 60B per TXbb. Memory barrier. Write all first 4B per TXbbs. > > The last 60 bytes of any TXWBB can be written in any order (before > > writing the first 4 bytes). > > Is your last statement (using lonely barrier) is in accordance with > > this limitation? Please explain. > > > > > There is also too many cases handled which are useless in bursts > situation, > > > this function needs to be re-written to its minimal use case i.e. > processing a > > > valid burst of packets/segments and triggering at the end of the burs= t the > Tx > > > doorbell. > > > >=20 > Regards, >=20 > -- > N=E9lio Laranjeiro > 6WIND