From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10074.outbound.protection.outlook.com [40.107.1.74]) by dpdk.org (Postfix) with ESMTP id F2A021BAE2 for ; Thu, 26 Oct 2017 14:31:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=VHkjrb/n5gX9gY9OaTiF1VDmIcfsAGiZE3oxU71Twjc=; b=fOTm+sLq2HGXLINpCqerZiOp859GE8ncxQQeN01lvxBlIpHlSgYvxur0Bity4TX+uH3PfgWXug+gcayULWa5xsFbwHBtXxv0VOQf3CJ4idDwgdp+yvuU9kjbH47MwRbmidqhePTmd9I4znn3vsoppPB3LfPjx73DUiHgM4BDK2c= Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com (10.167.127.17) by HE1PR05MB1260.eurprd05.prod.outlook.com (10.162.250.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Thu, 26 Oct 2017 12:30:54 +0000 Received: from HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::c524:908c:b99c:3f4b]) by HE1PR0502MB3659.eurprd05.prod.outlook.com ([fe80::c524:908c:b99c:3f4b%13]) with mapi id 15.20.0156.007; Thu, 26 Oct 2017 12:30:54 +0000 From: Matan Azrad To: =?iso-8859-1?Q?N=E9lio_Laranjeiro?= CC: Ophir Munk , Adrien Mazarguil , "dev@dpdk.org" , Thomas Monjalon , Olga Shern , Mordechay Haimovsky Thread-Topic: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions Thread-Index: AQHTTApr4imkSYrit0WXMr28NXPk2aLzBmWAgABxKwCAALwaAIABuooggAAhDoCAAAJoAA== Date: Thu, 26 Oct 2017 12:30:54 +0000 Message-ID: References: <1508752838-30408-1-git-send-email-ophirmu@mellanox.com> <1508768520-4810-1-git-send-email-ophirmu@mellanox.com> <1508768520-4810-5-git-send-email-ophirmu@mellanox.com> <20171024135149.fyg4nzcbygo2amtz@laranjeiro-vm> <20171025075006.znxl7mezy4pfyzsj@laranjeiro-vm> <20171026121219.ke3dz7hv4a5zfpih@laranjeiro-vm> In-Reply-To: <20171026121219.ke3dz7hv4a5zfpih@laranjeiro-vm> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR05MB1260; 6:DbXU7T/k5Qc/ixr6N3poaNUQxUXW0nL8cduZYK0FeZKtRTF1anmGchkRL31+uvVN1OBBfxYdTjgai2v1AwR1j0xz+rF7DUEUMt9UrMGzp4AcKeahs54qIY3c/5VQK50dIPrVwmnBR6KRe1fbmy8JiushWv2sSqA2BQJ9Mqn1dcSho7y0OwsDqRbXJbWVewFqLTclnlmjgh8ZyCBwrfudLoigq+QMFyUK19aFAOEwMPFkwfNruJ8OWZZi1WIi56Z/bNAa49x+uWcP5+fPNreRU0rD7wzkPELtwcumUPDcYxOAaWcGkRznZ/OwZKtys7Z2X+F4tzz/ixEjmZGEl6btb2N0kcQ6BdbFLW4vw5i/r48=; 5:ht4llM7ayRTnEN9knPIjtzY8HAQJlQ4XxwrmrVO06IfsqV4X/SFoGiWEEi4aqA6SLeyHt8d8trR9HrpvcwBo06HBD8v0bX8T/73+Xo1WC1ljE2w2E/eQk2D+vKQmB3olJtMea91O9YaMNGLV9gBQ/KAyF6ZJ275OfCpgzO1AYoY=; 24:aYzwIFat2Br9oCR/NzUlsRqS71CuVQFYwOMC2f9Fah88TMA8x8tWwerJKYUinYsSlR/MSTmXKd/2WbMFN0P0aBbE2FzKJhHSxcA2o5lwuZY=; 7:VQB/l1zoFS8MlVWy74VYkFi3NivZRZsH8b41ZULAJCQHVA7UiQLbXpDf0hYwqlBnOQzlPA5PUaxNSQsrmEF+SxUVrBlh99wnEARUB6MgOZRF+FPboaCu/OkwNAOpTGrTEr19ewlIu6fxZerRe7GSLqZdTASXnmA5Nad885KbN6Nolr5B4P/4DK7+OXsm2V3OMUxPZR/pAkxNdwFa4tgQVOOTGCz2cTJBWFaCnK+se62OZaGs1circ2hnjqhR5LrU x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 353f892e-c444-46a1-6ef8-08d51c6d66f6 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(4534020)(4602075)(4627075)(201703031133081)(201702281549075)(2017052603199); SRVR:HE1PR05MB1260; x-ms-traffictypediagnostic: HE1PR05MB1260: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(100000703101)(100105400095)(3231020)(3002001)(10201501046)(6055026)(6041248)(20161123558100)(20161123564025)(20161123560025)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR05MB1260; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR05MB1260; x-forefront-prvs: 04724A515E x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(376002)(346002)(43544003)(24454002)(199003)(13464003)(189002)(229853002)(55016002)(7696004)(2950100002)(6916009)(5660300001)(53936002)(99286003)(9686003)(8936002)(14454004)(478600001)(81156014)(81166006)(8676002)(97736004)(74316002)(33656002)(6246003)(76176999)(54356999)(4326008)(50986999)(101416001)(107886003)(106356001)(105586002)(93886005)(68736007)(305945005)(7736002)(54906003)(86362001)(53546010)(25786009)(6506006)(3660700001)(3280700002)(316002)(2900100001)(5250100002)(6436002)(66066001)(6116002)(3846002)(189998001)(102836003)(2906002)(427584002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR05MB1260; H:HE1PR0502MB3659.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 353f892e-c444-46a1-6ef8-08d51c6d66f6 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Oct 2017 12:30:54.5357 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR05MB1260 Subject: Re: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Oct 2017 12:31:03 -0000 Hi Nelio Please see my comments below (3). > -----Original Message----- > From: N=E9lio Laranjeiro [mailto:nelio.laranjeiro@6wind.com] > Sent: Thursday, October 26, 2017 3:12 PM > To: Matan Azrad > Cc: Ophir Munk ; Adrien Mazarguil > ; dev@dpdk.org; Thomas Monjalon > ; Olga Shern ; Mordechay > Haimovsky > Subject: Re: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path functions >=20 > On Thu, Oct 26, 2017 at 10:31:06AM +0000, Matan Azrad wrote: > > Hi Nelio > > > > I think the memory barrier discussion is not relevant for this patch > > (if it will be relevant I will create new one). > > Please see my comments inline. >=20 > It was not my single comment. There is also useless code like having nul= l > segments in the packets which is not allowed on DPDK. Sorry, but I can't find comments in the previous mails. Moreover this comment(first time I see it) is not relevant to this patch a= nd asking something else. All what this patch does is to merge 2 functions to prevent double asking about WQ remain space... Remove memory\compiler barriers or dealing with null segments are not in th= e scope here.=20 >=20 > > Regarding this specific patch, I didn't see any comment from you, Are > > you agree with it? > > > > > -----Original Message----- > > > From: N=E9lio Laranjeiro [mailto:nelio.laranjeiro@6wind.com] > > > Sent: Wednesday, October 25, 2017 10:50 AM > > > To: Ophir Munk > > > Cc: Adrien Mazarguil ; dev@dpdk.org; > > > Thomas Monjalon ; Olga Shern > > > ; Matan Azrad > > > Subject: Re: [dpdk-dev] [PATCH v2 4/7] net/mlx4: merge Tx path > > > functions > > > > > > On Tue, Oct 24, 2017 at 08:36:52PM +0000, Ophir Munk wrote: > > > > Hi, > > > > > > > > On Tuesday, October 24, 2017 4:52 PM, N=E9lio Laranjeiro wrote: > > > > > > > > > > On Mon, Oct 23, 2017 at 02:21:57PM +0000, Ophir Munk wrote: > > > > > > From: Matan Azrad > > > > > > > > > > > > Merge tx_burst and mlx4_post_send functions to prevent double > > > > > > asking about WQ remain space. > > > > > > > > > > > > This should improve performance. > > > > > > > > > > > > Signed-off-by: Matan Azrad > > > > > > --- > > > > > > drivers/net/mlx4/mlx4_rxtx.c | 353 > > > > > > +++++++++++++++++++++---------------------- > > > > > > 1 file changed, 170 insertions(+), 183 deletions(-) > > > > > > > > > > What are the real expectation you have on the remaining patches > > > > > of the series? > > > > > > > > > > According to the comment of this commit log "This should improve > > > > > performance" there are too many barriers at each packet/segment > > > > > level to improve something. > > > > > > > > > > The point is, mlx4_burst_tx() should write all the WQE without > > > > > any barrier as it is processing a burst of packets (whereas > > > > > Verbs functions which may only process a single packet). > > > > > > > > > The lonely barrier which should be present is the one to ensure > > > > > that all the host memory is flushed before triggering the Tx door= bell. > > > > > > > > > > > > > There is a known ConnectX-3 HW limitation: the first 4 bytes of > > > > every TXWBB (64 bytes chunks) should be written in a reversed > > > > order (from last TXWBB to first TXWBB). > > > > > > This means the first WQE filled by the burst function is the doorbell= . > > > In such situation, the first four bytes of it can be written before > > > leaving the burst function and after a write memory barrier. > > > > > > Until this first WQE is not complete, the NIC won't start processing > > > the packets. Memory barriers per packets becomes useless. > > > > I think this is not true, Since mlx4 HW can prefetch advanced TXbbs if > > their first 4 bytes are valid in spite of the first WQE is still not va= lid (please > read the spec). >=20 > A compiler barrier is enough on x86 to forbid the CPU to re-order the > instructions, on arm you need a memory barrier, there is a macro in DPDK = for > that, rte_io_wmb(). >=20 We are also using compiler barrier here. > Before triggering the doorbell you must flush the case, this is the only = place > where the rte_wmb() should be used. >=20 We are also using memory barrier only for this reason. > > > It gives something like: > > > > > > uint32_t tx_bb_db =3D 0; > > > void *first_wqe =3D NULL; > > > > > > /* > > > * Prepare all Packets by writing the WQEs without the 4 first bytes= of > > > * the first WQE. > > > */ > > > for () { > > > if (!wqe) { > > > first_wqe =3D wqe; > > > tx_bb_db =3D foo; > > > } > > > } > > > /* Leaving. */ > > > rte_wmb(); > > > *(uin32_t*)wqe =3D tx_bb_db; > > > return n; > > > > > > > I will take care to check if we can do 2 loops: > > Write all last 60B per TXbb. > > Memory barrier. > > Write all first 4B per TXbbs. > > > > > > The last 60 bytes of any TXWBB can be written in any order (before > > > > writing the first 4 bytes). > > > > Is your last statement (using lonely barrier) is in accordance > > > > with this limitation? Please explain. > > > > > > > > > There is also too many cases handled which are useless in bursts > > > situation, > > > > > this function needs to be re-written to its minimal use case i.e. > > > processing a > > > > > valid burst of packets/segments and triggering at the end of the > > > > > burst the > > > Tx > > > > > doorbell. > > > > > > > > > > > Regards, > > > > > > -- > > > N=E9lio Laranjeiro > > > 6WIND >=20 > Regards, >=20 > -- > N=E9lio Laranjeiro > 6WIND