From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0070.outbound.protection.outlook.com [157.56.110.70]) by dpdk.org (Postfix) with ESMTP id A0DFF6CB8 for ; Thu, 12 May 2016 16:51:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=TnTOsOAeDg00ICSSv8kJWL40xCx7pdmiflRGSSIWQGk=; b=fkNG7ulA/AnGwe7UfNZO39sjJkI5bo5tqK3DlsiIiNZENYBxo4KOKX68EmFRFcBr8gjdmstuAoZoEppQaAhp4gVLLvIw4iTWB5MuTUnFhvWgG9U+P1eL0FrsVeEj46n7tgcnhNY9HCO0+S1X0EnlWTP7q8Gmx020q2XvCmJVjio= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from localhost.localdomain (111.92.123.202) by BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) with Microsoft SMTP Server (TLS) id 15.1.497.12; Thu, 12 May 2016 14:51:13 +0000 Date: Thu, 12 May 2016 20:20:55 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" CC: "dev@dpdk.org" , "Richardson, Bruce" , "thomas.monjalon@6wind.com" Message-ID: <20160512145054.GA5784@localhost.localdomain> References: <20160512091349.GA10395@localhost.localdomain> <2601191342CEEE43887BDE71AB97725836B4FFE2@irsmsx105.ger.corp.intel.com> <20160512121719.GA1806@localhost.localdomain> <2601191342CEEE43887BDE71AB97725836B500B1@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2601191342CEEE43887BDE71AB97725836B500B1@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [111.92.123.202] X-ClientProxiedBy: MA1PR01CA0059.INDPRD01.PROD.OUTLOOK.COM (10.164.116.159) To BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) X-MS-Office365-Filtering-Correlation-Id: 6179d10d-5a03-4442-4713-08d37a74dedf X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 2:pZrbVK1Ouu9xI37g9RXusLuzpDN8M0ldjRhx2AJJODSyAkfsGCku7RFe8lWnaN0ZxkcGZXYQCMMhc0vG6jvMZcZPVqFcEryMJuNRGnWZs/EnIBSpPZh5V/em0ilWTzOedbnwjXZ0L7xE797ymD7ucjZom06em05QWzZj3uOGz2ZDyMKyoueQG6c5ozhtCQi0; 3:wuNyGRHcqtwOyi/6OYuzX6DtNUEM6Zx40nP6v4kRV3NNuiYmrUP+fW/9nLmpDLs79qZuQ9LB+mRtnQbx3GM0nhdwkTBvCW+eM+NCd3k0gZ27s+sdcTMPdLvOZ1I3PQT2 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 25:Z5iYq8x3e4SqvceqEoUBMPYRi0mtmoqyFg2MTESsZqGKvJH5lZ8s2Q8AzQEBEqQElXpi1ogkIZ/4QWsQYUClL2REtIygdgzsCtH/1aH/0rBRuO1iaFQ1DREJqsDBz3VMxqjXMEKs463YKIvUWDclrMZxd5a7jobwLgittBCAGek7rTfoHxndchb+Qta18VdO+IaGtKUQhPUPWLOWxgoiuO/2ujZDhcldNFuneqKQ4wXI51x9C7jO5kcz1tW9sPK6AEXb8t5ujy7viIJzVotGd7dMBoZUuO9PAmlFxRcmGOu38lU07Y7RFf8/guXEwd3pNXg4V7OgON7y8xImPE+lw/wDSfLK/xjxRa/fpXY9QQE0pJJ/s0x7NwLnYBVxpDL1Quo3VAA7nBX6Y/iIjjBIzlA6rGpyEiFV0UWDZe7g35HrbgWCaevpObBxx+IsFkmNvXHcBgaDVXp0ems4hLmWo0xvcZJOWB/trJ20YOhCfAYTdXOilWvaO5A2D1+agkUaYDGuROggWrlHuu3Q9fyt4tjzPRjk2e2wEJoNBMgOGPhJjbjH/CkawJt2qhWCjdz5yjVHkSwGT3FHE2IQa3NHO5sXwdKlNWelw55XLFZ/tlQaC3BSp4r5yugwcMHyma9lWu7hiAhKHdHdUYauHfwoJ5+E/6S9oIbq06zWNH0Rr4yF1CujuBXyovCM3dUGBgrTBuxRb7strxMevNMdH0VZITV1CSeCz37bpn1IropLVjStdAJBUuIZFXjbKovixD49MEgOvU3k5cSxK4JK0yHTmw== X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 20:+Jb72a5OZZALSIxSdK78ya0V+hDNd8f4KLGFT6ztHJ4iGfhM0ciDRvJK9Mu8mtGjzwKoEOZ5CMcWJU/rYoCksiiikd+/k+QAiCq92xWjlhSXQx7vKAS4mUs06/9++7074pDiyMjADjSUrXMOXe2kECsSphH0WG0xbhxL2BvGOVJCm1uiWUNPhjfN5OosEcH7eKSD03sj1fewBmocidZ7166Di+0XpRev/n1K94qkPtzDhvzTCBSAlrSJ4F1vGVJyqqzCu8gHgNekR/n17wYeOEmRlH+UoVW7Va8OTbHEw+ooikPkpgW49LKqUhxervG/1k2ef1MZGAQ+2RERMHi65TJ1gKXKGYWn8g8lleyJFWbTZ9WERqucbSeHuIuotw3E4c5YGo5LI3+3ClYfiD0jw8xjDU3vZw1m+Qbqcem3aDSAx8R2LjHijwvRc2lEb9UYN6CQb5OgzBY5W/tZ82cZaHRyO1Bl83aeJIDBzaLIGSKQVFVAvUoYDx+B5KsgNxrucGa4qWycYomvr2wsYYZv29XXw3l42a5qeYoL0tD9qasy5olEchT1MoaIZrlKfzl+BkW9C5J+4XS0WFgFUwfN3X9lIrFe6C2heAczWHkhYW8= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:BLUPR0701MB1714; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 4:/2Uuh1fENvTSbnyw5QMslCCZ3ToL4gihKbpuVXvaypREknV5J9bTot8hHOMyfFOlMzIxM+L+5tcNvlsEsUwSz2H10KfkxmxKL2MLHHq9LWnQ7KObZvXb3zcBVDurSHq9lnnVb38J1+52M/evZwhypIS0yTTGm3vKPTFxAUw50JvSS0qP7l/1sJ1EoyJetT8bE4U/iMrx4IjvodWaN695ol33lr2rO0Dkl+9i/SSxriUlzHG675xxjwU5WQA6DetJsOURfHoafNeSMqnkwpn80ISY4RLKwqkrb205uuwPQ0nhC69Zc6p5yiylUJfLZmrNAGQ3eSypWqZTPFOONSdKTwpCjDZg0HR4SGl/AITnKtcjH/Cmg5qjUJikH0ojOE+z X-Forefront-PRVS: 0940A19703 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(24454002)(53754006)(4001350100001)(9686002)(47776003)(50986999)(76176999)(1076002)(6116002)(3846002)(61506002)(42186005)(5004730100002)(110136002)(83506001)(2870700001)(92566002)(23676002)(81166006)(54356999)(189998001)(5008740100001)(50466002)(586003)(2950100001)(66066001)(77096005)(93886004)(33656002)(5009440100003)(2906002); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1714; H:localhost.localdomain; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 23:bAnzTKG9KIAOso82yKVR2g0HUs6KhJkx3Kgvx+ZRCb/+AP7KEUThMYEXDh9dnPcN2Sj0Iw56NFRjYbV4q2rSgEbChzKWLCpCOohrVtaqBcAanor2vw/njDTNY7OWHaKLeMm24aFXpMTaSBb3plR+hx7FqLNFntWGaYauxlb0Op0C6fzj1L+qZ886s9JuxtTm5DG9Gz40R+vojiQ+1oWC5bjHDgF8ZlIWl3c5WKq4LPWq890w9iLgGJNm+RKHYBT/VMT/8m3KYmfxIc/i75o0G0JbftAU3OjP2+OIUknGj9KuWCtUDVf9wEzkzlvh0In/IgyVsNtrKG5pRifRTNQ/jhKnHZYk5YA5SbPfv8fcht5OVM3xXNgaXJYWxoJyYhMaNPS5EcmH4lFbEvzi3kWvUL8ao4XLMgCBmm4oTcaG6gVjg3W8KH1Af4GVF9Gt1AVNu5gfBr+oNEV19yxJwt7hP5LvFRcU6TtXGrqIKlb16fnb40c0vA8IsdrhgC3G+kd+ZdLuinovLFH48lMNwTsNoDP05nyY24RvxIPcBAJarZYDeVHENpY7zXiUtBtbs07hpswXuh1Jv5st3pkPco7n73U/ZZg/5VL1ZjAEgE9cJy+824XZ4yw71AbXPN1bck7DSXVbBJybDO+35sjMbRAYy8RVz4y1HBoPlfSn/0qezNehMez6DP8goRUvFaknMf6qo/Rpu1QOtDCar3lUPh7Xt0VhAIdFj+zXM69wKYmRHsZO0czvgvXIocoF9cjvNqyevy2WkFmQCFwuNhdJLH1k7xAA61gA16oO9CYUUywDys0FZtEEOn2qbjIpMZmYHhaUu5+DYlFJISBr7DNLAhmKdV8tAL+oKdauO8ZdWZhdE64ghbBOk6UlUhpBaqhabQGm8Zx0k1iMuf3Yal2Qg6hakC6Jt3w6QQ4bVnyR02MLhHQ= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 5:KnyMhWrzX5juElrRr4elJo6h13pRF+lhGtgmlKhSRcKi+bRfC9AJbtnhiiY6Dddt6fhiBzwf8JU578xUGquYtI2v4iPM02M9+aWZd10VGatU0D6BGYt4XmBr/XfBI0eO/JHXNRtT0ZMYIGA65CuqZw==; 24:iRYfVn0uXjDlAVRq6a1XGAxqLoPDW66d+c99Xq7+Td+Oo9nCq/kV2eSBTAm7CJAhGjjS8M8zPSt2pIU+0hppaHQsP44EZwLYGwTncqtKbBg=; 7:MLpBFNSdxxV1Mhhp9sZd9OOznDESUug4u2v73FFkLz2M3/grPpKO66FiuObBQzsrD2JCv2H5/EDB9cNwzEFthyBQNsvb73WQqTkA5L1pRS0Iw50O+nu2GytDNdl0ucPqPEwLoAAOoPKioXi7NXn4sj+7K6ss3S2oETJ0f4kPNmu0ow57v8G1SDiw0IvrCnIt SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2016 14:51:13.4277 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1714 Subject: Re: [dpdk-dev] mbuff rearm_data aligmenet issue on non x86 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2016 14:51:18 -0000 On Thu, May 12, 2016 at 01:14:34PM +0000, Ananyev, Konstantin wrote: > > > > On Thu, May 12, 2016 at 10:07:09AM +0000, Ananyev, Konstantin wrote: > > > Hi Jerrin, > > > > > > > > > > > Hi All, > > > > > > > > I would like align mbuff rearm_data field to 8 byte aligned so that > > > > write to mbuf->rearm_data with uint64_t* will be naturally aligned. > > > > I am not sure about IA but some other architecture/implementation has overhead > > > > in non-naturally aligned stores. > > > > > > > > Proposed patch is something like this below, But open for any change to > > > > make fit for all other architectures/platform. > > > > > > > > Any thoughts ? > > > > > > > > ➜ [master] [dpdk-master] $ git diff > > > > diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h > > > > index 529debb..5a917d0 100644 > > > > --- a/lib/librte_mbuf/rte_mbuf.h > > > > +++ b/lib/librte_mbuf/rte_mbuf.h > > > > @@ -733,10 +733,8 @@ struct rte_mbuf { > > > > void *buf_addr; /**< Virtual address of segment > > > > buffer. */ > > > > phys_addr_t buf_physaddr; /**< Physical address of segment > > > > buffer. */ > > > > > > > > - uint16_t buf_len; /**< Length of segment buffer. */ > > > > - > > > > > > > > > There is no need to move buf_len itself, I think. > > > Just move rearm_data marker prior to buf_len is enough. > > > Though how do you suggest to deal with the fact, that right now we blindly > > > update the whole 64bits pointed by rearm_data: > > > > > > drivers/net/ixgbe/ixgbe_rxtx_vec.c: > > > /* > > > * Flush mbuf with pkt template. > > > * Data to be rearmed is 6 bytes long. > > > * Though, RX will overwrite ol_flags that are coming next > > > * anyway. So overwrite whole 8 bytes with one load: > > > * 6 bytes of rearm_data plus first 2 bytes of ol_flags. > > > */ > > > p0 = (uintptr_t)&mb0->rearm_data; > > > *(uint64_t *)p0 = rxq->mbuf_initializer; > > > > > > ? > > > > > > If buf_len will be inside these 64bits, we can't do it anymore. > > > > > > Are you suggesting something like: > > > > > > uint64_t *p0, v0; > > > > > > p0 = &mb0->rearm_data; > > > v0 = *p0 & REARM_MASK; > > > *p0 = v0 | rxq->mbuf_initializer; > > > ? > > > > Due to unaligned rearm_data issue, In ThunderX platform, we need to write > > multiple half word of aligned stores(so masking was better us). > > Ok, so what would be the gain on ARM if you'll make that change? ~4 cpu cycles per packet.Again it may not be ARM architecture specific as ARM architecture does not define instruction latency so it is more of a implementation specific data. > Again, what would be the drop (if any) on IA? > > > But I think, if we can put 16bit hole between port and ol_flags then > > we may not need the masking stuff in ixgbe. Right? > > You mean move buf_len somewhere else (end of cacheline0) and > introduce a 2B hole between port and ol_flags, right? Yes > Yep, that probably wouldn't have any performance impact. I will try two options and send a patch which don't have any performance impact on IA. > > > > > OR > > > > Even better, if we can fill in a uint16_t variable which will replaced > > later in the flow like "data_len"? > > data_len is grouped with rx_descriptor_fields1 on purpose - > so we can update packet_type,pkt_len, data_len,vlan_tci,rss with one 16B write. OK > > Konstantin > > > and move buf_len at end the first cache line? > >or any other thoughts to fix unaligned rearm_data issue? > > > > Jerin > > > > > > > > > > > > If so I wonder what would be the performance impact of that change. > > > Konstantin > > > > > > > > > > /* next 6 bytes are initialised on RX descriptor rearm */ > > > > - MARKER8 rearm_data; > > > > + MARKER64 rearm_data; > > > > uint16_t data_off; > > > > > > > > /** > > > > @@ -754,6 +752,7 @@ struct rte_mbuf { > > > > uint8_t nb_segs; /**< Number of segments. */ > > > > uint8_t port; /**< Input port. */ > > > > > > > > + uint16_t buf_len; /**< Length of segment buffer. */ > > > > uint64_t ol_flags; /**< Offload features. */ > > > > > > > > /* remaining bytes are set on RX when pulling packet from > > > > * descriptor > > > > > > > > /Jerin