From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0098.outbound.protection.outlook.com [207.46.100.98]) by dpdk.org (Postfix) with ESMTP id 8754F58DF for ; Wed, 4 May 2016 15:43:01 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-caviumnetworks-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=RORFIwTNfXZT6xWQkoFliCE6xFHvvehzhBAqMBOwT9o=; b=gw70R3VP72y5J2+Gi3r7jQ2saZX4VNn1jSirpKkbveTrI6PF9/Z1NsKNT7EYilcqWSmH9td1P1yURyX0uzMLS8KbMyG2dnvZ6jodhXz1jydJDkca3uXisu87m1sKWkVWe6SOJFk7FtFcXa7dxm31gdfXpo5oOeh5LWwVegM8/SY= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from localhost.localdomain (122.171.53.0) by BLUPR0701MB1715.namprd07.prod.outlook.com (10.163.85.141) with Microsoft SMTP Server (TLS) id 15.1.485.9; Wed, 4 May 2016 13:42:57 +0000 Date: Wed, 4 May 2016 19:12:32 +0530 From: Jerin Jacob To: Bruce Richardson CC: , Message-ID: <20160504134231.GA13071@localhost.localdomain> References: <1462176465-21562-1-git-send-email-jerin.jacob@caviumnetworks.com> <1462279327-9876-1-git-send-email-jerin.jacob@caviumnetworks.com> <20160504110950.GB25492@bricha3-MOBL3> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20160504110950.GB25492@bricha3-MOBL3> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [122.171.53.0] X-ClientProxiedBy: PN1PR01CA0018.INDPRD01.PROD.OUTLOOK.COM (10.164.137.25) To BLUPR0701MB1715.namprd07.prod.outlook.com (10.163.85.141) X-MS-Office365-Filtering-Correlation-Id: d9ceb288-3a4d-466f-0894-08d37422016c X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1715; 2:QtNqQ9RGUOvlh9KkkfaRcxtMjdbIdLHOqM+VWtsEZnPF/5tiKvQp7pw/v4j1mOmTclrHg03EceymIDpdisKl4qgyBsuZAJQ1/NMEj25/IxgLli6p7N+Ag1TZ8412VaNSmTeaZLOHifXNOTGSfBHjEZxfTScKwcOjD7Z3louBkIEuSWAa+xTF/UdE/dpJp3JY; 3:Oy79k8rLPJPQO4MLDsESkMSjTMK/hSLifZIZZGaFtxfHKunz223mifHpNh5PiBRAteLNJ+WPri4p7k/tl1gBPJn+o80JfZ1bRE5jrsHI9VCl7rwq2zeSqaXY28jBEGEd; 25:/q9lnDrM5S8j/ynkwbs061tR+Sh1cjNz50DUKqF3q7yCPdRmjTM1nz/H5engudwpE/M2tGktzAAvC5Ujzr5/L7LzD62OgNw3453BCW52w3CBFJF6vAZddqWUew6GJ0tX6rVYA4O0i6s++Jy4TxEl0PJvNCYlYUrmzsXgKIHqOupWh4wddMuwrkEAjYWH1GnHk8NuuoYoR13VupfT1JBbQDzSvwWdNi4SNzKFUVl/2Yllj/D0XPdLcpuiYk+SrKz1/vhmmSRVQe21v4rcmEf+wByq9Vmb3NEInRZjw9IWQYYPtLMUdO4kfIRm04HKT4aalfr4miINj+xjyk9jDs/j+TFhSc9mOWu6+uYWKh9feCLWCF7tmycVGl2WQJZTpyQonKVPCTDR9mliEiuKkJDF/qwUAdSFlHiWq9GiAq6yTcAoy9dJGENJ54/0a3daiVGg X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0701MB1715; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1715; 20:xHofRki3YTf4JyyRdh4Txecls2ryxkCSr/Vc4OW5qTSgOAMS7xKgvfUHfX9aX198O7W3rHNN7rDnE+H01ujBQ3Yk0MUmjrQYo4AkFhz2ErUehTrVDrr4STINGjfrZSAAswX7v6r1BmgyiUH4nOOMMNwphNdaDc4GeTA6zqawLgdBXcYF5rpNWZO5YWckmYImiqa68wNVKNbu20nuZ9xyssUU0J5UHqJZawdOi55gn7uxwNuT8PPE4o809peN0THfaqgq4M5GirBvtzMAspWuUbJaKB4+mza4ZM7t8eDYF/E1jKmLAFNtmSkQy/G+41DUjF68O8JVb5qQNKlEwIkS6tw5DPOOouWaPt4KEpuql68K3o8nt9DRxYMoPXej9sgkkUiNvsYH8/cjsPcbpKT7LAZZwmNs0t6u3oaef3o6apJYqiG1SLrVrOdoZX/mBcJ3znKArgRfOPn0NllXNX/23sgM7gxwzGne/bNQIkDSfGBuxMAU5udlwWrEWwnBr53Rvibe+gJRnez9BR15O5yo2hFymVK7oF1lY1YAu4O/jTn6KoFdH3UtO1NE0MY/eNBL9Fyq0fTqtiSXxFRCkjtrbE/imecNUL3+ve9z2ybRvwE= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(9101521098)(9101528026)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BLUPR0701MB1715; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1715; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1715; 4:GkCp+W8oR8kLq16I1bkFRdevfrNdzoeqXFIy45jkvRcamDv9+N83y1XgRz6m3XtKt0cECSrZBEA56zIF2I6/IOWFLZRmQnZ9662vAvA6VJTSCiIbB7S2ZmPramUIYW4UKTdhMaBdGAJo3Yb5OjXP+ud8K7yuA4rGYf94fiaq2q1Qs/fLzns2CIUXA+xoLPWmJSo7Ao3bgm+aWCl2G9RrViFRzRC2tRCP5TOu0v9LGyvwbGW/YjfxemXf6PYzREfUNk9VMMXXmLa0tYD+XmTtbYj6kSvuVUm+z+ayxV1pvtLr27bVQLU6tamPYK2eLrEbs0MvyeLpnVx39eMQq12rSCX/l16R5zzSOFJL/DwjRBWFoD+2VhqNy6Nec8Byt6hDKMF4XoHOD+z6u4j/UwquVpr4xiRgpi+c4cZ3VUiMEg0= X-Forefront-PRVS: 093290AD39 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(24454002)(24434003)(189998001)(66066001)(4326007)(76176999)(50986999)(15975445007)(54356999)(110136002)(5890100001)(9686002)(47776003)(5004730100002)(5008740100001)(46406003)(77096005)(86362001)(2906002)(19580395003)(3846002)(19580405001)(2950100001)(61506002)(4001350100001)(15395725005)(33656002)(6116002)(586003)(23726003)(1076002)(42186005)(97756001)(83506001)(81166005)(50466002)(92566002); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1715; H:localhost.localdomain; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0701MB1715; 23:G2vBaFC9I0clnSnzN5RDEU8hbNROz5akvRnK0br?= =?us-ascii?Q?PVQ1ojI3awA2haADZi8XbfESychGH+ECjq2wVInJVLVqqIKp8ACsej1yUYPf?= =?us-ascii?Q?srrzmVVbJQFruaygL/tjcFvXeweMg3PcyS7N6mPNv8b7t1SaCi+0lAjJvoRi?= =?us-ascii?Q?6H78K+M+biCbT1WARhzKjKI8f/sSFiBbED3Oq3EAhB08K6S3zD0CGeQISpgl?= =?us-ascii?Q?Qv9RDg8MYqFJ/06SStNImhu15naXozx5FpIntEBC2V2Vr0jVmQh+5LVCSxPD?= =?us-ascii?Q?Pzfg6oam+Q6noCv/h10VLk56jo4ZcKeV99znl4tiZDCA+Ulxnmd+/PCv979E?= =?us-ascii?Q?E6M59PFfmwS6RhxxTKhO+L/o34rWOuSfO3xftRpjiYm4nVZhoOe0z8ADJoPk?= =?us-ascii?Q?T39PXerbPlYGp8lRWkoRB9EvsLpo0rC5moUGkiHvhcpdie6xBP1pOgUsdSGu?= =?us-ascii?Q?+gIozzW1JAkGPFch14aOe1svLxLb+KkBEEjUmSqpNKM95PiSA8z8u7B6I4Nm?= =?us-ascii?Q?n3U/3sF9B+r93i3vSmcQ4Gw+l9Wm0YKyVRaxIJWCZgJLaJ4eDQngnsic4E3D?= =?us-ascii?Q?EgddtjadVTVitFDvzX+iqds7xdSx9p9DAOTLHPJMeWmFTi+qoH+RHgFo1d1y?= =?us-ascii?Q?l5AhjkcW8HDJBv7ksiRvXTBlzWDH06uqJTtFM34TiK3DIQE4hziVfSEkCIKS?= =?us-ascii?Q?8NCrjWj5G9FqegYTDZKX3aA7B7OZ8Fd3I8AxKmVJA+XEQLF1BylrxV02Aynn?= =?us-ascii?Q?wEU6QOZniYSRb4Fi4Hf8A+K0/yf5kdZRkcZz9+1C0NM4ADQYvpeotCr/4SOg?= =?us-ascii?Q?4vBrsUW0tP8j03QHNuQH/BCNZr/N4PllgsDA468NaUkP703n+jZbBOF22u0p?= =?us-ascii?Q?oPiUJtr8IqsvQ6iWocgyFjye/IZM1dE/rrXOoMwxG+PRtDX3RrFddMEie7fM?= =?us-ascii?Q?c5ltcKmcCA1KH9I3eozMwUwWIJwHnnZavg0wGrCfPQTJOJPQFTJTMBqeHQQZ?= =?us-ascii?Q?wzrCOFzdXfOVZHn0BXbcPsAwOtEK3i5L98RpbVLhH/yjFaPM3GVSBpBzAjU6?= =?us-ascii?Q?2iYitJUbzlk3zXY3M4MM4ia8QMpKZ?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1715; 5:UZXf8JKxYBdEqibSNl9/1IB3M+IEqEvJZZzxsgNx7jwPTBDdJ4XELy4BxGduScTqPUWLOGmMfIsjlY5RT1NnGnwi60QaGxe0Ql1PnUcoj26kQ+RkLj8OI+t+XC67MF+RtfsZsMYAzAe3JwTHL2yhkA==; 24:ToMVxAUvC+Nm48mtWl0f95kPfMCfa4GxLMxcMJcBY+95AyLD4+7vEQu7o7snF3QfaSyg1xA/KD+9ZkUYofr08vn9z8iag5LzRQP+3pjTEiI=; 7:PaQ96T/NXJNuTUE9dQQZknXfuTFaKaro3xSJ+tLfhCNcn8DzT8TNcJfLyJEkmV21YqK2WVoklfyy4PMFP8vZtb1R8HU2hiaDZ1WmCnIrnvZ45j7isggU5wy5tc6bjW8Gxj89Y0ga5pye7F2o6BCxBFPnOQ70FCBEDIrrLSK/8Rs1OQwWP0ObLDAD+IYkhJlv SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 May 2016 13:42:57.2656 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1715 Subject: Re: [dpdk-dev] [PATCH v2] ethdev: make struct rte_eth_dev cache aligned X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2016 13:43:02 -0000 On Wed, May 04, 2016 at 12:09:50PM +0100, Bruce Richardson wrote: > On Tue, May 03, 2016 at 06:12:07PM +0530, Jerin Jacob wrote: > > Elements of struct rte_eth_dev used in the fast path. > > Make struct rte_eth_dev cache aligned to avoid the cases where > > rte_eth_dev elements share the same cache line with other structures. > > > > Signed-off-by: Jerin Jacob > > --- > > v2: > > Remove __rte_cache_aligned from rte_eth_devices and keep > > it only at struct rte_eth_dev definition as suggested by Bruce > > http://dpdk.org/dev/patchwork/patch/12328/ > > --- > > lib/librte_ether/rte_ethdev.h | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/librte_ether/rte_ethdev.h b/lib/librte_ether/rte_ethdev.h > > index 2757510..48f14d5 100644 > > --- a/lib/librte_ether/rte_ethdev.h > > +++ b/lib/librte_ether/rte_ethdev.h > > @@ -1615,7 +1615,7 @@ struct rte_eth_dev { > > struct rte_eth_rxtx_callback *pre_tx_burst_cbs[RTE_MAX_QUEUES_PER_PORT]; > > uint8_t attached; /**< Flag indicating the port is attached */ > > enum rte_eth_dev_type dev_type; /**< Flag indicating the device type */ > > -}; > > +} __rte_cache_aligned; > > > > struct rte_eth_dev_sriov { > > uint8_t active; /**< SRIOV is active with 16, 32 or 64 pools */ > > -- > > Hi Jerin, Hi Bruce, > > have you seen a performance degradation due to ethdev elements sharing a cache No. Not because of sharing the cache line. > line? I ask because, surprisingly for me, I actually see a performance regression I see performance degradation in PMD in my setup where independent changes are causing the performance issue in PMD(~<100k). That's the reason I thought making aligned cache line stuff where ever it makes sense so that independent change shouldn't impact the PMD performance and this patch was an initiative for the same. > when I apply the above patch. It's not a big change - perf reduction of <1% - but > still noticable across multiple runs using testpmd. I'm using two 1x40G NICs > using i40e driver, and I see ~100kpps less traffic per port after applying the > patch. [CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz] This particular patch does not have any performance degradation in my setup. CPU: ThunderX > > Testpmd cmd and output shown below. > > Regards, > /Bruce > > $ sudo ./x86_64-native-linuxapp-gcc/app/testpmd -c C0000 -n 4 -- --rxd=512 --txd=512 --numa > EAL: Detected 36 lcore(s) > EAL: Probing VFIO support... > PMD: nfp_net_pmd_init(): librte_pmd_nfp_net version 0.1 > > EAL: PCI device 0000:01:00.0 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:01:00.1 on NUMA socket 0 > EAL: probe driver: 8086:1521 rte_igb_pmd > EAL: PCI device 0000:05:00.0 on NUMA socket 0 > EAL: probe driver: 8086:154a rte_ixgbe_pmd > EAL: PCI device 0000:05:00.1 on NUMA socket 0 > EAL: probe driver: 8086:154a rte_ixgbe_pmd > EAL: PCI device 0000:08:00.0 on NUMA socket 0 > EAL: probe driver: 8086:154a rte_ixgbe_pmd > EAL: PCI device 0000:08:00.1 on NUMA socket 0 > EAL: probe driver: 8086:154a rte_ixgbe_pmd > EAL: PCI device 0000:81:00.0 on NUMA socket 1 > EAL: probe driver: 8086:1584 rte_i40e_pmd > PMD: eth_i40e_dev_init(): FW 5.0 API 1.5 NVM 05.00.02 eetrack 80002281 > EAL: PCI device 0000:88:00.0 on NUMA socket 1 > EAL: probe driver: 8086:1584 rte_i40e_pmd > PMD: eth_i40e_dev_init(): FW 5.0 API 1.5 NVM 05.00.02 eetrack 80002281 > Configuring Port 0 (socket 1) > Port 0: 68:05:CA:27:D4:4E > Configuring Port 1 (socket 1) > Port 1: 68:05:CA:27:D2:0A > Checking link statuses... > Port 0 Link Up - speed 40000 Mbps - full-duplex > Port 1 Link Up - speed 40000 Mbps - full-duplex > Done > No commandline core given, start packet forwarding > io packet forwarding - CRC stripping disabled - packets/burst=32 > nb forwarding cores=1 - nb forwarding ports=2 > RX queues=1 - RX desc=512 - RX free threshold=32 > RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > TX queues=1 - TX desc=512 - TX free threshold=32 > TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > TX RS bit threshold=32 - TXQ flags=0xf01 > Press enter to exit > > Telling cores to stop... > Waiting for lcores to finish... > > ---------------------- Forward statistics for port 0 ---------------------- > RX-packets: 1940564672 RX-dropped: 1456035742 RX-total: 3396600414 > TX-packets: 1940564736 TX-dropped: 0 TX-total: 1940564736 > ---------------------------------------------------------------------------- > > ---------------------- Forward statistics for port 1 ---------------------- > RX-packets: 1940564671 RX-dropped: 1456036082 RX-total: 3396600753 > TX-packets: 1940564736 TX-dropped: 0 TX-total: 1940564736 > ---------------------------------------------------------------------------- > > +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ > RX-packets: 3881129343 RX-dropped: 2912071824 RX-total: 6793201167 > TX-packets: 3881129472 TX-dropped: 0 TX-total: 3881129472 > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Done. > > Shutting down port 0... > Stopping ports... > Done > Closing ports... > Done > > Shutting down port 1... > Stopping ports... > Done > Closing ports... > Done > > Bye... > >