From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0059.outbound.protection.outlook.com [207.46.100.59]) by dpdk.org (Postfix) with ESMTP id 36FF4902 for ; Tue, 8 Dec 2015 13:45:55 +0100 (CET) Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@caviumnetworks.com; Received: from localhost.localdomain (111.93.218.67) by BY1PR0701MB1721.namprd07.prod.outlook.com (10.162.111.140) with Microsoft SMTP Server (TLS) id 15.1.337.19; Tue, 8 Dec 2015 12:45:50 +0000 Date: Tue, 8 Dec 2015 18:15:31 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" Message-ID: <20151208124527.GA18192@localhost.localdomain> References: <1449417564-29600-1-git-send-email-jerin.jacob@caviumnetworks.com> <1449417564-29600-2-git-send-email-jerin.jacob@caviumnetworks.com> <2601191342CEEE43887BDE71AB97725836AD15BE@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB97725836AD15BE@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: MAXPR01CA0040.INDPRD01.PROD.OUTLOOK.COM (25.164.146.140) To BY1PR0701MB1721.namprd07.prod.outlook.com (25.162.111.140) X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1721; 2:e24iOS3/4qOEvrEd6bj36Rg9w/aHwExumSEc2jVNe9+WzmDbac6R+AyIp6FbJI+kbsByTFy+eqGLKWBO8iKsOnQlHr/x0H24lOOeOdIiqT7ij0DevGPPbVq0rCT1txbnYD+UNUkeOXdGC6kDLi9x0w==; 3:XnSb+jVcpv32UQOhe2dCzfyZBVPCRLs/hXv/jqM0AtFnGGFo8AaSSa6P8vIWiBux4NDLaDmGuAEz1NBbSMYhH/zFMjccoZdIN9ZzJldeHdeYd30kUJtAGTLrbwGdELfJ; 25:KehgqSASY1rL2yK0CpzGWd81sGRhazzk29aXedMyHltZNfW/cJF09fb8oQjrOgPt4jMkfNVluKt0bk0njIKuC6DSvk5hJicXbv81SqVKDOm9W3bYiTAu9o/pWOIxWCTLABEZrJdK0UnsH1T4ETal1nk7ePNUBk5vX2yPQm88DAay9dbt2XFv7KPMWOPk6+T3vZqyqJXz/W7+bpu0JCG1FYiVmjBZbSMVGuV0ZVtG5xVdYhzVX6cKuFSpfEuUqVYy6hfYUWOKQqClcp6ID3XEoQ== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY1PR0701MB1721; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1721; 20:Uw9znsFZeq56aGQ62rfijHJvLUcnL1QMS/L6wr7tsc4srEPjkP8Of/MLsesfwJk8eWy/xwq9Myrf0G48Nc6GRpaGx6Uynx2k857J3KL5H6D4eE4sKRcK3jfT7MtcoMp2dv+ty9BAHPqphIodTd1rO3vqVN+Js56DrVQo95zoaLFaRxq+XNnLZtYd2VbuSdIJOB130YDLyRcTPGAm7JQ3Uid4+iQxghuGM6R2no+ok1/MKRRFN3qJ/7y62i6/5kUcFfw5UpkulUhE0Bmc5FOxxRVxHy39CoqF9M3tuKL92QJW6hTfkemUyI1xsl1F27UlrNaew5L2TPkt9I3UMTFsNe+T0fuLJeptGYyt9eGbrHepQJvATH4383vW58HtgouYLis4+JHW/YNGzCfDLkgULurGlbXKznKd8K07i2vVcpt9f2i84D5HelBMgba9TTbcAQs+X+/uE6K34BMMRSsFRd0EtxRqKx5GtE/TEPhyP5kBBTML3D/E9pyf4mkyOiSSTw9LM+DvNsDAT9CRayVTo39tCtBSGUqwGvJYKGie181HH/ta20TXMF3BRRuUuN+f9KdRgjgsjVivwGvip5I8PUpudTyPr6owBViUpBW0unY= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(236414709691187); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(3002001)(10201501046); SRVR:BY1PR0701MB1721; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1721; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1721; 4:PQCjmTpOVBCBynT5ua3Sr9Ojj3D+MiL4KVGv0zxg/Z9t3Xj9c13aXi4FbYUryh8mf6fKRVhL9wHn1rfzEMk6NPQx3GpQy3pSY2eXRFAbK+tSl4jjXKwOwd2O7loJkTNtNXbMsdjjRH4RItk0YqxB5acKQfh7nuuDQjNtSdh4u5vbXtz9erNr9g8PBkh2naVBpR5KpeBhNOPFtI/DhPcT+o1ZTr2P6tvtoL3T84lBnKGTRFlmo4nrnOaU3AjZM4yCzi/Y4uUckIihoeQ126uzc7j+q+ujnQT6CsJdlGBSkcR146ZO5T/HYLv7iqstr/HG4HdpUWhXOoHkV85X8szuhsZSawvbDM+eR1kbBqRv528Ux+Dw9K2Kalw1ansniZPCyMEVlGsI/QkIWqnQXsMI/gsUOrP8BKI9d3t3oe+q0CIyjZ+6E76sCXqrGqDHOfae X-Forefront-PRVS: 0784C803FD X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(979002)(6009001)(6069001)(189002)(24454002)(199003)(13464003)(377454003)(105586002)(50466002)(122386002)(5009440100003)(5008740100001)(86362001)(586003)(2950100001)(23726003)(46406003)(6116002)(5004730100002)(1076002)(3846002)(1096002)(87976001)(189998001)(83506001)(101416001)(92566002)(33656002)(77096005)(106356001)(19580405001)(40100003)(19580395003)(76176999)(97736004)(61506002)(66066001)(54356999)(81156007)(47776003)(5001960100002)(50986999)(42186005)(4001350100001)(110136002)(97756001)(7099028)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1721; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: caviumnetworks.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY1PR0701MB1721; 23:nasQShiAfGJugb/zMyBaWZvT8nXUwHOMu/9qUGx?= =?us-ascii?Q?vaqUuuOur2mJeyExjU4YNttK2UPNGq9YRCTbkr1ue0RzdGNwhn1NAB6De7Oi?= =?us-ascii?Q?YJXBQ7a9sBmdCEPorBVBLcoU77Wmcp0DK1C0J2wWeKdQfbFMwFYG5pWrVAGm?= =?us-ascii?Q?0/XS0J6DBZfjB+99Oa+QtpDkwE8WDC/099dDOE6sLJR0Iz2WFGmMNziiWZTg?= =?us-ascii?Q?o2i3gUe66xi/i2V2J2Tt4aj5reUeVjj6RUtth+6LHrL6dqJXYVndb3ba0cGi?= =?us-ascii?Q?bcUHSKVrM7tZlXlxa0murDfnUjKeTpK1f9DWaxjUDkCt3i0IhL2v+PTWuMTt?= =?us-ascii?Q?D6f2MYjFd6ssfEuIprYSRY2h29XQ4C9P7mrbVH161Pj8DH9DzyCPoaRoND7i?= =?us-ascii?Q?9BArN7+F6QT5y9rcsNMhrF+46jJSEE9U7yzbIVc8PMaqv7OGxtBOHk1NBFPN?= =?us-ascii?Q?D3PfEyP4ey8HvyeUMGRqaYPs9PMayWJw6lgL/2E+CgqwnZg0Dh4wV76mionz?= =?us-ascii?Q?bXPNlkA8YnBOlX0xPar4ijzdAAMlosrxnIfAtTNUnBb1FWfoRoK6nsXkYc43?= =?us-ascii?Q?kylBOdhFGKVa0wSWspd2ZnXIfzC+NNqStCMDfgU5zT5qJreH002l4k5P8HKw?= =?us-ascii?Q?Bt4N8R0o+tg49P5/rd19Ed2ubBTVAjFu93Qdg42jxRB/tDFcXU31qdP3h+jL?= =?us-ascii?Q?Bk8uTr/I6mnvj8Pin54rup1jZ2BBe/yVAC6hz0qmlV1qUj3EtR/LyFMIL4Eb?= =?us-ascii?Q?P0lwmEwklXFgzbSeGL2t2qBBpUdbXx1Daz3avAINcJgA7ffeEmf3kyYVmk9/?= =?us-ascii?Q?poZnuaWtBFbnQH9JukijfdPKAswCCPHwdE/z530ysBJ3pyJAel4y5FEhNRhj?= =?us-ascii?Q?P/aQdwABU0Ra2pJG1qZBFsOAkUzkGNGEL2hD+2QwccUfHK3Rhze2nb/V2qmo?= =?us-ascii?Q?TSwhkUYPTCKWhfvwLJtdMJ7YQUVuUNolFlcDFVuPat6AGhYHrx63mntzPppP?= =?us-ascii?Q?hp2WOS2YDhxqzFf/p3+BHfKgJPICxcxE4Yf/XfwjktlLkzpOaxXoG5HMpWoc?= =?us-ascii?Q?FCbHFCORzJZw8/YwXF8IV4mFeNnYSXS2TxkDU4HhIXrj1Aq5EuorpZEs3npS?= =?us-ascii?Q?H6cetcHrMHGN+eBVZNoM6CifiM+Uczv0vBZ/CykU6TA3Ums382k33OgvRvMD?= =?us-ascii?Q?WlC5A8HhXcUwWEQ9WyOFR2TTVDytMyFHSD56vjn/plKFLkHqDFMWxJb3RESV?= =?us-ascii?Q?6HGF13gFmApX+5Sth8oqvcJsvtkZj3O6Lcnclv7ObG9Kp0JdKOLX5F/3PBah?= =?us-ascii?Q?8yWeC3c4RbSKu+PbWR50XoIM=3D?= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1721; 5:Y3LDThgS1iQ/VY/sTJNBrIwitqHCb5U5ZC5w5tbHF4DB0KHlpV8ZlhyTLKexu/pmLV4FdEtVb3/ktHPzx1kMufWyX6h6CGi9hVwfin/HkJo7blArJLnD6F2up6F2MIeAitn/rsRrnS1OpRIGnDgDUw==; 24:NHoy/xC3rV2ZOelJ+zpUP15tYw9evNW/TSgMK3UY2jELjRwh8m1YJfRcpu7Hi8IjlhNmtf6qzVz4JYGfjgDObSGMo8Zxd8VKFmL1ElllPIc= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2015 12:45:50.8734 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1721 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] [PATCH 1/2] mbuf: fix performance/cache resource issue with 128-byte cache line targets X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2015 12:45:55 -0000 On Mon, Dec 07, 2015 at 03:21:33PM +0000, Ananyev, Konstantin wrote: Hi Konstantin, > Hi Jerin, > > > -----Original Message----- > > From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com] > > Sent: Sunday, December 06, 2015 3:59 PM > > To: dev@dpdk.org > > Cc: thomas.monjalon@6wind.com; Richardson, Bruce; olivier.matz@6wind.com; Dumitrescu, Cristian; Ananyev, Konstantin; Jerin > > Jacob > > Subject: [dpdk-dev] [PATCH 1/2] mbuf: fix performance/cache resource issue with 128-byte cache line targets > > > > No need to split mbuf structure to two cache lines for 128-byte cache line > > size targets as it can fit on a single 128-byte cache line. > > > > Signed-off-by: Jerin Jacob > > --- > > app/test/test_mbuf.c | 4 ++++ > > lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h | 4 ++++ > > lib/librte_mbuf/rte_mbuf.h | 2 ++ > > 3 files changed, 10 insertions(+) > > > > diff --git a/app/test/test_mbuf.c b/app/test/test_mbuf.c > > index b32bef6..5e21075 100644 > > --- a/app/test/test_mbuf.c > > +++ b/app/test/test_mbuf.c > > @@ -930,7 +930,11 @@ test_failing_mbuf_sanity_check(void) > > static int > > test_mbuf(void) > > { > > +#if RTE_CACHE_LINE_SIZE == 64 > > RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != RTE_CACHE_LINE_SIZE * 2); > > +#elif RTE_CACHE_LINE_SIZE == 128 > > + RTE_BUILD_BUG_ON(sizeof(struct rte_mbuf) != RTE_CACHE_LINE_SIZE); > > +#endif > > > > /* create pktmbuf pool if it does not exist */ > > if (pktmbuf_pool == NULL) { > > diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h b/lib/librte_eal/linuxapp/eal/include/exec- > > env/rte_kni_common.h > > index bd1cc09..e724af7 100644 > > --- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h > > +++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h > > @@ -121,8 +121,12 @@ struct rte_kni_mbuf { > > uint32_t pkt_len; /**< Total pkt len: sum of all segment data_len. */ > > uint16_t data_len; /**< Amount of data in segment buffer. */ > > > > +#if RTE_CACHE_LINE_SIZE == 64 > > /* fields on second cache line */ > > char pad3[8] __attribute__((__aligned__(RTE_CACHE_LINE_SIZE))); > > +#elif RTE_CACHE_LINE_SIZE == 128 > > + char pad3[24]; > > +#endif > > void *pool; > > void *next; > > }; > > diff --git a/lib/librte_mbuf/rte_mbuf.h b/lib/librte_mbuf/rte_mbuf.h > > index f234ac9..0bf55e0 100644 > > --- a/lib/librte_mbuf/rte_mbuf.h > > +++ b/lib/librte_mbuf/rte_mbuf.h > > @@ -813,8 +813,10 @@ struct rte_mbuf { > > > > uint16_t vlan_tci_outer; /**< Outer VLAN Tag Control Identifier (CPU order) */ > > > > +#if RTE_CACHE_LINE_SIZE == 64 > > /* second cache line - fields only used in slow path or on TX */ > > MARKER cacheline1 __rte_cache_aligned; > > +#endif > > I suppose you'll need to keep same space reserved for first 64B even on systems with 128B cache-line. > Otherwise we can endup with different mbuf format for systems with 128B cache-line. Just to understand, Is there any issue in mbuf format being different across the systems. I think, we are not sending the mbuf over the wire or sharing with different system etc. right? Yes, I do understand the KNI dependency with mbuf. > Another thing - now we have __rte_cache_aligned all over the places, and I don't know is to double > sizes of all these structures is a good idea. I thought so, the only concern I have, what if, the struct split to 64 and one cache line is shared between two core/two different structs which have the different type of operation(most likely). One extensive write and other one read, The write makes the line dirty start evicting and read core is going to suffer. Any thoughts? If its tradeoff between amount memory and performance, I think, it makes sense to stick the performance in data plane, Hence split option may be not useful? right? > Again, #if RTE_CACHE_LINE_SIZE == 64 ... all over the places looks a bit clumsy. > Wonder can we have __rte_cache_aligned/ RTE_CACHE_LINE_SIZE architecture specific, I think, its architecture specific now > but introduce RTE_CACHE_MIN_LINE_SIZE(==64)/ __rte_cache_min_aligned and used it for mbuf > (and might be other places). Yes, it will help in this specific mbuf case and it make sense if mbuf going to stay within 2 x 64 CL. AND/OR can we introduce something like below to reduce the clutter in other places, macro name is just not correct, trying to share the view #define rte_cacheline_diff(for_64, for_128)\ do {\ #if RTE_CACHE_LINE_SIZE == 64\ for_64; #elif RTE_CACHE_LINE_SIZE == 128\ for_128;\ #endif OR Typedef struct rte_mbuf to new abstract type and define for 64 bytes and 128 byte Jerin > Konstantin > >