From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) by dpdk.org (Postfix) with ESMTP id DD70F595A for ; Mon, 2 Nov 2015 23:19:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=17874; q=dns/txt; s=iport; t=1446502759; x=1447712359; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=Gho2ge55PsRH+KW54RF9DruKaqcWUbnrQHisF92D2Fo=; b=fEwjVwZhkMaoy2L5D8NjvQtoIZ+OAsfPL26oFlqpna/7BqsOFFSNmMki XQ/ftUvCCqkgnl3uW2IwAz5TNc8S2ecOC9xszkLdza+bd9r82pSp7QA5H FsKOG5fBDElkS+wNrvHK6qOYeN0KaFRu8eIg9IsGABqWzOCGJtmRbg1bY w=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0D/AQC14DdW/5ldJa1egm5NgUIGv0EBDYFahhkCgTU4FAEBAQEBAQGBCoQ1AQEBAwEnTQUFCwIBCBEDAQIBJwcyFAkIAgQOBYgoCMIBAQEBAQEBAQEBAQEBAQEBAQEBAQEBGIt1hH4WhCwFjVSFEoNdAYgMhRiBWYQ/gySFa4khg3EBHwEBQoQEcoR3gQcBAQE X-IronPort-AV: E=Sophos;i="5.20,236,1444694400"; d="scan'208,217";a="204206561" Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by alln-iport-5.cisco.com with ESMTP; 02 Nov 2015 22:19:17 +0000 Received: from XCH-ALN-002.cisco.com (xch-aln-002.cisco.com [173.36.7.12]) by rcdn-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id tA2MJGud004304 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Mon, 2 Nov 2015 22:19:16 GMT Received: from xch-rcd-004.cisco.com (173.37.102.14) by XCH-ALN-002.cisco.com (173.36.7.12) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Mon, 2 Nov 2015 16:19:15 -0600 Received: from xch-rcd-004.cisco.com ([173.37.102.14]) by XCH-RCD-004.cisco.com ([173.37.102.14]) with mapi id 15.00.1104.000; Mon, 2 Nov 2015 16:19:15 -0600 From: "shesha Sreenivasamurthy (shesha)" To: Arnon Warshavsky Thread-Topic: [dpdk-dev] Reshuffling of rte_mbuf structure. Thread-Index: AQHRE5bISS1IQdF67k2r5hvr+GvFxJ6G/a2AgAJVlQD//505AIAAh1wA//+4BAA= Date: Mon, 2 Nov 2015 22:19:15 +0000 Message-ID: References: <20151102082420.768aea4a@xeon-e3> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.24.149.31] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Reshuffling of rte_mbuf structure. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2015 22:19:18 -0000 Ok, You are saying re-order the fields based on the configurations params. = I took word "NO" in the param to eliminate. Sure, this does not require and= change in the code that uses it. Will it not now boil down to same as havi= ng completely different layout definition and be more messier ? For example: Rather than having: #ifdef NO_TX_OFFLOAD Struct mbuf_rte { fieldA field1 field2 fieldB field4 filed5 }; #endif #ifdef NO_MULTISEG Struct mbuf_rte{ fieldA field2 field1 fieldB filed5 field4 } #endif We end up having Struct mbuf_rte { fieldA #ifdef NO_TX_OFFLOAD field1 field2 #endif #ifdef NO_MULTISEG field2 field1 #endif fieldB #ifdef NO_TX_OFFLOAD field4 field5 #endif #ifdef NO_MULTISEG field5 field4 #endif }; -- - Thanks char * (*shesha) (uint64_t cache, uint8_t F00D) { return 0x0000C0DE; } From: Arnon Warshavsky > Date: Monday, November 2, 2015 at 10:35 AM To: Cisco Employee > Cc: Stephen Hemminger >, "dev@dpdk.org" > Subject: Re: [dpdk-dev] Reshuffling of rte_mbuf structure. If NO_TX_OFFLOAD only changes the layout in terms of relative field locatio= n in cache lines, and does not eliminate the fields themselves why should the using code be affected? On Mon, Nov 2, 2015 at 8:30 PM, shesha Sreenivasamurthy (shesha) > wrote: One issue I see with optimization config options such as NO_TX_OFFLOAD, NO_= MULTISEG, NO_REFCOUNT is: It is not sufficient to have those "Ifdefs" insid= e mbuf structure, but should be sprinkled all over the code where correspon= ding fields are used. This may make the code messier. -- - Thanks char * (*shesha) (uint64_t cache, uint8_t F00D) { return 0x0000C0DE; } From: Stephen Hemminger > Date: Monday, November 2, 2015 at 8:24 AM To: Arnon Warshavsky > Cc: Cisco Employee >, "dev@dpdk.o= rg" > Subject: Re: [dpdk-dev] Reshuffling of rte_mbuf structure. On Sun, 1 Nov 2015 06:45:31 +0200 Arnon Warshavsky > wrote: My 2 cents, This was brought up in the recent user space summit, and it seems that indeed there is no one cache lines arrangement that fits all. OTOH multiple compile time options to suffice all flavors, would make it unpleasant to read maintain test and debug. (I think there was quiet a consensus in favor of reducing compile options in general) Currently I manage similar deviations via our own source control which I admit to be quite a pain. I would prefer an option of code manipulation/generation by some script during dpdk install, which takes the default version of rte_mbuf.h, along with an optional user file (json,xml,elvish,whatever) defining the structure replacements, creating your custom version, and placing it instead of the installed copy of rte_mbuf.h. Maybe the only facility required from dpdk is just the ability to register calls to such user scripts at some install stage(s), providing the mean along with responsibility to the user. /Arnon On Sat, Oct 31, 2015 at 6:44 AM, shesha Sreenivasamurthy (shesha) < shesha@cisco.com> wrote: > In Cisco, we are using DPDK for a very high speed packet processor > application. We don't use NIC TCP offload / RSS hashing. Putting those > fields in the first cache-line - and the obligatory mb->next datum in the > second cache line - causes significant LSU pressure and performance > degradation. If it does not affect other applications, I would like to > propose reshuffling of fields so that the obligator "next" field falls in > first cache line and RSS hashing goes to next. If this re-shuffling indee= d > hurts other applications, another idea is to make it compile time > configurable. Please provide feedback. > > -- > - Thanks > char * (*shesha) (uint64_t cache, uint8_t F00D) > { return 0x0000C0DE; } > Having different layouts will be a disaster for distro's they have to choos= e one. And I hate to introduce more configuration! But we see the same issue. It would make sense if there were configuration = options for some common optimizations NO_TX_OFFLOAD, NO_MULTISEG, NO_REFCOUNT and t= hen the mbuf got optimized for those combinations. Seems better than config opt= ions like LAYOUT1, LAYOUT2, ... In this specific case, I think lots of driver could be check nb_segs =3D=3D= 1 and avoiding the next field for simple packets. Long term, I think this will be losing battle. As DPDK grows more features,= the current mbuf structure will grow there is really nothing stopping the bloat of meta= data. -- Arnon Warshavsky Qwilt | work: +972-72-2221634 | mobile: +972-50-8583058 | arnon@qwilt.com