From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00044.outbound.protection.outlook.com [40.107.0.44]) by dpdk.org (Postfix) with ESMTP id A16647E6A for ; Wed, 19 Oct 2016 19:08:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=QWarx657Z0zXGCcncVKn9+IQoKrdCLxedLp01RIiB7I=; b=jlLpOLZ7Y5bt5oP9oPvms/iq1tAIGdSqlCZ04vdLk/evYUQBI3YnGtCC9gK5xt8+KlKdI+fgrx1YdlRxlcVQr86EPKhgBOwLluvyNyphGbdchUjHuZC6aNU+UPxGa+o/FTyHyC+jMh4IatM8XplPTjVgQOtKIh3xqwaKwi0SoSo= Received: from AM4PR0501MB2785.eurprd05.prod.outlook.com (10.172.216.17) by AM4PR0501MB2753.eurprd05.prod.outlook.com (10.172.216.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.659.11; Wed, 19 Oct 2016 17:08:43 +0000 Received: from AM4PR0501MB2785.eurprd05.prod.outlook.com ([10.172.216.17]) by AM4PR0501MB2785.eurprd05.prod.outlook.com ([10.172.216.17]) with mapi id 15.01.0659.025; Wed, 19 Oct 2016 17:08:43 +0000 From: Oleg Kuporosov To: Olivier Matz , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH 1/3] mbuf: embedding timestamp into the packet Thread-Index: AQHSJV8MMXGUm6DqMkSJqHfcr2VXyKCuYhiAgAGV7zA= Date: Wed, 19 Oct 2016 17:08:43 +0000 Message-ID: References: <1476369308-17021-1-git-send-email-olegk@mellanox.com> <1476369308-17021-2-git-send-email-olegk@mellanox.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=olegk@mellanox.com; x-originating-ip: [82.208.101.9] x-ms-office365-filtering-correlation-id: 42328bf6-3dcb-41fc-1119-08d3f84294ad x-microsoft-exchange-diagnostics: 1; AM4PR0501MB2753; 7:+bKfX7WkuaJgHIVZdzxTfbLXY7vb6dWeS0jgmr90AVrow6MAetGUnMqVfktr5XZdR1Dpnh/rkhGNww77IbXHokLlF3UoXRYJWswemmu7/iVdffRxyfzfcagYQMN/RUat7Nt92cNBfAntmc9/KFUj6VxX9LzDVgjcLuNrti19OlZ5xErgzL94JGAXUDAzrxedB/Hkz84C1fwOOe++MyLVbB8D+RyGz1P2b/vNA4aDVWqAMZpkX9f+AwX7kwigRgOrBq0IGhKFPdFwT1fgxr9PCup1NO0D++nEWFYMSS/edkB0YBq2NZLuxKlxHFDoxXOe5WNJUBO/tmFAJdFkZR/cz6Ot6Dr/w7m29YuvxHthAXQ= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM4PR0501MB2753; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397)(166708455590820); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026); SRVR:AM4PR0501MB2753; BCL:0; PCL:0; RULEID:; SRVR:AM4PR0501MB2753; x-forefront-prvs: 0100732B76 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(199003)(54094003)(189002)(2906002)(7736002)(5660300001)(101416001)(106116001)(81166006)(33656002)(2501003)(3660700001)(305945005)(3280700002)(81156014)(11100500001)(7696004)(5002640100001)(5001770100001)(8676002)(66066001)(92566002)(97736004)(7846002)(86362001)(107886002)(122556002)(76576001)(74316002)(102836003)(586003)(10400500002)(106356001)(54356999)(2950100002)(3846002)(77096005)(19580395003)(2900100001)(87936001)(9686002)(68736007)(105586002)(6116002)(76176999)(8936002)(15975445007)(50986999)(189998001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR0501MB2753; H:AM4PR0501MB2785.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Oct 2016 17:08:43.3290 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2753 Subject: Re: [dpdk-dev] [PATCH 1/3] mbuf: embedding timestamp into the packet X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Oct 2016 17:08:45 -0000 Hello Oliver Great thanks for your review and con > > - include uint64_t timestamp field into rte_mbuf with minimal impact to > > throughput/latency. Keep it just simple uint64_t in ns (more than 580 > > years) would be enough for immediate needs while using full > > struct timespec with twice bigger size would have much stronger > > performance impact as missed cacheline0. > > > > - it is possible as there is 6-bytes gap in 1st cacheline (fast path) > > and moving uint16_t vlan_tci_outer field to 2nd cacheline. > > > > - such move will only impact for pretty rare usable VLAN RX stripping > > mode for outer TCI (it used only for one NIC i40e from the whole set = and > > allows to keep minimal performance impact for RX/TX timestamps. >=20 > This argument is difficult to accept. One can say you are adding a field = for a > pretty rare case used by only one NIC :) It may be looks so, but in fact not only for one NIC. Absence of timestamp= there=20 Required from developers implement its support in out of DPDK data path wit= h Local DPDK patching which also may lead some penalty in accuracy. Good example here is open source implementation of tracing library -=20 https://github.com/wanduow/libtrace/tree/libtrace4 These folks patched DPDK to have timestamp for every packet (Intel DPDK Pat= ches folder) And used it in out of band to store and later processing (lib/format_dpdk.c= ). That was actually the starting point of my investigations. Such approach is working for 1GBb case but not for 10-100 cases. >=20 > Honestly, I'm not able to judge whether timestamp is more important than > vlan_tci_outer. As room is tight in the first cache line, your patch subm= ission > is the occasion to raise the question: how to decide what should be in th= e > first part of the mbuf? There are also some other candidates for moving: = m- > >seqn is only used in librte_reorder and it is not set in the RX part of = a driver. Agree, it is difficult to decide, my thoughts were: - there is hole (6 bytes) which wasn't marked as reserved for any planned e= xtensions; -vlan_tci_outer is being used by only one NIC (i40e) so far, 2nd NIC (fm10k= ) is using it With comment: * mbuf->vlan_tci_outer is an idle field in fm10k driver, * so it can be selected to store sglort value. To store some another value under some specific "if". Also for i40e it is under #ifndef RTE_LIBRTE_I40E_16BYTE_RX_DESC which per = doc should be Enabled for high throughput of small packets. So in default case (disabled)= it anyway has some performance penalty with using 32 bytes descriptor and moving it to 2n= d CL would not hit big additional penalty. Unfortunately I have no such NIC to measure= . Is there any data how often double tagging in being used in DPDK applicat= ions? Another my thought was to have at the end of 1st CL enum which may hold Reserved fields per specific use cases and data widths (uint8, 2xuint4, 4xu= int2, 8xbytes). =20 >=20 > About the timestamp, it would be valuable to have other opinions, not onl= y > about the placement of the field in the structure, but also to check that= this > API is also usable for other NICs. Sure, but I didn't change timesync/timestamping API itself. > Have you measured the impact of having the timestamp in the second part > of the mbuf? Yes, the worst case with prefetching of 2nd CL it is 5.7-5.9 % penalty for = combined RX+TX And expectedly much worse without prefetching. In the best case it is 0.3..0.5 % for RX only. It can be explained by much = harder cache trashing when TX is "on". =20 > Changing the mbuf structure should happen as rarely as possible, and we > have to make sure we take the correct decisions. I think we will discuss = this at > dpdk userland 2016. Oh, yes, please discuss, I would not be able to join. :( > Apart from that, I wonder if an ol_flag should be added to tell that the > timestamp field is valid in the mbuf. Oliver, there is PKT_RX_IEEE1588_TMST flag already. Best regards, Oleg.