From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2C7B6A0C41 for ; Thu, 30 Sep 2021 11:26:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B11A740DDA; Thu, 30 Sep 2021 11:26:40 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mails.dpdk.org (Postfix) with ESMTP id 6E6754067E for ; Thu, 30 Sep 2021 11:26:38 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10122"; a="205296339" X-IronPort-AV: E=Sophos;i="5.85,335,1624345200"; d="scan'208";a="205296339" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Sep 2021 02:26:36 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,335,1624345200"; d="scan'208";a="487256817" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by orsmga008.jf.intel.com with ESMTP; 30 Sep 2021 02:26:36 -0700 Received: from orsmsx607.amr.corp.intel.com (10.22.229.20) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Thu, 30 Sep 2021 02:26:36 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx607.amr.corp.intel.com (10.22.229.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12 via Frontend Transport; Thu, 30 Sep 2021 02:26:36 -0700 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (104.47.73.49) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.12; Thu, 30 Sep 2021 02:26:35 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NYY74RR6S9HSW5gIwvze9l/vZDV3TTTmzwWm8t0iZeagpLNQqC8BzSRAhuf6udDdrXZKyCLK/EmVXg6Npg5es8hSPZnfeCv0T6LsuYRnBAh1mX7acotAxLLpBqqPL6qPSHQXqzQ8cAZkO5+xw8y3Rtrkkej+GMX+WQtkA+2+mSLdiybMBbEuCOOPanBE4gafUhkGcubTNuNB+eQTIHVu4c6BQRDQTfa3IssVpKPAlUfrZ0EY+ouDErPZ6xkqu04uo8oSL8S3aWn/kCUoOxNnNKOPDHBRzfDUHqPDS+nTmXI8jk1WFGzKcTGW//xJEHmGAEC3nDtsHwLjviLBtxnVIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=qoj/0jvgnBlUWIPJQFX0/cFKoBvCS81DI7lm1U6oO1s=; b=Lc0pbOXmY/lirBPAcVhUM/d91gZHpIZ7ZEa9UWJNoGkhOGcCKDzlMOKD3k9/83Zu72yMJFiBE84/T0CKDSYQUwaMX3buDHYysL7x7cJZPX1cw5+AK1kaHQCzq34ntpSRfujDqV7SQr/zBOw67ID+uLMjUT3TJGtZz2nUqFUvFm1aPOeRMRdcCK+mHZqlOMKKfX6FqFhig5O6PbwPthtu0v0lIPJFSH6XRKOBwQbuAKaHsY5290a7GXD72AkeApW442o5OpOaqBj8qtZ4uzqKIGy/zAsTxmobWtt4USf+xW5e2qqZVLVjhVyUIQEDoi8MW4l3rjQ1eplLEgyKbSNUQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qoj/0jvgnBlUWIPJQFX0/cFKoBvCS81DI7lm1U6oO1s=; b=K8GMZPFzcBK4jDmxAZKdYKvh+gY+dsktZOIm34z8VL1NkhW2ZwSoGDpgeARjwOyjMPn0sxOnBBEsU72DDcLdg6k3ib9hadRJT9y63hTnEgw8EAneJnTBq6kY7+oE7efmWUj0QBeFq29xO7ozeWF1JVvsC8VrMqzgwI4s1U/N0+Q= Received: from PH0PR11MB4776.namprd11.prod.outlook.com (2603:10b6:510:30::14) by PH0PR11MB4886.namprd11.prod.outlook.com (2603:10b6:510:33::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.15; Thu, 30 Sep 2021 09:26:33 +0000 Received: from PH0PR11MB4776.namprd11.prod.outlook.com ([fe80::3174:7553:951:29e6]) by PH0PR11MB4776.namprd11.prod.outlook.com ([fe80::3174:7553:951:29e6%4]) with mapi id 15.20.4566.016; Thu, 30 Sep 2021 09:26:33 +0000 From: "Kinsella, Ray" To: Thomas Monjalon , Jared Brown CC: "users@dpdk.org" , "Richardson, Bruce" , "Ananyev, Konstantin" , "Yigit, Ferruh" Subject: RE: [dpdk-users] DPDK CPU selection criteria? Thread-Topic: [dpdk-users] DPDK CPU selection criteria? Thread-Index: AQHXtR0Jxs/zQOL85kqiEqo8VY3zaKu8QHzQ Date: Thu, 30 Sep 2021 09:26:33 +0000 Message-ID: References: <1988781.SuJIhfhpoT@thomas> In-Reply-To: <1988781.SuJIhfhpoT@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-version: 11.6.200.16 dlp-product: dlpe-windows dlp-reaction: no-action authentication-results: monjalon.net; dkim=none (message not signed) header.d=none;monjalon.net; dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b0a2bdb3-6227-4c2b-5fdd-08d983f464d5 x-ms-traffictypediagnostic: PH0PR11MB4886: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: A8seBWFJHkujT6uaiBbA1UDnDTq/YgyNV/htOsOmDkcbpn79NvDxQKajkHp8uHNBsgGO355T78LDxYH3iu5JIpfXPaOd5pssoR9X30ZQ2Q3hiOTD/iEVu/vMiQVeTvuvu+UbZw7Kd2KKTbrnwPkAGHRhJ9g/U5MV2IQPaZ2rrGY7Pwznl8kcpH4VdSLT//wN8pucmcziDfom+8a0Hy+w0EpjQ/NzxkZoq2+6uSiZRBMIHdMC40pORN+Ie6UM0VPrx6mvBfrvopi1vhBddCBEhSb+YfKaC9fzvN2d4o5+4d3xX/7Qxaz78TQwo6o22B7dQRNVLKiyr8EEowvojaozMDo57yPOzKLz0KvTCEyRPvjeU88hY+jJzKMN1j6LCReTz63hgdLfvf/RKf62zeAgGKKgP2YGkqIxBJJtThLEF7/Gp+red3kEcOWAjTErH5UZTr9/5kb7MjhFzz+39Ki6IZS22DQrB7R4VV22CJgMQzKqB3EmhV80MqT93vSZkKDFAVCRbjrManwQEzhlWn1qdPTfhwPg3t7AID1p+PKx0lbZoBy7GKoZC2n8f5y1JQQRUfm5PmrMnefPzkmW149Iv/TbFdkyrCY175mpJkf21iX1qAkN4xzgqOi8AFafebC4cW7ZY33Az2bGpIn36QQlpkufVPPnphMG5EdxG9B5J6i5DRVHCbR8xoVNJyD4XM76CFLgFE26loySEEM47mYAR7tyeLe+OT0L5aZgEAhPIpsrsv09t3q0NTINn9XcrsBxMOylOeVI5Z3LPh2RRbHL2EdwcvTEOMsnco2XEubjngZ534ynnR3RqvEpLEMBk1FddveeO0gU9Debpk6c/PfK/NoQ9hCix0intkm8tFXP8js= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB4776.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(966005)(76116006)(55016002)(66946007)(66476007)(64756008)(66556008)(38100700002)(186003)(26005)(66446008)(9686003)(19627235002)(52536014)(83380400001)(122000001)(33656002)(7696005)(86362001)(8676002)(5660300002)(53546011)(8936002)(6506007)(54906003)(316002)(110136005)(107886003)(2906002)(71200400001)(38070700005)(4326008)(508600001)(15398625002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?vZT0LA/oS/L3O2O0oNYRRHgzGz7mvxBXuQTDwyu7S7Cqd3wmS3quQeIW2gRb?= =?us-ascii?Q?dCTGPZDpllveie++vyEPjfpGu/ZUdzUPsC6ye1kr+dPL8qE5XkwlxpDobliv?= =?us-ascii?Q?eAf42V2Gu9D2KpqzF6SoDXhWbrkbjnpLkiGeOBaIWxXAimtLSZ0Vj7fZy84M?= =?us-ascii?Q?nWH7awTMVpBr3eGzQGlrTjARP4UhhpGfTBKEh1NwRjULvjQjMHw8NnXY4PAX?= =?us-ascii?Q?ve/zFSYZIRiicvUbi3zzS5XBDdU8VK5rlAfg3Ba/SuZl9w/B4n/eLukNaNgb?= =?us-ascii?Q?yy/2oN6om4Z3QefahmKZ2IHA+PvzWR1cgOZJEtZg+e0VkeHi5lcollut88iv?= =?us-ascii?Q?qqpoTC7UxSvpBpjy8f43fsZIfE6AWD4tHZMrH1K8217xO0rPcNb+6AI0fgrV?= =?us-ascii?Q?NGKFAEheXf8D0VrcvjxQUXwy3UXEzc5OadX9zMwPvZcN+MVp52pAThIMpCh+?= =?us-ascii?Q?5ZP9nN6cnAxsc7xY737bH9oYP/p0n1nn4yaRHEwmHHAAV2hpV9NuahapMQ4F?= =?us-ascii?Q?+WtLSAAyflgBMcPjmVH6yjoGod/mdWCMKxdP78U+lPIdXN6R5+g5PNEaKfs6?= =?us-ascii?Q?W8S82lQgy/cjLLxCACmcBiAyKdm3S92lrflOkcnsLfrkPY/afR/SndtPEU5R?= =?us-ascii?Q?fbgfDA1awjmkLDeR9h6joqW9F6J35R/2MxfvKzgjXzWqTWQyN/VlcUuekmKO?= =?us-ascii?Q?3hgln5CvhqI1Yq5Ji0l4AjkqZu43wMfCF7xdw4b1xCVswZVnCZ42cLqmNqCF?= =?us-ascii?Q?L/qBWWCa5uwnYMT55KBNoqSZF8SG+9pJHjlT8il437MyYOO5o/HyDLpNIlp9?= =?us-ascii?Q?uWSFNe6YB260LsTL6HUVMJMEzdRkPDmPbGJlZbFqVmciRKcHQUz9uI8KvQlv?= =?us-ascii?Q?GUjQm4qJCoR+ruLOJH5xXZuTJ2EY55ng0mj9tUY4ITmdGnGrsSDeOiQkp1Wy?= =?us-ascii?Q?2vglW8ivMPReNGjWuMm6pEwvW4vQrCgDMLsVRlbS2rl52+MxVFLWCecLH17b?= =?us-ascii?Q?QM4Xhyt7A6HSgDA9pn/J/mppXgaSQ9gqudgutlnc7fQNr961lLWbHm12MHc6?= =?us-ascii?Q?xx9qoRvNxl82BNezJe3QRCSf/u9usPyw4w9qMwPq426kE2CDoa9gWgZO51Va?= =?us-ascii?Q?Fjo49qqrFfi7S5vvg+eO113511YYUoGD1Yb5modnXVQwunHT2tWT40DV2EPa?= =?us-ascii?Q?HQEn3qggFINL0KKk87tqdA6uA5Sfvfs2fS+d36XEq2D6YrCbtyUfQlDnYqrG?= =?us-ascii?Q?VImZXSgp1yarfa83pTLGukm70scSMQtoW3rWw846FFrjxqiC/qGhd1clGov1?= =?us-ascii?Q?ZUM=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4776.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: b0a2bdb3-6227-4c2b-5fdd-08d983f464d5 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Sep 2021 09:26:33.5694 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qMa5mrXBDrNXRSXmJKaASOHtyn8O3RnoDthCto+6wpgjn8+S/3tQXMGWXQGiZBXU+hKJYD9d4uIZRIho3icdyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4886 X-OriginatorOrg: intel.com X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi Jared, inline > -----Original Message----- > From: Thomas Monjalon > Sent: Wednesday 29 September 2021 11:30 > To: Jared Brown > Cc: users@dpdk.org; Richardson, Bruce ; > Kinsella, Ray ; Ananyev, Konstantin > ; Yigit, Ferruh > Subject: Re: [dpdk-users] DPDK CPU selection criteria? >=20 > 17/09/2021 17:23, Jared Brown: > > Hello everybody! > > > > Is there some canonical resource or at least a recommendation on how > to evaluate different CPUs for suitability for use with DPDK? >=20 > There are some performance reports with details: > https://core.dpdk.org/perf-reports/ >=20 > It is difficult to give any generic info because it depends on CPU > architecture, use case, and application. > Please keep in mind there are hundreds of API functions so we cannot > have clues for all combinations. >=20 > > My use case is software routers (for example DANOS, 6WIND, TNRS), so I > am mainly interested in forwarding performance in terms of Mpps. > > > > What I am looking for is to develop some kind of heuristics to > evaluate CPUs in terms of $/Mpps without having to purchase hundreds of > SKUs and running tests on them. Maciek Konstantynowicz & Shrikant Shah have done extensive work in this are= a. I recommend reading the following. https://www.lfnetworking.org/wp-content/uploads/sites/55/2019/03/benchmarki= ng_sw_data_planes_skx_bdx_mar07_2019.pdf https://fd.io/docs/whitepapers/performance_analysis_sw_data_planes_dec21_20= 17.pdf Maciek and Shrikant analysis of the performance of DPDK (OVS and FD.io). I'd focus on their methodology here, it should resonate with you. Once you have done this ... you can take a look at my recent presentation w= ith Maciek for more pointers. How to build secure Terabit Network Services with FD.io technologies | Maci= ek Konstantynowicz & Ray Kinsella=20 Slides: https://drive.google.com/drive/folders/1KKxfu45xD785UeUra3-eBsn37OI= _syxp?usp=3Dsharing Video: https://www.youtube.com/watch?v=3DlZ_GqidsnTw > > > > The official DPDK documentation[0] states thus: > > > > "7.1. Hardware and Memory Requirements > > > > For best performance use an Intel Xeon class server system such as Ivy > Bridge, Haswell or newer." > > > > This is somewhat... vague. >=20 > This is only for Intel platforms. > DPDK is supported on AMD and Arm CPUs as well. >=20 > Are you interested only in Intel CPU? > All your questions below are interesting but are very hardware-specific. > +Cc few Intel engineers for specific questions. >=20 > > I suppose one could take [1] as a baseline, which states on page 2 > that an Ivy Bridge Xeon E3-1230 V2 is able to forward unidirectional > flows at linerate using 10G NICs at all frequencies above 1.6 GHz and > bidirectional flows at linerate using 10G NICs at 3.3 GHz. > > > > This however pales compared with [2] that on page 23 shows that a 3rd > Generation Scalable Xeon 8380 manages to very nearly saturate a 100G NIC > at all packet sizes. > > > > As there is almost a magnitude in difference in forwarding performance > per core, you can perhaps understand that I am somewhat at a loss when > trying to gauge the performance of a particular CPU model. Well I'd point out that there is a number of generations between those micr= oprocessors.=20 I'd also point out that considering microprocessors in your analysis, such as Ivybridge (9 years old) and even Skylake (6 years) might be kind of= dated. Having reviewed your other questions below. I'd point out that there are two things at work here which might be confusi= ng.=20 1. Platform improvement over time (Microprocessor & Network Card etc). 2. Software optimization over time, addition of AVX-512 instructions and so= on. > > > > Reading [3] one learns that several aspects of the CPU affect the > forwarding performance, but very little light is shed on how much each > feature on its own contributes. On page 172 one learns that CPU > frequency has a linear impact on the performance. This is borne out by > [1], but does not take into consideration inter-generational gaps as > witnessed by [2]. > > > > This begs the question, what are those inter-generational differences > made of? > > > > - L3 cache latency (p. 54) as an upper limit on Mpps. Do newer > generations have decidedly lower cache latencies and is this the > defining performance factor? > > > > - Direct Data I/O (p. 69)? Is DDIO combined with lower L3 cache > latency a major force multipler? Or is prefetch sufficient to keep > caches hot? This is somewhat confusing, as [3] states on page 62 that > DPDK can get one core to handleup to 33 mpps, on average. On one hand > this is the performance [1] demonstrated the better part of a decade > earlier, but on the other hand [2] demonstrates a magnitude larger > performance per core. > > > > - New instructions? On page 171 [3] notes that the AVX512 instruction > can move 64 bytes per cycle which [2] indicates has an almost 30% effect > on Mpps on page 22. How important is Transactional Synchronization > Extensions (TSX) support (see page 119 of [3]) for forwarding > performance? > > > > - Other factors are mentioned, such as memory frequency, memory size, > memory channels and cache sizes, but nothing is said how each of these > affect forwarding performance in terms of Mpps. The official > documentation [0] only states that: "Ensure that each memory channel has > at least one memory DIMM inserted, and that the memory size for each is > at least 4GB. Note: this has one of the most direct effects on > performance." > > > > - Turbo boost and hyperthreading? Are these supposed to be enabled or > disabled? I am getting conflicting information. Results listed in [2] > show increased Mpps by enabling, but [1] notes that they were disabled > due to them introducing measurement artifacts. I recall some > documentation recommending disabling, since enabling increases latency > and variance. > > > > - Xeon D, W, E3, E5, E7 and Scalable. Are these different processor > siblings observably different from each other from the perspective of > DPDK? Atoms certainly are as [3] notes on page 57, because they only > perform at 50% compared to an equivalent Xeon core. A reson isn't given, > but perhaps it is due to the missing L3 cache? > > > > - Something entirely else? Am I missing something completely obvious > that explains the inter-generational differences between CPUS in terms > of forwarding performance? > > > > > > So, given all this, how can I perform the mundane task of comparing > for example the Xeon W-1250P with the Xeon W-1350P? > > > > The 1250 is older, but has a larger L2 cache and a higher frequency. > > > > The 1350 is newer, uses faster memory, has a higher max memory > bandwidth, PCIe4.0, more PCI lanes and AVX-512. > > > > Or any other CPU model comparison, for that matter? > > > > - Jared > > > > > > [0] > > https://doc.dpdk.org/guides-16.04/linux_gsg/nic_perf_intel_platform.ht > > ml [1] > > https://www.net.in.tum.de/fileadmin/bibtex/publications/papers/ICN2015 > > .pdf [2] > > http://fast.dpdk.org/doc/perf/DPDK_21_05_Intel_NIC_performance_report. > > pdf [3] > > https://www.routledge.com/Data-Plane-Development-Kit-DPDK-A-Software-O > > ptimization-Guide-to-the/Zhu/p/book/9780367373955 > > >=20 >=20 >=20 >=20