From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E7B95A04F1 for ; Thu, 18 Jun 2020 12:26:29 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D27251BEE1; Thu, 18 Jun 2020 12:26:29 +0200 (CEST) Received: from alln-iport-1.cisco.com (alln-iport-1.cisco.com [173.37.142.88]) by dpdk.org (Postfix) with ESMTP id CAB521BED1 for ; Thu, 18 Jun 2020 12:26:28 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=822; q=dns/txt; s=iport; t=1592475988; x=1593685588; h=from:to:cc:subject:date:message-id: content-transfer-encoding:mime-version; bh=AXM/25J28r8GbglAIYnE6Ci3GySAzkooVp43ToLnHtY=; b=e4xfRdoYBO6uiaTc1xXKYGF0PuPLFi8bQ7rZ6MaMXogLTr1GcxseoRM5 TVvVmf8sCpvqj1cCb0/3/8yH3yenb4FolEQnwTM0q9fThT/ml1yRlEoFD sLpggIP04xuXYNMV44jmo6zx0kRUeTtaLtPxx8bZkztS6enTOL+D0MqmM M=; IronPort-PHdr: =?us-ascii?q?9a23=3An4Kp1hRYVmHLcu4+WqBzWKwmptpsv++ubAcI9p?= =?us-ascii?q?oqja5Pea2//pPkeVbS/uhpkESQBN+J5+pIiKzRqa+zEWAD4JPUtncEfdQMUh?= =?us-ascii?q?IekswZkkQmB9LNEkz0KvPmLklYVMRPXVNo5Te3ZE5SHsuta0XeoTu19zFBUh?= =?us-ascii?q?n6PBB+c+LyHIOahs+r1ue0rpvUZQgAhDe0bb5oahusqgCEvcgNiowkIaE0mR?= =?us-ascii?q?Y=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0DWHABZQOte/4MNJK1mHAEBATwBAQQ?= =?us-ascii?q?EAQECAQEHAQEVgUoCgVBRB4FHLywKh2ADjT+YUoEugSQDVQsBAQEMAQEtAgQ?= =?us-ascii?q?BAYREAoIlAiQ2Bw4CAwEBCwEBBQEBAQIBBgRthVsBC4VyAQEBAQIBEigGAQE?= =?us-ascii?q?3AQQNAQg2QiYBBA4NGoVQAw4gAQOsHAKBOYhhdIE0gwEBAQWFTBiCDgmBOAG?= =?us-ascii?q?CZol4GoFBP4FUgh8uPoQ/g0WCLbRNCoJamUGCcIkbkmORI545AgQCBAUCDgE?= =?us-ascii?q?BBYFaAy89gRlwFYI4bFAXAg1WkTmKVnQ3AgYIAQEDCXyOcgGBEAEB?= X-IronPort-AV: E=Sophos;i="5.73,526,1583193600"; d="scan'208";a="498443998" Received: from alln-core-1.cisco.com ([173.36.13.131]) by alln-iport-1.cisco.com with ESMTP/TLS/DHE-RSA-SEED-SHA; 18 Jun 2020 10:26:27 +0000 Received: from XCH-ALN-005.cisco.com (xch-aln-005.cisco.com [173.36.7.15]) by alln-core-1.cisco.com (8.15.2/8.15.2) with ESMTPS id 05IAQRrw000478 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Thu, 18 Jun 2020 10:26:27 GMT Received: from xhs-rtp-002.cisco.com (64.101.210.229) by XCH-ALN-005.cisco.com (173.36.7.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 18 Jun 2020 05:26:27 -0500 Received: from xhs-rtp-001.cisco.com (64.101.210.228) by xhs-rtp-002.cisco.com (64.101.210.229) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 18 Jun 2020 06:26:26 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (64.101.32.56) by xhs-rtp-001.cisco.com (64.101.210.228) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Thu, 18 Jun 2020 06:26:26 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KbxnjamFpBDcK6CxpG9Ej6AurAjuM1tZRyk8l98ZNQfzRaxM0l4Rq7QlNLgX8TEcqLtEpN2s58Rbj4WFUgoixJ3GnRhfXmbJLHiV82c0Rr3FK/OgZbTGUP62yxK/OAVNE9fpvNJyWt15O7GEEhuoyVwGKAyLtucTJwQO50n4UOsLdCaq205WXhuy+M16mRFpcqlVb6YyWyg0s6pXuYXRrjQwB0xFuHHBqe4jvXRb+w+jrENri4LGhpQzwKJ3SgM8pkswqb9sMTW9wbtQiVDwnlll4S2Pl6CKsk69/pE5jF/GaZfZokJhQNQE4a6VXTjHJjz08bDmZnVDVKCIFGPlBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AXM/25J28r8GbglAIYnE6Ci3GySAzkooVp43ToLnHtY=; b=J34mGyIXTeSeaZeb//vh//wfvbIrdaTFKsNBv6cjtbIUsGFdSC37FlKQmriPF55snPK6LLj1+RQrzmceQi9b6/uS7Ae/jvszJ3kX5MpQg/IJKe8z49Pw313dnr9hajettl2G9PnvDsv0LiPA2i64YcizWqllt9/DKhlTsIqhSZR63FziCq4iXLi3CTgr6gDkAM7jF0uy76MFukCaCTBxuhJfs/kCj9/9FOcfVE4IKKjnKhSEbHReJ2vMC0MFSZ3haz/UF64PsQumPwF4H4dBrUiRR6ugEMDrIdfhKUXWeu5iYwpPjxLJpEF6OnQ0c13Pt6+WPU93IslXr21YYJjnwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cisco.com; dmarc=pass action=none header.from=cisco.com; dkim=pass header.d=cisco.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.onmicrosoft.com; s=selector2-cisco-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AXM/25J28r8GbglAIYnE6Ci3GySAzkooVp43ToLnHtY=; b=A74+pg/Fs0S6CrS/l+u6B+RY2AYWv9vwR3gZFrcUCgtN6UVrBeJQWoBUZMwqIOQVmkosNArKcYlL/12ulpyU6SpnUV/VT8Q1p/SWzV12IdL3eucW5mzJdp3MPcUOE+/fJS+sPoKzBCBGQlspSKTu7jsGFYdfwWGlQZsoHLeh8bw= Received: from SN6PR11MB3101.namprd11.prod.outlook.com (2603:10b6:805:d8::23) by SA0PR11MB4687.namprd11.prod.outlook.com (2603:10b6:806:96::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3109.22; Thu, 18 Jun 2020 10:26:25 +0000 Received: from SN6PR11MB3101.namprd11.prod.outlook.com ([fe80::ac58:2cf3:c611:9b0]) by SN6PR11MB3101.namprd11.prod.outlook.com ([fe80::ac58:2cf3:c611:9b0%3]) with mapi id 15.20.3109.021; Thu, 18 Jun 2020 10:26:25 +0000 From: "Benoit Ganne (bganne)" To: "users@dpdk.org" CC: "contact@filipjaniszewski.com" Thread-Topic: Bigger mempool leads to worst performances. Thread-Index: AdZFWaVRN7ABblh2SESM4QBbxqfPsw== Date: Thu, 18 Jun 2020 10:26:25 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=cisco.com; x-originating-ip: [92.154.90.183] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e2e64aea-e054-43c5-310a-08d813720e2c x-ms-traffictypediagnostic: SA0PR11MB4687: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0438F90F17 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: omSv6MVHNlBgbqAHdlptgJN09AGUR8CgfUC046WH0fLzv+FYaqD1D+F1JppudpjPigos1sCk6T6Z6nB/sNE582loeqBLH41MxVzzUeyKnbD3sKDdUP4E/0W8ZIZqwSrxYFApXpxiK7CHTcuEGNhgzURHDr6uJTqNdqRMPTBXL7NHzRFLEKO0l1tr53SDGabsm/x0AYQSgB63W2SNWyxBCt31WefI7Iwaj+gs5H2xcYeAh21klzFrPC0kbw5rGWivI2gPDhTzfWF77dlBnQpkkl/H3/iTnh0NuFjLBJIRvr3xOr3kCK3a4tMQo76KrsvHQP2rd3qiJncXVfFcEKo52z3kU0dTjx2ooBIPTw0wThNPdZAJts5GR+Dfk6lSbX/0 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3101.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(346002)(376002)(136003)(39860400002)(366004)(396003)(76116006)(86362001)(66476007)(66946007)(71200400001)(4326008)(33656002)(83380400001)(316002)(8936002)(7696005)(5660300002)(6506007)(4744005)(2906002)(52536014)(26005)(55016002)(9686003)(186003)(8676002)(66556008)(64756008)(478600001)(66446008)(6916009)(135533001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: qb6mSP+iC671/A3tPV9YIulZIr8/b7AZ1GKnipvqTQ1S+0112C6rSCzwRznkF5M5tTiVcmIK7x3IfDDhIBJZzY4tIFBaTTewBk8hknsMYC+kkZsippIZKU8x3N/5qh+WljeEmReMOFLDGKzdcp2OwyyPx+PO/DNLWtZ/Tq9kiqrckQuPdWdznLXC/uwDnE7bA8+P7yCRPdBXlnduC1B1RYMbv3QtCeSXk8JULYU0BPymoiX7A3jfxFYapa8AVdzufowyjM/rQay56ACgjB84LboTjDjpe0pLATDiJdcCkGfQnZl99zNIUgy59gfrwxZkWKFBEWBaa+Iph0LcPylPM24p2vuKy3dL7zCYofsPB1GHo/n7ed5M+Z48woWhvA4M0k6AIrWKtFpk2D6BQkGUOn6caNNx6TwhkZQNxA56MIJinDwWyvXYZqneBr6NV+utVGDoRgMLtV7Y206jVJ5ovkuEmFKRPBhmBbdnnAyjpBA= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: e2e64aea-e054-43c5-310a-08d813720e2c X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jun 2020 10:26:25.6532 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 5ae1af62-9505-4097-a69a-c1553ef7840e X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TVIynIZy53MuaxpoKVIkAcZQAQoZzXQgsiq+008rQh97RqTnchD0lYudgfItleAIt/pGbtEfiUkUXIgNc7+qTQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4687 X-OriginatorOrg: cisco.com X-Outbound-SMTP-Client: 173.36.7.15, xch-aln-005.cisco.com X-Outbound-Node: alln-core-1.cisco.com Subject: Re: [dpdk-users] Bigger mempool leads to worst performances. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" > It looks like that in our environment increasing the mempool reduce the > capture performance, any suggestion on what I might look at to > troubleshoot the problem? Apparently we can't go beyond 4GiB mempool > without performance penalties. > (Please note that 1GiB hugepage are configured to serve all the require > additional memory.) I'd bet it on page walks because of TLB misses: if I am not mistaken, on In= tel you only have 4 TLB entries for 1GB hugepages, and prior to Skylake you= have no victim cache. So, as soon as you use more than 4 x 1GB hugepages, = you'll start triggering page walks because all your pages do no longer fit = in the TLB. You should be able to check that with 'perf stat -e dTLB-loads,dTLB-load-mi= sses,dTLB-stores,dTLB-store-misses -a -I1000' or similar. Best ben