From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F1A8EA0353 for ; Fri, 17 Dec 2021 10:58:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DF63040F35; Fri, 17 Dec 2021 10:58:41 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2060.outbound.protection.outlook.com [40.107.243.60]) by mails.dpdk.org (Postfix) with ESMTP id CA27640040; Fri, 17 Dec 2021 10:58:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V7OPJ+BquDHdihwl+dHcyzMl8HLSmJ3wTEklyuwBmKwdOeIjly3S4qzUM6Kthd161f3X2OKuzcQ0S52AAEC80A/Q/efRcBVeJvWYyyC/cqvOT8VEDAEzecXo/RtPL6DLW1YVl8OeIxNo/LhD+k7YXmEIWGCYe0X/2wu2bLt31ieqGIGxjiljZunTuMYU8Egs9wmR+BdgZ/G6oXjZa3uzJ65MbuWHdLMgpN+GoMnxyeOFCI9qaue5ahEQLyslI/J+WJfiMaSDKXs0f8gyMPf3U4MNYyv66tDhf8wB6OuXC9PxHqGsN0x7BuWm4L52YRrUSDebt612oQlBmS/L6JIX+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TTY9gVbGxjj/x2VIx6QsCBnIKxzVH/kG8CCndbmZS0w=; b=KFkXgJ+KOkO7Xl7DIRh95Ig2gdQsAaAJJHAINXEHGPnAdAB+cIbnQyl4tGiZHcs2r8UU4b2elR927R4NnlabdMuzKMF0EntZd6PYFJF03T4mkmUAINmnRwBQMK+jaVi0hGdJBQBpZyDu5D84oQfCOfmNc+c064UAdMk8uE79599rapHTQAtsIcFGEzj4bp8y40139mI1s+K87nwoZ2NUsIlWr7YjYvuRxJF7iXncALiOUuEfFRUT13k8AgNba3rIt5igrxEgTK3zWvCiCnvZ2YP8DXyQ/NYBz+WLnD8KvL05quhbI56xWPmklt3cqqWvSt5E8sc1Jhurzyky6NCbYw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TTY9gVbGxjj/x2VIx6QsCBnIKxzVH/kG8CCndbmZS0w=; b=OZYcG3W1olzoaubHbsiW+b0uRTcxOSeJtFa2qGSciojHneUhBWmYtOC9T61xqu/OEHYHF0nm39W9ALf7hS4i7SWEHKCK+OBu87d1L1kGN1sq3mHKeJ87vontCMGEbRD4unjwpIK6APgCPj7LXWGveCPcPBkkiYxylQIdeyzowRdLN1h8kBbIMaBXvKC0foBib0taPV9TTJmgwJ2PgmjYswVxfebJWG4UZGyRzupe+NTGpwYIcgYNvrqo3e0m8yQOStcYmFdQLvLnGUvuA3HAML3PZZ1uoxNEBtba4mVMeZjh8MTLHISx63uEQqEWyhaV89ead8wfOWu++kqO0dmrfQ== Received: from MWHPR2201CA0038.namprd22.prod.outlook.com (2603:10b6:301:16::12) by DM5PR12MB1450.namprd12.prod.outlook.com (2603:10b6:4:3::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4801.14; Fri, 17 Dec 2021 09:58:36 +0000 Received: from CO1NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:301:16:cafe::62) by MWHPR2201CA0038.outlook.office365.com (2603:10b6:301:16::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4801.15 via Frontend Transport; Fri, 17 Dec 2021 09:58:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by CO1NAM11FT042.mail.protection.outlook.com (10.13.174.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4801.14 via Frontend Transport; Fri, 17 Dec 2021 09:58:34 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 17 Dec 2021 09:58:33 +0000 Received: from nvidia.com (172.20.187.6) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Fri, 17 Dec 2021 01:58:31 -0800 From: Dmitry Kozlyuk To: CC: , Viacheslav Ovsiienko , "Xiaoyun Li" , Aman Singh Subject: [PATCH] app/testpmd: fix external buffer allocation Date: Fri, 17 Dec 2021 11:58:16 +0200 Message-ID: <20211217095816.2599242-1-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c97e7b17-4410-4f3b-a6fd-08d9c143c9c9 X-MS-TrafficTypeDiagnostic: DM5PR12MB1450:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cqWYfJnsmsn5susiTxO8y6cgRzxoRkCXZctswmD1TnJpJz4nx2ONnxDgP5347s3VeHS8bpO2Iu1PET0O9DJV5kdV6mZOa+MVaLPnLZrfX+RKmQh1+xX1/1HHNnZ3WjFwM96ZjQqxkyBMwLgIjUxt9sJVA6mS1MljsEITZCSEU8CsaiYmQbq5YhPiaiZkZMu/mab9b7hmlJAFQXPCI3zAofYhhcoFfQXYyHqsqhEDOBLHdD9cSTtfnIhfq0PgfiasL3VZJp7Ea1SZRi7l27nwp/0s8cduX6QRxVHM2LFQZDkrNNXcW1+rQX1fZEMdrl22OuyJWUHLxge6ksz/T64Dc0f9DL1z4of01psGcaLevcblyxi4eleWTbKLJ242JlJitk4sVXv5fff7kvdFTGxbIYevJxvBXpIE5hWcZyU1AwG8osHqqL39SVG+fARNtu8yvhNz/QeohkY1wRuO1QPgee2MNZKejsdT5yrcxMxYiJ8fhYqby+1gtTHlKvTZijlJ6JhODHoQTsXF00TdWub7DgTtEc/1WSw7JnbSa2Fc+X2FaOhtAUq022me7sWqKEU/ZKfa40jc78RqDk9gy+wqe9zUqklHdRpzTSTGjTPQ25njQz1wauO4tHHvN7e8k0w5pOpPIgL0JEYNTYjPyalJQ5qV6bj8uKX7B0IoB/RJ0QJKfDOCcxYl8n1+cHIzrDq+UygV9OxMF9Oj77t1VD6y7GYQNlvWJz1ZJIiMHkafHrK1tuwxpbPXhxdSxA6AnLmOeILIOcpxlveHy04g7q9hJfDPfeggdSx1PjQy7GvZEN6QXdC2pni54cy1cbktT1BjwIEL+lwoqX8mVbbA93OKDg== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(40470700001)(6286002)(5660300002)(47076005)(6666004)(34020700004)(4326008)(36860700001)(55016003)(82310400004)(8676002)(83380400001)(36756003)(356005)(1076003)(81166007)(8936002)(40460700001)(86362001)(316002)(6916009)(2906002)(70206006)(70586007)(2616005)(336012)(26005)(426003)(508600001)(54906003)(16526019)(186003)(7696005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 09:58:34.0166 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c97e7b17-4410-4f3b-a6fd-08d9c143c9c9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1450 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org External pinned buffer memory (--mp-alloc=xbuf) was allocated as multiple IOVA-contiguous memzones of 2M size and 2M alignment. Due to the malloc overhead and the alignment requirement, each 2M memzone consumed 4M of hugepage memory: 2M of usable memory + X of malloc overhead + (2M-X) padding. The allocation often failed with 2M hugepages and IOVA-as-PA if a PA-contiguous span of 2 hugepages could not be found. Also, with any hugepage size and IOVA mode memory consumption was almost 2x of the usable amount. Alignment requirement of 2M for external buffers is redundant. It was an attempt to ensure IOVA-contiguity by forcing memzones to start at hugepage boundaries, while 2M size intended to leave no unused space on the page. As shown above, this in fact caused excessive memory consumption and decreased the chance of a successful allocation. RTE_MEMZONE_F_IOVA_CONTIG already ensures IOVA-contiguity. Remove the alignment requirement. Reduce the memzone size by the malloc overhead size (4 cache lines), so that memory consumption for each memzone is (2M-X) of usable memory + X of malloc overhead = 2M. This also means that whenever there are free 2M hugepages, an IOVA-contiguous memzone can always be allocated. Fixes: 72512e1897b2 ("app/testpmd: add mempool with external data buffers") Cc: stable@dpdk.org Signed-off-by: Dmitry Kozlyuk Signed-off-by: Viacheslav Ovsiienko --- app/test-pmd/testpmd.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 55eb293cc0..fa04cc6be6 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -84,7 +84,13 @@ #endif #define EXTMEM_HEAP_NAME "extmem" -#define EXTBUF_ZONE_SIZE RTE_PGSIZE_2M +/* + * Zone size with the malloc overhead (max of debug and release variants) + * must fit into the smallest supported hugepage size (2M), + * so that an IOVA-contiguous zone of this size can always be allocated + * if there are free 2M hugepages. + */ +#define EXTBUF_ZONE_SIZE (RTE_PGSIZE_2M - 4 * RTE_CACHE_LINE_SIZE) uint16_t verbose_level = 0; /**< Silent by default. */ int testpmd_logtype; /**< Log type for testpmd logs */ @@ -1061,12 +1067,11 @@ setup_extbuf(uint32_t nb_mbufs, uint16_t mbuf_sz, unsigned int socket_id, ext_num = 0; break; } - mz = rte_memzone_reserve_aligned(mz_name, EXTBUF_ZONE_SIZE, - socket_id, - RTE_MEMZONE_IOVA_CONTIG | - RTE_MEMZONE_1GB | - RTE_MEMZONE_SIZE_HINT_ONLY, - EXTBUF_ZONE_SIZE); + mz = rte_memzone_reserve(mz_name, EXTBUF_ZONE_SIZE, + socket_id, + RTE_MEMZONE_IOVA_CONTIG | + RTE_MEMZONE_1GB | + RTE_MEMZONE_SIZE_HINT_ONLY); if (mz == NULL) { /* * The caller exits on external buffer creation -- 2.25.1