From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 959A5A0C4E; Tue, 2 Nov 2021 07:59:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 204E04069F; Tue, 2 Nov 2021 07:59:41 +0100 (CET) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id B29C14068F for ; Tue, 2 Nov 2021 07:59:39 +0100 (CET) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.173) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Mon, 1 Nov 2021 23:59:38 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UQA71bleg+/y/VaYDOF3ex9chJCc0ryscdJfkFNjCgMvhecNW8p6VuKhP2VXmtkyXeCy+186P1JWCu1R0HrO+Ep0bhiGTWB53PMexKoBR12IcWMwsZZz5BMhFub+cLUhk6X6JlryuozhmFVzHZAtPhXG0qSMp2I7SSASF7j8ZZ0HwboT4qUaTLX3OyVxF1Cu1AOdnQzGJklI8eeWDrLDKN4WKL29O3gFBhThx0d27oTKdWsTeUx/mPFoSVNmVzuAERFHkPzJXnn6Ul0AVs8ka2Y4JzLBkGH+WcCa1MMUiBluwM5mdiJIdNgTeonKzRyq2OgqQw7cY0FGXIA28W6aVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1zH29/xcb/+u0KsW10hcVoPQ0CQZ6WEx5N1gISyRvAo=; b=kJYkvkSyB4YfJBnqid/FbO28adhah1c+b3AfzINEY1gi9/k79ekSCJpK2i3vwsaxMkFcp2Mc0HrmZOCgRj+zx2bf1oTvI7cLSoO0Yvd+ch0B3hsQsbAgsZEZ+DhfbLw9tErUSqpukxCRhLXdrXpq6ukUbIzVynYBCrl/bXtA0LhkKsqJekU/t2K9k7Uh+CGxQDm4OCYh6vmr5RS85QtD5X5wUU+EnWQARxNM4gF07PcahXcaQsSZSB5AuyemLoy0FBTrEl2UtUSQfYfLgH80EobzcHxySngxlB3YQEXlgcvoh5rr7e00RcO89Q90GOOYEUPpYoLSjip2V9gcT98r5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1zH29/xcb/+u0KsW10hcVoPQ0CQZ6WEx5N1gISyRvAo=; b=esJlXf7mT2S1e94Kbk32dP/C2VXJUTn+X7wKjRkhGs+WU9w3W4wknnleBrs5XaV4iSoLWGSZGE5TM179GY7WFZ9xTBQnUaxMzqVz7fyEPvjaODoiJng4I58ti/Lntpxz5AWmwr/ycuB70OTcrH7bdey14bPg1nyY73zMnDlVDl6t0gOgb5VL3TRSmPGaeoJaX8i8n/0lybcLiwA7mvxbxXsQ8c0vdQo9KTl0Ol7Se0L7HALRBaw1ScMlq2soIOV4rx6y3ayAReBITc4sYyLdKytieQ7P7MGUPoGrWuE9wxuVYSrk2YHsKWNl644DbpZsHYz8JAnyssmQxa80/TB6IQ== Received: from DM5PR08CA0029.namprd08.prod.outlook.com (2603:10b6:4:60::18) by DM6PR12MB4154.namprd12.prod.outlook.com (2603:10b6:5:21d::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Tue, 2 Nov 2021 06:59:37 +0000 Received: from DM6NAM11FT037.eop-nam11.prod.protection.outlook.com (2603:10b6:4:60:cafe::aa) by DM5PR08CA0029.outlook.office365.com (2603:10b6:4:60::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.17 via Frontend Transport; Tue, 2 Nov 2021 06:59:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT037.mail.protection.outlook.com (10.13.172.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4649.14 via Frontend Transport; Tue, 2 Nov 2021 06:59:37 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 2 Nov 2021 06:59:35 +0000 From: Dmitry Kozlyuk To: CC: Raslan Darawsheh , Matan Azrad , Viacheslav Ovsiienko Date: Tue, 2 Nov 2021 08:59:17 +0200 Message-ID: <20211102065917.889267-1-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 44a37a54-b2ac-41c2-2b98-08d99dce5574 X-MS-TrafficTypeDiagnostic: DM6PR12MB4154: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:133; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QsQNt18Zb52xJLIVve+luaXpAsMULy7bRMCHyvZdB1DNpXn9JwryAlXZpbAxYsLOBuypzjbkr2cybwJL7x4LC8ZX5Bje1ATXJu08lseWsFbso8bSUkKWLZ8J6nk3jH22fQwjw+WmWvw43S+HboRF+IJEKUe3667mjZlgHOJciVvuRr66znAUAI40ktAMFtcLcO/3bL3UbjmRxDVD6MBDJmYa9WCSKiXcgML2XbcnHPrDDpaNhDc0fAIOU0z6fYGviLTQyb4tWWNgwDGkkDPwbXDhVqaZWnUbkvu+3HULh758JNVJq4BJvoAcIy85nETW35Rd1z3OrK5Tp6ScrqamQV4WWlizH9u/Urtq49WjIchIGWQ9aLXQYh/bsgAN0vT6toiaCNP4hofAyqcF+M7YaipdQTcBm1ozx3o2VYr7xLEfEkDiXOprxPVHhlgDKrjzKCRX0WCl9vsSfcWB3VDnuYP7dv2KtTejrKSGE/zgMX0csE5wKXbvmQGoXDkuAHvCs2fCiwXJlt5Sn9LcZJBk7tjStmAS8QVJ/v3vQ1laI/pH+TOwRhKIleTfcFROPmPRIDB3m+/URPaKFnZbwEwB/nVvDeydBZcn1CaSg1tx3HCfvVvuXfBcPcHEEaKznhCVNA3WxpFcuuuG5gR6kKWi8+/n3u5jxp+nn8+fekWGdl/t0aR4b2D6C9RMWpwoo0HFNssoPeazYce8Hl0Qkh73jA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(4326008)(2616005)(55016002)(7696005)(70206006)(1076003)(6286002)(26005)(6916009)(54906003)(8936002)(316002)(7636003)(186003)(16526019)(36906005)(356005)(47076005)(426003)(36860700001)(2906002)(36756003)(107886003)(508600001)(336012)(86362001)(8676002)(70586007)(83380400001)(82310400003)(5660300002)(6666004); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2021 06:59:37.0597 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 44a37a54-b2ac-41c2-2b98-08d99dce5574 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT037.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4154 Subject: [dpdk-dev] [PATCH] common/mlx5: fix external memory pool registration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Registration of packet mempools with RTE_PKTMBUF_POOL_PINNED_EXT_MEM was performed incorrectly: after population of such mempool chunks only contain memory for rte_mbuf structures, while pointers to actual external memory are not yet filled. MR LKeys could not be obtained for external memory addresses of such mempools. Rx datapath assumes all used mempools are registered and does not fallback to dynamic MR creation in such case, so no packets could be received. Skip registration of extmem pools on population because it is useless. If used for Rx, they are registered at port start. During registration, recognize such pools, inspect their mbufs and recover the pages they reside in. While MRs for these pages may already be created by rte_dev_dma_map(), they are not reused to avoid synchronization on Rx datapath in case these MRs are changed in the database. Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities") Signed-off-by: Dmitry Kozlyuk Reviewed-by: Matan Azrad Reviewed-by: Viacheslav Ovsiienko --- drivers/common/mlx5/mlx5_common.c | 4 + drivers/common/mlx5/mlx5_common_mr.c | 113 +++++++++++++++++++++++++-- drivers/net/mlx5/mlx5_trigger.c | 8 +- 3 files changed, 117 insertions(+), 8 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index e6ff045c95..97df19db4e 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -395,9 +395,13 @@ mlx5_dev_mempool_event_cb(enum rte_mempool_event event, struct rte_mempool *mp, void *arg) { struct mlx5_common_device *cdev = arg; + bool extmem = rte_pktmbuf_priv_flags(mp) & + RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF; switch (event) { case RTE_MEMPOOL_EVENT_READY: + if (extmem) + break; if (mlx5_dev_mempool_register(cdev, mp) < 0) DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s", diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c index 53a3e8565d..3d7a892c9b 100644 --- a/drivers/common/mlx5/mlx5_common_mr.c +++ b/drivers/common/mlx5/mlx5_common_mr.c @@ -1331,6 +1331,105 @@ mlx5_range_from_mempool_chunk(struct rte_mempool *mp, void *opaque, range->end = RTE_ALIGN_CEIL(range->start + memhdr->len, page_size); } +/** + * Collect page-aligned memory ranges of the mempool. + */ +static int +mlx5_mempool_get_chunks(struct rte_mempool *mp, struct mlx5_range **out, + unsigned int *out_n) +{ + struct mlx5_range *chunks; + unsigned int n; + + n = mp->nb_mem_chunks; + chunks = calloc(sizeof(chunks[0]), n); + if (chunks == NULL) + return -1; + rte_mempool_mem_iter(mp, mlx5_range_from_mempool_chunk, chunks); + *out = chunks; + *out_n = n; + return 0; +} + +struct mlx5_mempool_get_extmem_data { + struct mlx5_range *heap; + unsigned int heap_size; + int ret; +}; + +static void +mlx5_mempool_get_extmem_cb(struct rte_mempool *mp, void *opaque, + void *obj, unsigned int obj_idx) +{ + struct mlx5_mempool_get_extmem_data *data = opaque; + struct rte_mbuf *mbuf = obj; + uintptr_t addr = (uintptr_t)mbuf->buf_addr; + struct mlx5_range *seg, *heap; + struct rte_memseg_list *msl; + size_t page_size; + uintptr_t page_start; + unsigned int pos = 0, len = data->heap_size, delta; + + RTE_SET_USED(mp); + RTE_SET_USED(obj_idx); + if (data->ret < 0) + return; + /* Binary search for an already visited page. */ + while (len > 1) { + delta = len / 2; + if (addr < data->heap[pos + delta].start) { + len = delta; + } else { + pos += delta; + len -= delta; + } + } + if (data->heap != NULL) { + seg = &data->heap[pos]; + if (seg->start <= addr && addr < seg->end) + return; + } + /* Determine the page boundaries and remember them. */ + heap = realloc(data->heap, sizeof(heap[0]) * (data->heap_size + 1)); + if (heap == NULL) { + free(data->heap); + data->heap = NULL; + data->ret = -1; + return; + } + data->heap = heap; + data->heap_size++; + seg = &heap[data->heap_size - 1]; + msl = rte_mem_virt2memseg_list((void *)addr); + page_size = msl != NULL ? msl->page_sz : rte_mem_page_size(); + page_start = RTE_PTR_ALIGN_FLOOR(addr, page_size); + seg->start = page_start; + seg->end = page_start + page_size; + /* Maintain the heap order. */ + qsort(data->heap, data->heap_size, sizeof(heap[0]), + mlx5_range_compare_start); +} + +/** + * Recover pages of external memory as close as possible + * for a mempool with RTE_PKTMBUF_POOL_PINNED_EXT_BUF. + * Pages are stored in a heap for efficient search, for mbufs are many. + */ +static int +mlx5_mempool_get_extmem(struct rte_mempool *mp, struct mlx5_range **out, + unsigned int *out_n) +{ + struct mlx5_mempool_get_extmem_data data; + + memset(&data, 0, sizeof(data)); + rte_mempool_obj_iter(mp, mlx5_mempool_get_extmem_cb, &data); + if (data.ret < 0) + return -1; + *out = data.heap; + *out_n = data.heap_size; + return 0; +} + /** * Get VA-contiguous ranges of the mempool memory. * Each range start and end is aligned to the system page size. @@ -1350,13 +1449,15 @@ mlx5_get_mempool_ranges(struct rte_mempool *mp, struct mlx5_range **out, unsigned int *out_n) { struct mlx5_range *chunks; - unsigned int chunks_n = mp->nb_mem_chunks, contig_n, i; + unsigned int chunks_n, contig_n, i; + int ret; - /* Collect page-aligned memory ranges of the mempool. */ - chunks = calloc(sizeof(chunks[0]), chunks_n); - if (chunks == NULL) - return -1; - rte_mempool_mem_iter(mp, mlx5_range_from_mempool_chunk, chunks); + /* Collect the pool underlying memory. */ + ret = (rte_pktmbuf_priv_flags(mp) & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) ? + mlx5_mempool_get_extmem(mp, &chunks, &chunks_n) : + mlx5_mempool_get_chunks(mp, &chunks, &chunks_n); + if (ret < 0) + return ret; /* Merge adjacent chunks and place them at the beginning. */ qsort(chunks, chunks_n, sizeof(chunks[0]), mlx5_range_compare_start); contig_n = 1; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index d916c8addc..546b20bf86 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -146,14 +146,18 @@ mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl) return 0; } for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) { + uint32_t flags; + mp = rxq_ctrl->rxq.rxseg[s].mp; + flags = rte_pktmbuf_priv_flags(mp); ret = mlx5_mr_mempool_register(&priv->sh->cdev->mr_scache, priv->sh->cdev->pd, mp, &priv->mp_id); if (ret < 0 && rte_errno != EEXIST) return ret; - rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb, - &rxq_ctrl->rxq); + if ((flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) == 0) + rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb, + &rxq_ctrl->rxq); } return 0; } -- 2.25.1