From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9B1DA0C43; Mon, 18 Oct 2021 12:02:13 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 66C63410E4; Mon, 18 Oct 2021 12:02:11 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 46FB1410E1 for ; Mon, 18 Oct 2021 12:02:10 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.171) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Mon, 18 Oct 2021 03:02:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PvcTo22JKzGbDxc7eJK/AbHU8VyRNcDssz8PfyfocAPM13DBpoSrLXrIuo0ri4lVXwusNihlcIamGoGyGXh/GZmWaT4DTA9YbjoT0+vlDp0lD7i8sKY/rr5UtGnJp1nNtxJkVbMweQHWlosFR623U913zOIB5JnF1sVt5OE3hsZxE4tPxcNF6fvEpzjB53A0yVUJC2pk4J6o/s4HXAaucgdFgvKLa0whp2CFVNFhyOcSMbJuJih7K+Sh4RTrdKw57xJnReQkc3cV8XI5vHY3Mb01cc4raWSQO4ln5eM5aieWf5yxYRnH6C6X4bjIROnbuGwSgu8tCIP4ZqZBdfyEQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FlMQuTF3AYjYZPYMVM/BUfjL6dLmJ+MUVYnsfsZHrOA=; b=SSykoqC3Tt19SlMX/nSd/k+wY7KvyF7BzaUag8ElF+GwyWiUAtkcgGK+5lj+83K817rJx++W4yPqydUIedrG0MSZ6TyAv3zDFsWA2kG2cKCRxc+2XDxd/6YBK5RxIDEariiARFvFb2pOPM+Y/nTS4LLJERd9/nWjQPyCOEkqaQ+Jle1k1EORtft7W0X+RWwDyBN4ZH4/J3fOtbJOX1wE1tzkigb5RWqJhUDxOGQk9gZhRZkoX6207hLTMaZlsHTLolSATJ2uDgLk8KBcH7BPT8uYOu1d5ds0ytClFgqDqR66tbFjTTjWuMf0sLxHMrN5nzXwnwta578YsB1DQzKswQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=6wind.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FlMQuTF3AYjYZPYMVM/BUfjL6dLmJ+MUVYnsfsZHrOA=; b=EeJOYjOJcMFi9YRi9SweK+K5hhfoutc9w13wU7MojRwj3t0x700gINYkKVTQ2ZvdCdvczRL0OdLMnIk8ITLMDDftVPtAv8zfzPVBt6Cyrf7uD5zhGXwe9FdVzaCV+85Vi+j0iqFkP5DgcbHAy3h835GJZpn9w5jR5I/PIzeBP2dZT7wUh9cgI0p1ongny4dbmJ3bAps24802pHS3XgKMeMnska/FdS7lNKmp+tpSVWtpcpgBbs9Tvo4h2NRdM3smQQbmwBzaJlOi9lWSKHlzdnMyrcqJP1arwg+EXIEqbB0qZnQSkeyF/MxYhCg92dfKnCHfKzrsjKCJShVenFnvzQ== Received: from MW3PR06CA0024.namprd06.prod.outlook.com (2603:10b6:303:2a::29) by CH2PR12MB5003.namprd12.prod.outlook.com (2603:10b6:610:68::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17; Mon, 18 Oct 2021 10:02:07 +0000 Received: from CO1NAM11FT043.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2a:cafe::29) by MW3PR06CA0024.outlook.office365.com (2603:10b6:303:2a::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17 via Frontend Transport; Mon, 18 Oct 2021 10:02:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; 6wind.com; dkim=none (message not signed) header.d=none;6wind.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by CO1NAM11FT043.mail.protection.outlook.com (10.13.174.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4608.15 via Frontend Transport; Mon, 18 Oct 2021 10:02:07 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Mon, 18 Oct 2021 10:02:03 +0000 From: Dmitry Kozlyuk To: CC: Matan Azrad , Andrew Rybchenko , Olivier Matz , "Ray Kinsella" Date: Mon, 18 Oct 2021 13:01:38 +0300 Message-ID: <20211018100141.3290956-2-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20211018100141.3290956-1-dkozlyuk@nvidia.com> References: <20211016200046.3176642-1-dkozlyuk@nvidia.com> <20211018100141.3290956-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8b260c56-778a-49a6-471a-08d9921e5802 X-MS-TrafficTypeDiagnostic: CH2PR12MB5003: X-Microsoft-Antispam-PRVS: X-MS-Exchange-Transport-Forked: True X-MS-Oob-TLC-OOBClassifiers: OLM:2201; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: irTrvh1WQapS+Vz5M6u4bM0Gwgyp11VyPeTdCmVK/n/f7xLpcxvIFe7NLbUHjjvsJw2ZaDr76eodw3cauBr1DdQK8a3ALVvuXMJCk7x2u1s4HXA6KCLpqTMGwtH5A6ciPlnvO0pLEs9obD53X0M1ukrdolr2awH/cAfyIC9aEY9fT0aeAW9S5GaHPtWdp0cGOCcu3qjcLltAROfwnESOKwc5M3p0UxfgK9XORC6U6DFEYqOkd5DFX3nPo/25VqRG1xGR43M/bGjVFzEcYsWa+OBeZoFi3rTHUdWYyg1WsuSHi3jexUzhB7n+0mGopO+b66Drresh4aca80CvsUaJkCkjsAuT2apQJg3X6lDdnCGxOA36ggB39oMx75ZEntIfTUnXdVKJZaQyAO+nRupf3+Dv94nhqkfbYZejUqreA3cymhCpAlZfGQMN5P3oEJO7JxzvhTza4WqHEa6rG4EShhzz67SgkMv19P0gqcCKIv52oxAK+7haiiBkod19kTYZw0M6Kq1ih4Pao68ub05ncKDG4t+rreQjFn0favtbAknD/XzcwbsOorLPWc8wuIDp5SwAmuzmhZskhKeOFQcSZ8o3g6R2D9lTkl4+o2/kJqgcuYMVZ5Amjc5NVfCBJbBVRHn/VyjtFCvLOdMjarb2haB4vXZBmVTYstPySBCxnrPHO7qB3BLd3fskprxLvI9QaDN/XmwwWt5lVNmZv7dQng== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(6666004)(82310400003)(2906002)(6286002)(6916009)(36906005)(5660300002)(316002)(36860700001)(54906003)(30864003)(7636003)(107886003)(186003)(16526019)(508600001)(4326008)(83380400001)(86362001)(426003)(7696005)(8936002)(70206006)(8676002)(336012)(2616005)(70586007)(356005)(1076003)(47076005)(26005)(55016002)(36756003); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2021 10:02:07.0585 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8b260c56-778a-49a6-471a-08d9921e5802 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT043.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB5003 Subject: [dpdk-dev] [PATCH v7 1/4] mempool: add event callbacks X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Data path performance can benefit if the PMD knows which memory it will need to handle in advance, before the first mbuf is sent to the PMD. It is impractical, however, to consider all allocated memory for this purpose. Most often mbuf memory comes from mempools that can come and go. PMD can enumerate existing mempools on device start, but it also needs to track creation and destruction of mempools after the forwarding starts but before an mbuf from the new mempool is sent to the device. Add an API to register callback for mempool life cycle events: * rte_mempool_event_callback_register() * rte_mempool_event_callback_unregister() Currently tracked events are: * RTE_MEMPOOL_EVENT_READY (after populating a mempool) * RTE_MEMPOOL_EVENT_DESTROY (before freeing a mempool) Provide a unit test for the new API. The new API is internal, because it is primarily demanded by PMDs that may need to deal with any mempools and do not control their creation, while an application, on the other hand, knows which mempools it creates and doesn't care about internal mempools PMDs might create. Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad Reviewed-by: Andrew Rybchenko --- app/test/test_mempool.c | 248 ++++++++++++++++++++++++++++++++++++++ lib/mempool/rte_mempool.c | 124 +++++++++++++++++++ lib/mempool/rte_mempool.h | 62 ++++++++++ lib/mempool/version.map | 8 ++ 4 files changed, 442 insertions(+) diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c index 66bc8d86b7..c39c83256e 100644 --- a/app/test/test_mempool.c +++ b/app/test/test_mempool.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -489,6 +490,245 @@ test_mp_mem_init(struct rte_mempool *mp, data->ret = 0; } +struct test_mempool_events_data { + struct rte_mempool *mp; + enum rte_mempool_event event; + bool invoked; +}; + +static void +test_mempool_events_cb(enum rte_mempool_event event, + struct rte_mempool *mp, void *user_data) +{ + struct test_mempool_events_data *data = user_data; + + data->mp = mp; + data->event = event; + data->invoked = true; +} + +static int +test_mempool_events(int (*populate)(struct rte_mempool *mp)) +{ +#pragma push_macro("RTE_TEST_TRACE_FAILURE") +#undef RTE_TEST_TRACE_FAILURE +#define RTE_TEST_TRACE_FAILURE(...) do { goto fail; } while (0) + + static const size_t CB_NUM = 3; + static const size_t MP_NUM = 2; + + struct test_mempool_events_data data[CB_NUM]; + struct rte_mempool *mp[MP_NUM], *freed; + char name[RTE_MEMPOOL_NAMESIZE]; + size_t i, j; + int ret; + + memset(mp, 0, sizeof(mp)); + for (i = 0; i < CB_NUM; i++) { + ret = rte_mempool_event_callback_register + (test_mempool_events_cb, &data[i]); + RTE_TEST_ASSERT_EQUAL(ret, 0, "Failed to register the callback %zu: %s", + i, rte_strerror(rte_errno)); + } + ret = rte_mempool_event_callback_unregister(test_mempool_events_cb, mp); + RTE_TEST_ASSERT_NOT_EQUAL(ret, 0, "Unregistered a non-registered callback"); + /* NULL argument has no special meaning in this API. */ + ret = rte_mempool_event_callback_unregister(test_mempool_events_cb, + NULL); + RTE_TEST_ASSERT_NOT_EQUAL(ret, 0, "Unregistered a non-registered callback with NULL argument"); + + /* Create mempool 0 that will be observed by all callbacks. */ + memset(&data, 0, sizeof(data)); + strcpy(name, "empty0"); + mp[0] = rte_mempool_create_empty(name, MEMPOOL_SIZE, + MEMPOOL_ELT_SIZE, 0, 0, + SOCKET_ID_ANY, 0); + RTE_TEST_ASSERT_NOT_NULL(mp[0], "Cannot create mempool %s: %s", + name, rte_strerror(rte_errno)); + for (j = 0; j < CB_NUM; j++) + RTE_TEST_ASSERT_EQUAL(data[j].invoked, false, + "Callback %zu invoked on %s mempool creation", + j, name); + + rte_mempool_set_ops_byname(mp[0], rte_mbuf_best_mempool_ops(), NULL); + ret = populate(mp[0]); + RTE_TEST_ASSERT_EQUAL(ret, (int)mp[0]->size, "Failed to populate mempool %s: %s", + name, rte_strerror(rte_errno)); + for (j = 0; j < CB_NUM; j++) { + RTE_TEST_ASSERT_EQUAL(data[j].invoked, true, + "Callback %zu not invoked on mempool %s population", + j, name); + RTE_TEST_ASSERT_EQUAL(data[j].event, + RTE_MEMPOOL_EVENT_READY, + "Wrong callback invoked, expected READY"); + RTE_TEST_ASSERT_EQUAL(data[j].mp, mp[0], + "Callback %zu invoked for a wrong mempool instead of %s", + j, name); + } + + /* Check that unregistered callback 0 observes no events. */ + ret = rte_mempool_event_callback_unregister(test_mempool_events_cb, + &data[0]); + RTE_TEST_ASSERT_EQUAL(ret, 0, "Failed to unregister callback 0: %s", + rte_strerror(rte_errno)); + memset(&data, 0, sizeof(data)); + strcpy(name, "empty1"); + mp[1] = rte_mempool_create_empty(name, MEMPOOL_SIZE, + MEMPOOL_ELT_SIZE, 0, 0, + SOCKET_ID_ANY, 0); + RTE_TEST_ASSERT_NOT_NULL(mp[1], "Cannot create mempool %s: %s", + name, rte_strerror(rte_errno)); + rte_mempool_set_ops_byname(mp[1], rte_mbuf_best_mempool_ops(), NULL); + ret = populate(mp[1]); + RTE_TEST_ASSERT_EQUAL(ret, (int)mp[1]->size, "Failed to populate mempool %s: %s", + name, rte_strerror(rte_errno)); + RTE_TEST_ASSERT_EQUAL(data[0].invoked, false, + "Unregistered callback 0 invoked on %s mempool populaton", + name); + + for (i = 0; i < MP_NUM; i++) { + memset(&data, 0, sizeof(data)); + sprintf(name, "empty%zu", i); + rte_mempool_free(mp[i]); + /* + * Save pointer to check that it was passed to the callback, + * but put NULL into the array in case cleanup is called early. + */ + freed = mp[i]; + mp[i] = NULL; + for (j = 1; j < CB_NUM; j++) { + RTE_TEST_ASSERT_EQUAL(data[j].invoked, true, + "Callback %zu not invoked on mempool %s destruction", + j, name); + RTE_TEST_ASSERT_EQUAL(data[j].event, + RTE_MEMPOOL_EVENT_DESTROY, + "Wrong callback invoked, expected DESTROY"); + RTE_TEST_ASSERT_EQUAL(data[j].mp, freed, + "Callback %zu invoked for a wrong mempool instead of %s", + j, name); + } + RTE_TEST_ASSERT_EQUAL(data[0].invoked, false, + "Unregistered callback 0 invoked on %s mempool destruction", + name); + } + + for (j = 1; j < CB_NUM; j++) { + ret = rte_mempool_event_callback_unregister + (test_mempool_events_cb, &data[j]); + RTE_TEST_ASSERT_EQUAL(ret, 0, "Failed to unregister the callback %zu: %s", + j, rte_strerror(rte_errno)); + } + return TEST_SUCCESS; + +fail: + for (j = 0; j < CB_NUM; j++) + rte_mempool_event_callback_unregister + (test_mempool_events_cb, &data[j]); + for (i = 0; i < MP_NUM; i++) + rte_mempool_free(mp[i]); + return TEST_FAILED; + +#pragma pop_macro("RTE_TEST_TRACE_FAILURE") +} + +struct test_mempool_events_safety_data { + bool invoked; + int (*api_func)(rte_mempool_event_callback *func, void *user_data); + rte_mempool_event_callback *cb_func; + void *cb_user_data; + int ret; +}; + +static void +test_mempool_events_safety_cb(enum rte_mempool_event event, + struct rte_mempool *mp, void *user_data) +{ + struct test_mempool_events_safety_data *data = user_data; + + RTE_SET_USED(event); + RTE_SET_USED(mp); + data->invoked = true; + data->ret = data->api_func(data->cb_func, data->cb_user_data); +} + +static int +test_mempool_events_safety(void) +{ +#pragma push_macro("RTE_TEST_TRACE_FAILURE") +#undef RTE_TEST_TRACE_FAILURE +#define RTE_TEST_TRACE_FAILURE(...) do { \ + ret = TEST_FAILED; \ + goto exit; \ + } while (0) + + struct test_mempool_events_data data; + struct test_mempool_events_safety_data sdata[2]; + struct rte_mempool *mp; + size_t i; + int ret; + + /* removes itself */ + sdata[0].api_func = rte_mempool_event_callback_unregister; + sdata[0].cb_func = test_mempool_events_safety_cb; + sdata[0].cb_user_data = &sdata[0]; + sdata[0].ret = -1; + rte_mempool_event_callback_register(test_mempool_events_safety_cb, + &sdata[0]); + /* inserts a callback after itself */ + sdata[1].api_func = rte_mempool_event_callback_register; + sdata[1].cb_func = test_mempool_events_cb; + sdata[1].cb_user_data = &data; + sdata[1].ret = -1; + rte_mempool_event_callback_register(test_mempool_events_safety_cb, + &sdata[1]); + + mp = rte_mempool_create_empty("empty", MEMPOOL_SIZE, + MEMPOOL_ELT_SIZE, 0, 0, + SOCKET_ID_ANY, 0); + RTE_TEST_ASSERT_NOT_NULL(mp, "Cannot create mempool: %s", + rte_strerror(rte_errno)); + memset(&data, 0, sizeof(data)); + ret = rte_mempool_populate_default(mp); + RTE_TEST_ASSERT_EQUAL(ret, (int)mp->size, "Failed to populate mempool: %s", + rte_strerror(rte_errno)); + + RTE_TEST_ASSERT_EQUAL(sdata[0].ret, 0, "Callback failed to unregister itself: %s", + rte_strerror(rte_errno)); + RTE_TEST_ASSERT_EQUAL(sdata[1].ret, 0, "Failed to insert a new callback: %s", + rte_strerror(rte_errno)); + RTE_TEST_ASSERT_EQUAL(data.invoked, false, + "Inserted callback is invoked on mempool population"); + + memset(&data, 0, sizeof(data)); + sdata[0].invoked = false; + rte_mempool_free(mp); + mp = NULL; + RTE_TEST_ASSERT_EQUAL(sdata[0].invoked, false, + "Callback that unregistered itself was called"); + RTE_TEST_ASSERT_EQUAL(sdata[1].ret, -EEXIST, + "New callback inserted twice"); + RTE_TEST_ASSERT_EQUAL(data.invoked, true, + "Inserted callback is not invoked on mempool destruction"); + + rte_mempool_event_callback_unregister(test_mempool_events_cb, &data); + for (i = 0; i < RTE_DIM(sdata); i++) + rte_mempool_event_callback_unregister + (test_mempool_events_safety_cb, &sdata[i]); + ret = TEST_SUCCESS; + +exit: + /* cleanup, don't care which callbacks are already removed */ + rte_mempool_event_callback_unregister(test_mempool_events_cb, &data); + for (i = 0; i < RTE_DIM(sdata); i++) + rte_mempool_event_callback_unregister + (test_mempool_events_safety_cb, &sdata[i]); + /* in case of failure before the planned destruction */ + rte_mempool_free(mp); + return ret; + +#pragma pop_macro("RTE_TEST_TRACE_FAILURE") +} + static int test_mempool(void) { @@ -666,6 +906,14 @@ test_mempool(void) if (test_mempool_basic(default_pool, 1) < 0) GOTO_ERR(ret, err); + /* test mempool event callbacks */ + if (test_mempool_events(rte_mempool_populate_default) < 0) + GOTO_ERR(ret, err); + if (test_mempool_events(rte_mempool_populate_anon) < 0) + GOTO_ERR(ret, err); + if (test_mempool_events_safety() < 0) + GOTO_ERR(ret, err); + rte_mempool_list_dump(stdout); ret = 0; diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c index 607419ccaf..8810d08ab5 100644 --- a/lib/mempool/rte_mempool.c +++ b/lib/mempool/rte_mempool.c @@ -42,6 +42,18 @@ static struct rte_tailq_elem rte_mempool_tailq = { }; EAL_REGISTER_TAILQ(rte_mempool_tailq) +TAILQ_HEAD(mempool_callback_list, rte_tailq_entry); + +static struct rte_tailq_elem callback_tailq = { + .name = "RTE_MEMPOOL_CALLBACK", +}; +EAL_REGISTER_TAILQ(callback_tailq) + +/* Invoke all registered mempool event callbacks. */ +static void +mempool_event_callback_invoke(enum rte_mempool_event event, + struct rte_mempool *mp); + #define CACHE_FLUSHTHRESH_MULTIPLIER 1.5 #define CALC_CACHE_FLUSHTHRESH(c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) @@ -360,6 +372,10 @@ rte_mempool_populate_iova(struct rte_mempool *mp, char *vaddr, STAILQ_INSERT_TAIL(&mp->mem_list, memhdr, next); mp->nb_mem_chunks++; + /* Report the mempool as ready only when fully populated. */ + if (mp->populated_size >= mp->size) + mempool_event_callback_invoke(RTE_MEMPOOL_EVENT_READY, mp); + rte_mempool_trace_populate_iova(mp, vaddr, iova, len, free_cb, opaque); return i; @@ -722,6 +738,7 @@ rte_mempool_free(struct rte_mempool *mp) } rte_mcfg_tailq_write_unlock(); + mempool_event_callback_invoke(RTE_MEMPOOL_EVENT_DESTROY, mp); rte_mempool_trace_free(mp); rte_mempool_free_memchunks(mp); rte_mempool_ops_free(mp); @@ -1356,3 +1373,110 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *), rte_mcfg_mempool_read_unlock(); } + +struct mempool_callback_data { + rte_mempool_event_callback *func; + void *user_data; +}; + +static void +mempool_event_callback_invoke(enum rte_mempool_event event, + struct rte_mempool *mp) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te; + void *tmp_te; + + rte_mcfg_tailq_read_lock(); + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + RTE_TAILQ_FOREACH_SAFE(te, list, next, tmp_te) { + struct mempool_callback_data *cb = te->data; + rte_mcfg_tailq_read_unlock(); + cb->func(event, mp, cb->user_data); + rte_mcfg_tailq_read_lock(); + } + rte_mcfg_tailq_read_unlock(); +} + +int +rte_mempool_event_callback_register(rte_mempool_event_callback *func, + void *user_data) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te = NULL; + struct mempool_callback_data *cb; + void *tmp_te; + int ret; + + if (func == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + rte_mcfg_tailq_write_lock(); + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + RTE_TAILQ_FOREACH_SAFE(te, list, next, tmp_te) { + cb = te->data; + if (cb->func == func && cb->user_data == user_data) { + ret = -EEXIST; + goto exit; + } + } + + te = rte_zmalloc("mempool_cb_tail_entry", sizeof(*te), 0); + if (te == NULL) { + RTE_LOG(ERR, MEMPOOL, + "Cannot allocate event callback tailq entry!\n"); + ret = -ENOMEM; + goto exit; + } + + cb = rte_malloc("mempool_cb_data", sizeof(*cb), 0); + if (cb == NULL) { + RTE_LOG(ERR, MEMPOOL, + "Cannot allocate event callback!\n"); + rte_free(te); + ret = -ENOMEM; + goto exit; + } + + cb->func = func; + cb->user_data = user_data; + te->data = cb; + TAILQ_INSERT_TAIL(list, te, next); + ret = 0; + +exit: + rte_mcfg_tailq_write_unlock(); + rte_errno = -ret; + return ret; +} + +int +rte_mempool_event_callback_unregister(rte_mempool_event_callback *func, + void *user_data) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te = NULL; + struct mempool_callback_data *cb; + int ret = -ENOENT; + + rte_mcfg_tailq_write_lock(); + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + TAILQ_FOREACH(te, list, next) { + cb = te->data; + if (cb->func == func && cb->user_data == user_data) { + TAILQ_REMOVE(list, te, next); + ret = 0; + break; + } + } + rte_mcfg_tailq_write_unlock(); + + if (ret == 0) { + rte_free(te); + rte_free(cb); + } + rte_errno = -ret; + return ret; +} diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index 88bcbc51ef..5799d4a705 100644 --- a/lib/mempool/rte_mempool.h +++ b/lib/mempool/rte_mempool.h @@ -1769,6 +1769,68 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg), int rte_mempool_get_page_size(struct rte_mempool *mp, size_t *pg_sz); +/** + * Mempool event type. + * @internal + */ +enum rte_mempool_event { + /** Occurs after a mempool is fully populated. */ + RTE_MEMPOOL_EVENT_READY = 0, + /** Occurs before the destruction of a mempool begins. */ + RTE_MEMPOOL_EVENT_DESTROY = 1, +}; + +/** + * @internal + * Mempool event callback. + * + * rte_mempool_event_callback_register() may be called from within the callback, + * but the callbacks registered this way will not be invoked for the same event. + * rte_mempool_event_callback_unregister() may only be safely called + * to remove the running callback. + */ +typedef void (rte_mempool_event_callback)( + enum rte_mempool_event event, + struct rte_mempool *mp, + void *user_data); + +/** + * @internal + * Register a callback function invoked on mempool life cycle event. + * The function will be invoked in the process + * that performs an action which triggers the callback. + * + * @param func + * Callback function. + * @param user_data + * User data. + * + * @return + * 0 on success, negative on failure and rte_errno is set. + */ +__rte_internal +int +rte_mempool_event_callback_register(rte_mempool_event_callback *func, + void *user_data); + +/** + * @internal + * Unregister a callback added with rte_mempool_event_callback_register(). + * @p func and @p user_data must exactly match registration parameters. + * + * @param func + * Callback function. + * @param user_data + * User data. + * + * @return + * 0 on success, negative on failure and rte_errno is set. + */ +__rte_internal +int +rte_mempool_event_callback_unregister(rte_mempool_event_callback *func, + void *user_data); + #ifdef __cplusplus } #endif diff --git a/lib/mempool/version.map b/lib/mempool/version.map index 9f77da6fff..1b7d7c5456 100644 --- a/lib/mempool/version.map +++ b/lib/mempool/version.map @@ -64,3 +64,11 @@ EXPERIMENTAL { __rte_mempool_trace_ops_free; __rte_mempool_trace_set_ops_byname; }; + +INTERNAL { + global: + + # added in 21.11 + rte_mempool_event_callback_register; + rte_mempool_event_callback_unregister; +}; -- 2.25.1