From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id ACFE9A04BC; Fri, 9 Oct 2020 18:38:27 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 90F941D15D; Fri, 9 Oct 2020 18:38:26 +0200 (CEST) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id 0D3DC1C2F9 for ; Fri, 9 Oct 2020 18:38:23 +0200 (CEST) IronPort-SDR: Ub+Em/ZTJx82Qyr/gRjTK4TfvUPxg/7uqAZ/MnWPLeSiBTpLsrnUWSF4ErxuS18ltd71PDONFV UV/sY+MVjm9w== X-IronPort-AV: E=McAfee;i="6000,8403,9769"; a="227156424" X-IronPort-AV: E=Sophos;i="5.77,355,1596524400"; d="scan'208";a="227156424" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Oct 2020 09:38:21 -0700 IronPort-SDR: Y/dy/93f8jM9gS+QUbZRznOxbv70weIZ1TkjMKQ6X0JCvQflMqpGIcwe15ruIttV8OwjcOtdfw nl9D/167R3gQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,355,1596524400"; d="scan'208";a="354907340" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by FMSMGA003.fm.intel.com with ESMTP; 09 Oct 2020 09:38:21 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 9 Oct 2020 09:38:20 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5 via Frontend Transport; Fri, 9 Oct 2020 09:38:20 -0700 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (104.47.37.56) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.1713.5; Fri, 9 Oct 2020 09:38:18 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cWB/McR2HFRMHdB0lLQ1jmzq0q6y3aYY2TA9/3YFSilqewaPUTCmHBk2DkrVZ1jszcZ9+P/Z9479q74k1QIMJTJlJ481WIRBiVxAbW4evIbZ6mvR3QKTwa/YBSj72cKjRUOcbz2UUoZBXLsK7Cbo3C2YI2frTjMWF5emCp0+eV31Hy1+S0YSWOB4pqmhM6To1BhWYdDHGxv3TtCky1d4p6HGrPRzNWRd6X90zwd5OJeLvl94nbLLDxsSPn8ycZSmxbxaGCPwA7rk2sKclbtRnyVj3CzPhON9OlJFNHmeNZbdMCyya5hCUPdwgdK1L/YIKTRxHUtbSthpkEniUQLkPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tTVhTN0SFqF5yDUkXf0F+V6AFQzCVsZwK8QCQaxMA/U=; b=TJo8LMLs7um7Jqdwh7YC9vUpQAS/vD+8DohRmwLLJL679WaAW4CCJ5XBlp3WEO15vMiMONSWCB8lmkbboWux09PWZ05FfjphrKX3Edg3otWxiEJ9BmwpOwm+GdbIPTc8/G6rlWCDlz2Zb3P5Jpy0x8Ll7O4AbUbL02XxzXAaj1ZwE2iQjpXLKdbPkIS6wSLaZhmx2LkZg+3d5sib6o8qZzk48WSgaKUHkx0mLZCwT5rFG2V11dmnp53w0Uts1mbOLPatFN5UCbVhxl5IwX/0v9Plpmesv988+mK8jePyNZ6vFomdOwu7CW92QSk7/n9t88kj1fo6gJsZzXn2/CzVyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tTVhTN0SFqF5yDUkXf0F+V6AFQzCVsZwK8QCQaxMA/U=; b=tgPcvgjaFJVSZO1DdAepEFUSegnWkedmea5v+39VRlKF4eBKfWbuekYm45/xtfh1n6XQM5qSllt9MAdh/8ZjGv+Tdz3LIaXqW7edr8ECewKVASWVvQWODL1QFzvJP2e8V4Md2bLtqY98gzNpbqgEkk+KGU123FXFBRo0nsouj5g= Received: from BYAPR11MB3301.namprd11.prod.outlook.com (2603:10b6:a03:7f::26) by SJ0PR11MB5008.namprd11.prod.outlook.com (2603:10b6:a03:2d5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3433.34; Fri, 9 Oct 2020 16:38:14 +0000 Received: from BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b]) by BYAPR11MB3301.namprd11.prod.outlook.com ([fe80::f5a4:3f6b:ade3:296b%3]) with mapi id 15.20.3455.027; Fri, 9 Oct 2020 16:38:13 +0000 From: "Ananyev, Konstantin" To: "Ma, Liang J" , "dev@dpdk.org" CC: "Hunt, David" , "stephen@networkplumber.org" , "Burakov, Anatoly" Thread-Topic: [PATCH v4 04/10] power: add simple power management API and callback Thread-Index: AQHWmMYQwMLlC/ugfkqtcsvaQkLoF6mPfnfA Date: Fri, 9 Oct 2020 16:38:13 +0000 Message-ID: References: <1599214740-3927-1-git-send-email-liang.j.ma@intel.com> <1601647919-25312-1-git-send-email-liang.j.ma@intel.com> <1601647919-25312-4-git-send-email-liang.j.ma@intel.com> In-Reply-To: <1601647919-25312-4-git-send-email-liang.j.ma@intel.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [46.7.39.127] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d1ec0f41-0f3d-4272-3f32-08d86c71b791 x-ms-traffictypediagnostic: SJ0PR11MB5008: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 725+2gNJqTpM7cKqoehyamcBCFa/41FMfgz0GqCNK2BhoXZcacD0wxb0u/5dXDCBJpGpHZ/cdVhNjka7ZHSRe8lbXc3qbGoKXBW5V6gql3D/LzdKkYGZmYSE5wquYeS0Gh8SVGf3jxyNS/ldvE8aOcTP3Zmp1gfTkhuKzsroG+9CKzl/F8O/uWpRyJG3Z0wLzkaid+JPI/vSO+mGJbIALhmuqzp2uqRQyDjRC7ndO5YH7gLoQtEb2qXxtbCBBd1sVohVz7EFapFRlldKIF2BuMaRRVP0o0890ZCZrwnpGa2pItPOBbX4JXYj1eYRYAg4 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR11MB3301.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(39860400002)(136003)(376002)(366004)(396003)(66946007)(66476007)(52536014)(64756008)(66446008)(66556008)(76116006)(8676002)(4326008)(83380400001)(2906002)(107886003)(478600001)(5660300002)(9686003)(55016002)(71200400001)(8936002)(33656002)(186003)(54906003)(316002)(110136005)(86362001)(26005)(7696005)(6506007); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: mYMwNrUJ5W83WlynyOO/jp+s9QxAJthMENFnkJ9GP5nvCKkK3FfpSsqjYg+fwAqiIV275mhEX7G4UF3lnROWNIWB1AySVX9uxCxnuoEvm9mCArsq5smOHbAI8DjABkcakrnRt2a1ZoVfUvN9clrx+j6I5f5MrzEXH7TvBFUGNT22gt67RRU4tHruWMUTCYjMTmpNB6FZFKYXfvdzCaH3/EZ+LwU5BWMIChGOXnguGMAMlTr4vedk5JNJp2ORXcXfIy7xUsSZlGZEGkCm/BE9vTSG8HcDiMj9LRsUXg2swB13769D1Pf/k6damOWKg1f5OwVe2VuXR5mwzLFlJnuGlSjmNmrI7+wL5aLOJB8e8faSfee/1HgReIaISGLlHB3k58uS1o+4SqUVgDdFT6c42aFBExep4r28JjYpc2ZVlGVCZDZyQwq18EZ/q71X32ENeECSa1z1y+4avxIdg8Ez0GBtPs/dN1l3rLhZjwj93u0rOxBN0NNcpeTFUiCN9y2BZCLrSqaVDbxGs3aknBLEQ4jmEd/WwdMjoW45hygBi7xVu9C0pN3XkR/PNOtsLuZOS1vbjHGy8sY4DV5IaFvJ15FTo4i4PlOnGNzXf6ok3hr2Y771FxJOwcRcNuXQFnotwY3MJbZbf9hebbaesrpIRw== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3301.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1ec0f41-0f3d-4272-3f32-08d86c71b791 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Oct 2020 16:38:13.8699 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /GVjEn1WFQIe0rY13xmkLivVaPgmMLKJ+9/Oq8FFxZ/dlMt+7mhS13Ma560Xo3wU0SyAIPioTGaPRhcak8WHhP+CO0fos9kr+JIE6+2k+5k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5008 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v4 04/10] power: add simple power management API and callback X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > Add a simple on/off switch that will enable saving power when no > packets are arriving. It is based on counting the number of empty > polls and, when the number reaches a certain threshold, entering an > architecture-defined optimized power state that will either wait > until a TSC timestamp expires, or when packets arrive. >=20 > This API support 1 port to multiple core use case. >=20 > This design leverage RX Callback mechnaism which allow three > different power management methodology co exist. >=20 > 1. umwait/umonitor: >=20 > The TSC timestamp is automatically calculated using current > link speed and RX descriptor ring size, such that the sleep > time is not longer than it would take for a NIC to fill its > entire RX descriptor ring. >=20 > 2. Pause instruction >=20 > Instead of move the core into deeper C state, this lightweight > method use Pause instruction to relief the processor from > busy polling. >=20 > 3. Frequency Scaling > Reuse exist rte power library to scale up/down core frequency > depend on traffic volume. >=20 > Signed-off-by: Liang Ma > Signed-off-by: Anatoly Burakov > --- > lib/librte_power/meson.build | 5 +- > lib/librte_power/pmd_mgmt.h | 49 ++++++ > lib/librte_power/rte_power_pmd_mgmt.c | 208 +++++++++++++++++++++++++ > lib/librte_power/rte_power_pmd_mgmt.h | 88 +++++++++++ > lib/librte_power/rte_power_version.map | 4 + > 5 files changed, 352 insertions(+), 2 deletions(-) > create mode 100644 lib/librte_power/pmd_mgmt.h > create mode 100644 lib/librte_power/rte_power_pmd_mgmt.c > create mode 100644 lib/librte_power/rte_power_pmd_mgmt.h >=20 > diff --git a/lib/librte_power/meson.build b/lib/librte_power/meson.build > index 78c031c943..cc3c7a8646 100644 > --- a/lib/librte_power/meson.build > +++ b/lib/librte_power/meson.build > @@ -9,6 +9,7 @@ sources =3D files('rte_power.c', 'power_acpi_cpufreq.c', > 'power_kvm_vm.c', 'guest_channel.c', > 'rte_power_empty_poll.c', > 'power_pstate_cpufreq.c', > + 'rte_power_pmd_mgmt.c', > 'power_common.c') > -headers =3D files('rte_power.h','rte_power_empty_poll.h') > -deps +=3D ['timer'] > +headers =3D files('rte_power.h','rte_power_empty_poll.h','rte_power_pmd_= mgmt.h') > +deps +=3D ['timer' ,'ethdev'] > diff --git a/lib/librte_power/pmd_mgmt.h b/lib/librte_power/pmd_mgmt.h > new file mode 100644 > index 0000000000..756fbe20f7 > --- /dev/null > +++ b/lib/librte_power/pmd_mgmt.h > @@ -0,0 +1,49 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2010-2020 Intel Corporation > + */ > + > +#ifndef _PMD_MGMT_H > +#define _PMD_MGMT_H > + > +/** > + * @file > + * Power Management > + */ > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +/** > + * Possible power management states of an ethdev port. > + */ > +enum pmd_mgmt_state { > + /** Device power management is disabled. */ > + PMD_MGMT_DISABLED =3D 0, > + /** Device power management is enabled. */ > + PMD_MGMT_ENABLED, > +}; > + > +struct pmd_queue_cfg { > + enum pmd_mgmt_state pwr_mgmt_state; > + /**< Power mgmt Callback mode */ > + enum rte_power_pmd_mgmt_type cb_mode; > + /**< Empty poll number */ > + uint16_t empty_poll_stats; > + /**< Callback instance */ > + const struct rte_eth_rxtx_callback *cur_cb; > +} __rte_cache_aligned; > + > +struct pmd_port_cfg { > + int ref_cnt; > + struct pmd_queue_cfg *queue_cfg; > +} __rte_cache_aligned; > + > + > + > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif > diff --git a/lib/librte_power/rte_power_pmd_mgmt.c b/lib/librte_power/rte= _power_pmd_mgmt.c > new file mode 100644 > index 0000000000..35d2af46a4 > --- /dev/null > +++ b/lib/librte_power/rte_power_pmd_mgmt.c > @@ -0,0 +1,208 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2010-2020 Intel Corporation > + */ > + > +#include > +#include > +#include > +#include > +#include > + > +#include "rte_power_pmd_mgmt.h" > +#include "pmd_mgmt.h" > + > + > +#define EMPTYPOLL_MAX 512 > +#define PAUSE_NUM 64 > + > +static struct pmd_port_cfg port_cfg[RTE_MAX_ETHPORTS]; > + > +static uint16_t > +rte_power_mgmt_umwait(uint16_t port_id, uint16_t qidx, > + struct rte_mbuf **pkts __rte_unused, uint16_t nb_rx, > + uint16_t max_pkts __rte_unused, void *_ __rte_unused) > +{ > + > + struct pmd_queue_cfg *q_conf; > + q_conf =3D &port_cfg[port_id].queue_cfg[qidx]; > + > + if (unlikely(nb_rx =3D=3D 0)) { > + q_conf->empty_poll_stats++; > + if (unlikely(q_conf->empty_poll_stats > EMPTYPOLL_MAX)) { Here and in other places - wouldn't it be better to empty_poll_max as confi= gurable parameter, instead of constant value?=20 > + volatile void *target_addr; > + uint64_t expected, mask; > + uint16_t ret; > + > + /* > + * get address of next descriptor in the RX > + * ring for this queue, as well as expected > + * value and a mask. > + */ > + ret =3D rte_eth_get_wake_addr(port_id, qidx, > + &target_addr, &expected, > + &mask); > + if (ret =3D=3D 0) > + /* -1ULL is maximum value for TSC */ > + rte_power_monitor(target_addr, > + expected, mask, > + 0, -1ULL); Why not make timeout a user specified parameter? > + } > + } else > + q_conf->empty_poll_stats =3D 0; > + > + return nb_rx; > +} > + > +static uint16_t > +rte_power_mgmt_pause(uint16_t port_id, uint16_t qidx, > + struct rte_mbuf **pkts __rte_unused, uint16_t nb_rx, > + uint16_t max_pkts __rte_unused, void *_ __rte_unused) > +{ > + struct pmd_queue_cfg *q_conf; > + int i; > + q_conf =3D &port_cfg[port_id].queue_cfg[qidx]; > + > + if (unlikely(nb_rx =3D=3D 0)) { > + q_conf->empty_poll_stats++; > + if (unlikely(q_conf->empty_poll_stats > EMPTYPOLL_MAX)) { > + for (i =3D 0; i < PAUSE_NUM; i++) > + rte_pause(); Just rte_delay_us(timeout) instead of this loop? > + } > + } else > + q_conf->empty_poll_stats =3D 0; > + > + return nb_rx; > +} > + > +static uint16_t > +rte_power_mgmt_scalefreq(uint16_t port_id, uint16_t qidx, > + struct rte_mbuf **pkts __rte_unused, uint16_t nb_rx, > + uint16_t max_pkts __rte_unused, void *_ __rte_unused) > +{ > + struct pmd_queue_cfg *q_conf; > + q_conf =3D &port_cfg[port_id].queue_cfg[qidx]; > + > + if (unlikely(nb_rx =3D=3D 0)) { > + q_conf->empty_poll_stats++; > + if (unlikely(q_conf->empty_poll_stats > EMPTYPOLL_MAX)) { > + /*scale down freq */ > + rte_power_freq_min(rte_lcore_id()); > + > + } > + } else { > + q_conf->empty_poll_stats =3D 0; > + /* scal up freq */ > + rte_power_freq_max(rte_lcore_id()); > + } > + > + return nb_rx; > +} > + Probably worth to mention in comments that these functions enable/disable are not MT safe. > +int > +rte_power_pmd_mgmt_queue_enable(unsigned int lcore_id, > + uint16_t port_id, > + uint16_t queue_id, > + enum rte_power_pmd_mgmt_type mode) > +{ > + struct rte_eth_dev *dev; > + struct pmd_queue_cfg *queue_cfg; > + int ret =3D 0; > + > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); > + dev =3D &rte_eth_devices[port_id]; > + > + if (port_cfg[port_id].queue_cfg =3D=3D NULL) { > + port_cfg[port_id].ref_cnt =3D 0; > + /* allocate memory for empty poll stats */ > + port_cfg[port_id].queue_cfg =3D rte_malloc_socket(NULL, > + sizeof(struct pmd_queue_cfg) > + * RTE_MAX_QUEUES_PER_PORT, > + 0, dev->data->numa_node); > + if (port_cfg[port_id].queue_cfg =3D=3D NULL) > + return -ENOMEM; > + } > + > + queue_cfg =3D &port_cfg[port_id].queue_cfg[queue_id]; > + > + if (queue_cfg->pwr_mgmt_state =3D=3D PMD_MGMT_ENABLED) { > + ret =3D -EINVAL; > + goto failure_handler; > + } > + > + switch (mode) { > + case RTE_POWER_MGMT_TYPE_WAIT: > + if (!rte_cpu_get_flag_enabled(RTE_CPUFLAG_WAITPKG)) { > + ret =3D -ENOTSUP; > + goto failure_handler; > + } > + queue_cfg->cur_cb =3D rte_eth_add_rx_callback(port_id, queue_id, > + rte_power_mgmt_umwait, NULL); > + break; > + case RTE_POWER_MGMT_TYPE_SCALE: > + /* init scale freq */ > + if (rte_power_init(lcore_id)) { > + ret =3D -EINVAL; > + goto failure_handler; > + } > + queue_cfg->cur_cb =3D rte_eth_add_rx_callback(port_id, queue_id, > + rte_power_mgmt_scalefreq, NULL); > + break; > + case RTE_POWER_MGMT_TYPE_PAUSE: > + queue_cfg->cur_cb =3D rte_eth_add_rx_callback(port_id, queue_id, > + rte_power_mgmt_pause, NULL); > + break; default: .... > + } > + queue_cfg->cb_mode =3D mode; > + port_cfg[port_id].ref_cnt++; > + queue_cfg->pwr_mgmt_state =3D PMD_MGMT_ENABLED; > + return ret; > + > +failure_handler: > + if (port_cfg[port_id].ref_cnt =3D=3D 0) { > + rte_free(port_cfg[port_id].queue_cfg); > + port_cfg[port_id].queue_cfg =3D NULL; > + } > + return ret; > +} > + > +int > +rte_power_pmd_mgmt_queue_disable(unsigned int lcore_id, > + uint16_t port_id, > + uint16_t queue_id) > +{ > + struct pmd_queue_cfg *queue_cfg; > + > + if (port_cfg[port_id].ref_cnt <=3D 0) > + return -EINVAL; > + > + queue_cfg =3D &port_cfg[port_id].queue_cfg[queue_id]; > + > + if (queue_cfg->pwr_mgmt_state =3D=3D PMD_MGMT_DISABLED) > + return -EINVAL; > + > + switch (queue_cfg->cb_mode) { > + case RTE_POWER_MGMT_TYPE_WAIT: Think we need wakeup(lcore_id) here. > + case RTE_POWER_MGMT_TYPE_PAUSE: > + rte_eth_remove_rx_callback(port_id, queue_id, > + queue_cfg->cur_cb); > + break; > + case RTE_POWER_MGMT_TYPE_SCALE: > + rte_power_freq_max(lcore_id); > + rte_eth_remove_rx_callback(port_id, queue_id, > + queue_cfg->cur_cb); > + rte_power_exit(lcore_id); > + break; > + } > + /* it's not recommend to free callback instance here. > + * it cause memory leak which is a known issue. > + */ > + queue_cfg->cur_cb =3D NULL; > + queue_cfg->pwr_mgmt_state =3D PMD_MGMT_DISABLED; > + port_cfg[port_id].ref_cnt--; > + > + if (port_cfg[port_id].ref_cnt =3D=3D 0) { > + rte_free(port_cfg[port_id].queue_cfg); It is not safe to do so, unless device is already stopped. Otherwise you need some sync mechanism here (hand-made as bpf lib, or rcu o= nline/offline, or...) > + port_cfg[port_id].queue_cfg =3D NULL; > + } > + return 0; > +}