[PATCH] power: use hugepage memory for queue list entry structure

[PATCH] power: use hugepage memory for queue list entry structure

[Huisong Li]

The queue_list_entry structure data is used in rx_callback of io path
when enable PMD Power Management. However its memory is currently from
normal heap memory. For better performance, use hugepage memory to
replace it.

Signed-off-by: Huisong Li <lihuisong@huawei.com>
---
 lib/power/rte_power_pmd_mgmt.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/power/rte_power_pmd_mgmt.c b/lib/power/rte_power_pmd_mgmt.c
index a2fff3b765..c7bf57a910 100644
--- a/lib/power/rte_power_pmd_mgmt.c
+++ b/lib/power/rte_power_pmd_mgmt.c
@@ -97,7 +97,7 @@ queue_list_find(const struct pmd_core_cfg *cfg, const union queue *q)
 }
 
 static int
-queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
+queue_list_add(struct pmd_core_cfg *cfg, const union queue *q, unsigned int lcore_id)
 {
 	struct queue_list_entry *qle;
 
@@ -105,10 +105,10 @@ queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
 	if (queue_list_find(cfg, q) != NULL)
 		return -EEXIST;
 
-	qle = malloc(sizeof(*qle));
+	qle = rte_zmalloc_socket(NULL, sizeof(*qle), RTE_CACHE_LINE_SIZE,
+				 rte_lcore_to_socket_id(lcore_id));
 	if (qle == NULL)
 		return -ENOMEM;
-	memset(qle, 0, sizeof(*qle));
 
 	queue_copy(&qle->queue, q);
 	TAILQ_INSERT_TAIL(&cfg->head, qle, next);
@@ -570,7 +570,7 @@ rte_power_ethdev_pmgmt_queue_enable(unsigned int lcore_id, uint16_t port_id,
 		goto end;
 	}
 	/* add this queue to the list */
-	ret = queue_list_add(lcore_cfg, &qdata);
+	ret = queue_list_add(lcore_cfg, &qdata, lcore_id);
 	if (ret < 0) {
 		POWER_LOG(DEBUG, "Failed to add queue to list: %s",
 				strerror(-ret));
@@ -664,7 +664,7 @@ rte_power_ethdev_pmgmt_queue_disable(unsigned int lcore_id,
 	 * callbacks can be freed. we're intentionally casting away const-ness.
 	 */
 	rte_free((void *)(uintptr_t)queue_cfg->cb);
-	free(queue_cfg);
+	rte_free(queue_cfg);
 
 	return 0;
 }
-- 
2.22.0

Re: [PATCH] power: use hugepage memory for queue list entry structure

[lihuisong (C)]

Hi all,

Kindly ping for review.


在 2024/12/19 15:53, Huisong Li 写道:
> The queue_list_entry structure data is used in rx_callback of io path
> when enable PMD Power Management. However its memory is currently from
> normal heap memory. For better performance, use hugepage memory to
> replace it.
>
> Signed-off-by: Huisong Li <lihuisong@huawei.com>
> ---
>   lib/power/rte_power_pmd_mgmt.c | 10 +++++-----
>   1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/lib/power/rte_power_pmd_mgmt.c b/lib/power/rte_power_pmd_mgmt.c
> index a2fff3b765..c7bf57a910 100644
> --- a/lib/power/rte_power_pmd_mgmt.c
> +++ b/lib/power/rte_power_pmd_mgmt.c
> @@ -97,7 +97,7 @@ queue_list_find(const struct pmd_core_cfg *cfg, const union queue *q)
>   }
>   
>   static int
> -queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
> +queue_list_add(struct pmd_core_cfg *cfg, const union queue *q, unsigned int lcore_id)
>   {
>   	struct queue_list_entry *qle;
>   
> @@ -105,10 +105,10 @@ queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
>   	if (queue_list_find(cfg, q) != NULL)
>   		return -EEXIST;
>   
> -	qle = malloc(sizeof(*qle));
> +	qle = rte_zmalloc_socket(NULL, sizeof(*qle), RTE_CACHE_LINE_SIZE,
> +				 rte_lcore_to_socket_id(lcore_id));
>   	if (qle == NULL)
>   		return -ENOMEM;
> -	memset(qle, 0, sizeof(*qle));
>   
>   	queue_copy(&qle->queue, q);
>   	TAILQ_INSERT_TAIL(&cfg->head, qle, next);
> @@ -570,7 +570,7 @@ rte_power_ethdev_pmgmt_queue_enable(unsigned int lcore_id, uint16_t port_id,
>   		goto end;
>   	}
>   	/* add this queue to the list */
> -	ret = queue_list_add(lcore_cfg, &qdata);
> +	ret = queue_list_add(lcore_cfg, &qdata, lcore_id);
>   	if (ret < 0) {
>   		POWER_LOG(DEBUG, "Failed to add queue to list: %s",
>   				strerror(-ret));
> @@ -664,7 +664,7 @@ rte_power_ethdev_pmgmt_queue_disable(unsigned int lcore_id,
>   	 * callbacks can be freed. we're intentionally casting away const-ness.
>   	 */
>   	rte_free((void *)(uintptr_t)queue_cfg->cb);
> -	free(queue_cfg);
> +	rte_free(queue_cfg);
>   
>   	return 0;
>   }

RE: [PATCH] power: use hugepage memory for queue list entry structure

[Konstantin Ananyev]

Hi 

> 
> Hi all,
> 
> Kindly ping for review.
> 
> 
> 在 2024/12/19 15:53, Huisong Li 写道:
> > The queue_list_entry structure data is used in rx_callback of io path
> > when enable PMD Power Management. However its memory is currently from
> > normal heap memory. For better performance, use hugepage memory to
> > replace it.

Make sense to me.
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>

I  suppose it would also help if you can provide some numbers:
i.e.: how much exactly it is 'better'?
Did you see any changes in throughput/latency numbers, etc.

> > Signed-off-by: Huisong Li <lihuisong@huawei.com>
> > ---
> >   lib/power/rte_power_pmd_mgmt.c | 10 +++++-----
> >   1 file changed, 5 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/power/rte_power_pmd_mgmt.c b/lib/power/rte_power_pmd_mgmt.c
> > index a2fff3b765..c7bf57a910 100644
> > --- a/lib/power/rte_power_pmd_mgmt.c
> > +++ b/lib/power/rte_power_pmd_mgmt.c
> > @@ -97,7 +97,7 @@ queue_list_find(const struct pmd_core_cfg *cfg, const union queue *q)
> >   }
> >
> >   static int
> > -queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
> > +queue_list_add(struct pmd_core_cfg *cfg, const union queue *q, unsigned int lcore_id)
> >   {
> >   	struct queue_list_entry *qle;
> >
> > @@ -105,10 +105,10 @@ queue_list_add(struct pmd_core_cfg *cfg, const union queue *q)
> >   	if (queue_list_find(cfg, q) != NULL)
> >   		return -EEXIST;
> >
> > -	qle = malloc(sizeof(*qle));
> > +	qle = rte_zmalloc_socket(NULL, sizeof(*qle), RTE_CACHE_LINE_SIZE,
> > +				 rte_lcore_to_socket_id(lcore_id));
> >   	if (qle == NULL)
> >   		return -ENOMEM;
> > -	memset(qle, 0, sizeof(*qle));
> >
> >   	queue_copy(&qle->queue, q);
> >   	TAILQ_INSERT_TAIL(&cfg->head, qle, next);
> > @@ -570,7 +570,7 @@ rte_power_ethdev_pmgmt_queue_enable(unsigned int lcore_id, uint16_t port_id,
> >   		goto end;
> >   	}
> >   	/* add this queue to the list */
> > -	ret = queue_list_add(lcore_cfg, &qdata);
> > +	ret = queue_list_add(lcore_cfg, &qdata, lcore_id);
> >   	if (ret < 0) {
> >   		POWER_LOG(DEBUG, "Failed to add queue to list: %s",
> >   				strerror(-ret));
> > @@ -664,7 +664,7 @@ rte_power_ethdev_pmgmt_queue_disable(unsigned int lcore_id,
> >   	 * callbacks can be freed. we're intentionally casting away const-ness.
> >   	 */
> >   	rte_free((void *)(uintptr_t)queue_cfg->cb);
> > -	free(queue_cfg);
> > +	rte_free(queue_cfg);
> >
> >   	return 0;
> >   }

Re: [PATCH] power: use hugepage memory for queue list entry structure

[Stephen Hemminger]

On Thu, 20 Feb 2025 17:01:53 +0800
"lihuisong (C)" <lihuisong@huawei.com> wrote:

> > The queue_list_entry structure data is used in rx_callback of io path
> > when enable PMD Power Management. However its memory is currently from
> > normal heap memory. For better performance, use hugepage memory to
> > replace it.
> >
> > Signed-off-by: Huisong Li <lihuisong@huawei.com>

How is that in a hot path where this could matter?
The safety rails in rte_malloc() are much less than regular malloc().
I prefer some degree of safety from checkers and malloc library internals.

RE: [PATCH] power: use hugepage memory for queue list entry structure

[Konstantin Ananyev]

> -----Original Message-----
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Thursday, February 20, 2025 4:12 PM
> To: lihuisong (C) <lihuisong@huawei.com>
> Cc: dev@dpdk.org; thomas@monjalon.net; david.hunt@intel.com; anatoly.burakov@intel.com; sivaprasad.tummala@amd.com;
> liuyonglong <liuyonglong@huawei.com>
> Subject: Re: [PATCH] power: use hugepage memory for queue list entry structure
> 
> On Thu, 20 Feb 2025 17:01:53 +0800
> "lihuisong (C)" <lihuisong@huawei.com> wrote:
> 
> > > The queue_list_entry structure data is used in rx_callback of io path
> > > when enable PMD Power Management. However its memory is currently from
> > > normal heap memory. For better performance, use hugepage memory to
> > > replace it.
> > >
> > > Signed-off-by: Huisong Li <lihuisong@huawei.com>
> 
> How is that in a hot path where this could matter?

AFAIU - it is used in RX/TX callbacks that power library installs,
so I presume will get hit on every eth_rx_burst/tx_burst calls.

> The safety rails in rte_malloc() are much less than regular malloc().
> I prefer some degree of safety from checkers and malloc library internals.

Didn't get your point - what's suddenly wrong with rte_malloc()?

 

Re: [PATCH] power: use hugepage memory for queue list entry structure

[Stephen Hemminger]

On Thu, 20 Feb 2025 16:39:52 +0000
Konstantin Ananyev <konstantin.ananyev@huawei.com> wrote:

> > -----Original Message-----
> > From: Stephen Hemminger <stephen@networkplumber.org>
> > Sent: Thursday, February 20, 2025 4:12 PM
> > To: lihuisong (C) <lihuisong@huawei.com>
> > Cc: dev@dpdk.org; thomas@monjalon.net; david.hunt@intel.com; anatoly.burakov@intel.com; sivaprasad.tummala@amd.com;
> > liuyonglong <liuyonglong@huawei.com>
> > Subject: Re: [PATCH] power: use hugepage memory for queue list entry structure
> > 
> > On Thu, 20 Feb 2025 17:01:53 +0800
> > "lihuisong (C)" <lihuisong@huawei.com> wrote:
> >   
> > > > The queue_list_entry structure data is used in rx_callback of io path
> > > > when enable PMD Power Management. However its memory is currently from
> > > > normal heap memory. For better performance, use hugepage memory to
> > > > replace it.
> > > >
> > > > Signed-off-by: Huisong Li <lihuisong@huawei.com>  
> > 
> > How is that in a hot path where this could matter?  
> 
> AFAIU - it is used in RX/TX callbacks that power library installs,
> so I presume will get hit on every eth_rx_burst/tx_burst calls.
> 
> > The safety rails in rte_malloc() are much less than regular malloc().
> > I prefer some degree of safety from checkers and malloc library internals.  
> 
> Didn't get your point - what's suddenly wrong with rte_malloc()?

Coverity and Gcc analyzer treat malloc as special case.
With attributes rte_malloc gets similar treatment but not quite as much.
Also internally, malloc and free have more heap pool sanity checks.
In name of performance, those don't exist in rte_malloc().
Lastly hugepages are limited resource, so they should only be used when needed.

RE: [PATCH] power: use hugepage memory for queue list entry structure

[Konstantin Ananyev]

> > > -----Original Message-----
> > > From: Stephen Hemminger <stephen@networkplumber.org>
> > > Sent: Thursday, February 20, 2025 4:12 PM
> > > To: lihuisong (C) <lihuisong@huawei.com>
> > > Cc: dev@dpdk.org; thomas@monjalon.net; david.hunt@intel.com; anatoly.burakov@intel.com; sivaprasad.tummala@amd.com;
> > > liuyonglong <liuyonglong@huawei.com>
> > > Subject: Re: [PATCH] power: use hugepage memory for queue list entry structure
> > >
> > > On Thu, 20 Feb 2025 17:01:53 +0800
> > > "lihuisong (C)" <lihuisong@huawei.com> wrote:
> > >
> > > > > The queue_list_entry structure data is used in rx_callback of io path
> > > > > when enable PMD Power Management. However its memory is currently from
> > > > > normal heap memory. For better performance, use hugepage memory to
> > > > > replace it.
> > > > >
> > > > > Signed-off-by: Huisong Li <lihuisong@huawei.com>
> > >
> > > How is that in a hot path where this could matter?
> >
> > AFAIU - it is used in RX/TX callbacks that power library installs,
> > so I presume will get hit on every eth_rx_burst/tx_burst calls.
> >
> > > The safety rails in rte_malloc() are much less than regular malloc().
> > > I prefer some degree of safety from checkers and malloc library internals.
> >
> > Didn't get your point - what's suddenly wrong with rte_malloc()?
> 
> Coverity and Gcc analyzer treat malloc as special case.
> With attributes rte_malloc gets similar treatment but not quite as much.
> Also internally, malloc and free have more heap pool sanity checks.
> In name of performance, those don't exist in rte_malloc().
> Lastly hugepages are limited resource, so they should only be used when needed.

Ok, I understand all that, but what you suggest then - not use rte_malloc() anywhere
In DPDK code, even when it is a hot-path?

Re: [PATCH] power: use hugepage memory for queue list entry structure

[Burakov, Anatoly]

On 20/02/2025 17:45, Stephen Hemminger wrote:
> On Thu, 20 Feb 2025 16:39:52 +0000
> Konstantin Ananyev <konstantin.ananyev@huawei.com> wrote:
> 
>>> -----Original Message-----
>>> From: Stephen Hemminger <stephen@networkplumber.org>
>>> Sent: Thursday, February 20, 2025 4:12 PM
>>> To: lihuisong (C) <lihuisong@huawei.com>
>>> Cc: dev@dpdk.org; thomas@monjalon.net; david.hunt@intel.com; anatoly.burakov@intel.com; sivaprasad.tummala@amd.com;
>>> liuyonglong <liuyonglong@huawei.com>
>>> Subject: Re: [PATCH] power: use hugepage memory for queue list entry structure
>>>
>>> On Thu, 20 Feb 2025 17:01:53 +0800
>>> "lihuisong (C)" <lihuisong@huawei.com> wrote:
>>>    
>>>>> The queue_list_entry structure data is used in rx_callback of io path
>>>>> when enable PMD Power Management. However its memory is currently from
>>>>> normal heap memory. For better performance, use hugepage memory to
>>>>> replace it.
>>>>>
>>>>> Signed-off-by: Huisong Li <lihuisong@huawei.com>
>>>
>>> How is that in a hot path where this could matter?
>>
>> AFAIU - it is used in RX/TX callbacks that power library installs,
>> so I presume will get hit on every eth_rx_burst/tx_burst calls.
>>
>>> The safety rails in rte_malloc() are much less than regular malloc().
>>> I prefer some degree of safety from checkers and malloc library internals.
>>
>> Didn't get your point - what's suddenly wrong with rte_malloc()?
> 
> Coverity and Gcc analyzer treat malloc as special case.
> With attributes rte_malloc gets similar treatment but not quite as much.
> Also internally, malloc and free have more heap pool sanity checks.
> In name of performance, those don't exist in rte_malloc().
> Lastly hugepages are limited resource, so they should only be used when needed.

The last thing I would associate with rte_malloc is performance. I'm not 
sure I follow - which "sanity checks" were omitted from rte_malloc "in 
the name of performance" that are present in regular malloc?

-- 
Thanks,
Anatoly