DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>,
	"dev@dpdk.org" <dev@dpdk.org>
Cc: "Li, Xiaoyun" <xiaoyun.li@intel.com>,
	"anoobj@marvell.com" <anoobj@marvell.com>,
	"jerinj@marvell.com" <jerinj@marvell.com>,
	"ndabilpuram@marvell.com" <ndabilpuram@marvell.com>,
	"adwivedi@marvell.com" <adwivedi@marvell.com>,
	"shepard.siegel@atomicrules.com" <shepard.siegel@atomicrules.com>,
	"ed.czeck@atomicrules.com" <ed.czeck@atomicrules.com>,
	"john.miller@atomicrules.com" <john.miller@atomicrules.com>,
	"irusskikh@marvell.com" <irusskikh@marvell.com>,
	"ajit.khaparde@broadcom.com" <ajit.khaparde@broadcom.com>,
	"somnath.kotur@broadcom.com" <somnath.kotur@broadcom.com>,
	"rahul.lakkireddy@chelsio.com" <rahul.lakkireddy@chelsio.com>,
	"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
	"sachin.saxena@oss.nxp.com" <sachin.saxena@oss.nxp.com>,
	"Wang, Haiyue" <haiyue.wang@intel.com>,
	"Daley, John" <johndale@cisco.com>,
	"hyonkim@cisco.com" <hyonkim@cisco.com>,
	"Zhang, Qi Z" <qi.z.zhang@intel.com>,
	"Wang, Xiao W" <xiao.w.wang@intel.com>,
	"humin29@huawei.com" <humin29@huawei.com>,
	"yisen.zhuang@huawei.com" <yisen.zhuang@huawei.com>,
	"oulijun@huawei.com" <oulijun@huawei.com>,
	"Xing, Beilei" <beilei.xing@intel.com>,
	"Wu, Jingjing" <jingjing.wu@intel.com>,
	"Yang, Qiming" <qiming.yang@intel.com>,
	"matan@nvidia.com" <matan@nvidia.com>,
	"viacheslavo@nvidia.com" <viacheslavo@nvidia.com>,
	"sthemmin@microsoft.com" <sthemmin@microsoft.com>,
	"longli@microsoft.com" <longli@microsoft.com>,
	"heinrich.kuhn@corigine.com" <heinrich.kuhn@corigine.com>,
	"kirankumark@marvell.com" <kirankumark@marvell.com>,
	"mczekaj@marvell.com" <mczekaj@marvell.com>,
	"jiawenwu@trustnetic.com" <jiawenwu@trustnetic.com>,
	"jianwang@trustnetic.com" <jianwang@trustnetic.com>,
	"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
	"Xia, Chenbo" <chenbo.xia@intel.com>,
	"thomas@monjalon.net" <thomas@monjalon.net>,
	"Yigit, Ferruh" <ferruh.yigit@intel.com>,
	"mdr@ashroe.eu" <mdr@ashroe.eu>,
	"Jayatheerthan, Jay" <jay.jayatheerthan@intel.com>
Subject: Re: [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array
Date: Mon, 11 Oct 2021 23:06:34 +0000	[thread overview]
Message-ID: <DM6PR11MB4491FB3C1DB1D557CE1F06999AB59@DM6PR11MB4491.namprd11.prod.outlook.com> (raw)
In-Reply-To: <05fdac98-194d-94a8-0fa7-197353c27fe0@oktetlabs.ru>


> >>> At queue configure stage always allocate space for maximum possible
> >>> number (RTE_MAX_QUEUES_PER_PORT) of queue pointers.
> >>> That will allow 'fast' inline functions (eth_rx_burst, etc.) to refer
> >>> pointer to internal queue data without extra checking of current number
> >>> of configured queues.
> >>> That would help in future to hide rte_eth_dev and related structures.
> >>> It means that from now on, each ethdev port will always consume:
> >>> ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT)
> >>> bytes of memory for its queue pointers.
> >>> With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port.
> >>>
> >>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> >>> ---
> >>>   lib/ethdev/rte_ethdev.c | 36 +++++++++---------------------------
> >>>   1 file changed, 9 insertions(+), 27 deletions(-)
> >>>
> >>> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
> >>> index ed37f8871b..c8abda6dd7 100644
> >>> --- a/lib/ethdev/rte_ethdev.c
> >>> +++ b/lib/ethdev/rte_ethdev.c
> >>> @@ -897,7 +897,8 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
> >>>
> >>>   	if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */
> >>>   		dev->data->rx_queues = rte_zmalloc("ethdev->rx_queues",
> >>> -				sizeof(dev->data->rx_queues[0]) * nb_queues,
> >>> +				sizeof(dev->data->rx_queues[0]) *
> >>> +				RTE_MAX_QUEUES_PER_PORT,
> >>>   				RTE_CACHE_LINE_SIZE);
> >>
> >> Looking at it I have few questions:
> >> 1. Why is nb_queues == 0 case kept as an exception? Yes,
> >>     strictly speaking it is not the problem of the patch,
> >>     DPDK will still segfault (non-debug build) if I
> >>     allocate Tx queues only but call rte_eth_rx_burst().
> >
> > eth_dev_rx_queue_config(.., nb_queues=0) is used in few places to clean-up things.
> 
> No, as far as I know. For Tx only application (e.g. traffic generator)
> it is 100% legal to configure with tx_queues=X, rx_queues=0.
> The same is for Rx only application (e.g. packet capture).

Yes, that is valid config for sure.
I just pointed that simply ignoring 'nb_queues' value and
always allocating space for max possible queues, i.e:

eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues) 
{
....
- if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */
+ if (dev->data->rx_queues == NULL) {
wouldn't work, as right now nb_queues == 0 has extra special meaning -
do final cleanup and free dev->data->rx_queues.
But re-reading the text below, it seems that I misunderstood you
and it probably wasn't your intention anyway.

> 
> >
> >>     After reading the patch description I thought that
> >>     we're trying to address it.
> >
> > We do, though I can't see how we can address it in this patch.
> > Though it is a good idea - I think I can add extra check in eth_dev_fp_ops_setup()
> > or around and setup RX function pointers only when dev->data->rx_queues != NULL.
> > Same for TX.
> 
> You don't need to care about these pointers, if these arrays are
> always allocated. See (3) below.
> 
> >
> >> 2. Why do we need to allocate memory dynamically?
> >>     Can we just make rx_queues an array of appropriate size?
> >
> > Pavan already asked same question.
> > My answer to him:
> > Yep we can, and yes it will simplify this peace of code.
> > The main reason I decided no to do this change now -
> > it will change layout of the_eth_dev_data structure.
> > In this series I tried to mininize(/avoid) changes in rte_eth_dev and rte_eth_dev_data,
> > as much as possible to avoid any unforeseen performance and functional impacts.
> > If we'll manage to make rte_eth_dev and rte_eth_dev_data private we can in future
> > consider that one and other changes in rte_eth_dev and rte_eth_dev_data layouts
> > without worrying about ABI breakage
> 
> Thanks a lot. Makes sense.
> 
> >>     May be wasting 512K unconditionally is too much.
> >> 3. If wasting 512K is too much, I'd consider to move
> >>     allocation to eth_dev_get(). If
> >
> > Don't understand where 512KB came from.
> 
> 32 port * 1024 queues * 2 types * 8 pointer size
> if we allocate as in (2) above.
> 
> > each ethdev port will always consume:
> > ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT)
> > bytes of memory for its queue pointers.
> > With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port.
> 
> IMHO it will be a bit nicer if queue pointers arrays are allocated
> on device get if size is fixed. It is just a suggestion. If you
> disagree, feel free to drop it.

You mean - allocate these arrays somewhere at rte_eth_dev_allocate() path?
That sounds like an interesting idea, but seems too drastic to me at that stage.

> 
> >>>   		if (dev->data->rx_queues == NULL) {
> >>>   			dev->data->nb_rx_queues = 0;
> >>> @@ -908,21 +909,11 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
> >>>
> >>>   		rxq = dev->data->rx_queues;
> >>>
> >>> -		for (i = nb_queues; i < old_nb_queues; i++)
> >>> +		for (i = nb_queues; i < old_nb_queues; i++) {
> >>>   			(*dev->dev_ops->rx_queue_release)(rxq[i]);
> >>> -		rxq = rte_realloc(rxq, sizeof(rxq[0]) * nb_queues,
> >>> -				RTE_CACHE_LINE_SIZE);
> >>> -		if (rxq == NULL)
> >>> -			return -(ENOMEM);
> >>> -		if (nb_queues > old_nb_queues) {
> >>> -			uint16_t new_qs = nb_queues - old_nb_queues;
> >>> -
> >>> -			memset(rxq + old_nb_queues, 0,
> >>> -				sizeof(rxq[0]) * new_qs);
> >>> +			rxq[i] = NULL;
> >>
> >> It looks like the patch should be rebased on top of
> >> next-net main because of queue release patches.
> >>
> >> [snip]


  reply	other threads:[~2021-10-11 23:07 UTC|newest]

Thread overview: 112+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-20 16:28 [dpdk-dev] [RFC 0/7] hide eth dev related structures Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 1/7] eth: move ethdev 'burst' API into separate structure Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 2/7] eth: make drivers to use new API for Rx Konstantin Ananyev
2021-09-06 18:41   ` Ferruh Yigit
2021-09-14 14:28     ` Ananyev, Konstantin
2021-08-20 16:28 ` [dpdk-dev] [RFC 3/7] eth: make drivers to use new API for Tx Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 4/7] eth: make drivers to use new API for Tx prepare Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 5/7] eth: make drivers to use new API to obtain descriptor status Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 6/7] eth: make drivers to use new API for Rx queue count Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 7/7] eth: hide eth dev related structures Konstantin Ananyev
2021-08-26 12:37 ` [dpdk-dev] [RFC 0/7] " Jerin Jacob
2021-09-06 18:09   ` Ferruh Yigit
2021-09-14 13:33   ` Ananyev, Konstantin
2021-09-15  9:45     ` Jerin Jacob
2021-09-22 15:08       ` Ananyev, Konstantin
2021-09-27 16:14         ` Jerin Jacob
2021-09-28  9:37           ` Ananyev, Konstantin
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 0/5] " Konstantin Ananyev
2021-09-22 14:09   ` [dpdk-dev] [RFC v2 1/5] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-09-22 14:09   ` [dpdk-dev] [RFC v2 2/5] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-09-23  5:51     ` Wang, Haiyue
2021-09-22 14:09   ` [dpdk-dev] [RFC v2 3/5] ethdev: copy ethdev 'burst' API into separate structure Konstantin Ananyev
2021-09-23  5:58     ` Wang, Haiyue
2021-09-27 18:01       ` Jerin Jacob
2021-09-28  9:42         ` Ananyev, Konstantin
2021-09-22 14:09   ` [dpdk-dev] [RFC v2 4/5] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-09-22 14:09   ` [dpdk-dev] [RFC v2 5/5] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 14:02   ` [dpdk-dev] [PATCH v3 0/7] " Konstantin Ananyev
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-01 16:48       ` Ferruh Yigit
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-01 16:46       ` Ferruh Yigit
2021-10-01 17:40         ` Ananyev, Konstantin
2021-10-04  8:46           ` Ferruh Yigit
2021-10-04  9:20             ` Ananyev, Konstantin
2021-10-04 10:13               ` Ferruh Yigit
2021-10-04 11:17                 ` Ananyev, Konstantin
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-01 14:02     ` [dpdk-dev] [PATCH v3 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 16:53       ` Ferruh Yigit
2021-10-01 17:04         ` Ferruh Yigit
2021-10-01 17:02     ` [dpdk-dev] [PATCH v3 0/7] " Ferruh Yigit
2021-10-04 13:55     ` [dpdk-dev] [PATCH v4 " Konstantin Ananyev
2021-10-04 13:55       ` [dpdk-dev] [PATCH v4 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-05 12:09         ` Thomas Monjalon
2021-10-05 16:45           ` Ananyev, Konstantin
2021-10-05 16:49             ` Thomas Monjalon
2021-10-05 12:21         ` Thomas Monjalon
2021-10-04 13:55       ` [dpdk-dev] [PATCH v4 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-04 13:55       ` [dpdk-dev] [PATCH v4 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-05 13:09         ` Thomas Monjalon
2021-10-05 16:41           ` Ananyev, Konstantin
2021-10-05 16:48             ` Thomas Monjalon
2021-10-05 17:04               ` Ananyev, Konstantin
2021-10-04 13:56       ` [dpdk-dev] [PATCH v4 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-05  9:54         ` David Marchand
2021-10-05 10:13           ` Ananyev, Konstantin
2021-10-04 13:56       ` [dpdk-dev] [PATCH v4 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-05 13:13         ` Thomas Monjalon
2021-10-05 16:35           ` Ananyev, Konstantin
2021-10-05 16:45             ` Thomas Monjalon
2021-10-05 17:12               ` Ananyev, Konstantin
2021-10-05 17:41                 ` Thomas Monjalon
2021-10-04 13:56       ` [dpdk-dev] [PATCH v4 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-05 13:14         ` Thomas Monjalon
2021-10-05 16:21           ` Ananyev, Konstantin
2021-10-04 13:56       ` [dpdk-dev] [PATCH v4 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-05 10:04         ` David Marchand
2021-10-05 10:43           ` Ferruh Yigit
2021-10-05 11:37             ` David Marchand
2021-10-05 15:57               ` Ananyev, Konstantin
2021-10-05 13:24         ` Thomas Monjalon
2021-10-05 16:19           ` Ananyev, Konstantin
2021-10-05 16:25             ` Thomas Monjalon
2021-10-06 16:42       ` [dpdk-dev] [PATCH v4 0/7] " Ali Alnubani
2021-10-06 17:26         ` Ali Alnubani
2021-10-07 11:27       ` [dpdk-dev] [PATCH v5 " Konstantin Ananyev
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 1/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-11  9:20           ` Andrew Rybchenko
2021-10-11 16:25             ` Ananyev, Konstantin
2021-10-11 17:15               ` Andrew Rybchenko
2021-10-11 23:06                 ` Ananyev, Konstantin [this message]
2021-10-12  5:47                   ` Andrew Rybchenko
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 3/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-11  8:06           ` Andrew Rybchenko
2021-10-12 17:59           ` Hyong Youb Kim (hyonkim)
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 4/7] ethdev: copy fast-path API into separate structure Konstantin Ananyev
2021-10-09 12:05           ` fengchengwen
2021-10-11  1:18             ` fengchengwen
2021-10-11  8:39               ` Andrew Rybchenko
2021-10-11 15:24               ` Ananyev, Konstantin
2021-10-11  8:35             ` Andrew Rybchenko
2021-10-11 15:15             ` Ananyev, Konstantin
2021-10-11  8:25           ` Andrew Rybchenko
2021-10-11 16:52             ` Ananyev, Konstantin
2021-10-11 17:22               ` Andrew Rybchenko
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 5/7] ethdev: make fast-path functions to use new flat array Konstantin Ananyev
2021-10-11  9:02           ` Andrew Rybchenko
2021-10-11 15:47             ` Ananyev, Konstantin
2021-10-11 17:03               ` Andrew Rybchenko
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 6/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-11  9:09           ` Andrew Rybchenko
2021-10-07 11:27         ` [dpdk-dev] [PATCH v5 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-11  9:20           ` Andrew Rybchenko
2021-10-11 15:54             ` Ananyev, Konstantin
2021-10-11 17:04               ` Andrew Rybchenko
2021-10-08 18:13         ` [dpdk-dev] [PATCH v5 0/7] " Slava Ovsiienko
2021-10-11  9:22         ` Andrew Rybchenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DM6PR11MB4491FB3C1DB1D557CE1F06999AB59@DM6PR11MB4491.namprd11.prod.outlook.com \
    --to=konstantin.ananyev@intel.com \
    --cc=adwivedi@marvell.com \
    --cc=ajit.khaparde@broadcom.com \
    --cc=andrew.rybchenko@oktetlabs.ru \
    --cc=anoobj@marvell.com \
    --cc=beilei.xing@intel.com \
    --cc=chenbo.xia@intel.com \
    --cc=dev@dpdk.org \
    --cc=ed.czeck@atomicrules.com \
    --cc=ferruh.yigit@intel.com \
    --cc=haiyue.wang@intel.com \
    --cc=heinrich.kuhn@corigine.com \
    --cc=hemant.agrawal@nxp.com \
    --cc=humin29@huawei.com \
    --cc=hyonkim@cisco.com \
    --cc=irusskikh@marvell.com \
    --cc=jay.jayatheerthan@intel.com \
    --cc=jerinj@marvell.com \
    --cc=jianwang@trustnetic.com \
    --cc=jiawenwu@trustnetic.com \
    --cc=jingjing.wu@intel.com \
    --cc=john.miller@atomicrules.com \
    --cc=johndale@cisco.com \
    --cc=kirankumark@marvell.com \
    --cc=longli@microsoft.com \
    --cc=matan@nvidia.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=mczekaj@marvell.com \
    --cc=mdr@ashroe.eu \
    --cc=ndabilpuram@marvell.com \
    --cc=oulijun@huawei.com \
    --cc=qi.z.zhang@intel.com \
    --cc=qiming.yang@intel.com \
    --cc=rahul.lakkireddy@chelsio.com \
    --cc=sachin.saxena@oss.nxp.com \
    --cc=shepard.siegel@atomicrules.com \
    --cc=somnath.kotur@broadcom.com \
    --cc=sthemmin@microsoft.com \
    --cc=thomas@monjalon.net \
    --cc=viacheslavo@nvidia.com \
    --cc=xiao.w.wang@intel.com \
    --cc=xiaoyun.li@intel.com \
    --cc=yisen.zhuang@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).