From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>
Cc: "Li, Xiaoyun" <xiaoyun.li@intel.com>,
"anoobj@marvell.com" <anoobj@marvell.com>,
"jerinj@marvell.com" <jerinj@marvell.com>,
"ndabilpuram@marvell.com" <ndabilpuram@marvell.com>,
"adwivedi@marvell.com" <adwivedi@marvell.com>,
"shepard.siegel@atomicrules.com" <shepard.siegel@atomicrules.com>,
"ed.czeck@atomicrules.com" <ed.czeck@atomicrules.com>,
"john.miller@atomicrules.com" <john.miller@atomicrules.com>,
"irusskikh@marvell.com" <irusskikh@marvell.com>,
"ajit.khaparde@broadcom.com" <ajit.khaparde@broadcom.com>,
"somnath.kotur@broadcom.com" <somnath.kotur@broadcom.com>,
"rahul.lakkireddy@chelsio.com" <rahul.lakkireddy@chelsio.com>,
"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
"sachin.saxena@oss.nxp.com" <sachin.saxena@oss.nxp.com>,
"Wang, Haiyue" <haiyue.wang@intel.com>,
"Daley, John" <johndale@cisco.com>,
"hyonkim@cisco.com" <hyonkim@cisco.com>,
"Zhang, Qi Z" <qi.z.zhang@intel.com>,
"Wang, Xiao W" <xiao.w.wang@intel.com>,
"humin29@huawei.com" <humin29@huawei.com>,
"yisen.zhuang@huawei.com" <yisen.zhuang@huawei.com>,
"oulijun@huawei.com" <oulijun@huawei.com>,
"Xing, Beilei" <beilei.xing@intel.com>,
"Wu, Jingjing" <jingjing.wu@intel.com>,
"Yang, Qiming" <qiming.yang@intel.com>,
"matan@nvidia.com" <matan@nvidia.com>,
"viacheslavo@nvidia.com" <viacheslavo@nvidia.com>,
"sthemmin@microsoft.com" <sthemmin@microsoft.com>,
"longli@microsoft.com" <longli@microsoft.com>,
"heinrich.kuhn@corigine.com" <heinrich.kuhn@corigine.com>,
"kirankumark@marvell.com" <kirankumark@marvell.com>,
"mczekaj@marvell.com" <mczekaj@marvell.com>,
"jiawenwu@trustnetic.com" <jiawenwu@trustnetic.com>,
"jianwang@trustnetic.com" <jianwang@trustnetic.com>,
"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
"Xia, Chenbo" <chenbo.xia@intel.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"Yigit, Ferruh" <ferruh.yigit@intel.com>,
"mdr@ashroe.eu" <mdr@ashroe.eu>,
"Jayatheerthan, Jay" <jay.jayatheerthan@intel.com>
Subject: Re: [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array
Date: Tue, 12 Oct 2021 08:47:30 +0300 [thread overview]
Message-ID: <4fae7d1a-22de-f09f-d9a1-9e0b6378b995@oktetlabs.ru> (raw)
In-Reply-To: <DM6PR11MB4491FB3C1DB1D557CE1F06999AB59@DM6PR11MB4491.namprd11.prod.outlook.com>
On 10/12/21 2:06 AM, Ananyev, Konstantin wrote:
>>>>> At queue configure stage always allocate space for maximum possible
>>>>> number (RTE_MAX_QUEUES_PER_PORT) of queue pointers.
>>>>> That will allow 'fast' inline functions (eth_rx_burst, etc.) to refer
>>>>> pointer to internal queue data without extra checking of current number
>>>>> of configured queues.
>>>>> That would help in future to hide rte_eth_dev and related structures.
>>>>> It means that from now on, each ethdev port will always consume:
>>>>> ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT)
>>>>> bytes of memory for its queue pointers.
>>>>> With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port.
>>>>>
>>>>> Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
>>>>> ---
>>>>> lib/ethdev/rte_ethdev.c | 36 +++++++++---------------------------
>>>>> 1 file changed, 9 insertions(+), 27 deletions(-)
>>>>>
>>>>> diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c
>>>>> index ed37f8871b..c8abda6dd7 100644
>>>>> --- a/lib/ethdev/rte_ethdev.c
>>>>> +++ b/lib/ethdev/rte_ethdev.c
>>>>> @@ -897,7 +897,8 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
>>>>>
>>>>> if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */
>>>>> dev->data->rx_queues = rte_zmalloc("ethdev->rx_queues",
>>>>> - sizeof(dev->data->rx_queues[0]) * nb_queues,
>>>>> + sizeof(dev->data->rx_queues[0]) *
>>>>> + RTE_MAX_QUEUES_PER_PORT,
>>>>> RTE_CACHE_LINE_SIZE);
>>>>
>>>> Looking at it I have few questions:
>>>> 1. Why is nb_queues == 0 case kept as an exception? Yes,
>>>> strictly speaking it is not the problem of the patch,
>>>> DPDK will still segfault (non-debug build) if I
>>>> allocate Tx queues only but call rte_eth_rx_burst().
>>>
>>> eth_dev_rx_queue_config(.., nb_queues=0) is used in few places to clean-up things.
>>
>> No, as far as I know. For Tx only application (e.g. traffic generator)
>> it is 100% legal to configure with tx_queues=X, rx_queues=0.
>> The same is for Rx only application (e.g. packet capture).
>
> Yes, that is valid config for sure.
> I just pointed that simply ignoring 'nb_queues' value and
> always allocating space for max possible queues, i.e:
>
> eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
> {
> ....
> - if (dev->data->rx_queues == NULL && nb_queues != 0) { /* first time configuration */
> + if (dev->data->rx_queues == NULL) {
> wouldn't work, as right now nb_queues == 0 has extra special meaning -
> do final cleanup and free dev->data->rx_queues.
> But re-reading the text below, it seems that I misunderstood you
> and it probably wasn't your intention anyway.
>
>>
>>>
>>>> After reading the patch description I thought that
>>>> we're trying to address it.
>>>
>>> We do, though I can't see how we can address it in this patch.
>>> Though it is a good idea - I think I can add extra check in eth_dev_fp_ops_setup()
>>> or around and setup RX function pointers only when dev->data->rx_queues != NULL.
>>> Same for TX.
>>
>> You don't need to care about these pointers, if these arrays are
>> always allocated. See (3) below.
>>
>>>
>>>> 2. Why do we need to allocate memory dynamically?
>>>> Can we just make rx_queues an array of appropriate size?
>>>
>>> Pavan already asked same question.
>>> My answer to him:
>>> Yep we can, and yes it will simplify this peace of code.
>>> The main reason I decided no to do this change now -
>>> it will change layout of the_eth_dev_data structure.
>>> In this series I tried to mininize(/avoid) changes in rte_eth_dev and rte_eth_dev_data,
>>> as much as possible to avoid any unforeseen performance and functional impacts.
>>> If we'll manage to make rte_eth_dev and rte_eth_dev_data private we can in future
>>> consider that one and other changes in rte_eth_dev and rte_eth_dev_data layouts
>>> without worrying about ABI breakage
>>
>> Thanks a lot. Makes sense.
>>
>>>> May be wasting 512K unconditionally is too much.
>>>> 3. If wasting 512K is too much, I'd consider to move
>>>> allocation to eth_dev_get(). If
>>>
>>> Don't understand where 512KB came from.
>>
>> 32 port * 1024 queues * 2 types * 8 pointer size
>> if we allocate as in (2) above.
>>
>>> each ethdev port will always consume:
>>> ((2*sizeof(uintptr_t))* RTE_MAX_QUEUES_PER_PORT)
>>> bytes of memory for its queue pointers.
>>> With RTE_MAX_QUEUES_PER_PORT==1024 (default value) it is 16KB per port.
>>
>> IMHO it will be a bit nicer if queue pointers arrays are allocated
>> on device get if size is fixed. It is just a suggestion. If you
>> disagree, feel free to drop it.
>
> You mean - allocate these arrays somewhere at rte_eth_dev_allocate() path?
Yes, eth_dev_get() mentioned above is called from
rte_eth_dev_allocate().
> That sounds like an interesting idea, but seems too drastic to me at that stage.
Yes, of course, we can address it later.
>
>>
>>>>> if (dev->data->rx_queues == NULL) {
>>>>> dev->data->nb_rx_queues = 0;
>>>>> @@ -908,21 +909,11 @@ eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
>>>>>
>>>>> rxq = dev->data->rx_queues;
>>>>>
>>>>> - for (i = nb_queues; i < old_nb_queues; i++)
>>>>> + for (i = nb_queues; i < old_nb_queues; i++) {
>>>>> (*dev->dev_ops->rx_queue_release)(rxq[i]);
>>>>> - rxq = rte_realloc(rxq, sizeof(rxq[0]) * nb_queues,
>>>>> - RTE_CACHE_LINE_SIZE);
>>>>> - if (rxq == NULL)
>>>>> - return -(ENOMEM);
>>>>> - if (nb_queues > old_nb_queues) {
>>>>> - uint16_t new_qs = nb_queues - old_nb_queues;
>>>>> -
>>>>> - memset(rxq + old_nb_queues, 0,
>>>>> - sizeof(rxq[0]) * new_qs);
>>>>> + rxq[i] = NULL;
>>>>
>>>> It looks like the patch should be rebased on top of
>>>> next-net main because of queue release patches.
>>>>
>>>> [snip]
>
next prev parent reply other threads:[~2021-10-12 5:47 UTC|newest]
Thread overview: 112+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-20 16:28 [dpdk-dev] [RFC 0/7] hide eth dev related structures Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 1/7] eth: move ethdev 'burst' API into separate structure Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 2/7] eth: make drivers to use new API for Rx Konstantin Ananyev
2021-09-06 18:41 ` Ferruh Yigit
2021-09-14 14:28 ` Ananyev, Konstantin
2021-08-20 16:28 ` [dpdk-dev] [RFC 3/7] eth: make drivers to use new API for Tx Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 4/7] eth: make drivers to use new API for Tx prepare Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 5/7] eth: make drivers to use new API to obtain descriptor status Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 6/7] eth: make drivers to use new API for Rx queue count Konstantin Ananyev
2021-08-20 16:28 ` [dpdk-dev] [RFC 7/7] eth: hide eth dev related structures Konstantin Ananyev
2021-08-26 12:37 ` [dpdk-dev] [RFC 0/7] " Jerin Jacob
2021-09-06 18:09 ` Ferruh Yigit
2021-09-14 13:33 ` Ananyev, Konstantin
2021-09-15 9:45 ` Jerin Jacob
2021-09-22 15:08 ` Ananyev, Konstantin
2021-09-27 16:14 ` Jerin Jacob
2021-09-28 9:37 ` Ananyev, Konstantin
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 0/5] " Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 1/5] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 2/5] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-09-23 5:51 ` Wang, Haiyue
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 3/5] ethdev: copy ethdev 'burst' API into separate structure Konstantin Ananyev
2021-09-23 5:58 ` Wang, Haiyue
2021-09-27 18:01 ` Jerin Jacob
2021-09-28 9:42 ` Ananyev, Konstantin
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 4/5] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-09-22 14:09 ` [dpdk-dev] [RFC v2 5/5] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 0/7] " Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-01 16:48 ` Ferruh Yigit
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-01 16:46 ` Ferruh Yigit
2021-10-01 17:40 ` Ananyev, Konstantin
2021-10-04 8:46 ` Ferruh Yigit
2021-10-04 9:20 ` Ananyev, Konstantin
2021-10-04 10:13 ` Ferruh Yigit
2021-10-04 11:17 ` Ananyev, Konstantin
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-01 14:02 ` [dpdk-dev] [PATCH v3 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-01 16:53 ` Ferruh Yigit
2021-10-01 17:04 ` Ferruh Yigit
2021-10-01 17:02 ` [dpdk-dev] [PATCH v3 0/7] " Ferruh Yigit
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 " Konstantin Ananyev
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 1/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-05 12:09 ` Thomas Monjalon
2021-10-05 16:45 ` Ananyev, Konstantin
2021-10-05 16:49 ` Thomas Monjalon
2021-10-05 12:21 ` Thomas Monjalon
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 2/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-04 13:55 ` [dpdk-dev] [PATCH v4 3/7] ethdev: copy ethdev 'fast' API into separate structure Konstantin Ananyev
2021-10-05 13:09 ` Thomas Monjalon
2021-10-05 16:41 ` Ananyev, Konstantin
2021-10-05 16:48 ` Thomas Monjalon
2021-10-05 17:04 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 4/7] ethdev: make burst functions to use new flat array Konstantin Ananyev
2021-10-05 9:54 ` David Marchand
2021-10-05 10:13 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 5/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-05 13:13 ` Thomas Monjalon
2021-10-05 16:35 ` Ananyev, Konstantin
2021-10-05 16:45 ` Thomas Monjalon
2021-10-05 17:12 ` Ananyev, Konstantin
2021-10-05 17:41 ` Thomas Monjalon
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 6/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-05 13:14 ` Thomas Monjalon
2021-10-05 16:21 ` Ananyev, Konstantin
2021-10-04 13:56 ` [dpdk-dev] [PATCH v4 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-05 10:04 ` David Marchand
2021-10-05 10:43 ` Ferruh Yigit
2021-10-05 11:37 ` David Marchand
2021-10-05 15:57 ` Ananyev, Konstantin
2021-10-05 13:24 ` Thomas Monjalon
2021-10-05 16:19 ` Ananyev, Konstantin
2021-10-05 16:25 ` Thomas Monjalon
2021-10-06 16:42 ` [dpdk-dev] [PATCH v4 0/7] " Ali Alnubani
2021-10-06 17:26 ` Ali Alnubani
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 " Konstantin Ananyev
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 1/7] ethdev: remove legacy Rx descriptor done API Konstantin Ananyev
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 2/7] ethdev: allocate max space for internal queue array Konstantin Ananyev
2021-10-11 9:20 ` Andrew Rybchenko
2021-10-11 16:25 ` Ananyev, Konstantin
2021-10-11 17:15 ` Andrew Rybchenko
2021-10-11 23:06 ` Ananyev, Konstantin
2021-10-12 5:47 ` Andrew Rybchenko [this message]
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 3/7] ethdev: change input parameters for rx_queue_count Konstantin Ananyev
2021-10-11 8:06 ` Andrew Rybchenko
2021-10-12 17:59 ` Hyong Youb Kim (hyonkim)
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 4/7] ethdev: copy fast-path API into separate structure Konstantin Ananyev
2021-10-09 12:05 ` fengchengwen
2021-10-11 1:18 ` fengchengwen
2021-10-11 8:39 ` Andrew Rybchenko
2021-10-11 15:24 ` Ananyev, Konstantin
2021-10-11 8:35 ` Andrew Rybchenko
2021-10-11 15:15 ` Ananyev, Konstantin
2021-10-11 8:25 ` Andrew Rybchenko
2021-10-11 16:52 ` Ananyev, Konstantin
2021-10-11 17:22 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 5/7] ethdev: make fast-path functions to use new flat array Konstantin Ananyev
2021-10-11 9:02 ` Andrew Rybchenko
2021-10-11 15:47 ` Ananyev, Konstantin
2021-10-11 17:03 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 6/7] ethdev: add API to retrieve multiple ethernet addresses Konstantin Ananyev
2021-10-11 9:09 ` Andrew Rybchenko
2021-10-07 11:27 ` [dpdk-dev] [PATCH v5 7/7] ethdev: hide eth dev related structures Konstantin Ananyev
2021-10-11 9:20 ` Andrew Rybchenko
2021-10-11 15:54 ` Ananyev, Konstantin
2021-10-11 17:04 ` Andrew Rybchenko
2021-10-08 18:13 ` [dpdk-dev] [PATCH v5 0/7] " Slava Ovsiienko
2021-10-11 9:22 ` Andrew Rybchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4fae7d1a-22de-f09f-d9a1-9e0b6378b995@oktetlabs.ru \
--to=andrew.rybchenko@oktetlabs.ru \
--cc=adwivedi@marvell.com \
--cc=ajit.khaparde@broadcom.com \
--cc=anoobj@marvell.com \
--cc=beilei.xing@intel.com \
--cc=chenbo.xia@intel.com \
--cc=dev@dpdk.org \
--cc=ed.czeck@atomicrules.com \
--cc=ferruh.yigit@intel.com \
--cc=haiyue.wang@intel.com \
--cc=heinrich.kuhn@corigine.com \
--cc=hemant.agrawal@nxp.com \
--cc=humin29@huawei.com \
--cc=hyonkim@cisco.com \
--cc=irusskikh@marvell.com \
--cc=jay.jayatheerthan@intel.com \
--cc=jerinj@marvell.com \
--cc=jianwang@trustnetic.com \
--cc=jiawenwu@trustnetic.com \
--cc=jingjing.wu@intel.com \
--cc=john.miller@atomicrules.com \
--cc=johndale@cisco.com \
--cc=kirankumark@marvell.com \
--cc=konstantin.ananyev@intel.com \
--cc=longli@microsoft.com \
--cc=matan@nvidia.com \
--cc=maxime.coquelin@redhat.com \
--cc=mczekaj@marvell.com \
--cc=mdr@ashroe.eu \
--cc=ndabilpuram@marvell.com \
--cc=oulijun@huawei.com \
--cc=qi.z.zhang@intel.com \
--cc=qiming.yang@intel.com \
--cc=rahul.lakkireddy@chelsio.com \
--cc=sachin.saxena@oss.nxp.com \
--cc=shepard.siegel@atomicrules.com \
--cc=somnath.kotur@broadcom.com \
--cc=sthemmin@microsoft.com \
--cc=thomas@monjalon.net \
--cc=viacheslavo@nvidia.com \
--cc=xiao.w.wang@intel.com \
--cc=xiaoyun.li@intel.com \
--cc=yisen.zhuang@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).