From: Ilya Maximets <i.maximets@samsung.com>
To: Jan Blunck <jblunck@infradead.org>
Cc: Eric Kinzie <ehkinzie@gmail.com>,
Dyasly Sergey <s.dyasly@samsung.com>,
Heetae Ahn <heetae82.ahn@samsung.com>,
stable@dpdk.org, dev@dpdk.org, bruce.richardson@intel.com,
Bernard Iremonger <bernard.iremonger@intel.com>,
Declan Doherty <declan.doherty@intel.com>
Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH] Revert "bonding: use existing enslaved device queues"
Date: Fri, 28 Oct 2016 09:14:17 +0300 [thread overview]
Message-ID: <8507e1f4-328e-a13d-cee2-69db06fe30be@samsung.com> (raw)
In-Reply-To: <03e11be9-45b2-e05f-99f1-8be1bcc05f19@samsung.com>
On 25.10.2016 09:26, Ilya Maximets wrote:
> On 24.10.2016 17:54, Jan Blunck wrote:
>> On Wed, Oct 19, 2016 at 5:47 AM, Ilya Maximets <i.maximets@samsung.com> wrote:
>>> On 18.10.2016 18:19, Jan Blunck wrote:
>>>> On Tue, Oct 18, 2016 at 2:49 PM, Ilya Maximets <i.maximets@samsung.com> wrote:
>>>>> On 18.10.2016 15:28, Jan Blunck wrote:
>>>>>> If the application already configured queues the PMD should not
>>>>>> silently claim ownership and reset them.
>>>>>>
>>>>>> What exactly is the problem when changing MTU? This works fine from
>>>>>> what I can tell.
>>>>>
>>>>> Following scenario leads to APP PANIC:
>>>>>
>>>>> 1. mempool_1 = rte_mempool_create()
>>>>> 2. rte_eth_rx_queue_setup(bond0, ..., mempool_1);
>>>>> 3. rte_eth_dev_start(bond0);
>>>>> 4. mempool_2 = rte_mempool_create();
>>>>> 5. rte_eth_dev_stop(bond0);
>>>>> 6. rte_eth_rx_queue_setup(bond0, ..., mempool_2);
>>>>> 7. rte_eth_dev_start(bond0);
>>>>> * RX queues still use 'mempool_1' because reconfiguration doesn't affect them. *
>>>>> 8. rte_mempool_free(mempool_1);
>>>>> 9. On any rx operation we'll get PANIC because of using freed 'mempool_1':
>>>>> PANIC in rte_mempool_get_ops():
>>>>> assert "(ops_index >= 0) && (ops_index < RTE_MEMPOOL_MAX_OPS_IDX)" failed
>>>>>
>>>>> You may just start OVS 2.6 with DPDK bonding device and attempt to change MTU via 'mtu_request'.
>>>>> Bug is easily reproducible.
>>>>>
>>>>
>>>> I see. I'm not 100% that this is expected to work without leaking the
>>>> driver's queues though. The driver is allowed to do allocations in
>>>> its rx_queue_setup() function that are being freed via
>>>> rx_queue_release() later. But rx_queue_release() is only called if you
>>>> reconfigure the
>>>> device with 0 queues.
>
> It's not true. Drivers usually calls 'rx_queue_release()' inside
> 'rx_queue_setup()' function while reallocating the already allocated
> queue. (See ixgbe driver for example). Also all HW queues are
> usually destroyed inside 'eth_dev_stop()' and reallocated in
> 'eth_dev_start()' or '{rx,tx}_queue_setup()'.
> So, there is no leaks at all.
>
>>>> From what I understand there is no other way to
>>>> reconfigure a device to use another mempool.
>>>>
>>>> But ... even that wouldn't work with the bonding driver right now: the
>>>> bonding master only configures the slaves during startup. I can put
>>>> that on my todo list.
>
> No, bonding master reconfigures new slaves in 'rte_eth_bond_slave_add()'
> if needed.
>
>>>> Coming back to your original problem: changing the MTU for the bond
>>>> does work through rte_eth_dev_set_mtu() for slaves supporting that. In
>>>> any other case you could (re-)configure rxmode.max_rx_pkt_len (and
>>>> jumbo_frame / enable_scatter accordingly). This does work without a
>>>> call to rte_eth_rx_queue_setup().
>>>
>>> Thanks for suggestion, but using of rte_eth_dev_set_mtu() without
>>> reconfiguration will require to have mempools with huge mbufs (9KB)
>>> for all ports from the start. This is unacceptable because leads to
>>> significant performance regressions because of fast cache exhausting.
>>> Also this will require big work to rewrite OVS reconfiguration code
>>> this way.
>>> Anyway, it isn't the MTU only problem. Number of rx/tx descriptors
>>> also can't be changed in runtime.
>>>
>>>
>>> I'm not fully understand what is the use case for this 'reusing' code.
>>> Could you, please, describe situation where this behaviour is necessary?
>>
>> The device that is added to the bond was used before and therefore
>> already has allocated queues. Therefore we reuse the existing queues
>> of the devices instead of borrowing the queues of the bond device. If
>> the slave is removed from the bond again there is no need to allocate
>> the queues again.
>>
>> Hope that clarifies the usecase
>
> So, As I see, this is just an optimization that leads to differently
> configured queues of same device and possible application crash if the
> old configuration doesn't valid any more.
>
> With revert applied in your usecase while adding the device to bond
> it's queues will be automatically reconfigured according to configuration
> of the bond device. If the slave is removed from the bond all its'
> queues will remain as configured by bond. You can reconfigure them if
> needed. I guess, that in your case configuration of slave devices,
> actually, matches configuration of bond device. In that case slave
> will remain in the same state after removing from bond as it was
> before adding.
So, Jan, Declan, what do you think about this?
Best regards, Ilya Maximets.
next prev parent reply other threads:[~2016-10-28 6:14 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-09-07 12:28 [dpdk-stable] " Ilya Maximets
[not found] ` <CGME20160916050359eucas1p22998d07e190781e165082cdd9c917470@eucas1p2.samsung.com>
2016-09-16 5:03 ` Ilya Maximets
2016-10-06 14:32 ` Declan Doherty
2016-10-19 9:55 ` [dpdk-stable] [dpdk-dev] " Bruce Richardson
2016-10-25 13:59 ` Bruce Richardson
2016-10-07 2:02 ` [dpdk-stable] " Eric Kinzie
2016-10-12 13:24 ` Ilya Maximets
2016-10-12 15:24 ` [dpdk-stable] [dpdk-dev] " Bruce Richardson
2016-10-13 23:37 ` Eric Kinzie
2016-10-24 11:02 ` Declan Doherty
2016-10-24 14:51 ` Jan Blunck
2016-10-24 15:07 ` Declan Doherty
2016-10-25 12:57 ` Bruce Richardson
2016-10-25 13:48 ` Declan Doherty
2016-10-25 14:00 ` Bruce Richardson
2016-11-21 11:30 ` Ferruh Yigit
2016-11-21 11:39 ` Jan Blunck
2016-11-21 12:49 ` Ilya Maximets
2016-11-21 13:11 ` Ilya Maximets
2016-11-23 20:35 ` Jan Blunck
2016-10-18 12:28 ` Jan Blunck
2016-10-18 12:49 ` Ilya Maximets
2016-10-18 15:19 ` Jan Blunck
2016-10-19 9:47 ` Ilya Maximets
2016-10-24 14:54 ` Jan Blunck
2016-10-25 6:26 ` Ilya Maximets
2016-10-28 6:14 ` Ilya Maximets [this message]
2016-11-11 9:16 ` Ilya Maximets
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8507e1f4-328e-a13d-cee2-69db06fe30be@samsung.com \
--to=i.maximets@samsung.com \
--cc=bernard.iremonger@intel.com \
--cc=bruce.richardson@intel.com \
--cc=declan.doherty@intel.com \
--cc=dev@dpdk.org \
--cc=ehkinzie@gmail.com \
--cc=heetae82.ahn@samsung.com \
--cc=jblunck@infradead.org \
--cc=s.dyasly@samsung.com \
--cc=stable@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).