From: Ivan Boule <ivan.boule@6wind.com>
To: "Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
"Richardson, Bruce" <bruce.richardson@intel.com>
Cc: Thomas Monjalon <thomas.monjalon@6wind.com>,
"Pattan, Reshma" <reshma.pattan@intel.com>,
"dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists
Date: Wed, 15 Jun 2016 17:33:18 +0200 [thread overview]
Message-ID: <5761753E.5030805@6wind.com> (raw)
In-Reply-To: <2601191342CEEE43887BDE71AB97725836B7194A@irsmsx105.ger.corp.intel.com>
On 06/15/2016 04:27 PM, Ananyev, Konstantin wrote:
>
>
>> -----Original Message-----
>> From: Richardson, Bruce
>> Sent: Wednesday, June 15, 2016 3:22 PM
>> To: Ananyev, Konstantin
>> Cc: Ivan Boule; Thomas Monjalon; Pattan, Reshma; dev@dpdk.org
>> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists
>>
>> On Wed, Jun 15, 2016 at 03:20:27PM +0100, Ananyev, Konstantin wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Ivan Boule [mailto:ivan.boule@6wind.com]
>>>> Sent: Wednesday, June 15, 2016 3:07 PM
>>>> To: Richardson, Bruce; Ananyev, Konstantin
>>>> Cc: Thomas Monjalon; Pattan, Reshma; dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists
>>>>
>>>> On 06/15/2016 03:29 PM, Bruce Richardson wrote:
>>>>> On Wed, Jun 15, 2016 at 12:40:16PM +0000, Ananyev, Konstantin wrote:
>>>>>> Hi Ivan,
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Ivan Boule [mailto:ivan.boule@6wind.com]
>>>>>>> Sent: Wednesday, June 15, 2016 1:15 PM
>>>>>>> To: Thomas Monjalon; Ananyev, Konstantin
>>>>>>> Cc: Pattan, Reshma; dev@dpdk.org
>>>>>>> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists
>>>>>>>
>>>>>>> On 06/15/2016 10:48 AM, Thomas Monjalon wrote:
>>>>>>>
>>>>>>>>>
>>>>>>>>>> I think the read access would need locking but we do not want it
>>>>>>>>>> in fast path.
>>>>>>>>>
>>>>>>>>> I don't think it would be needed.
>>>>>>>>> As I said - read/write interaction didn't change from what we have right now.
>>>>>>>>> But if you have some particular scenario in mind that you believe would cause
>>>>>>>>> a race condition - please speak up.
>>>>>>>>
>>>>>>>> If we add/remove a callback during a burst? Is it possible that the next
>>>>>>>> pointer would have a wrong value leading to a crash?
>>>>>>>> Maybe we need a comment to state that we should not alter burst
>>>>>>>> callbacks while running burst functions.
>>>>>>>>
>>>>>>>
>>>>>>> Hi Reshma,
>>>>>>>
>>>>>>> You claim that the "rte_eth_rx_cb_lock" does not need to be hold in the
>>>>>>> function "rte_eth_rx_burst()" while parsing the "post_rx_burst_cbs" list
>>>>>>> of RX callbacks associated with the polled RX queue to safely remove RX
>>>>>>> callback(s) in parallel.
>>>>>>> The problem is not [only] with the setting and the loading of "cb->next"
>>>>>>> that you assume to be atomic operations, which is certainly true on most
>>>>>>> CPUs.
>>>>>>> I see the 2 important following issues:
>>>>>>>
>>>>>>> 1) the "rte_eth_rxtx_callback" data structure associated with a removed
>>>>>>> RX callback could still be accessed in the callback parsing loop of the
>>>>>>> function "rte_eth_rx_burst()" after having been freed in parallel.
>>>>>>>
>>>>>>> BUT SUCH A BAD SITUATION WILL NOT CURRENTLY HAPPEN, THANKS TO THE NICE
>>>>>>> MEMORY LEAK BUG in the function "rte_eth_remove_rx_callback()" that
>>>>>>> does not free the "rte_eth_rxtx_callback" data structure associated with
>>>>>>> the removed callback !
>>>>>>
>>>>>> Yes, though it is documented behaviour, someone can probably
>>>>>> refer it as a feature, not a bug ;)
>>>>>>
>>>>>
>>>>> +1
>>>>> This is definitely not a bug, this is absolutely by design. One may argue with
>>>>> the design, but it was done for a definite reason, so as to avoid paying the
>>>>> penalty of having locks. It pushes more responsibility onto the app, but it
>>>>> does allow the app to choose the best solution for managing the freeing of
>>>>> memory for its situation. The alternative is to force all apps to pay the cost
>>>>> of having locks, even if better options for freeing the memory are available.
>>>>>
>>>>> /Bruce
>>>>>
>>>>
>>>> -1 (not to say 0xFFFFFFFF)
>>>>
>>>> This is definitely an API design bug !
>>>> I would say that if you don't know how to free a resource that you
>>>> allocate, it is very likely that you are wrong allocating it.
>>>> And this is exactly what happens here with RX/TX callback data structures.
>>>> This problem can easily be addressed by just changing the API as follows:
>>>>
>>>> Change
>>>> void *
>>>> rte_eth_add_rx_callback(uint8_t port_id, uint16_t queue_id,
>>>> rte_rx_callback_fn fn, void *user_param)
>>>>
>>>> to
>>>> int
>>>> rte_eth_add_rx_callback(uint8_t port_id, uint16_t queue_id,
>>>> struct rte_eth_rxtx_callback *cb)
>>>>
>>>> In addition of solving the problem, this approach makes the API
>>>> consistent and let the application allocate "rte_eth_rxtx_callback" data
>>>> structures through any appropriate mean.
>>>
>>> That might make API a bit cleaner, but I don't see how it fixes the generic problem:
>>> there is still no way to know by the upper layer when it is safe to free/re-use
>>> removed callback, but to make sure that all IO on that queue is stopped
>>> (I.E. some external synchronisation around the queue).
>>
>> Actually, it allows other, more creative solutions, like an app having a
>> statically allocated set/pool of callback structures that it doesn't need to
>> allocate or free at all.
>
> I understand that, and as I said I am not against that change.
> But it doesn't solve the problem in general.
> Ok, if you have a static object, you wouldn't need to call free() for it,
> but you still want to re-use it after remove_cb() has finished, right?
> So there is no actual difference - the main question is:
> at what point after remove_cb() it is safe to modify it's contents.
> Konstantin
>
>>
>> /Bruce
>
Hi Bruce/Konstantin
There is no need to use locks to ensure that a RX callback being removed
is not executed/invoked and will never be invoked again.
This issue can be easily addressed at application level through the
common synchronization mechanism that consists in having the control
thread of the application that manages the adding/removal of RX
callbacks to:
1) Ask the packet processing function of the application to stop
invoking the function rte_eth_rx_burst() on the target RX queue.
2) Once the packet processing function acked it stopped polling the
target RX queue, safely remove the RX callback and free whatever
resource needs to be freed.
3) Enable the packet processing function of the application to invoke
again the function rte_eth_rx_burst() on the target RX queue.
Enjoy :-)
Ivan
--
Ivan Boule
6WIND Development Engineer
next prev parent reply other threads:[~2016-06-15 15:33 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1465487895-5870-1-git-send-email-reshma.pattan@intel.com>
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 0/8] add packet capture framework Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 1/8] librte_ether: protect add/remove of rxtx callbacks with spinlocks Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 2/8] librte_ether: add new api rte_eth_add_first_rx_callback Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 3/8] librte_ether: add new fields to rte_eth_dev_info struct Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 4/8] librte_ether: make rte_eth_dev_get_port_by_name rte_eth_dev_get_name_by_port public Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 5/8] lib/librte_pdump: add new library for packet capturing support Reshma Pattan
2016-06-10 18:48 ` Aaron Conole
2016-06-10 22:14 ` Pattan, Reshma
2016-06-13 13:28 ` Aaron Conole
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 6/8] app/pdump: add pdump tool for packet capturing Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 7/8] app/test-pmd: add pdump initialization uninitialization Reshma Pattan
2016-06-10 16:18 ` [dpdk-dev] [PATCH v8 8/8] doc: update doc for packet capture framework Reshma Pattan
2016-06-10 23:23 ` [dpdk-dev] [PATCH v8 0/8] add " Neil Horman
2016-06-13 8:47 ` Pattan, Reshma
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 " Reshma Pattan
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists Reshma Pattan
2016-06-14 19:59 ` Thomas Monjalon
2016-06-15 5:30 ` Pattan, Reshma
2016-06-15 8:19 ` Thomas Monjalon
2016-06-15 8:37 ` Ananyev, Konstantin
2016-06-15 8:48 ` Thomas Monjalon
2016-06-15 9:54 ` Ananyev, Konstantin
2016-06-15 11:17 ` Thomas Monjalon
2016-06-15 13:49 ` Thomas Monjalon
2016-06-15 12:15 ` Ivan Boule
2016-06-15 12:40 ` Ananyev, Konstantin
2016-06-15 13:29 ` Bruce Richardson
2016-06-15 14:07 ` Ivan Boule
2016-06-15 14:19 ` Bruce Richardson
2016-06-15 14:20 ` Ananyev, Konstantin
2016-06-15 14:22 ` Bruce Richardson
2016-06-15 14:27 ` Ananyev, Konstantin
2016-06-15 15:33 ` Ivan Boule [this message]
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 2/8] ethdev: add new api to add Rx callback as head of the list Reshma Pattan
2016-06-14 20:01 ` Thomas Monjalon
2016-06-14 21:43 ` Pattan, Reshma
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 3/8] ethdev: add new fields to ethdev info struct Reshma Pattan
2016-06-14 20:10 ` Thomas Monjalon
2016-06-14 21:57 ` Pattan, Reshma
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 4/8] ethdev: make get port by name and get name by port public Reshma Pattan
2016-06-14 20:23 ` Thomas Monjalon
2016-06-14 21:55 ` Pattan, Reshma
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 5/8] pdump: add new library for packet capturing support Reshma Pattan
2016-06-14 20:28 ` Thomas Monjalon
2016-06-14 21:59 ` Pattan, Reshma
2016-06-15 9:05 ` Mcnamara, John
2016-06-15 9:32 ` Thomas Monjalon
2016-06-15 9:43 ` Bruce Richardson
2016-06-15 15:44 ` Mcnamara, John
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 6/8] app/pdump: add pdump tool for packet capturing Reshma Pattan
2016-06-14 19:56 ` Thomas Monjalon
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 7/8] app/testpmd: add pdump initialization uninitialization Reshma Pattan
2016-06-14 9:38 ` [dpdk-dev] [PATCH v9 8/8] doc: update doc for packet capture framework Reshma Pattan
2016-06-14 20:41 ` Thomas Monjalon
2016-06-15 5:44 ` Pattan, Reshma
2016-06-15 8:24 ` Thomas Monjalon
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 0/7] add " Reshma Pattan
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 1/7] ethdev: use locks to protect Rx/Tx callback lists Reshma Pattan
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 2/7] ethdev: add new api to add Rx callback as head of the list Reshma Pattan
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 3/7] ethdev: add new fields to ethdev info struct Reshma Pattan
2016-06-16 19:14 ` Thomas Monjalon
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 4/7] ethdev: make get port by name and get name by port public Reshma Pattan
2016-06-16 20:27 ` Thomas Monjalon
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 5/7] pdump: add new library for packet capturing support Reshma Pattan
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 6/7] app/pdump: add pdump tool for packet capturing Reshma Pattan
2016-06-15 14:06 ` [dpdk-dev] [PATCH v10 7/7] app/testpmd: add pdump initialization uninitialization Reshma Pattan
2016-06-16 21:55 ` [dpdk-dev] [PATCH v10 0/7] add packet capture framework Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5761753E.5030805@6wind.com \
--to=ivan.boule@6wind.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=konstantin.ananyev@intel.com \
--cc=reshma.pattan@intel.com \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).