From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f172.google.com (mail-lb0-f172.google.com [209.85.217.172]) by dpdk.org (Postfix) with ESMTP id 00B506C98 for ; Wed, 15 Jun 2016 17:33:21 +0200 (CEST) Received: by mail-lb0-f172.google.com with SMTP id ak10so3828557lbc.3 for ; Wed, 15 Jun 2016 08:33:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=HB3a31RjEXrwllULif5A8N65VIqenpFAYN7RmGdcAXE=; b=sfi59aOLVzowatwmbTbtXVR7FddgvNM94dN6GAhrYiuVy9Kk1TH4d7C8zw+HDSIv8u T2X2sGaSCKv8aEF+uvKuPuHOXjorMIvxcrgYKWTBROsEc583JmeA97wWmMxAkWzv206n ia5wN0URggoDtecywMyLq2y0B0m7j9Kqf3NT7wv/meOA5VG3A1caL8VChdUVN6dKdgVD JiaE269aGPALsDsEnD6EpZYL9bDq9vTit21iC8ZuOMoN7+KzVMF5I+cvy2Qcasl3fpIa iydYE5pQDY2/5Eg48MU1BqguEygDwfyw6fIgtwVrOMms/f7NjQ26m8IF/PVF3lnMjVHq 5qxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=HB3a31RjEXrwllULif5A8N65VIqenpFAYN7RmGdcAXE=; b=dehNBjoRwcI9y0LX0d63QQC90MVWfTwnWzex9qHBE4FyRWFz/9cCABWGGelY506DfR 0WJRkIv8jBRaYAEd7iVy+O9pfmWKGamIznc2sH1onkhVizCWenFZVd3fS+MTyvlf+Jf5 oqdypmNjuS9m3qm8o3qKvK+icvJ/hWelhyJE37Su+wd4Izi2L4PBJR2Ekjbc3GsGkuFT gBPhSEJAa4v5EjXzEgxJ+RLHC2FBOfPzQNPZN4hPIYg8vp72WSg6YCB8U8ETjc5EAnX3 MQjp+X+bo8iZTaAua9qE4yDKpygpyfaOatzhIFo/oumtr+dw9b87tFxW0VoUQV9LQK4L HuHg== X-Gm-Message-State: ALyK8tKZgJ0lftkiVfKgaN73nbimcQzWQf9AaU/xjtt6+VrIUHerLszrXovW7SQnD6P7io1g X-Received: by 10.28.67.69 with SMTP id q66mr5298449wma.81.1466004800292; Wed, 15 Jun 2016 08:33:20 -0700 (PDT) Received: from [10.16.0.189] (guy78-3-82-239-227-177.fbx.proxad.net. [82.239.227.177]) by smtp.gmail.com with ESMTPSA id db6sm38979103wjb.2.2016.06.15.08.33.18 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 15 Jun 2016 08:33:18 -0700 (PDT) To: "Ananyev, Konstantin" , "Richardson, Bruce" References: <1465575534-23605-1-git-send-email-reshma.pattan@intel.com> <10886152.VH5xYhdqG2@xps13> <2601191342CEEE43887BDE71AB97725836B714ED@irsmsx105.ger.corp.intel.com> <2907169.iIEIeOfXh7@xps13> <576146CD.8060008@6wind.com> <2601191342CEEE43887BDE71AB97725836B717B0@irsmsx105.ger.corp.intel.com> <20160615132928.GA11536@bricha3-MOBL3> <57616118.7080806@6wind.com> <2601191342CEEE43887BDE71AB97725836B7191F@irsmsx105.ger.corp.intel.com> <20160615142223.GC11536@bricha3-MOBL3> <2601191342CEEE43887BDE71AB97725836B7194A@irsmsx105.ger.corp.intel.com> Cc: Thomas Monjalon , "Pattan, Reshma" , "dev@dpdk.org" From: Ivan Boule Message-ID: <5761753E.5030805@6wind.com> Date: Wed, 15 Jun 2016 17:33:18 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Icedove/38.8.0 MIME-Version: 1.0 In-Reply-To: <2601191342CEEE43887BDE71AB97725836B7194A@irsmsx105.ger.corp.intel.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jun 2016 15:33:21 -0000 On 06/15/2016 04:27 PM, Ananyev, Konstantin wrote: > > >> -----Original Message----- >> From: Richardson, Bruce >> Sent: Wednesday, June 15, 2016 3:22 PM >> To: Ananyev, Konstantin >> Cc: Ivan Boule; Thomas Monjalon; Pattan, Reshma; dev@dpdk.org >> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists >> >> On Wed, Jun 15, 2016 at 03:20:27PM +0100, Ananyev, Konstantin wrote: >>> >>> >>>> -----Original Message----- >>>> From: Ivan Boule [mailto:ivan.boule@6wind.com] >>>> Sent: Wednesday, June 15, 2016 3:07 PM >>>> To: Richardson, Bruce; Ananyev, Konstantin >>>> Cc: Thomas Monjalon; Pattan, Reshma; dev@dpdk.org >>>> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists >>>> >>>> On 06/15/2016 03:29 PM, Bruce Richardson wrote: >>>>> On Wed, Jun 15, 2016 at 12:40:16PM +0000, Ananyev, Konstantin wrote: >>>>>> Hi Ivan, >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Ivan Boule [mailto:ivan.boule@6wind.com] >>>>>>> Sent: Wednesday, June 15, 2016 1:15 PM >>>>>>> To: Thomas Monjalon; Ananyev, Konstantin >>>>>>> Cc: Pattan, Reshma; dev@dpdk.org >>>>>>> Subject: Re: [dpdk-dev] [PATCH v9 1/8] ethdev: use locks to protect Rx/Tx callback lists >>>>>>> >>>>>>> On 06/15/2016 10:48 AM, Thomas Monjalon wrote: >>>>>>> >>>>>>>>> >>>>>>>>>> I think the read access would need locking but we do not want it >>>>>>>>>> in fast path. >>>>>>>>> >>>>>>>>> I don't think it would be needed. >>>>>>>>> As I said - read/write interaction didn't change from what we have right now. >>>>>>>>> But if you have some particular scenario in mind that you believe would cause >>>>>>>>> a race condition - please speak up. >>>>>>>> >>>>>>>> If we add/remove a callback during a burst? Is it possible that the next >>>>>>>> pointer would have a wrong value leading to a crash? >>>>>>>> Maybe we need a comment to state that we should not alter burst >>>>>>>> callbacks while running burst functions. >>>>>>>> >>>>>>> >>>>>>> Hi Reshma, >>>>>>> >>>>>>> You claim that the "rte_eth_rx_cb_lock" does not need to be hold in the >>>>>>> function "rte_eth_rx_burst()" while parsing the "post_rx_burst_cbs" list >>>>>>> of RX callbacks associated with the polled RX queue to safely remove RX >>>>>>> callback(s) in parallel. >>>>>>> The problem is not [only] with the setting and the loading of "cb->next" >>>>>>> that you assume to be atomic operations, which is certainly true on most >>>>>>> CPUs. >>>>>>> I see the 2 important following issues: >>>>>>> >>>>>>> 1) the "rte_eth_rxtx_callback" data structure associated with a removed >>>>>>> RX callback could still be accessed in the callback parsing loop of the >>>>>>> function "rte_eth_rx_burst()" after having been freed in parallel. >>>>>>> >>>>>>> BUT SUCH A BAD SITUATION WILL NOT CURRENTLY HAPPEN, THANKS TO THE NICE >>>>>>> MEMORY LEAK BUG in the function "rte_eth_remove_rx_callback()" that >>>>>>> does not free the "rte_eth_rxtx_callback" data structure associated with >>>>>>> the removed callback ! >>>>>> >>>>>> Yes, though it is documented behaviour, someone can probably >>>>>> refer it as a feature, not a bug ;) >>>>>> >>>>> >>>>> +1 >>>>> This is definitely not a bug, this is absolutely by design. One may argue with >>>>> the design, but it was done for a definite reason, so as to avoid paying the >>>>> penalty of having locks. It pushes more responsibility onto the app, but it >>>>> does allow the app to choose the best solution for managing the freeing of >>>>> memory for its situation. The alternative is to force all apps to pay the cost >>>>> of having locks, even if better options for freeing the memory are available. >>>>> >>>>> /Bruce >>>>> >>>> >>>> -1 (not to say 0xFFFFFFFF) >>>> >>>> This is definitely an API design bug ! >>>> I would say that if you don't know how to free a resource that you >>>> allocate, it is very likely that you are wrong allocating it. >>>> And this is exactly what happens here with RX/TX callback data structures. >>>> This problem can easily be addressed by just changing the API as follows: >>>> >>>> Change >>>> void * >>>> rte_eth_add_rx_callback(uint8_t port_id, uint16_t queue_id, >>>> rte_rx_callback_fn fn, void *user_param) >>>> >>>> to >>>> int >>>> rte_eth_add_rx_callback(uint8_t port_id, uint16_t queue_id, >>>> struct rte_eth_rxtx_callback *cb) >>>> >>>> In addition of solving the problem, this approach makes the API >>>> consistent and let the application allocate "rte_eth_rxtx_callback" data >>>> structures through any appropriate mean. >>> >>> That might make API a bit cleaner, but I don't see how it fixes the generic problem: >>> there is still no way to know by the upper layer when it is safe to free/re-use >>> removed callback, but to make sure that all IO on that queue is stopped >>> (I.E. some external synchronisation around the queue). >> >> Actually, it allows other, more creative solutions, like an app having a >> statically allocated set/pool of callback structures that it doesn't need to >> allocate or free at all. > > I understand that, and as I said I am not against that change. > But it doesn't solve the problem in general. > Ok, if you have a static object, you wouldn't need to call free() for it, > but you still want to re-use it after remove_cb() has finished, right? > So there is no actual difference - the main question is: > at what point after remove_cb() it is safe to modify it's contents. > Konstantin > >> >> /Bruce > Hi Bruce/Konstantin There is no need to use locks to ensure that a RX callback being removed is not executed/invoked and will never be invoked again. This issue can be easily addressed at application level through the common synchronization mechanism that consists in having the control thread of the application that manages the adding/removal of RX callbacks to: 1) Ask the packet processing function of the application to stop invoking the function rte_eth_rx_burst() on the target RX queue. 2) Once the packet processing function acked it stopped polling the target RX queue, safely remove the RX callback and free whatever resource needs to be freed. 3) Enable the packet processing function of the application to invoke again the function rte_eth_rx_burst() on the target RX queue. Enjoy :-) Ivan -- Ivan Boule 6WIND Development Engineer