From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <nhorman@tuxdriver.com>
Received: from smtp.tuxdriver.com (charlotte.tuxdriver.com [70.61.120.58])
by dpdk.org (Postfix) with ESMTP id 50845B3A5
for <dev@dpdk.org>; Fri, 19 Sep 2014 19:23:25 +0200 (CEST)
Received: from hmsreliant.think-freely.org
([2001:470:8:a08:7aac:c0ff:fec2:933b] helo=localhost)
by smtp.tuxdriver.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.63)
(envelope-from <nhorman@tuxdriver.com>)
id 1XV1zY-0002mM-PE; Fri, 19 Sep 2014 13:29:14 -0400
Date: Fri, 19 Sep 2014 13:29:07 -0400
From: Neil Horman <nhorman@tuxdriver.com>
To: "Wodkowski, PawelX" <pawelx.wodkowski@intel.com>
Message-ID: <20140919172907.GE12897@hmsreliant.think-freely.org>
References: <1410963713-13837-1-git-send-email-pawelx.wodkowski@intel.com>
<1410963713-13837-3-git-send-email-pawelx.wodkowski@intel.com>
<20140917151304.GD4213@localhost.localdomain>
<F6F2A6264E145F47A18AB6DF8E87425D12B24CC2@IRSMSX102.ger.corp.intel.com>
<20140918160234.GJ20389@hmsreliant.think-freely.org>
<F6F2A6264E145F47A18AB6DF8E87425D12B2513B@IRSMSX102.ger.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <F6F2A6264E145F47A18AB6DF8E87425D12B2513B@IRSMSX102.ger.corp.intel.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-Spam-Score: -2.9 (--)
X-Spam-Status: No
Cc: "dev@dpdk.org" <dev@dpdk.org>, "Jastrzebski,
MichalX K" <michalx.k.jastrzebski@intel.com>
Subject: Re: [dpdk-dev] [PATCH 2/2] bond: add mode 4 support
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 19 Sep 2014 17:23:25 -0000
On Fri, Sep 19, 2014 at 12:47:35PM +0000, Wodkowski, PawelX wrote:
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Thursday, September 18, 2014 18:03
> > To: Wodkowski, PawelX
> > Cc: dev@dpdk.org; Jastrzebski, MichalX K; Doherty, Declan
> > Subject: Re: [dpdk-dev] [PATCH 2/2] bond: add mode 4 support
> >
> > On Thu, Sep 18, 2014 at 08:07:31AM +0000, Wodkowski, PawelX wrote:
> > > > > +int
> > > > > +bond_mode_8023ad_deactivate_slave(struct rte_eth_dev *bond_dev,
> > > > > + uint8_t slave_pos)
> > > > > +{
> > > > > + struct bond_dev_private *internals = bond_dev->data->dev_private;
> > > > > + struct mode8023ad_data *data = &internals->mode4;
> > > > > + struct port *port;
> > > > > + uint8_t i;
> > > > > +
> > > > > + bond_mode_8023ad_stop(bond_dev);
> > > > > +
> > > > > + /* Exclude slave from transmit policy. If this slave is an aggregator
> > > > > + * make all aggregated slaves unselected to force sellection logic
> > > > > + * to select suitable aggregator for this port */
> > > > > + for (i = 0; i < internals->active_slave_count; i++) {
> > > > > + port = &data->port_list[slave_pos];
> > > > > + if (port->used_agregator_idx == slave_pos) {
> > > > > + port->selected = UNSELECTED;
> > > > > + port->actor_state &= ~(STATE_SYNCHRONIZATION |
> > > > STATE_DISTRIBUTING |
> > > > > + STATE_COLLECTING);
> > > > > +
> > > > > + /* Use default aggregator */
> > > > > + port->used_agregator_idx = i;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > + port = &data->port_list[slave_pos];
> > > > > + timer_cancel(&port->current_while_timer);
> > > > > + timer_cancel(&port->periodic_timer);
> > > > > + timer_cancel(&port->wait_while_timer);
> > > > > + timer_cancel(&port->tx_machine_timer);
> > > > > +
> > > > These all seem rather racy. Alarm callbacks are executed with the alarm list
> > > > locks not held. So there is every possibility that you could execute these (or
> > > > any timer_cancel calls in this PMD in parallel with the internal state machine
> > > > timer callback, and leave either with a corrupted timer list (resulting from a
> > > > double free between here, and the actual callback site),
> > >
> > > I don't think so. Yes, callbacks are executed with alarm list locks not held, but
> > > this is not the issue because access to list itself is guarded by lock and
> > > ap->executing variable. So list will not be trashed. Check source of
> > > eal_alarm_callback(), rte_eal_alarm_set() and rte_eal_alarm_cancel().
> > >
> > Yes, you're right, the list is probably safe wht the executing bit.
> >
> > > > or a timer that is
> > > > actually still pending when a slave is removed.
> > > >
> > > This is not the issue also, but problem might be similar. I assumed that alarms
> > > are atomic but when I looked at rte alarms closer I saw a race condition
> > > between and rte_eal_alarm_cancel() from bond_mode_8023ad_stop()
> > > and rte_eal_alarm_set() from state machines callback. This need to be
> > > reworked in some way.
> >
> > Yes, this is what I was referring to:
> >
> > CPU0 CPU1
> > rte_eal_alarm_callback bond_8023ad_deactivate_slave
> > -bond_8023_ad_periodic_cb timer_cancel
> > timer_set
> >
> > If those timer functions operate on the same timer, the result is that you can
> > leave the stop/deactivate slave paths with a timer function for that slave still
> > pending. The bonding mode needs some internal state to serialize those
> > operations and determine if the timer should be reactivated.
> >
> > Neil
>
> I did rethink the issue and problem is much simpler than it looks like. I did the
> following:
> 1. Change internal state machine alarms to use rte_rdtsc(). This makes all
> mode 4 internal timer_*() function not affected by any race condition.
> 2. Do a busy loop when canceling main callback timer until cancel is successfull.
> This should do the trick about race condition. Do you agree?
>
I think that will work, but I believe you're making it more complicated (and
less reusable) than it needs to be. What I think you really need to do is
create a new rte api call, rte_eal_alarm_cancel_sync (something like the
equivalent of del_timer_sync in linux, that wraps up the
while(rte_eal_alarm_cancel(...) == 0) {rte_pause} in its own function (so other
call sites can use it, as I don't think this is an uncommon problem), Then just
create a bonding-internal state flag to signal the periodic callback that it
shouldn't re-arm the timer. That way all you have to do is set the flag, and
call rte_eal_alarm_cancel_sync, and you're done. And other applications will be
able to handle this common type of operation as well
Neil