DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Zhang, Qi Z" <qi.z.zhang@intel.com>
To: Ilya Maximets <i.maximets@samsung.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "Lu, Wenzhuo" <wenzhuo.lu@intel.com>,
	"Ananyev, Konstantin" <konstantin.ananyev@intel.com>,
	Laurent Hardy <laurent.hardy@6wind.com>,
	"Dai, Wei" <wei.dai@intel.com>,
	"stable@dpdk.org" <stable@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: fix busy polling while fiber link update
Date: Wed, 12 Sep 2018 08:28:49 +0000	[thread overview]
Message-ID: <039ED4275CED7440929022BC67E7061153284408@SHSMSX103.ccr.corp.intel.com> (raw)
In-Reply-To: <20180912080338eucas1p1bfdacb30aa969cd607ccf99f64d6bf80~TmFveK2Dy2157121571eucas1p1U@eucas1p1.samsung.com>



> -----Original Message-----
> From: Ilya Maximets [mailto:i.maximets@samsung.com]
> Sent: Wednesday, September 12, 2018 4:05 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org
> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; Laurent Hardy
> <laurent.hardy@6wind.com>; Dai, Wei <wei.dai@intel.com>;
> stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: fix busy polling while fiber link
> update
> 
> On 12.09.2018 09:49, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Ilya Maximets [mailto:i.maximets@samsung.com]
> >> Sent: Monday, September 10, 2018 11:09 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org
> >> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>; Ananyev, Konstantin
> >> <konstantin.ananyev@intel.com>; Laurent Hardy
> >> <laurent.hardy@6wind.com>; Dai, Wei <wei.dai@intel.com>;
> >> stable@dpdk.org
> >> Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: fix busy polling while
> >> fiber link update
> >>
> >> On 04.09.2018 09:08, Zhang, Qi Z wrote:
> >>> Hi Ilya:
> >>>
> >>>> -----Original Message-----
> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ilya Maximets
> >>>> Sent: Friday, August 31, 2018 8:40 PM
> >>>> To: dev@dpdk.org
> >>>> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>; Ananyev, Konstantin
> >>>> <konstantin.ananyev@intel.com>; Laurent Hardy
> >>>> <laurent.hardy@6wind.com>; Dai, Wei <wei.dai@intel.com>; Ilya
> >>>> Maximets <i.maximets@samsung.com>; stable@dpdk.org
> >>>> Subject: [dpdk-dev] [PATCH] net/ixgbe: fix busy polling while fiber
> >>>> link update
> >>>>
> >>>> If the multispeed fiber link is in DOWN state, ixgbe_setup_link
> >>>> could take around a second of busy polling. This is highly
> >>>> inconvenient for the case where single thread periodically checks the link
> statuses.
> >>>> For example, OVS main thread periodically updates the link statuses
> >>>> and hangs for a really long time busy waiting on ixgbe_setup_link()
> >>>> for a DOWN fiber ports. For case with 3 down ports it hangs for a 3
> >>>> seconds and unable to do anything including packet processing.
> >>>> Fix that by shifting that workaround to a separate thread by alarm
> >>>> handler that will try to set up link if it is DOWN.
> >>>
> >>> Does that mean we will block the interrupt thread for 3 seconds?
> >>
> >> Three times for one second. Other work could be scheduled between.
> >> IMHO, it's much better than blocking usual caller for 3 seconds.
> >>
> >>> Also, can we guarantee there will not be any race condition if we
> >>> call
> >> ixgbe_setup_link at another thread, the base code API is not assumed
> >> to be thread-safe as I know.
> >>
> >> The only user of 'ixgbe_setup_link' is 'ixgbe_dev_start', but it
> >> could be called only if device stopped. 'ixgbe_dev_stop' cancels the alarm.
> >> Race with 'link_update' avoided by 'IXGBE_FLAG_NEED_LINK_CONFIG' flag.
> >
> > I guess, it' not only about when ixgb_setup_link race with itself, but also
> when it race with other APIs.
> > Also the concern is, even in current version, we can prove there is no issue,
> how can we guarantee we are safe for future base code update? It's not
> designed as thread-safe.
> > For my option, the change is risky.
> 
> In current implementation interrupt handler already calls the
> 'ixgbe_dev_link_update' which subsequently calls 'ixgbe_setup_link'
> in our case if LSC interrupts enabled. So, my change makes the driver even
> safer by moving 'ixgbe_setup_link' to the same interrupt thread.
> Otherwise two threads (interrupts handler and the link status checking thread)
> could call 'ixgbe_setup_link' simultaneously.

Ok, you are right, seems the concern I have is already exist , your patch does not introduce new issue.
So I have no objection if this will fix some issue.
But let's check if any ixgbe experts will comment.

Regards
Qi

> 
> >
> > Btw, since ixgbe support LSC, it is not necessary for "single thread
> periodically checks the link statuses", right?
> 
> In current implementation it will take at least 5 seconds (4 + 1) for the interrupt
> handler to detect DOWN link state for ixgbe multispeed fiber. This is too much
> for many real world cases.
> 
> >
> >>
> >>>
> >>> Regards
> >>> Qi
> >>>
> >>>>
> >>>> Fixes: c12d22f65b13 ("net/ixgbe: ensure link status is updated")
> >>>> CC: stable@dpdk.org
> >>>>
> >>>> Signed-off-by: Ilya Maximets <i.maximets@samsung.com>
> >>>> ---
> >>>>  drivers/net/ixgbe/ixgbe_ethdev.c | 43
> >>>> ++++++++++++++++++++++++--------
> >>>>  1 file changed, 32 insertions(+), 11 deletions(-)
> >>>>
> >>>> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> >>>> b/drivers/net/ixgbe/ixgbe_ethdev.c
> >>>> index 26b192737..a33b9a6e8 100644
> >>>> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> >>>> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> >>>> @@ -221,6 +221,8 @@ static int ixgbe_dev_interrupt_action(struct
> >>>> rte_eth_dev *dev,
> >>>>  				      struct rte_intr_handle *handle);  static void
> >>>> ixgbe_dev_interrupt_handler(void *param);  static void
> >>>> ixgbe_dev_interrupt_delayed_handler(void *param);
> >>>> +static void ixgbe_dev_setup_link_alarm_handler(void *param);
> >>>> +
> >>>>  static int ixgbe_add_rar(struct rte_eth_dev *dev, struct
> >>>> ether_addr *mac_addr,
> >>>>  			 uint32_t index, uint32_t pool);  static void
> >>>> ixgbe_remove_rar(struct rte_eth_dev *dev, uint32_t index); @@
> >>>> -2791,6 +2793,8 @@ ixgbe_dev_stop(struct rte_eth_dev *dev)
> >>>>
> >>>>  	PMD_INIT_FUNC_TRACE();
> >>>>
> >>>> +	rte_eal_alarm_cancel(ixgbe_dev_setup_link_alarm_handler, dev);
> >>>> +
> >>>>  	/* disable interrupts */
> >>>>  	ixgbe_disable_intr(hw);
> >>>>
> >>>> @@ -3969,6 +3973,25 @@ ixgbevf_check_link(struct ixgbe_hw *hw,
> >>>> ixgbe_link_speed *speed,
> >>>>  	return ret_val;
> >>>>  }
> >>>>
> >>>> +static void
> >>>> +ixgbe_dev_setup_link_alarm_handler(void *param) {
> >>>> +	struct rte_eth_dev *dev = (struct rte_eth_dev *)param;
> >>>> +	struct ixgbe_hw *hw =
> >>>> IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> >>>> +	struct ixgbe_interrupt *intr =
> >>>> +		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> >>>> +	u32 speed;
> >>>> +	bool autoneg = false;
> >>>> +
> >>>> +	speed = hw->phy.autoneg_advertised;
> >>>> +	if (!speed)
> >>>> +		ixgbe_get_link_capabilities(hw, &speed, &autoneg);
> >>>> +
> >>>> +	ixgbe_setup_link(hw, speed, true);
> >>>> +
> >>>> +	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG; }
> >>>> +
> >>>>  /* return 0 means link status changed, -1 means not changed */
> >>>> int ixgbe_dev_link_update_share(struct rte_eth_dev *dev, @@ -3981,9
> >>>> +4004,7 @@ ixgbe_dev_link_update_share(struct rte_eth_dev *dev,
> >>>>  		IXGBE_DEV_PRIVATE_TO_INTR(dev->data->dev_private);
> >>>>  	int link_up;
> >>>>  	int diag;
> >>>> -	u32 speed = 0;
> >>>>  	int wait = 1;
> >>>> -	bool autoneg = false;
> >>>>
> >>>>  	memset(&link, 0, sizeof(link));
> >>>>  	link.link_status = ETH_LINK_DOWN; @@ -3993,13 +4014,8 @@
> >>>> ixgbe_dev_link_update_share(struct
> >> rte_eth_dev
> >>>> *dev,
> >>>>
> >>>>  	hw->mac.get_link_status = true;
> >>>>
> >>>> -	if ((intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG) &&
> >>>> -		ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
> >>>> -		speed = hw->phy.autoneg_advertised;
> >>>> -		if (!speed)
> >>>> -			ixgbe_get_link_capabilities(hw, &speed, &autoneg);
> >>>> -		ixgbe_setup_link(hw, speed, true);
> >>>> -	}
> >>>> +	if (intr->flags & IXGBE_FLAG_NEED_LINK_CONFIG)
> >>>> +		return rte_eth_linkstatus_set(dev, &link);
> >>>>
> >>>>  	/* check if it needs to wait to complete, if lsc interrupt is enabled */
> >>>>  	if (wait_to_complete == 0 || dev->data->dev_conf.intr_conf.lsc !=
> >>>> 0) @@
> >>>> -4017,11 +4033,14 @@ ixgbe_dev_link_update_share(struct rte_eth_dev
> >> *dev,
> >>>>  	}
> >>>>
> >>>>  	if (link_up == 0) {
> >>>> -		intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
> >>>> +		if (ixgbe_get_media_type(hw) == ixgbe_media_type_fiber) {
> >>>> +			intr->flags |= IXGBE_FLAG_NEED_LINK_CONFIG;
> >>>> +			rte_eal_alarm_set(10,
> >>>> +				ixgbe_dev_setup_link_alarm_handler, dev);
> >>>> +		}
> >>>>  		return rte_eth_linkstatus_set(dev, &link);
> >>>>  	}
> >>>>
> >>>> -	intr->flags &= ~IXGBE_FLAG_NEED_LINK_CONFIG;
> >>>>  	link.link_status = ETH_LINK_UP;
> >>>>  	link.link_duplex = ETH_LINK_FULL_DUPLEX;
> >>>>
> >>>> @@ -5128,6 +5147,8 @@ ixgbevf_dev_stop(struct rte_eth_dev *dev)
> >>>>
> >>>>  	PMD_INIT_FUNC_TRACE();
> >>>>
> >>>> +	rte_eal_alarm_cancel(ixgbe_dev_setup_link_alarm_handler, dev);
> >>>> +
> >>>>  	ixgbevf_intr_disable(dev);
> >>>>
> >>>>  	hw->adapter_stopped = 1;
> >>>> --
> >>>> 2.17.1
> >>>

  reply	other threads:[~2018-09-12  8:29 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CGME20180831123824eucas1p1cd2981c716c4764703e24c3eeb4d33c7@eucas1p1.samsung.com>
2018-08-31 12:39 ` Ilya Maximets
2018-09-04  6:08   ` Zhang, Qi Z
2018-09-10 15:08     ` Ilya Maximets
2018-09-12  6:49       ` Zhang, Qi Z
2018-09-12  8:05         ` Ilya Maximets
2018-09-12  8:28           ` Zhang, Qi Z [this message]
2018-09-21 14:25             ` Zhang, Qi Z
2018-10-03  7:43               ` Ilya Maximets
2018-10-09 10:22                 ` Zhao1, Wei
2018-10-11  9:56           ` Zhao1, Wei
2018-10-11 10:26             ` Ilya Maximets
2018-10-11 12:21               ` Laurent Hardy
2018-10-12  7:36                 ` Zhao1, Wei
2018-10-15 10:43                   ` Laurent Hardy
2018-10-16  8:29                     ` Zhao1, Wei
2018-10-12  9:19               ` Zhao1, Wei
2018-10-12 10:14                 ` Ilya Maximets
2018-10-15  3:03                   ` Zhao1, Wei
2018-10-15  8:40                     ` Ilya Maximets
2018-10-16  8:59                       ` Zhao1, Wei
2018-10-30 10:20   ` Ilya Maximets
2018-11-01 15:45     ` Zhang, Qi Z
2018-11-01 16:05       ` Ilya Maximets
     [not found]   ` <CGME20181101160505eucas1p1fcf268f3febaa80dcbb3e573b2fc2c68@eucas1p1.samsung.com>
2018-11-01 16:04     ` [dpdk-dev] [PATCH v2] " Ilya Maximets
2018-11-02 13:49       ` Zhang, Qi Z
2018-11-07 15:52       ` Burakov, Anatoly
2018-11-08 10:27         ` Ilya Maximets

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=039ED4275CED7440929022BC67E7061153284408@SHSMSX103.ccr.corp.intel.com \
    --to=qi.z.zhang@intel.com \
    --cc=dev@dpdk.org \
    --cc=i.maximets@samsung.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=laurent.hardy@6wind.com \
    --cc=stable@dpdk.org \
    --cc=wei.dai@intel.com \
    --cc=wenzhuo.lu@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).