From: "Mody, Rasesh" <Rasesh.Mody@cavium.com>
To: Ferruh Yigit <ferruh.yigit@intel.com>,
Stephen Hemminger <stephen@networkplumber.org>
Cc: "Tan, Jianfeng" <jianfeng.tan@intel.com>,
Jingjing Wu <jingjing.wu@intel.com>,
"Thotton, Shijith" <Shijith.Thotton@cavium.com>,
Gregory Etelson <gregory@weka.io>,
"Patil, Harish" <Harish.Patil@cavium.com>,
Thomas Monjalon <thomas@monjalon.net>,
"dev@dpdk.org" <dev@dpdk.org>,
"stable@dpdk.org" <stable@dpdk.org>,
George Prekas <george.prekas@epfl.ch>,
Sergio Gonzalez Monroy <sergio.gonzalez.monroy@intel.com>
Subject: Re: [dpdk-stable] [dpdk-dev] [PATCH] igb_uio: remove device reset in open
Date: Fri, 3 Nov 2017 19:18:31 +0000 [thread overview]
Message-ID: <CY4PR0701MB382781A0DF18B5100CCD21109F5D0@CY4PR0701MB3827.namprd07.prod.outlook.com> (raw)
In-Reply-To: <506f596c-f7e3-56b1-d13a-17b49bc3945a@intel.com>
> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> Sent: Thursday, November 02, 2017 5:32 PM
>
> On 11/2/2017 11:45 AM, Mody, Rasesh wrote:
> >> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> >> Sent: Thursday, November 02, 2017 11:10 AM
> >>
> >> On 11/2/2017 10:34 AM, Mody, Rasesh wrote:
> >>>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> >>>> Sent: Thursday, November 02, 2017 1:55 AM
> >>>>
> >>>> On 11/2/2017 1:03 AM, Mody, Rasesh wrote:
> >>>>>> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> >>>>>> Sent: Wednesday, November 01, 2017 7:12 AM
> >>>>>>
> >>>>>> On Wed, 1 Nov 2017 06:58:53 +0000 "Mody, Rasesh"
> >>>>>> <Rasesh.Mody@cavium.com> wrote:
> >>>>>>
> >>>>>>> Hi Jianfeng and Ferruh,
> >>>>>>>
> >>>>>>>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> >>>>>>>> Sent: Thursday, October 26, 2017 5:50 PM
> >>>>>>>>
> >>>>>>>> On 10/26/2017 2:28 AM, Tan, Jianfeng wrote:
> >>>>>>>>> Hi Rasesh,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/26/2017 7:43 AM, Mody, Rasesh wrote:
> >>>>>>>>>> Hi Ferruh,
> >>>>>>>>>>
> >>>>>>>>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of
> Ferruh
> >>>>>>>>>>> Yigit
> >>>>>>>>>>> Sent: Friday, October 20, 2017 9:58 AM
> >>>>>>>>>>>
> >>>>>>>>>>> On 10/20/2017 9:55 AM, Ferruh Yigit wrote:
> >>>>>>>>>>>> Remove device reset during application start, the reset for
> >>>>>>>>>>>> application exit still there.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Reset in open removed because of following comments:
> >>>>>>>>>>>> 1- Device reset not completed when VF driver loaded, which
> >>>>>>>>>>>> cause VF
> >>>>>>>> PMD
> >>>>>>>>>>>> initialization error.
> >>>>>>>>>>>> Adding delay can solve the issue but will increase
> >>>>>>>>>>>> driver load
> >>>> time.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2- Reset will be issues all devices unconditionally, not
> >>>>>>>>>>>> very
> >>>> efficient
> >>>>>>>>>>>> way.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and
> >>>>>>>>>>>> release of device file")
> >>>>>>>>>>>> Cc: stable@dpdk.org
> >>>>>>>>>>>>
> >>>>>>>>>>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >>>>>>>>>>> Hi Jingjing, Shijith, Gregory, Harish,
> >>>>>>>>>>>
> >>>>>>>>>>> Can you please test this on top of current master (which has
> >>>>>>>>>>> already Jingjin's
> >>>>>>>>>>> fix) ?
> >>>>>>>>>> The original FLR change during igb_uio open()/release() in
> >>>>>>>>>> DPDK17.08 also
> >>>>>>>> impacts BNX2X PMD and it exhibits the issues with bare metal
> >> testing.
> >>>>>>>>>>
> >>>>>>>>>> Now, we tested this change for BNX2X PMD using latest dpdk,
> >>>>>>>>>> which has
> >>>>>>>> this fix where FLR is invoked only in the release().
> >>>>>>>>
> >>>>>>>> Good to hear this fixed the problem.
> >>>>>>>
> >>>>>>> Yes, it fixed the issue caused by pci reset during application start.
> >>>>>>>
> >>>>>>>>
> >>>>>>>>>> However, we ran into an issue when trying to reload the
> >>>>>>>>>> testpmd
> >>>>>>>> application in quick succession. The pci reset, called during
> >>>>>>>> the igb_uio
> >>>>>>>> release() operation, is taking longer time and adapter is still
> >>>>>>>> doing the FLR when we relaunch the application. We see this
> >>>>>>>> behavior with bare metal testing.
> >>>>>>>>>
> >>>>>>>>> If we don't reset that device, it will continue working which
> >>>>>>>>> is a more serious issue IMO.
> >>>>>>>>
> >>>>>>>> +1
> >>>>>>>
> >>>>>>> I think, it would better for the individual PMDs to take care of
> >>>>>>> the reset
> >>>>>> during the application exit.
> >>>>>>
> >>>>>> That will never be possible. Poll Mode Drivers are userspace
> >>>>>> entities and part of the application. If application crashes,
> >>>>>> there is no way for PMD to do cleanup, it must be handled by kernel.
> >>>>>
> >>>>> The pci reset in release is breaking the BNX2X PMD. Could we
> >>>>> revert this
> >>>> reset and get it included with a solution that works for all in the
> >>>> next
> >> release?
> >>>>
> >>>> Hi Rasesh,
> >>>>
> >>>> I am not sure if there is more to do for solution for next
> >>>> releases, and related to your case, indeed I wasn't expecting a
> >>>> device reset will take more than five minutes...
> >>>>
> >>>> Would you be OK to control the reset via a compile time config
> >>>> option, which is enabled by default. So you will need to disable it
> >>>> to
> >> prevent the reset?
> >>>
> >>> Hi Ferruh,
> >>>
> >>> As I understand, we will have a compile time config option, enabled
> >>> by
> >> default, to guard the pci_reset_function() in the
> >> igbuio_pci_release(). We will disable this config option to prevent the
> reset when using BNX2X.
> >>
> >> Yep, this is the idea.
> >>
> >>> The controlled reset should work for us.
> >>
> >> If there is no objection, I can send a patch for this.
> >
> > We are ok as we have at least some way to disable the reset, please send a
> patch.
>
> Sent http://dpdk.org/dev/patchwork/patch/31143/, can you please test?
The testing of BNX2X looks OK with this patch. However, the solution has following drawbacks:
- an application will need to be recompiled to have the igb_uio kernel module rebuilt to support bnx2x devices
- this will break pre-compiled solutions that are provided with an OS such as RHOSP or as part of a pre-compiled VNF
We can live with this temporary solution for now. In the long term, we may have to revisit this.
We are also looking at why bnx2x FLR is taking this long.
Thanks!
-Rasesh
next prev parent reply other threads:[~2017-11-03 19:18 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-17 20:14 [dpdk-stable] [PATCH] igb_uio: revert open and release operations Ferruh Yigit
2017-10-17 20:33 ` Thomas Monjalon
2017-10-18 4:50 ` Patil, Harish
2017-10-19 22:43 ` Patil, Harish
2017-10-20 1:15 ` Ferruh Yigit
2017-10-20 15:26 ` Tan, Jianfeng
2017-10-20 16:32 ` Ferruh Yigit
2017-10-20 16:55 ` [dpdk-stable] [PATCH] igb_uio: remove device reset in open Ferruh Yigit
2017-10-20 16:57 ` Ferruh Yigit
2017-10-20 19:01 ` Gregory Etelson
2017-10-20 22:18 ` Patil, Harish
2017-10-23 12:28 ` Shijith Thotton
2017-10-23 16:36 ` Ferruh Yigit
2017-10-23 19:03 ` Shijith Thotton
2017-10-24 2:45 ` Wu, Jingjing
2017-10-25 23:43 ` [dpdk-stable] [dpdk-dev] " Mody, Rasesh
2017-10-26 9:28 ` Tan, Jianfeng
2017-10-27 0:49 ` Ferruh Yigit
2017-11-01 6:58 ` Mody, Rasesh
2017-11-01 14:12 ` Stephen Hemminger
2017-11-02 8:03 ` Mody, Rasesh
2017-11-02 8:55 ` Ferruh Yigit
2017-11-02 17:34 ` Mody, Rasesh
2017-11-02 18:09 ` Ferruh Yigit
2017-11-02 18:45 ` Mody, Rasesh
2017-11-03 0:31 ` Ferruh Yigit
2017-11-03 19:18 ` Mody, Rasesh [this message]
2017-10-24 21:38 ` [dpdk-stable] " Ferruh Yigit
2017-10-18 0:14 ` [dpdk-stable] [PATCH] igb_uio: revert open and release operations Wu, Jingjing
2017-10-18 6:27 ` Shijith Thotton
2017-10-18 20:47 ` Ferruh Yigit
2017-10-24 21:32 ` Ferruh Yigit
2017-10-22 16:06 [dpdk-stable] [dpdk-dev] [PATCH] igb_uio: remove device reset in open Patil, Harish
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CY4PR0701MB382781A0DF18B5100CCD21109F5D0@CY4PR0701MB3827.namprd07.prod.outlook.com \
--to=rasesh.mody@cavium.com \
--cc=Harish.Patil@cavium.com \
--cc=Shijith.Thotton@cavium.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=george.prekas@epfl.ch \
--cc=gregory@weka.io \
--cc=jianfeng.tan@intel.com \
--cc=jingjing.wu@intel.com \
--cc=sergio.gonzalez.monroy@intel.com \
--cc=stable@dpdk.org \
--cc=stephen@networkplumber.org \
--cc=thomas@monjalon.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).