From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B6E72A034E; Tue, 1 Feb 2022 14:20:02 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3AAB140698; Tue, 1 Feb 2022 14:20:02 +0100 (CET) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mails.dpdk.org (Postfix) with ESMTP id 7DC8B40691 for ; Tue, 1 Feb 2022 14:20:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1643721600; x=1675257600; h=message-id:date:to:cc:references:from:subject: in-reply-to:content-transfer-encoding:mime-version; bh=bGxaWU6sG3/k3xn0QuO65enxl0ABZcQ9LWChtDbc5/4=; b=daNK1gyZQEFQXf8Bl61HvKquo3VUlcM1p1tCEvTFCwjVWXJw+R0HO30W 13hJya7NGz1ym/9f8GNkSdANC5cRgXqtuJ7ZhDPe+IsxjEoOZwa1/wf+w A8el706FjFMESHlAYPRIXsjUwyLIm/VKqglAI+2OhMrODgfWBTxWekVHZ Dwm3m8QZmkfdUUviFB+Vw0r0K63+lMuTOiYk1TfaUdaT1qwyqS29uhWrR Fy5vA12oMHpo7mNT/P2noF1QNJvyO2KPmCaFLsuuDsE8WIOSrm4Qh8DwY Nvu3QVApn3crW1gTOW2wcqb+XW0V0/ZP00GJksJ6F1d4V4stmuxWIIU6I g==; X-IronPort-AV: E=McAfee;i="6200,9189,10244"; a="247904663" X-IronPort-AV: E=Sophos;i="5.88,333,1635231600"; d="scan'208";a="247904663" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Feb 2022 05:19:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,333,1635231600"; d="scan'208";a="626729399" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga002.fm.intel.com with ESMTP; 01 Feb 2022 05:19:59 -0800 Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 1 Feb 2022 05:19:58 -0800 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20 via Frontend Transport; Tue, 1 Feb 2022 05:19:58 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.172) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.20; Tue, 1 Feb 2022 05:19:58 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jgjjR64T1QCAtei+GJadaAWRhelXdrBOmwcoqaw1PjqMnQkjKs0c6kf0IxSwcGuHCXjNwbuYgwsh2f8oXWfXrFZTr6jeGC+Pvj6+AB0U2Qa9WFVinOMfAZ4bUDG9q0/d8bZGGeCmfJ0Q8GFNzKzHCdlVRQSjR+R88tExo1v1VPSmwGCBd0EvoZK2oknzvAfQBrDdDmjDLEwKMqiXLrNb3GXiLQjZptQVLfoctQGLWgS5OqjKrtIVuZ254f2Eh7jNbVgz40d5tjpe3uZBCP8MlRV2Qti+byWHxYwNhznIvdRQ25kJcRWyUvmAUqoGctaqwNba85wN373w6/tgM1TFDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aHY+XT8aZNK8b1BVsHT0E/J1V/jcI3eEejje8mvhGgU=; b=QSufMlOnkfPDSb441t8sy0He4ycnA8yPZqfbKV/Azj8mnJwqGtZOLnQ6aDMgxMf92y2+6S9U3koc8RlHY6oYMoWXjIHH98rivR7exFUq/IEPMLCLKBfTpaMqklEEmtM1dL1p5qJB6z8U773bc8tH9Wyb+naeqrtctDMKq6gQWzFarbKbwOqjahz2CK+G01Axs3Q1ir7vUbeLoNEdkqEvp1/eRizjY8zGhpuGieiMyK4CqJmWTFU5qyUMg8WSQAXkJCwu9QaOr+Zjg1XsfwDd76q8q3XsUIxZl7fOS6AktCi1tBkxQz+aKxfHEc4oxZF7+hAwDQ6hMyVB/TIF8OSJOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) by MWHPR11MB1968.namprd11.prod.outlook.com (2603:10b6:300:113::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4951.11; Tue, 1 Feb 2022 13:19:56 +0000 Received: from PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e]) by PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e%7]) with mapi id 15.20.4930.022; Tue, 1 Feb 2022 13:19:56 +0000 Message-ID: Date: Tue, 1 Feb 2022 13:19:51 +0000 Content-Language: en-US To: Kalesh Anakkur Purayil CC: dpdk-dev , Thomas Monjalon , "Andrew Rybchenko" , Konstantin Ananyev , Ajit Kumar Khaparde , , Stephen Hemminger , "jerinj@marvell.com" References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-1-kalesh-anakkur.purayil@broadcom.com> <20220128124830.427-2-kalesh-anakkur.purayil@broadcom.com> <1bd15a49-b53c-b1e7-bed4-2f8aef55a0f9@intel.com> From: Ferruh Yigit Subject: Re: [dpdk-dev] [PATCH v7 1/4] ethdev: support device reset and recovery events X-User: ferruhy In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0333.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18c::14) To PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 65c94298-cbc9-4263-c8a8-08d9e5858a25 X-MS-TrafficTypeDiagnostic: MWHPR11MB1968:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qSjxQXJHoPXrEmxfrceHpLY4/D7WKJ1wHssr/nlosP0BrMOJ4dO9tUaNd1K18RdJizJHG21DfuOJGVdX0ltpNaGBsx8N5HeHkPx/BmmHAxN0pQHbn0VD/CqhB7MB1x2Pz255FmsljPXeZ0awdf/icq3F4qJZafY+cS5svccuRAwyM+NUKbpwwczwyQJdQ8vavI5tLNCQ97Z9vEHtG/r8FepnvSAu6JKKsgsoslJJPLnfvNIbeWU+CoTHlXMBHiLQQVt17CHRreyjd6y/0GDzlNtdLN38VwF3tIWJOQ0nrSEaen5si4uuNQ+rYAVCjLeL9iNzCRQnCowZxvmp8dGBucepxISDMEI57GGQrAlV/LcfuP6/cIWi2tLtT0VJuV4anNybw4VM8s6EHmDKmntvYgJ5FeCp0NvgdkwKKQoqwTck2zL3+IlSrZLovqUlbjj8wYQ8PE/xjK2dE3vipiagJdkHPA5uz8tPVqUVjNWGU+QMMECxLTIu/K2+UtLn4hOrQ1XkOJFkVVB6+W4pw4eR6tR/w0Wr3jPKLK4T/QKn4HicDESqJwJwCnqCpVl/H4ysinkicNbgQ9Nq7FUmkWv9FqYwDBLfuZgAe1JcGPZocb0NVh7MY8v/4WTCuMcGDMf9gQMLbyoDu7OAzHeowOIlYRWmpNR2JPvy7PyR7QxsXU5US2cFANYlac7awsBkX+VXEcXCPmaFIsOxcJKua2zkhQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5000.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(82960400001)(6486002)(31686004)(83380400001)(36756003)(186003)(6916009)(26005)(38100700002)(54906003)(316002)(44832011)(2906002)(66476007)(8936002)(8676002)(31696002)(2616005)(4326008)(66556008)(66946007)(5660300002)(6666004)(508600001)(6506007)(6512007)(86362001)(55236004)(53546011)(45980500001)(20210929001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TXZWUE41U2hxa1dYNHY1UHozcStrQkFuSGc5Y0E5ZzFGbXBXam5yUS9VZjVi?= =?utf-8?B?ajBlV01DQVZuVks4Z2dJaXliSVdhdUFJMkcyY1N5dFo5OEpuMjlCTEpHVXN5?= =?utf-8?B?TExvcjFEaDhNL0lCWVlOeWtYb3JoNjV1VHd4Q3lLYTJXYU5vNkpjbHZQSnM1?= =?utf-8?B?cXdtQ2RkWndZOHRoaE9tNHRlWHJNeW9UVExsYjlkSUtvYXpvcGI3UTh4OXBK?= =?utf-8?B?cC81L01EVkVweHhGVTE0OTdWVUtITWs1WDFQcURVNWJVS05rYnlJYXVaTlln?= =?utf-8?B?OG80dExiL2dpVDh2UFA2cWNTVDVucjJoRDlSOUJqMW9ZNHNYTjI5bjczZEJt?= =?utf-8?B?RkFEK2lERnNlK2dVTHpvWGhYdzVsVWpJT2psSEh6cGFKQkFCb1YzOUdRODlP?= =?utf-8?B?WnRZN2VXL2V2ZlVCaHhFaDN2MExmVzJ5Q25pQkw3WW5JNnd4UkxtVDJOY3Fy?= =?utf-8?B?d0RTTTd0MVVVMWJOZDhHOEhLSlJtdjlPSllubmpQVVcvRjA4K1NxaGFTcE5P?= =?utf-8?B?b2dTOFQ1bThWcnpJK2EvUzZCTjdtaTJ2ejZsVjBwdkJ3UFAwcUh6OTFycVdD?= =?utf-8?B?SU4wUVhpdzRIalA2bEtrZ2dFUDlNWFlPYXNmS1ZDK0hoOTg3N2VDQ3pDRFha?= =?utf-8?B?L2VaV3Q1VzNWQXR2OTVpaFlOc2dLaFZUS3J1aWM5MlNYQnF5NCtZN1JKZDEv?= =?utf-8?B?NkNMdEdRdUdzeTVMMlF3bnpXMjFvdXYwUmVLaC9LcU1MOWV0TmljS2JxNXQw?= =?utf-8?B?M2FLUjNnc1pwYk1CQ2hRaGZjS05jaG1kQjkxaGV6azNaODFVczk5WWtLRUVT?= =?utf-8?B?VkFPamhkeXdycTVxaDkzRjV2bzFMWlVRbkpHWWFWTktHK2ZMaWczS2R1b1dh?= =?utf-8?B?ZlZIOWVFVkJIRkh5MldJS3h0U05VQ0NOMElDMkNPZHZOOE1sL01xb001R0Nk?= =?utf-8?B?ZjNiZVZacDN0OUIzL3JmbXMrTG9FSUFDS2RLRkpsS0s4WFpnd3hpbW1aM0lQ?= =?utf-8?B?SVZpVFU1RThSQWZxNzR0SmUvOHlna25BVVE3TElGRVBsbWVWNFplbHpLRGZw?= =?utf-8?B?dWwyTHJCRzc2RHhPZUhpRVhwMTJSaTVLbDdBQmZPU0d6a1RSdHZCOXZPZ2VD?= =?utf-8?B?elNhdU40ZFFQOUNldUdpVHR6VXY3azd0ZTZHY0FGcWhoOHRtSU1yK3pya0xk?= =?utf-8?B?bjZUR0dRZzI1OVFZSG1LSWhqbjdkMk14Vmh2Q1BZOWtXYTA1Tm5zc1d6YjhZ?= =?utf-8?B?MlFyZHl4ZHJQc0x1enZxODBTL3BKK1lOMDV2SHY2c3FxcUNrSFppRVdrRmJn?= =?utf-8?B?ZmtDVnQzK2h2eG5MOGhXTzFadTYrT1c0K2ZQMnRFWFFXTGIwaTlOMnRXUE1G?= =?utf-8?B?bVgvdk1CZW5WVUJBallIb1BOdnhHTlVVOXNFU1NRUXgxUXJRbVdPK1lXRURT?= =?utf-8?B?ZmROUGF3dWo5cW8wVlJJcGdSQi9ieFRZMVlCRC83bmRTTnd4N0Z5Wm4zOCtQ?= =?utf-8?B?eWFKWVlsWS9lNkFETVE5V2ZwVjBmOEY4UUJGbk9mVE1PS2daRHJ2V2s4aWtQ?= =?utf-8?B?c01vMUlTL2lFOVJVYlcyQWhiZGp6ZlBIMVFGbDVjVlhjb3pwU3pmSmhDNGg3?= =?utf-8?B?Slh3WCt2dUdHa204OHdqaGRBWE5UaW1ZOFY3WHdUZEtaVjl3L1FVeFdVbnZV?= =?utf-8?B?QWpXS2V1ZWVjN0xTdUlSa0pHWXk2T3Rzanh2R04yTDlZWEZSYnlzZGJ1cXVu?= =?utf-8?B?b3dpZGs4MEpsbFIyQkN5d2ZDdXhBTlRTWCtTQXN1SUkyTkVzQmE2OUROM05r?= =?utf-8?B?dGRURHhJL2Myb0x0UDVYYTNUeHdiTnhwYmhnNDV2UzZ5aWsya1l1dXgxaTZ1?= =?utf-8?B?bUNyeVdVeDIxU3NCRzNmSWdsNmUzeVBoSDBMT0FySGJjOHlOK0RlSVYyQWVL?= =?utf-8?B?VkdDRDZFNXh4S0lrTGFHanBLMHdEOFN1dGloc2pucGh6TW5rMGc3a3VyS2E5?= =?utf-8?B?TnlLaUtSN3lHY1VSTmV3NFgvV3FybVBPU0ZlUkVHZU9RMTRMNk9jTEJ0SXBN?= =?utf-8?B?eWNxYk9oOHJtRnZ3SkZRdUI4cEpnOFlzWFR5dC95UVJYdTN4Z2l0Vys5aXE4?= =?utf-8?B?VDF5Q2tCTEhBTCtkclV0US82d2dCYXJta2dNN1hiai9CVXFiZm9CTkMyZ3VO?= =?utf-8?Q?3ShwTtfpmJlVqLZ6//49qsI=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 65c94298-cbc9-4263-c8a8-08d9e5858a25 X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5000.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2022 13:19:56.3590 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: e3E+BGkaFaqribAaPQkNmPF66nvyxpvkHBVzIMabcvyBaAyQx1SKP+rlMQ3J1qpikML6SjltW2LQzqT9db3I9Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1968 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/1/2022 1:09 PM, Kalesh Anakkur Purayil wrote: > Thank you Ferruh for the review. Please see inline. > > On Tue, Feb 1, 2022 at 5:41 PM Ferruh Yigit > wrote: > > On 1/28/2022 12:48 PM, Kalesh A P wrote: > > From: Kalesh AP > > > > > Adding support for the device reset and recovery events in the > > rte_eth_event framework. FW error and FW reset conditions would be > > managed internally by the PMD without needing application intervention. > > In such cases, PMD would need reset/recovery events to notify application > > that PMD is undergoing a reset. > > > > While most of the recovery process is transparent to the application since > > most of the driver ensures recovery from FW reset or FW error conditions, > > the application will have to reprogram any flows which were offloaded to > > the underlying hardware. > > > > Signed-off-by: Kalesh AP > > > Signed-off-by: Somnath Kotur > > > Reviewed-by: Ajit Khaparde > > > More developer cc'ed. > > > --- > >   doc/guides/prog_guide/poll_mode_drv.rst | 24 ++++++++++++++++++++++++ > >   lib/ethdev/rte_ethdev.h                 | 18 ++++++++++++++++++ > >   2 files changed, 42 insertions(+) > > > > diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst > > index 6831289..9ecc0e4 100644 > > --- a/doc/guides/prog_guide/poll_mode_drv.rst > > +++ b/doc/guides/prog_guide/poll_mode_drv.rst > > @@ -623,3 +623,27 @@ by application. > >   The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger > >   the application to handle reset event. It is duty of application to > >   handle all synchronization before it calls rte_eth_dev_reset(). > > + > > +Error recovery support > > +~~~~~~~~~~~~~~~~~~~~~~ > > + > > +When the PMD detects a FW reset or error condition, it may try to recover > > +from the error without needing the application intervention. In such cases, > > +PMD would need events to notify the application that it is undergoing > > +an error recovery. > > + > > +The PMD should trigger RTE_ETH_EVENT_ERR_RECOVERING event to notify the > > +application that PMD detected a FW reset or FW error condition. PMD may > > +try to recover from the error by itself. Data path may be quiesced and > > +control path operations may fail during the recovery period. The application > > +should stop polling till it receives RTE_ETH_EVENT_RECOVERED event from the PMD. > > + > > Between the time FW error occurred and the application receive the RECOVERING event, > datapath will continue to poll and application may call control APIs, so the event > really is not solving the issue and driver somehow should be sure this won't crash > the application, in that case not sure about the benefit of this event. > > [Kalesh]: As soon as the driver detects a FW dead or reset condition, it sets the fastpath pointers to dummy functions. This will prevent the crash. All control path operations would fail with -EBUSY. This change is already there in bnxt PMD. This event is a notification to the application that the PMD is recovering from a FW error condition so that it can stop polling and issue control path operations. > This was my point indeed. PMD can't rely on this event and should take actions (for both datapath and control path) to prevent possible crash/error. If that is the case event doesn't have that much benefit since PMD already has to protect itself. > > > +The PMD should trigger RTE_ETH_EVENT_RECOVERED event to notify the application > > +that the it has recovered from the error condition. PMD re-configures the port > > +to the state prior to the error condition. Control path and data path are up now. > > +Since the device has undergone a reset, flow rules offloaded prior to reset > > +may be lost and the application should recreate the rules again. > > + > > I think the most difficult part here is clarify what application should do > when this event received consistent for all devices, "flow rules may be lost" > looks very vague to me. > Unless it is not clear for application what to do when this event is received, > it is not that useful or it will be specific to some PMDs. And I can see it is > hard to clarify this but perhaps we can define a set of common behavior. > Not sure what others are thinking. > > [Kalesh]: Sure, let's wait for others' opinions as well. > > > > +The PMD should trigger RTE_ETH_EVENT_INTR_RMV event to notify the application > > +that it has failed to recover from the error condition. The device may not be > > +usable anymore. > > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h > > index 147cc1c..a46819f 100644 > > --- a/lib/ethdev/rte_ethdev.h > > +++ b/lib/ethdev/rte_ethdev.h > > @@ -3818,6 +3818,24 @@ enum rte_eth_event_type { > >       RTE_ETH_EVENT_DESTROY,  /**< port is released */ > >       RTE_ETH_EVENT_IPSEC,    /**< IPsec offload related event */ > >       RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */ > > +     RTE_ETH_EVENT_ERR_RECOVERING, > > +                     /**< port recovering from an error > > +                      * > > +                      * PMD detected a FW reset or error condition. > > +                      * PMD will try to recover from the error. > > +                      * Data path may be quiesced and Control path operations > > +                      * may fail at this time. > > +                      */ > > Please put multi line comments before enum, Andrew did a set of cleanups for these. > > [Kalesh]: Sure, will do. > > > > +     RTE_ETH_EVENT_RECOVERED, > > +                     /**< port recovered from an error > > +                      * > > +                      * PMD has recovered from the error condition. > > +                      * Control path and Data path are up now. > > +                      * PMD re-configures the port to the state prior to the error. > > +                      * Since the device has undergone a reset, flow rules > > +                      * offloaded prior to reset may be lost and > > +                      * the application should recreate the rules again. > > +                      */ > >       RTE_ETH_EVENT_MAX       /**< max value of this enum */ > >   }; > > > > > > -- > Regards, > Kalesh A P