From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 420E4A055F;
	Fri, 10 Jun 2022 02:16:07 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 294E64069C;
	Fri, 10 Jun 2022 02:16:07 +0200 (CEST)
Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187])
 by mails.dpdk.org (Postfix) with ESMTP id A911C40689
 for <dev@dpdk.org>; Fri, 10 Jun 2022 02:16:05 +0200 (CEST)
Received: from dggpeml500024.china.huawei.com (unknown [172.30.72.54])
 by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4LK1gT64cfzjXND;
 Fri, 10 Jun 2022 08:15:01 +0800 (CST)
Received: from [127.0.0.1] (10.67.100.224) by dggpeml500024.china.huawei.com
 (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Fri, 10 Jun
 2022 08:16:01 +0800
Subject: Re: [dpdk-dev] [PATCH v7 1/4] ethdev: support device reset and
 recovery events
To: Ray Kinsella <mdr@ashroe.eu>
CC: Thomas Monjalon <thomas@monjalon.net>, Ferruh Yigit
 <ferruh.yigit@intel.com>, Kalesh A P <kalesh-anakkur.purayil@broadcom.com>,
 <dev@dpdk.org>, <ajit.khaparde@broadcom.com>, <asafp@nvidia.com>, David
 Marchand <david.marchand@redhat.com>, Andrew Rybchenko
 <andrew.rybchenko@oktetlabs.ru>, <lizhenyi1@huawei.com>, shuliubin 00419723
 <shuliubin@huawei.com>
References: <20201009034832.10302-1-kalesh-anakkur.purayil@broadcom.com>
 <87sfspiuj1.fsf@mdr78.vserver.site> <878rudiwhb.fsf@mdr78.vserver.site>
 <45691978.XUcTiDjVJD@thomas> <875yphigb6.fsf@mdr78.vserver.site>
 <c27dbe0a-b5be-a246-eae4-960f06d583e4@huawei.com>
 <87k0abynhp.fsf@mdr78.vserver.site>
From: fengchengwen <fengchengwen@huawei.com>
Message-ID: <0ae97fbc-1ebe-a8d8-b121-071e9d5111ae@huawei.com>
Date: Fri, 10 Jun 2022 08:16:00 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
 Thunderbird/68.11.0
MIME-Version: 1.0
In-Reply-To: <87k0abynhp.fsf@mdr78.vserver.site>
Content-Type: text/plain; charset="utf-8"
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-Originating-IP: [10.67.100.224]
X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To
 dggpeml500024.china.huawei.com (7.185.36.10)
X-CFilter-Loop: Reflected
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

On 2022/5/24 23:11, Ray Kinsella wrote:
> 
> fengchengwen <fengchengwen@huawei.com> writes:
> 
>> Hi all,
>>
>>   This patch lasts for a long time. Are we waiting for 22.11 to deal with it?
> 
> That was my read, as can't reliably change the value of _MAX at this
> stage without it having impact elsewhere. 
> 
> 
>>   We have the same requirements for the reset or recovery mechanism, but there are differences:
>>
>>     APP                                    PMD
>>      |                                      |
>>      |                                  detect error
>>      |     <---report error event---        |
>>      |                                      |
>> do error stats                              |
>> and report                                  |
>>      |      ---start recover-->             |
>>      |                                  do recover
>>      |     <---report recover result        |
>>      |                                      |
>> if succ just log
>> else may migrate
>> service
>>
>> Can we generalize these processes(means that the implementation is at the framework layer)? or only at PMD API?
>>
>>
>> On 2022/2/15 0:06, Ray Kinsella wrote:
>>>
>>> Thomas Monjalon <thomas@monjalon.net> writes:
>>>
>>>> 14/02/2022 11:16, Ray Kinsella:
>>>>> Ray Kinsella <mdr@ashroe.eu> writes:
>>>>>> Thomas Monjalon <thomas@monjalon.net> writes:
>>>>>>> 02/02/2022 12:44, Ray Kinsella:
>>>>>>>> Ferruh Yigit <ferruh.yigit@intel.com> writes:
>>>>>>>>> On 1/28/2022 12:48 PM, Kalesh A P wrote:
>>>>>>>>>> --- a/lib/ethdev/rte_ethdev.h
>>>>>>>>>> +++ b/lib/ethdev/rte_ethdev.h
>>>>>>>>>> @@ -3818,6 +3818,24 @@ enum rte_eth_event_type {
>>>>>>>>>>   	RTE_ETH_EVENT_DESTROY,  /**< port is released */
>>>>>>>>>>   	RTE_ETH_EVENT_IPSEC,    /**< IPsec offload related event */
>>>>>>>>>>   	RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */
>>>>>>>>>> +	RTE_ETH_EVENT_ERR_RECOVERING,
>>>>>>>>>> +			/**< port recovering from an error
>>>>>>>>>> +			 *
>>>>>>>>>> +			 * PMD detected a FW reset or error condition.
>>>>>>>>>> +			 * PMD will try to recover from the error.
>>>>>>>>>> +			 * Data path may be quiesced and Control path operations
>>>>>>>>>> +			 * may fail at this time.
>>>>>>>>>> +			 */

I think we should standard error reason which could pass to application, so that
application know the really reason. the error reason could as the ret_param of
rte_eth_dev_callback_process().

But I think it could be done later.

>>>>>>>>>> +	RTE_ETH_EVENT_RECOVERED,
>>>>>>>>>> +			/**< port recovered from an error
>>>>>>>>>> +			 *
>>>>>>>>>> +			 * PMD has recovered from the error condition.
>>>>>>>>>> +			 * Control path and Data path are up now.
>>>>>>>>>> +			 * PMD re-configures the port to the state prior to the error.
>>>>>>>>>> +			 * Since the device has undergone a reset, flow rules
>>>>>>>>>> +			 * offloaded prior to reset may be lost and
>>>>>>>>>> +			 * the application should recreate the rules again.
>>>>>>>>>> +			 */

please add RTE_ETH_EVENT_RECOVER_FAIL event, the RTE_ETH_EVENT_INTR_RMV event is
a big event, it has its own usecase. So please add the RECOVER_FAIL event to let
application decide remove or keep it.

>>>>>>>>>>   	RTE_ETH_EVENT_MAX       /**< max value of this enum */
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also ABI check complains about 'RTE_ETH_EVENT_MAX' value check, cc'ed more people
>>>>>>>>> to evaluate if it is a false positive:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1 function with some indirect sub-type change:
>>>>>>>>>   [C] 'function int rte_eth_dev_callback_register(uint16_t, rte_eth_event_type, rte_eth_dev_cb_fn, void*)' at rte_ethdev.c:4637:1 has some indirect sub-type changes:
>>>>>>>>>     parameter 3 of type 'typedef rte_eth_dev_cb_fn' has sub-type changes:
>>>>>>>>>       underlying type 'int (typedef uint16_t, enum rte_eth_event_type, void*, void*)*' changed:
>>>>>>>>>         in pointed to type 'function type int (typedef uint16_t, enum rte_eth_event_type, void*, void*)':
>>>>>>>>>           parameter 2 of type 'enum rte_eth_event_type' has sub-type changes:
>>>>>>>>>             type size hasn't changed
>>>>>>>>>             2 enumerator insertions:
>>>>>>>>>               'rte_eth_event_type::RTE_ETH_EVENT_ERR_RECOVERING' value '11'
>>>>>>>>>               'rte_eth_event_type::RTE_ETH_EVENT_RECOVERED' value '12'
>>>>>>>>>             1 enumerator change:
>>>>>>>>>               'rte_eth_event_type::RTE_ETH_EVENT_MAX' from value '11' to '13' at rte_ethdev.h:3807:1
>>>>>>>>
>>>>>>>> I don't immediately see the problem that this would cause.
>>>>>>>> There are no array sizes etc dependent on the value of MAX for instance.
>>>>>>>>
>>>>>>>> Looks safe?
>>>>>>>
>>>>>>> We never know how this enum will be used by the application.
>>>>>>> The max value may be used for the size of an event array.
>>>>>>> It looks a real ABI issue unfortunately.
>>>>>>
>>>>>> Right - but we only really care about it when an array size based on MAX
>>>>>> is likely to be passed to DPDK, which doesn't apply in this case.
>>>>
>>>> I don't completely agree.
>>>> A developer may assume an event will never exceed MAX value.
>>>> However, after an upgrade of DPDK without app rebuild,
>>>> a higher event value may be received in the app,
>>>> breaking the assumption.
>>>> Should we consider this case as an ABI breakage?
>>>
>>> Nope - I think we should explicitly exclude MAX values from any
>>> ABI guarantee, as being able to change them is key to our be able to
>>> evolve DPDK while maintaining ABI stability. 
>>>
>>> Consider what it means applying the ABI policy to a MAX value, you are
>>> in effect saying that that no value can be added to this enumeration
>>> until the next ABI version, for me this is very restrictive without a
>>> solid reason. 
>>>
>>>>
>>>>>> I noted that some Linux folks explicitly mark similar MAX values as not
>>>>>> part of the ABI.
>>>>>>
>>>>>> /usr/include/linux/perf_event.h
>>>>>> 37:     PERF_TYPE_MAX,                          /* non-ABI */
>>>>>> 60:     PERF_COUNT_HW_MAX,                      /* non-ABI */
>>>>>> 79:     PERF_COUNT_HW_CACHE_MAX,                /* non-ABI */
>>>>>> 87:     PERF_COUNT_HW_CACHE_OP_MAX,             /* non-ABI */
>>>>>> 94:     PERF_COUNT_HW_CACHE_RESULT_MAX,         /* non-ABI */
>>>>>> 116:    PERF_COUNT_SW_MAX,                      /* non-ABI */
>>>>>> 149:    PERF_SAMPLE_MAX = 1U << 24,             /* non-ABI */
>>>>>> 151:    __PERF_SAMPLE_CALLCHAIN_EARLY           = 1ULL << 63, /*
>>>>>> non-ABI; internal use */
>>>>>> 189:    PERF_SAMPLE_BRANCH_MAX_SHIFT            /* non-ABI */
>>>>>> 267:    PERF_TXN_MAX            = (1 << 8), /* non-ABI */
>>>>>> 301:    PERF_FORMAT_MAX = 1U << 4,              /* non-ABI */
>>>>>> 1067:   PERF_RECORD_MAX,                        /* non-ABI */
>>>>>> 1078:   PERF_RECORD_KSYMBOL_TYPE_MAX            /* non-ABI */
>>>>>> 1087:   PERF_BPF_EVENT_MAX,             /* non-ABI */
>>>>>
>>>>> Any thoughts on similarly annotating all our _MAX enums in the same way?
>>>>> We could also add a section in the ABI Policy to make it explicit _MAX
>>>>> enum values are not part of the ABI - what do folks think?
>>>>
>>>> Interesting. I am not sure it is always ABI-safe though.
>>>
>>>
> 
>