Re: [PATCH v12 2/5] ethdev: support proactive error handling mode

[fengchengwen <fengchengwen@huawei.com>]

Hi Andrew,

  I rework part of rst according your comments, sent by v13, please take a look.

Thanks.

On 2022/10/13 16:58, Andrew Rybchenko wrote:
> On 10/12/22 06:45, Chengwen Feng wrote:
>> From: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
>>
>> Some PMDs (e.g. hns3) could detect hardware or firmware errors, one
>> error recovery mode is to report RTE_ETH_EVENT_INTR_RESET event, and
>> wait for application invoke rte_eth_dev_reset() to recover the port,
>> however, this mode has the following weaknesses:
>>
>> 1) Due to different hardware and software design, some NIC port recovery
>> process requires multiple handshakes with the firmware and PF (when the
>> port is VF). It takes a long time to complete the entire operation for
>> one port, If multiple ports (for example, multiple VFs of a PF) are
>> reset at the same time, other VFs may fail to be reset. (Because the
>> reset processing is serial, the previous VFs must be processed before
>> the subsequent VFs).
>>
>> 2) The impact on the application layer is great, and it should stop
>> working queues, stop calling Rx and Tx functions, and then call
>> rte_eth_dev_reset(), and re-setup all again.
>>
>> This patch introduces proactive error handling mode, the PMD will try
>> to recover from the errors itself. In this process, the PMD sets the
>> data path pointers to dummy functions (which will prevent the crash),
>> and also make sure the control path operations failed with retcode
>> -EBUSY.
>>
>> Because the PMD recovers automatically, the application can only sense
>> that the data flow is disconnected for a while and the control API
>> returns an error in this period.
>>
>> In order to sense the error happening/recovering, three events were
>> introduced:
>>
>> 1) RTE_ETH_EVENT_ERR_RECOVERING: used to notify the application that it
>> detected an error and the recovery is being started. Upon receiving the
>> event, the application should not invoke any control path APIs until
>> receiving RTE_ETH_EVENT_RECOVERY_SUCCESS or
>> RTE_ETH_EVENT_RECOVERY_FAILED event.
>>
>> 2) RTE_ETH_EVENT_RECOVERY_SUCCESS: used to notify the application that
>> it recovers successful from the error, the PMD already re-configures the
>> port, and the effect is the same as that of the restart operation.
>>
>> 3) RTE_ETH_EVENT_RECOVERY_FAILED: used to notify the application that it
>> recovers failed from the error, the port should not usable anymore. The
>> application should close the port.
>>
>> Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
>> Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com>
>> Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
>> Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> 
> With few nits below,
> 
> Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> 
> [snip]
> 
>> diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst
>> index 9d081b1cba..73941a74bd 100644
>> --- a/doc/guides/prog_guide/poll_mode_drv.rst
>> +++ b/doc/guides/prog_guide/poll_mode_drv.rst
>> @@ -627,3 +627,41 @@ by application.
>>   The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger
>>   the application to handle reset event. It is duty of application to
>>   handle all synchronization before it calls rte_eth_dev_reset().
>> +
>> +The above error handling mode is known as ``RTE_ETH_ERROR_HANDLE_MODE_PASSIVE``.
>> +
>> +Proactive Error Handling Mode
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +If PMD supports ``RTE_ETH_ERROR_HANDLE_MODE_PROACTIVE``, it means once detect
>> +hardware or firmware errors, the PMD will try to recover from the errors. In
>> +this process, the PMD sets the data path pointers to dummy functions (which
>> +will prevent the crash), and also make sure the control path operations failed
>> +with retcode -EBUSY.
>> +
>> +Also in this process, from the perspective of application, services are
>> +affected. For example, the Rx/Tx bust APIs cannot receive and send packets,
> 
> bust -> burst
> 
>> +and the control plane API return failure.
> 
> I think we need to highlight here that the key advantage of the
> proactive error recover that it requires nothing from PMD by
> default. The recover simply happens.
> 
>> +
>> +In some service scenarios, application needs to be aware of the event to
>> +determine whether to migrate services. So three events were introduced:
>> +
>> +* RTE_ETH_EVENT_ERR_RECOVERING: used to notify the application that it detected
>> +  an error and the recovery is being started. Upon receiving the event, the
>> +  application should not invoke any control path APIs until receiving
>> +  RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_RECOVERY_FAILED event.
>> +
>> +* RTE_ETH_EVENT_RECOVERY_SUCCESS: used to notify the application that it
>> +  recovers successful from the error, the PMD already re-configures the port,
>> +  and the effect is the same as that of the restart operation.
>> +
>> +* RTE_ETH_EVENT_RECOVERY_FAILED: used to notify the application that it
>> +  recovers failed from the error, the port should not usable anymore. the
>> +  application should close the port.
>> +
>> +.. note::
>> +        * Before the PMD reports the recovery result, the PMD may report the
>> +          ``RTE_ETH_EVENT_ERR_RECOVERING`` event again, because a larger error
>> +          may occur during the recovery.
>> +        * The error handling mode supported by the PMD can be reported through
>> +          the ``rte_eth_dev_info_get`` API.
>> diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst
>> +     *    - LRO configuration
>> +     *    - LSC configuration
>> +     *    - MTU
>> +     *    - Mac address (default and those supplied by MAC address array)
>> +     *    - Promiscuous and allmulticast mode
>> +     *    - PTP configuration
>> +     *    - Queue (Rx/Tx) settings
>> +     *    - Queue statistics mappings
>> +     *    - RSS configuration by rte_eth_dev_rss_xxx() family
>> +     *    - Rx checksum configuration
>> +     *    - Rx interrupt settings
>> +     *    - Traffic management configuration
>> +     *    - VLAN configuration (including filtering, tpid, strip, pvid)
>> +     *    - VMDq configuration
>> +     * b) the following configuration maybe retained or not depending on the
>> +     *    device capabilities:
>> +     *    - flow rules
>> +     *      @see RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP
>> +     *    - shared flow objects
>> +     *      @see RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP
>> +     * c) the other configuration will not be stored and will need to be
>> +     *    re-configured.
>> +     */
>> +    RTE_ETH_EVENT_RECOVERY_SUCCESS,
>> +    /** Port recovers failed from the error.
>> +     * It means that the port should not usable anymore. The application
>> +     * should close the port.
>> +     */
>> +    RTE_ETH_EVENT_RECOVERY_FAILED,
>>       RTE_ETH_EVENT_MAX       /**< max value of this enum */
>>   };
> 
> [snip]
> 
> 
> .