Re: [PATCH 1/5] ethdev: fix race-condition of proactive error handling mode

[fengchengwen <fengchengwen@huawei.com>]

On 2023/3/6 19:13, Konstantin Ananyev wrote:
> 
> 
>>>>>>>>> In the proactive error handling mode, the PMD will set the data path
>>>>>>>>> pointers to dummy functions and then try recovery, in this period the
>>>>>>>>> application may still invoking data path API. This will introduce a
>>>>>>>>> race-condition with data path which may lead to crash [1].
>>>>>>>>>
>>>>>>>>> Although the PMD added delay after setting data path pointers to cover
>>>>>>>>> the above race-condition, it reduces the probability, but it doesn't
>>>>>>>>> solve the problem.
>>>>>>>>>
>>>>>>>>> To solve the race-condition problem fundamentally, the following
>>>>>>>>> requirements are added:
>>>>>>>>> 1. The PMD should set the data path pointers to dummy functions after
>>>>>>>>>     report RTE_ETH_EVENT_ERR_RECOVERING event.
>>>>>>>>> 2. The application should stop data path API invocation when process
>>>>>>>>>     the RTE_ETH_EVENT_ERR_RECOVERING event.
>>>>>>>>> 3. The PMD should set the data path pointers to valid functions before
>>>>>>>>>     report RTE_ETH_EVENT_RECOVERY_SUCCESS event.
>>>>>>>>> 4. The application should enable data path API invocation when process
>>>>>>>>>     the RTE_ETH_EVENT_RECOVERY_SUCCESS event.
>>>>>>>>>
>>>>>>>
>>>>>>> How this is solving the race-condition, by pushing responsibility to
>>>>>>> stop data path to application?
>>>>>>
>>>>>> Exactly, it becomes application responsibility to make sure data-path is
>>>>>> stopped/suspended before recovery will continue.
>>>>>>
>>>>>
>>>>> From documentation of the feature:
>>>>>
>>>>> ``
>>>>> Because the PMD recovers automatically,
>>>>> the application can only sense that the data flow is disconnected for a
>>>>> while and the control API returns an error in this period.
>>>>>
>>>>> In order to sense the error happening/recovering, as well as to restore
>>>>> some additional configuration, three events are available:
>>>>> ``
>>>>>
>>>>> It looks like initial design is to use events mainly inform application
>>>>> about what happened and mainly for re-configuration.
>>>>>
>>>>> Although I am don't disagree to involve the application, I am not sure
>>>>> that is part of current design.
>>>>
>>>> I thought we all agreed that initial design contain some fallacies that
>>>> need to fixed, no?
>>>> Statement that with current rte_ethdev design error recovery can be done
>>>> without interaction with the app (to stop/suspend data/control path)
>>>> is the main one I think.
>>>> It needs some interaction with app layer, one way or another.
>>>>
>>>>>>>
>>>>>>> What if application is not interested in recovery modes at all and not
>>>>>>> registered any callback for the recovery?
>>>>>>
>>>>>>
>>>>>> Are you saying there is no way for application to disable
>>>>>> automatic recovery in PMD if it is not interested
>>>>>> (or can't full-fill per-requesties for it)?
>>>>>> If so, then yes it is a problem and we need to fix it.
>>>>>> I assumed that such mechanism to disable unwanted events already exists,
>>>>>> but I can't find anything.
>>>>>> Wonder what would be the easiest way here - can PMD make a decision
>>>>>> based on callback return value, or do we need a new API to
>>>>>> enable/disable callbacks, or ...?
>>>>>>
>>>>>>
>>>>>
>>>>> As far as I can see automatic recovery is not configurable by app.
>>>>>
>>>>> But that is not all, PMD sends events to application but PMD can't know
>>>>> if application is handling them or not, so with current design PMD can't
>>>>> rely on to app.
>>>>
>>>> Well, PMD invokes user provided callback.
>>>> One way to fix that problem - if there is no callback provided,
>>>> or callback returns an error code - PMD can assume that recovery
>>>> should not be done.
>>>> That is probably not the best design choice, but at least it will allow
>>>> to fix the problem without too many changes and introducing new API.
>>>> That could be sort of a 'quick fix'.
>>>> In a meanwhile we can think about new/better approach for that.
>>>>
>>>
>>> -rc2 for 23.03 is a few days away.
>>>
>>> What do you think to have 'quick fix' as modifying how driver updates
>>> burst ops to prevent the race condition, for this release?

The 'quick fix', do you mean only update function pointer (without rxq setting) ?
Currently the PMDs which announced support "proactive error handling mode" already
do this.

>>>
>>> And plan a design update for the next release?
>> +1 on the overall approach.
> 
> Yep, agree.

Hope for better solution.
And also, I notice only the openvswitch (from all open-source software which based-on DPDK)
registers RTE_ETH_EVENT_INTR_RESET callback .

Therefore, hope we build a recovery framework at the DPDK SDK level and be compatible
with RTE_ETH_EVENT_INTR_RESET and RTE_ETH_EVENT_ERR_RECOVERING mechanism.

>  
>>
>>>
>>>
>>>>>
>>>>>>> I think driver should not rely on application for this, unless
>>>>>>> application explicitly says (to driver) that it is handling recovery,
>>>>>>> right now there is no way for driver to know this.
>>>>>>
>>>>>> I think it is visa-versa:
>>>>>> application should not enable auto-recovery if it can't meet
>>>>>> per-requeststies for it (provide appropriate callback).
>>>>>>
>>>>>
>>>>> I agree on above, we are saying similar thing in different perspective.
>>>>
>>>> Ok, that's good we are on the same page.
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>> Also, this patch introduce a driver internal function
>>>>>>>>> rte_eth_fp_ops_setup which used as an help function for PMD.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://patchwork.dpdk.org/project/dpdk/patch/20230220060839.1267349-2-ashok.k.kaladi@intel.com/
>>>>>>>>>
>>>>>>>>> Fixes: eb0d471a8941 ("ethdev: add proactive error handling mode")
>>>>>>>>> Cc: stable@dpdk.org
>>>>>>>>>
>>>>>>>>> Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
>>>>>>>>> ---
>>>>>>>>>   doc/guides/prog_guide/poll_mode_drv.rst | 20 +++++++---------
>>>>>>>>>   lib/ethdev/ethdev_driver.c              |  8 +++++++
>>>>>>>>>   lib/ethdev/ethdev_driver.h              | 10 ++++++++
>>>>>>>>>   lib/ethdev/rte_ethdev.h                 | 32
>>>>>>>>> +++++++++++++++----------
>>>>>>>>>   lib/ethdev/version.map                  |  1 +
>>>>>>>>>   5 files changed, 46 insertions(+), 25 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/doc/guides/prog_guide/poll_mode_drv.rst
>>>>>>>>> b/doc/guides/prog_guide/poll_mode_drv.rst
>>>>>>>>> index c145a9066c..e380ff135a 100644
>>>>>>>>> --- a/doc/guides/prog_guide/poll_mode_drv.rst
>>>>>>>>> +++ b/doc/guides/prog_guide/poll_mode_drv.rst
>>>>>>>>> @@ -638,14 +638,9 @@ different from the application invokes recovery
>>>>>>>>> in PASSIVE mode,
>>>>>>>>>   the PMD automatically recovers from error in PROACTIVE mode,
>>>>>>>>>   and only a small amount of work is required for the application.
>>>>>>>>>
>>>>>>>>> -During error detection and automatic recovery,
>>>>>>>>> -the PMD sets the data path pointers to dummy functions
>>>>>>>>> -(which will prevent the crash),
>>>>>>>>> -and also make sure the control path operations fail with a return
>>>>>>>>> code ``-EBUSY``.
>>>>>>>>> -
>>>>>>>>> -Because the PMD recovers automatically,
>>>>>>>>> -the application can only sense that the data flow is disconnected
>>>>>>>>> for a while
>>>>>>>>> -and the control API returns an error in this period.
>>>>>>>>> +During error detection and automatic recovery, the PMD sets the
>>>>>>>>> data path
>>>>>>>>> +pointers to dummy functions and also make sure the control path
>>>>>>>>> operations
>>>>>>>>> +failed with a return code ``-EBUSY``.
>>>>>>>>>
>>>>>>>>>   In order to sense the error happening/recovering,
>>>>>>>>>   as well as to restore some additional configuration,
>>>>>>>>> @@ -653,9 +648,9 @@ three events are available:
>>>>>>>>>
>>>>>>>>>   ``RTE_ETH_EVENT_ERR_RECOVERING``
>>>>>>>>>      Notify the application that an error is detected
>>>>>>>>> -   and the recovery is being started.
>>>>>>>>> +   and the recovery is about to start.
>>>>>>>>>      Upon receiving the event, the application should not invoke
>>>>>>>>> -   any control path function until receiving
>>>>>>>>> +   any control and data path API until receiving
>>>>>>>>>      ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` or
>>>>>>>>> ``RTE_ETH_EVENT_RECOVERY_FAILED`` event.
>>>>>>>>>
>>>>>>>>>   .. note::
>>>>>>>>> @@ -666,8 +661,9 @@ three events are available:
>>>>>>>>>
>>>>>>>>>   ``RTE_ETH_EVENT_RECOVERY_SUCCESS``
>>>>>>>>>      Notify the application that the recovery from error is successful,
>>>>>>>>> -   the PMD already re-configures the port,
>>>>>>>>> -   and the effect is the same as a restart operation.
>>>>>>>>> +   the PMD already re-configures the port.
>>>>>>>>> +   The application should restore some additional configuration,
>>>>>>>>> and then
>>>>>>>>> +   enable data path API invocation.
>>>>>>>>>
>>>>>>>>>   ``RTE_ETH_EVENT_RECOVERY_FAILED``
>>>>>>>>>      Notify the application that the recovery from error failed,
>>>>>>>>> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev_driver.c
>>>>>>>>> index 0be1e8ca04..f994653fe9 100644
>>>>>>>>> --- a/lib/ethdev/ethdev_driver.c
>>>>>>>>> +++ b/lib/ethdev/ethdev_driver.c
>>>>>>>>> @@ -515,6 +515,14 @@ rte_eth_dma_zone_free(const struct rte_eth_dev
>>>>>>>>> *dev, const char *ring_name,
>>>>>>>>>       return rc;
>>>>>>>>>   }
>>>>>>>>>
>>>>>>>>> +void
>>>>>>>>> +rte_eth_fp_ops_setup(struct rte_eth_dev *dev)
>>>>>>>>> +{
>>>>>>>>> +    if (dev == NULL)
>>>>>>>>> +        return;
>>>>>>>>> +    eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_id, dev);
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>>   const struct rte_memzone *
>>>>>>>>>   rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char
>>>>>>>>> *ring_name,
>>>>>>>>>                uint16_t queue_id, size_t size, unsigned int align,
>>>>>>>>> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h
>>>>>>>>> index 2c9d615fb5..0d964d1f67 100644
>>>>>>>>> --- a/lib/ethdev/ethdev_driver.h
>>>>>>>>> +++ b/lib/ethdev/ethdev_driver.h
>>>>>>>>> @@ -1621,6 +1621,16 @@ int
>>>>>>>>>   rte_eth_dma_zone_free(const struct rte_eth_dev *eth_dev, const
>>>>>>>>> char *name,
>>>>>>>>>            uint16_t queue_id);
>>>>>>>>>
>>>>>>>>> +/**
>>>>>>>>> + * @internal
>>>>>>>>> + * Setup eth fast-path API to ethdev values.
>>>>>>>>> + *
>>>>>>>>> + * @param dev
>>>>>>>>> + *  Pointer to struct rte_eth_dev.
>>>>>>>>> + */
>>>>>>>>> +__rte_internal
>>>>>>>>> +void rte_eth_fp_ops_setup(struct rte_eth_dev *dev);
>>>>>>>>> +
>>>>>>>>>   /**
>>>>>>>>>    * @internal
>>>>>>>>>    * Atomically set the link status for the specific device.
>>>>>>>>> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
>>>>>>>>> index 049641d57c..44ee7229c1 100644
>>>>>>>>> --- a/lib/ethdev/rte_ethdev.h
>>>>>>>>> +++ b/lib/ethdev/rte_ethdev.h
>>>>>>>>> @@ -3944,25 +3944,28 @@ enum rte_eth_event_type {
>>>>>>>>>        */
>>>>>>>>>       RTE_ETH_EVENT_RX_AVAIL_THRESH,
>>>>>>>>>       /** Port recovering from a hardware or firmware error.
>>>>>>>>> -     * If PMD supports proactive error recovery,
>>>>>>>>> -     * it should trigger this event to notify application
>>>>>>>>> -     * that it detected an error and the recovery is being started.
>>>>>>>>> -     * Upon receiving the event, the application should not invoke
>>>>>>>>> any control path API
>>>>>>>>> -     * (such as rte_eth_dev_configure/rte_eth_dev_stop...) until
>>>>>>>>> receiving
>>>>>>>>> -     * RTE_ETH_EVENT_RECOVERY_SUCCESS or
>>>>>>>>> RTE_ETH_EVENT_RECOVERY_FAILED event.
>>>>>>>>> -     * The PMD will set the data path pointers to dummy functions,
>>>>>>>>> -     * and re-set the data path pointers to non-dummy functions
>>>>>>>>> -     * before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS event.
>>>>>>>>> -     * It means that the application cannot send or receive any
>>>>>>>>> packets
>>>>>>>>> -     * during this period.
>>>>>>>>> +     *
>>>>>>>>> +     * If PMD supports proactive error recovery, it should trigger
>>>>>>>>> this
>>>>>>>>> +     * event to notify application that it detected an error and the
>>>>>>>>> +     * recovery is about to start.
>>>>>>>>> +     *
>>>>>>>>> +     * Upon receiving the event, the application should not invoke any
>>>>>>>>> +     * control and data path API until receiving
>>>>>>>>> +     * RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_RECOVERY_FAILED
>>>>>>>>> +     * event.
>>>>>>>>> +     *
>>>>>>>>> +     * Once this event is reported, the PMD will set the data path
>>>>>>>>> pointers
>>>>>>>>> +     * to dummy functions, and re-set the data path pointers to valid
>>>>>>>>> +     * functions before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS
>>>>>>>>> event.
>>>>>>>>> +     *
>>>>>>>>>        * @note Before the PMD reports the recovery result,
>>>>>>>>>        * the PMD may report the RTE_ETH_EVENT_ERR_RECOVERING event
>>>>>>>>> again,
>>>>>>>>>        * because a larger error may occur during the recovery.
>>>>>>>>>        */
>>>>>>>>>       RTE_ETH_EVENT_ERR_RECOVERING,
>>>>>>>>>       /** Port recovers successfully from the error.
>>>>>>>>> -     * The PMD already re-configured the port,
>>>>>>>>> -     * and the effect is the same as a restart operation.
>>>>>>>>> +     *
>>>>>>>>> +     * The PMD already re-configured the port:
>>>>>>>>>        * a) The following operation will be retained: (alphabetically)
>>>>>>>>>        *    - DCB configuration
>>>>>>>>>        *    - FEC configuration
>>>>>>>>> @@ -3989,6 +3992,9 @@ enum rte_eth_event_type {
>>>>>>>>>        *      (@see RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP)
>>>>>>>>>        * c) Any other configuration will not be stored
>>>>>>>>>        *    and will need to be re-configured.
>>>>>>>>> +     *
>>>>>>>>> +     * The application should restore some additional configuration
>>>>>>>>> +     * (see above case b/c), and then enable data path API invocation.
>>>>>>>>>        */
>>>>>>>>>       RTE_ETH_EVENT_RECOVERY_SUCCESS,
>>>>>>>>>       /** Port recovery failed.
>>>>>>>>> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map
>>>>>>>>> index 357d1a88c0..c273e0bdae 100644
>>>>>>>>> --- a/lib/ethdev/version.map
>>>>>>>>> +++ b/lib/ethdev/version.map
>>>>>>>>> @@ -320,6 +320,7 @@ INTERNAL {
>>>>>>>>>       rte_eth_devices;
>>>>>>>>>       rte_eth_dma_zone_free;
>>>>>>>>>       rte_eth_dma_zone_reserve;
>>>>>>>>> +    rte_eth_fp_ops_setup;
>>>>>>>>>       rte_eth_hairpin_queue_peer_bind;
>>>>>>>>>       rte_eth_hairpin_queue_peer_unbind;
>>>>>>>>>       rte_eth_hairpin_queue_peer_update;
>>>>>>>>> --
>>>>>>>>   Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
>>>>>>>>
>>>>>>>>> 2.17.1
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>