Re: [dpdk-dev] [PATCH v7 1/4] ethdev: support device reset and recovery events

[Ajit Khaparde <ajit.khaparde@broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 8887 bytes --]

On Tue, Feb 1, 2022 at 5:20 AM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> On 2/1/2022 1:09 PM, Kalesh Anakkur Purayil wrote:
> > Thank you Ferruh for the review. Please see inline.
> >
> > On Tue, Feb 1, 2022 at 5:41 PM Ferruh Yigit <ferruh.yigit@intel.com <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 1/28/2022 12:48 PM, Kalesh A P wrote:
> >      > From: Kalesh AP <kalesh-anakkur.purayil@broadcom.com <mailto:kalesh-anakkur.purayil@broadcom.com>>
> >      >
> >      > Adding support for the device reset and recovery events in the
> >      > rte_eth_event framework. FW error and FW reset conditions would be
> >      > managed internally by the PMD without needing application intervention.
> >      > In such cases, PMD would need reset/recovery events to notify application
> >      > that PMD is undergoing a reset.
> >      >
> >      > While most of the recovery process is transparent to the application since
> >      > most of the driver ensures recovery from FW reset or FW error conditions,
> >      > the application will have to reprogram any flows which were offloaded to
> >      > the underlying hardware.
> >      >
> >      > Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com <mailto:kalesh-anakkur.purayil@broadcom.com>>
> >      > Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com <mailto:somnath.kotur@broadcom.com>>
> >      > Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com <mailto:ajit.khaparde@broadcom.com>>
> >
> >     More developer cc'ed.
> >
> >      > ---
> >      >   doc/guides/prog_guide/poll_mode_drv.rst | 24 ++++++++++++++++++++++++
> >      >   lib/ethdev/rte_ethdev.h                 | 18 ++++++++++++++++++
> >      >   2 files changed, 42 insertions(+)
> >      >
> >      > diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst
> >      > index 6831289..9ecc0e4 100644
> >      > --- a/doc/guides/prog_guide/poll_mode_drv.rst
> >      > +++ b/doc/guides/prog_guide/poll_mode_drv.rst
> >      > @@ -623,3 +623,27 @@ by application.
> >      >   The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger
> >      >   the application to handle reset event. It is duty of application to
> >      >   handle all synchronization before it calls rte_eth_dev_reset().
> >      > +
> >      > +Error recovery support
> >      > +~~~~~~~~~~~~~~~~~~~~~~
> >      > +
> >      > +When the PMD detects a FW reset or error condition, it may try to recover
> >      > +from the error without needing the application intervention. In such cases,
> >      > +PMD would need events to notify the application that it is undergoing
> >      > +an error recovery.
> >      > +
> >      > +The PMD should trigger RTE_ETH_EVENT_ERR_RECOVERING event to notify the
> >      > +application that PMD detected a FW reset or FW error condition. PMD may
> >      > +try to recover from the error by itself. Data path may be quiesced and
> >      > +control path operations may fail during the recovery period. The application
> >      > +should stop polling till it receives RTE_ETH_EVENT_RECOVERED event from the PMD.
> >      > +
> >
> >     Between the time FW error occurred and the application receive the RECOVERING event,
> >     datapath will continue to poll and application may call control APIs, so the event
> >     really is not solving the issue and driver somehow should be sure this won't crash
> >     the application, in that case not sure about the benefit of this event.
> >
> > [Kalesh]: As soon as the driver detects a FW dead or reset condition, it sets the fastpath pointers to dummy functions. This will prevent the crash. All control path operations would fail with -EBUSY. This change is already there in bnxt PMD. This event is a notification to the application that the PMD is recovering from a FW error condition so that it can stop polling and issue control path operations.
> >
>
> This was my point indeed. PMD can't rely on this event and should take actions (for both
> datapath and control path) to prevent possible crash/error. If that is the case event
> doesn't have that much benefit since PMD already has to protect itself.

This is an attempt to prevent PMD and applications from encountering
unexpected situations because of an error in hardware or firmware.
The unexpected scenarios can include lockups to segfaults apart from
other situations.

When hardware or firmware encounters and error, propagating it to the
application for it to react can be slow, time consuming and could be
vendor specific.

Instead in the case of Broadcom devices we are trying to contain the
detection and recovery within the hardware, firmware mostly and to a
small extent driver framework.
In the case of kernel drivers, this allows avoiding a system reset as
much as possible while in the case of DPDK PMD, it can prevent
application reloads or ungraceful exits.

The PMD generated notification to the application will be more like a
"For Your Information" for most of the applications.
But applications can decide to act on the notification and take
specific steps - like flushing or deleting all the existing flow
rules, meters etc.. without waiting for success or failure indication.
Applications can also re-offload the flow rules, recreate meters based
on the notification allowing them to sync with the fresh hardware,
firmware state.

We think other PMD can also take advantage of these notifications to
provide better reliability, accessibility and serviceability in
general.

Thanks

Ajit
>
> >
> >      > +The PMD should trigger RTE_ETH_EVENT_RECOVERED event to notify the application
> >      > +that the it has recovered from the error condition. PMD re-configures the port
> >      > +to the state prior to the error condition. Control path and data path are up now.
> >      > +Since the device has undergone a reset, flow rules offloaded prior to reset
> >      > +may be lost and the application should recreate the rules again.
> >      > +
> >
> >     I think the most difficult part here is clarify what application should do
> >     when this event received consistent for all devices, "flow rules may be lost"
> >     looks very vague to me.
> >     Unless it is not clear for application what to do when this event is received,
> >     it is not that useful or it will be specific to some PMDs. And I can see it is
> >     hard to clarify this but perhaps we can define a set of common behavior.
> >     Not sure what others are thinking.
> >
> > [Kalesh]: Sure, let's wait for others' opinions as well.
> >
> >
> >      > +The PMD should trigger RTE_ETH_EVENT_INTR_RMV event to notify the application
> >      > +that it has failed to recover from the error condition. The device may not be
> >      > +usable anymore.
> >      > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h
> >      > index 147cc1c..a46819f 100644
> >      > --- a/lib/ethdev/rte_ethdev.h
> >      > +++ b/lib/ethdev/rte_ethdev.h
> >      > @@ -3818,6 +3818,24 @@ enum rte_eth_event_type {
> >      >       RTE_ETH_EVENT_DESTROY,  /**< port is released */
> >      >       RTE_ETH_EVENT_IPSEC,    /**< IPsec offload related event */
> >      >       RTE_ETH_EVENT_FLOW_AGED,/**< New aged-out flows is detected */
> >      > +     RTE_ETH_EVENT_ERR_RECOVERING,
> >      > +                     /**< port recovering from an error
> >      > +                      *
> >      > +                      * PMD detected a FW reset or error condition.
> >      > +                      * PMD will try to recover from the error.
> >      > +                      * Data path may be quiesced and Control path operations
> >      > +                      * may fail at this time.
> >      > +                      */
> >
> >     Please put multi line comments before enum, Andrew did a set of cleanups for these.
> >
> > [Kalesh]: Sure, will do.
> >
> >
> >      > +     RTE_ETH_EVENT_RECOVERED,
> >      > +                     /**< port recovered from an error
> >      > +                      *
> >      > +                      * PMD has recovered from the error condition.
> >      > +                      * Control path and Data path are up now.
> >      > +                      * PMD re-configures the port to the state prior to the error.
> >      > +                      * Since the device has undergone a reset, flow rules
> >      > +                      * offloaded prior to reset may be lost and
> >      > +                      * the application should recreate the rules again.
> >      > +                      */
> >      >       RTE_ETH_EVENT_MAX       /**< max value of this enum */
> >      >   };
> >      >
> >
> >
> >
> > --
> > Regards,
> > Kalesh A P
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]