From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB8B141E34; Thu, 9 Mar 2023 03:05:30 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7501040ED7; Thu, 9 Mar 2023 03:05:30 +0100 (CET) Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by mails.dpdk.org (Postfix) with ESMTP id 4A60340A7E for ; Thu, 9 Mar 2023 03:05:26 +0100 (CET) Received: by mail-qk1-f171.google.com with SMTP id g20so220968qkm.7 for ; Wed, 08 Mar 2023 18:05:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1678327525; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=7m3HQOx2WascNIli8c7UZ+RYJoH2Q7N1IEifz4VhTjs=; b=dxljXryYQ24Fr3eDfNlILo+7ZecxD/rgU7Px3w2cukBE5aiukmlmUEvnkehKbgqgID 9sWyAl2ftB7LM6KVyzHo+49gUI1YcN4WVrpi5qSmMz0w99xNUkKT/43nAA0z0vDTMI8+ 2Od/EdBvEmSL02SHDQetTEOQ9tWg8t5kh/zqo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678327525; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7m3HQOx2WascNIli8c7UZ+RYJoH2Q7N1IEifz4VhTjs=; b=nAREOnPrSmY+7kaOqqFO8u7AUJpura97cAS6pYmqpy0cOyxl/qpHxVLzd4SFTdj3c1 yKVAkrcAY9pzQIWSft9ielWEu+IzU5lNBIp29Dp0cB4985lQZU+YbbYNSUeaR7idPSMe CK4DlQki/g+kqBFA7eboln5pFOy4cU1LZdBX70aaXmuZ+DhJTY8t7cLSRoSkDcUVwmJs 4i5zaGRotc+v4Atmcz1WokKy5qLnT/JPO0L7hBP3z0ASn+kQmuhYGbacUIQZfNopX2z6 N3cJhH1HGlcrVZcY1WSH/lXOqat16RsFskas02n4rLBBHaYjbGZBqUaQBF7exGJ+BxD+ 1oaw== X-Gm-Message-State: AO0yUKWC4Rx/I/c9bXBZPgi4Wl83F5+oR1Sx6GvCwA3YGyw1Fnieqzeu KrjofvUAsYqK4h1abirfT2wtrHQcXr4+eV92279v6g== X-Google-Smtp-Source: AK7set8XmzxjsMB+xLji9KFysiN2GZxMxfHzwCI860MewFa+MikMFGILCeH45G71gzWF9HqpfaSNZBIYHfSWJUSVgxI= X-Received: by 2002:a05:620a:ce4:b0:742:72c6:a140 with SMTP id c4-20020a05620a0ce400b0074272c6a140mr5791382qkj.7.1678327525436; Wed, 08 Mar 2023 18:05:25 -0800 (PST) MIME-Version: 1.0 References: <20230301030610.49468-1-fengchengwen@huawei.com> <20230301030610.49468-2-fengchengwen@huawei.com> <0f387ca1eee34a7f92745de7b59a71a1@huawei.com> <18c5b676-ae72-e646-89af-d6cd636d923f@yandex.ru> <5b40049f-6f22-fc3f-b13e-da1793c2fd1a@amd.com> <9092fdae9c1d4c53a00a8f23eb1129ec@huawei.com> <0effdaa9-5045-0635-775c-6e1eda0d5dba@amd.com> <9dc5d714-07a4-c32e-e557-efe8e7fe2d16@huawei.com> <5b110017-8679-4603-5c25-742ed83e4bda@amd.com> <16463f17-6f71-308b-c27b-9134346002c9@huawei.com> In-Reply-To: From: Ajit Khaparde Date: Wed, 8 Mar 2023 18:05:06 -0800 Message-ID: Subject: Re: [PATCH 1/5] ethdev: fix race-condition of proactive error handling mode To: Konstantin Ananyev Cc: Fengchengwen , Ferruh Yigit , Konstantin Ananyev , Thomas Monjalon , Andrew Rybchenko , "dev@dpdk.org" Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="000000000000fec2dd05f66e134f" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000fec2dd05f66e134f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Mar 7, 2023 at 4:40=E2=80=AFAM Konstantin Ananyev wrote: > > > > > >>>>>>>>>>> In the proactive error handling mode, the PMD will set the = data path > > >>>>>>>>>>> pointers to dummy functions and then try recovery, in this = period the > > >>>>>>>>>>> application may still invoking data path API. This will int= roduce a > > >>>>>>>>>>> race-condition with data path which may lead to crash [1]. > > >>>>>>>>>>> > > >>>>>>>>>>> Although the PMD added delay after setting data path pointe= rs to cover > > >>>>>>>>>>> the above race-condition, it reduces the probability, but i= t doesn't > > >>>>>>>>>>> solve the problem. > > >>>>>>>>>>> > > >>>>>>>>>>> To solve the race-condition problem fundamentally, the foll= owing > > >>>>>>>>>>> requirements are added: > > >>>>>>>>>>> 1. The PMD should set the data path pointers to dummy funct= ions after > > >>>>>>>>>>> report RTE_ETH_EVENT_ERR_RECOVERING event. > > >>>>>>>>>>> 2. The application should stop data path API invocation whe= n process > > >>>>>>>>>>> the RTE_ETH_EVENT_ERR_RECOVERING event. > > >>>>>>>>>>> 3. The PMD should set the data path pointers to valid funct= ions before > > >>>>>>>>>>> report RTE_ETH_EVENT_RECOVERY_SUCCESS event. > > >>>>>>>>>>> 4. The application should enable data path API invocation w= hen process > > >>>>>>>>>>> the RTE_ETH_EVENT_RECOVERY_SUCCESS event. > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> How this is solving the race-condition, by pushing responsibi= lity to > > >>>>>>>>> stop data path to application? > > >>>>>>>> > > >>>>>>>> Exactly, it becomes application responsibility to make sure da= ta-path is > > >>>>>>>> stopped/suspended before recovery will continue. > > >>>>>>>> > > >>>>>>> > > >>>>>>> From documentation of the feature: > > >>>>>>> > > >>>>>>> `` > > >>>>>>> Because the PMD recovers automatically, > > >>>>>>> the application can only sense that the data flow is disconnect= ed for a > > >>>>>>> while and the control API returns an error in this period. > > >>>>>>> > > >>>>>>> In order to sense the error happening/recovering, as well as to= restore > > >>>>>>> some additional configuration, three events are available: > > >>>>>>> `` > > >>>>>>> > > >>>>>>> It looks like initial design is to use events mainly inform app= lication > > >>>>>>> about what happened and mainly for re-configuration. > > >>>>>>> > > >>>>>>> Although I am don't disagree to involve the application, I am n= ot sure > > >>>>>>> that is part of current design. > > >>>>>> > > >>>>>> I thought we all agreed that initial design contain some fallaci= es that > > >>>>>> need to fixed, no? > > >>>>>> Statement that with current rte_ethdev design error recovery can= be done > > >>>>>> without interaction with the app (to stop/suspend data/control p= ath) > > >>>>>> is the main one I think. > > >>>>>> It needs some interaction with app layer, one way or another. > > >>>>>> > > >>>>>>>>> > > >>>>>>>>> What if application is not interested in recovery modes at al= l and not > > >>>>>>>>> registered any callback for the recovery? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Are you saying there is no way for application to disable > > >>>>>>>> automatic recovery in PMD if it is not interested > > >>>>>>>> (or can't full-fill per-requesties for it)? > > >>>>>>>> If so, then yes it is a problem and we need to fix it. > > >>>>>>>> I assumed that such mechanism to disable unwanted events alrea= dy exists, > > >>>>>>>> but I can't find anything. > > >>>>>>>> Wonder what would be the easiest way here - can PMD make a dec= ision > > >>>>>>>> based on callback return value, or do we need a new API to > > >>>>>>>> enable/disable callbacks, or ...? > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> As far as I can see automatic recovery is not configurable by a= pp. > > >>>>>>> > > >>>>>>> But that is not all, PMD sends events to application but PMD ca= n't know > > >>>>>>> if application is handling them or not, so with current design = PMD can't > > >>>>>>> rely on to app. > > >>>>>> > > >>>>>> Well, PMD invokes user provided callback. > > >>>>>> One way to fix that problem - if there is no callback provided, > > >>>>>> or callback returns an error code - PMD can assume that recovery > > >>>>>> should not be done. > > >>>>>> That is probably not the best design choice, but at least it wil= l allow > > >>>>>> to fix the problem without too many changes and introducing new = API. > > >>>>>> That could be sort of a 'quick fix'. > > >>>>>> In a meanwhile we can think about new/better approach for that. > > >>>>>> > > >>>>> > > >>>>> -rc2 for 23.03 is a few days away. > > >>>>> > > >>>>> What do you think to have 'quick fix' as modifying how driver upd= ates > > >>>>> burst ops to prevent the race condition, for this release? > > >> > > >> The 'quick fix', do you mean only update function pointer (without r= xq setting) ? > > >> Currently the PMDs which announced support "proactive error handling= mode" already > > >> do this. > > >> > > > > > > Yes. > > > I checked hns3, it does as you said, hns3_eth_dev_fp_ops_config()' > > > updates all fields in 'rte_eth_fp_ops' but only function pointer seem= s > > > changed in the driver, resulting only function pointers to be updated= . > > > > > > The discussion about race condition started with patch [1], which > > > mentions a crash because of a race condition. Later in discussions, > > > recovery event given as a sample for where the race can occur, that i= s > > > why we are here. > > > > > > But after above info, although there is race condition and a bigger > > > update (that needs application involvement) is required for recovery > > > mechanism, there is no crash and NO 'quick fix' is required for recov= ery. > > > > > > @Konstantin, @Chengwen, can you please confirm above understanding is > > > correct? > > > > Yes, that's what. > > Yes, I think with Chengwen patch the race condition problem should be fix= ed. > Though for that user has to provide a properly implemented callback. > What is not currently addressed - user can not disable this auto-recovery= procedure on his will. > So if user will not provide a proper call-back the recovery can still pro= ceed and race can happen. Ideally the user or the application should participate in the recovery to prevent more catastrophic results which may need a system reboot. Not all scenarios are recoverable, but based on implementation that could be a very small percentage. But the application awareness and participation as an end goal is a good idea nevertheless. > > > > > > > > > > > > > > > [1] > > > https://patches.dpdk.org/project/dpdk/patch/20230220060839.1267349-2-= ashok.k.kaladi@intel.com/ > > > > > >>>>> > > >>>>> And plan a design update for the next release? > > >>>> +1 on the overall approach. > > >>> > > >>> Yep, agree. > > >> > > >> Hope for better solution. > > >> And also, I notice only the openvswitch (from all open-source softwa= re which based-on DPDK) > > >> registers RTE_ETH_EVENT_INTR_RESET callback . > > >> > > >> Therefore, hope we build a recovery framework at the DPDK SDK level = and be compatible > > >> with RTE_ETH_EVENT_INTR_RESET and RTE_ETH_EVENT_ERR_RECOVERING mecha= nism. > > >> > > >>> > > >>>> > > >>>>> > > >>>>> > > >>>>>>> > > >>>>>>>>> I think driver should not rely on application for this, unles= s > > >>>>>>>>> application explicitly says (to driver) that it is handling r= ecovery, > > >>>>>>>>> right now there is no way for driver to know this. > > >>>>>>>> > > >>>>>>>> I think it is visa-versa: > > >>>>>>>> application should not enable auto-recovery if it can't meet > > >>>>>>>> per-requeststies for it (provide appropriate callback). > > >>>>>>>> > > >>>>>>> > > >>>>>>> I agree on above, we are saying similar thing in different pers= pective. > > >>>>>> > > >>>>>> Ok, that's good we are on the same page. > > >>>>>> > > >>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>>> > > >>>>>>>>>>> Also, this patch introduce a driver internal function > > >>>>>>>>>>> rte_eth_fp_ops_setup which used as an help function for PMD= . > > >>>>>>>>>>> > > >>>>>>>>>>> [1] > > >>>>>>>>>>> http://patchwork.dpdk.org/project/dpdk/patch/20230220060839= .1267349-2-ashok.k.kaladi@intel.com/ > > >>>>>>>>>>> > > >>>>>>>>>>> Fixes: eb0d471a8941 ("ethdev: add proactive error handling = mode") > > >>>>>>>>>>> Cc: stable@dpdk.org > > >>>>>>>>>>> > > >>>>>>>>>>> Signed-off-by: Chengwen Feng > > >>>>>>>>>>> --- > > >>>>>>>>>>> doc/guides/prog_guide/poll_mode_drv.rst | 20 +++++++-----= ---- > > >>>>>>>>>>> lib/ethdev/ethdev_driver.c | 8 +++++++ > > >>>>>>>>>>> lib/ethdev/ethdev_driver.h | 10 ++++++++ > > >>>>>>>>>>> lib/ethdev/rte_ethdev.h | 32 > > >>>>>>>>>>> +++++++++++++++---------- > > >>>>>>>>>>> lib/ethdev/version.map | 1 + > > >>>>>>>>>>> 5 files changed, 46 insertions(+), 25 deletions(-) > > >>>>>>>>>>> > > >>>>>>>>>>> diff --git a/doc/guides/prog_guide/poll_mode_drv.rst > > >>>>>>>>>>> b/doc/guides/prog_guide/poll_mode_drv.rst > > >>>>>>>>>>> index c145a9066c..e380ff135a 100644 > > >>>>>>>>>>> --- a/doc/guides/prog_guide/poll_mode_drv.rst > > >>>>>>>>>>> +++ b/doc/guides/prog_guide/poll_mode_drv.rst > > >>>>>>>>>>> @@ -638,14 +638,9 @@ different from the application invokes= recovery > > >>>>>>>>>>> in PASSIVE mode, > > >>>>>>>>>>> the PMD automatically recovers from error in PROACTIVE mo= de, > > >>>>>>>>>>> and only a small amount of work is required for the appli= cation. > > >>>>>>>>>>> > > >>>>>>>>>>> -During error detection and automatic recovery, > > >>>>>>>>>>> -the PMD sets the data path pointers to dummy functions > > >>>>>>>>>>> -(which will prevent the crash), > > >>>>>>>>>>> -and also make sure the control path operations fail with a= return > > >>>>>>>>>>> code ``-EBUSY``. > > >>>>>>>>>>> - > > >>>>>>>>>>> -Because the PMD recovers automatically, > > >>>>>>>>>>> -the application can only sense that the data flow is disco= nnected > > >>>>>>>>>>> for a while > > >>>>>>>>>>> -and the control API returns an error in this period. > > >>>>>>>>>>> +During error detection and automatic recovery, the PMD set= s the > > >>>>>>>>>>> data path > > >>>>>>>>>>> +pointers to dummy functions and also make sure the control= path > > >>>>>>>>>>> operations > > >>>>>>>>>>> +failed with a return code ``-EBUSY``. > > >>>>>>>>>>> > > >>>>>>>>>>> In order to sense the error happening/recovering, > > >>>>>>>>>>> as well as to restore some additional configuration, > > >>>>>>>>>>> @@ -653,9 +648,9 @@ three events are available: > > >>>>>>>>>>> > > >>>>>>>>>>> ``RTE_ETH_EVENT_ERR_RECOVERING`` > > >>>>>>>>>>> Notify the application that an error is detected > > >>>>>>>>>>> - and the recovery is being started. > > >>>>>>>>>>> + and the recovery is about to start. > > >>>>>>>>>>> Upon receiving the event, the application should not i= nvoke > > >>>>>>>>>>> - any control path function until receiving > > >>>>>>>>>>> + any control and data path API until receiving > > >>>>>>>>>>> ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` or > > >>>>>>>>>>> ``RTE_ETH_EVENT_RECOVERY_FAILED`` event. > > >>>>>>>>>>> > > >>>>>>>>>>> .. note:: > > >>>>>>>>>>> @@ -666,8 +661,9 @@ three events are available: > > >>>>>>>>>>> > > >>>>>>>>>>> ``RTE_ETH_EVENT_RECOVERY_SUCCESS`` > > >>>>>>>>>>> Notify the application that the recovery from error is= successful, > > >>>>>>>>>>> - the PMD already re-configures the port, > > >>>>>>>>>>> - and the effect is the same as a restart operation. > > >>>>>>>>>>> + the PMD already re-configures the port. > > >>>>>>>>>>> + The application should restore some additional configur= ation, > > >>>>>>>>>>> and then > > >>>>>>>>>>> + enable data path API invocation. > > >>>>>>>>>>> > > >>>>>>>>>>> ``RTE_ETH_EVENT_RECOVERY_FAILED`` > > >>>>>>>>>>> Notify the application that the recovery from error fa= iled, > > >>>>>>>>>>> diff --git a/lib/ethdev/ethdev_driver.c b/lib/ethdev/ethdev= _driver.c > > >>>>>>>>>>> index 0be1e8ca04..f994653fe9 100644 > > >>>>>>>>>>> --- a/lib/ethdev/ethdev_driver.c > > >>>>>>>>>>> +++ b/lib/ethdev/ethdev_driver.c > > >>>>>>>>>>> @@ -515,6 +515,14 @@ rte_eth_dma_zone_free(const struct rte= _eth_dev > > >>>>>>>>>>> *dev, const char *ring_name, > > >>>>>>>>>>> return rc; > > >>>>>>>>>>> } > > >>>>>>>>>>> > > >>>>>>>>>>> +void > > >>>>>>>>>>> +rte_eth_fp_ops_setup(struct rte_eth_dev *dev) > > >>>>>>>>>>> +{ > > >>>>>>>>>>> + if (dev =3D=3D NULL) > > >>>>>>>>>>> + return; > > >>>>>>>>>>> + eth_dev_fp_ops_setup(rte_eth_fp_ops + dev->data->port_= id, dev); > > >>>>>>>>>>> +} > > >>>>>>>>>>> + > > >>>>>>>>>>> const struct rte_memzone * > > >>>>>>>>>>> rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, c= onst char > > >>>>>>>>>>> *ring_name, > > >>>>>>>>>>> uint16_t queue_id, size_t size, unsigned int= align, > > >>>>>>>>>>> diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev= _driver.h > > >>>>>>>>>>> index 2c9d615fb5..0d964d1f67 100644 > > >>>>>>>>>>> --- a/lib/ethdev/ethdev_driver.h > > >>>>>>>>>>> +++ b/lib/ethdev/ethdev_driver.h > > >>>>>>>>>>> @@ -1621,6 +1621,16 @@ int > > >>>>>>>>>>> rte_eth_dma_zone_free(const struct rte_eth_dev *eth_dev, = const > > >>>>>>>>>>> char *name, > > >>>>>>>>>>> uint16_t queue_id); > > >>>>>>>>>>> > > >>>>>>>>>>> +/** > > >>>>>>>>>>> + * @internal > > >>>>>>>>>>> + * Setup eth fast-path API to ethdev values. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * @param dev > > >>>>>>>>>>> + * Pointer to struct rte_eth_dev. > > >>>>>>>>>>> + */ > > >>>>>>>>>>> +__rte_internal > > >>>>>>>>>>> +void rte_eth_fp_ops_setup(struct rte_eth_dev *dev); > > >>>>>>>>>>> + > > >>>>>>>>>>> /** > > >>>>>>>>>>> * @internal > > >>>>>>>>>>> * Atomically set the link status for the specific device= . > > >>>>>>>>>>> diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethde= v.h > > >>>>>>>>>>> index 049641d57c..44ee7229c1 100644 > > >>>>>>>>>>> --- a/lib/ethdev/rte_ethdev.h > > >>>>>>>>>>> +++ b/lib/ethdev/rte_ethdev.h > > >>>>>>>>>>> @@ -3944,25 +3944,28 @@ enum rte_eth_event_type { > > >>>>>>>>>>> */ > > >>>>>>>>>>> RTE_ETH_EVENT_RX_AVAIL_THRESH, > > >>>>>>>>>>> /** Port recovering from a hardware or firmware error= . > > >>>>>>>>>>> - * If PMD supports proactive error recovery, > > >>>>>>>>>>> - * it should trigger this event to notify application > > >>>>>>>>>>> - * that it detected an error and the recovery is being= started. > > >>>>>>>>>>> - * Upon receiving the event, the application should no= t invoke > > >>>>>>>>>>> any control path API > > >>>>>>>>>>> - * (such as rte_eth_dev_configure/rte_eth_dev_stop...)= until > > >>>>>>>>>>> receiving > > >>>>>>>>>>> - * RTE_ETH_EVENT_RECOVERY_SUCCESS or > > >>>>>>>>>>> RTE_ETH_EVENT_RECOVERY_FAILED event. > > >>>>>>>>>>> - * The PMD will set the data path pointers to dummy fu= nctions, > > >>>>>>>>>>> - * and re-set the data path pointers to non-dummy func= tions > > >>>>>>>>>>> - * before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS eve= nt. > > >>>>>>>>>>> - * It means that the application cannot send or receiv= e any > > >>>>>>>>>>> packets > > >>>>>>>>>>> - * during this period. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * If PMD supports proactive error recovery, it should= trigger > > >>>>>>>>>>> this > > >>>>>>>>>>> + * event to notify application that it detected an err= or and the > > >>>>>>>>>>> + * recovery is about to start. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * Upon receiving the event, the application should no= t invoke any > > >>>>>>>>>>> + * control and data path API until receiving > > >>>>>>>>>>> + * RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_REC= OVERY_FAILED > > >>>>>>>>>>> + * event. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * Once this event is reported, the PMD will set the d= ata path > > >>>>>>>>>>> pointers > > >>>>>>>>>>> + * to dummy functions, and re-set the data path pointe= rs to valid > > >>>>>>>>>>> + * functions before reporting RTE_ETH_EVENT_RECOVERY_S= UCCESS > > >>>>>>>>>>> event. > > >>>>>>>>>>> + * > > >>>>>>>>>>> * @note Before the PMD reports the recovery result, > > >>>>>>>>>>> * the PMD may report the RTE_ETH_EVENT_ERR_RECOVERIN= G event > > >>>>>>>>>>> again, > > >>>>>>>>>>> * because a larger error may occur during the recove= ry. > > >>>>>>>>>>> */ > > >>>>>>>>>>> RTE_ETH_EVENT_ERR_RECOVERING, > > >>>>>>>>>>> /** Port recovers successfully from the error. > > >>>>>>>>>>> - * The PMD already re-configured the port, > > >>>>>>>>>>> - * and the effect is the same as a restart operation. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * The PMD already re-configured the port: > > >>>>>>>>>>> * a) The following operation will be retained: (alph= abetically) > > >>>>>>>>>>> * - DCB configuration > > >>>>>>>>>>> * - FEC configuration > > >>>>>>>>>>> @@ -3989,6 +3992,9 @@ enum rte_eth_event_type { > > >>>>>>>>>>> * (@see RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEE= P) > > >>>>>>>>>>> * c) Any other configuration will not be stored > > >>>>>>>>>>> * and will need to be re-configured. > > >>>>>>>>>>> + * > > >>>>>>>>>>> + * The application should restore some additional conf= iguration > > >>>>>>>>>>> + * (see above case b/c), and then enable data path API= invocation. > > >>>>>>>>>>> */ > > >>>>>>>>>>> RTE_ETH_EVENT_RECOVERY_SUCCESS, > > >>>>>>>>>>> /** Port recovery failed. > > >>>>>>>>>>> diff --git a/lib/ethdev/version.map b/lib/ethdev/version.ma= p > > >>>>>>>>>>> index 357d1a88c0..c273e0bdae 100644 > > >>>>>>>>>>> --- a/lib/ethdev/version.map > > >>>>>>>>>>> +++ b/lib/ethdev/version.map > > >>>>>>>>>>> @@ -320,6 +320,7 @@ INTERNAL { > > >>>>>>>>>>> rte_eth_devices; > > >>>>>>>>>>> rte_eth_dma_zone_free; > > >>>>>>>>>>> rte_eth_dma_zone_reserve; > > >>>>>>>>>>> + rte_eth_fp_ops_setup; > > >>>>>>>>>>> rte_eth_hairpin_queue_peer_bind; > > >>>>>>>>>>> rte_eth_hairpin_queue_peer_unbind; > > >>>>>>>>>>> rte_eth_hairpin_queue_peer_update; > > >>>>>>>>>>> -- > > >>>>>>>>>> Acked-by: Konstantin Ananyev > > >>>>>>>>>> > > >>>>>>>>>>> 2.17.1 > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>> > > > > > > . > > > --000000000000fec2dd05f66e134f Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQdgYJKoZIhvcNAQcCoIIQZzCCEGMCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg3NMIIFDTCCA/WgAwIBAgIQeEqpED+lv77edQixNJMdADANBgkqhkiG9w0BAQsFADBMMSAwHgYD VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0yMDA5MTYwMDAwMDBaFw0yODA5MTYwMDAwMDBaMFsxCzAJBgNVBAYT AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBS MyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA vbCmXCcsbZ/a0fRIQMBxp4gJnnyeneFYpEtNydrZZ+GeKSMdHiDgXD1UnRSIudKo+moQ6YlCOu4t rVWO/EiXfYnK7zeop26ry1RpKtogB7/O115zultAz64ydQYLe+a1e/czkALg3sgTcOOcFZTXk38e aqsXsipoX1vsNurqPtnC27TWsA7pk4uKXscFjkeUE8JZu9BDKaswZygxBOPBQBwrA5+20Wxlk6k1 e6EKaaNaNZUy30q3ArEf30ZDpXyfCtiXnupjSK8WU2cK4qsEtj09JS4+mhi0CTCrCnXAzum3tgcH cHRg0prcSzzEUDQWoFxyuqwiwhHu3sPQNmFOMwIDAQABo4IB2jCCAdYwDgYDVR0PAQH/BAQDAgGG MGAGA1UdJQRZMFcGCCsGAQUFBwMCBggrBgEFBQcDBAYKKwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJ KwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcDBwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB /wIBADAdBgNVHQ4EFgQUljPR5lgXWzR1ioFWZNW+SN6hj88wHwYDVR0jBBgwFoAUj/BLf6guRSSu TVD6Y5qL3uLdG7wwegYIKwYBBQUHAQEEbjBsMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9i YWxzaWduLmNvbS9yb290cjMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5j b20vY2FjZXJ0L3Jvb3QtcjMuY3J0MDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFs c2lnbi5jb20vcm9vdC1yMy5jcmwwWgYDVR0gBFMwUTALBgkrBgEEAaAyASgwQgYKKwYBBAGgMgEo CjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAN BgkqhkiG9w0BAQsFAAOCAQEAdAXk/XCnDeAOd9nNEUvWPxblOQ/5o/q6OIeTYvoEvUUi2qHUOtbf jBGdTptFsXXe4RgjVF9b6DuizgYfy+cILmvi5hfk3Iq8MAZsgtW+A/otQsJvK2wRatLE61RbzkX8 9/OXEZ1zT7t/q2RiJqzpvV8NChxIj+P7WTtepPm9AIj0Keue+gS2qvzAZAY34ZZeRHgA7g5O4TPJ /oTd+4rgiU++wLDlcZYd/slFkaT3xg4qWDepEMjT4T1qFOQIL+ijUArYS4owpPg9NISTKa1qqKWJ jFoyms0d0GwOniIIbBvhI2MJ7BSY9MYtWVT5jJO3tsVHwj4cp92CSFuGwunFMzCCA18wggJHoAMC AQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUAMEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9v dCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5 MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0E XyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuul9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+J J5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJpij2aTv2y8gokeWdimFXN6x0FNx04Druci8u nPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTv riBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGj QjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5N UPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEAS0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigH M8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9ubG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmU Y/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaMld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V 14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcy a5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/fhO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/ XzCCBVUwggQ9oAMCAQICDAzZWuPidkrRZaiw2zANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTExMC8GA1UEAxMoR2xvYmFsU2lnbiBHQ0MgUjMg UGVyc29uYWxTaWduIDIgQ0EgMjAyMDAeFw0yMjA5MTAwODE4NDVaFw0yNTA5MTAwODE4NDVaMIGW MQswCQYDVQQGEwJJTjESMBAGA1UECBMJS2FybmF0YWthMRIwEAYDVQQHEwlCYW5nYWxvcmUxFjAU BgNVBAoTDUJyb2FkY29tIEluYy4xHDAaBgNVBAMTE0FqaXQgS3VtYXIgS2hhcGFyZGUxKTAnBgkq hkiG9w0BCQEWGmFqaXQua2hhcGFyZGVAYnJvYWRjb20uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOC AQ8AMIIBCgKCAQEArZ/Aqg34lMOo2BabvAa+dRThl9OeUUJMob125dz+jvS78k4NZn1mYrHu53Dn YycqjtuSMlJ6vJuwN2W6QpgTaA2SDt5xTB7CwA2urpcm7vWxxLOszkr5cxMB1QBbTd77bXFuyTqW jrer3VIWqOujJ1n+n+1SigMwEr7PKQR64YKq2aRYn74ukY3DlQdKUrm2yUkcA7aExLcAwHWUna/u pZEyqKnwS1lKCzjX7mV5W955rFsFxChdAKfw0HilwtqdY24mhy62+GeaEkD0gYIj1tCmw9gnQToc K+0s7xEunfR9pBrzmOwS3OQbcP0nJ8SmQ8R+reroH6LYuFpaqK1rgQIDAQABo4IB2zCCAdcwDgYD VR0PAQH/BAQDAgWgMIGjBggrBgEFBQcBAQSBljCBkzBOBggrBgEFBQcwAoZCaHR0cDovL3NlY3Vy ZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NnY2NyM3BlcnNvbmFsc2lnbjJjYTIwMjAuY3J0MEEG CCsGAQUFBzABhjVodHRwOi8vb2NzcC5nbG9iYWxzaWduLmNvbS9nc2djY3IzcGVyc29uYWxzaWdu MmNhMjAyMDBNBgNVHSAERjBEMEIGCisGAQQBoDIBKAowNDAyBggrBgEFBQcCARYmaHR0cHM6Ly93 d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wCQYDVR0TBAIwADBJBgNVHR8EQjBAMD6gPKA6 hjhodHRwOi8vY3JsLmdsb2JhbHNpZ24uY29tL2dzZ2NjcjNwZXJzb25hbHNpZ24yY2EyMDIwLmNy bDAlBgNVHREEHjAcgRphaml0LmtoYXBhcmRlQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEF BQcDBDAfBgNVHSMEGDAWgBSWM9HmWBdbNHWKgVZk1b5I3qGPzzAdBgNVHQ4EFgQUbrcTuh0mr2qP xYdtyDgFeRIiE/gwDQYJKoZIhvcNAQELBQADggEBALrc1TljKrDhXicOaZlzIQyqOEkKAZ324i8X OwzA0n2EcPGmMZvgARurvanSLD3mLeeuyq1feCcjfGM1CJFh4+EY7EkbFbpVPOIdstSBhbnAJnOl aC/q0wTndKoC/xXBhXOZB8YL/Zq4ZclQLMUO6xi/fFRyHviI5/IrosdrpniXFJ9ukJoOXtvdrEF+ KlMYg/Deg9xo3wddCqQIsztHSkR4XaANdn+dbLRQpctZ13BY1lim4uz5bYn3M0IxyZWkQ1JuPHCK aRJv0SfR88PoI4RB7NCEHqFwARTj1KvFPQi8pK/YISFydZYbZrxQdyWDidqm4wSuJfpE6i0cWvCd u50xggJtMIICaQIBATBrMFsxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNh MTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBSMyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwAgwM2Vrj 4nZK0WWosNswDQYJYIZIAWUDBAIBBQCggdQwLwYJKoZIhvcNAQkEMSIEIE/IqNeoJ2Xj46It+dst fs4I4W/PL0iVgKG87WXNH7ruMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF MQ8XDTIzMDMwOTAyMDUyNVowaQYJKoZIhvcNAQkPMVwwWjALBglghkgBZQMEASowCwYJYIZIAWUD BAEWMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzALBgkqhkiG9w0BAQowCwYJKoZIhvcNAQEHMAsG CWCGSAFlAwQCATANBgkqhkiG9w0BAQEFAASCAQB3lft3GQN9Mr8dNFD7723RRDyFQTEXJwIDVNpU wLpzeHLbKQZmKvp6bPEFllPrQ9cthM0H7jVM27uFOFjHPeFbbWxvZT8Ofc8GD9wSwdtGdt8uQBh8 oUNJWTvrgsrTJEAlyGD/DKIeM9/c9LRSWn/kZN8rEEcDgjnBJoNn8JeNMzflO2oLzDhpO3b4Sh0c SHVo8GOQSvwrCxPkV+DyNC9cwRHdGrAHB+dFWk0u/3aptKGGYXUCDjwP+PArQJSKXb15ngQdMCg3 v1tNLMWxM+AL6v2Vknlh41xXVQ3QGuAnWR0io3pQxF8IWKqZVSVqmfXzZl28QWI/CANcFBKfWjB1 --000000000000fec2dd05f66e134f--