* [dpdk-dev] [PATCH 0/3] bnxt patches @ 2020-12-24 6:31 Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur ` (3 more replies) 0 siblings, 4 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 6:31 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur Some fixes and enchancements in the core bnxt PMD Somnath Kotur (3): net/bnxt: fix to init/destroy locks only once net/bnxt: fix error path handling of dev start op net/bnxt: check for chip reset in dev stop/close ops drivers/net/bnxt/bnxt.h | 5 + drivers/net/bnxt/bnxt_cpr.c | 2 + drivers/net/bnxt/bnxt_ethdev.c | 225 +++++++++++++++++++-------------- 3 files changed, 135 insertions(+), 97 deletions(-) -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once 2020-12-24 6:31 [dpdk-dev] [PATCH 0/3] bnxt patches Somnath Kotur @ 2020-12-24 6:31 ` Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur ` (2 subsequent siblings) 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 6:31 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur, stable Invoking init/uninit locks in init_resources and uninit_resources would end up initializing and destroying locks on every port start stop which is not desired. Move the 2 routines to dev_init and dev_close respectively as locks need to be initialized and destroyed only once during the lifetime of the driver. Fixes: 1cb3d39a48f7 ("net/bnxt: synchronize between flow related functions") Cc: stable@dpdk.org Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt_ethdev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 0788d263d8..40e15c69d3 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1419,6 +1419,18 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +static void +bnxt_uninit_locks(struct bnxt *bp) +{ + pthread_mutex_destroy(&bp->flow_lock); + pthread_mutex_destroy(&bp->def_cp_lock); + pthread_mutex_destroy(&bp->health_check_lock); + if (bp->rep_info) { + pthread_mutex_destroy(&bp->rep_info->vfr_lock); + pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); + } +} + static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1444,6 +1456,7 @@ static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) bnxt_free_link_info(bp); bnxt_free_pf_info(bp); bnxt_free_parent_info(bp); + bnxt_uninit_locks(bp); rte_memzone_free((const struct rte_memzone *)bp->tx_mem_zone); bp->tx_mem_zone = NULL; @@ -4790,10 +4803,6 @@ static int bnxt_init_resources(struct bnxt *bp, bool reconfig_dev) return rc; } - rc = bnxt_init_locks(bp); - if (rc) - return rc; - return 0; } @@ -5280,6 +5289,10 @@ bnxt_dev_init(struct rte_eth_dev *eth_dev, void *params __rte_unused) if (rc) goto error_free; + rc = bnxt_init_locks(bp); + if (rc) + return rc; + rc = bnxt_init_resources(bp, false); if (rc) goto error_free; @@ -5371,18 +5384,6 @@ bnxt_free_error_recovery_info(struct bnxt *bp) bp->fw_cap &= ~BNXT_FW_CAP_ERROR_RECOVERY; } -static void -bnxt_uninit_locks(struct bnxt *bp) -{ - pthread_mutex_destroy(&bp->flow_lock); - pthread_mutex_destroy(&bp->def_cp_lock); - pthread_mutex_destroy(&bp->health_check_lock); - if (bp->rep_info) { - pthread_mutex_destroy(&bp->rep_info->vfr_lock); - pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); - } -} - static int bnxt_uninit_resources(struct bnxt *bp, bool reconfig_dev) { @@ -5404,7 +5405,6 @@ bnxt_uninit_resources(struct bnxt *bp, bool reconfig_dev) bnxt_uninit_ctx_mem(bp); - bnxt_uninit_locks(bp); bnxt_free_flow_stats_info(bp); bnxt_free_rep_info(bp); rte_free(bp->ptp_cfg); -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op 2020-12-24 6:31 [dpdk-dev] [PATCH 0/3] bnxt patches Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur @ 2020-12-24 6:31 ` Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 6:31 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur, stable Call bnxt_dev_stop in error path of bnxt_dev_start_op() to keep it simple and consistent Fixes: c09f57b49c13 ("net/bnxt: add start/stop/link update operations") Cc: stable@dpdk.org Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt_ethdev.c | 144 ++++++++++++++++----------------- 1 file changed, 70 insertions(+), 74 deletions(-) diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 40e15c69d3..17dc49628d 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1239,80 +1239,6 @@ static int bnxt_handle_if_change_status(struct bnxt *bp) return rc; } -static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) -{ - struct bnxt *bp = eth_dev->data->dev_private; - uint64_t rx_offloads = eth_dev->data->dev_conf.rxmode.offloads; - int vlan_mask = 0; - int rc, retry_cnt = BNXT_IF_CHANGE_RETRY_COUNT; - - if (!eth_dev->data->nb_tx_queues || !eth_dev->data->nb_rx_queues) { - PMD_DRV_LOG(ERR, "Queues are not configured yet!\n"); - return -EINVAL; - } - - if (bp->rx_cp_nr_rings > RTE_ETHDEV_QUEUE_STAT_CNTRS) - PMD_DRV_LOG(ERR, - "RxQ cnt %d > RTE_ETHDEV_QUEUE_STAT_CNTRS %d\n", - bp->rx_cp_nr_rings, RTE_ETHDEV_QUEUE_STAT_CNTRS); - - do { - rc = bnxt_hwrm_if_change(bp, true); - if (rc == 0 || rc != -EAGAIN) - break; - - rte_delay_ms(BNXT_IF_CHANGE_RETRY_INTERVAL); - } while (retry_cnt--); - - if (rc) - return rc; - - if (bp->flags & BNXT_FLAG_IF_CHANGE_HOT_FW_RESET_DONE) { - rc = bnxt_handle_if_change_status(bp); - if (rc) - return rc; - } - - bnxt_enable_int(bp); - - rc = bnxt_init_chip(bp); - if (rc) - goto error; - - eth_dev->data->scattered_rx = bnxt_scattered_rx(eth_dev); - eth_dev->data->dev_started = 1; - - bnxt_link_update_op(eth_dev, 1); - - if (rx_offloads & DEV_RX_OFFLOAD_VLAN_FILTER) - vlan_mask |= ETH_VLAN_FILTER_MASK; - if (rx_offloads & DEV_RX_OFFLOAD_VLAN_STRIP) - vlan_mask |= ETH_VLAN_STRIP_MASK; - rc = bnxt_vlan_offload_set_op(eth_dev, vlan_mask); - if (rc) - goto error; - - /* Initialize bnxt ULP port details */ - rc = bnxt_ulp_port_init(bp); - if (rc) - goto error; - - eth_dev->rx_pkt_burst = bnxt_receive_function(eth_dev); - eth_dev->tx_pkt_burst = bnxt_transmit_function(eth_dev); - - bnxt_schedule_fw_health_check(bp); - - return 0; - -error: - bnxt_shutdown_nic(bp); - bnxt_free_tx_mbufs(bp); - bnxt_free_rx_mbufs(bp); - bnxt_hwrm_if_change(bp, false); - eth_dev->data->dev_started = 0; - return rc; -} - static int bnxt_dev_set_link_up_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1419,6 +1345,76 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) +{ + struct bnxt *bp = eth_dev->data->dev_private; + uint64_t rx_offloads = eth_dev->data->dev_conf.rxmode.offloads; + int vlan_mask = 0; + int rc, retry_cnt = BNXT_IF_CHANGE_RETRY_COUNT; + + if (!eth_dev->data->nb_tx_queues || !eth_dev->data->nb_rx_queues) { + PMD_DRV_LOG(ERR, "Queues are not configured yet!\n"); + return -EINVAL; + } + + if (bp->rx_cp_nr_rings > RTE_ETHDEV_QUEUE_STAT_CNTRS) + PMD_DRV_LOG(ERR, + "RxQ cnt %d > RTE_ETHDEV_QUEUE_STAT_CNTRS %d\n", + bp->rx_cp_nr_rings, RTE_ETHDEV_QUEUE_STAT_CNTRS); + + do { + rc = bnxt_hwrm_if_change(bp, true); + if (rc == 0 || rc != -EAGAIN) + break; + + rte_delay_ms(BNXT_IF_CHANGE_RETRY_INTERVAL); + } while (retry_cnt--); + + if (rc) + return rc; + + if (bp->flags & BNXT_FLAG_IF_CHANGE_HOT_FW_RESET_DONE) { + rc = bnxt_handle_if_change_status(bp); + if (rc) + return rc; + } + + bnxt_enable_int(bp); + + rc = bnxt_init_chip(bp); + if (rc) + goto error; + + eth_dev->data->scattered_rx = bnxt_scattered_rx(eth_dev); + eth_dev->data->dev_started = 1; + + bnxt_link_update_op(eth_dev, 1); + + if (rx_offloads & DEV_RX_OFFLOAD_VLAN_FILTER) + vlan_mask |= ETH_VLAN_FILTER_MASK; + if (rx_offloads & DEV_RX_OFFLOAD_VLAN_STRIP) + vlan_mask |= ETH_VLAN_STRIP_MASK; + rc = bnxt_vlan_offload_set_op(eth_dev, vlan_mask); + if (rc) + goto error; + + /* Initialize bnxt ULP port details */ + rc = bnxt_ulp_port_init(bp); + if (rc) + goto error; + + eth_dev->rx_pkt_burst = bnxt_receive_function(eth_dev); + eth_dev->tx_pkt_burst = bnxt_transmit_function(eth_dev); + + bnxt_schedule_fw_health_check(bp); + + return 0; + +error: + bnxt_dev_stop_op(eth_dev); + return rc; +} + static void bnxt_uninit_locks(struct bnxt *bp) { -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops 2020-12-24 6:31 [dpdk-dev] [PATCH 0/3] bnxt patches Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur @ 2020-12-24 6:31 ` Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 6:31 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur While the error recovery thread is running, an application can invoke dev_stop or dev_close_op thus triggering a race and unwanted consequences if dev_close is invoked while the recovery is not yet completed. Fix by having another lock to synchronize between the 2 threads and return EGAIN if adapter is in the middle of recovery when dev_stop or dev_close ops are invoked Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt.h | 5 ++++ drivers/net/bnxt/bnxt_cpr.c | 2 ++ drivers/net/bnxt/bnxt_ethdev.c | 49 +++++++++++++++++++++++++++++----- 3 files changed, 49 insertions(+), 7 deletions(-) diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h index 8374e9fadc..7c135370f0 100644 --- a/drivers/net/bnxt/bnxt.h +++ b/drivers/net/bnxt/bnxt.h @@ -719,6 +719,11 @@ struct bnxt { * health_check_lock */ pthread_mutex_t health_check_lock; + /* synchronize between dev_stop/dev_close_op and + * error recovery thread triggered as part of + * HWRM_ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY + */ + pthread_mutex_t err_recovery_lock; uint16_t max_req_len; uint16_t max_resp_len; uint16_t hwrm_max_ext_req_len; diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index ee96ae81bf..6e172a9eea 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -133,6 +133,7 @@ void bnxt_handle_async_event(struct bnxt *bp, return; } + pthread_mutex_lock(&bp->err_recovery_lock); event_data = rte_le_to_cpu_32(async_cmp->event_data1); /* timestamp_lo/hi values are in units of 100ms */ bp->fw_reset_max_msecs = async_cmp->timestamp_hi ? @@ -152,6 +153,7 @@ void bnxt_handle_async_event(struct bnxt *bp, } bp->flags |= BNXT_FLAG_FW_RESET; + pthread_mutex_unlock(&bp->err_recovery_lock); rte_eal_alarm_set(US_PER_MS, bnxt_dev_reset_and_resume, (void *)bp); break; diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 17dc49628d..0bd0762859 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1276,8 +1276,7 @@ static void bnxt_free_switch_domain(struct bnxt *bp) } } -/* Unload the driver, release resources */ -static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) +static int bnxt_dev_stop(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev); @@ -1345,6 +1344,22 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +/* Unload the driver, release resources */ +static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) +{ + struct bnxt *bp = eth_dev->data->dev_private; + + pthread_mutex_lock(&bp->err_recovery_lock); + if (bp->flags & BNXT_FLAG_FW_RESET) { + PMD_DRV_LOG(ERR, + "Adapter recovering from error..Please retry\n"); + return -EAGAIN; + } + pthread_mutex_unlock(&bp->err_recovery_lock); + + return bnxt_dev_stop(eth_dev); +} + static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1411,7 +1426,7 @@ static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) return 0; error: - bnxt_dev_stop_op(eth_dev); + bnxt_dev_stop(eth_dev); return rc; } @@ -1421,6 +1436,7 @@ bnxt_uninit_locks(struct bnxt *bp) pthread_mutex_destroy(&bp->flow_lock); pthread_mutex_destroy(&bp->def_cp_lock); pthread_mutex_destroy(&bp->health_check_lock); + pthread_mutex_destroy(&bp->err_recovery_lock); if (bp->rep_info) { pthread_mutex_destroy(&bp->rep_info->vfr_lock); pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); @@ -1435,13 +1451,21 @@ static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; + pthread_mutex_lock(&bp->err_recovery_lock); + if (bp->flags & BNXT_FLAG_FW_RESET) { + PMD_DRV_LOG(ERR, + "Adapter recovering from error...Please retry\n"); + return -EAGAIN; + } + pthread_mutex_unlock(&bp->err_recovery_lock); + /* cancel the recovery handler before remove dev */ rte_eal_alarm_cancel(bnxt_dev_reset_and_resume, (void *)bp); rte_eal_alarm_cancel(bnxt_dev_recover, (void *)bp); bnxt_cancel_fc_thread(bp); if (eth_dev->data->dev_started) - ret = bnxt_dev_stop_op(eth_dev); + ret = bnxt_dev_stop(eth_dev); bnxt_free_switch_domain(bp); @@ -3655,7 +3679,7 @@ static void bnxt_dev_cleanup(struct bnxt *bp) bp->eth_dev->data->dev_link.link_status = 0; bp->link_info->link_up = 0; if (bp->eth_dev->data->dev_started) - bnxt_dev_stop_op(bp->eth_dev); + bnxt_dev_stop(bp->eth_dev); bnxt_uninit_resources(bp, true); } @@ -3756,6 +3780,7 @@ static void bnxt_dev_recover(void *arg) int timeout = bp->fw_reset_max_msecs; int rc = 0; + pthread_mutex_lock(&bp->err_recovery_lock); /* Clear Error flag so that device re-init should happen */ bp->flags &= ~BNXT_FLAG_FATAL_ERROR; @@ -3792,12 +3817,15 @@ static void bnxt_dev_recover(void *arg) goto err_start; PMD_DRV_LOG(INFO, "Recovered from FW reset\n"); + pthread_mutex_unlock(&bp->err_recovery_lock); + return; err_start: - bnxt_dev_stop_op(bp->eth_dev); + bnxt_dev_stop(bp->eth_dev); err: bp->flags |= BNXT_FLAG_FATAL_ERROR; bnxt_uninit_resources(bp, false); + pthread_mutex_unlock(&bp->err_recovery_lock); PMD_DRV_LOG(ERR, "Failed to recover from FW reset\n"); } @@ -4733,8 +4761,15 @@ bnxt_init_locks(struct bnxt *bp) } err = pthread_mutex_init(&bp->health_check_lock, NULL); - if (err) + if (err) { PMD_DRV_LOG(ERR, "Unable to initialize health_check_lock\n"); + return err; + } + + err = pthread_mutex_init(&bp->err_recovery_lock, NULL); + if (err) + PMD_DRV_LOG(ERR, "Unable to initialize err_recovery_lock\n"); + return err; } -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH v2 0/3] bnxt patches 2020-12-24 6:31 [dpdk-dev] [PATCH 0/3] bnxt patches Somnath Kotur ` (2 preceding siblings ...) 2020-12-24 6:31 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur @ 2020-12-24 9:37 ` Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur ` (3 more replies) 3 siblings, 4 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 9:37 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur A couple of bnxt PMD fixes and an enhancement Somnath Kotur (3): net/bnxt: fix to init/destroy locks only once net/bnxt: fix error path handling of dev start op net/bnxt: check for chip reset in dev stop/close ops drivers/net/bnxt/bnxt.h | 5 + drivers/net/bnxt/bnxt_cpr.c | 2 + drivers/net/bnxt/bnxt_ethdev.c | 225 +++++++++++++++++++-------------- 3 files changed, 135 insertions(+), 97 deletions(-) -- v1->v2: Modified patch 1/3 to return error 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur @ 2020-12-24 9:37 ` Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur ` (2 subsequent siblings) 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 9:37 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur Invoking init/uninit locks in init_resources and uninit_resources would end up initializing and destroying locks on every port start stop which is not desired. Move the 2 routines to dev_init and dev_close respectively as locks need to be initialized and destroyed only once during the lifetime of the driver. Fixes: 1cb3d39a48f7 ("net/bnxt: synchronize between flow related functions") Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt_ethdev.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 0788d263d8..6c38155da6 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1419,6 +1419,18 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +static void +bnxt_uninit_locks(struct bnxt *bp) +{ + pthread_mutex_destroy(&bp->flow_lock); + pthread_mutex_destroy(&bp->def_cp_lock); + pthread_mutex_destroy(&bp->health_check_lock); + if (bp->rep_info) { + pthread_mutex_destroy(&bp->rep_info->vfr_lock); + pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); + } +} + static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1444,6 +1456,7 @@ static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) bnxt_free_link_info(bp); bnxt_free_pf_info(bp); bnxt_free_parent_info(bp); + bnxt_uninit_locks(bp); rte_memzone_free((const struct rte_memzone *)bp->tx_mem_zone); bp->tx_mem_zone = NULL; @@ -4790,10 +4803,6 @@ static int bnxt_init_resources(struct bnxt *bp, bool reconfig_dev) return rc; } - rc = bnxt_init_locks(bp); - if (rc) - return rc; - return 0; } @@ -5280,6 +5289,10 @@ bnxt_dev_init(struct rte_eth_dev *eth_dev, void *params __rte_unused) if (rc) goto error_free; + rc = bnxt_init_locks(bp); + if (rc) + goto error_free; + rc = bnxt_init_resources(bp, false); if (rc) goto error_free; @@ -5371,18 +5384,6 @@ bnxt_free_error_recovery_info(struct bnxt *bp) bp->fw_cap &= ~BNXT_FW_CAP_ERROR_RECOVERY; } -static void -bnxt_uninit_locks(struct bnxt *bp) -{ - pthread_mutex_destroy(&bp->flow_lock); - pthread_mutex_destroy(&bp->def_cp_lock); - pthread_mutex_destroy(&bp->health_check_lock); - if (bp->rep_info) { - pthread_mutex_destroy(&bp->rep_info->vfr_lock); - pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); - } -} - static int bnxt_uninit_resources(struct bnxt *bp, bool reconfig_dev) { @@ -5404,7 +5405,6 @@ bnxt_uninit_resources(struct bnxt *bp, bool reconfig_dev) bnxt_uninit_ctx_mem(bp); - bnxt_uninit_locks(bp); bnxt_free_flow_stats_info(bp); bnxt_free_rep_info(bp); rte_free(bp->ptp_cfg); -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur @ 2020-12-24 9:37 ` Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur 2021-01-12 3:41 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Ajit Khaparde 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 9:37 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur Call bnxt_dev_stop in error path of bnxt_dev_start_op() to keep it simple and consistent Fixes: c09f57b49c13 ("net/bnxt: add start/stop/link update operations") Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt_ethdev.c | 144 ++++++++++++++++----------------- 1 file changed, 70 insertions(+), 74 deletions(-) diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index 6c38155da6..b6a9db1b66 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1239,80 +1239,6 @@ static int bnxt_handle_if_change_status(struct bnxt *bp) return rc; } -static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) -{ - struct bnxt *bp = eth_dev->data->dev_private; - uint64_t rx_offloads = eth_dev->data->dev_conf.rxmode.offloads; - int vlan_mask = 0; - int rc, retry_cnt = BNXT_IF_CHANGE_RETRY_COUNT; - - if (!eth_dev->data->nb_tx_queues || !eth_dev->data->nb_rx_queues) { - PMD_DRV_LOG(ERR, "Queues are not configured yet!\n"); - return -EINVAL; - } - - if (bp->rx_cp_nr_rings > RTE_ETHDEV_QUEUE_STAT_CNTRS) - PMD_DRV_LOG(ERR, - "RxQ cnt %d > RTE_ETHDEV_QUEUE_STAT_CNTRS %d\n", - bp->rx_cp_nr_rings, RTE_ETHDEV_QUEUE_STAT_CNTRS); - - do { - rc = bnxt_hwrm_if_change(bp, true); - if (rc == 0 || rc != -EAGAIN) - break; - - rte_delay_ms(BNXT_IF_CHANGE_RETRY_INTERVAL); - } while (retry_cnt--); - - if (rc) - return rc; - - if (bp->flags & BNXT_FLAG_IF_CHANGE_HOT_FW_RESET_DONE) { - rc = bnxt_handle_if_change_status(bp); - if (rc) - return rc; - } - - bnxt_enable_int(bp); - - rc = bnxt_init_chip(bp); - if (rc) - goto error; - - eth_dev->data->scattered_rx = bnxt_scattered_rx(eth_dev); - eth_dev->data->dev_started = 1; - - bnxt_link_update_op(eth_dev, 1); - - if (rx_offloads & DEV_RX_OFFLOAD_VLAN_FILTER) - vlan_mask |= ETH_VLAN_FILTER_MASK; - if (rx_offloads & DEV_RX_OFFLOAD_VLAN_STRIP) - vlan_mask |= ETH_VLAN_STRIP_MASK; - rc = bnxt_vlan_offload_set_op(eth_dev, vlan_mask); - if (rc) - goto error; - - /* Initialize bnxt ULP port details */ - rc = bnxt_ulp_port_init(bp); - if (rc) - goto error; - - eth_dev->rx_pkt_burst = bnxt_receive_function(eth_dev); - eth_dev->tx_pkt_burst = bnxt_transmit_function(eth_dev); - - bnxt_schedule_fw_health_check(bp); - - return 0; - -error: - bnxt_shutdown_nic(bp); - bnxt_free_tx_mbufs(bp); - bnxt_free_rx_mbufs(bp); - bnxt_hwrm_if_change(bp, false); - eth_dev->data->dev_started = 0; - return rc; -} - static int bnxt_dev_set_link_up_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1419,6 +1345,76 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) +{ + struct bnxt *bp = eth_dev->data->dev_private; + uint64_t rx_offloads = eth_dev->data->dev_conf.rxmode.offloads; + int vlan_mask = 0; + int rc, retry_cnt = BNXT_IF_CHANGE_RETRY_COUNT; + + if (!eth_dev->data->nb_tx_queues || !eth_dev->data->nb_rx_queues) { + PMD_DRV_LOG(ERR, "Queues are not configured yet!\n"); + return -EINVAL; + } + + if (bp->rx_cp_nr_rings > RTE_ETHDEV_QUEUE_STAT_CNTRS) + PMD_DRV_LOG(ERR, + "RxQ cnt %d > RTE_ETHDEV_QUEUE_STAT_CNTRS %d\n", + bp->rx_cp_nr_rings, RTE_ETHDEV_QUEUE_STAT_CNTRS); + + do { + rc = bnxt_hwrm_if_change(bp, true); + if (rc == 0 || rc != -EAGAIN) + break; + + rte_delay_ms(BNXT_IF_CHANGE_RETRY_INTERVAL); + } while (retry_cnt--); + + if (rc) + return rc; + + if (bp->flags & BNXT_FLAG_IF_CHANGE_HOT_FW_RESET_DONE) { + rc = bnxt_handle_if_change_status(bp); + if (rc) + return rc; + } + + bnxt_enable_int(bp); + + rc = bnxt_init_chip(bp); + if (rc) + goto error; + + eth_dev->data->scattered_rx = bnxt_scattered_rx(eth_dev); + eth_dev->data->dev_started = 1; + + bnxt_link_update_op(eth_dev, 1); + + if (rx_offloads & DEV_RX_OFFLOAD_VLAN_FILTER) + vlan_mask |= ETH_VLAN_FILTER_MASK; + if (rx_offloads & DEV_RX_OFFLOAD_VLAN_STRIP) + vlan_mask |= ETH_VLAN_STRIP_MASK; + rc = bnxt_vlan_offload_set_op(eth_dev, vlan_mask); + if (rc) + goto error; + + /* Initialize bnxt ULP port details */ + rc = bnxt_ulp_port_init(bp); + if (rc) + goto error; + + eth_dev->rx_pkt_burst = bnxt_receive_function(eth_dev); + eth_dev->tx_pkt_burst = bnxt_transmit_function(eth_dev); + + bnxt_schedule_fw_health_check(bp); + + return 0; + +error: + bnxt_dev_stop_op(eth_dev); + return rc; +} + static void bnxt_uninit_locks(struct bnxt *bp) { -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur @ 2020-12-24 9:37 ` Somnath Kotur 2021-01-12 3:41 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Ajit Khaparde 3 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 9:37 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur While the error recovery thread is running, an application can invoke dev_stop or dev_close_op thus triggering a race and unwanted consequences if dev_close is invoked while the recovery is not yet completed. Fix by having another lock to synchronize between the 2 threads and return EGAIN if adapter is in the middle of recovery when dev_stop or dev_close ops are invoked Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com> --- drivers/net/bnxt/bnxt.h | 5 ++++ drivers/net/bnxt/bnxt_cpr.c | 2 ++ drivers/net/bnxt/bnxt_ethdev.c | 49 +++++++++++++++++++++++++++++----- 3 files changed, 49 insertions(+), 7 deletions(-) diff --git a/drivers/net/bnxt/bnxt.h b/drivers/net/bnxt/bnxt.h index 8374e9fadc..7c135370f0 100644 --- a/drivers/net/bnxt/bnxt.h +++ b/drivers/net/bnxt/bnxt.h @@ -719,6 +719,11 @@ struct bnxt { * health_check_lock */ pthread_mutex_t health_check_lock; + /* synchronize between dev_stop/dev_close_op and + * error recovery thread triggered as part of + * HWRM_ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY + */ + pthread_mutex_t err_recovery_lock; uint16_t max_req_len; uint16_t max_resp_len; uint16_t hwrm_max_ext_req_len; diff --git a/drivers/net/bnxt/bnxt_cpr.c b/drivers/net/bnxt/bnxt_cpr.c index ee96ae81bf..6e172a9eea 100644 --- a/drivers/net/bnxt/bnxt_cpr.c +++ b/drivers/net/bnxt/bnxt_cpr.c @@ -133,6 +133,7 @@ void bnxt_handle_async_event(struct bnxt *bp, return; } + pthread_mutex_lock(&bp->err_recovery_lock); event_data = rte_le_to_cpu_32(async_cmp->event_data1); /* timestamp_lo/hi values are in units of 100ms */ bp->fw_reset_max_msecs = async_cmp->timestamp_hi ? @@ -152,6 +153,7 @@ void bnxt_handle_async_event(struct bnxt *bp, } bp->flags |= BNXT_FLAG_FW_RESET; + pthread_mutex_unlock(&bp->err_recovery_lock); rte_eal_alarm_set(US_PER_MS, bnxt_dev_reset_and_resume, (void *)bp); break; diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index b6a9db1b66..a6794a417d 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -1276,8 +1276,7 @@ static void bnxt_free_switch_domain(struct bnxt *bp) } } -/* Unload the driver, release resources */ -static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) +static int bnxt_dev_stop(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev); @@ -1345,6 +1344,22 @@ static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) return 0; } +/* Unload the driver, release resources */ +static int bnxt_dev_stop_op(struct rte_eth_dev *eth_dev) +{ + struct bnxt *bp = eth_dev->data->dev_private; + + pthread_mutex_lock(&bp->err_recovery_lock); + if (bp->flags & BNXT_FLAG_FW_RESET) { + PMD_DRV_LOG(ERR, + "Adapter recovering from error..Please retry\n"); + return -EAGAIN; + } + pthread_mutex_unlock(&bp->err_recovery_lock); + + return bnxt_dev_stop(eth_dev); +} + static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) { struct bnxt *bp = eth_dev->data->dev_private; @@ -1411,7 +1426,7 @@ static int bnxt_dev_start_op(struct rte_eth_dev *eth_dev) return 0; error: - bnxt_dev_stop_op(eth_dev); + bnxt_dev_stop(eth_dev); return rc; } @@ -1421,6 +1436,7 @@ bnxt_uninit_locks(struct bnxt *bp) pthread_mutex_destroy(&bp->flow_lock); pthread_mutex_destroy(&bp->def_cp_lock); pthread_mutex_destroy(&bp->health_check_lock); + pthread_mutex_destroy(&bp->err_recovery_lock); if (bp->rep_info) { pthread_mutex_destroy(&bp->rep_info->vfr_lock); pthread_mutex_destroy(&bp->rep_info->vfr_start_lock); @@ -1435,13 +1451,21 @@ static int bnxt_dev_close_op(struct rte_eth_dev *eth_dev) if (rte_eal_process_type() != RTE_PROC_PRIMARY) return 0; + pthread_mutex_lock(&bp->err_recovery_lock); + if (bp->flags & BNXT_FLAG_FW_RESET) { + PMD_DRV_LOG(ERR, + "Adapter recovering from error...Please retry\n"); + return -EAGAIN; + } + pthread_mutex_unlock(&bp->err_recovery_lock); + /* cancel the recovery handler before remove dev */ rte_eal_alarm_cancel(bnxt_dev_reset_and_resume, (void *)bp); rte_eal_alarm_cancel(bnxt_dev_recover, (void *)bp); bnxt_cancel_fc_thread(bp); if (eth_dev->data->dev_started) - ret = bnxt_dev_stop_op(eth_dev); + ret = bnxt_dev_stop(eth_dev); bnxt_free_switch_domain(bp); @@ -3655,7 +3679,7 @@ static void bnxt_dev_cleanup(struct bnxt *bp) bp->eth_dev->data->dev_link.link_status = 0; bp->link_info->link_up = 0; if (bp->eth_dev->data->dev_started) - bnxt_dev_stop_op(bp->eth_dev); + bnxt_dev_stop(bp->eth_dev); bnxt_uninit_resources(bp, true); } @@ -3756,6 +3780,7 @@ static void bnxt_dev_recover(void *arg) int timeout = bp->fw_reset_max_msecs; int rc = 0; + pthread_mutex_lock(&bp->err_recovery_lock); /* Clear Error flag so that device re-init should happen */ bp->flags &= ~BNXT_FLAG_FATAL_ERROR; @@ -3792,12 +3817,15 @@ static void bnxt_dev_recover(void *arg) goto err_start; PMD_DRV_LOG(INFO, "Recovered from FW reset\n"); + pthread_mutex_unlock(&bp->err_recovery_lock); + return; err_start: - bnxt_dev_stop_op(bp->eth_dev); + bnxt_dev_stop(bp->eth_dev); err: bp->flags |= BNXT_FLAG_FATAL_ERROR; bnxt_uninit_resources(bp, false); + pthread_mutex_unlock(&bp->err_recovery_lock); PMD_DRV_LOG(ERR, "Failed to recover from FW reset\n"); } @@ -4733,8 +4761,15 @@ bnxt_init_locks(struct bnxt *bp) } err = pthread_mutex_init(&bp->health_check_lock, NULL); - if (err) + if (err) { PMD_DRV_LOG(ERR, "Unable to initialize health_check_lock\n"); + return err; + } + + err = pthread_mutex_init(&bp->err_recovery_lock, NULL); + if (err) + PMD_DRV_LOG(ERR, "Unable to initialize err_recovery_lock\n"); + return err; } -- 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [dpdk-dev] [PATCH v2 0/3] bnxt patches 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur ` (2 preceding siblings ...) 2020-12-24 9:37 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur @ 2021-01-12 3:41 ` Ajit Khaparde 3 siblings, 0 replies; 10+ messages in thread From: Ajit Khaparde @ 2021-01-12 3:41 UTC (permalink / raw) To: Somnath Kotur; +Cc: dev, Ferruh Yigit On Thu, Dec 24, 2020 at 1:46 AM Somnath Kotur <somnath.kotur@broadcom.com> wrote: > > A couple of bnxt PMD fixes and an enhancement > > Somnath Kotur (3): > net/bnxt: fix to init/destroy locks only once > net/bnxt: fix error path handling of dev start op > net/bnxt: check for chip reset in dev stop/close ops Patchset applied to dpdk-next-net-brcm. Thanks > > drivers/net/bnxt/bnxt.h | 5 + > drivers/net/bnxt/bnxt_cpr.c | 2 + > drivers/net/bnxt/bnxt_ethdev.c | 225 +++++++++++++++++++-------------- > 3 files changed, 135 insertions(+), 97 deletions(-) > > -- > v1->v2: Modified patch 1/3 to return error > 2.28.0.497.g54e85e7 > ^ permalink raw reply [flat|nested] 10+ messages in thread
* [dpdk-dev] [PATCH v2 0/3] bnxt patches @ 2020-12-24 9:35 Somnath Kotur 0 siblings, 0 replies; 10+ messages in thread From: Somnath Kotur @ 2020-12-24 9:35 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, Somnath Kotur A couple of bnxt PMD fixes and an enhancement Somnath Kotur (3): net/bnxt: fix to init/destroy locks only once net/bnxt: fix error path handling of dev start op net/bnxt: check for chip reset in dev stop/close ops drivers/net/bnxt/bnxt.h | 5 + drivers/net/bnxt/bnxt_cpr.c | 2 + drivers/net/bnxt/bnxt_ethdev.c | 225 +++++++++++++++++++-------------- 3 files changed, 135 insertions(+), 97 deletions(-) -- v1->v2: Modified patch 1/3 to return error 2.28.0.497.g54e85e7 ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-01-12 3:42 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-12-24 6:31 [dpdk-dev] [PATCH 0/3] bnxt patches Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur 2020-12-24 6:31 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 1/3] net/bnxt: fix to init/destroy locks only once Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 2/3] net/bnxt: fix error path handling of dev start op Somnath Kotur 2020-12-24 9:37 ` [dpdk-dev] [PATCH 3/3] net/bnxt: check for chip reset in dev stop/close ops Somnath Kotur 2021-01-12 3:41 ` [dpdk-dev] [PATCH v2 0/3] bnxt patches Ajit Khaparde 2020-12-24 9:35 Somnath Kotur
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).