From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D999A0351 for ; Mon, 21 Feb 2022 16:43:15 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7EB224114B; Mon, 21 Feb 2022 16:43:15 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 157144013F for ; Mon, 21 Feb 2022 16:43:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645458193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3xfNXCRTcMsJPWQn1bY1rw9AWxMsm6nSCxTs8dqh5Kc=; b=NcEP9N+t9y/GJNQBnaY4OH5z2JmW3dAXSEeumnzExP2XcfCecv/zUHkQ4d2TOImKceXco7 X8d7NjlGzukP8jhj4ZzkPkR+RaGb5CiR/7gs1ufav7/gRV1FACfiBK1ZZiiXa6jJjN59p9 Z74kzjz5v/Iwtvo1aDARD6yaCuS7es4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-626-783DQpUbNH-m3EJSD69BjA-1; Mon, 21 Feb 2022 10:43:10 -0500 X-MC-Unique: 783DQpUbNH-m3EJSD69BjA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3DDC91853026; Mon, 21 Feb 2022 15:43:09 +0000 (UTC) Received: from rh.Home (unknown [10.39.195.12]) by smtp.corp.redhat.com (Postfix) with ESMTP id 642D27E2E8; Mon, 21 Feb 2022 15:43:08 +0000 (UTC) From: Kevin Traynor To: Jiawen Wu Cc: dpdk stable Subject: patch 'net/ngbe: fix Tx hang on queue disable' has been queued to stable release 21.11.1 Date: Mon, 21 Feb 2022 15:36:10 +0000 Message-Id: <20220221153625.152324-181-ktraynor@redhat.com> In-Reply-To: <20220221153625.152324-1-ktraynor@redhat.com> References: <20220221153625.152324-1-ktraynor@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ktraynor@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 21.11.1 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 02/26/22. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/kevintraynor/dpdk-stable This queued commit can be viewed at: https://github.com/kevintraynor/dpdk-stable/commit/b9c20ea8f087834d6e5518bb28f5b188f50a9bbb Thanks. Kevin --- >From b9c20ea8f087834d6e5518bb28f5b188f50a9bbb Mon Sep 17 00:00:00 2001 From: Jiawen Wu Date: Wed, 9 Feb 2022 18:42:04 +0800 Subject: [PATCH] net/ngbe: fix Tx hang on queue disable [ upstream commit ac6c5e9af56a741479b8361014bec124aed4ab28 ] Add commands requesting firmware to enable or disable PCIe bus master. Disable PCIe master access to clear BME when stop hardware, and verify there are no pending requests. Move disabling Tx queue after disabling PCIe bus master, to ensure that there are no packets left to cause Tx hang. Fixes: 78710873c2f3 ("net/ngbe: add HW initialization") Signed-off-by: Jiawen Wu --- drivers/net/ngbe/base/ngbe_hw.c | 76 +++++++++++++++++++++++++++---- drivers/net/ngbe/base/ngbe_hw.h | 1 + drivers/net/ngbe/base/ngbe_mng.c | 57 +++++++++++++++++++++++ drivers/net/ngbe/base/ngbe_mng.h | 21 +++++++++ drivers/net/ngbe/base/ngbe_regs.h | 3 ++ drivers/net/ngbe/base/ngbe_type.h | 3 ++ drivers/net/ngbe/ngbe_ethdev.c | 7 ++- 7 files changed, 158 insertions(+), 10 deletions(-) diff --git a/drivers/net/ngbe/base/ngbe_hw.c b/drivers/net/ngbe/base/ngbe_hw.c index 0b22ea0fb3..782fd71d29 100644 --- a/drivers/net/ngbe/base/ngbe_hw.c +++ b/drivers/net/ngbe/base/ngbe_hw.c @@ -351,6 +351,6 @@ void ngbe_set_lan_id_multi_port(struct ngbe_hw *hw) s32 ngbe_stop_hw(struct ngbe_hw *hw) { - u32 reg_val; u16 i; + s32 status = 0; DEBUGFUNC("ngbe_stop_hw"); @@ -373,14 +373,25 @@ s32 ngbe_stop_hw(struct ngbe_hw *hw) wr32(hw, NGBE_ICR(0), NGBE_ICR_MASK); + wr32(hw, NGBE_BMECTL, 0x3); + + /* Disable the receive unit by stopping each queue */ + for (i = 0; i < hw->mac.max_rx_queues; i++) + wr32(hw, NGBE_RXCFG(i), 0); + + /* flush all queues disables */ + ngbe_flush(hw); + msec_delay(2); + + /* + * Prevent the PCI-E bus from hanging by disabling PCI-E master + * access and verify no pending requests + */ + status = ngbe_set_pcie_master(hw, false); + if (status) + return status; + /* Disable the transmit unit. Each queue must be disabled. */ for (i = 0; i < hw->mac.max_tx_queues; i++) - wr32(hw, NGBE_TXCFG(i), NGBE_TXCFG_FLUSH); - - /* Disable the receive unit by stopping each queue */ - for (i = 0; i < hw->mac.max_rx_queues; i++) { - reg_val = rd32(hw, NGBE_RXCFG(i)); - reg_val &= ~NGBE_RXCFG_ENA; - wr32(hw, NGBE_RXCFG(i), reg_val); - } + wr32(hw, NGBE_TXCFG(i), 0); /* flush all queues disables */ @@ -1077,4 +1088,51 @@ out: } +/** + * ngbe_set_pcie_master - Disable or Enable PCI-express master access + * @hw: pointer to hardware structure + * + * Disables PCI-Express master access and verifies there are no pending + * requests. NGBE_ERR_MASTER_REQUESTS_PENDING is returned if master disable + * bit hasn't caused the master requests to be disabled, else 0 + * is returned signifying master requests disabled. + **/ +s32 ngbe_set_pcie_master(struct ngbe_hw *hw, bool enable) +{ + s32 status = 0; + u16 addr = 0x04; + u32 data, i; + + DEBUGFUNC("ngbe_set_pcie_master"); + + ngbe_hic_pcie_read(hw, addr, &data, 4); + if (enable) + data |= 0x04; + else + data &= ~0x04; + + ngbe_hic_pcie_write(hw, addr, &data, 4); + + if (enable) + goto out; + + /* Exit if master requests are blocked */ + if (!(rd32(hw, NGBE_BMEPEND)) || + NGBE_REMOVED(hw->hw_addr)) + goto out; + + /* Poll for master request bit to clear */ + for (i = 0; i < NGBE_PCI_MASTER_DISABLE_TIMEOUT; i++) { + usec_delay(100); + if (!(rd32(hw, NGBE_BMEPEND))) + goto out; + } + + DEBUGOUT("PCIe transaction pending bit also did not clear.\n"); + status = NGBE_ERR_MASTER_REQUESTS_PENDING; + +out: + return status; +} + /** * ngbe_acquire_swfw_sync - Acquire SWFW semaphore diff --git a/drivers/net/ngbe/base/ngbe_hw.h b/drivers/net/ngbe/base/ngbe_hw.h index b32cf87ff4..7e0e23b195 100644 --- a/drivers/net/ngbe/base/ngbe_hw.h +++ b/drivers/net/ngbe/base/ngbe_hw.h @@ -55,4 +55,5 @@ s32 ngbe_validate_mac_addr(u8 *mac_addr); s32 ngbe_acquire_swfw_sync(struct ngbe_hw *hw, u32 mask); void ngbe_release_swfw_sync(struct ngbe_hw *hw, u32 mask); +s32 ngbe_set_pcie_master(struct ngbe_hw *hw, bool enable); s32 ngbe_set_vmdq(struct ngbe_hw *hw, u32 rar, u32 vmdq); diff --git a/drivers/net/ngbe/base/ngbe_mng.c b/drivers/net/ngbe/base/ngbe_mng.c index a3dd8093ce..68e06e2c24 100644 --- a/drivers/net/ngbe/base/ngbe_mng.c +++ b/drivers/net/ngbe/base/ngbe_mng.c @@ -244,4 +244,61 @@ s32 ngbe_hic_sr_write(struct ngbe_hw *hw, u32 addr, u8 *buf, int len) } +s32 ngbe_hic_pcie_read(struct ngbe_hw *hw, u16 addr, u32 *buf, int len) +{ + struct ngbe_hic_read_pcie command; + u32 value = 0; + int err, i = 0; + + if (len > NGBE_PMMBX_DATA_SIZE) + return NGBE_ERR_HOST_INTERFACE_COMMAND; + + memset(&command, 0, sizeof(command)); + command.hdr.cmd = FW_PCIE_READ_CMD; + command.hdr.buf_len = sizeof(command) - sizeof(command.hdr); + command.hdr.checksum = FW_DEFAULT_CHECKSUM; + command.lan_id = hw->bus.lan_id; + command.addr = addr; + + err = ngbe_host_interface_command(hw, (u32 *)&command, + sizeof(command), NGBE_HI_COMMAND_TIMEOUT, false); + if (err) + return err; + + while (i < (len >> 2)) { + value = rd32a(hw, NGBE_MNGMBX, FW_PCIE_BUSMASTER_OFFSET + i); + ((u32 *)buf)[i] = value; + i++; + } + + return 0; +} + +s32 ngbe_hic_pcie_write(struct ngbe_hw *hw, u16 addr, u32 *buf, int len) +{ + struct ngbe_hic_write_pcie command; + u32 value = 0; + int err, i = 0; + + while (i < (len >> 2)) { + value = ((u32 *)buf)[i]; + i++; + } + + memset(&command, 0, sizeof(command)); + command.hdr.cmd = FW_PCIE_WRITE_CMD; + command.hdr.buf_len = sizeof(command) - sizeof(command.hdr); + command.hdr.checksum = FW_DEFAULT_CHECKSUM; + command.lan_id = hw->bus.lan_id; + command.addr = addr; + command.data = value; + + err = ngbe_host_interface_command(hw, (u32 *)&command, + sizeof(command), NGBE_HI_COMMAND_TIMEOUT, false); + if (err) + return err; + + return 0; +} + s32 ngbe_hic_check_cap(struct ngbe_hw *hw) { diff --git a/drivers/net/ngbe/base/ngbe_mng.h b/drivers/net/ngbe/base/ngbe_mng.h index e3d0309cbc..321338a051 100644 --- a/drivers/net/ngbe/base/ngbe_mng.h +++ b/drivers/net/ngbe/base/ngbe_mng.h @@ -21,4 +21,7 @@ #define FW_WRITE_SHADOW_RAM_CMD 0x33 #define FW_WRITE_SHADOW_RAM_LEN 0xA /* 8 plus 1 WORD to write */ +#define FW_PCIE_READ_CMD 0xEC +#define FW_PCIE_WRITE_CMD 0xED +#define FW_PCIE_BUSMASTER_OFFSET 2 #define FW_DEFAULT_CHECKSUM 0xFF /* checksum always 0xFF */ #define FW_NVM_DATA_OFFSET 3 @@ -77,6 +80,24 @@ struct ngbe_hic_write_shadow_ram { }; +struct ngbe_hic_read_pcie { + struct ngbe_hic_hdr hdr; + u8 lan_id; + u8 rsvd; + u16 addr; + u32 data; +}; + +struct ngbe_hic_write_pcie { + struct ngbe_hic_hdr hdr; + u8 lan_id; + u8 rsvd; + u16 addr; + u32 data; +}; + s32 ngbe_hic_sr_read(struct ngbe_hw *hw, u32 addr, u8 *buf, int len); s32 ngbe_hic_sr_write(struct ngbe_hw *hw, u32 addr, u8 *buf, int len); +s32 ngbe_hic_pcie_read(struct ngbe_hw *hw, u16 addr, u32 *buf, int len); +s32 ngbe_hic_pcie_write(struct ngbe_hw *hw, u16 addr, u32 *buf, int len); s32 ngbe_hic_check_cap(struct ngbe_hw *hw); diff --git a/drivers/net/ngbe/base/ngbe_regs.h b/drivers/net/ngbe/base/ngbe_regs.h index 872b008c46..e84bfdf88a 100644 --- a/drivers/net/ngbe/base/ngbe_regs.h +++ b/drivers/net/ngbe/base/ngbe_regs.h @@ -867,4 +867,7 @@ enum ngbe_5tuple_protocol { ******************************************************************************/ /* Interrupt */ +#define NGBE_BMECTL 0x012020 +#define NGBE_BMECTL_VFDRP MS(1, 0x1) +#define NGBE_BMECTL_PFDRP MS(0, 0x1) #define NGBE_ICRMISC 0x000100 #define NGBE_ICRMISC_MASK MS(8, 0xFFFFFF) diff --git a/drivers/net/ngbe/base/ngbe_type.h b/drivers/net/ngbe/base/ngbe_type.h index 269e087d50..4c995e7397 100644 --- a/drivers/net/ngbe/base/ngbe_type.h +++ b/drivers/net/ngbe/base/ngbe_type.h @@ -18,4 +18,7 @@ #define NGBE_MAX_UTA 128 +#define NGBE_PCI_MASTER_DISABLE_TIMEOUT 800 + + #define NGBE_ALIGN 128 /* as intel did */ #define NGBE_ISB_SIZE 16 diff --git a/drivers/net/ngbe/ngbe_ethdev.c b/drivers/net/ngbe/ngbe_ethdev.c index 8e31234442..30c9e68579 100644 --- a/drivers/net/ngbe/ngbe_ethdev.c +++ b/drivers/net/ngbe/ngbe_ethdev.c @@ -951,5 +951,4 @@ ngbe_dev_start(struct rte_eth_dev *dev) /* stop adapter */ hw->adapter_stopped = 0; - ngbe_stop_hw(hw); /* reinitialize adapter, this calls reset and start */ @@ -962,4 +961,6 @@ ngbe_dev_start(struct rte_eth_dev *dev) hw->mac.get_link_status = true; + ngbe_set_pcie_master(hw, true); + /* configure PF module if SRIOV enabled */ ngbe_pf_host_configure(dev); @@ -1175,4 +1176,6 @@ ngbe_dev_stop(struct rte_eth_dev *dev) rte_intr_vec_list_free(intr_handle); + ngbe_set_pcie_master(hw, true); + adapter->rss_reta_updated = 0; @@ -1203,4 +1206,6 @@ ngbe_dev_close(struct rte_eth_dev *dev) ngbe_dev_free_queues(dev); + ngbe_set_pcie_master(hw, false); + /* reprogram the RAR[0] in case user changed it. */ ngbe_set_rar(hw, 0, hw->mac.addr, 0, true); -- 2.34.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2022-02-21 15:22:48.460985206 +0000 +++ 0181-net-ngbe-fix-Tx-hang-on-queue-disable.patch 2022-02-21 15:22:44.357704725 +0000 @@ -1 +1 @@ -From ac6c5e9af56a741479b8361014bec124aed4ab28 Mon Sep 17 00:00:00 2001 +From b9c20ea8f087834d6e5518bb28f5b188f50a9bbb Mon Sep 17 00:00:00 2001 @@ -5,0 +6,2 @@ +[ upstream commit ac6c5e9af56a741479b8361014bec124aed4ab28 ] + @@ -14 +15,0 @@ -Cc: stable@dpdk.org