From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6A7A0A0543 for ; Tue, 21 Jun 2022 10:05:31 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 64DB140151; Tue, 21 Jun 2022 10:05:31 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2057.outbound.protection.outlook.com [40.107.220.57]) by mails.dpdk.org (Postfix) with ESMTP id 36E2F4281A for ; Tue, 21 Jun 2022 10:05:30 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P6b6Cu5ZngQp+e6DkUJ9WLmyKovyt0VUBbR2240zSndOR7FplCcbfIHMQ4XKsfpEp4iTBrwIlCoGHSHROVtCAu4rwSKPbD/P5R3d8sDFyx9vgCe7UhAYSl3OkXNp5/Ks/8dQ9aWO2Hm/wGVD5VHBmaU4hrxCGzbpm2MMnRDWxhjgw0VRG7siDbt2uONwvvNN64cKgn6hdpH+KLmZNqpfkLWd0oe0obwf43jwtyLievVpMW9yk9BzJqJOUjc+8hypwcBAqbIfnN8DDXsEhR+V++pO1QIXAk+cw7v3gsgB12bLHeR4YHM8Tfjj0kWtTTbxZBUZejCudLWprktSyqaMEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sZV4kqwQP8ZropVBPDHTepH5ymWCyk4AyOjsALcjiv0=; b=nuSeRlG4kjWPDYdu4bQv49rerqnKD9vfFCNwQryLDVeaxVmQTpMGFVwIKXg4+Q9x/dwJtL4O/+tGHsP/vXfL6lnFrMnnR0BXHQrgW73lzNbmjPv65I03X5pJSpyXsSrQFmLXgdpNo2i/TXnrJqFMLQWh4lovmaqyr7fYgcOgLNtpfNxDs30Ad2AuSITfyD+GoAcRiwu60Hfr6K5cdWfSQi2AsvnMWGbXe9trEXx744dZ7ivyYFRsDMhi/71O1q+dQOs6U9I46O4dhCZaXVQpQxL4Z+XGuWFwbK0WSSZ28iP3i96dmnYdxUTiHUokBNBnIDgEsChpxZon40L8Zy4rcQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sZV4kqwQP8ZropVBPDHTepH5ymWCyk4AyOjsALcjiv0=; b=pTcfYAnq3KRqdEIGqyBP2JCzUxyPBSGlEFoD6u2/6r45N6q5JjKgAZ/PipTCKEhrwX7jZMNj7UdXye6Pvs1Kgj5VdoYGSUcmfCS4cOHjUFZsovAn8DjXDwKlTB/fJOIWk9momBIrcI/nJ+5zxOCjJUlvwtkYhKTFGL+KgfCUctZCUsffEpPCWPh02k15W4jyKXdMkUr5mVv0O0yBKZwWPdhSoHNydoh4cmV4MKfx9RHtYmQjO3Bt0CEsnZTydHhU5weViZY4ugBZ0FzApgGukZK+s5SuW363PUL/wYDcRqDEuSn3E9qHsaY2AdlVoxt84NlYK4KUFCVt2FCd6OJQZQ== Received: from MWHPR12CA0052.namprd12.prod.outlook.com (2603:10b6:300:103::14) by DM6PR12MB3484.namprd12.prod.outlook.com (2603:10b6:5:3c::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.18; Tue, 21 Jun 2022 08:05:28 +0000 Received: from CO1NAM11FT004.eop-nam11.prod.protection.outlook.com (2603:10b6:300:103:cafe::23) by MWHPR12CA0052.outlook.office365.com (2603:10b6:300:103::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.14 via Frontend Transport; Tue, 21 Jun 2022 08:05:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT004.mail.protection.outlook.com (10.13.175.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5353.14 via Frontend Transport; Tue, 21 Jun 2022 08:05:28 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Tue, 21 Jun 2022 08:05:27 +0000 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Tue, 21 Jun 2022 01:05:26 -0700 From: Xueming Li To: Rahul Lakkireddy CC: dpdk stable Subject: patch 'net/cxgbe: fix Tx queue stuck with mbuf chain coalescing' has been queued to stable release 20.11.6 Date: Tue, 21 Jun 2022 11:01:32 +0300 Message-ID: <20220621080301.2315720-27-xuemingl@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220621080301.2315720-1-xuemingl@nvidia.com> References: <20220621080301.2315720-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 14b81498-4377-4bd2-d69d-08da535ccdd4 X-MS-TrafficTypeDiagnostic: DM6PR12MB3484:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iMTFus9z0hSrsgqEr3YOa+QNmNYpfXP79BeRUwsWfVehzQ1nlFCvBij/bJWX7G5UbhSDl1B037AGxsnFsSA8vXIvJ9rmAyo5h/1sQQf7P+RH4Hg19Cncie+Jqh0rvV6Jgm0TklXZGBc6/eSRJg6y2bmw5DL00NTa3qIl7ySiIb1EI+1IU3o6lKR8rHC+aZywOA1vcZ3PAjtg3Hk/uNysKxq/DJui2qPg0FgUV30rTvfS7Ezx/t2QF/gm5nfWz8TtCghQTTzn5IFSr3Umgc4QS61m1zGH8c4l8tEa4VUxo0LJnWOlstmB0QhRvV4Wvnk/C2IwswiIkJYfqDU8bgDt1i0MbNskqmLfhytwtXX4+j06RFQKgfdzitk9ZWOlagXBIDI4XMSLIvZ5r9Ub2o87Kpn9Lwk/kN87YNt0TbhMzljsjziq9YwWn2FJLB6hIn4I6L1IXJZzUsAHwG9z791s1BaZbZezMtFV7cuob2ldcOtMu09W7+C1YJKeiS6J9py5LARXEYq16Wfu2fUaAIW/tW14Gr5eTf5lBXgF+dA/YmImrvrLGtWCX4uIShL/0kcp0eBdPB3wWWfyq1N3lgmK73thJ6+Oz+7yFjoyVLNoyhspjSUewJAJoA+vMHVAzuoYqXOptCHssJc5Jedz9Soss5CmhYzbUbPco9Oouu+JepxrfN2kpz+jvWLqzjtkId7nwIXIHNWnYnBorMmzmE53ytkdMEHPTnACdvB/luVLAF+WooThOXEco3+z4HbstIDRWyjdVhdmU8aHq0HiKqSHxfBHoZb0f0mW07cXKt4tXqN65M0jBxkL7DFUTcZfOh2ogAZAKuIbjrlgDnlhhGxPT4XqWmaUshEVnqtNUo5tsEqjtKRA2y++t5TCRtkyua2t5xTLlNuu1CvMA2mbru6KDhVUCe32GqeSw8D5EzXRv4M= X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230016)(4636009)(376002)(396003)(39860400002)(136003)(346002)(36840700001)(46966006)(40470700004)(7696005)(6916009)(82310400005)(55016003)(8676002)(40480700001)(966005)(478600001)(86362001)(82740400003)(41300700001)(2906002)(316002)(6666004)(356005)(81166007)(53546011)(36756003)(8936002)(47076005)(40460700003)(6286002)(186003)(2616005)(1076003)(426003)(16526019)(336012)(83380400001)(36860700001)(5660300002)(26005)(40140700001)(70206006)(70586007)(4326008)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 08:05:28.0783 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 14b81498-4377-4bd2-d69d-08da535ccdd4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT004.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3484 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 20.11.6 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 06/23/22. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/730c2afc40bc64bb590c545b0221dfb94fef0425 Thanks. Xueming Li --- >From 730c2afc40bc64bb590c545b0221dfb94fef0425 Mon Sep 17 00:00:00 2001 From: Rahul Lakkireddy Date: Tue, 19 Apr 2022 03:54:19 +0530 Subject: [PATCH] net/cxgbe: fix Tx queue stuck with mbuf chain coalescing Cc: Xueming Li [ upstream commit 151e828f6427667faf3fdfaa00d14a65c7f57cd6 ] When trying to coalesce mbufs with chain on Tx side, it is possible to get stuck during queue wrap around. When coalescing this mbuf chain fails, the Tx path returns EBUSY and when the same packet is retried again, it couldn't get coalesced again, and the loop repeats. Fix by pushing the packet through the normal Tx path. Also use FW_ETH_TX_PKTS_WR to handle mbufs with chain for FW to optimize. Fixes: 6c2809628cd5 ("net/cxgbe: improve latency for slow traffic") Signed-off-by: Rahul Lakkireddy --- drivers/net/cxgbe/sge.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/drivers/net/cxgbe/sge.c b/drivers/net/cxgbe/sge.c index c18ea7c248..0998864269 100644 --- a/drivers/net/cxgbe/sge.c +++ b/drivers/net/cxgbe/sge.c @@ -793,9 +793,9 @@ static inline void txq_advance(struct sge_txq *q, unsigned int n) #define MAX_COALESCE_LEN 64000 -static inline int wraps_around(struct sge_txq *q, int ndesc) +static inline bool wraps_around(struct sge_txq *q, int ndesc) { - return (q->pidx + ndesc) > q->size ? 1 : 0; + return (q->pidx + ndesc) > q->size ? true : false; } static void tx_timer_cb(void *data) @@ -846,7 +846,6 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, /* fill the pkts WR header */ wr = (void *)&q->desc[q->pidx]; - wr->op_pkd = htonl(V_FW_WR_OP(FW_ETH_TX_PKTS2_WR)); vmwr = (void *)&q->desc[q->pidx]; wr_mid = V_FW_WR_LEN16(DIV_ROUND_UP(q->coalesce.flits, 2)); @@ -856,8 +855,11 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, wr->npkt = q->coalesce.idx; wr->r3 = 0; if (is_pf4(adap)) { - wr->op_pkd = htonl(V_FW_WR_OP(FW_ETH_TX_PKTS2_WR)); wr->type = q->coalesce.type; + if (likely(wr->type != 0)) + wr->op_pkd = htonl(V_FW_WR_OP(FW_ETH_TX_PKTS2_WR)); + else + wr->op_pkd = htonl(V_FW_WR_OP(FW_ETH_TX_PKTS_WR)); } else { wr->op_pkd = htonl(V_FW_WR_OP(FW_ETH_TX_PKTS_VM_WR)); vmwr->r4 = 0; @@ -936,13 +938,16 @@ static inline int should_tx_packet_coalesce(struct sge_eth_txq *txq, ndesc = DIV_ROUND_UP(q->coalesce.flits + flits, 8); credits = txq_avail(q) - ndesc; + if (unlikely(wraps_around(q, ndesc))) + return 0; + /* If we are wrapping or this is last mbuf then, send the * already coalesced mbufs and let the non-coalesce pass * handle the mbuf. */ - if (unlikely(credits < 0 || wraps_around(q, ndesc))) { + if (unlikely(credits < 0)) { ship_tx_pkt_coalesce_wr(adap, txq); - return 0; + return -EBUSY; } /* If the max coalesce len or the max WR len is reached @@ -966,8 +971,12 @@ new: ndesc = flits_to_desc(q->coalesce.flits + flits); credits = txq_avail(q) - ndesc; - if (unlikely(credits < 0 || wraps_around(q, ndesc))) + if (unlikely(wraps_around(q, ndesc))) return 0; + + if (unlikely(credits < 0)) + return -EBUSY; + q->coalesce.flits += wr_size / sizeof(__be64); q->coalesce.type = type; q->coalesce.ptr = (unsigned char *)&q->desc[q->pidx] + @@ -1110,7 +1119,7 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf, unsigned int flits, ndesc, cflits; int l3hdr_len, l4hdr_len, eth_xtra_len; int len, last_desc; - int credits; + int should_coal, credits; u32 wr_mid; u64 cntrl, *end; bool v6; @@ -1141,9 +1150,9 @@ out_free: /* align the end of coalesce WR to a 512 byte boundary */ txq->q.coalesce.max = (8 - (txq->q.pidx & 7)) * 8; - if (!((m->ol_flags & PKT_TX_TCP_SEG) || - m->pkt_len > RTE_ETHER_MAX_LEN)) { - if (should_tx_packet_coalesce(txq, mbuf, &cflits, adap)) { + if ((m->ol_flags & PKT_TX_TCP_SEG) == 0) { + should_coal = should_tx_packet_coalesce(txq, mbuf, &cflits, adap); + if (should_coal > 0) { if (unlikely(map_mbuf(mbuf, addr) < 0)) { dev_warn(adap, "%s: mapping err for coalesce\n", __func__); @@ -1152,8 +1161,8 @@ out_free: } return tx_do_packet_coalesce(txq, mbuf, cflits, adap, pi, addr, nb_pkts); - } else { - return -EBUSY; + } else if (should_coal < 0) { + return should_coal; } } @@ -1200,8 +1209,7 @@ out_free: end = (u64 *)vmwr + flits; } - len = 0; - len += sizeof(*cpl); + len = sizeof(*cpl); /* Coalescing skipped and we send through normal path */ if (!(m->ol_flags & PKT_TX_TCP_SEG)) { -- 2.35.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2022-06-21 15:37:50.523416752 +0800 +++ 0026-net-cxgbe-fix-Tx-queue-stuck-with-mbuf-chain-coalesc.patch 2022-06-21 15:37:49.004451134 +0800 @@ -1 +1 @@ -From 151e828f6427667faf3fdfaa00d14a65c7f57cd6 Mon Sep 17 00:00:00 2001 +From 730c2afc40bc64bb590c545b0221dfb94fef0425 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit 151e828f6427667faf3fdfaa00d14a65c7f57cd6 ] @@ -15 +17,0 @@ -Cc: stable@dpdk.org @@ -23 +25 @@ -index 5c176004f9..566cd48406 100644 +index c18ea7c248..0998864269 100644 @@ -26 +28 @@ -@@ -789,9 +789,9 @@ static inline void txq_advance(struct sge_txq *q, unsigned int n) +@@ -793,9 +793,9 @@ static inline void txq_advance(struct sge_txq *q, unsigned int n) @@ -38 +40 @@ -@@ -842,7 +842,6 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, +@@ -846,7 +846,6 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, @@ -46 +48 @@ -@@ -852,8 +851,11 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, +@@ -856,8 +855,11 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, @@ -59 +61 @@ -@@ -932,13 +934,16 @@ static inline int should_tx_packet_coalesce(struct sge_eth_txq *txq, +@@ -936,13 +938,16 @@ static inline int should_tx_packet_coalesce(struct sge_eth_txq *txq, @@ -78 +80 @@ -@@ -962,8 +967,12 @@ new: +@@ -966,8 +971,12 @@ new: @@ -92 +94 @@ -@@ -1106,7 +1115,7 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf, +@@ -1110,7 +1119,7 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf, @@ -101 +103 @@ -@@ -1138,9 +1147,9 @@ out_free: +@@ -1141,9 +1150,9 @@ out_free: @@ -105 +107 @@ -- if (!((m->ol_flags & RTE_MBUF_F_TX_TCP_SEG) || +- if (!((m->ol_flags & PKT_TX_TCP_SEG) || @@ -108 +110 @@ -+ if ((m->ol_flags & RTE_MBUF_F_TX_TCP_SEG) == 0) { ++ if ((m->ol_flags & PKT_TX_TCP_SEG) == 0) { @@ -114 +116 @@ -@@ -1149,8 +1158,8 @@ out_free: +@@ -1152,8 +1161,8 @@ out_free: @@ -125 +127 @@ -@@ -1197,8 +1206,7 @@ out_free: +@@ -1200,8 +1209,7 @@ out_free: @@ -134 +136 @@ - if (!(m->ol_flags & RTE_MBUF_F_TX_TCP_SEG)) { + if (!(m->ol_flags & PKT_TX_TCP_SEG)) {