From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 760F141D44 for ; Mon, 27 Feb 2023 16:42:09 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7152D40A84; Mon, 27 Feb 2023 16:42:09 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2057.outbound.protection.outlook.com [40.107.93.57]) by mails.dpdk.org (Postfix) with ESMTP id 656B740A7D for ; Mon, 27 Feb 2023 16:42:08 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RTNZ0iHdiC7KAaZW+c1GNL20Hr0nmEuGMBUL6ncvtYKe3V+U89DBgoZORfZCHNPTRgrAxyPdkLY0hrBKuZNw+HTd9eKYKdMZ9dAClJCgpS79P8PpJeR4km+1TO7UKmTPphy8Rs7EZiS7eJWHQWUbBPM+pLg4CxnwBTYnNlO0KRJL10Jg/9CDRNEvgEttQGKRvndwZW1Kny62LkGpxOBkchykz9t0JH4f2CgicN1BrbJx28OVNEdRDYqNLYhesAjU11xFF+gdEGqdhab+niD6Jm1ViTiLCdvTYM5k0gpMM3xF+lWLqhsXtxHciGOtmM423PXIHqRIrp3DziW6pGqItA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=32JTe4YDrKSk5kxQ7aOKvx1qZqpeUMGStWaI1Rz8k6s=; b=lvJNDsoQL3iaG4+jPFfmGuSbciLRi84Yq4U5GXva8sYsdn1ZWVnkj2HgFsRWYRZA+gg2wDh+fxQogJczYxYunsmvW/uvjqIxzlJikJ1s+ke9yNJYwH4WfXSkO/UQ30og/I8jIB1nc7WiUQJqidMwfBzF8fOan4TrQ89mmhvXU2Djke5DLSliajHagYZPrBO80jxFeOr1WEeYkDX6bgopUVL0V9U3dDJ98cB8dd49kzkvBZJ/BOS/Pd1fBiWeno5qaLrHvaX0wHcKJiVKh/gWq4yg8KnP45DO4ffgaeZ4HTfe/DqfiZZNMrcyXqucDTLOcr3dzpUmbfGtJHquPIne8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=32JTe4YDrKSk5kxQ7aOKvx1qZqpeUMGStWaI1Rz8k6s=; b=BkS1SaUlWWbu2rtiCWtUu4F78r29r4vt1AjONo7SxkZw1/TSwfPAflANzns840D1bUSNDUQzkRfyWsv2MsFw8ONHkzzyKoy2HieNUVGyGRi2kHYkKl+CMurXGXuhDyVSMsrimkhw7ks1F90AHuOUSDYrXY4zUovqgM+IfsGXTmy4rFSmCJqdCou5MkuFXUMj3cRWetytgMq/xXnwKgWDvQs07LiSgXk3UmcjY9atm5scXGaaOFSxwJo+EEERnTDFyP0fCzasqliZQq38Nx9oQHu8DRRRFqyXyVqVHaaDUkC8ExmL43sHj1GITeqBu8m6D5VyFkPYWbXKMpWQEXNxDg== Received: from MN2PR20CA0063.namprd20.prod.outlook.com (2603:10b6:208:235::32) by DS7PR12MB5815.namprd12.prod.outlook.com (2603:10b6:8:77::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.29; Mon, 27 Feb 2023 15:42:06 +0000 Received: from BL02EPF00010208.namprd05.prod.outlook.com (2603:10b6:208:235:cafe::fa) by MN2PR20CA0063.outlook.office365.com (2603:10b6:208:235::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.29 via Frontend Transport; Mon, 27 Feb 2023 15:42:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL02EPF00010208.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.12 via Frontend Transport; Mon, 27 Feb 2023 15:42:06 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Mon, 27 Feb 2023 07:41:53 -0800 Received: from pegasus01.mtr.labs.mlnx (10.126.231.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 27 Feb 2023 07:41:51 -0800 From: Alexander Kozyrev To: CC: , , , Subject: [PATCH 20.11] net/mlx5: check compressed CQE opcode in vectorized Rx Date: Mon, 27 Feb 2023 17:41:30 +0200 Message-ID: <20230227154130.619504-1-akozyrev@nvidia.com> X-Mailer: git-send-email 2.18.2 MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.126.231.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00010208:EE_|DS7PR12MB5815:EE_ X-MS-Office365-Filtering-Correlation-Id: 77d85270-8807-4dde-272b-08db18d92e45 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V7h7lYJL7LX76TFVhOtwEiUym9q+IM++jUPhw1p0ESEv1wWsJ7C705zxb88OIRK+nAA5TfRSp6Yse9IR6I0gkRsizhJaVj2LTCdy6k3VmcmTHz7EsD+THkziQY9JW9oKasmoSDNi0ptf67QA5xGhOtqDOUoTXZvqQvSdfWw9srtUTWwXPkC0QwUsMfEcheCVnzPitXwOFzRjo0zHzftzzBqsRfxWOovZcxj6YH1QrcG1TDORlS5mHUy/Cs19pdMTkJnUdQ5IGOjGi5bT1R5oYm9QAXhF4EPm03BNjjXvxkaIOD+CQltetl7Um+MdGmT47x2MpoYANvhQ0f471S897YhScXWFaS8oXAosmKtbiOZHj8c8xQq9QA8dvVJyxAfMm6sQl4DdjpGV31W9vdpDWVSXRBu4I4t8J/Hthy2gBDAzmqyznYE3oEwJV/3aXREtjtWcA10wdtlu/xLy4ANCjRpzGaSoQKLQ9Kdh/pSslBWlQxdZygAlCTQU9Lkmjv+r7+DDYgCDZ6gxBh/SwpsnLZpg5HPczNLMDbnHfG+3vCQYFA7QA7MJxblZVbLLgzK8XES6dG6L8Cl5UKjD9a5Pxi9KlusuR4rXRoHdiBZ2dlSHlwqvoCvI9pnBhrchm4yB6MHq+faWJ/KT+Hr2mlWIwnI+pVl4P82wpOrD3td/Tjw3Ej22VPFw5NeL5OnENsw1CTRdUTfeTi4UvsQ5T4x5qYWTW6emB6LZ7lCylF/vGMI= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(376002)(346002)(39860400002)(136003)(451199018)(40470700004)(46966006)(36840700001)(16526019)(26005)(186003)(1076003)(6666004)(36860700001)(426003)(47076005)(82310400005)(356005)(83380400001)(40460700003)(86362001)(7636003)(40480700001)(36756003)(82740400003)(2616005)(336012)(70586007)(4326008)(6916009)(8676002)(70206006)(2906002)(8936002)(5660300002)(41300700001)(316002)(54906003)(478600001)(83323001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2023 15:42:06.4435 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 77d85270-8807-4dde-272b-08db18d92e45 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00010208.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5815 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org [ upstream commit 1f903ebe2c05366aed6a453b6e7636945fc87285 ] The CQE opcode is never checked for a compressed CQE in the vectorized Rx burst routines. It is assumed that compressed CQEs are always valid and skipped error checking. This is obviously not the case and error CQEs may be compressed together as well. Need to check for the MLX5_CQE_RESP_ERR opcode and mark all the packets as bad ones in the compression session if it is there. Note that this issue is not applicable to the scalar Rx burst. Fixes: 6cb559d67b ("net/mlx5: add vectorized Rx/Tx burst for x86") Signed-off-by: Alexander Kozyrev --- drivers/net/mlx5/mlx5_rxtx_vec_altivec.h | 16 +++++++++++++--- drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 10 +++++++--- drivers/net/mlx5/mlx5_rxtx_vec_sse.h | 9 ++++++--- 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h index 418e6ae23b..bb584e3627 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h @@ -783,7 +783,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, { const uint16_t q_n = 1 << rxq->cqe_n; const uint16_t q_mask = q_n - 1; - unsigned int pos; + unsigned int pos, adj; uint64_t n = 0; uint64_t comp_idx = MLX5_VPMD_DESCS_PER_LOOP; uint16_t nocmp_n = 0; @@ -866,7 +866,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, vector unsigned char pkt_mb0, pkt_mb1, pkt_mb2, pkt_mb3; vector unsigned char op_own, op_own_tmp1, op_own_tmp2; vector unsigned char opcode, owner_mask, invalid_mask; - vector unsigned char comp_mask; + vector unsigned char comp_mask, mini_mask; vector unsigned char mask; #ifdef MLX5_PMD_SOFT_COUNTERS const vector unsigned char lower_half = { @@ -1174,6 +1174,16 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, (vector unsigned long)mask); /* D.3 check error in opcode. */ + adj = (comp_idx != MLX5_VPMD_DESCS_PER_LOOP && comp_idx == n); + mask = (vector unsigned char)(vector unsigned long){ + (adj * sizeof(uint16_t) * 8), 0}; + lshift = vec_splat((vector unsigned long)mask, 0); + shmask = vec_cmpgt(shmax, lshift); + mini_mask = (vector unsigned char) + vec_sl((vector unsigned long)invalid_mask, lshift); + mini_mask = (vector unsigned char) + vec_sel((vector unsigned long)shmask, + (vector unsigned long)mini_mask, shmask); opcode = (vector unsigned char) vec_cmpeq((vector unsigned int)resp_err_check, (vector unsigned int)opcode); @@ -1182,7 +1192,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, (vector unsigned int)zero); opcode = (vector unsigned char) vec_andc((vector unsigned long)opcode, - (vector unsigned long)invalid_mask); + (vector unsigned long)mini_mask); /* D.4 mark if any error is set */ *err |= ((vector unsigned long)opcode)[0]; diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h index aa60ee8b92..e4b74c40d1 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h @@ -524,7 +524,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, { const uint16_t q_n = 1 << rxq->cqe_n; const uint16_t q_mask = q_n - 1; - unsigned int pos; + unsigned int pos, adj; uint64_t n = 0; uint64_t comp_idx = MLX5_VPMD_DESCS_PER_LOOP; uint16_t nocmp_n = 0; @@ -616,7 +616,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, pos += MLX5_VPMD_DESCS_PER_LOOP) { uint16x4_t op_own; uint16x4_t opcode, owner_mask, invalid_mask; - uint16x4_t comp_mask; + uint16x4_t comp_mask, mini_mask; uint16x4_t mask; uint16x4_t byte_cnt; uint32x4_t ptype_info, flow_tag; @@ -780,8 +780,12 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, -1UL >> (n * sizeof(uint16_t) * 8) : 0); invalid_mask = vorr_u16(invalid_mask, mask); /* D.3 check error in opcode. */ + adj = (comp_idx != MLX5_VPMD_DESCS_PER_LOOP && comp_idx == n); + mask = vcreate_u16(adj ? + -1UL >> ((n + 1) * sizeof(uint16_t) * 8) : -1UL); + mini_mask = vand_u16(invalid_mask, mask); opcode = vceq_u16(resp_err_check, opcode); - opcode = vbic_u16(opcode, invalid_mask); + opcode = vbic_u16(opcode, mini_mask); /* D.4 mark if any error is set */ *err |= vget_lane_u64(vreinterpret_u64_u16(opcode), 0); /* C.4 fill in mbuf - rearm_data and packet_type. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h index 9b812e3844..b40bb39650 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h @@ -523,7 +523,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, { const uint16_t q_n = 1 << rxq->cqe_n; const uint16_t q_mask = q_n - 1; - unsigned int pos; + unsigned int pos, adj; uint64_t n = 0; uint64_t comp_idx = MLX5_VPMD_DESCS_PER_LOOP; uint16_t nocmp_n = 0; @@ -591,7 +591,7 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, __m128i pkt_mb0, pkt_mb1, pkt_mb2, pkt_mb3; __m128i op_own, op_own_tmp1, op_own_tmp2; __m128i opcode, owner_mask, invalid_mask; - __m128i comp_mask; + __m128i comp_mask, mini_mask; __m128i mask; #ifdef MLX5_PMD_SOFT_COUNTERS __m128i byte_cnt; @@ -729,9 +729,12 @@ rxq_cq_process_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, mask = _mm_sll_epi64(ones, mask); invalid_mask = _mm_or_si128(invalid_mask, mask); /* D.3 check error in opcode. */ + adj = (comp_idx != MLX5_VPMD_DESCS_PER_LOOP && comp_idx == n); + mask = _mm_set_epi64x(0, adj * sizeof(uint16_t) * 8); + mini_mask = _mm_sll_epi64(invalid_mask, mask); opcode = _mm_cmpeq_epi32(resp_err_check, opcode); opcode = _mm_packs_epi32(opcode, zero); - opcode = _mm_andnot_si128(invalid_mask, opcode); + opcode = _mm_andnot_si128(mini_mask, opcode); /* D.4 mark if any error is set */ *err |= _mm_cvtsi128_si64(opcode); /* D.5 fill in mbuf - rearm_data and packet_type. */ -- 2.18.2