From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 454C5A0A0E for ; Mon, 10 May 2021 18:22:25 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3FE1040F35; Mon, 10 May 2021 18:22:25 +0200 (CEST) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-eopbgr750085.outbound.protection.outlook.com [40.107.75.85]) by mails.dpdk.org (Postfix) with ESMTP id 6289E410E3 for ; Mon, 10 May 2021 18:22:24 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K7eyDg60aAHRQtPs3SMbUUePGlV9tdLiw2wwEV21tW4XIZPgWeSvUgJjTGNUok96wD9eoTOAA2QCyA80tQYcRVwpdL5Wu6BNvR4Wh9MgpHA0IWF99gIutNaiZZXEZCxq24aeI3EgmdUPRIAKdHPkH2QMOxj+DjD/I7xx6Gr9nDmPZtPv0rUc9a7kG77iF/ibT54UNYpZVMKAcX7QQG7AcNRhSK6OeDASbbPD1fkJiGeEU5rTJ6oS/cuaQC/LmgKGx4SfM4iULMxRGteeSXb1V8D8i1hrtCdPOq1gti/OXywtoZ2eVA1WF8ooS4RE4BmkwxBkms+JIAmhI39MMhmR8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M/JsBSxdLTRGZ8ZOke9x0FS5yMvWUZuueaX3RW1QEZU=; b=FMaNSSYfhS+yktqnSaDiwkPlyRmwrX2cjnUEiijExpOrHhEm+VXVcrIXErrSV9WyTXDmLetULpIyC8FmP1I+gLQTGI7hwFGSk1ak+5bPF3J8TXcBTTSgRYjD7Z9yS3oB8bvAUv3Tgy/WN0NV8aXxLdfnqfR29EKt8HBFI+VNYXe7iI7q3SMemQsnAc8c4QdhK0+FISC1AqNld5Wh4K7KItwX/5dQngAWGkX+SfrDExmyMvdobHoxjOPf0DFT5hCuhlV7Quw06/AlB85jMH6ojJf2SZPEu0An0lRJr0LYkw8JvvonMGNsp04YjXHxW4i+KalLsMDs1Rns4tb2MlZLaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M/JsBSxdLTRGZ8ZOke9x0FS5yMvWUZuueaX3RW1QEZU=; b=bVr3f0RSl+Ge4ZxQD6vCn5+AepFZHarz60hWOgTo3CPP8n6FROyzjrTUtbHqmhnEzm/7jcPh9keIAJhSjBQbF89Q4/MtI3XQU4wfL3zR+AJAQTfcQm6YQ+p/K9fX9crQ0BEhjpyPd4OP0GaOzGeOIwswkO8rs11JkUjb9u3YycKzmc60I4kOMoJ6uUbOXzMvtW8JzkTAfAfy94yXlRd6MlGt9VP5yT+U6Uje/I9MtIBj/d1fUjQ0z8W3SwiVsfzsA4t2xdMepmqrcJaDlvmZ4al7OwO2zSCruBor6/UNh6HgnXpGxcBM4LvNvC+g3PRUKx7QdoK8LiKlAR+LBjyV8g== Received: from BN0PR03CA0060.namprd03.prod.outlook.com (2603:10b6:408:e7::35) by DM8PR12MB5398.namprd12.prod.outlook.com (2603:10b6:8:3f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.26; Mon, 10 May 2021 16:22:23 +0000 Received: from BN8NAM11FT031.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e7:cafe::d8) by BN0PR03CA0060.outlook.office365.com (2603:10b6:408:e7::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4108.25 via Frontend Transport; Mon, 10 May 2021 16:22:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT031.mail.protection.outlook.com (10.13.177.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4108.25 via Frontend Transport; Mon, 10 May 2021 16:22:22 +0000 Received: from nvidia.com (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 10 May 2021 16:22:20 +0000 From: Xueming Li To: Wenzhuo Lu CC: Luca Boccassi , David Coyle , dpdk stable Date: Tue, 11 May 2021 00:02:21 +0800 Message-ID: <20210510160258.30982-192-xuemingl@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210510160258.30982-1-xuemingl@nvidia.com> References: <20210510160258.30982-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 11823d01-b49c-4472-6309-08d913cfcaa9 X-MS-TrafficTypeDiagnostic: DM8PR12MB5398: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: meNmfcvspj/5gv/ju7VVwinBCugixMl0GTwAm/Wekbm1E2xuE+VZKSCKprn9oMKaYDYodQdGhdEYQkOajBxXrRImBBxYkLPO0XhxOGYl6cg61HwVwzll2bEzw5bOwGJTpRygC3agZuZ3/mCjkn+CVgbZ+QNMMSe/gc+/d1uTrWvkwc+MpqIvJxn/TmHfWwfop01qBlSlJ2LKZ3p4sL7TbSVVIOMqGWrPAykVJtD++5qqVB5VVqsv1W/VqgCZaH58Adf+eo4VVXaypY/GiGo1w9CG1Xk0dmx//HyvOY5c3ShE6YjcSmiB97oU7QrXP9Plp9CMHFp+EGm37zNB5oclcYRbF7cfSstU5syilwcpDx/o3Ep/NzVn7SFw3fFHCZgmEdG3li3GwEAut20tLdJ4LIfiJkfjUSeWQ60rb8vsVm8haXsh+D0sbN4Olrj2+bdcVC10xMhUwoik0kj3kb4hv7w+Yh4V6iP94tUvUjscfm771OiUpOhVSv1j282XrbxP0SC9kj4U0yL5J0lzA3uJ+pQ1IIJa460ekmbWq1FYQaA2w/wUUiFswTJ81i3SYcmT48giOhpwTALsAVhFAqE8CBUDPXXqmsYI2X8/qHhKGsZKv11UI9oPxY/P5toDX0AwrYBWiVTIa+56vNUcjpWKbWOJLO41ZOucksDX1ijuBagk2n+hxs4wo6Uc4LTk4WuUzkiAzxqLcvNd95+QDXu6k/OATGNBWn1TrZCpAtSD1xo= X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(498600001)(54906003)(36906005)(6916009)(70586007)(70206006)(8936002)(82310400003)(5660300002)(4326008)(30864003)(966005)(86362001)(356005)(8676002)(1076003)(6286002)(36756003)(186003)(16526019)(36860700001)(47076005)(7696005)(2616005)(336012)(426003)(55016002)(2906002)(53546011)(7636003)(83380400001)(26005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2021 16:22:22.6124 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 11823d01-b49c-4472-6309-08d913cfcaa9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT031.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR12MB5398 Subject: [dpdk-stable] patch 'net/ice: fix crash in AVX512' has been queued to stable release 20.11.2 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to stable release 20.11.2 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 05/12/21. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/a0cffe358de79e28b22072dbfc32ed7b84eb89b0 Thanks. Xueming Li --- >From a0cffe358de79e28b22072dbfc32ed7b84eb89b0 Mon Sep 17 00:00:00 2001 From: Wenzhuo Lu Date: Wed, 14 Apr 2021 15:25:25 +0800 Subject: [PATCH] net/ice: fix crash in AVX512 Cc: Luca Boccassi [ upstream commit 20daa1c978b7d34b447453b3f25483c700745456 ] Fix segment fault when failing to get the memory from the pool. If there's no memory in the default cache, fall back to the previous process. The previous AVX2 rearm function is changed to add some AVX512 instructions and changed to a callee of the AVX2 and AVX512 rearm functions. Fixes: 7f85d5ebcfe1 ("net/ice: add AVX512 vector path") Reported-by: David Coyle Signed-off-by: Wenzhuo Lu Tested-by: David Coyle --- drivers/net/ice/ice_rxtx_vec_avx2.c | 120 +-------------- drivers/net/ice/ice_rxtx_vec_avx512.c | 5 +- drivers/net/ice/ice_rxtx_vec_common.h | 203 ++++++++++++++++++++++++++ drivers/net/ice/meson.build | 2 + 4 files changed, 211 insertions(+), 119 deletions(-) diff --git a/drivers/net/ice/ice_rxtx_vec_avx2.c b/drivers/net/ice/ice_rxtx_vec_avx2.c index 7838e17787..e4edcd38a9 100644 --- a/drivers/net/ice/ice_rxtx_vec_avx2.c +++ b/drivers/net/ice/ice_rxtx_vec_avx2.c @@ -10,126 +10,10 @@ #pragma GCC diagnostic ignored "-Wcast-qual" #endif -static inline void +static __rte_always_inline void ice_rxq_rearm(struct ice_rx_queue *rxq) { - int i; - uint16_t rx_id; - volatile union ice_rx_flex_desc *rxdp; - struct ice_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; - - rxdp = rxq->rx_ring + rxq->rxrearm_start; - - /* Pull 'n' more MBUFs into the software ring */ - if (rte_mempool_get_bulk(rxq->mp, - (void *)rxep, - ICE_RXQ_REARM_THRESH) < 0) { - if (rxq->rxrearm_nb + ICE_RXQ_REARM_THRESH >= - rxq->nb_rx_desc) { - __m128i dma_addr0; - - dma_addr0 = _mm_setzero_si128(); - for (i = 0; i < ICE_DESCS_PER_LOOP; i++) { - rxep[i].mbuf = &rxq->fake_mbuf; - _mm_store_si128((__m128i *)&rxdp[i].read, - dma_addr0); - } - } - rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += - ICE_RXQ_REARM_THRESH; - return; - } - -#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC - struct rte_mbuf *mb0, *mb1; - __m128i dma_addr0, dma_addr1; - __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, - RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 2 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; i += 2, rxep += 2) { - __m128i vaddr0, vaddr1; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - - /* convert pa to dma_addr hdr/data */ - dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); - dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); - - /* add headroom to pa values */ - dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); - dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); - - /* flush desc with pa dma_addr */ - _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); - _mm_store_si128((__m128i *)&rxdp++->read, dma_addr1); - } -#else - struct rte_mbuf *mb0, *mb1, *mb2, *mb3; - __m256i dma_addr0_1, dma_addr2_3; - __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); - /* Initialize the mbufs in vector, process 4 mbufs in one loop */ - for (i = 0; i < ICE_RXQ_REARM_THRESH; - i += 4, rxep += 4, rxdp += 4) { - __m128i vaddr0, vaddr1, vaddr2, vaddr3; - __m256i vaddr0_1, vaddr2_3; - - mb0 = rxep[0].mbuf; - mb1 = rxep[1].mbuf; - mb2 = rxep[2].mbuf; - mb3 = rxep[3].mbuf; - - /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ - RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != - offsetof(struct rte_mbuf, buf_addr) + 8); - vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); - vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); - vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); - vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); - - /** - * merge 0 & 1, by casting 0 to 256-bit and inserting 1 - * into the high lanes. Similarly for 2 & 3 - */ - vaddr0_1 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), - vaddr1, 1); - vaddr2_3 = - _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), - vaddr3, 1); - - /* convert pa to dma_addr hdr/data */ - dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); - dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); - - /* add headroom to pa values */ - dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); - dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); - - /* flush desc with pa dma_addr */ - _mm256_store_si256((__m256i *)&rxdp->read, dma_addr0_1); - _mm256_store_si256((__m256i *)&(rxdp + 2)->read, dma_addr2_3); - } - -#endif - - rxq->rxrearm_start += ICE_RXQ_REARM_THRESH; - if (rxq->rxrearm_start >= rxq->nb_rx_desc) - rxq->rxrearm_start = 0; - - rxq->rxrearm_nb -= ICE_RXQ_REARM_THRESH; - - rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? - (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); - - /* Update the tail pointer on the NIC */ - ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); + return ice_rxq_rearm_common(rxq, false); } static inline __m256i diff --git a/drivers/net/ice/ice_rxtx_vec_avx512.c b/drivers/net/ice/ice_rxtx_vec_avx512.c index fd5d724329..533da22bc4 100644 --- a/drivers/net/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/ice/ice_rxtx_vec_avx512.c @@ -12,7 +12,7 @@ #define ICE_DESCS_PER_LOOP_AVX 8 -static inline void +static __rte_always_inline void ice_rxq_rearm(struct ice_rx_queue *rxq) { int i; @@ -24,6 +24,9 @@ ice_rxq_rearm(struct ice_rx_queue *rxq) rxdp = rxq->rx_ring + rxq->rxrearm_start; + if (unlikely(!cache)) + return ice_rxq_rearm_common(rxq, true); + /* We need to pull 'n' more MBUFs into the software ring */ if (cache->len < ICE_RXQ_REARM_THRESH) { uint32_t req = ICE_RXQ_REARM_THRESH + (cache->size - diff --git a/drivers/net/ice/ice_rxtx_vec_common.h b/drivers/net/ice/ice_rxtx_vec_common.h index c09ac7f667..a5d76a2936 100644 --- a/drivers/net/ice/ice_rxtx_vec_common.h +++ b/drivers/net/ice/ice_rxtx_vec_common.h @@ -7,6 +7,10 @@ #include "ice_rxtx.h" +#ifndef __INTEL_COMPILER +#pragma GCC diagnostic ignored "-Wcast-qual" +#endif + static inline uint16_t ice_rx_reassemble_packets(struct ice_rx_queue *rxq, struct rte_mbuf **rx_bufs, uint16_t nb_bufs, uint8_t *split_flags) @@ -318,4 +322,203 @@ ice_tx_vec_dev_check_default(struct rte_eth_dev *dev) return 0; } +#ifdef CC_AVX2_SUPPORT +static __rte_always_inline void +ice_rxq_rearm_common(struct ice_rx_queue *rxq, __rte_unused bool avx512) +{ + int i; + uint16_t rx_id; + volatile union ice_rx_flex_desc *rxdp; + struct ice_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + + /* Pull 'n' more MBUFs into the software ring */ + if (rte_mempool_get_bulk(rxq->mp, + (void *)rxep, + ICE_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + ICE_RXQ_REARM_THRESH >= + rxq->nb_rx_desc) { + __m128i dma_addr0; + + dma_addr0 = _mm_setzero_si128(); + for (i = 0; i < ICE_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + _mm_store_si128((__m128i *)&rxdp[i].read, + dma_addr0); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += + ICE_RXQ_REARM_THRESH; + return; + } + +#ifndef RTE_LIBRTE_ICE_16BYTE_RX_DESC + struct rte_mbuf *mb0, *mb1; + __m128i dma_addr0, dma_addr1; + __m128i hdr_room = _mm_set_epi64x(RTE_PKTMBUF_HEADROOM, + RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < ICE_RXQ_REARM_THRESH; i += 2, rxep += 2) { + __m128i vaddr0, vaddr1; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = _mm_unpackhi_epi64(vaddr0, vaddr0); + dma_addr1 = _mm_unpackhi_epi64(vaddr1, vaddr1); + + /* add headroom to pa values */ + dma_addr0 = _mm_add_epi64(dma_addr0, hdr_room); + dma_addr1 = _mm_add_epi64(dma_addr1, hdr_room); + + /* flush desc with pa dma_addr */ + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr0); + _mm_store_si128((__m128i *)&rxdp++->read, dma_addr1); + } +#else +#ifdef CC_AVX512_SUPPORT + if (avx512) { + struct rte_mbuf *mb0, *mb1, *mb2, *mb3; + struct rte_mbuf *mb4, *mb5, *mb6, *mb7; + __m512i dma_addr0_3, dma_addr4_7; + __m512i hdr_room = _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 8 mbufs in one loop */ + for (i = 0; i < ICE_RXQ_REARM_THRESH; + i += 8, rxep += 8, rxdp += 8) { + __m128i vaddr0, vaddr1, vaddr2, vaddr3; + __m128i vaddr4, vaddr5, vaddr6, vaddr7; + __m256i vaddr0_1, vaddr2_3; + __m256i vaddr4_5, vaddr6_7; + __m512i vaddr0_3, vaddr4_7; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + mb2 = rxep[2].mbuf; + mb3 = rxep[3].mbuf; + mb4 = rxep[4].mbuf; + mb5 = rxep[5].mbuf; + mb6 = rxep[6].mbuf; + mb7 = rxep[7].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + vaddr4 = _mm_loadu_si128((__m128i *)&mb4->buf_addr); + vaddr5 = _mm_loadu_si128((__m128i *)&mb5->buf_addr); + vaddr6 = _mm_loadu_si128((__m128i *)&mb6->buf_addr); + vaddr7 = _mm_loadu_si128((__m128i *)&mb7->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3, and so on. + */ + vaddr0_1 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), + vaddr1, 1); + vaddr2_3 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), + vaddr3, 1); + vaddr4_5 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr4), + vaddr5, 1); + vaddr6_7 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr6), + vaddr7, 1); + vaddr0_3 = + _mm512_inserti64x4(_mm512_castsi256_si512(vaddr0_1), + vaddr2_3, 1); + vaddr4_7 = + _mm512_inserti64x4(_mm512_castsi256_si512(vaddr4_5), + vaddr6_7, 1); + + /* convert pa to dma_addr hdr/data */ + dma_addr0_3 = _mm512_unpackhi_epi64(vaddr0_3, vaddr0_3); + dma_addr4_7 = _mm512_unpackhi_epi64(vaddr4_7, vaddr4_7); + + /* add headroom to pa values */ + dma_addr0_3 = _mm512_add_epi64(dma_addr0_3, hdr_room); + dma_addr4_7 = _mm512_add_epi64(dma_addr4_7, hdr_room); + + /* flush desc with pa dma_addr */ + _mm512_store_si512((__m512i *)&rxdp->read, dma_addr0_3); + _mm512_store_si512((__m512i *)&(rxdp + 4)->read, dma_addr4_7); + } + } else +#endif + { + struct rte_mbuf *mb0, *mb1, *mb2, *mb3; + __m256i dma_addr0_1, dma_addr2_3; + __m256i hdr_room = _mm256_set1_epi64x(RTE_PKTMBUF_HEADROOM); + /* Initialize the mbufs in vector, process 4 mbufs in one loop */ + for (i = 0; i < ICE_RXQ_REARM_THRESH; + i += 4, rxep += 4, rxdp += 4) { + __m128i vaddr0, vaddr1, vaddr2, vaddr3; + __m256i vaddr0_1, vaddr2_3; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + mb2 = rxep[2].mbuf; + mb3 = rxep[3].mbuf; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, buf_iova) != + offsetof(struct rte_mbuf, buf_addr) + 8); + vaddr0 = _mm_loadu_si128((__m128i *)&mb0->buf_addr); + vaddr1 = _mm_loadu_si128((__m128i *)&mb1->buf_addr); + vaddr2 = _mm_loadu_si128((__m128i *)&mb2->buf_addr); + vaddr3 = _mm_loadu_si128((__m128i *)&mb3->buf_addr); + + /** + * merge 0 & 1, by casting 0 to 256-bit and inserting 1 + * into the high lanes. Similarly for 2 & 3 + */ + vaddr0_1 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr0), + vaddr1, 1); + vaddr2_3 = + _mm256_inserti128_si256(_mm256_castsi128_si256(vaddr2), + vaddr3, 1); + + /* convert pa to dma_addr hdr/data */ + dma_addr0_1 = _mm256_unpackhi_epi64(vaddr0_1, vaddr0_1); + dma_addr2_3 = _mm256_unpackhi_epi64(vaddr2_3, vaddr2_3); + + /* add headroom to pa values */ + dma_addr0_1 = _mm256_add_epi64(dma_addr0_1, hdr_room); + dma_addr2_3 = _mm256_add_epi64(dma_addr2_3, hdr_room); + + /* flush desc with pa dma_addr */ + _mm256_store_si256((__m256i *)&rxdp->read, dma_addr0_1); + _mm256_store_si256((__m256i *)&(rxdp + 2)->read, dma_addr2_3); + } + } + +#endif + + rxq->rxrearm_start += ICE_RXQ_REARM_THRESH; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= ICE_RXQ_REARM_THRESH; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + ICE_PCI_REG_WC_WRITE(rxq->qrx_tail, rx_id); +} +#endif + #endif diff --git a/drivers/net/ice/meson.build b/drivers/net/ice/meson.build index 7b291269dc..4638011cbc 100644 --- a/drivers/net/ice/meson.build +++ b/drivers/net/ice/meson.build @@ -24,8 +24,10 @@ if arch_subdir == 'x86' # a. we have AVX supported in minimum instruction set baseline # b. it's not minimum instruction set, but supported by compiler if cc.get_define('__AVX2__', args: machine_args) != '' + cflags += ['-DCC_AVX2_SUPPORT'] sources += files('ice_rxtx_vec_avx2.c') elif cc.has_argument('-mavx2') + cflags += ['-DCC_AVX2_SUPPORT'] ice_avx2_lib = static_library('ice_avx2_lib', 'ice_rxtx_vec_avx2.c', dependencies: [static_rte_ethdev, -- 2.25.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2021-05-10 23:59:31.555983600 +0800 +++ 0193-net-ice-fix-crash-in-AVX512.patch 2021-05-10 23:59:26.650000000 +0800 @@ -1 +1 @@ -From 20daa1c978b7d34b447453b3f25483c700745456 Mon Sep 17 00:00:00 2001 +From a0cffe358de79e28b22072dbfc32ed7b84eb89b0 Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Luca Boccassi + +[ upstream commit 20daa1c978b7d34b447453b3f25483c700745456 ] @@ -15 +17,0 @@ -Cc: stable@dpdk.org @@ -28 +30 @@ -index 25efd30e6b..83dcdf15d4 100644 +index 7838e17787..e4edcd38a9 100644 @@ -161 +163 @@ -index 0a3e8da024..a668b82232 100644 +index fd5d724329..533da22bc4 100644 @@ -403 +405 @@ -index 44ef64b4cf..b2d0b66219 100644 +index 7b291269dc..4638011cbc 100644 @@ -406 +408 @@ -@@ -28,8 +28,10 @@ if arch_subdir == 'x86' +@@ -24,8 +24,10 @@ if arch_subdir == 'x86'