From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id E4DE0A0096 for ; Tue, 9 Apr 2019 21:07:45 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1D0625B34; Tue, 9 Apr 2019 21:06:57 +0200 (CEST) Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by dpdk.org (Postfix) with ESMTP id 510445689 for ; Tue, 9 Apr 2019 21:06:42 +0200 (CEST) Received: by mail-qt1-f195.google.com with SMTP id v20so21113561qtv.12 for ; Tue, 09 Apr 2019 12:06:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:reply-to; bh=MrwCmaa8+w4XyKn78I56FttoUZvb8kjswQcVZcFFDZQ=; b=l6Ktc9pnUVtyyOLFis/pkHQvA8ElRDK+HcFjQPCn1o1Zt88IaBQKjFIpI3qUDVPtYK yaGVbDbRmi450+mhEWryFzVENcAihLBIZfwR0y6sddCIEazQXJcGMTJAsyVcTdx4MNrk ZvOPD82ekgJLbLV8Lkrc6rFI6bqzKmc+zexyAH530vS9MSlQLIu/or1PRbhqJbju8PUN c3rw26NePCZFzbyt4o1baC2BsTPaGspa6FHQHs7RpXTDiYemB+z/JwOKT0jYLJ0Uapey JBYSoF01sML93d5fujLzUGf9QjG2Cjp/xkQI3FeN1DBTl0NbCquFm3nOPACnt8F9BtZ3 dKEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:reply-to; bh=MrwCmaa8+w4XyKn78I56FttoUZvb8kjswQcVZcFFDZQ=; b=lZ2bv8YsOxLucXYmFsGvxwvcfVt5VyBZHg93NGssciZ6KIc1o2y0Kj18DwAThxoanT ijERp07jhky4OannDOa1n106O5vT7mXaRAdLJzskiAw355kvAqt9lNdqPbCUByUHA7xj x6kbaULT9wVXrdKSYr26EcTLddh3KDRzUqF9cigKJWcDwvTxBbinv3N8hpijXwS4f9kw NXphcjPjUWVVwQ1oO7jeyxZtl84dih2DzZ+6cgxK5Q94VkBvCCjQZQPshMzu2YPJKerD /33QvRirzCNU1FOzsvUVF2hlJBxD2clZG0Ptb5Gk83e0lzuxDTVVjI3uoeMp2ZCgv+jx DZlQ== X-Gm-Message-State: APjAAAUlhIGionQuYmqP7RP3sYR3acJIuJIPdxadfoNg1TulW2rWDwqq /1I6WP5zYgs0ScWPTl+c/T2mrtzm6Mg= X-Google-Smtp-Source: APXvYqyBpaUw4lKvYSulISN2RZazJNpQPgpGV13hpcyNl3CVLTzC5Yp3DezSvxrpUeYqX07dnd8Z2A== X-Received: by 2002:ac8:8b9:: with SMTP id v54mr32965250qth.64.1554836800965; Tue, 09 Apr 2019 12:06:40 -0700 (PDT) Received: from csz25116.canlab.ibm.com ([199.246.40.57]) by smtp.gmail.com with ESMTPSA id q23sm17934789qkc.16.2019.04.09.12.06.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 12:06:39 -0700 (PDT) From: Vivian Kong X-Google-Original-From: Vivian Kong To: dev@dpdk.org Date: Tue, 9 Apr 2019 15:06:24 -0400 Message-Id: <20190409190630.31975-7-vivkong@ca.ibm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190409190630.31975-1-vivkong@ca.ibm.com> References: <20190409190630.31975-1-vivkong@ca.ibm.com> Subject: [dpdk-dev] [RFC 06/12] net/i40e: add support for s390x architecture X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: vivkong@ca.ibm.com List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Content-Type: text/plain; charset="UTF-8" Message-ID: <20190409190624.ulIw8Xuq6_k3wubqbyjSTDfWbWohIu1sgwY9mtfO6nA@z> Enable i40e and i40e vector support on s390x. Signed-off-by: Vivian Kong --- doc/guides/nics/features/i40e.ini | 1 + doc/guides/nics/features/i40e_vec.ini | 1 + drivers/net/i40e/Makefile | 2 + drivers/net/i40e/i40e_rxtx_vec_s390x.c | 631 +++++++++++++++++++++++++ 4 files changed, 635 insertions(+) create mode 100644 drivers/net/i40e/i40e_rxtx_vec_s390x.c diff --git a/doc/guides/nics/features/i40e.ini b/doc/guides/nics/features/i40e.ini index 16eab7f43..0c1acfc23 100644 --- a/doc/guides/nics/features/i40e.ini +++ b/doc/guides/nics/features/i40e.ini @@ -55,3 +55,4 @@ x86-32 = Y x86-64 = Y ARMv8 = Y Power8 = Y +s390x = Y diff --git a/doc/guides/nics/features/i40e_vec.ini b/doc/guides/nics/features/i40e_vec.ini index c65e8b036..8c5062698 100644 --- a/doc/guides/nics/features/i40e_vec.ini +++ b/doc/guides/nics/features/i40e_vec.ini @@ -43,3 +43,4 @@ x86-32 = Y x86-64 = Y ARMv8 = Y Power8 = Y +s390x = Y diff --git a/drivers/net/i40e/Makefile b/drivers/net/i40e/Makefile index 3f869a8d6..d13223cc9 100644 --- a/drivers/net/i40e/Makefile +++ b/drivers/net/i40e/Makefile @@ -78,6 +78,8 @@ ifeq ($(CONFIG_RTE_ARCH_ARM64),y) SRCS-$(CONFIG_RTE_LIBRTE_I40E_INC_VECTOR) += i40e_rxtx_vec_neon.c else ifeq ($(CONFIG_RTE_ARCH_PPC_64),y) SRCS-$(CONFIG_RTE_LIBRTE_I40E_INC_VECTOR) += i40e_rxtx_vec_altivec.c +else ifeq ($(CONFIG_RTE_ARCH_S390X),y) +SRCS-$(CONFIG_RTE_LIBRTE_I40E_INC_VECTOR) += i40e_rxtx_vec_s390x.c else SRCS-$(CONFIG_RTE_LIBRTE_I40E_INC_VECTOR) += i40e_rxtx_vec_sse.c endif diff --git a/drivers/net/i40e/i40e_rxtx_vec_s390x.c b/drivers/net/i40e/i40e_rxtx_vec_s390x.c new file mode 100644 index 000000000..b35ab0678 --- /dev/null +++ b/drivers/net/i40e/i40e_rxtx_vec_s390x.c @@ -0,0 +1,631 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2010-2015 Intel Corporation. All rights reserved. + * (c) Copyright IBM Corp. 2017, 2019 + */ + +#include +#include +#include +#include + +#include "base/i40e_prototype.h" +#include "base/i40e_type.h" +#include "i40e_ethdev.h" +#include "i40e_rxtx.h" +#include "i40e_rxtx_vec_common.h" + +#pragma GCC diagnostic ignored "-Wcast-qual" + +typedef unsigned long long vector_unsigned_long_long + __attribute__((vector_size(2 * sizeof(unsigned long long)))); +typedef unsigned int vector_unsigned_int + __attribute__((vector_size(4 * sizeof(unsigned int)))); +typedef unsigned short vector_unsigned_short + __attribute__((vector_size(8 * sizeof(unsigned short)))); +typedef unsigned char vector_unsigned_char + __attribute__((vector_size(16 * sizeof(unsigned char)))); + + +static inline void +i40e_rxq_rearm(struct i40e_rx_queue *rxq) +{ + int i; + uint16_t rx_id; + volatile union i40e_rx_desc *rxdp; + + struct i40e_rx_entry *rxep = &rxq->sw_ring[rxq->rxrearm_start]; + struct rte_mbuf *mb0, *mb1; + + vector_unsigned_long_long hdr_room = (vector_unsigned_long_long){ + RTE_PKTMBUF_HEADROOM, + RTE_PKTMBUF_HEADROOM}; + vector_unsigned_long_long dma_addr0, dma_addr1; + + rxdp = rxq->rx_ring + rxq->rxrearm_start; + + /* Pull 'n' more MBUFs into the software ring */ + if (rte_mempool_get_bulk(rxq->mp, + (void *)rxep, + RTE_I40E_RXQ_REARM_THRESH) < 0) { + if (rxq->rxrearm_nb + RTE_I40E_RXQ_REARM_THRESH >= + rxq->nb_rx_desc) { + dma_addr0 = (vector_unsigned_long_long){}; + for (i = 0; i < RTE_I40E_DESCS_PER_LOOP; i++) { + rxep[i].mbuf = &rxq->fake_mbuf; + vec_xstd2(dma_addr0, 0, + (unsigned long long *)&rxdp[i].read); + } + } + rte_eth_devices[rxq->port_id].data->rx_mbuf_alloc_failed += + RTE_I40E_RXQ_REARM_THRESH; + return; + } + + /* Initialize the mbufs in vector, process 2 mbufs in one loop */ + for (i = 0; i < RTE_I40E_RXQ_REARM_THRESH; i += 2, rxep += 2) { + vector_unsigned_long_long vaddr0, vaddr1; + uintptr_t p0, p1; + + mb0 = rxep[0].mbuf; + mb1 = rxep[1].mbuf; + + /* Flush mbuf with pkt template. + * Data to be rearmed is 6 bytes long. + * Though, RX will overwrite ol_flags that are coming next + * anyway. So overwrite whole 8 bytes with one load: + * 6 bytes of rearm_data plus first 2 bytes of ol_flags. + */ + p0 = (uintptr_t)&mb0->rearm_data; + *(uint64_t *)p0 = rxq->mbuf_initializer; + p1 = (uintptr_t)&mb1->rearm_data; + *(uint64_t *)p1 = rxq->mbuf_initializer; + + /* load buf_addr(lo 64bit) and buf_iova(hi 64bit) */ + vaddr0 = vec_xld2(0, (unsigned long long *)&mb0->buf_addr); + vaddr1 = vec_xld2(0, (unsigned long long *)&mb1->buf_addr); + + /* convert pa to dma_addr hdr/data */ + dma_addr0 = vec_mergel(vaddr0, vaddr0); + dma_addr1 = vec_mergel(vaddr1, vaddr1); + + /* add headroom to pa values */ + dma_addr0 = dma_addr0 + hdr_room; + dma_addr1 = dma_addr1 + hdr_room; + + /* flush desc with pa dma_addr */ + vec_xstd2(dma_addr0, 0, (unsigned long long *)&rxdp++->read); + vec_xstd2(dma_addr1, 0, (unsigned long long *)&rxdp++->read); + } + + rxq->rxrearm_start += RTE_I40E_RXQ_REARM_THRESH; + if (rxq->rxrearm_start >= rxq->nb_rx_desc) + rxq->rxrearm_start = 0; + + rxq->rxrearm_nb -= RTE_I40E_RXQ_REARM_THRESH; + + rx_id = (uint16_t)((rxq->rxrearm_start == 0) ? + (rxq->nb_rx_desc - 1) : (rxq->rxrearm_start - 1)); + + /* Update the tail pointer on the NIC */ + I40E_PCI_REG_WRITE(rxq->qrx_tail, rx_id); +} + +static inline void +desc_to_olflags_v(vector_unsigned_long_long descs[4], struct rte_mbuf **rx_pkts) +{ + vector_unsigned_int vlan0, vlan1, rss, l3_l4e; + + /* mask everything except RSS, flow director and VLAN flags + * bit2 is for VLAN tag, bit11 for flow director indication + * bit13:12 for RSS indication. + */ + const vector_unsigned_int rss_vlan_msk = (vector_unsigned_int){ + (int32_t)0x1c03804, (int32_t)0x1c03804, + (int32_t)0x1c03804, (int32_t)0x1c03804}; + + /* map rss and vlan type to rss hash and vlan flag */ + const vector_unsigned_char vlan_flags = (vector_unsigned_char){ + 0, 0, 0, 0, + PKT_RX_VLAN | PKT_RX_VLAN_STRIPPED, 0, 0, 0, + 0, 0, 0, 0, + 0, 0, 0, 0}; + + const vector_unsigned_char rss_flags = (vector_unsigned_char){ + 0, PKT_RX_FDIR, 0, 0, + 0, 0, PKT_RX_RSS_HASH, PKT_RX_RSS_HASH | PKT_RX_FDIR, + 0, 0, 0, 0, + 0, 0, 0, 0}; + + const vector_unsigned_char l3_l4e_flags = (vector_unsigned_char){ + 0, + PKT_RX_IP_CKSUM_BAD, + PKT_RX_L4_CKSUM_BAD, + PKT_RX_L4_CKSUM_BAD | PKT_RX_IP_CKSUM_BAD, + PKT_RX_EIP_CKSUM_BAD, + PKT_RX_EIP_CKSUM_BAD | PKT_RX_IP_CKSUM_BAD, + PKT_RX_EIP_CKSUM_BAD | PKT_RX_L4_CKSUM_BAD, + PKT_RX_EIP_CKSUM_BAD | PKT_RX_L4_CKSUM_BAD + | PKT_RX_IP_CKSUM_BAD, + 0, 0, 0, 0, 0, 0, 0, 0}; + + vlan0 = (vector_unsigned_int)vec_mergel(descs[0], descs[1]); + vlan1 = (vector_unsigned_int)vec_mergel(descs[2], descs[3]); + vlan0 = (vector_unsigned_int)vec_mergeh(vlan0, vlan1); + + vlan1 = vec_and(vlan0, rss_vlan_msk); + vlan0 = (vector_unsigned_int)vec_perm(vlan_flags, + (vector_unsigned_char){}, + *(vector_unsigned_char *)&vlan1); + + rss[0] = (uint32_t)vlan1[0] >> 11; + rss[1] = (uint32_t)vlan1[1] >> 11; + rss[2] = (uint32_t)vlan1[2] >> 11; + rss[3] = (uint32_t)vlan1[3] >> 11; + rss = (vector_unsigned_int)vec_perm(rss_flags, (vector_unsigned_char){}, + *(vector_unsigned_char *)&rss); + + l3_l4e[0] = (uint32_t)vlan1[0] >> 22; + l3_l4e[1] = (uint32_t)vlan1[1] >> 22; + l3_l4e[2] = (uint32_t)vlan1[2] >> 22; + l3_l4e[3] = (uint32_t)vlan1[3] >> 22; + + l3_l4e = (vector_unsigned_int)vec_perm(l3_l4e_flags, + (vector_unsigned_char){}, + *(vector_unsigned_char *)&l3_l4e); + + vlan0 = vec_or(vlan0, rss); + vlan0 = vec_or(vlan0, l3_l4e); + + rx_pkts[0]->ol_flags = (uint64_t)vlan0[2]; + rx_pkts[1]->ol_flags = (uint64_t)vlan0[3]; + rx_pkts[2]->ol_flags = (uint64_t)vlan0[0]; + rx_pkts[3]->ol_flags = (uint64_t)vlan0[1]; +} + +#define PKTLEN_SHIFT 10 + +static inline void +desc_to_ptype_v(vector_unsigned_long_long descs[4], struct rte_mbuf **rx_pkts, + uint32_t *ptype_tbl) +{ + vector_unsigned_long_long ptype0 = vec_mergel(descs[0], descs[1]); + vector_unsigned_long_long ptype1 = vec_mergel(descs[2], descs[3]); + + ptype0[0] = ptype0[0] >> 30; + ptype0[1] = ptype0[1] >> 30; + + ptype1[0] = ptype1[0] >> 30; + ptype1[1] = ptype1[1] >> 30; + + rx_pkts[0]->packet_type = + ptype_tbl[(*(vector_unsigned_char *)&ptype0)[0]]; + rx_pkts[1]->packet_type = + ptype_tbl[(*(vector_unsigned_char *)&ptype0)[8]]; + rx_pkts[2]->packet_type = + ptype_tbl[(*(vector_unsigned_char *)&ptype1)[0]]; + rx_pkts[3]->packet_type = + ptype_tbl[(*(vector_unsigned_char *)&ptype1)[8]]; +} + + /* Notice: + * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts > RTE_I40E_VPMD_RX_BURST, only scan RTE_I40E_VPMD_RX_BURST + * numbers of DD bits + */ +static inline uint16_t +_recv_raw_pkts_vec(struct i40e_rx_queue *rxq, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts, uint8_t *split_packet) +{ + volatile union i40e_rx_desc *rxdp; + struct i40e_rx_entry *sw_ring; + uint16_t nb_pkts_recd; + int pos; + uint64_t var; + vector_unsigned_char shuf_msk; + uint32_t *ptype_tbl = rxq->vsi->adapter->ptype_tbl; + + vector_unsigned_short crc_adjust = (vector_unsigned_short){ + 0, 0, /* ignore pkt_type field */ + rxq->crc_len, /* sub crc on pkt_len */ + 0, /* ignore high-16bits of pkt_len */ + rxq->crc_len, /* sub crc on data_len */ + 0, 0, 0 /* ignore non-length fields */ + }; + vector_unsigned_long_long dd_check, eop_check; + + /* nb_pkts shall be less equal than RTE_I40E_MAX_RX_BURST */ + nb_pkts = RTE_MIN(nb_pkts, RTE_I40E_MAX_RX_BURST); + + /* nb_pkts has to be floor-aligned to RTE_I40E_DESCS_PER_LOOP */ + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_I40E_DESCS_PER_LOOP); + + /* Just the act of getting into the function from the application is + * going to cost about 7 cycles + */ + rxdp = rxq->rx_ring + rxq->rx_tail; + + rte_prefetch0(rxdp); + + /* See if we need to rearm the RX queue - gives the prefetch a bit + * of time to act + */ + if (rxq->rxrearm_nb > RTE_I40E_RXQ_REARM_THRESH) + i40e_rxq_rearm(rxq); + + /* Before we start moving massive data around, check to see if + * there is actually a packet available + */ + if (!(rxdp->wb.qword1.status_error_len & + rte_cpu_to_le_32(1 << I40E_RX_DESC_STATUS_DD_SHIFT))) + return 0; + + /* 4 packets DD mask */ + dd_check = (vector_unsigned_long_long){0x0000000100000001ULL, + 0x0000000100000001ULL}; + + /* 4 packets EOP mask */ + eop_check = (vector_unsigned_long_long){0x0000000200000002ULL, + 0x0000000200000002ULL}; + + /* mask to shuffle from desc. to mbuf */ + shuf_msk = (vector_unsigned_char){ + 0xFF, 0xFF, /* pkt_type set as unknown */ + 0xFF, 0xFF, /* pkt_type set as unknown */ + 14, 15, /* octet 15~14, low 16 bits pkt_len */ + 0xFF, 0xFF, /* skip high 16 bits pkt_len, zero out */ + 14, 15, /* octet 15~14, 16 bits data_len */ + 2, 3, /* octet 2~3, low 16 bits vlan_macip */ + 4, 5, 6, 7 /* octet 4~7, 32bits rss */ + }; + + /* Cache is empty -> need to scan the buffer rings, but first move + * the next 'n' mbufs into the cache + */ + sw_ring = &rxq->sw_ring[rxq->rx_tail]; + + /* A. load 4 packet in one loop + * [A*. mask out 4 unused dirty field in desc] + * B. copy 4 mbuf point from swring to rx_pkts + * C. calc the number of DD bits among the 4 packets + * [C*. extract the end-of-packet bit, if requested] + * D. fill info. from desc to mbuf + */ + + for (pos = 0, nb_pkts_recd = 0; pos < nb_pkts; + pos += RTE_I40E_DESCS_PER_LOOP, + rxdp += RTE_I40E_DESCS_PER_LOOP) { + vector_unsigned_long_long descs[RTE_I40E_DESCS_PER_LOOP]; + vector_unsigned_char pkt_mb1, pkt_mb2, pkt_mb3, pkt_mb4; + vector_unsigned_short staterr, sterr_tmp1, sterr_tmp2; + vector_unsigned_long_long mbp1, mbp2; /* two mbuf pointer + * in one XMM reg. + */ + + /* B.1 load 1 mbuf point */ + mbp1 = *(vector_unsigned_long_long *)&sw_ring[pos]; + /* Read desc statuses backwards to avoid race condition */ + /* A.1 load 4 pkts desc */ + descs[3] = *(vector_unsigned_long_long *)(rxdp + 3); + rte_compiler_barrier(); + + /* B.2 copy 2 mbuf point into rx_pkts */ + *(vector_unsigned_long_long *)&rx_pkts[pos] = mbp1; + + /* B.1 load 1 mbuf point */ + mbp2 = *(vector_unsigned_long_long *)&sw_ring[pos + 2]; + + descs[2] = *(vector_unsigned_long_long *)(rxdp + 2); + rte_compiler_barrier(); + /* B.1 load 2 mbuf point */ + descs[1] = *(vector_unsigned_long_long *)(rxdp + 1); + rte_compiler_barrier(); + descs[0] = *(vector_unsigned_long_long *)(rxdp); + + /* B.2 copy 2 mbuf point into rx_pkts */ + *(vector_unsigned_long_long *)&rx_pkts[pos + 2] = mbp2; + + if (split_packet) { + rte_mbuf_prefetch_part2(rx_pkts[pos]); + rte_mbuf_prefetch_part2(rx_pkts[pos + 1]); + rte_mbuf_prefetch_part2(rx_pkts[pos + 2]); + rte_mbuf_prefetch_part2(rx_pkts[pos + 3]); + } + + /* avoid compiler reorder optimization */ + rte_compiler_barrier(); + + /* pkt 3,4 shift the pktlen field to be 16-bit aligned*/ + vector_unsigned_int len3_temp = vec_xld2(0, + (unsigned int *)&descs[3]); + len3_temp[3] = len3_temp[3] << PKTLEN_SHIFT; + const vector_unsigned_int len3 = len3_temp; + + vector_unsigned_int len2_temp = vec_xld2(0, + (unsigned int *)&descs[2]); + len2_temp[3] = len2_temp[3] << PKTLEN_SHIFT; + const vector_unsigned_int len2 = len2_temp; + + /* merge the now-aligned packet length fields back in */ + descs[3] = (vector_unsigned_long_long)len3; + descs[2] = (vector_unsigned_long_long)len2; + + /* D.1 pkt 3,4 convert format from desc to pktmbuf */ + pkt_mb4 = vec_perm((vector_unsigned_char)descs[3], + (vector_unsigned_char){}, shuf_msk); + pkt_mb3 = vec_perm((vector_unsigned_char)descs[2], + (vector_unsigned_char){}, shuf_msk); + + /* C.1 4=>2 filter staterr info only */ + sterr_tmp2 = vec_mergel((vector_unsigned_short)descs[3], + (vector_unsigned_short)descs[2]); + /* C.1 4=>2 filter staterr info only */ + sterr_tmp1 = vec_mergel((vector_unsigned_short)descs[1], + (vector_unsigned_short)descs[0]); + /* D.2 pkt 3,4 set in_port/nb_seg and remove crc */ + pkt_mb4 = (vector_unsigned_char)((vector_unsigned_short)pkt_mb4 + - crc_adjust); + pkt_mb3 = (vector_unsigned_char)((vector_unsigned_short)pkt_mb3 + - crc_adjust); + + /* pkt 1,2 shift the pktlen field to be 16-bit aligned*/ + const vector_unsigned_int len1 = + vec_sll(vec_xld2(0, (unsigned int *)&descs[1]), + (vector_unsigned_int){0, 0, 0, PKTLEN_SHIFT}); + const vector_unsigned_int len0 = + vec_sll(vec_xld2(0, (unsigned int *)&descs[0]), + (vector_unsigned_int){0, 0, 0, PKTLEN_SHIFT}); + + /* merge the now-aligned packet length fields back in */ + descs[1] = (vector_unsigned_long_long)len1; + descs[0] = (vector_unsigned_long_long)len0; + + /* D.1 pkt 1,2 convert format from desc to pktmbuf */ + pkt_mb2 = vec_perm((vector_unsigned_char)descs[1], + (vector_unsigned_char){}, shuf_msk); + pkt_mb1 = vec_perm((vector_unsigned_char)descs[0], + (vector_unsigned_char){}, shuf_msk); + + /* C.2 get 4 pkts staterr value */ + staterr = (vector_unsigned_short)vec_mergeh(sterr_tmp1, + sterr_tmp2); + + /* D.3 copy final 3,4 data to rx_pkts */ + vec_xstd2(pkt_mb4, 0, (unsigned char *)&rx_pkts[pos + 3] + ->rx_descriptor_fields1); + vec_xstd2(pkt_mb3, 0, (unsigned char *)&rx_pkts[pos + 2] + ->rx_descriptor_fields1); + + /* D.2 pkt 1,2 set in_port/nb_seg and remove crc */ + pkt_mb2 = (vector_unsigned_char)((vector_unsigned_short)pkt_mb2 + - crc_adjust); + pkt_mb1 = (vector_unsigned_char)((vector_unsigned_short)pkt_mb1 + - crc_adjust); + + /* C* extract and record EOP bit */ + if (split_packet) { + vector_unsigned_char eop_shuf_mask = + (vector_unsigned_char){ + 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, + 0x04, 0x0C, 0x00, 0x08 + }; + + /* and with mask to extract bits, flipping 1-0 */ + vector_unsigned_char eop_bits = + vec_and((vector_unsigned_char)vec_nor(staterr, + staterr), (vector_unsigned_char)eop_check); + /* the staterr values are not in order, as the count + * count of dd bits doesn't care. However, for end of + * packet tracking, we do care, so shuffle. This also + * compresses the 32-bit values to 8-bit + */ + eop_bits = vec_perm(eop_bits, (vector_unsigned_char){}, + eop_shuf_mask); + /* store the resulting 32-bit value */ + *split_packet = (vec_xld2(0, + (unsigned int *)&eop_bits))[0]; + split_packet += RTE_I40E_DESCS_PER_LOOP; + + /* zero-out next pointers */ + rx_pkts[pos]->next = NULL; + rx_pkts[pos + 1]->next = NULL; + rx_pkts[pos + 2]->next = NULL; + rx_pkts[pos + 3]->next = NULL; + } + + /* C.3 calc available number of desc */ + staterr = vec_and(staterr, (vector_unsigned_short)dd_check); + + /* D.3 copy final 1,2 data to rx_pkts */ + vec_xstd2(pkt_mb2, 0, (unsigned char *)&rx_pkts[pos + 1] + ->rx_descriptor_fields1); + vec_xstd2(pkt_mb1, 0, (unsigned char *)&rx_pkts[pos] + ->rx_descriptor_fields1); + + desc_to_ptype_v(descs, &rx_pkts[pos], ptype_tbl); + desc_to_olflags_v(descs, &rx_pkts[pos]); + + /* C.4 calc avaialbe number of desc */ + var = __builtin_popcountll((vec_xld2(0, + (unsigned long long *)&staterr)[0])); + nb_pkts_recd += var; + if (likely(var != RTE_I40E_DESCS_PER_LOOP)) + break; + } + + /* Update our internal tail pointer */ + rxq->rx_tail = (uint16_t)(rxq->rx_tail + nb_pkts_recd); + rxq->rx_tail = (uint16_t)(rxq->rx_tail & (rxq->nb_rx_desc - 1)); + rxq->rxrearm_nb = (uint16_t)(rxq->rxrearm_nb + nb_pkts_recd); + + return nb_pkts_recd; +} + + /* Notice: + * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts > RTE_I40E_VPMD_RX_BURST, only scan RTE_I40E_VPMD_RX_BURST + * numbers of DD bits + */ +uint16_t +i40e_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + return _recv_raw_pkts_vec(rx_queue, rx_pkts, nb_pkts, NULL); +} + + /* vPMD receive routine that reassembles scattered packets + * Notice: + * - nb_pkts < RTE_I40E_DESCS_PER_LOOP, just return no packet + * - nb_pkts > RTE_I40E_VPMD_RX_BURST, only scan RTE_I40E_VPMD_RX_BURST + * numbers of DD bits + */ +uint16_t +i40e_recv_scattered_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct i40e_rx_queue *rxq = rx_queue; + uint8_t split_flags[RTE_I40E_VPMD_RX_BURST] = {0}; + + /* get some new buffers */ + uint16_t nb_bufs = _recv_raw_pkts_vec(rxq, rx_pkts, nb_pkts, + split_flags); + if (nb_bufs == 0) + return 0; + + /* happy day case, full burst + no packets to be joined */ + const uint64_t *split_fl64 = (uint64_t *)split_flags; + + if (rxq->pkt_first_seg == NULL && + split_fl64[0] == 0 && split_fl64[1] == 0 && + split_fl64[2] == 0 && split_fl64[3] == 0) + return nb_bufs; + + /* reassemble any packets that need reassembly*/ + unsigned int i = 0; + + if (!rxq->pkt_first_seg) { + /* find the first split flag, and only reassemble then*/ + while (i < nb_bufs && !split_flags[i]) + i++; + if (i == nb_bufs) + return nb_bufs; + } + return i + reassemble_packets(rxq, &rx_pkts[i], nb_bufs - i, + &split_flags[i]); +} + +static inline void +vtx1(volatile struct i40e_tx_desc *txdp, + struct rte_mbuf *pkt, uint64_t flags) +{ + uint64_t high_qw = (I40E_TX_DESC_DTYPE_DATA | + ((uint64_t)flags << I40E_TXD_QW1_CMD_SHIFT) | + ((uint64_t)pkt->data_len << I40E_TXD_QW1_TX_BUF_SZ_SHIFT)); + + vector_unsigned_long_long descriptor = (vector_unsigned_long_long){ + pkt->buf_iova + pkt->data_off, high_qw}; + *(vector_unsigned_long_long *)txdp = descriptor; +} + +static inline void +vtx(volatile struct i40e_tx_desc *txdp, + struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags) +{ + int i; + + for (i = 0; i < nb_pkts; ++i, ++txdp, ++pkt) + vtx1(txdp, *pkt, flags); +} + +uint16_t +i40e_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts) +{ + struct i40e_tx_queue *txq = (struct i40e_tx_queue *)tx_queue; + volatile struct i40e_tx_desc *txdp; + struct i40e_tx_entry *txep; + uint16_t n, nb_commit, tx_id; + uint64_t flags = I40E_TD_CMD; + uint64_t rs = I40E_TX_DESC_CMD_RS | I40E_TD_CMD; + int i; + + /* cross rx_thresh boundary is not allowed */ + nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh); + + if (txq->nb_tx_free < txq->tx_free_thresh) + i40e_tx_free_bufs(txq); + + nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); + nb_commit = nb_pkts; + if (unlikely(nb_pkts == 0)) + return 0; + + tx_id = txq->tx_tail; + txdp = &txq->tx_ring[tx_id]; + txep = &txq->sw_ring[tx_id]; + + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts); + + n = (uint16_t)(txq->nb_tx_desc - tx_id); + if (nb_commit >= n) { + tx_backlog_entry(txep, tx_pkts, n); + + for (i = 0; i < n - 1; ++i, ++tx_pkts, ++txdp) + vtx1(txdp, *tx_pkts, flags); + + vtx1(txdp, *tx_pkts++, rs); + + nb_commit = (uint16_t)(nb_commit - n); + + tx_id = 0; + txq->tx_next_rs = (uint16_t)(txq->tx_rs_thresh - 1); + + /* avoid reach the end of ring */ + txdp = &txq->tx_ring[tx_id]; + txep = &txq->sw_ring[tx_id]; + } + + tx_backlog_entry(txep, tx_pkts, nb_commit); + + vtx(txdp, tx_pkts, nb_commit, flags); + + tx_id = (uint16_t)(tx_id + nb_commit); + if (tx_id > txq->tx_next_rs) { + txq->tx_ring[txq->tx_next_rs].cmd_type_offset_bsz |= + rte_cpu_to_le_64(((uint64_t)I40E_TX_DESC_CMD_RS) << + I40E_TXD_QW1_CMD_SHIFT); + txq->tx_next_rs = + (uint16_t)(txq->tx_next_rs + txq->tx_rs_thresh); + } + + txq->tx_tail = tx_id; + + I40E_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); + + return nb_pkts; +} + +void __attribute__((cold)) +i40e_rx_queue_release_mbufs_vec(struct i40e_rx_queue *rxq) +{ + _i40e_rx_queue_release_mbufs_vec(rxq); +} + +int __attribute__((cold)) +i40e_rxq_vec_setup(struct i40e_rx_queue *rxq) +{ + return i40e_rxq_vec_setup_default(rxq); +} + +int __attribute__((cold)) +i40e_txq_vec_setup(struct i40e_tx_queue __rte_unused * txq) +{ + return 0; +} + +int __attribute__((cold)) +i40e_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev) +{ + return i40e_rx_vec_dev_conf_condition_check_default(dev); +} -- 2.17.1