From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0063.outbound.protection.outlook.com [104.47.33.63]) by dpdk.org (Postfix) with ESMTP id 3A77937B4 for ; Fri, 1 Jul 2016 13:18:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=B3cfLGAFawybZ25mxw04hYxdyaB9/YsB6YFnnHVBHDs=; b=X7c+k59lvmC1qFH1Ui/KPnVbdQtyu+lAsRY6/1sOss4nsoesgsEYndBgPni7KKdR93s2MGCmILB/nCWfb55cRQD73Q2P1Vyt1eUnPkoyQLr8Ds/+grrqbE5xxlOFEGiURCU5ytgPW36XQu5SkXrNX3YWGTnJIbm6iadG4P+xirY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain.localdomain (122.167.11.22) by BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) with Microsoft SMTP Server (TLS) id 15.1.523.12; Fri, 1 Jul 2016 11:18:04 +0000 From: Jerin Jacob To: CC: , , , , , Jerin Jacob Date: Fri, 1 Jul 2016 16:46:38 +0530 Message-ID: <1467371814-26754-4-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1467371814-26754-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1467028448-8914-1-git-send-email-jerin.jacob@caviumnetworks.com> <1467371814-26754-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [122.167.11.22] X-ClientProxiedBy: PN1PR01CA0027.INDPRD01.PROD.OUTLOOK.COM (10.164.137.34) To BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) X-MS-Office365-Filtering-Correlation-Id: c9a81095-5332-48f4-b97e-08d3a1a160dd X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 2:9ybnkqh9g+eW2+BlsKiqLm6M1weSuI3HX5gufF1UgAmgBLuGRst8kdHzAF4Li5okZ/Q/0D9oGKEI91bAOqkDIFgF13q7iRyjaaj2SHDepTNyY5Yl5OLiJeA1Y1fzfj+jGU4HTAuUmlnkUWF514+uKSTwo8H90AGpGIw4LOdC4Ev4qwqu1tGfJuaKb11iDczf; 3:cVykDShPaL/axdAUB7HrER09VAsUdV4snFl+YCChZbsHsxp4TRESrQQ81yCxLS9hilzLTW+b7NsunK+lkbyvPlW+RYmRj+AkWF6KYnhHtesrSdOMFioKUVAoF3GfZr/V; 25:7vrlZh4dog5dnAgAWli5hbyvSc8uRDdQhYZrqPSLHND3Li6zEFkRysjjZDGCAjr8AiLn6zjJ7r+kws4xZA8QuoLGBe71cjDbLyaMtc1DxzTuyJDw+NO1UQQgKB74tL9jras2UXcf5KQrcP657WFUMjLdjifUxXZ3qCDGVKuzyB190O1e6WuqXjR0FTqM6X//bSqVkgvqg6R/sVYNk1nZHd1x370D6uyeQepa/17GlREaoNZ02lUQIyouaPYAiBERG5R2Tuh05sNuKALPhfYgkynDKWolRhDwvPpz69SD42arRWXdm65UEuo1iNsMD2WWVEYU7QPXyyFGDHhgEnSbrtQnE2g4UkjcpagblFakiznSOrN/mQ4iqv2D4TSV3ao8BgUMNYje61e1ZOgQ0M35jX4H7GwP+flqmVSfWl+VY+4= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 20:UjTuH8dKKUaYNlJYFNDiB1MHL6GrGJ2/1/NNRGPYfLc6yqEekQyIcComvxGosYkG98nhprOBzauiKUDiT6lVEnuuS4pZOAFVsMPrNv9tdi9S8zeWm941JYVwzZ1bTAi6p7NZ/+ThmsvsmzMe1tX1mNyssEU0V/gmak0VP9cqz5nXd9PClfuvhVy+xa+at6JjP4fVr6yBB+6XdelLFiGIhdflA6gYtYGQen1x+M1ac6davEuF0OGkH9MZ2VEv4lx4ViFxhjZHR2OWle60VbWmY009LysqIv/Yk18FiCbXc/dRmUiAMrcUHNfovD+ycU/gg9zq7Ep1oKCkyewH0T8BgR+Wqb1tqNzaATjezbmypOu6kd1MT/MphTvmDPgGUF7cyEEf1zHuqewC6+0UNDtJqZ0wH1CEypTpDjafOxt+Z3ZYXueVWWiIlhJ1c5K7aTygx5mYp4yK0pixyjU5fH5mzM5pDrdJSF7YVfcQaiXLf1XhHIQVajJ96giOz1fN1JFyRo/2yR3+15mpJIHyBhmG3nwNkXteJdOhohbzUVbme75HmdQRUIQeiutyFYRLXU1tqJzKBTJ6+DFydsbkOKBD74/e4LRLGrrz9eVrGZNNMgE= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BLUPR0701MB1714; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 4:Zcg77nfBszNFfw9JziX/7m08hFv+3LejQRqghbTNV/nphxe4Ih2z32z3f0iRO6M7dmpwAD4dclwuP4oJm03tQRzrRoqYFPANd9SyniLeUrU59EukrPumPHUlOpKQOMezqN9dJ65fKJk0ilQkj3W2rLkpZTH6btMXKrrEn1W0jKrMr0VLEpdFgY3+2HsTDSucsQiMIr2hcgqHbfxcpj+Snq7Yl81C0WcXIP7dsASw/UOF7Pr5luobzTAKglP9eTntafl95wooK2KAO91Z1p1SefYrIOo4S4aOAAb2dpCT3nk1HkTT/EH0nUKDQhXl49NDhisXvOpeaikuT2nKQYK5Q23yUDKEV/0X8fBYmxv3FelmA6qf1Q9bf48rlqzxWm0x X-Forefront-PRVS: 0990C54589 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6069001)(7916002)(189002)(199003)(57704003)(68736007)(8676002)(5003940100001)(81166006)(81156014)(66066001)(7846002)(76176999)(50466002)(4001430100002)(50986999)(305945005)(586003)(2906002)(4326007)(92566002)(48376002)(3846002)(6116002)(101416001)(77096005)(2950100001)(36756003)(42186005)(107886002)(110136002)(97736004)(19580395003)(189998001)(7736002)(229853001)(33646002)(105586002)(2351001)(19580405001)(106356001)(47776003)(50226002)(7099028); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1714; H:localhost.localdomain.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0701MB1714; 23:fxgSAVBkWZpHsUW4asAlcisS5XIhTfZlBZ7EjnO?= =?us-ascii?Q?OuuAwjKMP2L3vlXMHg8tqJeV5/nUp0F9OnJ+jOf5hv5WUW0z/dUDglAOjgD6?= =?us-ascii?Q?6FkYpphvFsVlZPsd74v99bhCdC0SVR+1om/ZiPHaLKe4eNSqaGVZJGWqAymA?= =?us-ascii?Q?08+hm6u66UcW6rldWMaa8lQK0+JLss5i8lwmyssGSSZ5/TVY3H2+iApW+vd2?= =?us-ascii?Q?BjdteqzuEaZCjW83Q+GoehCfYrGTV22EFOD1tR8NmXKGt2dVeNuP5aEbof4g?= =?us-ascii?Q?Eb6vkwHIBkkZdn0GGlEc/kROwJGlJmp3CiM19T6PzGra9aCX1TiFrS+l1lKy?= =?us-ascii?Q?TKBbAHn+Hrji2Q1LfpLdBp/SnhJXQYGIxGqATxhG5E1k82NNpz6JRKLM7jQo?= =?us-ascii?Q?lGFPdNkZYJcbGlIOz1KWvUtpr4o6bsyJihsltu0SDVf7c3vwTnN4PAn9MW69?= =?us-ascii?Q?odQmsmrjRsbwIsjAfrGQ8bVbbrRNd7QX1KN9u6ISSW9Br4i99v5MzaodiojI?= =?us-ascii?Q?u4kWCyRs+s0p4bAE5QvnsZnc5LpGADo7c/Li/LXUYfRLQ4PDZJh/lA0bzNxL?= =?us-ascii?Q?mnEuX+gKSYdg++D66RY9rmTqBp8ziJFW83InQGVqSBTZfVHCwZPAgLI30J5w?= =?us-ascii?Q?7LqDP/ek8ngQZ7idGurddPORjdTK+kqLSs/52LUoKnFKpENL+5c/9pFyyzcB?= =?us-ascii?Q?cH38Kekvpv82Of+ZRHo9Ii6tf6mmFlo3SXBD+E4WbIOmvSDcNSO/j62Gq5ty?= =?us-ascii?Q?7zXy28qPt3EE1vQ6TnMpPq9IvNuGm0OyTv/ypUQK34CzcYEGpHLJVL40jzYy?= =?us-ascii?Q?7Imhj7WAqbrUomC3083J3aJEw1pFE8FwGRmlomvkFN5wKfG6pUoVt4kYSj9L?= =?us-ascii?Q?bj671tG45fD1MyrgLqqwUa3t8eKEtM2UbGg350WDUT0YJRh/obtd3RXEN/KC?= =?us-ascii?Q?m5VX4GGbY2CTPZRNgepnNV2dypvuCCr2JbvgxSt91hhZL138BnDFWi2sZP79?= =?us-ascii?Q?m0Vfl85HrJGUVLsPfAuinszHtqjbQ84lfz0+RJKg0EV8jlXUuVAp/jT5NboR?= =?us-ascii?Q?LAVmsUpisumt88HzlQ0/vMgKnZ7nVyyBRHdmUCBBExmj08XVR8SeXb6tPY2e?= =?us-ascii?Q?VX6PXzlEd4jIQKyEo8EgDLTsnGWX4wbs7XtSxdwIRXQuLYv+SavnuPQ=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 6:2xh751wD1zijZnhla67zWqcIN4FfXHCKLDWoS4HIns8HjkSAv8Ya+oq4spbq9XOKA6F43m7JFmbUiwySkTdFsJDS2+fv3pmXSYpzL6w1/XcagePKb/FnAkNVB3yFJFG8Gf61Dz10knWPnVtsqrBzjL6ZoDxcwkd6NtaYS7duqCuOVBfqZ6kqf3Iz9WF7IvGDfWAyIZ43JP2YxVN++J272eL9AbeRozLLncnLlPx16ynQjqBszAMjYV4uZ2mp7iliCFRCxq4eXkH4Y4Ra8yeRMDUrcERTDnz8vLJMOuYEUkfvB+xbzo53fpIZnrHs469/; 5:XyovNaeDb5abJ269zaFh9oNE4RdeVS/8jkUz2vP2iglPhYXDwNYDw51LLGJVQK1O2cz3lFzlaVoN9ixnreyIKEW9oXcwZx0eMp0GC57fs65HI4wuiRD5zdb7DJ/r0xsNAtINPIg38Q6ijW9A4QP68A==; 24:gRNubtw8lObF6pOiQVvCluCBcs/1F6EWWpY4RKaxdHS3j7/1muckaCe7QeoU+hdqgIE9a4ESlDwrDcg00AuLzPOwC6ZnCfGolchQwyE1lpc=; 7:7m0egIeKBvKgOSlkfiQeg0kwey1KNMd0ZxI6yLwJnVSSxz1BP7U8YQfHxvolZ0MFXI6bLItNgNOBuY5ct8hydaM+Tu3utXev6Ec6NSgOJTtEdHKVgjdigVgN4mBUEv7rumMbICXuPFXiGZHhCVxledQrAMW6VUl/5K+tdnsE/ujxB81UTrZ59oWeqdQI7UW7orjiOWnWf9MX8SzmObSj/XlBGQNGbvEH7nqYQVoDYjNrGjQ5c4KUsj6QOdQjKfBtOrJjM4V/O0evwXARKiQ8MA== SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jul 2016 11:18:04.7035 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1714 Subject: [dpdk-dev] [PATCH v2 3/3] virtio: add neon support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jul 2016 11:18:09 -0000 Added neon based Rx vector implementation. Selection of the new handler based neon availability at runtime. Updated the release notes and MAINTAINERS file. Signed-off-by: Jerin Jacob --- MAINTAINERS | 1 + doc/guides/rel_notes/release_16_07.rst | 2 + drivers/net/virtio/Makefile | 2 + drivers/net/virtio/virtio_rxtx.c | 3 + drivers/net/virtio/virtio_rxtx_simple_neon.c | 235 +++++++++++++++++++++++++++ 5 files changed, 243 insertions(+) create mode 100644 drivers/net/virtio/virtio_rxtx_simple_neon.c diff --git a/MAINTAINERS b/MAINTAINERS index a59191e..ab04cee 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -143,6 +143,7 @@ F: lib/librte_acl/acl_run_neon.* F: lib/librte_lpm/rte_lpm_neon.h F: lib/librte_hash/rte*_arm64.h F: drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c +F: drivers/net/virtio/virtio_rxtx_simple_neon.c EZchip TILE-Gx M: Zhigang Lu diff --git a/doc/guides/rel_notes/release_16_07.rst b/doc/guides/rel_notes/release_16_07.rst index 9e2a817..3a5add5 100644 --- a/doc/guides/rel_notes/release_16_07.rst +++ b/doc/guides/rel_notes/release_16_07.rst @@ -174,6 +174,8 @@ New Features section of the "Network Interface Controller Drivers" document. +* **Virtio NEON support for ARM.** + Resolved Issues --------------- diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile index c4103b7..97972a6 100644 --- a/drivers/net/virtio/Makefile +++ b/drivers/net/virtio/Makefile @@ -54,6 +54,8 @@ SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple.c ifeq ($(CONFIG_RTE_ARCH_X86),y) SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple_sse.c +else ifneq ($(filter y,$(CONFIG_RTE_ARCH_ARM) $(CONFIG_RTE_ARCH_ARM64)),) +SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple_neon.c endif ifeq ($(CONFIG_RTE_VIRTIO_USER),y) diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c index a4d4a57..19d1742 100644 --- a/drivers/net/virtio/virtio_rxtx.c +++ b/drivers/net/virtio/virtio_rxtx.c @@ -481,6 +481,9 @@ virtio_update_rxtx_handler(struct rte_eth_dev *dev, #if defined RTE_ARCH_X86 if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_SSE3)) use_simple_rxtx = 1; +#elif defined RTE_ARCH_ARM64 || defined CONFIG_RTE_ARCH_ARM + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_NEON)) + use_simple_rxtx = 1; #endif /* Use simple rx/tx func if single segment and no offloads */ if (use_simple_rxtx && diff --git a/drivers/net/virtio/virtio_rxtx_simple_neon.c b/drivers/net/virtio/virtio_rxtx_simple_neon.c new file mode 100644 index 0000000..793eefb --- /dev/null +++ b/drivers/net/virtio/virtio_rxtx_simple_neon.c @@ -0,0 +1,235 @@ +/* + * BSD LICENSE + * + * Copyright (C) Cavium networks Ltd. 2016 + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "virtio_rxtx_simple.h" + +#define RTE_VIRTIO_VPMD_RX_BURST 32 +#define RTE_VIRTIO_DESC_PER_LOOP 8 +#define RTE_VIRTIO_VPMD_RX_REARM_THRESH RTE_VIRTIO_VPMD_RX_BURST + +/* virtio vPMD receive routine, only accept(nb_pkts >= RTE_VIRTIO_DESC_PER_LOOP) + * + * This routine is for non-mergeable RX, one desc for each guest buffer. + * This routine is based on the RX ring layout optimization. Each entry in the + * avail ring points to the desc with the same index in the desc ring and this + * will never be changed in the driver. + * + * - nb_pkts < RTE_VIRTIO_DESC_PER_LOOP, just return no packet + */ +uint16_t +virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct virtnet_rx *rxvq = rx_queue; + struct virtqueue *vq = rxvq->vq; + uint16_t nb_used; + uint16_t desc_idx; + struct vring_used_elem *rused; + struct rte_mbuf **sw_ring; + struct rte_mbuf **sw_ring_end; + uint16_t nb_pkts_received; + + uint8x16_t shuf_msk1 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 4, 5, 0xFF, 0xFF, /* pkt len */ + 4, 5, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + uint8x16_t shuf_msk2 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 12, 13, 0xFF, 0xFF, /* pkt len */ + 12, 13, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Subtract the header length. + * In which case do we need the header length in used->len ? + */ + uint16x8_t len_adjust = { + 0, 0, + (uint16_t)vq->hw->vtnet_hdr_size, 0, + (uint16_t)vq->hw->vtnet_hdr_size, + 0, + 0, 0 + }; + + if (unlikely(nb_pkts < RTE_VIRTIO_DESC_PER_LOOP)) + return 0; + + nb_used = VIRTQUEUE_NUSED(vq); + + rte_rmb(); + + if (unlikely(nb_used == 0)) + return 0; + + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_VIRTIO_DESC_PER_LOOP); + nb_used = RTE_MIN(nb_used, nb_pkts); + + desc_idx = (uint16_t)(vq->vq_used_cons_idx & (vq->vq_nentries - 1)); + rused = &vq->vq_ring.used->ring[desc_idx]; + sw_ring = &vq->sw_ring[desc_idx]; + sw_ring_end = &vq->sw_ring[vq->vq_nentries]; + + rte_prefetch_non_temporal(rused); + + if (vq->vq_free_cnt >= RTE_VIRTIO_VPMD_RX_REARM_THRESH) { + virtio_rxq_rearm_vec(rxvq); + if (unlikely(virtqueue_kick_prepare(vq))) + virtqueue_notify(vq); + } + + for (nb_pkts_received = 0; + nb_pkts_received < nb_used;) { + uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP]; + + mbp[0] = vld1q_u64((uint64_t *)(sw_ring + 0)); + desc[0] = vld1q_u64((uint64_t *)(rused + 0)); + vst1q_u64((uint64_t *)&rx_pkts[0], mbp[0]); + + mbp[1] = vld1q_u64((uint64_t *)(sw_ring + 2)); + desc[1] = vld1q_u64((uint64_t *)(rused + 2)); + vst1q_u64((uint64_t *)&rx_pkts[2], mbp[1]); + + mbp[2] = vld1q_u64((uint64_t *)(sw_ring + 4)); + desc[2] = vld1q_u64((uint64_t *)(rused + 4)); + vst1q_u64((uint64_t *)&rx_pkts[4], mbp[2]); + + mbp[3] = vld1q_u64((uint64_t *)(sw_ring + 6)); + desc[3] = vld1q_u64((uint64_t *)(rused + 6)); + vst1q_u64((uint64_t *)&rx_pkts[6], mbp[3]); + + pkt_mb[1] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk2)); + pkt_mb[0] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk1)); + pkt_mb[1] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[1]), len_adjust)); + pkt_mb[0] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[0]), len_adjust)); + vst1q_u64((void *)&rx_pkts[1]->rx_descriptor_fields1, + pkt_mb[1]); + vst1q_u64((void *)&rx_pkts[0]->rx_descriptor_fields1, + pkt_mb[0]); + + pkt_mb[3] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk2)); + pkt_mb[2] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk1)); + pkt_mb[3] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[3]), len_adjust)); + pkt_mb[2] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[2]), len_adjust)); + vst1q_u64((void *)&rx_pkts[3]->rx_descriptor_fields1, + pkt_mb[3]); + vst1q_u64((void *)&rx_pkts[2]->rx_descriptor_fields1, + pkt_mb[2]); + + pkt_mb[5] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk2)); + pkt_mb[4] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk1)); + pkt_mb[5] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[5]), len_adjust)); + pkt_mb[4] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[4]), len_adjust)); + vst1q_u64((void *)&rx_pkts[5]->rx_descriptor_fields1, + pkt_mb[5]); + vst1q_u64((void *)&rx_pkts[4]->rx_descriptor_fields1, + pkt_mb[4]); + + pkt_mb[7] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk2)); + pkt_mb[6] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk1)); + pkt_mb[7] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[7]), len_adjust)); + pkt_mb[6] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[6]), len_adjust)); + vst1q_u64((void *)&rx_pkts[7]->rx_descriptor_fields1, + pkt_mb[7]); + vst1q_u64((void *)&rx_pkts[6]->rx_descriptor_fields1, + pkt_mb[6]); + + if (unlikely(nb_used <= RTE_VIRTIO_DESC_PER_LOOP)) { + if (sw_ring + nb_used <= sw_ring_end) + nb_pkts_received += nb_used; + else + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + if (unlikely(sw_ring + RTE_VIRTIO_DESC_PER_LOOP >= + sw_ring_end)) { + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + nb_pkts_received += RTE_VIRTIO_DESC_PER_LOOP; + + rx_pkts += RTE_VIRTIO_DESC_PER_LOOP; + sw_ring += RTE_VIRTIO_DESC_PER_LOOP; + rused += RTE_VIRTIO_DESC_PER_LOOP; + nb_used -= RTE_VIRTIO_DESC_PER_LOOP; + } + } + } + + vq->vq_used_cons_idx += nb_pkts_received; + vq->vq_free_cnt += nb_pkts_received; + rxvq->stats.packets += nb_pkts_received; + return nb_pkts_received; +} -- 2.5.5