From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0043.outbound.protection.outlook.com [104.47.38.43]) by dpdk.org (Postfix) with ESMTP id 238E75960 for ; Tue, 5 Jul 2016 14:50:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=S04YXyEIhGEwrArqAfBK3ULM0pVbCw8MGDP0MUdlD2E=; b=Nc0BojGDyyKaQaVhkjS7aHUd/rA30hixzUohcx8fp+kGWOnDyAnJidbp9FL1rgAs20xKcinsruVtMji6ERixFUD2HR3GP21uLYzsSl6v+VUupUaGbCdWmKj+OZT7sNCOuY1BeZWrA5Lo62tQOcRS2io8GP0PtkTkSm0eFwW55Po= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain.com (111.93.218.67) by BLUPR0701MB1716.namprd07.prod.outlook.com (10.163.85.142) with Microsoft SMTP Server (TLS) id 15.1.534.14; Tue, 5 Jul 2016 12:50:29 +0000 From: Jerin Jacob To: CC: , , , , , Jerin Jacob Date: Tue, 5 Jul 2016 18:19:26 +0530 Message-ID: <1467722966-2572-5-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1467722966-2572-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1467371814-26754-1-git-send-email-jerin.jacob@caviumnetworks.com> <1467722966-2572-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: MAXPR01CA0020.INDPRD01.PROD.OUTLOOK.COM (10.164.147.27) To BLUPR0701MB1716.namprd07.prod.outlook.com (10.163.85.142) X-MS-Office365-Filtering-Correlation-Id: a659e77d-7f6b-46d5-635a-08d3a4d2f394 X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1716; 2:T94e5FqsMqDr2FwXJuerNBO0Zz1DcPDw6BODnJ13Ue0/Y6nNbuIZez1X/T1NytEEy9qzHi+qtuXo+OnOmfYsD3D1wneoi0xNVDTt/eCG3VtHO6oGN1lNLvbV7ZrAORHruMq1MHVw8pbF7uiVKkNhtzuLRo7Ujhf7WHwWdq59EAuLRofK+TkzDPm6KxYkjW2H; 3:3IsllpjsS1QhqjJJWUMk0L/xClU+cJ7TnRW6U/dXbQJjG9wdKiWKTO3MK3ZrBsATlxr0gW0wS078RDcywrDcl8WzxFtE/Ri/CL+St6tSTBE6dZarx5JVKcLfVT3OXvMf X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR0701MB1716; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1716; 25:0XMt2JwkBpbRr6rb2uYz4kedaOPZQwHThGWxb62Thd1m3se7j/Euil8Q4FBGCx539chwcKdSoHTFBXaG9CC/jHslor0m/VCseqKHAigpanq9c9WcCZX9/o2JUbF+I1rghMvMFACoj/ie3MDLNICzuJIy9jfDjxn8/r32sbpC7XE/pAixC4MaWTBNKAVYS6RJhEqGSArzmzqIP5Rs7I+CL30oAU8dz+s4L0jPj1OXjudLwmlqdkLLNDnhBJnVgBUTm6tHSBuJv/QvvC9FM5Ky+bu8DDSnCEVH4OKWMz+VsgRiSlZRa6V+4pddNuzjrIx8TrKpK/WqJWP7CItR7EosjW0w0OMTBffbfGbw4vyAxtbhYE+sk2XuHcITB9PsxUvNntJaumV6W6QPU2BAW1S/EEaVh/tHHe3GghKAPOXhbggPjDJwKik7+yOqhSZMeMhwk40NZ26ZX7Xcg0phVnE3E6d1mySDAnc4NlPMuRneaWZ4kKWjwDPvKAL/pzzp7zukKIUWYUSeSf1h1lB7nJ7OEPr+9a0x2k44r63fOMfgcoqVICEzVmiC/zqUiRyZKmBBoKgRTpCdGYHzvEJiUlBy7Wcwr8AK86YNB/gWT/SFpqyF6VAb6K0PaZH5gHN0NYpXH9ztlWHRpQpkW1J+Uzd1DtxSiKws4wTYKt7CNKJBJT0+/QMNF9wsXTJkRGz8Anh5ZUWR8XYmdah2k0NdnxOz4aLSC46b34XEBRqJVftsViJQAJXf4s4/b1V637WPJNPWvjwDclF2Mx1Wz86AVgBfpg== X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1716; 31:7s1+V4FZ+gPU+LJYz8tPWGFskWGPY89OwCZd1493e6DFbobPQGLJ9sQImS3/zyINIh+uLsHzH4bTfT+Nlx/LN7EX99hwJCTVc4yh5ADSxKw4B2YsgbENHvRCV0ln0yUauJobre71ZToPz0Jk1p8ViuItOvGzbpNSNk9Cto690g7i8CnZNPxoFXJ60xWiEcpZP2bB3ZsU6aQe7+0Iz7gvzQ==; 20:k59Ux2oika2zHmk7VB8RHudGtY7jiw0U7WXcTTNkdIl8pPiW0ivKBvoDozlJGjGQPl70yfXc7apqlHxVv1f0Ps880rNRuqXPS3tIqIPhJkbRAFy4iyDKUiKY0CMYZtnMXquMlPLDzz/Rn6OMbMyhaQIvlbF6ky6OEGES9y1fxVbhjKhOaH2baZ6KhIYPqiXsDH4lW+pq97mAPNxZDPf9xfxBhBptx/nHS2CfrUBMG9kcbYUA2bX8/k0VBe7JXDnGrg3jF1UZuCBdeFzrq+BtTpiX5qyg9a8kR2cltEOOpttLBgRVe2+7aDtL5IWockQ1tmpjdhzl6N8RmP8JwAdZkTjx8yBAoBJCGcYlqozKGYg9mxRkdAVaPUXLUy9WL5Hb2ArGS57vlEUuN+iMFyGfp7ewJVWiVOZQRgue8r3139bDi37qZvQ+l6U9lt6FNpwFtdB+nBSWsAAxAO2B++I6aJ+1UnmVn/5YEVmp+WFXqMBfor17z1Cd9RHElXdQwL4eCbbz3T8HHMi7bE95Jx8t++fwEsvIXJCavx7bRGLo6n6+/59T52Oov+d+RClo9Y1KUQ5mAJgP2ZWoTJBoTMlZtTeA2fsXBV54YxW46PiqtdM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001); SRVR:BLUPR0701MB1716; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1716; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1716; 4:TSCKRR4dfJlVFk/LQrx1cMNnHjdmMHXfuTPP5Tw5Pxo1whQ69lbl3HdO1J4216G4qhjxUGcG+bylFW5x8qXwIhzBkYuQU4nYGr3Ml6ILFx0ym8H/6aJl1sGuqmMjcAeL9rvBCJanH9uduztBO6jCCmKH2SCke+5UCgsAXImw9FRU5YNZFEiPVlxwlTy/24c41Db79gpmaVVQU2VCfeUxkdchLy5qm1qtULWvTpGd9jjOWO+3m5C+x9/EiEiMi1rZmQlSnlN6eGtreom0XzgsVgazs+lf0zpsdnWFvSciEXv/ewLwSPFaZOKXA+xQS3tTQa5YgxWk5opDYTB3b97Z1N8fjcZyMiPsvF7gA5kjypdfwoMCvYBw6AtuhPay4GZJ X-Forefront-PRVS: 0994F5E0C5 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(6069001)(7916002)(199003)(189002)(57704003)(305945005)(7846002)(101416001)(7736002)(19580395003)(107886002)(97736004)(19580405001)(189998001)(69596002)(110136002)(68736007)(50226002)(42186005)(92566002)(53416004)(48376002)(4001430100002)(2950100001)(50466002)(33646002)(76176999)(5009440100003)(81156014)(8676002)(81166006)(586003)(47776003)(105586002)(2351001)(5003940100001)(229853001)(106356001)(4326007)(6116002)(3846002)(66066001)(36756003)(77096005)(50986999)(2906002)(7099028); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1716; H:localhost.localdomain.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0701MB1716; 23:Z0KggpidauB4jp/VL1bHcgC4NEOEBHMbVw6y27k?= =?us-ascii?Q?S5JcJCgJaoBzIgiUaaHG5bnASL9dWpM75i3gYa2xZ5pn272/j7E35trO1gfQ?= =?us-ascii?Q?8SvZ9vn8/Zrfbfyda1f6nOhlzUpUAJCkVtou98giEj4GQ8GDSiD3VVl1wRTh?= =?us-ascii?Q?pEzTMTQ0gssuUiJNP9BIULnQmU27qGZWdhbn5gS87D+stCJ4qhilHmfZ1Dtt?= =?us-ascii?Q?QjLEqYTGWiTWGpVIPdQ4uuQlqJKgnkB8u8bhNJGbiIrRz7QGpPdtSLu9R+jo?= =?us-ascii?Q?GlwjNzpGjCN0NTVZh6kvvx9ojNm5n9aiqj+lITVamR/l8xtXTSfIDBJtox+M?= =?us-ascii?Q?LY+J6MlNgBMRciqyLeXN7JPmatWrEUIvHrYyH23Xg77dGsy/CongmDK2Yzta?= =?us-ascii?Q?0V3smgjH+aaHOfX3j5JYv7iUvlq1eGLu8iIFdcO6HKfYmYF+xfiXG40eqFG7?= =?us-ascii?Q?T96+QJEVa9MGItmreNVcal1PQ5pfW6BKSKPAx0ypz0nYKrwtZMchbqMGCpVR?= =?us-ascii?Q?aj3YLJx6aDocli178jnClyNdFGrxpQBTwkEbyjdt9Ied2NEKdFSz3MjvII7o?= =?us-ascii?Q?8fb5o95IprlWNxODXq3YxML9T1tUZQie/IuRjdDdc350CR+YqP3CkxvmcwNP?= =?us-ascii?Q?7qVyZYxwnegLUZ+UxboZNibLGgdlkeE/9xOiVdeqZHqT5gw2BncV/b1+xKkI?= =?us-ascii?Q?dohGs5Y6pgwChc/L1XINZYwO91DdMlkj4ROnvW0WhmdTsnAUhFPCEDpT+RxS?= =?us-ascii?Q?fuwlKqai0aRgEt0y9IC5GZxFoApCTypjVvcqdpQfSFx8s934YK1Kj++VlcJm?= =?us-ascii?Q?Yc9NEaWktRLLXwjWQw7aM8AvskvzTFqA6jqaYF4S3Mbh0HHAa2zoO7FMXWrp?= =?us-ascii?Q?DX2K3/MGMJw6icnDLWNLecCKDhtI/i7GKoERw6iW8Z/qxKal4COo56YwNkt9?= =?us-ascii?Q?f6y0hRgfjE72PYQGbmoCdHRZ+1xH+82r+1y0sCpElH4RGbx9XmI+tOs2N5T9?= =?us-ascii?Q?40ViWeBQPSDYzpWpd2RjZZj8YRS0w6HzVXYEF8kUT6wP9ySO843Xz98N7Y/9?= =?us-ascii?Q?O4vdprgR7Eccw3PBXOHJ/7nCYHXEbdH6KWns2otpnp8JXgtFSu6n2PIGurtS?= =?us-ascii?Q?LUwc+xdT2o7WDGY6v6CXJwLMk6EaxMhGgdNe6em4Nja1tf8RKsaAvQuq0WHg?= =?us-ascii?Q?Wsuf12HUmR09BzZCYx2fPIi9jLpj9vfQw69dLul73dqbRf+i6I+XaZURiUg?= =?us-ascii?Q?=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1716; 6:G5z7cLJryoHVEjWZRXLPBGxI0iCL+B4mNigBHSZ2/r21DU3Ajaly8njGnBA58BAOOv5zUsYBP2WDFkFbgp/XBPomlJ58iilKN+AmddN2o1CojMztFDRQ308/yvR355tf0X1Yp0+GyGjq6YinpK8w/CS9JJXb5IShwiMLc2AjxLXCn0A6napjd5VHDAdGXvf+b9oalDoSdm6xdsEz9T+hm3StqjcG3peyHZq+A2RUHQXYw/yNDUle2nYKSIEv4V3kMOlB9pc/uDL2NhRGKInEGYkgyskT8zPpAVGs6y4KPSw=; 5:WaTXm3CqHKBfN+s4LsJCBIO0SAa54FeJLrilav+ZoBORYqUbKvdUEIiOMdDqrNr0hC+bBdGm6OuStkyPfmmwlkiPHwbfqMgZuA/PncWR24hKcJHKBIb2o/AKK+NTzF6DbySGsGkUFCzxwf0DzuNIqQ==; 24:qnIBGpsFpQUqQWBPmSPWxJ8zyrMoWFZOXxC10ZqhRoiGRD3htDWWs6oFnm9MFaZPVPM4+Uk+FvuQhXkLgmAqo17HKtx4Ib2cmnGajJkWerk=; 7:hSk2ZW2PswaFYL8nPnbRv6nMselwHL2gJvBYv4U6oxd4DQAUmduquiWvk12fPuvu+Qh3ptzopvOPnfLOhMT5UhcWunfSM6EgfzjCMzNXft2HMJfXqY90N0tHepsfBgaaKjMArHXFmLcDK3m050SFU5m3az8lPzNgUdkc8Nj3wBNf9IahZU1rzsUAu80b2FRCCwzdokPWTNo+aiCGXEtvsuKkeI5PTvxk3nfyIMZ/zG++WqjiP0S2o+zahmCEVlpA SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2016 12:50:29.3698 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1716 Subject: [dpdk-dev] [PATCH v3 4/4] virtio: add neon support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jul 2016 12:50:34 -0000 Added neon based Rx vector implementation. Selection of the new handler based neon availability at runtime. Updated the release notes and MAINTAINERS file. Signed-off-by: Jerin Jacob --- MAINTAINERS | 1 + doc/guides/rel_notes/release_16_07.rst | 2 + drivers/net/virtio/Makefile | 2 + drivers/net/virtio/virtio_rxtx.c | 3 + drivers/net/virtio/virtio_rxtx_simple_neon.c | 235 +++++++++++++++++++++++++++ 5 files changed, 243 insertions(+) create mode 100644 drivers/net/virtio/virtio_rxtx_simple_neon.c diff --git a/MAINTAINERS b/MAINTAINERS index a59191e..ab04cee 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -143,6 +143,7 @@ F: lib/librte_acl/acl_run_neon.* F: lib/librte_lpm/rte_lpm_neon.h F: lib/librte_hash/rte*_arm64.h F: drivers/net/ixgbe/ixgbe_rxtx_vec_neon.c +F: drivers/net/virtio/virtio_rxtx_simple_neon.c EZchip TILE-Gx M: Zhigang Lu diff --git a/doc/guides/rel_notes/release_16_07.rst b/doc/guides/rel_notes/release_16_07.rst index 569f562..57f3d28 100644 --- a/doc/guides/rel_notes/release_16_07.rst +++ b/doc/guides/rel_notes/release_16_07.rst @@ -180,6 +180,8 @@ New Features section of the "Network Interface Controller Drivers" document. +* **Virtio NEON support for ARM.** + Resolved Issues --------------- diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile index c4103b7..97972a6 100644 --- a/drivers/net/virtio/Makefile +++ b/drivers/net/virtio/Makefile @@ -54,6 +54,8 @@ SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple.c ifeq ($(CONFIG_RTE_ARCH_X86),y) SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple_sse.c +else ifneq ($(filter y,$(CONFIG_RTE_ARCH_ARM) $(CONFIG_RTE_ARCH_ARM64)),) +SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple_neon.c endif ifeq ($(CONFIG_RTE_VIRTIO_USER),y) diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c index adc3457..8f6cad8 100644 --- a/drivers/net/virtio/virtio_rxtx.c +++ b/drivers/net/virtio/virtio_rxtx.c @@ -481,6 +481,9 @@ virtio_update_rxtx_handler(struct rte_eth_dev *dev, #if defined RTE_ARCH_X86 if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_SSE3)) use_simple_rxtx = 1; +#elif defined RTE_ARCH_ARM64 || defined CONFIG_RTE_ARCH_ARM + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_NEON)) + use_simple_rxtx = 1; #endif /* Use simple rx/tx func if single segment and no offloads */ if (use_simple_rxtx && diff --git a/drivers/net/virtio/virtio_rxtx_simple_neon.c b/drivers/net/virtio/virtio_rxtx_simple_neon.c new file mode 100644 index 0000000..793eefb --- /dev/null +++ b/drivers/net/virtio/virtio_rxtx_simple_neon.c @@ -0,0 +1,235 @@ +/* + * BSD LICENSE + * + * Copyright (C) Cavium networks Ltd. 2016 + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "virtio_rxtx_simple.h" + +#define RTE_VIRTIO_VPMD_RX_BURST 32 +#define RTE_VIRTIO_DESC_PER_LOOP 8 +#define RTE_VIRTIO_VPMD_RX_REARM_THRESH RTE_VIRTIO_VPMD_RX_BURST + +/* virtio vPMD receive routine, only accept(nb_pkts >= RTE_VIRTIO_DESC_PER_LOOP) + * + * This routine is for non-mergeable RX, one desc for each guest buffer. + * This routine is based on the RX ring layout optimization. Each entry in the + * avail ring points to the desc with the same index in the desc ring and this + * will never be changed in the driver. + * + * - nb_pkts < RTE_VIRTIO_DESC_PER_LOOP, just return no packet + */ +uint16_t +virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct virtnet_rx *rxvq = rx_queue; + struct virtqueue *vq = rxvq->vq; + uint16_t nb_used; + uint16_t desc_idx; + struct vring_used_elem *rused; + struct rte_mbuf **sw_ring; + struct rte_mbuf **sw_ring_end; + uint16_t nb_pkts_received; + + uint8x16_t shuf_msk1 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 4, 5, 0xFF, 0xFF, /* pkt len */ + 4, 5, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + uint8x16_t shuf_msk2 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 12, 13, 0xFF, 0xFF, /* pkt len */ + 12, 13, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Subtract the header length. + * In which case do we need the header length in used->len ? + */ + uint16x8_t len_adjust = { + 0, 0, + (uint16_t)vq->hw->vtnet_hdr_size, 0, + (uint16_t)vq->hw->vtnet_hdr_size, + 0, + 0, 0 + }; + + if (unlikely(nb_pkts < RTE_VIRTIO_DESC_PER_LOOP)) + return 0; + + nb_used = VIRTQUEUE_NUSED(vq); + + rte_rmb(); + + if (unlikely(nb_used == 0)) + return 0; + + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_VIRTIO_DESC_PER_LOOP); + nb_used = RTE_MIN(nb_used, nb_pkts); + + desc_idx = (uint16_t)(vq->vq_used_cons_idx & (vq->vq_nentries - 1)); + rused = &vq->vq_ring.used->ring[desc_idx]; + sw_ring = &vq->sw_ring[desc_idx]; + sw_ring_end = &vq->sw_ring[vq->vq_nentries]; + + rte_prefetch_non_temporal(rused); + + if (vq->vq_free_cnt >= RTE_VIRTIO_VPMD_RX_REARM_THRESH) { + virtio_rxq_rearm_vec(rxvq); + if (unlikely(virtqueue_kick_prepare(vq))) + virtqueue_notify(vq); + } + + for (nb_pkts_received = 0; + nb_pkts_received < nb_used;) { + uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP]; + + mbp[0] = vld1q_u64((uint64_t *)(sw_ring + 0)); + desc[0] = vld1q_u64((uint64_t *)(rused + 0)); + vst1q_u64((uint64_t *)&rx_pkts[0], mbp[0]); + + mbp[1] = vld1q_u64((uint64_t *)(sw_ring + 2)); + desc[1] = vld1q_u64((uint64_t *)(rused + 2)); + vst1q_u64((uint64_t *)&rx_pkts[2], mbp[1]); + + mbp[2] = vld1q_u64((uint64_t *)(sw_ring + 4)); + desc[2] = vld1q_u64((uint64_t *)(rused + 4)); + vst1q_u64((uint64_t *)&rx_pkts[4], mbp[2]); + + mbp[3] = vld1q_u64((uint64_t *)(sw_ring + 6)); + desc[3] = vld1q_u64((uint64_t *)(rused + 6)); + vst1q_u64((uint64_t *)&rx_pkts[6], mbp[3]); + + pkt_mb[1] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk2)); + pkt_mb[0] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk1)); + pkt_mb[1] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[1]), len_adjust)); + pkt_mb[0] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[0]), len_adjust)); + vst1q_u64((void *)&rx_pkts[1]->rx_descriptor_fields1, + pkt_mb[1]); + vst1q_u64((void *)&rx_pkts[0]->rx_descriptor_fields1, + pkt_mb[0]); + + pkt_mb[3] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk2)); + pkt_mb[2] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk1)); + pkt_mb[3] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[3]), len_adjust)); + pkt_mb[2] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[2]), len_adjust)); + vst1q_u64((void *)&rx_pkts[3]->rx_descriptor_fields1, + pkt_mb[3]); + vst1q_u64((void *)&rx_pkts[2]->rx_descriptor_fields1, + pkt_mb[2]); + + pkt_mb[5] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk2)); + pkt_mb[4] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk1)); + pkt_mb[5] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[5]), len_adjust)); + pkt_mb[4] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[4]), len_adjust)); + vst1q_u64((void *)&rx_pkts[5]->rx_descriptor_fields1, + pkt_mb[5]); + vst1q_u64((void *)&rx_pkts[4]->rx_descriptor_fields1, + pkt_mb[4]); + + pkt_mb[7] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk2)); + pkt_mb[6] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk1)); + pkt_mb[7] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[7]), len_adjust)); + pkt_mb[6] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[6]), len_adjust)); + vst1q_u64((void *)&rx_pkts[7]->rx_descriptor_fields1, + pkt_mb[7]); + vst1q_u64((void *)&rx_pkts[6]->rx_descriptor_fields1, + pkt_mb[6]); + + if (unlikely(nb_used <= RTE_VIRTIO_DESC_PER_LOOP)) { + if (sw_ring + nb_used <= sw_ring_end) + nb_pkts_received += nb_used; + else + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + if (unlikely(sw_ring + RTE_VIRTIO_DESC_PER_LOOP >= + sw_ring_end)) { + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + nb_pkts_received += RTE_VIRTIO_DESC_PER_LOOP; + + rx_pkts += RTE_VIRTIO_DESC_PER_LOOP; + sw_ring += RTE_VIRTIO_DESC_PER_LOOP; + rused += RTE_VIRTIO_DESC_PER_LOOP; + nb_used -= RTE_VIRTIO_DESC_PER_LOOP; + } + } + } + + vq->vq_used_cons_idx += nb_pkts_received; + vq->vq_free_cnt += nb_pkts_received; + rxvq->stats.packets += nb_pkts_received; + return nb_pkts_received; +} -- 2.5.5