From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0084.outbound.protection.outlook.com [157.56.110.84]) by dpdk.org (Postfix) with ESMTP id 882C05955 for ; Mon, 27 Jun 2016 13:55:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=9ypEGhIbXZ3ryPqSvPMNWdCb3mZf/Lu9ivWg09MIr5A=; b=TekQuvfIPPxIJp9s+3deWdeiaVXKZkc1iheu+qUB8GxVVxjSx9L8NLQlvvnHnXLJRfnLJiFiCXzyHayvnHOvhgEffAiD/eVvIFBKdTZV3JnryMkTaY0yb+sF4rKgFt2qa4z6W37DF8P6a1ovnw9S5DGgMdUP3NGLif2/Krd/OY4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain.localdomain (171.48.12.117) by BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19) with Microsoft SMTP Server (TLS) id 15.1.528.16; Mon, 27 Jun 2016 11:55:05 +0000 From: Jerin Jacob To: CC: , , , , , Jerin Jacob Date: Mon, 27 Jun 2016 17:24:08 +0530 Message-ID: <1467028448-8914-5-git-send-email-jerin.jacob@caviumnetworks.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1467028448-8914-1-git-send-email-jerin.jacob@caviumnetworks.com> References: <1467028448-8914-1-git-send-email-jerin.jacob@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [171.48.12.117] X-ClientProxiedBy: MA1PR01CA0014.INDPRD01.PROD.OUTLOOK.COM (10.164.117.21) To BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19) X-MS-Office365-Filtering-Correlation-Id: a1dd4c16-ab1c-4f76-bcfa-08d39e81e2a6 X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 2:lCI+uKCAunNLdO2QkYuOHvK/eQSd7tw99O7osRKes+0Tm1Ij5gxgTwyqhlFsgNpIdc9sSmJm4lLGRi44w8AwaSlqo0lnygkoq7HzAE5WXag5w0zdoQV2uuNdvtQJL2Wpp4s/mVVaxe18ibdeY1u21To2EkvKupSGY6Q00ZUVBAauC764+YHpD+VUVxwVP2Nw; 3:GjfriBtr9v8jHxrxrowqa1IsEeo7DwaaEWOOfXoRi9ImpOeBW0LKbro/sXuxzdhYyc7O3UROAdcNTmi0clJbMX89H+Spa8Izy2Pl3Ro+oc53F8VaOUaSB+1UxnbkkgQe; 25:5s1GVFb2z/ZmsD5DLni3I5IubbR/3yg3IDINA7lM8SNRPh9IOtfnfNGtvQD2WYMhEqAlPBxbKBOFhww+V2DM7yRkYT8tShbSANtsxcQVucC4SI5AdHYZJ3SpCe3VPrWzz9arylugOT6EF1k82dbQqtrmIXLHOvVq6VM8WzVxq/IEPlCQFDAaYHKqnXm2+Fe+hVykrCkamxooFiI9dBkChoQX17WM7LWieKLaf4PDlRLTTUjPJoHHf5Qws3e5+lb0tt9dIMtJD6CCyd6N9ehNIP2PiEKNIaYLn49uwS+H0YNtc6/lX52M1Dkg6BK6pxzXSqvYec1Dx6EvNuvC48xuNwJwlKaAPLhpb/pJ0jLcP0ufAG4zZRrhIQ3ifIOI91K2nHxaGzPkr49hbb8xOZpUeGmVyy6d72ZnV6POLNxnHx0= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0701MB1720; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 31:BrxEIWS5FOg+87Xbp3fbBi6eDjVsvfY9JhRW4lOcUIFojn0exkCc2Reo72A3jxW/jwsekDyE9uwj46FYNBW3cq6vkQiZyO4uenZKCq1oDPB9AMiawzjmps3uRTkCsGbLXxVuJJnNMowScM4QXBR6Mst2eWNWiAjJj3/ZoW+bp/XxuYQCkWWv3nS2dBfVelLwhWe5pCGXhD3VNN1de7PLsQ==; 20:2wjzzG8ogC2Nr2di2k45QfJWy8xRCha+CRQ5FyFwgxQUzRJWtK8p4TsVrcA2/78Udk/SZxlb3DswsfTclXT0FqYGd74jxmRg9PSZ3uUzS4Rm8vimV5lwyr71o3ShDt2DYbwNHxCGtLzl3xRvYLZRFtl2VZYpFii3SVsgD8BkzjJANwU3suQTwKH/iCNuu6vTLdJ5/LG0NaqqFo55GtY49aQFC63UBVOxSqB4TSsY48nHU2ikGUUqhFqBKigE4V9FSVs7xyzAyaHmNbd7X723SB/yCngwk71yJE65AvTGTDuqW9BOvMGubOE19eNNmhNCDL7oP2X4U5chSW8QDb0uWtn7OOq2wvk/uxjBdy+lMm9Cp97bAxvoDveSmYfWmphC/p+xnz2K31yNlHg5xRbx/DPULHcU8dhzKobVjIZ6YsNZMRoRz6EyQMXXhmqMJ1HAuS38eqXA5jw5mfcFs+ysffQnRe52ga+LvS5XDfXpt0GDEQM5T7WVluYzTEFGCFxEUvA76+/kTS9W0DwZardmMY8EkEep4zdv91r4x+tK6QNQD7EkROSDpi47XUKF+jlLm0EMl2FM8c4x8jBF1kXLL611sZd8q/zaoZxvDdM8vX8= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BN3PR0701MB1720; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1720; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 4:DG8L8K7hGWmo3QG3kQlUGfbPzaCYmilblUuGzDLsDXLnLoXshoQIrg7yk7qr2MLFwiRHqix9uQ8F5wEBRO6HYNYRqG5W+s5qTNzttBJt+BcEvTwmNRxqC7LhQ+6DKX+GjSHEqz84fkdnVbUNKvQ3pB3TdvoEK8B0Hxv2962bYkQHl9pE2flYfiqFbyG1QXjqqNTzhTYmocnnGI/nm/QjLmE16MjX3e4sUwlgn43h1uzPySmTU989UlU0JhpS5KVu3IGPOE1kytNlzqHXC4VA8H2UgVY5RKo06i23nGQxWVdKNKqnd5lLJxGd789M6ElHv45lk9mJCrP8oitoSCPt8Z/veVEEugUMuCYcU8zIr8z1oXtwKQHb2++AqubLvyIW X-Forefront-PRVS: 09860C2161 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(7916002)(189002)(199003)(57704003)(50226002)(4001430100002)(97736004)(105586002)(50466002)(50986999)(76176999)(575784001)(586003)(3846002)(48376002)(42186005)(106356001)(101416001)(107886002)(5003940100001)(110136002)(2950100001)(19580395003)(19580405001)(305945005)(77096005)(36756003)(92566002)(2906002)(81166006)(47776003)(68736007)(4326007)(7846002)(66066001)(229853001)(8676002)(2351001)(33646002)(6116002)(81156014)(7736002)(189998001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1720; H:localhost.localdomain.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1720; 23:2AvtXzHNe6rvrvC9ILwOkbHvsUOah0+L+jzkWzW?= =?us-ascii?Q?0/xkKOUo9suqbapQdvs1uUyh6or+sH43HSRWy8cQqTDRofx2bwsCmHGrzFTW?= =?us-ascii?Q?lhp0XxD8hSpCCrpIbaQM1at/lD9ayjUMKhgnBvNTPZXmdRBeb+0pJEf5TlBp?= =?us-ascii?Q?hSPtjhuPeKYwPK9mwriHLqQruhjYJVT7BsOaO28tNzs6OpIqsshOnFj3Bomm?= =?us-ascii?Q?QQs4M1zzmagh7e4d5rtIHBEIGAkY2dOGkQuUgDfCvsQbSV7P6uZyMCfEjwhx?= =?us-ascii?Q?YIWSocyT7AdA7a4CYWwGtwyFhY8IKQtQN2v0CDCl+xwDuBfR8VJl09WvDJ7d?= =?us-ascii?Q?Sgmn4fcfldAFfejtI4dLelj/PBMEwxaY3nzGWoVVgjv86/CeI1veNjKRopIg?= =?us-ascii?Q?a33z8XdqELIwmGyI4Bea3fIdWyv6Ld0MksIpsWHl+nI3G6m+GgYeEJZ8XA5k?= =?us-ascii?Q?fGjOdTncaku8sy878+R+T0IXk5GeW/1VepECOoukd1m/2OKQN8UhX9ftmtHr?= =?us-ascii?Q?xYeCgrNsEzPVg4VZf6jrsBziBJZpWFbva/s1Cd2N9Nqu3UiJe6SG+TTB9f4P?= =?us-ascii?Q?Pic9AHqQwDsJ2RE0B2Mpy1nKCrnITYI8/t1NFf5TaU2EA8ubFvoAzbeEIXrT?= =?us-ascii?Q?cZDtYyWf6g8fcWKd3nN2gXQ6BdvqI1uB0pfMHeM/9UYVglN0ZMMo8cuJu3Rn?= =?us-ascii?Q?HmOeulxsG9zYBnZYESi7sCWRGf182jt0gP24JDe7dIn57716hu9/zCvFvbUh?= =?us-ascii?Q?zwheSU+UOGregHUy7ZxGccBACYk6PLSjLReM3Y8e4s7qpr0Iyhe7lwqqMql5?= =?us-ascii?Q?oZovDsaeC7h7aOa+PPEa1BeDBDeIt6XwdOgPuMPM7OS2zmIWhILX22ajuXRc?= =?us-ascii?Q?xKdki2HRP+OONKTeYKKWgjvU0hGKvy2/M5zJhpF6g1sMuYgR5ucviaDR0zxT?= =?us-ascii?Q?zJj69j1NTUiQdp9MAKdICv8hV9gqPDHQHHpugOrnEAPQkrMcFkbvVsyLtqOJ?= =?us-ascii?Q?OlNHm9ItA3Ozj7bQy+KsPsu5LKvxqAOa8lKUkXVmO4dZZvVQjbA8bylEQbKn?= =?us-ascii?Q?vVG0gYurYGoVXjKDrXUQjY5VP9/P4Rdidow5anSY7c1TkZWIZFNBdQdyd0ws?= =?us-ascii?Q?eG2zlIsFLWgFXsUsIASj375TZ1RSpMdOeZwUjA6K4lX4IHR//fGcxVg=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720; 6:/iGmQTh64rTVkZ5c681X+yc5JDINh39GH76eOGkU0F4Fph4YkT9ibhu02mA7WrzOz5TmVDLZDxQuWnTJLqcy5EXs/COegWrhd5wFZF3a4gvBQvIQtlvQVPcjuHcwYdJr/bDn+HlNZsZVFhSOvdPidFQztUMwwlByfVRZzFv/WOnlHwezUe7uAqfuUgzTO8AoYwyg4upwDP9TvQY0bkrIOOCfRZ29YLWuraVxAdzOfBautpIwn66nBtVfOjmt8SDpI1d9Z7Y7Zbl+iDNzxz+/uwdtp06QAf/a6xeODlz4LAI=; 5:aCUWvTk/+Zpl8Ka9+JBWFOad+X4+lne2+Ud2q5746ZE/bp2Z9u3a17NFnaBNu9k47ntm0lEfy2rCIHUahtZLAcp5PYEkm3VNq8mL5J5kHV5REoMRUK60G3+dnRWZKQsm07GKiKkNn+WoTKfC9RtIHA==; 24:1wIMS0gVrDhLjwGpuRplVIOwksMT6MuQhdmEebws2HJEmyiaNE0WIjtReyehvPHOrieFRLw4I1C1/HDcD0L9gS1JpWDn7fYz84pMejgBdDo=; 7:j+RGuUMPILazgIFLOgBG7kbqxECeBitmKBGWSktorcu1I9t73scJTw+bPSBE9DaJ4sI5XHDRIf9b0BrPKnkjbsvvVlt9BQX5xzTfXsB2eQpy0YTKYrjaJFc72H0ffgyLH/pkdabZ3CW+DkyVlJjLdtoNJQiKNUBkT0ljrl6/CYFtQZonTZWjIV3dcYu472CalHg1gq79UATLfgLIfA2AW8Yv/+B4HUVg1AQPM/gqlihkivfXYshOaxRfshb2y3g2 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2016 11:55:05.3678 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1720 Subject: [dpdk-dev] [PATCH 4/4] virtio: add neon support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jun 2016 11:55:10 -0000 Added neon based Rx vector implementation for virtio. Selected neon based virtio implementation for ARM64 as default and updated the MAINTAINERS file. Signed-off-by: Jerin Jacob --- MAINTAINERS | 1 + config/defconfig_arm64-armv8a-linuxapp-gcc | 1 - doc/guides/rel_notes/release_16_07.rst | 2 + drivers/net/virtio/virtio_rxtx.c | 2 +- drivers/net/virtio/virtio_rxtx_simple.c | 2 + drivers/net/virtio/virtio_rxtx_simple_neon.h | 238 +++++++++++++++++++++++++++ 6 files changed, 244 insertions(+), 2 deletions(-) create mode 100644 drivers/net/virtio/virtio_rxtx_simple_neon.h diff --git a/MAINTAINERS b/MAINTAINERS index f6c0d3d..2bb12aa 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -142,6 +142,7 @@ F: lib/librte_eal/common/include/arch/arm/*_64.h F: lib/librte_acl/acl_run_neon.* F: lib/librte_lpm/rte_lpm_neon.h F: lib/librte_hash/rte*_arm64.h +F: drivers/net/virtio/virtio_rxtx_simple_neon.h EZchip TILE-Gx M: Zhigang Lu diff --git a/config/defconfig_arm64-armv8a-linuxapp-gcc b/config/defconfig_arm64-armv8a-linuxapp-gcc index 95ed30e..a786562 100644 --- a/config/defconfig_arm64-armv8a-linuxapp-gcc +++ b/config/defconfig_arm64-armv8a-linuxapp-gcc @@ -48,6 +48,5 @@ CONFIG_RTE_IXGBE_INC_VECTOR=n CONFIG_RTE_LIBRTE_IVSHMEM=n CONFIG_RTE_LIBRTE_FM10K_PMD=n CONFIG_RTE_LIBRTE_I40E_PMD=n -CONFIG_RTE_LIBRTE_VIRTIO_INC_VECTOR=n CONFIG_RTE_SCHED_VECTOR=n diff --git a/doc/guides/rel_notes/release_16_07.rst b/doc/guides/rel_notes/release_16_07.rst index 2694f50..3187a33 100644 --- a/doc/guides/rel_notes/release_16_07.rst +++ b/doc/guides/rel_notes/release_16_07.rst @@ -118,6 +118,8 @@ New Features * Root privilege is a must for sorting hugepages by physical address. * Can only be used with vhost user backend. +* **Virtio NEON support for ARM.** + Resolved Issues --------------- diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c index e9b42f3..ca25db3 100644 --- a/drivers/net/virtio/virtio_rxtx.c +++ b/drivers/net/virtio/virtio_rxtx.c @@ -500,7 +500,7 @@ virtio_dev_tx_queue_setup(struct rte_eth_dev *dev, } #ifdef RTE_LIBRTE_VIRTIO_INC_VECTOR -#ifdef RTE_MACHINE_CPUFLAG_SSSE3 +#if defined RTE_MACHINE_CPUFLAG_SSSE3 || defined RTE_MACHINE_CPUFLAG_NEON struct virtio_hw *hw = dev->data->dev_private; /* Use simple rx/tx func if single segment and no offloads */ if ((tx_conf->txq_flags & VIRTIO_SIMPLE_FLAGS) == VIRTIO_SIMPLE_FLAGS && diff --git a/drivers/net/virtio/virtio_rxtx_simple.c b/drivers/net/virtio/virtio_rxtx_simple.c index ca87605..e5dc010 100644 --- a/drivers/net/virtio/virtio_rxtx_simple.c +++ b/drivers/net/virtio/virtio_rxtx_simple.c @@ -131,6 +131,8 @@ virtio_rxq_rearm_vec(struct virtnet_rx *rxvq) #ifdef RTE_MACHINE_CPUFLAG_SSSE3 #include "virtio_rxtx_simple_sse.h" +#elif RTE_MACHINE_CPUFLAG_NEON +#include "virtio_rxtx_simple_neon.h" #endif #define VIRTIO_TX_FREE_THRESH 32 diff --git a/drivers/net/virtio/virtio_rxtx_simple_neon.h b/drivers/net/virtio/virtio_rxtx_simple_neon.h new file mode 100644 index 0000000..41f347d --- /dev/null +++ b/drivers/net/virtio/virtio_rxtx_simple_neon.h @@ -0,0 +1,238 @@ +/* + * BSD LICENSE + * + * Copyright (C) Cavium networks Ltd. 2016 + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Cavium networks nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +*/ + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "virtio_logs.h" +#include "virtio_ethdev.h" +#include "virtqueue.h" +#include "virtio_rxtx.h" + +#define RTE_VIRTIO_VPMD_RX_BURST 32 +#define RTE_VIRTIO_DESC_PER_LOOP 8 +#define RTE_VIRTIO_VPMD_RX_REARM_THRESH RTE_VIRTIO_VPMD_RX_BURST + +/* virtio vPMD receive routine, only accept(nb_pkts >= RTE_VIRTIO_DESC_PER_LOOP) + * + * This routine is for non-mergeable RX, one desc for each guest buffer. + * This routine is based on the RX ring layout optimization. Each entry in the + * avail ring points to the desc with the same index in the desc ring and this + * will never be changed in the driver. + * + * - nb_pkts < RTE_VIRTIO_DESC_PER_LOOP, just return no packet + */ +uint16_t +virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts, + uint16_t nb_pkts) +{ + struct virtnet_rx *rxvq = rx_queue; + struct virtqueue *vq = rxvq->vq; + uint16_t nb_used; + uint16_t desc_idx; + struct vring_used_elem *rused; + struct rte_mbuf **sw_ring; + struct rte_mbuf **sw_ring_end; + uint16_t nb_pkts_received; + + uint8x16_t shuf_msk1 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 4, 5, 0xFF, 0xFF, /* pkt len */ + 4, 5, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + uint8x16_t shuf_msk2 = { + 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */ + 12, 13, 0xFF, 0xFF, /* pkt len */ + 12, 13, /* dat len */ + 0xFF, 0xFF, /* vlan tci */ + 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Subtract the header length. + * In which case do we need the header length in used->len ? + */ + uint16x8_t len_adjust = { + 0, 0, + (uint16_t)vq->hw->vtnet_hdr_size, 0, + (uint16_t)vq->hw->vtnet_hdr_size, + 0, + 0, 0 + }; + + if (unlikely(nb_pkts < RTE_VIRTIO_DESC_PER_LOOP)) + return 0; + + nb_used = VIRTQUEUE_NUSED(vq); + + rte_rmb(); + + if (unlikely(nb_used == 0)) + return 0; + + nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_VIRTIO_DESC_PER_LOOP); + nb_used = RTE_MIN(nb_used, nb_pkts); + + desc_idx = (uint16_t)(vq->vq_used_cons_idx & (vq->vq_nentries - 1)); + rused = &vq->vq_ring.used->ring[desc_idx]; + sw_ring = &vq->sw_ring[desc_idx]; + sw_ring_end = &vq->sw_ring[vq->vq_nentries]; + + rte_prefetch_non_temporal(rused); + + if (vq->vq_free_cnt >= RTE_VIRTIO_VPMD_RX_REARM_THRESH) { + virtio_rxq_rearm_vec(rxvq); + if (unlikely(virtqueue_kick_prepare(vq))) + virtqueue_notify(vq); + } + + for (nb_pkts_received = 0; + nb_pkts_received < nb_used;) { + uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP / 2]; + uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP]; + + mbp[0] = vld1q_u64((uint64_t *)(sw_ring + 0)); + desc[0] = vld1q_u64((uint64_t *)(rused + 0)); + vst1q_u64((uint64_t *)&rx_pkts[0], mbp[0]); + + mbp[1] = vld1q_u64((uint64_t *)(sw_ring + 2)); + desc[1] = vld1q_u64((uint64_t *)(rused + 2)); + vst1q_u64((uint64_t *)&rx_pkts[2], mbp[1]); + + mbp[2] = vld1q_u64((uint64_t *)(sw_ring + 4)); + desc[2] = vld1q_u64((uint64_t *)(rused + 4)); + vst1q_u64((uint64_t *)&rx_pkts[4], mbp[2]); + + mbp[3] = vld1q_u64((uint64_t *)(sw_ring + 6)); + desc[3] = vld1q_u64((uint64_t *)(rused + 6)); + vst1q_u64((uint64_t *)&rx_pkts[6], mbp[3]); + + pkt_mb[1] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk2)); + pkt_mb[0] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[0]), shuf_msk1)); + pkt_mb[1] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[1]), len_adjust)); + pkt_mb[0] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[0]), len_adjust)); + vst1q_u64((void *)&rx_pkts[1]->rx_descriptor_fields1, + pkt_mb[1]); + vst1q_u64((void *)&rx_pkts[0]->rx_descriptor_fields1, + pkt_mb[0]); + + pkt_mb[3] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk2)); + pkt_mb[2] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[1]), shuf_msk1)); + pkt_mb[3] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[3]), len_adjust)); + pkt_mb[2] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[2]), len_adjust)); + vst1q_u64((void *)&rx_pkts[3]->rx_descriptor_fields1, + pkt_mb[3]); + vst1q_u64((void *)&rx_pkts[2]->rx_descriptor_fields1, + pkt_mb[2]); + + pkt_mb[5] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk2)); + pkt_mb[4] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[2]), shuf_msk1)); + pkt_mb[5] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[5]), len_adjust)); + pkt_mb[4] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[4]), len_adjust)); + vst1q_u64((void *)&rx_pkts[5]->rx_descriptor_fields1, + pkt_mb[5]); + vst1q_u64((void *)&rx_pkts[4]->rx_descriptor_fields1, + pkt_mb[4]); + + pkt_mb[7] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk2)); + pkt_mb[6] = vreinterpretq_u64_u8(vqtbl1q_u8( + vreinterpretq_u8_u64(desc[3]), shuf_msk1)); + pkt_mb[7] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[7]), len_adjust)); + pkt_mb[6] = vreinterpretq_u64_u16(vsubq_u16( + vreinterpretq_u16_u64(pkt_mb[6]), len_adjust)); + vst1q_u64((void *)&rx_pkts[7]->rx_descriptor_fields1, + pkt_mb[7]); + vst1q_u64((void *)&rx_pkts[6]->rx_descriptor_fields1, + pkt_mb[6]); + + if (unlikely(nb_used <= RTE_VIRTIO_DESC_PER_LOOP)) { + if (sw_ring + nb_used <= sw_ring_end) + nb_pkts_received += nb_used; + else + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + if (unlikely(sw_ring + RTE_VIRTIO_DESC_PER_LOOP >= + sw_ring_end)) { + nb_pkts_received += sw_ring_end - sw_ring; + break; + } else { + nb_pkts_received += RTE_VIRTIO_DESC_PER_LOOP; + + rx_pkts += RTE_VIRTIO_DESC_PER_LOOP; + sw_ring += RTE_VIRTIO_DESC_PER_LOOP; + rused += RTE_VIRTIO_DESC_PER_LOOP; + nb_used -= RTE_VIRTIO_DESC_PER_LOOP; + } + } + } + + vq->vq_used_cons_idx += nb_pkts_received; + vq->vq_free_cnt += nb_pkts_received; + rxvq->stats.packets += nb_pkts_received; + return nb_pkts_received; +} -- 2.5.5