From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <Jerin.Jacob@cavium.com>
Received: from na01-bn1-obe.outbound.protection.outlook.com
(mail-bn1on0084.outbound.protection.outlook.com [157.56.110.84])
by dpdk.org (Postfix) with ESMTP id 882C05955
for <dev@dpdk.org>; Mon, 27 Jun 2016 13:55:09 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
bh=9ypEGhIbXZ3ryPqSvPMNWdCb3mZf/Lu9ivWg09MIr5A=;
b=TekQuvfIPPxIJp9s+3deWdeiaVXKZkc1iheu+qUB8GxVVxjSx9L8NLQlvvnHnXLJRfnLJiFiCXzyHayvnHOvhgEffAiD/eVvIFBKdTZV3JnryMkTaY0yb+sF4rKgFt2qa4z6W37DF8P6a1ovnw9S5DGgMdUP3NGLif2/Krd/OY4=
Authentication-Results: spf=none (sender IP is )
smtp.mailfrom=Jerin.Jacob@cavium.com;
Received: from localhost.localdomain.localdomain (171.48.12.117) by
BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19) with Microsoft SMTP
Server (TLS) id 15.1.528.16; Mon, 27 Jun 2016 11:55:05 +0000
From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
To: <dev@dpdk.org>
CC: <thomas.monjalon@6wind.com>, <bruce.richardson@intel.com>,
<jianbo.liu@linaro.org>, <huawei.xie@intel.com>,
<yuanhan.liu@linux.intel.com>, Jerin Jacob <jerin.jacob@caviumnetworks.com>
Date: Mon, 27 Jun 2016 17:24:08 +0530
Message-ID: <1467028448-8914-5-git-send-email-jerin.jacob@caviumnetworks.com>
X-Mailer: git-send-email 2.5.5
In-Reply-To: <1467028448-8914-1-git-send-email-jerin.jacob@caviumnetworks.com>
References: <1467028448-8914-1-git-send-email-jerin.jacob@caviumnetworks.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [171.48.12.117]
X-ClientProxiedBy: MA1PR01CA0014.INDPRD01.PROD.OUTLOOK.COM (10.164.117.21) To
BN3PR0701MB1720.namprd07.prod.outlook.com (10.163.39.19)
X-MS-Office365-Filtering-Correlation-Id: a1dd4c16-ab1c-4f76-bcfa-08d39e81e2a6
X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720;
2:lCI+uKCAunNLdO2QkYuOHvK/eQSd7tw99O7osRKes+0Tm1Ij5gxgTwyqhlFsgNpIdc9sSmJm4lLGRi44w8AwaSlqo0lnygkoq7HzAE5WXag5w0zdoQV2uuNdvtQJL2Wpp4s/mVVaxe18ibdeY1u21To2EkvKupSGY6Q00ZUVBAauC764+YHpD+VUVxwVP2Nw;
3:GjfriBtr9v8jHxrxrowqa1IsEeo7DwaaEWOOfXoRi9ImpOeBW0LKbro/sXuxzdhYyc7O3UROAdcNTmi0clJbMX89H+Spa8Izy2Pl3Ro+oc53F8VaOUaSB+1UxnbkkgQe;
25:5s1GVFb2z/ZmsD5DLni3I5IubbR/3yg3IDINA7lM8SNRPh9IOtfnfNGtvQD2WYMhEqAlPBxbKBOFhww+V2DM7yRkYT8tShbSANtsxcQVucC4SI5AdHYZJ3SpCe3VPrWzz9arylugOT6EF1k82dbQqtrmIXLHOvVq6VM8WzVxq/IEPlCQFDAaYHKqnXm2+Fe+hVykrCkamxooFiI9dBkChoQX17WM7LWieKLaf4PDlRLTTUjPJoHHf5Qws3e5+lb0tt9dIMtJD6CCyd6N9ehNIP2PiEKNIaYLn49uwS+H0YNtc6/lX52M1Dkg6BK6pxzXSqvYec1Dx6EvNuvC48xuNwJwlKaAPLhpb/pJ0jLcP0ufAG4zZRrhIQ3ifIOI91K2nHxaGzPkr49hbb8xOZpUeGmVyy6d72ZnV6POLNxnHx0=
X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0701MB1720;
X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720;
31:BrxEIWS5FOg+87Xbp3fbBi6eDjVsvfY9JhRW4lOcUIFojn0exkCc2Reo72A3jxW/jwsekDyE9uwj46FYNBW3cq6vkQiZyO4uenZKCq1oDPB9AMiawzjmps3uRTkCsGbLXxVuJJnNMowScM4QXBR6Mst2eWNWiAjJj3/ZoW+bp/XxuYQCkWWv3nS2dBfVelLwhWe5pCGXhD3VNN1de7PLsQ==;
20:2wjzzG8ogC2Nr2di2k45QfJWy8xRCha+CRQ5FyFwgxQUzRJWtK8p4TsVrcA2/78Udk/SZxlb3DswsfTclXT0FqYGd74jxmRg9PSZ3uUzS4Rm8vimV5lwyr71o3ShDt2DYbwNHxCGtLzl3xRvYLZRFtl2VZYpFii3SVsgD8BkzjJANwU3suQTwKH/iCNuu6vTLdJ5/LG0NaqqFo55GtY49aQFC63UBVOxSqB4TSsY48nHU2ikGUUqhFqBKigE4V9FSVs7xyzAyaHmNbd7X723SB/yCngwk71yJE65AvTGTDuqW9BOvMGubOE19eNNmhNCDL7oP2X4U5chSW8QDb0uWtn7OOq2wvk/uxjBdy+lMm9Cp97bAxvoDveSmYfWmphC/p+xnz2K31yNlHg5xRbx/DPULHcU8dhzKobVjIZ6YsNZMRoRz6EyQMXXhmqMJ1HAuS38eqXA5jw5mfcFs+ysffQnRe52ga+LvS5XDfXpt0GDEQM5T7WVluYzTEFGCFxEUvA76+/kTS9W0DwZardmMY8EkEep4zdv91r4x+tK6QNQD7EkROSDpi47XUKF+jlLm0EMl2FM8c4x8jBF1kXLL611sZd8q/zaoZxvDdM8vX8=
X-Microsoft-Antispam-PRVS: <BN3PR0701MB1720A7BC0D58F4CB8A3295E381210@BN3PR0701MB1720.namprd07.prod.outlook.com>
X-Exchange-Antispam-Report-Test: UriScan:;
X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0;
RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046);
SRVR:BN3PR0701MB1720; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0701MB1720;
X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720;
4:DG8L8K7hGWmo3QG3kQlUGfbPzaCYmilblUuGzDLsDXLnLoXshoQIrg7yk7qr2MLFwiRHqix9uQ8F5wEBRO6HYNYRqG5W+s5qTNzttBJt+BcEvTwmNRxqC7LhQ+6DKX+GjSHEqz84fkdnVbUNKvQ3pB3TdvoEK8B0Hxv2962bYkQHl9pE2flYfiqFbyG1QXjqqNTzhTYmocnnGI/nm/QjLmE16MjX3e4sUwlgn43h1uzPySmTU989UlU0JhpS5KVu3IGPOE1kytNlzqHXC4VA8H2UgVY5RKo06i23nGQxWVdKNKqnd5lLJxGd789M6ElHv45lk9mJCrP8oitoSCPt8Z/veVEEugUMuCYcU8zIr8z1oXtwKQHb2++AqubLvyIW
X-Forefront-PRVS: 09860C2161
X-Forefront-Antispam-Report: SFV:NSPM;
SFS:(10009020)(4630300001)(6069001)(6009001)(7916002)(189002)(199003)(57704003)(50226002)(4001430100002)(97736004)(105586002)(50466002)(50986999)(76176999)(575784001)(586003)(3846002)(48376002)(42186005)(106356001)(101416001)(107886002)(5003940100001)(110136002)(2950100001)(19580395003)(19580405001)(305945005)(77096005)(36756003)(92566002)(2906002)(81166006)(47776003)(68736007)(4326007)(7846002)(66066001)(229853001)(8676002)(2351001)(33646002)(6116002)(81156014)(7736002)(189998001);
DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR0701MB1720;
H:localhost.localdomain.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1;
A:1; LANG:en;
Received-SPF: None (protection.outlook.com: cavium.com does not designate
permitted sender hosts)
X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0701MB1720;
23:2AvtXzHNe6rvrvC9ILwOkbHvsUOah0+L+jzkWzW?=
=?us-ascii?Q?0/xkKOUo9suqbapQdvs1uUyh6or+sH43HSRWy8cQqTDRofx2bwsCmHGrzFTW?=
=?us-ascii?Q?lhp0XxD8hSpCCrpIbaQM1at/lD9ayjUMKhgnBvNTPZXmdRBeb+0pJEf5TlBp?=
=?us-ascii?Q?hSPtjhuPeKYwPK9mwriHLqQruhjYJVT7BsOaO28tNzs6OpIqsshOnFj3Bomm?=
=?us-ascii?Q?QQs4M1zzmagh7e4d5rtIHBEIGAkY2dOGkQuUgDfCvsQbSV7P6uZyMCfEjwhx?=
=?us-ascii?Q?YIWSocyT7AdA7a4CYWwGtwyFhY8IKQtQN2v0CDCl+xwDuBfR8VJl09WvDJ7d?=
=?us-ascii?Q?Sgmn4fcfldAFfejtI4dLelj/PBMEwxaY3nzGWoVVgjv86/CeI1veNjKRopIg?=
=?us-ascii?Q?a33z8XdqELIwmGyI4Bea3fIdWyv6Ld0MksIpsWHl+nI3G6m+GgYeEJZ8XA5k?=
=?us-ascii?Q?fGjOdTncaku8sy878+R+T0IXk5GeW/1VepECOoukd1m/2OKQN8UhX9ftmtHr?=
=?us-ascii?Q?xYeCgrNsEzPVg4VZf6jrsBziBJZpWFbva/s1Cd2N9Nqu3UiJe6SG+TTB9f4P?=
=?us-ascii?Q?Pic9AHqQwDsJ2RE0B2Mpy1nKCrnITYI8/t1NFf5TaU2EA8ubFvoAzbeEIXrT?=
=?us-ascii?Q?cZDtYyWf6g8fcWKd3nN2gXQ6BdvqI1uB0pfMHeM/9UYVglN0ZMMo8cuJu3Rn?=
=?us-ascii?Q?HmOeulxsG9zYBnZYESi7sCWRGf182jt0gP24JDe7dIn57716hu9/zCvFvbUh?=
=?us-ascii?Q?zwheSU+UOGregHUy7ZxGccBACYk6PLSjLReM3Y8e4s7qpr0Iyhe7lwqqMql5?=
=?us-ascii?Q?oZovDsaeC7h7aOa+PPEa1BeDBDeIt6XwdOgPuMPM7OS2zmIWhILX22ajuXRc?=
=?us-ascii?Q?xKdki2HRP+OONKTeYKKWgjvU0hGKvy2/M5zJhpF6g1sMuYgR5ucviaDR0zxT?=
=?us-ascii?Q?zJj69j1NTUiQdp9MAKdICv8hV9gqPDHQHHpugOrnEAPQkrMcFkbvVsyLtqOJ?=
=?us-ascii?Q?OlNHm9ItA3Ozj7bQy+KsPsu5LKvxqAOa8lKUkXVmO4dZZvVQjbA8bylEQbKn?=
=?us-ascii?Q?vVG0gYurYGoVXjKDrXUQjY5VP9/P4Rdidow5anSY7c1TkZWIZFNBdQdyd0ws?=
=?us-ascii?Q?eG2zlIsFLWgFXsUsIASj375TZ1RSpMdOeZwUjA6K4lX4IHR//fGcxVg=3D?=
=?us-ascii?Q?=3D?=
X-Microsoft-Exchange-Diagnostics: 1; BN3PR0701MB1720;
6:/iGmQTh64rTVkZ5c681X+yc5JDINh39GH76eOGkU0F4Fph4YkT9ibhu02mA7WrzOz5TmVDLZDxQuWnTJLqcy5EXs/COegWrhd5wFZF3a4gvBQvIQtlvQVPcjuHcwYdJr/bDn+HlNZsZVFhSOvdPidFQztUMwwlByfVRZzFv/WOnlHwezUe7uAqfuUgzTO8AoYwyg4upwDP9TvQY0bkrIOOCfRZ29YLWuraVxAdzOfBautpIwn66nBtVfOjmt8SDpI1d9Z7Y7Zbl+iDNzxz+/uwdtp06QAf/a6xeODlz4LAI=;
5:aCUWvTk/+Zpl8Ka9+JBWFOad+X4+lne2+Ud2q5746ZE/bp2Z9u3a17NFnaBNu9k47ntm0lEfy2rCIHUahtZLAcp5PYEkm3VNq8mL5J5kHV5REoMRUK60G3+dnRWZKQsm07GKiKkNn+WoTKfC9RtIHA==;
24:1wIMS0gVrDhLjwGpuRplVIOwksMT6MuQhdmEebws2HJEmyiaNE0WIjtReyehvPHOrieFRLw4I1C1/HDcD0L9gS1JpWDn7fYz84pMejgBdDo=;
7:j+RGuUMPILazgIFLOgBG7kbqxECeBitmKBGWSktorcu1I9t73scJTw+bPSBE9DaJ4sI5XHDRIf9b0BrPKnkjbsvvVlt9BQX5xzTfXsB2eQpy0YTKYrjaJFc72H0ffgyLH/pkdabZ3CW+DkyVlJjLdtoNJQiKNUBkT0ljrl6/CYFtQZonTZWjIV3dcYu472CalHg1gq79UATLfgLIfA2AW8Yv/+B4HUVg1AQPM/gqlihkivfXYshOaxRfshb2y3g2
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-OriginatorOrg: caviumnetworks.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2016 11:55:05.3678 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0701MB1720
Subject: [dpdk-dev] [PATCH 4/4] virtio: add neon support
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 27 Jun 2016 11:55:10 -0000
Added neon based Rx vector implementation for virtio.
Selected neon based virtio implementation for ARM64 as
default and updated the MAINTAINERS file.
Signed-off-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
MAINTAINERS | 1 +
config/defconfig_arm64-armv8a-linuxapp-gcc | 1 -
doc/guides/rel_notes/release_16_07.rst | 2 +
drivers/net/virtio/virtio_rxtx.c | 2 +-
drivers/net/virtio/virtio_rxtx_simple.c | 2 +
drivers/net/virtio/virtio_rxtx_simple_neon.h | 238 +++++++++++++++++++++++++++
6 files changed, 244 insertions(+), 2 deletions(-)
create mode 100644 drivers/net/virtio/virtio_rxtx_simple_neon.h
diff --git a/MAINTAINERS b/MAINTAINERS
index f6c0d3d..2bb12aa 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -142,6 +142,7 @@ F: lib/librte_eal/common/include/arch/arm/*_64.h
F: lib/librte_acl/acl_run_neon.*
F: lib/librte_lpm/rte_lpm_neon.h
F: lib/librte_hash/rte*_arm64.h
+F: drivers/net/virtio/virtio_rxtx_simple_neon.h
EZchip TILE-Gx
M: Zhigang Lu <zlu@ezchip.com>
diff --git a/config/defconfig_arm64-armv8a-linuxapp-gcc b/config/defconfig_arm64-armv8a-linuxapp-gcc
index 95ed30e..a786562 100644
--- a/config/defconfig_arm64-armv8a-linuxapp-gcc
+++ b/config/defconfig_arm64-armv8a-linuxapp-gcc
@@ -48,6 +48,5 @@ CONFIG_RTE_IXGBE_INC_VECTOR=n
CONFIG_RTE_LIBRTE_IVSHMEM=n
CONFIG_RTE_LIBRTE_FM10K_PMD=n
CONFIG_RTE_LIBRTE_I40E_PMD=n
-CONFIG_RTE_LIBRTE_VIRTIO_INC_VECTOR=n
CONFIG_RTE_SCHED_VECTOR=n
diff --git a/doc/guides/rel_notes/release_16_07.rst b/doc/guides/rel_notes/release_16_07.rst
index 2694f50..3187a33 100644
--- a/doc/guides/rel_notes/release_16_07.rst
+++ b/doc/guides/rel_notes/release_16_07.rst
@@ -118,6 +118,8 @@ New Features
* Root privilege is a must for sorting hugepages by physical address.
* Can only be used with vhost user backend.
+* **Virtio NEON support for ARM.**
+
Resolved Issues
---------------
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index e9b42f3..ca25db3 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -500,7 +500,7 @@ virtio_dev_tx_queue_setup(struct rte_eth_dev *dev,
}
#ifdef RTE_LIBRTE_VIRTIO_INC_VECTOR
-#ifdef RTE_MACHINE_CPUFLAG_SSSE3
+#if defined RTE_MACHINE_CPUFLAG_SSSE3 || defined RTE_MACHINE_CPUFLAG_NEON
struct virtio_hw *hw = dev->data->dev_private;
/* Use simple rx/tx func if single segment and no offloads */
if ((tx_conf->txq_flags & VIRTIO_SIMPLE_FLAGS) == VIRTIO_SIMPLE_FLAGS &&
diff --git a/drivers/net/virtio/virtio_rxtx_simple.c b/drivers/net/virtio/virtio_rxtx_simple.c
index ca87605..e5dc010 100644
--- a/drivers/net/virtio/virtio_rxtx_simple.c
+++ b/drivers/net/virtio/virtio_rxtx_simple.c
@@ -131,6 +131,8 @@ virtio_rxq_rearm_vec(struct virtnet_rx *rxvq)
#ifdef RTE_MACHINE_CPUFLAG_SSSE3
#include "virtio_rxtx_simple_sse.h"
+#elif RTE_MACHINE_CPUFLAG_NEON
+#include "virtio_rxtx_simple_neon.h"
#endif
#define VIRTIO_TX_FREE_THRESH 32
diff --git a/drivers/net/virtio/virtio_rxtx_simple_neon.h b/drivers/net/virtio/virtio_rxtx_simple_neon.h
new file mode 100644
index 0000000..41f347d
--- /dev/null
+++ b/drivers/net/virtio/virtio_rxtx_simple_neon.h
@@ -0,0 +1,238 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright (C) Cavium networks Ltd. 2016
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Cavium networks nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*/
+
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+#include <rte_byteorder.h>
+#include <rte_branch_prediction.h>
+#include <rte_cycles.h>
+#include <rte_ether.h>
+#include <rte_ethdev.h>
+#include <rte_errno.h>
+#include <rte_memory.h>
+#include <rte_memzone.h>
+#include <rte_mempool.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_prefetch.h>
+#include <rte_string_fns.h>
+#include <rte_vect.h>
+
+#include "virtio_logs.h"
+#include "virtio_ethdev.h"
+#include "virtqueue.h"
+#include "virtio_rxtx.h"
+
+#define RTE_VIRTIO_VPMD_RX_BURST 32
+#define RTE_VIRTIO_DESC_PER_LOOP 8
+#define RTE_VIRTIO_VPMD_RX_REARM_THRESH RTE_VIRTIO_VPMD_RX_BURST
+
+/* virtio vPMD receive routine, only accept(nb_pkts >= RTE_VIRTIO_DESC_PER_LOOP)
+ *
+ * This routine is for non-mergeable RX, one desc for each guest buffer.
+ * This routine is based on the RX ring layout optimization. Each entry in the
+ * avail ring points to the desc with the same index in the desc ring and this
+ * will never be changed in the driver.
+ *
+ * - nb_pkts < RTE_VIRTIO_DESC_PER_LOOP, just return no packet
+ */
+uint16_t
+virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkts,
+ uint16_t nb_pkts)
+{
+ struct virtnet_rx *rxvq = rx_queue;
+ struct virtqueue *vq = rxvq->vq;
+ uint16_t nb_used;
+ uint16_t desc_idx;
+ struct vring_used_elem *rused;
+ struct rte_mbuf **sw_ring;
+ struct rte_mbuf **sw_ring_end;
+ uint16_t nb_pkts_received;
+
+ uint8x16_t shuf_msk1 = {
+ 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */
+ 4, 5, 0xFF, 0xFF, /* pkt len */
+ 4, 5, /* dat len */
+ 0xFF, 0xFF, /* vlan tci */
+ 0xFF, 0xFF, 0xFF, 0xFF
+ };
+
+ uint8x16_t shuf_msk2 = {
+ 0xFF, 0xFF, 0xFF, 0xFF, /* packet type */
+ 12, 13, 0xFF, 0xFF, /* pkt len */
+ 12, 13, /* dat len */
+ 0xFF, 0xFF, /* vlan tci */
+ 0xFF, 0xFF, 0xFF, 0xFF
+ };
+
+ /* Subtract the header length.
+ * In which case do we need the header length in used->len ?
+ */
+ uint16x8_t len_adjust = {
+ 0, 0,
+ (uint16_t)vq->hw->vtnet_hdr_size, 0,
+ (uint16_t)vq->hw->vtnet_hdr_size,
+ 0,
+ 0, 0
+ };
+
+ if (unlikely(nb_pkts < RTE_VIRTIO_DESC_PER_LOOP))
+ return 0;
+
+ nb_used = VIRTQUEUE_NUSED(vq);
+
+ rte_rmb();
+
+ if (unlikely(nb_used == 0))
+ return 0;
+
+ nb_pkts = RTE_ALIGN_FLOOR(nb_pkts, RTE_VIRTIO_DESC_PER_LOOP);
+ nb_used = RTE_MIN(nb_used, nb_pkts);
+
+ desc_idx = (uint16_t)(vq->vq_used_cons_idx & (vq->vq_nentries - 1));
+ rused = &vq->vq_ring.used->ring[desc_idx];
+ sw_ring = &vq->sw_ring[desc_idx];
+ sw_ring_end = &vq->sw_ring[vq->vq_nentries];
+
+ rte_prefetch_non_temporal(rused);
+
+ if (vq->vq_free_cnt >= RTE_VIRTIO_VPMD_RX_REARM_THRESH) {
+ virtio_rxq_rearm_vec(rxvq);
+ if (unlikely(virtqueue_kick_prepare(vq)))
+ virtqueue_notify(vq);
+ }
+
+ for (nb_pkts_received = 0;
+ nb_pkts_received < nb_used;) {
+ uint64x2_t desc[RTE_VIRTIO_DESC_PER_LOOP / 2];
+ uint64x2_t mbp[RTE_VIRTIO_DESC_PER_LOOP / 2];
+ uint64x2_t pkt_mb[RTE_VIRTIO_DESC_PER_LOOP];
+
+ mbp[0] = vld1q_u64((uint64_t *)(sw_ring + 0));
+ desc[0] = vld1q_u64((uint64_t *)(rused + 0));
+ vst1q_u64((uint64_t *)&rx_pkts[0], mbp[0]);
+
+ mbp[1] = vld1q_u64((uint64_t *)(sw_ring + 2));
+ desc[1] = vld1q_u64((uint64_t *)(rused + 2));
+ vst1q_u64((uint64_t *)&rx_pkts[2], mbp[1]);
+
+ mbp[2] = vld1q_u64((uint64_t *)(sw_ring + 4));
+ desc[2] = vld1q_u64((uint64_t *)(rused + 4));
+ vst1q_u64((uint64_t *)&rx_pkts[4], mbp[2]);
+
+ mbp[3] = vld1q_u64((uint64_t *)(sw_ring + 6));
+ desc[3] = vld1q_u64((uint64_t *)(rused + 6));
+ vst1q_u64((uint64_t *)&rx_pkts[6], mbp[3]);
+
+ pkt_mb[1] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[0]), shuf_msk2));
+ pkt_mb[0] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[0]), shuf_msk1));
+ pkt_mb[1] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[1]), len_adjust));
+ pkt_mb[0] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[0]), len_adjust));
+ vst1q_u64((void *)&rx_pkts[1]->rx_descriptor_fields1,
+ pkt_mb[1]);
+ vst1q_u64((void *)&rx_pkts[0]->rx_descriptor_fields1,
+ pkt_mb[0]);
+
+ pkt_mb[3] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[1]), shuf_msk2));
+ pkt_mb[2] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[1]), shuf_msk1));
+ pkt_mb[3] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[3]), len_adjust));
+ pkt_mb[2] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[2]), len_adjust));
+ vst1q_u64((void *)&rx_pkts[3]->rx_descriptor_fields1,
+ pkt_mb[3]);
+ vst1q_u64((void *)&rx_pkts[2]->rx_descriptor_fields1,
+ pkt_mb[2]);
+
+ pkt_mb[5] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[2]), shuf_msk2));
+ pkt_mb[4] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[2]), shuf_msk1));
+ pkt_mb[5] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[5]), len_adjust));
+ pkt_mb[4] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[4]), len_adjust));
+ vst1q_u64((void *)&rx_pkts[5]->rx_descriptor_fields1,
+ pkt_mb[5]);
+ vst1q_u64((void *)&rx_pkts[4]->rx_descriptor_fields1,
+ pkt_mb[4]);
+
+ pkt_mb[7] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[3]), shuf_msk2));
+ pkt_mb[6] = vreinterpretq_u64_u8(vqtbl1q_u8(
+ vreinterpretq_u8_u64(desc[3]), shuf_msk1));
+ pkt_mb[7] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[7]), len_adjust));
+ pkt_mb[6] = vreinterpretq_u64_u16(vsubq_u16(
+ vreinterpretq_u16_u64(pkt_mb[6]), len_adjust));
+ vst1q_u64((void *)&rx_pkts[7]->rx_descriptor_fields1,
+ pkt_mb[7]);
+ vst1q_u64((void *)&rx_pkts[6]->rx_descriptor_fields1,
+ pkt_mb[6]);
+
+ if (unlikely(nb_used <= RTE_VIRTIO_DESC_PER_LOOP)) {
+ if (sw_ring + nb_used <= sw_ring_end)
+ nb_pkts_received += nb_used;
+ else
+ nb_pkts_received += sw_ring_end - sw_ring;
+ break;
+ } else {
+ if (unlikely(sw_ring + RTE_VIRTIO_DESC_PER_LOOP >=
+ sw_ring_end)) {
+ nb_pkts_received += sw_ring_end - sw_ring;
+ break;
+ } else {
+ nb_pkts_received += RTE_VIRTIO_DESC_PER_LOOP;
+
+ rx_pkts += RTE_VIRTIO_DESC_PER_LOOP;
+ sw_ring += RTE_VIRTIO_DESC_PER_LOOP;
+ rused += RTE_VIRTIO_DESC_PER_LOOP;
+ nb_used -= RTE_VIRTIO_DESC_PER_LOOP;
+ }
+ }
+ }
+
+ vq->vq_used_cons_idx += nb_pkts_received;
+ vq->vq_free_cnt += nb_pkts_received;
+ rxvq->stats.packets += nb_pkts_received;
+ return nb_pkts_received;
+}
--
2.5.5