From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (xvm-189-124.dc0.ghst.net [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2824EA09FF; Tue, 5 Jan 2021 15:17:01 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E622A1607C1; Tue, 5 Jan 2021 15:17:00 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mails.dpdk.org (Postfix) with ESMTP id EC588160737 for ; Tue, 5 Jan 2021 15:16:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1609856218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8xX09gIoR5KB/MjZZRkVCnW2UVJ5bU8AJb/pu7FqdIs=; b=ZnmKrIsYNtWQNSDOmo/S1Gzt1mXoiA8gIgsdYiTQ35KdQrPyVlWZ7Msf6LdHLcaN+P1xJV 15VeOotP02RzMyfvxtqPSiBlv6J2HH3xy/iMTsNplhV1n+XhH5W35wPnZRzXUtuYrNDaup G4Tq8BR9M0S1drCcvR44L8itgOjJX/I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-441-VXwVEFHXNIqnkKPuFlBIEA-1; Tue, 05 Jan 2021 09:16:54 -0500 X-MC-Unique: VXwVEFHXNIqnkKPuFlBIEA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1FF5E15726; Tue, 5 Jan 2021 14:16:52 +0000 (UTC) Received: from [10.36.110.9] (unknown [10.36.110.9]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DCF5D60BE5; Tue, 5 Jan 2021 14:16:49 +0000 (UTC) To: Joyce Kong , chenbo.xia@intel.com, jerinj@marvell.com, ruifeng.wang@arm.com, honnappa.nagarahalli@arm.com Cc: dev@dpdk.org, nd@arm.com References: <20200911120906.45995-1-joyce.kong@arm.com> <20201117100635.27690-1-joyce.kong@arm.com> <20201117100635.27690-3-joyce.kong@arm.com> From: Maxime Coquelin Message-ID: Date: Tue, 5 Jan 2021 15:16:47 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20201117100635.27690-3-joyce.kong@arm.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v1 2/4] net/virtio: add vectorized packed ring Rx NEON path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 11/17/20 11:06 AM, Joyce Kong wrote: > Optimize packed ring Rx batch path with NEON instructions. > > Signed-off-by: Joyce Kong > Reviewed-by: Ruifeng Wang > --- > drivers/net/virtio/virtio_rxtx_packed.h | 15 ++ > drivers/net/virtio/virtio_rxtx_packed_neon.h | 150 +++++++++++++++++++ > 2 files changed, 165 insertions(+) > create mode 100644 drivers/net/virtio/virtio_rxtx_packed_neon.h > > diff --git a/drivers/net/virtio/virtio_rxtx_packed.h b/drivers/net/virtio/virtio_rxtx_packed.h > index b0b1d63ec..8f5198ad7 100644 > --- a/drivers/net/virtio/virtio_rxtx_packed.h > +++ b/drivers/net/virtio/virtio_rxtx_packed.h > @@ -19,9 +19,16 @@ > #include "virtqueue.h" > > #define BYTE_SIZE 8 > + > +#ifdef CC_AVX512_SUPPORT > /* flag bits offset in packed ring desc higher 64bits */ > #define FLAGS_BITS_OFFSET ((offsetof(struct vring_packed_desc, flags) - \ > offsetof(struct vring_packed_desc, len)) * BYTE_SIZE) > +#elif defined(RTE_ARCH_ARM) > +/* flag bits offset in packed ring desc from ID */ > +#define FLAGS_BITS_OFFSET ((offsetof(struct vring_packed_desc, flags) - \ > + offsetof(struct vring_packed_desc, id)) * BYTE_SIZE) > +#endif > > #define PACKED_FLAGS_MASK ((0ULL | VRING_PACKED_DESC_F_AVAIL_USED) << \ > FLAGS_BITS_OFFSET) > @@ -44,8 +51,16 @@ > /* net hdr short size mask */ > #define NET_HDR_MASK 0x3F > > +#ifdef RTE_ARCH_ARM > +/* The cache line size on different Arm platforms are different, so > + * put a four batch size here to match with the minimum cache line > + * size and accommodate NEON register size. > + */ > +#define PACKED_BATCH_SIZE 4 > +#else > #define PACKED_BATCH_SIZE (RTE_CACHE_LINE_SIZE / \ > sizeof(struct vring_packed_desc)) > +#endif > #define PACKED_BATCH_MASK (PACKED_BATCH_SIZE - 1) > > #ifdef VIRTIO_GCC_UNROLL_PRAGMA > diff --git a/drivers/net/virtio/virtio_rxtx_packed_neon.h b/drivers/net/virtio/virtio_rxtx_packed_neon.h > new file mode 100644 > index 000000000..fb1e49909 > --- /dev/null > +++ b/drivers/net/virtio/virtio_rxtx_packed_neon.h > @@ -0,0 +1,150 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(c) 2020 Arm Corporation > + */ > + > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > + > +#include "virtio_ethdev.h" > +#include "virtio_pci.h" > +#include "virtio_rxtx_packed.h" > +#include "virtqueue.h" > + > +static inline uint16_t > +virtqueue_dequeue_batch_packed_vec(struct virtnet_rx *rxvq, > + struct rte_mbuf **rx_pkts) > +{ > + struct virtqueue *vq = rxvq->vq; > + struct virtio_hw *hw = vq->hw; > + uint16_t head_size = hw->vtnet_hdr_size; > + uint16_t id = vq->vq_used_cons_idx; > + struct vring_packed_desc *p_desc; > + uint16_t i; > + > + if (id & PACKED_BATCH_MASK) > + return -1; > + > + if (unlikely((id + PACKED_BATCH_SIZE) > vq->vq_nentries)) > + return -1; This function returns an unsigned short, I think you should return 0 here since it failed to dequeue packets. > + /* Map packed descriptor to mbuf fields. */ > + uint8x16_t shuf_msk1 = { > + 0xFF, 0xFF, 0xFF, 0xFF, /* pkt_type set as unknown */ > + 0, 1, /* octet 1~0, low 16 bits pkt_len */ > + 0xFF, 0xFF, /* skip high 16 bits of pkt_len, zero out */ > + 0, 1, /* octet 1~0, 16 bits data_len */ > + 0xFF, 0xFF, /* vlan tci set as unknown */ > + 0xFF, 0xFF, 0xFF, 0xFF > + }; > + > + uint8x16_t shuf_msk2 = { > + 0xFF, 0xFF, 0xFF, 0xFF, /* pkt_type set as unknown */ > + 8, 9, /* octet 9~8, low 16 bits pkt_len */ > + 0xFF, 0xFF, /* skip high 16 bits of pkt_len, zero out */ > + 8, 9, /* octet 9~8, 16 bits data_len */ > + 0xFF, 0xFF, /* vlan tci set as unknown */ > + 0xFF, 0xFF, 0xFF, 0xFF > + }; > + > + /* Subtract the header length. */ > + uint16x8_t len_adjust = { > + 0, 0, /* ignore pkt_type field */ > + head_size, /* sub head_size on pkt_len */ > + 0, /* ignore high 16 bits of pkt_len */ > + head_size, /* sub head_size on data_len */ > + 0, 0, 0 /* ignore non-length fields */ > + }; > + > + uint64x2_t desc[PACKED_BATCH_SIZE / 2]; > + uint64x2x2_t mbp[PACKED_BATCH_SIZE / 2]; > + uint64x2_t pkt_mb[PACKED_BATCH_SIZE]; > + > + p_desc = &vq->vq_packed.ring.desc[id]; > + /* Load high 64 bits of packed descriptor 0,1. */ > + desc[0] = vld2q_u64((uint64_t *)(p_desc)).val[1]; > + /* Load high 64 bits of packed descriptor 2,3. */ > + desc[1] = vld2q_u64((uint64_t *)(p_desc + 2)).val[1]; > + > + /* Only care avail/used bits. */ > + uint32x4_t v_mask = vdupq_n_u32(PACKED_FLAGS_MASK); > + /* Extract high 32 bits of packed descriptor (id, flags). */ > + uint32x4_t v_desc = vuzp2q_u32(vreinterpretq_u32_u64(desc[0]), > + vreinterpretq_u32_u64(desc[1])); > + uint32x4_t v_flag = vandq_u32(v_desc, v_mask); > + > + uint32x4_t v_used_flag = vdupq_n_u32(0); > + if (vq->vq_packed.used_wrap_counter) > + v_used_flag = vdupq_n_u32(PACKED_FLAGS_MASK); > + > + poly128_t desc_stats = vreinterpretq_p128_u32(~vceqq_u32(v_flag, v_used_flag)); > + > + /* Check all descs are used. */ > + if (desc_stats) > + return -1; Same here. You should return 0 here as the queue is full. > + > + /* Load 2 mbuf pointers per time. */ > + mbp[0] = vld2q_u64((uint64_t *)&vq->vq_descx[id]); > + vst1q_u64((uint64_t *)&rx_pkts[0], mbp[0].val[0]); > + > + mbp[1] = vld2q_u64((uint64_t *)&vq->vq_descx[id + 2]); > + vst1q_u64((uint64_t *)&rx_pkts[2], mbp[1].val[0]); > + > + /** > + * Update data length and packet length for descriptor. > + * structure of pkt_mb: > + * -------------------------------------------------------------------- > + * |32 bits pkt_type|32 bits pkt_len|16 bits data_len|16 bits vlan_tci| > + * -------------------------------------------------------------------- > + */ > + pkt_mb[0] = vreinterpretq_u64_u8(vqtbl1q_u8( > + vreinterpretq_u8_u64(desc[0]), shuf_msk1)); > + pkt_mb[1] = vreinterpretq_u64_u8(vqtbl1q_u8( > + vreinterpretq_u8_u64(desc[0]), shuf_msk2)); > + pkt_mb[2] = vreinterpretq_u64_u8(vqtbl1q_u8( > + vreinterpretq_u8_u64(desc[1]), shuf_msk1))' > + pkt_mb[3] = vreinterpretq_u64_u8(vqtbl1q_u8( > + vreinterpretq_u8_u64(desc[1]), shuf_msk2)); > + > + pkt_mb[0] = vreinterpretq_u64_u16(vsubq_u16( > + vreinterpretq_u16_u64(pkt_mb[0]), len_adjust)); > + pkt_mb[1] = vreinterpretq_u64_u16(vsubq_u16( > + vreinterpretq_u16_u64(pkt_mb[1]), len_adjust)); > + pkt_mb[2] = vreinterpretq_u64_u16(vsubq_u16( > + vreinterpretq_u16_u64(pkt_mb[2]), len_adjust)); > + pkt_mb[3] = vreinterpretq_u64_u16(vsubq_u16( > + vreinterpretq_u16_u64(pkt_mb[3]), len_adjust)); > + > + vst1q_u64((void *)&rx_pkts[0]->rx_descriptor_fields1, pkt_mb[0]); > + vst1q_u64((void *)&rx_pkts[1]->rx_descriptor_fields1, pkt_mb[1]); > + vst1q_u64((void *)&rx_pkts[2]->rx_descriptor_fields1, pkt_mb[2]); > + vst1q_u64((void *)&rx_pkts[3]->rx_descriptor_fields1, pkt_mb[3]); > + > + if (hw->has_rx_offload) { > + virtio_for_each_try_unroll(i, 0, PACKED_BATCH_SIZE) { > + char *addr = (char *)rx_pkts[i]->buf_addr + > + RTE_PKTMBUF_HEADROOM - head_size; > + virtio_vec_rx_offload(rx_pkts[i], > + (struct virtio_net_hdr *)addr); > + } > + } > + > + virtio_update_batch_stats(&rxvq->stats, rx_pkts[0]->pkt_len, > + rx_pkts[1]->pkt_len, rx_pkts[2]->pkt_len, > + rx_pkts[3]->pkt_len); > + > + vq->vq_free_cnt += PACKED_BATCH_SIZE; > + > + vq->vq_used_cons_idx += PACKED_BATCH_SIZE; > + if (vq->vq_used_cons_idx >= vq->vq_nentries) { > + vq->vq_used_cons_idx -= vq->vq_nentries; > + vq->vq_packed.used_wrap_counter ^= 1; > + } > + > + return 0; > +} >