From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5BC61A0C43 for ; Thu, 30 Sep 2021 10:13:44 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 45098410F1; Thu, 30 Sep 2021 10:13:44 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id DDBF54067E for ; Thu, 30 Sep 2021 10:13:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632989622; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sIj1G70Rd0lpkqP1/uQJTfb5XozOGmEUSLOUutrQfqw=; b=dl/LBuN6pPQ4X/ocHx/Ac93Uao8PkglLve6oVHLcYYD3ubxv/XGt/JsZNsgS60Y5GMzEil 2Z4iFk6kO/dk8dRB5LGxCDOzGxjhdFEB9qj0ow0/UIASN0af5L710utmk/eO8dx1Z/7FaW O8XBhsY074jLT3KlbfPdnBf2rmOv1+8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-6-thUT_RKmNEep8iePRQX3FQ-1; Thu, 30 Sep 2021 04:13:39 -0400 X-MC-Unique: thUT_RKmNEep8iePRQX3FQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 151431006AA3; Thu, 30 Sep 2021 08:13:38 +0000 (UTC) Received: from max-t490s.redhat.com (unknown [10.39.208.6]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E0CA5F70B; Thu, 30 Sep 2021 08:13:02 +0000 (UTC) From: Maxime Coquelin To: olivier.matz@6wind.com, david.marchand@redhat.com, chenbo.xia@intel.com, dev@dpdk.org Cc: Maxime Coquelin , stable@dpdk.org Date: Thu, 30 Sep 2021 10:12:59 +0200 Message-Id: <20210930081259.232706-1-maxime.coquelin@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" Subject: [dpdk-stable] [PATCH v2] net/virtio: revert forcing IOVA as VA mode for virtio-user X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" This patch removes the simplification in Virtio descriptors handling, where their buffer addresses are IOVAs for Virtio PCI devices, and VA-only for Virtio-user devices, which added a requirement on Virtio-user that it only supported IOVA as VA. This change introduced a regression for applications using Virtio-user and other physical PMDs that require IOVA as PA because they don't use an IOMMU. This patch reverts to the old behaviour, but needed to be reworked because of the refactoring that happened in v21.02. Fixes: 17043a2909bb ("net/virtio: force IOVA as VA mode for virtio-user") Cc: stable@dpdk.org Reported-by: Olivier Matz Signed-off-by: Maxime Coquelin --- Changes in v2: ============== - Fix cosmetics issues reported by David --- drivers/net/virtio/virtio.h | 1 + drivers/net/virtio/virtio_ethdev.c | 27 ++++++++++++----- drivers/net/virtio/virtio_rxtx.c | 31 +++++++++----------- drivers/net/virtio/virtio_rxtx_packed.h | 2 +- drivers/net/virtio/virtio_rxtx_packed_avx.h | 8 ++--- drivers/net/virtio/virtio_rxtx_packed_neon.h | 8 ++--- drivers/net/virtio/virtio_rxtx_simple.h | 2 +- drivers/net/virtio/virtio_user_ethdev.c | 7 ++++- drivers/net/virtio/virtqueue.h | 22 +++++++++++++- 9 files changed, 72 insertions(+), 36 deletions(-) diff --git a/drivers/net/virtio/virtio.h b/drivers/net/virtio/virtio.h index 525e2dad4c..e78b2e429e 100644 --- a/drivers/net/virtio/virtio.h +++ b/drivers/net/virtio/virtio.h @@ -192,6 +192,7 @@ struct virtio_hw { uint16_t max_queue_pairs; uint64_t req_guest_features; struct virtnet_ctl *cvq; + bool use_va; }; struct virtio_ops { diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c index b08109c61c..b60eeb24ab 100644 --- a/drivers/net/virtio/virtio_ethdev.c +++ b/drivers/net/virtio/virtio_ethdev.c @@ -515,12 +515,14 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t queue_idx) memset(mz->addr, 0, mz->len); - vq->vq_ring_mem = mz->iova; + if (hw->use_va) + vq->vq_ring_mem = (uintptr_t)mz->addr; + else + vq->vq_ring_mem = mz->iova; + vq->vq_ring_virt_mem = mz->addr; - PMD_INIT_LOG(DEBUG, "vq->vq_ring_mem: 0x%" PRIx64, - (uint64_t)mz->iova); - PMD_INIT_LOG(DEBUG, "vq->vq_ring_virt_mem: 0x%" PRIx64, - (uint64_t)(uintptr_t)mz->addr); + PMD_INIT_LOG(DEBUG, "vq->vq_ring_mem: 0x%" PRIx64, vq->vq_ring_mem); + PMD_INIT_LOG(DEBUG, "vq->vq_ring_virt_mem: %p", vq->vq_ring_virt_mem); virtio_init_vring(vq); @@ -570,17 +572,28 @@ virtio_init_queue(struct rte_eth_dev *dev, uint16_t queue_idx) txvq->port_id = dev->data->port_id; txvq->mz = mz; txvq->virtio_net_hdr_mz = hdr_mz; - txvq->virtio_net_hdr_mem = hdr_mz->iova; + if (hw->use_va) + txvq->virtio_net_hdr_mem = (uintptr_t)hdr_mz->addr; + else + txvq->virtio_net_hdr_mem = hdr_mz->iova; } else if (queue_type == VTNET_CQ) { cvq = &vq->cq; cvq->mz = mz; cvq->virtio_net_hdr_mz = hdr_mz; - cvq->virtio_net_hdr_mem = hdr_mz->iova; + if (hw->use_va) + cvq->virtio_net_hdr_mem = (uintptr_t)hdr_mz->addr; + else + cvq->virtio_net_hdr_mem = hdr_mz->iova; memset(cvq->virtio_net_hdr_mz->addr, 0, rte_mem_page_size()); hw->cvq = cvq; } + if (hw->use_va) + vq->mbuf_addr_offset = offsetof(struct rte_mbuf, buf_addr); + else + vq->mbuf_addr_offset = offsetof(struct rte_mbuf, buf_iova); + if (queue_type == VTNET_TQ) { struct virtio_tx_region *txr; unsigned int i; diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c index b9d7c8d18f..e8e6ed20a5 100644 --- a/drivers/net/virtio/virtio_rxtx.c +++ b/drivers/net/virtio/virtio_rxtx.c @@ -271,7 +271,7 @@ virtqueue_enqueue_refill_inorder(struct virtqueue *vq, dxp->cookie = (void *)cookies[i]; dxp->ndescs = 1; - start_dp[idx].addr = cookies[i]->buf_iova + + start_dp[idx].addr = VIRTIO_MBUF_ADDR(cookies[i], vq) + RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size; start_dp[idx].len = cookies[i]->buf_len - RTE_PKTMBUF_HEADROOM + hw->vtnet_hdr_size; @@ -310,10 +310,10 @@ virtqueue_enqueue_recv_refill(struct virtqueue *vq, struct rte_mbuf **cookie, dxp->cookie = (void *)cookie[i]; dxp->ndescs = 1; - start_dp[idx].addr = cookie[i]->buf_iova + + start_dp[idx].addr = VIRTIO_MBUF_ADDR(cookie[i], vq) + RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size; - start_dp[idx].len = cookie[i]->buf_len - - RTE_PKTMBUF_HEADROOM + hw->vtnet_hdr_size; + start_dp[idx].len = cookie[i]->buf_len - RTE_PKTMBUF_HEADROOM + + hw->vtnet_hdr_size; start_dp[idx].flags = VRING_DESC_F_WRITE; vq->vq_desc_head_idx = start_dp[idx].next; vq_update_avail_ring(vq, idx); @@ -336,13 +336,10 @@ virtqueue_refill_single_packed(struct virtqueue *vq, uint16_t flags = vq->vq_packed.cached_flags; struct virtio_hw *hw = vq->hw; - dp->addr = cookie->buf_iova + - RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size; - dp->len = cookie->buf_len - - RTE_PKTMBUF_HEADROOM + hw->vtnet_hdr_size; + dp->addr = VIRTIO_MBUF_ADDR(cookie, vq) + RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size; + dp->len = cookie->buf_len - RTE_PKTMBUF_HEADROOM + hw->vtnet_hdr_size; - virtqueue_store_flags_packed(dp, flags, - hw->weak_barriers); + virtqueue_store_flags_packed(dp, flags, hw->weak_barriers); if (++vq->vq_avail_idx >= vq->vq_nentries) { vq->vq_avail_idx -= vq->vq_nentries; @@ -482,8 +479,8 @@ virtqueue_enqueue_xmit_inorder(struct virtnet_tx *txvq, else virtqueue_xmit_offload(hdr, cookies[i]); - start_dp[idx].addr = rte_mbuf_data_iova(cookies[i]) - head_size; - start_dp[idx].len = cookies[i]->data_len + head_size; + start_dp[idx].addr = VIRTIO_MBUF_DATA_DMA_ADDR(cookies[i], vq) - head_size; + start_dp[idx].len = cookies[i]->data_len + head_size; start_dp[idx].flags = 0; @@ -529,9 +526,9 @@ virtqueue_enqueue_xmit_packed_fast(struct virtnet_tx *txvq, else virtqueue_xmit_offload(hdr, cookie); - dp->addr = rte_mbuf_data_iova(cookie) - head_size; - dp->len = cookie->data_len + head_size; - dp->id = id; + dp->addr = VIRTIO_MBUF_DATA_DMA_ADDR(cookie, vq) - head_size; + dp->len = cookie->data_len + head_size; + dp->id = id; if (++vq->vq_avail_idx >= vq->vq_nentries) { vq->vq_avail_idx -= vq->vq_nentries; @@ -617,8 +614,8 @@ virtqueue_enqueue_xmit(struct virtnet_tx *txvq, struct rte_mbuf *cookie, virtqueue_xmit_offload(hdr, cookie); do { - start_dp[idx].addr = rte_mbuf_data_iova(cookie); - start_dp[idx].len = cookie->data_len; + start_dp[idx].addr = VIRTIO_MBUF_DATA_DMA_ADDR(cookie, vq); + start_dp[idx].len = cookie->data_len; if (prepend_header) { start_dp[idx].addr -= head_size; start_dp[idx].len += head_size; diff --git a/drivers/net/virtio/virtio_rxtx_packed.h b/drivers/net/virtio/virtio_rxtx_packed.h index 1d1db60da8..77e5cb37e7 100644 --- a/drivers/net/virtio/virtio_rxtx_packed.h +++ b/drivers/net/virtio/virtio_rxtx_packed.h @@ -288,7 +288,7 @@ virtio_recv_refill_packed_vec(struct virtnet_rx *rxvq, dxp = &vq->vq_descx[idx + i]; dxp->cookie = (void *)cookie[total_num + i]; - addr = cookie[total_num + i]->buf_iova + + addr = VIRTIO_MBUF_ADDR(cookie[total_num + i], vq) + RTE_PKTMBUF_HEADROOM - hw->vtnet_hdr_size; start_dp[idx + i].addr = addr; start_dp[idx + i].len = cookie[total_num + i]->buf_len diff --git a/drivers/net/virtio/virtio_rxtx_packed_avx.h b/drivers/net/virtio/virtio_rxtx_packed_avx.h index c819d2e4f2..8cb71f3fe6 100644 --- a/drivers/net/virtio/virtio_rxtx_packed_avx.h +++ b/drivers/net/virtio/virtio_rxtx_packed_avx.h @@ -71,13 +71,13 @@ virtqueue_enqueue_batch_packed_vec(struct virtnet_tx *txvq, } __m512i descs_base = _mm512_set_epi64(tx_pkts[3]->data_len, - tx_pkts[3]->buf_iova, + VIRTIO_MBUF_ADDR(tx_pkts[3], vq), tx_pkts[2]->data_len, - tx_pkts[2]->buf_iova, + VIRTIO_MBUF_ADDR(tx_pkts[2], vq), tx_pkts[1]->data_len, - tx_pkts[1]->buf_iova, + VIRTIO_MBUF_ADDR(tx_pkts[1], vq), tx_pkts[0]->data_len, - tx_pkts[0]->buf_iova); + VIRTIO_MBUF_ADDR(tx_pkts[0], vq)); /* id offset and data offset */ __m512i data_offsets = _mm512_set_epi64((uint64_t)3 << ID_BITS_OFFSET, diff --git a/drivers/net/virtio/virtio_rxtx_packed_neon.h b/drivers/net/virtio/virtio_rxtx_packed_neon.h index f19e618635..c222ebf00c 100644 --- a/drivers/net/virtio/virtio_rxtx_packed_neon.h +++ b/drivers/net/virtio/virtio_rxtx_packed_neon.h @@ -97,12 +97,12 @@ virtqueue_enqueue_batch_packed_vec(struct virtnet_tx *txvq, uint64x2x2_t desc[PACKED_BATCH_SIZE / 2]; uint64x2_t base_addr0 = { - tx_pkts[0]->buf_iova + tx_pkts[0]->data_off, - tx_pkts[1]->buf_iova + tx_pkts[1]->data_off + VIRTIO_MBUF_ADDR(tx_pkts[0], vq) + tx_pkts[0]->data_off, + VIRTIO_MBUF_ADDR(tx_pkts[1], vq) + tx_pkts[1]->data_off }; uint64x2_t base_addr1 = { - tx_pkts[2]->buf_iova + tx_pkts[2]->data_off, - tx_pkts[3]->buf_iova + tx_pkts[3]->data_off + VIRTIO_MBUF_ADDR(tx_pkts[2], vq) + tx_pkts[2]->data_off, + VIRTIO_MBUF_ADDR(tx_pkts[3], vq) + tx_pkts[3]->data_off }; desc[0].val[0] = base_addr0; diff --git a/drivers/net/virtio/virtio_rxtx_simple.h b/drivers/net/virtio/virtio_rxtx_simple.h index f258771fcf..d8f96e0434 100644 --- a/drivers/net/virtio/virtio_rxtx_simple.h +++ b/drivers/net/virtio/virtio_rxtx_simple.h @@ -43,7 +43,7 @@ virtio_rxq_rearm_vec(struct virtnet_rx *rxvq) p = (uintptr_t)&sw_ring[i]->rearm_data; *(uint64_t *)p = rxvq->mbuf_initializer; - start_dp[i].addr = sw_ring[i]->buf_iova + + start_dp[i].addr = VIRTIO_MBUF_ADDR(sw_ring[i], vq) + RTE_PKTMBUF_HEADROOM - vq->hw->vtnet_hdr_size; start_dp[i].len = sw_ring[i]->buf_len - RTE_PKTMBUF_HEADROOM + vq->hw->vtnet_hdr_size; diff --git a/drivers/net/virtio/virtio_user_ethdev.c b/drivers/net/virtio/virtio_user_ethdev.c index 688c1104d5..0271098f0d 100644 --- a/drivers/net/virtio/virtio_user_ethdev.c +++ b/drivers/net/virtio/virtio_user_ethdev.c @@ -657,6 +657,12 @@ virtio_user_pmd_probe(struct rte_vdev_device *vdev) goto end; } + /* + * Virtio-user requires using virtual addresses for the descriptors + * buffers, whatever other devices require + */ + hw->use_va = true; + /* previously called by pci probing for physical dev */ if (eth_virtio_dev_init(eth_dev) < 0) { PMD_INIT_LOG(ERR, "eth_virtio_dev_init fails"); @@ -769,7 +775,6 @@ static struct rte_vdev_driver virtio_user_driver = { .remove = virtio_user_pmd_remove, .dma_map = virtio_user_pmd_dma_map, .dma_unmap = virtio_user_pmd_dma_unmap, - .drv_flags = RTE_VDEV_DRV_NEED_IOVA_AS_VA, }; RTE_PMD_REGISTER_VDEV(net_virtio_user, virtio_user_driver); diff --git a/drivers/net/virtio/virtqueue.h b/drivers/net/virtio/virtqueue.h index d0c48ca415..5baac221f7 100644 --- a/drivers/net/virtio/virtqueue.h +++ b/drivers/net/virtio/virtqueue.h @@ -113,6 +113,25 @@ virtqueue_store_flags_packed(struct vring_packed_desc *dp, #define VIRTQUEUE_MAX_NAME_SZ 32 +/** + * Return the IOVA (or virtual address in case of virtio-user) of mbuf + * data buffer. + * + * The address is firstly casted to the word size (sizeof(uintptr_t)) + * before casting it to uint64_t. This is to make it work with different + * combination of word size (64 bit and 32 bit) and virtio device + * (virtio-pci and virtio-user). + */ +#define VIRTIO_MBUF_ADDR(mb, vq) \ + ((uint64_t)(*(uintptr_t *)((uintptr_t)(mb) + (vq)->mbuf_addr_offset))) + +/** + * Return the physical address (or virtual address in case of + * virtio-user) of mbuf data buffer, taking care of mbuf data offset + */ +#define VIRTIO_MBUF_DATA_DMA_ADDR(mb, vq) \ + (VIRTIO_MBUF_ADDR(mb, vq) + (mb)->data_off) + #define VTNET_SQ_RQ_QUEUE_IDX 0 #define VTNET_SQ_TQ_QUEUE_IDX 1 #define VTNET_SQ_CQ_QUEUE_IDX 2 @@ -255,6 +274,7 @@ struct virtqueue { void *vq_ring_virt_mem; /**< linear address of vring*/ unsigned int vq_ring_size; + uint16_t mbuf_addr_offset; union { struct virtnet_rx rxq; @@ -739,7 +759,7 @@ virtqueue_enqueue_xmit_packed(struct virtnet_tx *txvq, struct rte_mbuf *cookie, do { uint16_t flags; - start_dp[idx].addr = rte_mbuf_data_iova(cookie); + start_dp[idx].addr = VIRTIO_MBUF_DATA_DMA_ADDR(cookie, vq); start_dp[idx].len = cookie->data_len; if (prepend_header) { start_dp[idx].addr -= head_size; -- 2.31.1