From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 59E00467D6; Sat, 24 May 2025 11:14:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 01D9440E24; Sat, 24 May 2025 11:13:57 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 79E0C40E49 for ; Sat, 24 May 2025 11:13:55 +0200 (CEST) Received: from pps.filterd (m0431383.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 54O9DLDJ012478; Sat, 24 May 2025 02:13:55 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pfpt0220; bh=Z 3KXSvyk+KriLRh/KVv7BecYZ3BFEihAPNDP+AUadwI=; b=Cs/RapLWEONa1JKXt F2co5niKEEAcky3WHhS2zm7Dyn0s2lhd0pqP7HM+/j5kGlCB0sBnzdl6SVCzHezf QY9NCZ08YDEr0w+lP00IC1nkfgNOszHPgTu0/ztGPoJQ5eZuhW0h6kiWSNJ2DmgB 0cwGT12Uomlv9BhXE0HQddFy1xuaKvvMf8pmesrT3wpct6Be2xNLyU0gz5MtooXN d3ApoI5ILnQfJaQJXGeyx54mbT8S+FaU48Ist7iibb0I8s6L/2MFl7vW4Q+xQVfR Bl4SLC4yKQhNDk/ih70CR4WB0WG7AYD2mCnH7on+oGHu6r66KArxUbJNUv9DlMUV W6y1g== Received: from dc6wp-exch02.marvell.com ([4.21.29.225]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 46u1pc0maf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 24 May 2025 02:13:54 -0700 (PDT) Received: from DC6WP-EXCH02.marvell.com (10.76.176.209) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sat, 24 May 2025 02:13:53 -0700 Received: from maili.marvell.com (10.69.176.80) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sat, 24 May 2025 02:13:53 -0700 Received: from MININT-80QBFE8.corp.innovium.com (MININT-80QBFE8.marvell.com [10.28.164.118]) by maili.marvell.com (Postfix) with ESMTP id 79CDF3F7079; Sat, 24 May 2025 02:13:48 -0700 (PDT) From: To: , Vamsi Attunuru CC: , , , , , , , , , , , "Pavan Nikhilesh" Subject: [25.11 PATCH v3 4/5] dma/cnxk: implement enqueue dequeue ops Date: Sat, 24 May 2025 14:43:14 +0530 Message-ID: <20250524091316.10056-5-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250524091316.10056-1-pbhagavatula@marvell.com> References: <20250519185604.5584-1-pbhagavatula@marvell.com> <20250524091316.10056-1-pbhagavatula@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-ORIG-GUID: sv1CtyTEeTPElYtXhEf5nqWeXuFJU_iL X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwNTI0MDA4MyBTYWx0ZWRfX4WdwSHHhKFVC eYiIGN91ua+rNskskTIXn6t2i+YMDn7XZf4upHWc9f1U2lNSTc97PpBDTxQVyrgrlTmfWhU9wAa DedRRUuRrWFQrysfbZkyi8eegilSJyIw5ZPeWz1jnnFpkpU72BrQyECpF3LZDha5JYP4w+jIAA8 bm3hKwrpGm7kVu7kMgadKu08Trkma0YMzZRHM6WoJx4qklA8UfXdOAhwvLHZHalbSs9C2ebZFzp 5AHvg3ZERzCbkMAMDT6vmFC6Ono5N6MsdV6Kx5SZ1roMG3lFhxcrHe2PPXFw5caih7r9UIcgfxL ZWHMF2zWISIejnUxMGlFKkRmX5+J0g3nM38JZFDfGWmasny9tn10/dUpi3oPCgCC8HNI2fsQwaH 6yp8LydZ/Xtn5r3THDvY14KCaRMXJ0S+BNEKWsVP8LRosBnCE+d7YJdt2Jvh1M+VuQeck6l+ X-Authority-Analysis: v=2.4 cv=JovxrN4C c=1 sm=1 tr=0 ts=68318dd2 cx=c_pps a=gIfcoYsirJbf48DBMSPrZA==:117 a=gIfcoYsirJbf48DBMSPrZA==:17 a=dt9VzEwgFbYA:10 a=M5GUcnROAAAA:8 a=xKmJUHBexbacZ4eocFsA:9 a=OBjm3rFKGHvpk9ecZwUJ:22 X-Proofpoint-GUID: sv1CtyTEeTPElYtXhEf5nqWeXuFJU_iL X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.0.736,FMLib:17.12.80.40 definitions=2025-05-24_04,2025-05-22_01,2025-03-28_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Pavan Nikhilesh Implement DMA enqueue/dequeue operations when application enables it via configuration. Signed-off-by: Pavan Nikhilesh --- drivers/dma/cnxk/cnxk_dmadev.c | 78 ++++++++---- drivers/dma/cnxk/cnxk_dmadev.h | 7 ++ drivers/dma/cnxk/cnxk_dmadev_fp.c | 189 ++++++++++++++++++++++++++---- 3 files changed, 232 insertions(+), 42 deletions(-) diff --git a/drivers/dma/cnxk/cnxk_dmadev.c b/drivers/dma/cnxk/cnxk_dmadev.c index 90bb69011f..f12a15aef7 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.c +++ b/drivers/dma/cnxk/cnxk_dmadev.c @@ -7,6 +7,7 @@ #include static int cnxk_stats_reset(struct rte_dma_dev *dev, uint16_t vchan); +static void cnxk_set_fp_ops(struct rte_dma_dev *dev, uint8_t enable_enq_deq); static int cnxk_dmadev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *dev_info, uint32_t size) @@ -19,7 +20,7 @@ cnxk_dmadev_info_get(const struct rte_dma_dev *dev, struct rte_dma_info *dev_inf dev_info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM | RTE_DMA_CAPA_MEM_TO_DEV | RTE_DMA_CAPA_DEV_TO_MEM | RTE_DMA_CAPA_DEV_TO_DEV | RTE_DMA_CAPA_OPS_COPY | RTE_DMA_CAPA_OPS_COPY_SG | - RTE_DMA_CAPA_M2D_AUTO_FREE; + RTE_DMA_CAPA_M2D_AUTO_FREE | RTE_DMA_CAPA_OPS_ENQ_DEQ; if (roc_feature_dpi_has_priority()) { dev_info->dev_capa |= RTE_DMA_CAPA_PRI_POLICY_SP; dev_info->nb_priorities = CN10K_DPI_MAX_PRI; @@ -114,6 +115,8 @@ cnxk_dmadev_configure(struct rte_dma_dev *dev, const struct rte_dma_conf *conf, if (roc_feature_dpi_has_priority()) dpivf->rdpi.priority = conf->priority; + cnxk_set_fp_ops(dev, conf->flags & RTE_DMA_CFG_FLAG_ENQ_DEQ); + return 0; } @@ -270,6 +273,14 @@ cnxk_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, return -ENOMEM; } + size = (max_desc * sizeof(struct rte_dma_op *)); + dpi_conf->c_desc.ops = rte_zmalloc(NULL, size, RTE_CACHE_LINE_SIZE); + if (dpi_conf->c_desc.ops == NULL) { + plt_err("Failed to allocate for ops array"); + rte_free(dpi_conf->c_desc.compl_ptr); + return -ENOMEM; + } + for (i = 0; i < max_desc; i++) dpi_conf->c_desc.compl_ptr[i * CNXK_DPI_COMPL_OFFSET] = CNXK_DPI_REQ_CDATA; @@ -382,23 +393,25 @@ cnxk_dmadev_completed(void *dev_private, uint16_t vchan, const uint16_t nb_cpls, struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; struct cnxk_dpi_cdesc_data_s *c_desc = &dpi_conf->c_desc; + const uint16_t max_cnt = c_desc->max_cnt; uint8_t status; int cnt; for (cnt = 0; cnt < nb_cpls; cnt++) { - status = c_desc->compl_ptr[c_desc->head * CNXK_DPI_COMPL_OFFSET]; + status = c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET]; if (status) { if (status == CNXK_DPI_REQ_CDATA) break; *has_error = 1; dpi_conf->stats.errors++; - c_desc->compl_ptr[c_desc->head * CNXK_DPI_COMPL_OFFSET] = + c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET] = CNXK_DPI_REQ_CDATA; - CNXK_DPI_STRM_INC(*c_desc, head); + c_desc->head++; break; } - c_desc->compl_ptr[c_desc->head * CNXK_DPI_COMPL_OFFSET] = CNXK_DPI_REQ_CDATA; - CNXK_DPI_STRM_INC(*c_desc, head); + c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET] = + CNXK_DPI_REQ_CDATA; + c_desc->head++; } dpi_conf->stats.completed += cnt; @@ -414,18 +427,20 @@ cnxk_dmadev_completed_status(void *dev_private, uint16_t vchan, const uint16_t n struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; struct cnxk_dpi_cdesc_data_s *c_desc = &dpi_conf->c_desc; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; int cnt; for (cnt = 0; cnt < nb_cpls; cnt++) { - status[cnt] = c_desc->compl_ptr[c_desc->head * CNXK_DPI_COMPL_OFFSET]; + status[cnt] = c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET]; if (status[cnt]) { if (status[cnt] == CNXK_DPI_REQ_CDATA) break; dpi_conf->stats.errors++; } - c_desc->compl_ptr[c_desc->head * CNXK_DPI_COMPL_OFFSET] = CNXK_DPI_REQ_CDATA; - CNXK_DPI_STRM_INC(*c_desc, head); + c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET] = + CNXK_DPI_REQ_CDATA; + c_desc->head++; } dpi_conf->stats.completed += cnt; @@ -528,6 +543,37 @@ cnxk_stats_reset(struct rte_dma_dev *dev, uint16_t vchan) return 0; } +static void +cnxk_set_fp_ops(struct rte_dma_dev *dev, uint8_t ena_enq_deq) +{ + + dev->fp_obj->copy = cnxk_dmadev_copy; + dev->fp_obj->copy_sg = cnxk_dmadev_copy_sg; + dev->fp_obj->submit = cnxk_dmadev_submit; + dev->fp_obj->completed = cnxk_dmadev_completed; + dev->fp_obj->completed_status = cnxk_dmadev_completed_status; + dev->fp_obj->burst_capacity = cnxk_damdev_burst_capacity; + + if (roc_model_is_cn10k()) { + dev->fp_obj->copy = cn10k_dmadev_copy; + dev->fp_obj->copy_sg = cn10k_dmadev_copy_sg; + } + + if (ena_enq_deq) { + dev->fp_obj->copy = NULL; + dev->fp_obj->copy_sg = NULL; + dev->fp_obj->submit = NULL; + dev->fp_obj->completed = NULL; + dev->fp_obj->completed_status = NULL; + + dev->fp_obj->enqueue = cnxk_dma_ops_enqueue; + dev->fp_obj->dequeue = cnxk_dma_ops_dequeue; + + if (roc_model_is_cn10k()) + dev->fp_obj->enqueue = cn10k_dma_ops_enqueue; + } +} + static const struct rte_dma_dev_ops cnxk_dmadev_ops = { .dev_close = cnxk_dmadev_close, .dev_configure = cnxk_dmadev_configure, @@ -571,19 +617,7 @@ cnxk_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_de dmadev->fp_obj->dev_private = dpivf; dmadev->dev_ops = &cnxk_dmadev_ops; - dmadev->fp_obj->copy = cnxk_dmadev_copy; - dmadev->fp_obj->copy_sg = cnxk_dmadev_copy_sg; - dmadev->fp_obj->submit = cnxk_dmadev_submit; - dmadev->fp_obj->completed = cnxk_dmadev_completed; - dmadev->fp_obj->completed_status = cnxk_dmadev_completed_status; - dmadev->fp_obj->burst_capacity = cnxk_damdev_burst_capacity; - - if (roc_model_is_cn10k()) { - dpivf->is_cn10k = true; - dmadev->fp_obj->copy = cn10k_dmadev_copy; - dmadev->fp_obj->copy_sg = cn10k_dmadev_copy_sg; - } - + dpivf->is_cn10k = roc_model_is_cn10k(); dpivf->mcs_lock = NULL; rdpi = &dpivf->rdpi; diff --git a/drivers/dma/cnxk/cnxk_dmadev.h b/drivers/dma/cnxk/cnxk_dmadev.h index 9a232a5464..18039e43fb 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.h +++ b/drivers/dma/cnxk/cnxk_dmadev.h @@ -93,6 +93,7 @@ struct cnxk_dpi_cdesc_data_s { uint16_t head; uint16_t tail; uint8_t *compl_ptr; + struct rte_dma_op **ops; }; struct cnxk_dpi_conf { @@ -131,5 +132,11 @@ int cn10k_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iov int cn10k_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge *src, const struct rte_dma_sge *dst, uint16_t nb_src, uint16_t nb_dst, uint64_t flags); +uint16_t cnxk_dma_ops_enqueue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, + uint16_t nb_ops); +uint16_t cn10k_dma_ops_enqueue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, + uint16_t nb_ops); +uint16_t cnxk_dma_ops_dequeue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, + uint16_t nb_ops); #endif diff --git a/drivers/dma/cnxk/cnxk_dmadev_fp.c b/drivers/dma/cnxk/cnxk_dmadev_fp.c index dea73c5b41..579aafa894 100644 --- a/drivers/dma/cnxk/cnxk_dmadev_fp.c +++ b/drivers/dma/cnxk/cnxk_dmadev_fp.c @@ -246,15 +246,16 @@ cnxk_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iova_t d struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; uint64_t cmd[CNXK_DPI_DW_PER_SINGLE_CMD]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; uint8_t *comp_ptr; int rc; - if (unlikely(((dpi_conf->c_desc.tail + 1) & dpi_conf->c_desc.max_cnt) == - dpi_conf->c_desc.head)) + if (unlikely(((dpi_conf->c_desc.tail + 1) & max_cnt) == (dpi_conf->c_desc.head & max_cnt))) return -ENOSPC; - comp_ptr = &dpi_conf->c_desc.compl_ptr[dpi_conf->c_desc.tail * CNXK_DPI_COMPL_OFFSET]; - CNXK_DPI_STRM_INC(dpi_conf->c_desc, tail); + comp_ptr = &dpi_conf->c_desc + .compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.tail++; cmd[0] = (1UL << 54) | (1UL << 48); cmd[1] = dpi_conf->cmd.u | ((flags & RTE_DMA_OP_FLAG_AUTO_FREE) << 37); @@ -276,7 +277,7 @@ cnxk_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iova_t d rc = __dpi_queue_write_single(dpivf, cmd); if (unlikely(rc)) { - CNXK_DPI_STRM_DEC(dpi_conf->c_desc, tail); + dpi_conf->c_desc.tail--; return rc; } @@ -300,17 +301,18 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge { struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; const struct rte_dma_sge *fptr, *lptr; uint8_t *comp_ptr; uint64_t hdr[4]; int rc; - if (unlikely(((dpi_conf->c_desc.tail + 1) & dpi_conf->c_desc.max_cnt) == - dpi_conf->c_desc.head)) + if (unlikely(((dpi_conf->c_desc.tail + 1) & max_cnt) == (dpi_conf->c_desc.head & max_cnt))) return -ENOSPC; - comp_ptr = &dpi_conf->c_desc.compl_ptr[dpi_conf->c_desc.tail * CNXK_DPI_COMPL_OFFSET]; - CNXK_DPI_STRM_INC(dpi_conf->c_desc, tail); + comp_ptr = &dpi_conf->c_desc + .compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.tail++; hdr[1] = dpi_conf->cmd.u | ((flags & RTE_DMA_OP_FLAG_AUTO_FREE) << 37); hdr[2] = (uint64_t)comp_ptr; @@ -331,7 +333,7 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge rc = __dpi_queue_write_sg(dpivf, hdr, fptr, lptr, nb_src, nb_dst); if (unlikely(rc)) { - CNXK_DPI_STRM_DEC(dpi_conf->c_desc, tail); + dpi_conf->c_desc.tail--; return rc; } @@ -355,16 +357,17 @@ cn10k_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iova_t { struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; uint64_t cmd[CNXK_DPI_DW_PER_SINGLE_CMD]; uint8_t *comp_ptr; int rc; - if (unlikely(((dpi_conf->c_desc.tail + 1) & dpi_conf->c_desc.max_cnt) == - dpi_conf->c_desc.head)) + if (unlikely(((dpi_conf->c_desc.tail + 1) & max_cnt) == (dpi_conf->c_desc.head & max_cnt))) return -ENOSPC; - comp_ptr = &dpi_conf->c_desc.compl_ptr[dpi_conf->c_desc.tail * CNXK_DPI_COMPL_OFFSET]; - CNXK_DPI_STRM_INC(dpi_conf->c_desc, tail); + comp_ptr = &dpi_conf->c_desc + .compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.tail++; cmd[0] = dpi_conf->cmd.u | (1U << 6) | 1U; cmd[1] = (uint64_t)comp_ptr; @@ -376,7 +379,7 @@ cn10k_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, rte_iova_t rc = __dpi_queue_write_single(dpivf, cmd); if (unlikely(rc)) { - CNXK_DPI_STRM_DEC(dpi_conf->c_desc, tail); + dpi_conf->c_desc.tail--; return rc; } @@ -401,16 +404,17 @@ cn10k_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge { struct cnxk_dpi_vf_s *dpivf = dev_private; struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; uint8_t *comp_ptr; uint64_t hdr[4]; int rc; - if (unlikely(((dpi_conf->c_desc.tail + 1) & dpi_conf->c_desc.max_cnt) == - dpi_conf->c_desc.head)) + if (unlikely(((dpi_conf->c_desc.tail + 1) & max_cnt) == (dpi_conf->c_desc.head & max_cnt))) return -ENOSPC; - comp_ptr = &dpi_conf->c_desc.compl_ptr[dpi_conf->c_desc.tail * CNXK_DPI_COMPL_OFFSET]; - CNXK_DPI_STRM_INC(dpi_conf->c_desc, tail); + comp_ptr = &dpi_conf->c_desc + .compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.tail++; hdr[0] = dpi_conf->cmd.u | (nb_dst << 6) | nb_src; hdr[1] = (uint64_t)comp_ptr; @@ -418,7 +422,7 @@ cn10k_dmadev_copy_sg(void *dev_private, uint16_t vchan, const struct rte_dma_sge rc = __dpi_queue_write_sg(dpivf, hdr, src, dst, nb_src, nb_dst); if (unlikely(rc)) { - CNXK_DPI_STRM_DEC(dpi_conf->c_desc, tail); + dpi_conf->c_desc.tail--; return rc; } @@ -670,3 +674,148 @@ cnxk_dma_adapter_dequeue(uintptr_t get_work1) return (uintptr_t)op; } + +uint16_t +cnxk_dma_ops_enqueue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, uint16_t nb_ops) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; + const struct rte_dma_sge *fptr, *lptr; + uint16_t src, dst, nwords = 0; + struct rte_dma_op *op; + uint16_t space, i; + uint8_t *comp_ptr; + uint64_t hdr[4]; + int rc; + + space = ((dpi_conf->c_desc.max_cnt + dpi_conf->c_desc.head - dpi_conf->c_desc.tail) & + max_cnt); + space = RTE_MIN(space, nb_ops); + + for (i = 0; i < space; i++) { + op = ops[i]; + comp_ptr = &dpi_conf->c_desc.compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * + CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.ops[dpi_conf->c_desc.tail & max_cnt] = op; + dpi_conf->c_desc.tail++; + + hdr[1] = dpi_conf->cmd.u | ((op->flags & RTE_DMA_OP_FLAG_AUTO_FREE) << 37); + hdr[2] = (uint64_t)comp_ptr; + + src = op->nb_src; + dst = op->nb_dst; + /* + * For inbound case, src pointers are last pointers. + * For all other cases, src pointers are first pointers. + */ + if (((dpi_conf->cmd.u >> 48) & DPI_HDR_XTYPE_MASK) == DPI_XTYPE_INBOUND) { + fptr = &op->src_dst_seg[src]; + lptr = &op->src_dst_seg[0]; + RTE_SWAP(src, dst); + } else { + fptr = &op->src_dst_seg[0]; + lptr = &op->src_dst_seg[src]; + } + hdr[0] = ((uint64_t)dst << 54) | (uint64_t)src << 48; + + rc = __dpi_queue_write_sg(dpivf, hdr, fptr, lptr, src, dst); + if (rc) { + dpi_conf->c_desc.tail--; + goto done; + } + nwords += CNXK_DPI_CMD_LEN(src, dst); + } + +done: + if (nwords) { + rte_wmb(); + plt_write64(nwords, dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpi_conf->stats.submitted += i; + } + + return i; +} + +uint16_t +cn10k_dma_ops_enqueue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, uint16_t nb_ops) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + const uint16_t max_cnt = dpi_conf->c_desc.max_cnt; + uint16_t space, i, nwords = 0; + struct rte_dma_op *op; + uint16_t src, dst; + uint8_t *comp_ptr; + uint64_t hdr[4]; + int rc; + + space = (max_cnt + dpi_conf->c_desc.head - dpi_conf->c_desc.tail) & max_cnt; + space = RTE_MIN(space, nb_ops); + + for (i = 0; i < space; i++) { + op = ops[i]; + src = op->nb_src; + dst = op->nb_dst; + + comp_ptr = &dpi_conf->c_desc.compl_ptr[(dpi_conf->c_desc.tail & max_cnt) * + CNXK_DPI_COMPL_OFFSET]; + dpi_conf->c_desc.ops[dpi_conf->c_desc.tail & max_cnt] = op; + dpi_conf->c_desc.tail++; + + hdr[0] = dpi_conf->cmd.u | (dst << 6) | src; + hdr[1] = (uint64_t)comp_ptr; + hdr[2] = (1UL << 47) | ((op->flags & RTE_DMA_OP_FLAG_AUTO_FREE) << 43); + + rc = __dpi_queue_write_sg(dpivf, hdr, &op->src_dst_seg[0], &op->src_dst_seg[src], + src, dst); + if (rc) { + dpi_conf->c_desc.tail--; + goto done; + } + nwords += CNXK_DPI_CMD_LEN(src, dst); + } + +done: + if (nwords) { + rte_wmb(); + plt_write64(nwords, dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpi_conf->stats.submitted += i; + } + + return i; +} + +uint16_t +cnxk_dma_ops_dequeue(void *dev_private, uint16_t vchan, struct rte_dma_op **ops, uint16_t nb_ops) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + struct cnxk_dpi_conf *dpi_conf = &dpivf->conf[vchan]; + struct cnxk_dpi_cdesc_data_s *c_desc = &dpi_conf->c_desc; + const uint16_t max_cnt = c_desc->max_cnt; + struct rte_dma_op *op; + uint16_t space, cnt; + uint8_t status; + + space = (c_desc->tail - c_desc->head) & max_cnt; + space = RTE_MIN(nb_ops, space); + + for (cnt = 0; cnt < space; cnt++) { + status = c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET]; + op = c_desc->ops[c_desc->head & max_cnt]; + if (status) { + if (status == CNXK_DPI_REQ_CDATA) + break; + dpi_conf->stats.errors++; + } + op->status = status; + ops[cnt] = op; + c_desc->compl_ptr[(c_desc->head & max_cnt) * CNXK_DPI_COMPL_OFFSET] = + CNXK_DPI_REQ_CDATA; + c_desc->head++; + } + + dpi_conf->stats.completed += cnt; + + return cnt; +} -- 2.43.0