From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 53446A00C4; Fri, 5 Aug 2022 21:30:14 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2797542C0C; Fri, 5 Aug 2022 21:30:14 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 5E0CD400D6 for ; Fri, 5 Aug 2022 21:30:12 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 275JTnH4007880; Fri, 5 Aug 2022 12:30:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=Uu17ACdtZjuPFH89mWeV5gOmRqLjfIKMAQamHYMmTgc=; b=G3n+bvvVlrJboMVTw8XJv0xkih7Z5EBu47o1MwIRn0ZM7ZhDlPtMWTD6blpju/nFexby fB1v6BVDNYLVJVLBYZXLLYdZn4yAG9gi891JqeaUNF+4DwOLSWbYoXQeXJiveSBXKfhE V1ddyg97BgWMVmwJ0aVI49CeMJ+jXI494/cOQy3SHvYpLmSR1unUl7lyDxdyb5/JTWlX vgN3LWzPSvIran6iL5VyH1L7Bu5SQ28cLxqM8jP0Hscpo3j26yLUKwNbm+wQwUQhImHc YsZyd//pHyB/38PMrXTYJ0Xt/IdlVLPRwtAL/OXI8GVb0JztIqe0SyBvVblr0NIqsVJq bQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 3hrg4cnf97-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 05 Aug 2022 12:30:10 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 5 Aug 2022 12:30:08 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Fri, 5 Aug 2022 12:30:08 -0700 Received: from rchintakuntla-lnx3.caveonetworks.com (unknown [10.111.140.81]) by maili.marvell.com (Postfix) with ESMTP id B26DB5B6955; Fri, 5 Aug 2022 12:30:07 -0700 (PDT) From: Radha Mohan Chintakuntla To: CC: , Radha Mohan Chintakuntla , Nithin Dabilpuram , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , Veerasenareddy Burru Subject: [PATCH v3] dma/cnxk: add support for CN10K DMA engine Date: Fri, 5 Aug 2022 12:30:07 -0700 Message-ID: <20220805193007.26020-1-radhac@marvell.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220805185715.22862-1-radhac@marvell.com> References: <20220805185715.22862-1-radhac@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-GUID: q_6ycWzjeRtlGxMSl1cHKpjFUgA-WL1U X-Proofpoint-ORIG-GUID: q_6ycWzjeRtlGxMSl1cHKpjFUgA-WL1U X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-08-05_09,2022-08-05_01,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added support for CN10K SoC DMA engine to dmadev. Signed-off-by: Radha Mohan Chintakuntla Reviewed-by: Jerin Jacob Kollanukkaran --- Changes from v2: - Added missing files required in the patch Changes from v1: - Removed gerrit changeID drivers/common/cnxk/hw/dpi.h | 16 +- drivers/common/cnxk/roc_dpi_priv.h | 2 +- drivers/dma/cnxk/cnxk_dmadev.c | 248 +++++++++++++++++++++++++---- drivers/dma/cnxk/cnxk_dmadev.h | 2 +- 4 files changed, 225 insertions(+), 43 deletions(-) diff --git a/drivers/common/cnxk/hw/dpi.h b/drivers/common/cnxk/hw/dpi.h index 2da123228f..a34713dde6 100644 --- a/drivers/common/cnxk/hw/dpi.h +++ b/drivers/common/cnxk/hw/dpi.h @@ -61,7 +61,7 @@ */ union dpi_instr_hdr_s { uint64_t u[4]; - struct dpi_dma_instr_hdr_s_s { + struct dpi_cn9k_instr_hdr_s_s { uint64_t tag : 32; uint64_t tt : 2; uint64_t grp : 10; @@ -93,17 +93,9 @@ union dpi_instr_hdr_s { /* Word 2 - End */ uint64_t reserved_192_255 : 64; /* Word 3 - End */ - } s; -}; + } cn9k; -/** - * Structure dpi_cn10k_instr_hdr_s for CN10K - * - * DPI DMA Instruction Header Format - */ -union dpi_cn10k_instr_hdr_s { - uint64_t u[4]; - struct dpi_cn10k_dma_instr_hdr_s_s { + struct dpi_cn10k_instr_hdr_s_s { uint64_t nfst : 4; uint64_t reserved_4_5 : 2; uint64_t nlst : 4; @@ -135,7 +127,7 @@ union dpi_cn10k_instr_hdr_s { /* Word 2 - End */ uint64_t reserved_192_255 : 64; /* Word 3 - End */ - } s; + } cn10k; }; #endif /*__DEV_DPI_HW_H__*/ diff --git a/drivers/common/cnxk/roc_dpi_priv.h b/drivers/common/cnxk/roc_dpi_priv.h index 92953fbcfc..1fa1a715d3 100644 --- a/drivers/common/cnxk/roc_dpi_priv.h +++ b/drivers/common/cnxk/roc_dpi_priv.h @@ -23,7 +23,7 @@ typedef union dpi_mbox_msg_t { uint64_t u[2]; struct dpi_mbox_message_s { /* VF ID to configure */ - uint64_t vfid : 4; + uint64_t vfid : 8; /* Command code */ uint64_t cmd : 4; /* Command buffer size in 8-byte words */ diff --git a/drivers/dma/cnxk/cnxk_dmadev.c b/drivers/dma/cnxk/cnxk_dmadev.c index 2824c1b44f..a67bcba9d4 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.c +++ b/drivers/dma/cnxk/cnxk_dmadev.c @@ -70,31 +70,31 @@ cnxk_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, RTE_SET_USED(vchan); RTE_SET_USED(conf_sz); - header->s.pt = DPI_HDR_PT_ZBW_CA; + header->cn9k.pt = DPI_HDR_PT_ZBW_CA; switch (conf->direction) { case RTE_DMA_DIR_DEV_TO_MEM: - header->s.xtype = DPI_XTYPE_INBOUND; - header->s.lport = conf->src_port.pcie.coreid; - header->s.fport = 0; - header->s.pvfe = 1; + header->cn9k.xtype = DPI_XTYPE_INBOUND; + header->cn9k.lport = conf->src_port.pcie.coreid; + header->cn9k.fport = 0; + header->cn9k.pvfe = 1; break; case RTE_DMA_DIR_MEM_TO_DEV: - header->s.xtype = DPI_XTYPE_OUTBOUND; - header->s.lport = 0; - header->s.fport = conf->dst_port.pcie.coreid; - header->s.pvfe = 1; + header->cn9k.xtype = DPI_XTYPE_OUTBOUND; + header->cn9k.lport = 0; + header->cn9k.fport = conf->dst_port.pcie.coreid; + header->cn9k.pvfe = 1; break; case RTE_DMA_DIR_MEM_TO_MEM: - header->s.xtype = DPI_XTYPE_INTERNAL_ONLY; - header->s.lport = 0; - header->s.fport = 0; - header->s.pvfe = 0; + header->cn9k.xtype = DPI_XTYPE_INTERNAL_ONLY; + header->cn9k.lport = 0; + header->cn9k.fport = 0; + header->cn9k.pvfe = 0; break; case RTE_DMA_DIR_DEV_TO_DEV: - header->s.xtype = DPI_XTYPE_EXTERNAL_ONLY; - header->s.lport = conf->src_port.pcie.coreid; - header->s.fport = conf->dst_port.pcie.coreid; + header->cn9k.xtype = DPI_XTYPE_EXTERNAL_ONLY; + header->cn9k.lport = conf->src_port.pcie.coreid; + header->cn9k.fport = conf->dst_port.pcie.coreid; }; for (i = 0; i < conf->nb_desc; i++) { @@ -103,6 +103,63 @@ cnxk_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, plt_err("Failed to allocate for comp_data"); return -ENOMEM; } + comp_data->cdata = DPI_REQ_CDATA; + dpivf->conf.c_desc.compl_ptr[i] = comp_data; + }; + dpivf->conf.c_desc.max_cnt = DPI_MAX_DESC; + dpivf->conf.c_desc.head = 0; + dpivf->conf.c_desc.tail = 0; + + return 0; +} + +static int +cn10k_dmadev_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan, + const struct rte_dma_vchan_conf *conf, + uint32_t conf_sz) +{ + struct cnxk_dpi_vf_s *dpivf = dev->fp_obj->dev_private; + struct cnxk_dpi_compl_s *comp_data; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + int i; + + RTE_SET_USED(vchan); + RTE_SET_USED(conf_sz); + + header->cn10k.pt = DPI_HDR_PT_ZBW_CA; + + switch (conf->direction) { + case RTE_DMA_DIR_DEV_TO_MEM: + header->cn10k.xtype = DPI_XTYPE_INBOUND; + header->cn10k.lport = conf->src_port.pcie.coreid; + header->cn10k.fport = 0; + header->cn10k.pvfe = 1; + break; + case RTE_DMA_DIR_MEM_TO_DEV: + header->cn10k.xtype = DPI_XTYPE_OUTBOUND; + header->cn10k.lport = 0; + header->cn10k.fport = conf->dst_port.pcie.coreid; + header->cn10k.pvfe = 1; + break; + case RTE_DMA_DIR_MEM_TO_MEM: + header->cn10k.xtype = DPI_XTYPE_INTERNAL_ONLY; + header->cn10k.lport = 0; + header->cn10k.fport = 0; + header->cn10k.pvfe = 0; + break; + case RTE_DMA_DIR_DEV_TO_DEV: + header->cn10k.xtype = DPI_XTYPE_EXTERNAL_ONLY; + header->cn10k.lport = conf->src_port.pcie.coreid; + header->cn10k.fport = conf->dst_port.pcie.coreid; + }; + + for (i = 0; i < conf->nb_desc; i++) { + comp_data = rte_zmalloc(NULL, sizeof(*comp_data), 0); + if (comp_data == NULL) { + plt_err("Failed to allocate for comp_data"); + return -ENOMEM; + } + comp_data->cdata = DPI_REQ_CDATA; dpivf->conf.c_desc.compl_ptr[i] = comp_data; }; dpivf->conf.c_desc.max_cnt = DPI_MAX_DESC; @@ -237,17 +294,17 @@ cnxk_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; comp_ptr->cdata = DPI_REQ_CDATA; - header->s.ptr = (uint64_t)comp_ptr; + header->cn9k.ptr = (uint64_t)comp_ptr; STRM_INC(dpivf->conf.c_desc); - header->s.nfst = 1; - header->s.nlst = 1; + header->cn9k.nfst = 1; + header->cn9k.nlst = 1; /* * For inbound case, src pointers are last pointers. * For all other cases, src pointers are first pointers. */ - if (header->s.xtype == DPI_XTYPE_INBOUND) { + if (header->cn9k.xtype == DPI_XTYPE_INBOUND) { fptr = dst; lptr = src; } else { @@ -296,21 +353,21 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; comp_ptr->cdata = DPI_REQ_CDATA; - header->s.ptr = (uint64_t)comp_ptr; + header->cn9k.ptr = (uint64_t)comp_ptr; STRM_INC(dpivf->conf.c_desc); /* * For inbound case, src pointers are last pointers. * For all other cases, src pointers are first pointers. */ - if (header->s.xtype == DPI_XTYPE_INBOUND) { - header->s.nfst = nb_dst & 0xf; - header->s.nlst = nb_src & 0xf; + if (header->cn9k.xtype == DPI_XTYPE_INBOUND) { + header->cn9k.nfst = nb_dst & 0xf; + header->cn9k.nlst = nb_src & 0xf; fptr = &dst[0]; lptr = &src[0]; } else { - header->s.nfst = nb_src & 0xf; - header->s.nlst = nb_dst & 0xf; + header->cn9k.nfst = nb_src & 0xf; + header->cn9k.nlst = nb_dst & 0xf; fptr = &src[0]; lptr = &dst[0]; } @@ -319,13 +376,13 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, dpivf->cmd[1] = header->u[1]; dpivf->cmd[2] = header->u[2]; num_words += 4; - for (i = 0; i < header->s.nfst; i++) { + for (i = 0; i < header->cn9k.nfst; i++) { dpivf->cmd[num_words++] = (uint64_t)fptr->length; dpivf->cmd[num_words++] = fptr->addr; fptr++; } - for (i = 0; i < header->s.nlst; i++) { + for (i = 0; i < header->cn9k.nlst; i++) { dpivf->cmd[num_words++] = (uint64_t)lptr->length; dpivf->cmd[num_words++] = lptr->addr; lptr++; @@ -342,9 +399,113 @@ cnxk_dmadev_copy_sg(void *dev_private, uint16_t vchan, dpivf->num_words += num_words; } + return (rc < 0) ? rc : dpivf->desc_idx++; +} + +static int +cn10k_dmadev_copy(void *dev_private, uint16_t vchan, rte_iova_t src, + rte_iova_t dst, uint32_t length, uint64_t flags) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + struct cnxk_dpi_compl_s *comp_ptr; + rte_iova_t fptr, lptr; + int num_words = 0; + int rc; + + RTE_SET_USED(vchan); + + comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; + comp_ptr->cdata = DPI_REQ_CDATA; + header->cn10k.ptr = (uint64_t)comp_ptr; + STRM_INC(dpivf->conf.c_desc); + + header->cn10k.nfst = 1; + header->cn10k.nlst = 1; + + fptr = src; + lptr = dst; + + dpivf->cmd[0] = header->u[0]; + dpivf->cmd[1] = header->u[1]; + dpivf->cmd[2] = header->u[2]; + /* word3 is always 0 */ + num_words += 4; + dpivf->cmd[num_words++] = length; + dpivf->cmd[num_words++] = fptr; + dpivf->cmd[num_words++] = length; + dpivf->cmd[num_words++] = lptr; + + rc = __dpi_queue_write(&dpivf->rdpi, dpivf->cmd, num_words); + if (!rc) { + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + plt_write64(num_words, + dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpivf->stats.submitted++; + } + dpivf->num_words += num_words; + } + return dpivf->desc_idx++; } +static int +cn10k_dmadev_copy_sg(void *dev_private, uint16_t vchan, + const struct rte_dma_sge *src, + const struct rte_dma_sge *dst, uint16_t nb_src, + uint16_t nb_dst, uint64_t flags) +{ + struct cnxk_dpi_vf_s *dpivf = dev_private; + union dpi_instr_hdr_s *header = &dpivf->conf.hdr; + const struct rte_dma_sge *fptr, *lptr; + struct cnxk_dpi_compl_s *comp_ptr; + int num_words = 0; + int i, rc; + + RTE_SET_USED(vchan); + + comp_ptr = dpivf->conf.c_desc.compl_ptr[dpivf->conf.c_desc.tail]; + comp_ptr->cdata = DPI_REQ_CDATA; + header->cn10k.ptr = (uint64_t)comp_ptr; + STRM_INC(dpivf->conf.c_desc); + + header->cn10k.nfst = nb_src & 0xf; + header->cn10k.nlst = nb_dst & 0xf; + fptr = &src[0]; + lptr = &dst[0]; + + dpivf->cmd[0] = header->u[0]; + dpivf->cmd[1] = header->u[1]; + dpivf->cmd[2] = header->u[2]; + num_words += 4; + + for (i = 0; i < header->cn10k.nfst; i++) { + dpivf->cmd[num_words++] = (uint64_t)fptr->length; + dpivf->cmd[num_words++] = fptr->addr; + fptr++; + } + + for (i = 0; i < header->cn10k.nlst; i++) { + dpivf->cmd[num_words++] = (uint64_t)lptr->length; + dpivf->cmd[num_words++] = lptr->addr; + lptr++; + } + + rc = __dpi_queue_write(&dpivf->rdpi, dpivf->cmd, num_words); + if (!rc) { + if (flags & RTE_DMA_OP_FLAG_SUBMIT) { + rte_wmb(); + plt_write64(num_words, + dpivf->rdpi.rbase + DPI_VDMA_DBELL); + dpivf->stats.submitted += nb_src; + } + dpivf->num_words += num_words; + } + + return (rc < 0) ? rc : dpivf->desc_idx++; +} + static uint16_t cnxk_dmadev_completed(void *dev_private, uint16_t vchan, const uint16_t nb_cpls, uint16_t *last_idx, bool *has_error) @@ -353,11 +514,17 @@ cnxk_dmadev_completed(void *dev_private, uint16_t vchan, const uint16_t nb_cpls, int cnt; RTE_SET_USED(vchan); + + if (dpivf->stats.submitted == dpivf->stats.completed) + return 0; + for (cnt = 0; cnt < nb_cpls; cnt++) { struct cnxk_dpi_compl_s *comp_ptr = dpivf->conf.c_desc.compl_ptr[cnt]; if (comp_ptr->cdata) { + if (comp_ptr->cdata == DPI_REQ_CDATA) + break; *has_error = 1; dpivf->stats.errors++; break; @@ -385,8 +552,12 @@ cnxk_dmadev_completed_status(void *dev_private, uint16_t vchan, struct cnxk_dpi_compl_s *comp_ptr = dpivf->conf.c_desc.compl_ptr[cnt]; status[cnt] = comp_ptr->cdata; - if (comp_ptr->cdata) + if (status[cnt]) { + if (status[cnt] == DPI_REQ_CDATA) + break; + dpivf->stats.errors++; + } } *last_idx = cnt - 1; @@ -435,6 +606,17 @@ cnxk_stats_reset(struct rte_dma_dev *dev, uint16_t vchan __rte_unused) return 0; } +static const struct rte_dma_dev_ops cn10k_dmadev_ops = { + .dev_close = cnxk_dmadev_close, + .dev_configure = cnxk_dmadev_configure, + .dev_info_get = cnxk_dmadev_info_get, + .dev_start = cnxk_dmadev_start, + .dev_stop = cnxk_dmadev_stop, + .stats_get = cnxk_stats_get, + .stats_reset = cnxk_stats_reset, + .vchan_setup = cn10k_dmadev_vchan_setup, +}; + static const struct rte_dma_dev_ops cnxk_dmadev_ops = { .dev_close = cnxk_dmadev_close, .dev_configure = cnxk_dmadev_configure, @@ -486,6 +668,14 @@ cnxk_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused, dmadev->fp_obj->completed = cnxk_dmadev_completed; dmadev->fp_obj->completed_status = cnxk_dmadev_completed_status; + if (pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CN10KA || + pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CNF10KA || + pci_dev->id.subsystem_device_id == PCI_SUBSYSTEM_DEVID_CN10KB) { + dmadev->dev_ops = &cn10k_dmadev_ops; + dmadev->fp_obj->copy = cn10k_dmadev_copy; + dmadev->fp_obj->copy_sg = cn10k_dmadev_copy_sg; + } + rdpi = &dpivf->rdpi; rdpi->pci_dev = pci_dev; diff --git a/drivers/dma/cnxk/cnxk_dmadev.h b/drivers/dma/cnxk/cnxk_dmadev.h index 5fc241b55e..e1f5694f50 100644 --- a/drivers/dma/cnxk/cnxk_dmadev.h +++ b/drivers/dma/cnxk/cnxk_dmadev.h @@ -8,7 +8,7 @@ #define DPI_QUEUE_STOP 0x0 #define DPI_QUEUE_START 0x1 #define STRM_INC(s) ((s).tail = ((s).tail + 1) % (s).max_cnt) -#define DPI_MAX_DESC DPI_MAX_POINTER +#define DPI_MAX_DESC 1024 /* Set Completion data to 0xFF when request submitted, * upon successful request completion engine reset to completion status -- 2.17.1