[dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver

DPDK patches and discussions
 help / color / mirror / Atom feed

* [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver
@ 2021-10-30 10:36 Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
                   ` (6 more replies)
  0 siblings, 7 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch set add hisilicon DMA driver.

Chengwen Feng (6):
  dma/hisilicon: add device probe and remove functions
  dma/hisilicon: add dmadev instances create and destroy
  dma/hisilicon: add control path functions
  dma/hisilicon: add data path functions
  dma/hisilicon: support multi-process
  devbind: add Kunpeng DMA to dmadev category

 MAINTAINERS                            |   5 +
 doc/guides/dmadevs/hisilicon.rst       |  41 ++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 925 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    | 236 +++++++
 drivers/dma/hisilicon/meson.build      |   7 +
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 usertools/dpdk-devbind.py              |   6 +-
 10 files changed, 1228 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

Add the basic device probe and remove functions and initial
documentation for new hisilicon DMA drivers. Maintainers update is also
included in this patch.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 MAINTAINERS                            |   5 ++
 doc/guides/dmadevs/hisilicon.rst       |  21 +++++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 119 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    |  24 +++++
 drivers/dma/hisilicon/meson.build      |   7 ++
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 9 files changed, 185 insertions(+)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0e5951f8f1..1567f7b695 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1206,6 +1206,11 @@ M: Conor Walsh <conor.walsh@intel.com>
 F: drivers/dma/ioat/
 F: doc/guides/dmadevs/ioat.rst
 
+Hisilicon DMA
+M: Chengwen Feng <fengchengwen@huawei.com>
+F: drivers/dma/hisilicon
+F: doc/guides/dmadevs/hisilicon.rst
+
 
 RegEx Drivers
 -------------
diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
new file mode 100644
index 0000000000..4cbaac4204
--- /dev/null
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -0,0 +1,21 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2021 HiSilicon Limited.
+
+HISILICON Kunpeng DMA Driver
+============================
+
+Kunpeng SoC has an internal DMA unit which can be used by application to
+accelerate data copies. The DMA PF function supports multiple DMA channels.
+
+
+Supported Kunpeng SoCs
+----------------------
+
+* Kunpeng 920
+
+
+Device Setup
+-------------
+
+Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
+user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 20476039a5..6b04276524 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -13,3 +13,4 @@ an application through DMA API.
 
    idxd
    ioat
+   hisilicon
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 502cc5ceb2..00a45475be 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -86,6 +86,10 @@ New Features
   driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake,
   Skylake and Broadwell. This device driver can be used through the generic dmadev API.
 
+* **Added hisilicon dmadev driver implementation.**
+  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
+  This device driver can be used through the generic dmadev API.
+
 * **Added support to get all MAC addresses of a device.**
 
   Added ``rte_eth_macaddrs_get`` to allow user to retrieve all Ethernet
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
new file mode 100644
index 0000000000..e6fb8a0fc8
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 HiSilicon Limited
+ */
+
+#include <inttypes.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <rte_eal.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_dmadev_pmd.h>
+
+#include "hisi_dmadev.h"
+
+RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
+#define HISI_DMA_LOG(level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s(): " fmt "\n", __func__, ##args)
+#define HISI_DMA_LOG_RAW(hw, level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s %s(): " fmt "\n", (hw)->data->dev_name, \
+		__func__, ##args)
+#define HISI_DMA_DEBUG(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, DEBUG, fmt, ## args)
+#define HISI_DMA_INFO(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, INFO, fmt, ## args)
+#define HISI_DMA_WARN(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, WARNING, fmt, ## args)
+#define HISI_DMA_ERR(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
+
+static uint8_t
+hisi_dma_reg_layout(uint8_t revision)
+{
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		return HISI_DMA_REG_LAYOUT_HIP08;
+	else
+		return HISI_DMA_REG_LAYOUT_INVALID;
+}
+
+static void
+hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
+			     char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function);
+}
+
+static int
+hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
+			uint8_t *out_revision)
+{
+	uint8_t revision;
+	int ret;
+
+	ret = rte_pci_read_config(pci_dev, &revision, 1,
+				  HISI_DMA_PCI_REVISION_ID_REG);
+	if (ret != 1) {
+		HISI_DMA_LOG(ERR, "%s read PCI revision failed!", name);
+		return -EINVAL;
+	}
+	if (hisi_dma_reg_layout(revision) == HISI_DMA_REG_LAYOUT_INVALID) {
+		HISI_DMA_LOG(ERR, "%s revision: 0x%x not supported!",
+			     name, revision);
+		return -EINVAL;
+	}
+
+	*out_revision = revision;
+	return 0;
+}
+
+static int
+hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
+	       struct rte_pci_device *pci_dev)
+{
+	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
+	uint8_t revision;
+	int ret;
+
+	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
+
+	if (pci_dev->mem_resource[2].addr == NULL) {
+		HISI_DMA_LOG(ERR, "%s BAR2 is NULL!\n", name);
+		return -ENODEV;
+	}
+
+	ret = hisi_dma_check_revision(pci_dev, name, &revision);
+	if (ret)
+		return ret;
+	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
+
+	return ret;
+}
+
+static int
+hisi_dma_remove(struct rte_pci_device *pci_dev)
+{
+	RTE_SET_USED(pci_dev);
+	return 0;
+}
+
+static const struct rte_pci_id pci_id_hisi_dma_map[] = {
+	{ RTE_PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HISI_DMA_DEVICE_ID) },
+	{ .vendor_id = 0, }, /* sentinel */
+};
+
+static struct rte_pci_driver hisi_dma_pmd_drv = {
+	.id_table  = pci_id_hisi_dma_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe     = hisi_dma_probe,
+	.remove    = hisi_dma_remove,
+};
+
+RTE_PMD_REGISTER_PCI(dma_hisilicon, hisi_dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(dma_hisilicon, pci_id_hisi_dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(dma_hisilicon, "vfio-pci");
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
new file mode 100644
index 0000000000..114b9dcb5b
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 HiSilicon Limited
+ */
+
+#ifndef HISI_DMADEV_H
+#define HISI_DMADEV_H
+
+#define PCI_VENDOR_ID_HUAWEI			0x19e5
+#define HISI_DMA_DEVICE_ID			0xA122
+#define HISI_DMA_PCI_REVISION_ID_REG		0x08
+#define HISI_DMA_REVISION_HIP08B		0x21
+
+/**
+ * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
+ * they have the same pci device id but with different pci revision.
+ * Unfortunately, they have different register layouts, so the layout
+ * enumerations are defined.
+ */
+enum {
+	HISI_DMA_REG_LAYOUT_INVALID = 0,
+	HISI_DMA_REG_LAYOUT_HIP08
+};
+
+#endif /* HISI_DMADEV_H */
diff --git a/drivers/dma/hisilicon/meson.build b/drivers/dma/hisilicon/meson.build
new file mode 100644
index 0000000000..9e31b47141
--- /dev/null
+++ b/drivers/dma/hisilicon/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 HiSilicon Limited
+
+deps += ['bus_pci', 'dmadev']
+sources = files(
+        'hisi_dmadev.c',
+)
diff --git a/drivers/dma/hisilicon/version.map b/drivers/dma/hisilicon/version.map
new file mode 100644
index 0000000000..c2e0723b4c
--- /dev/null
+++ b/drivers/dma/hisilicon/version.map
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index a69418ce9b..6d6c054f51 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -4,6 +4,7 @@
 drivers = [
         'idxd',
         'ioat',
+        'hisilicon',
         'skeleton',
 ]
 std_deps = ['dmadev']
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add dmadev instances create during the PCI probe, and
destroy them during the PCI remove. Internal structures and HW
definitions was also included.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 ++
 drivers/dma/hisilicon/hisi_dmadev.c | 212 +++++++++++++++++++++++++++-
 drivers/dma/hisilicon/hisi_dmadev.h |  97 +++++++++++++
 3 files changed, 318 insertions(+), 1 deletion(-)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 4cbaac4204..65138a8365 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -19,3 +19,13 @@ Device Setup
 
 Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
 user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once probed successfully, the device will appear as four ``dmadev`` which can be
+accessed using API from the ``rte_dmadev`` library.
+
+The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
+will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
+2nd ``dmadev`` name is "7b:00.0-ch1".
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index e6fb8a0fc8..b8369e7e71 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -6,7 +6,9 @@
 #include <string.h>
 
 #include <rte_bus_pci.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
+#include <rte_io.h>
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
@@ -30,6 +32,141 @@ RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
 #define HISI_DMA_ERR(hw, fmt, args...) \
 		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
 
+static uint32_t
+hisi_dma_queue_base(struct hisi_dma_dev *hw)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08)
+		return HISI_DMA_HIP08_QUEUE_BASE;
+	else
+		return 0;
+}
+
+static void
+hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
+{
+	rte_write32(rte_cpu_to_le_32(val),
+		    (volatile void *)((char *)base + off));
+}
+
+static void
+hisi_dma_write_dev(struct hisi_dma_dev *hw, uint32_t off, uint32_t val)
+{
+	hisi_dma_write_reg(hw->io_base, off, val);
+}
+
+static void
+hisi_dma_write_queue(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t val)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	hisi_dma_write_dev(hw, off, val);
+}
+
+static uint32_t
+hisi_dma_read_reg(void *base, uint32_t off)
+{
+	uint32_t val = rte_read32((volatile void *)((char *)base + off));
+	return rte_le_to_cpu_32(val);
+}
+
+static uint32_t
+hisi_dma_read_dev(struct hisi_dma_dev *hw, uint32_t off)
+{
+	return hisi_dma_read_reg(hw->io_base, off);
+}
+
+static uint32_t
+hisi_dma_read_queue(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return hisi_dma_read_dev(hw, off);
+}
+
+static void
+hisi_dma_update_bit(struct hisi_dma_dev *hw, uint32_t off, uint32_t pos,
+		    bool set)
+{
+	uint32_t tmp = hisi_dma_read_dev(hw, off);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_dev(hw, off, tmp);
+}
+
+static void
+hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
+			  bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
+#define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
+	uint32_t timeout = 0; \
+	while (timeout++ <= (timeout_us)) { \
+		(val) = hisi_dma_read_queue(hw, HISI_DMA_QUEUE_FSM_REG); \
+		if (cond) \
+			break; \
+		rte_delay_us(sleep_us); \
+	} \
+	(cond) ? 0 : -ETIME; \
+})
+
+static int
+hisi_dma_reset_hw(struct hisi_dma_dev *hw)
+{
+#define POLL_SLEEP_US	100
+#define POLL_TIMEOUT_US	10000
+
+	uint32_t tmp;
+	int ret;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, true);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) != HISI_DMA_STATE_RUN,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "disable dma timeout!");
+		return ret;
+	}
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL1_REG,
+				  HISI_DMA_QUEUE_CTRL1_RESET_B, true);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) == HISI_DMA_STATE_IDLE,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "reset dma timeout!");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void
+hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
+{
+	struct hisi_dma_dev hw;
+
+	memset(&hw, 0, sizeof(hw));
+	hw.io_base = pci_bar;
+
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		hisi_dma_update_bit(&hw, HISI_DMA_HIP08_MODE_REG,
+				    HISI_DMA_HIP08_MODE_SEL_B, true);
+}
+
 static uint8_t
 hisi_dma_reg_layout(uint8_t revision)
 {
@@ -49,6 +186,57 @@ hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function);
 }
 
+static void
+hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
+		      uint8_t queue_id, char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x-ch%u",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function, queue_id);
+}
+
+static int
+hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
+		uint8_t revision)
+{
+#define REG_PCI_BAR_INDEX	2
+
+	char name[RTE_DEV_NAME_MAX_LEN];
+	struct rte_dma_dev *dev;
+	struct hisi_dma_dev *hw;
+	int ret;
+
+	hisi_dma_gen_dev_name(pci_dev, queue_id, name, sizeof(name));
+	dev = rte_dma_pmd_allocate(name, pci_dev->device.numa_node,
+				   sizeof(*hw));
+	if (dev == NULL) {
+		HISI_DMA_LOG(ERR, "%s allocate dmadev fail!", name);
+		return -EINVAL;
+	}
+
+	dev->device = &pci_dev->device;
+
+	hw = dev->data->dev_private;
+	hw->data = dev->data;
+	hw->revision = revision;
+	hw->reg_layout = hisi_dma_reg_layout(revision);
+	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
+	hw->queue_id = queue_id;
+
+	ret = hisi_dma_reset_hw(hw);
+	if (ret) {
+		HISI_DMA_LOG(ERR, "%s init device fail!", name);
+		(void)rte_dma_pmd_release(name);
+		return -EIO;
+	}
+
+	dev->state = RTE_DMA_DEV_READY;
+	HISI_DMA_LOG(DEBUG, "%s create dmadev success!", name);
+
+	return 0;
+}
+
 static int
 hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
 			uint8_t *out_revision)
@@ -78,6 +266,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
 	uint8_t revision;
+	uint8_t i;
 	int ret;
 
 	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
@@ -92,13 +281,34 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
+	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		ret = hisi_dma_create(pci_dev, i, revision);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s create dmadev %u failed!",
+				     name, i);
+			break;
+		}
+	}
+
 	return ret;
 }
 
 static int
 hisi_dma_remove(struct rte_pci_device *pci_dev)
 {
-	RTE_SET_USED(pci_dev);
+	char name[RTE_DEV_NAME_MAX_LEN];
+	uint8_t i;
+	int ret;
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		hisi_dma_gen_dev_name(pci_dev, i, name, sizeof(name));
+		ret = rte_dma_pmd_release(name);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 114b9dcb5b..50aaa38b72 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -5,11 +5,24 @@
 #ifndef HISI_DMADEV_H
 #define HISI_DMADEV_H
 
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#define BIT(x)	(1ul << (x))
+#define BITS_PER_LONG	(__SIZEOF_LONG__ * 8)
+#define GENMASK(h, l) \
+		(((~0UL) << (l)) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
+#define BF_SHF(x) (__builtin_ffsll(x) - 1)
+#define FIELD_GET(mask, reg) \
+		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
+#define HISI_DMA_MAX_HW_QUEUES			4
+
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
  * they have the same pci device id but with different pci revision.
@@ -21,4 +34,88 @@ enum {
 	HISI_DMA_REG_LAYOUT_HIP08
 };
 
+/**
+ * Hardware PCI bar register MAP:
+ *
+ *     --------------
+ *     | Misc-reg-0 |
+ *     |            |
+ *     --------------   -> Queue base
+ *     |            |
+ *     | Queue-0    |
+ *     |            |
+ *     --------------   ---
+ *     |            |    ^
+ *     | Queue-1    |   Queue region
+ *     |            |    v
+ *     --------------   ---
+ *     | ...        |
+ *     | Queue-x    |
+ *     | ...        |
+ *     --------------
+ *     | Misc-reg-1 |
+ *     --------------
+ *
+ * As described above, a single queue register is continuous and occupies the
+ * length of queue-region. The global offset for a single queue register is
+ * calculated by:
+ *     offset = queue-base + (queue-id * queue-region) + reg-offset-in-region.
+ *
+ * The first part of queue region is basically the same for HIP08 and later chip
+ * register layouts, therefore, HISI_QUEUE_* registers are defined for it.
+ */
+#define HISI_DMA_QUEUE_SQ_BASE_L_REG		0x0
+#define HISI_DMA_QUEUE_SQ_BASE_H_REG		0x4
+#define HISI_DMA_QUEUE_SQ_DEPTH_REG		0x8
+#define HISI_DMA_QUEUE_SQ_TAIL_REG		0xC
+#define HISI_DMA_QUEUE_CQ_BASE_L_REG		0x10
+#define HISI_DMA_QUEUE_CQ_BASE_H_REG		0x14
+#define HISI_DMA_QUEUE_CQ_DEPTH_REG		0x18
+#define HISI_DMA_QUEUE_CQ_HEAD_REG		0x1C
+#define HISI_DMA_QUEUE_CTRL0_REG		0x20
+#define HISI_DMA_QUEUE_CTRL0_EN_B		0
+#define HISI_DMA_QUEUE_CTRL0_PAUSE_B		4
+#define HISI_DMA_QUEUE_CTRL1_REG		0x24
+#define HISI_DMA_QUEUE_CTRL1_RESET_B		0
+#define HISI_DMA_QUEUE_FSM_REG			0x30
+#define HISI_DMA_QUEUE_FSM_STS_M		GENMASK(3, 0)
+#define HISI_DMA_QUEUE_INT_STATUS_REG		0x40
+#define HISI_DMA_QUEUE_ERR_INT_NUM0_REG		0x84
+#define HISI_DMA_QUEUE_ERR_INT_NUM1_REG		0x88
+#define HISI_DMA_QUEUE_ERR_INT_NUM2_REG		0x8C
+#define HISI_DMA_QUEUE_REGION_SIZE		0x100
+
+/**
+ * HiSilicon IP08 DMA register and field define:
+ */
+#define HISI_DMA_HIP08_QUEUE_BASE			0x0
+#define HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B		2
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_REG		0x44
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_M			GENMASK(14, 0)
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG		0x90
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG		0x94
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG		0x98
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG		0x48
+#define HISI_DMA_HIP08_MODE_REG				0x217C
+#define HISI_DMA_HIP08_MODE_SEL_B			0
+#define HISI_DMA_HIP08_DUMP_START_REG			0x2000
+#define HISI_DMA_HIP08_DUMP_END_REG			0x2280
+
+/**
+ * In fact, there are multiple states, but it need to pay attention to
+ * the following two states for the driver:
+ */
+enum {
+	HISI_DMA_STATE_IDLE = 0,
+	HISI_DMA_STATE_RUN,
+};
+
+struct hisi_dma_dev {
+	struct rte_dma_dev_data *data;
+	uint8_t revision; /**< PCI revision. */
+	uint8_t reg_layout; /**< hardware register layout. */
+	void *io_base;
+	uint8_t queue_id; /**< hardware DMA queue index. */
+};
+
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add control path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 +
 drivers/dma/hisilicon/hisi_dmadev.c | 385 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  99 +++++++
 3 files changed, 494 insertions(+)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 65138a8365..24bae86bdc 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -29,3 +29,13 @@ accessed using API from the ``rte_dmadev`` library.
 The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
 will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
 2nd ``dmadev`` name is "7b:00.0-ch1".
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Kunpeng DMA configuration requirements:
+
+* ``ring_size`` must be a power of two, between 32 and 8192.
+* Only one ``vchan`` is supported per ``dmadev``.
+* Silent mode is not supported.
+* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM``.
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index b8369e7e71..bcdcf4de4b 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -10,6 +10,8 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
 
@@ -41,6 +43,14 @@ hisi_dma_queue_base(struct hisi_dma_dev *hw)
 		return 0;
 }
 
+static volatile void *
+hisi_dma_queue_regaddr(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return (volatile void *)((char *)hw->io_base + off);
+}
+
 static void
 hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
 {
@@ -103,6 +113,15 @@ hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
 	hisi_dma_write_queue(hw, qoff, tmp);
 }
 
+static void
+hisi_dma_update_queue_mbit(struct hisi_dma_dev *hw, uint32_t qoff,
+			   uint32_t mask, bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
 #define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
 	uint32_t timeout = 0; \
 	while (timeout++ <= (timeout_us)) { \
@@ -154,6 +173,45 @@ hisi_dma_reset_hw(struct hisi_dma_dev *hw)
 	return 0;
 }
 
+static void
+hisi_dma_init_hw(struct hisi_dma_dev *hw)
+{
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_L_REG,
+			     lower_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_H_REG,
+			     upper_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_L_REG,
+			     lower_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_H_REG,
+			     upper_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_DEPTH_REG,
+			     hw->sq_depth_mask);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_DEPTH_REG, hw->cq_depth - 1);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM0_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM1_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM2_REG, 0);
+
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG,
+				     0);
+		hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B, false);
+		hisi_dma_update_queue_mbit(hw, HISI_DMA_QUEUE_INT_STATUS_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+		hisi_dma_update_queue_mbit(hw,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+	}
+}
+
 static void
 hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
 {
@@ -176,6 +234,301 @@ hisi_dma_reg_layout(uint8_t revision)
 		return HISI_DMA_REG_LAYOUT_INVALID;
 }
 
+static void
+hisi_dma_zero_iomem(struct hisi_dma_dev *hw)
+{
+	memset(hw->iomz->addr, 0, hw->iomz_sz);
+}
+
+static int
+hisi_dma_alloc_iomem(struct hisi_dma_dev *hw, uint16_t ring_size,
+		     const char *dev_name)
+{
+	uint32_t sq_size = sizeof(struct hisi_dma_sqe) * ring_size;
+	uint32_t cq_size = sizeof(struct hisi_dma_cqe) *
+			   (ring_size + HISI_DMA_CQ_RESERVED);
+	uint32_t status_size = sizeof(uint16_t) * ring_size;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	const struct rte_memzone *iomz;
+	uint32_t total_size;
+
+	sq_size = RTE_CACHE_LINE_ROUNDUP(sq_size);
+	cq_size = RTE_CACHE_LINE_ROUNDUP(cq_size);
+	status_size = RTE_CACHE_LINE_ROUNDUP(status_size);
+	total_size = sq_size + cq_size + status_size;
+
+	(void)snprintf(mz_name, sizeof(mz_name), "hisi_dma:%s", dev_name);
+	iomz = rte_memzone_reserve(mz_name, total_size, hw->data->numa_node,
+				   RTE_MEMZONE_IOVA_CONTIG);
+	if (iomz == NULL) {
+		HISI_DMA_ERR(hw, "malloc %s iomem fail!", mz_name);
+		return -ENOMEM;
+	}
+
+	hw->iomz = iomz;
+	hw->iomz_sz = total_size;
+	hw->sqe = iomz->addr;
+	hw->cqe = (void *)((char *)iomz->addr + sq_size);
+	hw->status = (void *)((char *)iomz->addr + sq_size + cq_size);
+	hw->sqe_iova = iomz->iova;
+	hw->cqe_iova = iomz->iova + sq_size;
+	hw->sq_depth_mask = ring_size - 1;
+	hw->cq_depth = ring_size + HISI_DMA_CQ_RESERVED;
+	hisi_dma_zero_iomem(hw);
+
+	return 0;
+}
+
+static void
+hisi_dma_free_iomem(struct hisi_dma_dev *hw)
+{
+	if (hw->iomz != NULL)
+		rte_memzone_free(hw->iomz);
+
+	hw->iomz = NULL;
+	hw->sqe = NULL;
+	hw->cqe = NULL;
+	hw->status = NULL;
+	hw->sqe_iova = 0;
+	hw->cqe_iova = 0;
+	hw->sq_depth_mask = 0;
+	hw->cq_depth = 0;
+}
+
+static int
+hisi_dma_info_get(const struct rte_dma_dev *dev,
+		  struct rte_dma_info *dev_info,
+		  uint32_t info_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info_sz);
+
+	dev_info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM |
+			     RTE_DMA_CAPA_OPS_COPY;
+	dev_info->max_vchans = 1;
+	dev_info->max_desc = HISI_DMA_MAX_DESC_NUM;
+	dev_info->min_desc = HISI_DMA_MIN_DESC_NUM;
+
+	return 0;
+}
+
+static int
+hisi_dma_configure(struct rte_dma_dev *dev,
+		   const struct rte_dma_conf *conf,
+		   uint32_t conf_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(conf);
+	RTE_SET_USED(conf_sz);
+	return 0;
+}
+
+static int
+hisi_dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan,
+		     const struct rte_dma_vchan_conf *conf,
+		     uint32_t conf_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+	int ret;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(conf_sz);
+
+	if (!rte_is_power_of_2(conf->nb_desc)) {
+		HISI_DMA_ERR(hw, "Number of desc must be power of 2!");
+		return -EINVAL;
+	}
+
+	hisi_dma_free_iomem(hw);
+	ret = hisi_dma_alloc_iomem(hw, conf->nb_desc, dev->data->dev_name);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int
+hisi_dma_start(struct rte_dma_dev *dev)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	if (hw->iomz == NULL) {
+		HISI_DMA_ERR(hw, "Vchan was not setup, start fail!\n");
+		return -EINVAL;
+	}
+
+	/* Reset the dmadev to a known state, include:
+	 *   1) zero iomem, also include status fields.
+	 *   2) init hardware register.
+	 *   3) init index values to zero.
+	 *   4) init running statistics.
+	 */
+	hisi_dma_zero_iomem(hw);
+	hisi_dma_init_hw(hw);
+	hw->ridx = 0;
+	hw->cridx = 0;
+	hw->sq_head = 0;
+	hw->sq_tail = 0;
+	hw->cq_sq_head = 0;
+	hw->cq_head = 0;
+	hw->cqs_completed = 0;
+	hw->cqe_vld = 1;
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, true);
+
+	return 0;
+}
+
+static int
+hisi_dma_stop(struct rte_dma_dev *dev)
+{
+	return hisi_dma_reset_hw(dev->data->dev_private);
+}
+
+static int
+hisi_dma_close(struct rte_dma_dev *dev)
+{
+	/* The dmadev already stopped */
+	hisi_dma_free_iomem(dev->data->dev_private);
+	return 0;
+}
+
+static int
+hisi_dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan,
+		   struct rte_dma_stats *stats,
+		   uint32_t stats_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(stats_sz);
+	stats->submitted = hw->submitted;
+	stats->completed = hw->completed;
+	stats->errors = hw->errors;
+
+	return 0;
+}
+
+static int
+hisi_dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	return 0;
+}
+
+static void
+hisi_dma_get_dump_range(struct hisi_dma_dev *hw, uint32_t *start, uint32_t *end)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		*start = HISI_DMA_HIP08_DUMP_START_REG;
+		*end = HISI_DMA_HIP08_DUMP_END_REG;
+	} else {
+		*start = 0;
+		*end = 0;
+	}
+}
+
+static void
+hisi_dma_dump_common(struct hisi_dma_dev *hw, FILE *f)
+{
+#define DUMP_REGNUM_PER_LINE	4
+
+	uint32_t start, end;
+	uint32_t cnt, i;
+
+	hisi_dma_get_dump_range(hw, &start, &end);
+
+	(void)fprintf(f, "    common-register:\n");
+
+	cnt = 0;
+	for (i = start; i <= end; i += sizeof(uint32_t)) {
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "      [%4x]:", i);
+		(void)fprintf(f, " 0x%08x", hisi_dma_read_dev(hw, i));
+		cnt++;
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "\n");
+	}
+	if (cnt % DUMP_REGNUM_PER_LINE)
+		(void)fprintf(f, "\n");
+}
+
+static void
+hisi_dma_dump_read_queue(struct hisi_dma_dev *hw, uint32_t qoff,
+			 char *buffer, int max_sz)
+{
+	memset(buffer, 0, max_sz);
+
+	/* Address-related registers are not printed for security reasons. */
+	if (qoff == HISI_DMA_QUEUE_SQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_SQ_BASE_H_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_H_REG) {
+		(void)snprintf(buffer, max_sz, "**********");
+		return;
+	}
+
+	(void)snprintf(buffer, max_sz, "0x%08x", hisi_dma_read_queue(hw, qoff));
+}
+
+static void
+hisi_dma_dump_queue(struct hisi_dma_dev *hw, FILE *f)
+{
+#define REG_FMT_LEN	32
+	char buf[REG_FMT_LEN] = { 0 };
+	uint32_t i;
+
+	(void)fprintf(f, "    queue-register:\n");
+	for (i = 0; i < HISI_DMA_QUEUE_REGION_SIZE; ) {
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, "      [%2x]: %s", i, buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s\n", buf);
+		i += sizeof(uint32_t);
+	}
+}
+
+static int
+hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	(void)fprintf(f,
+		"    revision: 0x%x queue_id: %u ring_size: %u\n"
+		"    ridx: %u cridx: %u\n"
+		"    sq_head: %u sq_tail: %u cq_sq_head: %u\n"
+		"    cq_head: %u cqs_completed: %u cqe_vld: %u\n"
+		"    submitted: %" PRIu64 " completed: %" PRIu64 " errors %"
+		PRIu64"\n",
+		hw->revision, hw->queue_id,
+		hw->sq_depth_mask > 0 ? hw->sq_depth_mask + 1 : 0,
+		hw->ridx, hw->cridx,
+		hw->sq_head, hw->sq_tail, hw->cq_sq_head,
+		hw->cq_head, hw->cqs_completed, hw->cqe_vld,
+		hw->submitted, hw->completed, hw->errors);
+	hisi_dma_dump_queue(hw, f);
+	hisi_dma_dump_common(hw, f);
+
+	return 0;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -196,6 +549,33 @@ hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function, queue_id);
 }
 
+/**
+ * Hardware queue state machine:
+ *
+ *   -----------  dmadev_create	  ------------------
+ *   | Unknown | ---------------> |      IDLE      |
+ *   -----------                  ------------------
+ *                                   ^          |
+ *                                   |          |dev_start
+ *                           dev_stop|          |
+ *                                   |          v
+ *                                ------------------
+ *                                |      RUN       |
+ *                                ------------------
+ *
+ */
+static const struct rte_dma_dev_ops hisi_dmadev_ops = {
+	.dev_info_get     = hisi_dma_info_get,
+	.dev_configure    = hisi_dma_configure,
+	.dev_start        = hisi_dma_start,
+	.dev_stop         = hisi_dma_stop,
+	.dev_close        = hisi_dma_close,
+	.vchan_setup      = hisi_dma_vchan_setup,
+	.stats_get        = hisi_dma_stats_get,
+	.stats_reset      = hisi_dma_stats_reset,
+	.dev_dump         = hisi_dma_dump,
+};
+
 static int
 hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 		uint8_t revision)
@@ -216,6 +596,7 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	}
 
 	dev->device = &pci_dev->device;
+	dev->dev_ops = &hisi_dmadev_ops;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
@@ -223,6 +604,10 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->reg_layout = hisi_dma_reg_layout(revision);
 	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
 	hw->queue_id = queue_id;
+	hw->sq_tail_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_SQ_TAIL_REG);
+	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
 	ret = hisi_dma_reset_hw(hw);
 	if (ret) {
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 50aaa38b72..dd0315cd31 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -16,12 +16,17 @@
 #define FIELD_GET(mask, reg) \
 		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
 
+#define lower_32_bits(x) ((uint32_t)(x))
+#define upper_32_bits(x) ((uint32_t)(((x) >> 16) >> 16))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
 #define HISI_DMA_MAX_HW_QUEUES			4
+#define HISI_DMA_MAX_DESC_NUM			8192
+#define HISI_DMA_MIN_DESC_NUM			32
 
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
@@ -110,12 +115,106 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * After scanning the CQ array, the CQ head register needs to be updated.
+ * Updating the register involves write memory barrier operations.
+ * Here use the following method to reduce WMB operations:
+ *   a) malloc more CQEs, which correspond to the macro HISI_DMA_CQ_RESERVED.
+ *   b) update the CQ head register after accumulated number of completed CQs
+ *      is greater than or equal to HISI_DMA_CQ_RESERVED.
+ */
+#define HISI_DMA_CQ_RESERVED		64
+
+struct hisi_dma_sqe {
+	uint32_t dw0;
+#define SQE_FENCE_FLAG	BIT(10)
+#define SQE_OPCODE_M2M	0x4
+	uint32_t dw1;
+	uint32_t dw2;
+	uint32_t length;
+	uint64_t src_addr;
+	uint64_t dst_addr;
+};
+
+struct hisi_dma_cqe {
+	uint64_t rsv;
+	uint64_t misc;
+#define CQE_SQ_HEAD_MASK	GENMASK(15, 0)
+#define CQE_VALID_B		BIT(48)
+#define CQE_STATUS_MASK		GENMASK(63, 49)
+};
+
 struct hisi_dma_dev {
+	struct hisi_dma_sqe *sqe;
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t *status; /* the completion status array of SQEs. */
+
+	volatile void *sq_tail_reg; /**< register address for doorbell. */
+	volatile void *cq_head_reg; /**< register address for answer CQ. */
+
+	uint16_t sq_depth_mask; /**< SQ depth - 1, the SQ depth is power of 2 */
+	uint16_t cq_depth; /* CQ depth */
+
+	uint16_t ridx; /**< ring index which will assign to the next request. */
+	/** ring index which returned by hisi_dmadev_completed APIs. */
+	uint16_t cridx;
+
+	/**
+	 * SQE array management fields:
+	 *
+	 *  -----------------------------------------------------
+	 *  | SQE0 | SQE1 | SQE2 |   ...  | SQEx | ... | SQEn-1 |
+	 *  -----------------------------------------------------
+	 *     ^             ^               ^
+	 *     |             |               |
+	 *   sq_head     cq_sq_head       sq_tail
+	 *
+	 *  sq_head: index to the oldest completed request, this filed was
+	 *           updated by hisi_dmadev_completed* APIs.
+	 *  sq_tail: index of the next new request, this field was updated by
+	 *           hisi_dmadev_copy API.
+	 *  cq_sq_head: next index of index that has been completed by hardware,
+	 *              this filed was updated by hisi_dmadev_completed* APIs.
+	 *
+	 *  [sq_head, cq_sq_head): the SQEs that hardware already completed.
+	 *  [cq_sq_head, sq_tail): the SQEs that hardware processing.
+	 */
+	uint16_t sq_head;
+	uint16_t sq_tail;
+	uint16_t cq_sq_head;
+	/**
+	 * The driver scans the CQE array, if the valid bit changes, the CQE is
+	 * considered valid.
+	 * Note: One CQE is corresponding to one or several SQEs, e.g. app
+	 *       submits two copy requests, the hardware processes the two SQEs,
+	 *       but it may write back only one CQE and the CQE's sq_head field
+	 *       indicates the index of the second copy request in the SQE
+	 *       array.
+	 */
+	uint16_t cq_head; /**< CQ index for next scans. */
+	/** accumulated number of completed CQs
+	 * @see HISI_DMA_CQ_RESERVED
+	 */
+	uint16_t cqs_completed;
+	uint8_t cqe_vld; /**< valid bit for CQE, will change for every round. */
+
+	uint64_t submitted;
+	uint64_t completed;
+	uint64_t errors;
+
+	/**
+	 * The following fields are not accessed in the I/O path, so they are
+	 * placed at the end.
+	 */
 	struct rte_dma_dev_data *data;
 	uint8_t revision; /**< PCI revision. */
 	uint8_t reg_layout; /**< hardware register layout. */
 	void *io_base;
 	uint8_t queue_id; /**< hardware DMA queue index. */
+	const struct rte_memzone *iomz;
+	uint32_t iomz_sz;
+	rte_iova_t sqe_iova;
+	rte_iova_t cqe_iova;
 };
 
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data path functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (2 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add data path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 206 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  16 +++
 2 files changed, 222 insertions(+)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index bcdcf4de4b..d03967cae3 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -529,6 +529,206 @@ hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
 	return 0;
 }
 
+static int
+hisi_dma_copy(void *dev_private, uint16_t vchan,
+		 rte_iova_t src, rte_iova_t dst,
+		 uint32_t length, uint64_t flags)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	struct hisi_dma_sqe *sqe = &hw->sqe[hw->sq_tail];
+
+	RTE_SET_USED(vchan);
+
+	if (((hw->sq_tail + 1) & hw->sq_depth_mask) == hw->sq_head)
+		return -ENOSPC;
+
+	sqe->dw0 = rte_cpu_to_le_32(SQE_OPCODE_M2M);
+	sqe->dw1 = 0;
+	sqe->dw2 = 0;
+	sqe->length = rte_cpu_to_le_32(length);
+	sqe->src_addr = rte_cpu_to_le_64(src);
+	sqe->dst_addr = rte_cpu_to_le_64(dst);
+	hw->sq_tail = (hw->sq_tail + 1) & hw->sq_depth_mask;
+	hw->submitted++;
+
+	if (flags & RTE_DMA_OP_FLAG_FENCE)
+		sqe->dw0 |= rte_cpu_to_le_32(SQE_FENCE_FLAG);
+	if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+		rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return hw->ridx++;
+}
+
+static int
+hisi_dma_submit(void *dev_private, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev_private;
+
+	RTE_SET_USED(vchan);
+	rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return 0;
+}
+
+static inline void
+hisi_dma_scan_cq(struct hisi_dma_dev *hw)
+{
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t csq_head = hw->cq_sq_head;
+	uint16_t cq_head = hw->cq_head;
+	uint16_t count = 0;
+	uint64_t misc;
+
+	while (true) {
+		cqe = &hw->cqe[cq_head];
+		misc = cqe->misc;
+		misc = rte_le_to_cpu_64(misc);
+		if (FIELD_GET(CQE_VALID_B, misc) != hw->cqe_vld)
+			break;
+
+		csq_head = FIELD_GET(CQE_SQ_HEAD_MASK, misc);
+		if (unlikely(misc & CQE_STATUS_MASK))
+			hw->status[csq_head] = FIELD_GET(CQE_STATUS_MASK,
+							 misc);
+
+		count++;
+		cq_head++;
+		if (cq_head == hw->cq_depth) {
+			hw->cqe_vld = !hw->cqe_vld;
+			cq_head = 0;
+		}
+	}
+
+	if (count == 0)
+		return;
+
+	hw->cq_head = cq_head;
+	hw->cq_sq_head = (csq_head + 1) & hw->sq_depth_mask;
+	hw->cqs_completed += count;
+	if (hw->cqs_completed >= HISI_DMA_CQ_RESERVED) {
+		rte_write32(rte_cpu_to_le_32(cq_head), hw->cq_head_reg);
+		hw->cqs_completed = 0;
+	}
+}
+
+static inline uint16_t
+hisi_dma_calc_cpls(struct hisi_dma_dev *hw, const uint16_t nb_cpls)
+{
+	uint16_t cpl_num;
+
+	if (hw->cq_sq_head >= hw->sq_head)
+		cpl_num = hw->cq_sq_head - hw->sq_head;
+	else
+		cpl_num = hw->sq_depth_mask + 1 - hw->sq_head + hw->cq_sq_head;
+
+	if (cpl_num > nb_cpls)
+		cpl_num = nb_cpls;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_completed(void *dev_private,
+		   uint16_t vchan, const uint16_t nb_cpls,
+		   uint16_t *last_idx, bool *has_error)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		if (hw->status[sq_head]) {
+			*has_error = true;
+			break;
+		}
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (i > 0) {
+		hw->cridx += i;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += i;
+
+	return i;
+}
+
+static enum rte_dma_status_code
+hisi_dma_convert_status(uint16_t status)
+{
+	switch (status) {
+	case HISI_DMA_STATUS_SUCCESS:
+		return RTE_DMA_STATUS_SUCCESSFUL;
+	case HISI_DMA_STATUS_INVALID_OPCODE:
+		return RTE_DMA_STATUS_INVALID_OPCODE;
+	case HISI_DMA_STATUS_INVALID_LENGTH:
+		return RTE_DMA_STATUS_INVALID_LENGTH;
+	case HISI_DMA_STATUS_USER_ABORT:
+		return RTE_DMA_STATUS_USER_ABORT;
+	case HISI_DMA_STATUS_REMOTE_READ_ERROR:
+	case HISI_DMA_STATUS_AXI_READ_ERROR:
+		return RTE_DMA_STATUS_BUS_READ_ERROR;
+	case HISI_DMA_STATUS_AXI_WRITE_ERROR:
+		return RTE_DMA_STATUS_BUS_WRITE_ERROR;
+	case HISI_DMA_STATUS_DATA_POISON:
+	case HISI_DMA_STATUS_REMOTE_DATA_POISION:
+		return RTE_DMA_STATUS_DATA_POISION;
+	case HISI_DMA_STATUS_SQE_READ_ERROR:
+	case HISI_DMA_STATUS_SQE_READ_POISION:
+		return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR;
+	case HISI_DMA_STATUS_LINK_DOWN_ERROR:
+		return RTE_DMA_STATUS_DEV_LINK_ERROR;
+	default:
+		return RTE_DMA_STATUS_ERROR_UNKNOWN;
+	}
+}
+
+static uint16_t
+hisi_dma_completed_status(void *dev_private,
+			  uint16_t vchan, const uint16_t nb_cpls,
+			  uint16_t *last_idx, enum rte_dma_status_code *status)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		status[i] = hisi_dma_convert_status(hw->status[sq_head]);
+		hw->errors += !!status[i];
+		hw->status[sq_head] = HISI_DMA_STATUS_SUCCESS;
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (likely(cpl_num > 0)) {
+		hw->cridx += cpl_num;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += cpl_num;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_burst_capacity(const void *dev_private, uint16_t vchan)
+{
+	const struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t sq_tail = hw->sq_tail;
+
+	RTE_SET_USED(vchan);
+
+	return (sq_tail >= sq_head) ? hw->sq_depth_mask - sq_tail + sq_head :
+				      sq_head - 1 - sq_tail;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -597,6 +797,12 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 
 	dev->device = &pci_dev->device;
 	dev->dev_ops = &hisi_dmadev_ops;
+	dev->fp_obj->dev_private = dev->data->dev_private;
+	dev->fp_obj->copy = hisi_dma_copy;
+	dev->fp_obj->submit = hisi_dma_submit;
+	dev->fp_obj->completed = hisi_dma_completed;
+	dev->fp_obj->completed_status = hisi_dma_completed_status;
+	dev->fp_obj->burst_capacity = hisi_dma_burst_capacity;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index dd0315cd31..12e209c86e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -115,6 +115,22 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * Hardware complete status define:
+ */
+#define HISI_DMA_STATUS_SUCCESS			0x0
+#define HISI_DMA_STATUS_INVALID_OPCODE		0x1
+#define HISI_DMA_STATUS_INVALID_LENGTH		0x2
+#define HISI_DMA_STATUS_USER_ABORT		0x4
+#define HISI_DMA_STATUS_REMOTE_READ_ERROR	0x10
+#define HISI_DMA_STATUS_AXI_READ_ERROR		0x20
+#define HISI_DMA_STATUS_AXI_WRITE_ERROR		0x40
+#define HISI_DMA_STATUS_DATA_POISON		0x80
+#define HISI_DMA_STATUS_SQE_READ_ERROR		0x100
+#define HISI_DMA_STATUS_SQE_READ_POISION	0x200
+#define HISI_DMA_STATUS_REMOTE_DATA_POISION	0x400
+#define HISI_DMA_STATUS_LINK_DOWN_ERROR		0x800
+
 /**
  * After scanning the CQ array, the CQ head register needs to be updated.
  * Updating the register involves write memory barrier operations.
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (3 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add multi-process support for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index d03967cae3..05066b4d0e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -392,8 +392,10 @@ hisi_dma_stop(struct rte_dma_dev *dev)
 static int
 hisi_dma_close(struct rte_dma_dev *dev)
 {
-	/* The dmadev already stopped */
-	hisi_dma_free_iomem(dev->data->dev_private);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		/* The dmadev already stopped */
+		hisi_dma_free_iomem(dev->data->dev_private);
+	}
 	return 0;
 }
 
@@ -815,11 +817,13 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
 						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
-	ret = hisi_dma_reset_hw(hw);
-	if (ret) {
-		HISI_DMA_LOG(ERR, "%s init device fail!", name);
-		(void)rte_dma_pmd_release(name);
-		return -EIO;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = hisi_dma_reset_hw(hw);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s init device fail!", name);
+			(void)rte_dma_pmd_release(name);
+			return -EIO;
+		}
 	}
 
 	dev->state = RTE_DMA_DEV_READY;
@@ -872,7 +876,8 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
-	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
 
 	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
 		ret = hisi_dma_create(pci_dev, i, revision);
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (4 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

add Kunpeng DMA device ID to dmadev category.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 usertools/dpdk-devbind.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index bb00f43702..a74a68ed82 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -68,10 +68,14 @@
 intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
                  'SVendor': None, 'SDevice': None}
 
+hisilicon_dma = {'Class': '08', 'Vendor': '19e5', 'Device': 'a122',
+                 'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx]
+dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
+               hisilicon_dma]
 eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (5 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
@ 2021-11-02 12:37 ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
                     ` (6 more replies)
  6 siblings, 7 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch set add hisilicon DMA driver.

Chengwen Feng (6):
  dma/hisilicon: add device probe and remove functions
  dma/hisilicon: add dmadev instances create and destroy
  dma/hisilicon: add control path functions
  dma/hisilicon: add data path functions
  dma/hisilicon: support multi-process
  devbind: add Kunpeng DMA to dmadev category

---
v2: fix compile error on non-Linux OS.

 MAINTAINERS                            |   5 +
 doc/guides/dmadevs/hisilicon.rst       |  41 ++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 925 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    | 236 +++++++
 drivers/dma/hisilicon/meson.build      |  19 +
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 usertools/dpdk-devbind.py              |   6 +-
 10 files changed, 1240 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-07 17:28     ` Thomas Monjalon
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
                     ` (5 subsequent siblings)
  6 siblings, 1 reply; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

Add the basic device probe and remove functions and initial
documentation for new hisilicon DMA drivers. Maintainers update is also
included in this patch.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 MAINTAINERS                            |   5 ++
 doc/guides/dmadevs/hisilicon.rst       |  21 +++++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 119 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    |  24 +++++
 drivers/dma/hisilicon/meson.build      |  19 ++++
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 9 files changed, 197 insertions(+)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0e5951f8f1..1567f7b695 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1206,6 +1206,11 @@ M: Conor Walsh <conor.walsh@intel.com>
 F: drivers/dma/ioat/
 F: doc/guides/dmadevs/ioat.rst
 
+Hisilicon DMA
+M: Chengwen Feng <fengchengwen@huawei.com>
+F: drivers/dma/hisilicon
+F: doc/guides/dmadevs/hisilicon.rst
+
 
 RegEx Drivers
 -------------
diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
new file mode 100644
index 0000000000..4cbaac4204
--- /dev/null
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -0,0 +1,21 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2021 HiSilicon Limited.
+
+HISILICON Kunpeng DMA Driver
+============================
+
+Kunpeng SoC has an internal DMA unit which can be used by application to
+accelerate data copies. The DMA PF function supports multiple DMA channels.
+
+
+Supported Kunpeng SoCs
+----------------------
+
+* Kunpeng 920
+
+
+Device Setup
+-------------
+
+Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
+user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 20476039a5..6b04276524 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -13,3 +13,4 @@ an application through DMA API.
 
    idxd
    ioat
+   hisilicon
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 502cc5ceb2..00a45475be 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -86,6 +86,10 @@ New Features
   driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake,
   Skylake and Broadwell. This device driver can be used through the generic dmadev API.
 
+* **Added hisilicon dmadev driver implementation.**
+  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
+  This device driver can be used through the generic dmadev API.
+
 * **Added support to get all MAC addresses of a device.**
 
   Added ``rte_eth_macaddrs_get`` to allow user to retrieve all Ethernet
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
new file mode 100644
index 0000000000..e6fb8a0fc8
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 HiSilicon Limited
+ */
+
+#include <inttypes.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <rte_eal.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_dmadev_pmd.h>
+
+#include "hisi_dmadev.h"
+
+RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
+#define HISI_DMA_LOG(level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s(): " fmt "\n", __func__, ##args)
+#define HISI_DMA_LOG_RAW(hw, level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s %s(): " fmt "\n", (hw)->data->dev_name, \
+		__func__, ##args)
+#define HISI_DMA_DEBUG(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, DEBUG, fmt, ## args)
+#define HISI_DMA_INFO(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, INFO, fmt, ## args)
+#define HISI_DMA_WARN(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, WARNING, fmt, ## args)
+#define HISI_DMA_ERR(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
+
+static uint8_t
+hisi_dma_reg_layout(uint8_t revision)
+{
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		return HISI_DMA_REG_LAYOUT_HIP08;
+	else
+		return HISI_DMA_REG_LAYOUT_INVALID;
+}
+
+static void
+hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
+			     char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function);
+}
+
+static int
+hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
+			uint8_t *out_revision)
+{
+	uint8_t revision;
+	int ret;
+
+	ret = rte_pci_read_config(pci_dev, &revision, 1,
+				  HISI_DMA_PCI_REVISION_ID_REG);
+	if (ret != 1) {
+		HISI_DMA_LOG(ERR, "%s read PCI revision failed!", name);
+		return -EINVAL;
+	}
+	if (hisi_dma_reg_layout(revision) == HISI_DMA_REG_LAYOUT_INVALID) {
+		HISI_DMA_LOG(ERR, "%s revision: 0x%x not supported!",
+			     name, revision);
+		return -EINVAL;
+	}
+
+	*out_revision = revision;
+	return 0;
+}
+
+static int
+hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
+	       struct rte_pci_device *pci_dev)
+{
+	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
+	uint8_t revision;
+	int ret;
+
+	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
+
+	if (pci_dev->mem_resource[2].addr == NULL) {
+		HISI_DMA_LOG(ERR, "%s BAR2 is NULL!\n", name);
+		return -ENODEV;
+	}
+
+	ret = hisi_dma_check_revision(pci_dev, name, &revision);
+	if (ret)
+		return ret;
+	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
+
+	return ret;
+}
+
+static int
+hisi_dma_remove(struct rte_pci_device *pci_dev)
+{
+	RTE_SET_USED(pci_dev);
+	return 0;
+}
+
+static const struct rte_pci_id pci_id_hisi_dma_map[] = {
+	{ RTE_PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HISI_DMA_DEVICE_ID) },
+	{ .vendor_id = 0, }, /* sentinel */
+};
+
+static struct rte_pci_driver hisi_dma_pmd_drv = {
+	.id_table  = pci_id_hisi_dma_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe     = hisi_dma_probe,
+	.remove    = hisi_dma_remove,
+};
+
+RTE_PMD_REGISTER_PCI(dma_hisilicon, hisi_dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(dma_hisilicon, pci_id_hisi_dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(dma_hisilicon, "vfio-pci");
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
new file mode 100644
index 0000000000..114b9dcb5b
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 HiSilicon Limited
+ */
+
+#ifndef HISI_DMADEV_H
+#define HISI_DMADEV_H
+
+#define PCI_VENDOR_ID_HUAWEI			0x19e5
+#define HISI_DMA_DEVICE_ID			0xA122
+#define HISI_DMA_PCI_REVISION_ID_REG		0x08
+#define HISI_DMA_REVISION_HIP08B		0x21
+
+/**
+ * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
+ * they have the same pci device id but with different pci revision.
+ * Unfortunately, they have different register layouts, so the layout
+ * enumerations are defined.
+ */
+enum {
+	HISI_DMA_REG_LAYOUT_INVALID = 0,
+	HISI_DMA_REG_LAYOUT_HIP08
+};
+
+#endif /* HISI_DMADEV_H */
diff --git a/drivers/dma/hisilicon/meson.build b/drivers/dma/hisilicon/meson.build
new file mode 100644
index 0000000000..0786772442
--- /dev/null
+++ b/drivers/dma/hisilicon/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 HiSilicon Limited
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+if (arch_subdir != 'x86' and arch_subdir != 'arm') or (not dpdk_conf.get('RTE_ARCH_64'))
+    build = false
+    reason = 'only supported on x86_64 and aarch64'
+    subdir_done()
+endif
+
+deps += ['bus_pci', 'dmadev']
+sources = files(
+        'hisi_dmadev.c',
+)
diff --git a/drivers/dma/hisilicon/version.map b/drivers/dma/hisilicon/version.map
new file mode 100644
index 0000000000..c2e0723b4c
--- /dev/null
+++ b/drivers/dma/hisilicon/version.map
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index a69418ce9b..6d6c054f51 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -4,6 +4,7 @@
 drivers = [
         'idxd',
         'ioat',
+        'hisilicon',
         'skeleton',
 ]
 std_deps = ['dmadev']
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-11-07 17:28     ` Thomas Monjalon
  0 siblings, 0 replies; 16+ messages in thread
From: Thomas Monjalon @ 2021-11-07 17:28 UTC (permalink / raw)
  To: Chengwen Feng; +Cc: dev

02/11/2021 13:37, Chengwen Feng:
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> +Hisilicon DMA
> +M: Chengwen Feng <fengchengwen@huawei.com>
> +F: drivers/dma/hisilicon

Final slash is missing (and it is mandatory for the parsing).

> --- a/doc/guides/dmadevs/index.rst
> +++ b/doc/guides/dmadevs/index.rst
> @@ -13,3 +13,4 @@ an application through DMA API.
>  
>     idxd
>     ioat
> +   hisilicon

Let's keep an alphabetical order.

> --- a/doc/guides/rel_notes/release_21_11.rst
> +++ b/doc/guides/rel_notes/release_21_11.rst
> +* **Added hisilicon dmadev driver implementation.**

HiSilicon DMA driver
blank line missing

> +  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
> +  This device driver can be used through the generic dmadev API.

> --- a/drivers/dma/meson.build
> +++ b/drivers/dma/meson.build
> @@ -4,6 +4,7 @@
>  drivers = [
>          'idxd',
>          'ioat',
> +        'hisilicon',
>          'skeleton',
>  ]

alphabetical order



^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
                     ` (4 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add dmadev instances create during the PCI probe, and
destroy them during the PCI remove. Internal structures and HW
definitions was also included.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 ++
 drivers/dma/hisilicon/hisi_dmadev.c | 212 +++++++++++++++++++++++++++-
 drivers/dma/hisilicon/hisi_dmadev.h |  97 +++++++++++++
 3 files changed, 318 insertions(+), 1 deletion(-)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 4cbaac4204..65138a8365 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -19,3 +19,13 @@ Device Setup
 
 Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
 user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once probed successfully, the device will appear as four ``dmadev`` which can be
+accessed using API from the ``rte_dmadev`` library.
+
+The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
+will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
+2nd ``dmadev`` name is "7b:00.0-ch1".
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index e6fb8a0fc8..b8369e7e71 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -6,7 +6,9 @@
 #include <string.h>
 
 #include <rte_bus_pci.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
+#include <rte_io.h>
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
@@ -30,6 +32,141 @@ RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
 #define HISI_DMA_ERR(hw, fmt, args...) \
 		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
 
+static uint32_t
+hisi_dma_queue_base(struct hisi_dma_dev *hw)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08)
+		return HISI_DMA_HIP08_QUEUE_BASE;
+	else
+		return 0;
+}
+
+static void
+hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
+{
+	rte_write32(rte_cpu_to_le_32(val),
+		    (volatile void *)((char *)base + off));
+}
+
+static void
+hisi_dma_write_dev(struct hisi_dma_dev *hw, uint32_t off, uint32_t val)
+{
+	hisi_dma_write_reg(hw->io_base, off, val);
+}
+
+static void
+hisi_dma_write_queue(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t val)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	hisi_dma_write_dev(hw, off, val);
+}
+
+static uint32_t
+hisi_dma_read_reg(void *base, uint32_t off)
+{
+	uint32_t val = rte_read32((volatile void *)((char *)base + off));
+	return rte_le_to_cpu_32(val);
+}
+
+static uint32_t
+hisi_dma_read_dev(struct hisi_dma_dev *hw, uint32_t off)
+{
+	return hisi_dma_read_reg(hw->io_base, off);
+}
+
+static uint32_t
+hisi_dma_read_queue(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return hisi_dma_read_dev(hw, off);
+}
+
+static void
+hisi_dma_update_bit(struct hisi_dma_dev *hw, uint32_t off, uint32_t pos,
+		    bool set)
+{
+	uint32_t tmp = hisi_dma_read_dev(hw, off);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_dev(hw, off, tmp);
+}
+
+static void
+hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
+			  bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
+#define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
+	uint32_t timeout = 0; \
+	while (timeout++ <= (timeout_us)) { \
+		(val) = hisi_dma_read_queue(hw, HISI_DMA_QUEUE_FSM_REG); \
+		if (cond) \
+			break; \
+		rte_delay_us(sleep_us); \
+	} \
+	(cond) ? 0 : -ETIME; \
+})
+
+static int
+hisi_dma_reset_hw(struct hisi_dma_dev *hw)
+{
+#define POLL_SLEEP_US	100
+#define POLL_TIMEOUT_US	10000
+
+	uint32_t tmp;
+	int ret;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, true);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) != HISI_DMA_STATE_RUN,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "disable dma timeout!");
+		return ret;
+	}
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL1_REG,
+				  HISI_DMA_QUEUE_CTRL1_RESET_B, true);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) == HISI_DMA_STATE_IDLE,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "reset dma timeout!");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void
+hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
+{
+	struct hisi_dma_dev hw;
+
+	memset(&hw, 0, sizeof(hw));
+	hw.io_base = pci_bar;
+
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		hisi_dma_update_bit(&hw, HISI_DMA_HIP08_MODE_REG,
+				    HISI_DMA_HIP08_MODE_SEL_B, true);
+}
+
 static uint8_t
 hisi_dma_reg_layout(uint8_t revision)
 {
@@ -49,6 +186,57 @@ hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function);
 }
 
+static void
+hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
+		      uint8_t queue_id, char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x-ch%u",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function, queue_id);
+}
+
+static int
+hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
+		uint8_t revision)
+{
+#define REG_PCI_BAR_INDEX	2
+
+	char name[RTE_DEV_NAME_MAX_LEN];
+	struct rte_dma_dev *dev;
+	struct hisi_dma_dev *hw;
+	int ret;
+
+	hisi_dma_gen_dev_name(pci_dev, queue_id, name, sizeof(name));
+	dev = rte_dma_pmd_allocate(name, pci_dev->device.numa_node,
+				   sizeof(*hw));
+	if (dev == NULL) {
+		HISI_DMA_LOG(ERR, "%s allocate dmadev fail!", name);
+		return -EINVAL;
+	}
+
+	dev->device = &pci_dev->device;
+
+	hw = dev->data->dev_private;
+	hw->data = dev->data;
+	hw->revision = revision;
+	hw->reg_layout = hisi_dma_reg_layout(revision);
+	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
+	hw->queue_id = queue_id;
+
+	ret = hisi_dma_reset_hw(hw);
+	if (ret) {
+		HISI_DMA_LOG(ERR, "%s init device fail!", name);
+		(void)rte_dma_pmd_release(name);
+		return -EIO;
+	}
+
+	dev->state = RTE_DMA_DEV_READY;
+	HISI_DMA_LOG(DEBUG, "%s create dmadev success!", name);
+
+	return 0;
+}
+
 static int
 hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
 			uint8_t *out_revision)
@@ -78,6 +266,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
 	uint8_t revision;
+	uint8_t i;
 	int ret;
 
 	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
@@ -92,13 +281,34 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
+	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		ret = hisi_dma_create(pci_dev, i, revision);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s create dmadev %u failed!",
+				     name, i);
+			break;
+		}
+	}
+
 	return ret;
 }
 
 static int
 hisi_dma_remove(struct rte_pci_device *pci_dev)
 {
-	RTE_SET_USED(pci_dev);
+	char name[RTE_DEV_NAME_MAX_LEN];
+	uint8_t i;
+	int ret;
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		hisi_dma_gen_dev_name(pci_dev, i, name, sizeof(name));
+		ret = rte_dma_pmd_release(name);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 114b9dcb5b..50aaa38b72 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -5,11 +5,24 @@
 #ifndef HISI_DMADEV_H
 #define HISI_DMADEV_H
 
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#define BIT(x)	(1ul << (x))
+#define BITS_PER_LONG	(__SIZEOF_LONG__ * 8)
+#define GENMASK(h, l) \
+		(((~0UL) << (l)) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
+#define BF_SHF(x) (__builtin_ffsll(x) - 1)
+#define FIELD_GET(mask, reg) \
+		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
+#define HISI_DMA_MAX_HW_QUEUES			4
+
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
  * they have the same pci device id but with different pci revision.
@@ -21,4 +34,88 @@ enum {
 	HISI_DMA_REG_LAYOUT_HIP08
 };
 
+/**
+ * Hardware PCI bar register MAP:
+ *
+ *     --------------
+ *     | Misc-reg-0 |
+ *     |            |
+ *     --------------   -> Queue base
+ *     |            |
+ *     | Queue-0    |
+ *     |            |
+ *     --------------   ---
+ *     |            |    ^
+ *     | Queue-1    |   Queue region
+ *     |            |    v
+ *     --------------   ---
+ *     | ...        |
+ *     | Queue-x    |
+ *     | ...        |
+ *     --------------
+ *     | Misc-reg-1 |
+ *     --------------
+ *
+ * As described above, a single queue register is continuous and occupies the
+ * length of queue-region. The global offset for a single queue register is
+ * calculated by:
+ *     offset = queue-base + (queue-id * queue-region) + reg-offset-in-region.
+ *
+ * The first part of queue region is basically the same for HIP08 and later chip
+ * register layouts, therefore, HISI_QUEUE_* registers are defined for it.
+ */
+#define HISI_DMA_QUEUE_SQ_BASE_L_REG		0x0
+#define HISI_DMA_QUEUE_SQ_BASE_H_REG		0x4
+#define HISI_DMA_QUEUE_SQ_DEPTH_REG		0x8
+#define HISI_DMA_QUEUE_SQ_TAIL_REG		0xC
+#define HISI_DMA_QUEUE_CQ_BASE_L_REG		0x10
+#define HISI_DMA_QUEUE_CQ_BASE_H_REG		0x14
+#define HISI_DMA_QUEUE_CQ_DEPTH_REG		0x18
+#define HISI_DMA_QUEUE_CQ_HEAD_REG		0x1C
+#define HISI_DMA_QUEUE_CTRL0_REG		0x20
+#define HISI_DMA_QUEUE_CTRL0_EN_B		0
+#define HISI_DMA_QUEUE_CTRL0_PAUSE_B		4
+#define HISI_DMA_QUEUE_CTRL1_REG		0x24
+#define HISI_DMA_QUEUE_CTRL1_RESET_B		0
+#define HISI_DMA_QUEUE_FSM_REG			0x30
+#define HISI_DMA_QUEUE_FSM_STS_M		GENMASK(3, 0)
+#define HISI_DMA_QUEUE_INT_STATUS_REG		0x40
+#define HISI_DMA_QUEUE_ERR_INT_NUM0_REG		0x84
+#define HISI_DMA_QUEUE_ERR_INT_NUM1_REG		0x88
+#define HISI_DMA_QUEUE_ERR_INT_NUM2_REG		0x8C
+#define HISI_DMA_QUEUE_REGION_SIZE		0x100
+
+/**
+ * HiSilicon IP08 DMA register and field define:
+ */
+#define HISI_DMA_HIP08_QUEUE_BASE			0x0
+#define HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B		2
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_REG		0x44
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_M			GENMASK(14, 0)
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG		0x90
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG		0x94
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG		0x98
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG		0x48
+#define HISI_DMA_HIP08_MODE_REG				0x217C
+#define HISI_DMA_HIP08_MODE_SEL_B			0
+#define HISI_DMA_HIP08_DUMP_START_REG			0x2000
+#define HISI_DMA_HIP08_DUMP_END_REG			0x2280
+
+/**
+ * In fact, there are multiple states, but it need to pay attention to
+ * the following two states for the driver:
+ */
+enum {
+	HISI_DMA_STATE_IDLE = 0,
+	HISI_DMA_STATE_RUN,
+};
+
+struct hisi_dma_dev {
+	struct rte_dma_dev_data *data;
+	uint8_t revision; /**< PCI revision. */
+	uint8_t reg_layout; /**< hardware register layout. */
+	void *io_base;
+	uint8_t queue_id; /**< hardware DMA queue index. */
+};
+
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
                     ` (3 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add control path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 +
 drivers/dma/hisilicon/hisi_dmadev.c | 385 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  99 +++++++
 3 files changed, 494 insertions(+)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 65138a8365..24bae86bdc 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -29,3 +29,13 @@ accessed using API from the ``rte_dmadev`` library.
 The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
 will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
 2nd ``dmadev`` name is "7b:00.0-ch1".
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Kunpeng DMA configuration requirements:
+
+* ``ring_size`` must be a power of two, between 32 and 8192.
+* Only one ``vchan`` is supported per ``dmadev``.
+* Silent mode is not supported.
+* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM``.
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index b8369e7e71..bcdcf4de4b 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -10,6 +10,8 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
 
@@ -41,6 +43,14 @@ hisi_dma_queue_base(struct hisi_dma_dev *hw)
 		return 0;
 }
 
+static volatile void *
+hisi_dma_queue_regaddr(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return (volatile void *)((char *)hw->io_base + off);
+}
+
 static void
 hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
 {
@@ -103,6 +113,15 @@ hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
 	hisi_dma_write_queue(hw, qoff, tmp);
 }
 
+static void
+hisi_dma_update_queue_mbit(struct hisi_dma_dev *hw, uint32_t qoff,
+			   uint32_t mask, bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
 #define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
 	uint32_t timeout = 0; \
 	while (timeout++ <= (timeout_us)) { \
@@ -154,6 +173,45 @@ hisi_dma_reset_hw(struct hisi_dma_dev *hw)
 	return 0;
 }
 
+static void
+hisi_dma_init_hw(struct hisi_dma_dev *hw)
+{
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_L_REG,
+			     lower_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_H_REG,
+			     upper_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_L_REG,
+			     lower_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_H_REG,
+			     upper_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_DEPTH_REG,
+			     hw->sq_depth_mask);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_DEPTH_REG, hw->cq_depth - 1);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM0_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM1_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM2_REG, 0);
+
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG,
+				     0);
+		hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B, false);
+		hisi_dma_update_queue_mbit(hw, HISI_DMA_QUEUE_INT_STATUS_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+		hisi_dma_update_queue_mbit(hw,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+	}
+}
+
 static void
 hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
 {
@@ -176,6 +234,301 @@ hisi_dma_reg_layout(uint8_t revision)
 		return HISI_DMA_REG_LAYOUT_INVALID;
 }
 
+static void
+hisi_dma_zero_iomem(struct hisi_dma_dev *hw)
+{
+	memset(hw->iomz->addr, 0, hw->iomz_sz);
+}
+
+static int
+hisi_dma_alloc_iomem(struct hisi_dma_dev *hw, uint16_t ring_size,
+		     const char *dev_name)
+{
+	uint32_t sq_size = sizeof(struct hisi_dma_sqe) * ring_size;
+	uint32_t cq_size = sizeof(struct hisi_dma_cqe) *
+			   (ring_size + HISI_DMA_CQ_RESERVED);
+	uint32_t status_size = sizeof(uint16_t) * ring_size;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	const struct rte_memzone *iomz;
+	uint32_t total_size;
+
+	sq_size = RTE_CACHE_LINE_ROUNDUP(sq_size);
+	cq_size = RTE_CACHE_LINE_ROUNDUP(cq_size);
+	status_size = RTE_CACHE_LINE_ROUNDUP(status_size);
+	total_size = sq_size + cq_size + status_size;
+
+	(void)snprintf(mz_name, sizeof(mz_name), "hisi_dma:%s", dev_name);
+	iomz = rte_memzone_reserve(mz_name, total_size, hw->data->numa_node,
+				   RTE_MEMZONE_IOVA_CONTIG);
+	if (iomz == NULL) {
+		HISI_DMA_ERR(hw, "malloc %s iomem fail!", mz_name);
+		return -ENOMEM;
+	}
+
+	hw->iomz = iomz;
+	hw->iomz_sz = total_size;
+	hw->sqe = iomz->addr;
+	hw->cqe = (void *)((char *)iomz->addr + sq_size);
+	hw->status = (void *)((char *)iomz->addr + sq_size + cq_size);
+	hw->sqe_iova = iomz->iova;
+	hw->cqe_iova = iomz->iova + sq_size;
+	hw->sq_depth_mask = ring_size - 1;
+	hw->cq_depth = ring_size + HISI_DMA_CQ_RESERVED;
+	hisi_dma_zero_iomem(hw);
+
+	return 0;
+}
+
+static void
+hisi_dma_free_iomem(struct hisi_dma_dev *hw)
+{
+	if (hw->iomz != NULL)
+		rte_memzone_free(hw->iomz);
+
+	hw->iomz = NULL;
+	hw->sqe = NULL;
+	hw->cqe = NULL;
+	hw->status = NULL;
+	hw->sqe_iova = 0;
+	hw->cqe_iova = 0;
+	hw->sq_depth_mask = 0;
+	hw->cq_depth = 0;
+}
+
+static int
+hisi_dma_info_get(const struct rte_dma_dev *dev,
+		  struct rte_dma_info *dev_info,
+		  uint32_t info_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info_sz);
+
+	dev_info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM |
+			     RTE_DMA_CAPA_OPS_COPY;
+	dev_info->max_vchans = 1;
+	dev_info->max_desc = HISI_DMA_MAX_DESC_NUM;
+	dev_info->min_desc = HISI_DMA_MIN_DESC_NUM;
+
+	return 0;
+}
+
+static int
+hisi_dma_configure(struct rte_dma_dev *dev,
+		   const struct rte_dma_conf *conf,
+		   uint32_t conf_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(conf);
+	RTE_SET_USED(conf_sz);
+	return 0;
+}
+
+static int
+hisi_dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan,
+		     const struct rte_dma_vchan_conf *conf,
+		     uint32_t conf_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+	int ret;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(conf_sz);
+
+	if (!rte_is_power_of_2(conf->nb_desc)) {
+		HISI_DMA_ERR(hw, "Number of desc must be power of 2!");
+		return -EINVAL;
+	}
+
+	hisi_dma_free_iomem(hw);
+	ret = hisi_dma_alloc_iomem(hw, conf->nb_desc, dev->data->dev_name);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int
+hisi_dma_start(struct rte_dma_dev *dev)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	if (hw->iomz == NULL) {
+		HISI_DMA_ERR(hw, "Vchan was not setup, start fail!\n");
+		return -EINVAL;
+	}
+
+	/* Reset the dmadev to a known state, include:
+	 *   1) zero iomem, also include status fields.
+	 *   2) init hardware register.
+	 *   3) init index values to zero.
+	 *   4) init running statistics.
+	 */
+	hisi_dma_zero_iomem(hw);
+	hisi_dma_init_hw(hw);
+	hw->ridx = 0;
+	hw->cridx = 0;
+	hw->sq_head = 0;
+	hw->sq_tail = 0;
+	hw->cq_sq_head = 0;
+	hw->cq_head = 0;
+	hw->cqs_completed = 0;
+	hw->cqe_vld = 1;
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, true);
+
+	return 0;
+}
+
+static int
+hisi_dma_stop(struct rte_dma_dev *dev)
+{
+	return hisi_dma_reset_hw(dev->data->dev_private);
+}
+
+static int
+hisi_dma_close(struct rte_dma_dev *dev)
+{
+	/* The dmadev already stopped */
+	hisi_dma_free_iomem(dev->data->dev_private);
+	return 0;
+}
+
+static int
+hisi_dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan,
+		   struct rte_dma_stats *stats,
+		   uint32_t stats_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(stats_sz);
+	stats->submitted = hw->submitted;
+	stats->completed = hw->completed;
+	stats->errors = hw->errors;
+
+	return 0;
+}
+
+static int
+hisi_dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	return 0;
+}
+
+static void
+hisi_dma_get_dump_range(struct hisi_dma_dev *hw, uint32_t *start, uint32_t *end)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		*start = HISI_DMA_HIP08_DUMP_START_REG;
+		*end = HISI_DMA_HIP08_DUMP_END_REG;
+	} else {
+		*start = 0;
+		*end = 0;
+	}
+}
+
+static void
+hisi_dma_dump_common(struct hisi_dma_dev *hw, FILE *f)
+{
+#define DUMP_REGNUM_PER_LINE	4
+
+	uint32_t start, end;
+	uint32_t cnt, i;
+
+	hisi_dma_get_dump_range(hw, &start, &end);
+
+	(void)fprintf(f, "    common-register:\n");
+
+	cnt = 0;
+	for (i = start; i <= end; i += sizeof(uint32_t)) {
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "      [%4x]:", i);
+		(void)fprintf(f, " 0x%08x", hisi_dma_read_dev(hw, i));
+		cnt++;
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "\n");
+	}
+	if (cnt % DUMP_REGNUM_PER_LINE)
+		(void)fprintf(f, "\n");
+}
+
+static void
+hisi_dma_dump_read_queue(struct hisi_dma_dev *hw, uint32_t qoff,
+			 char *buffer, int max_sz)
+{
+	memset(buffer, 0, max_sz);
+
+	/* Address-related registers are not printed for security reasons. */
+	if (qoff == HISI_DMA_QUEUE_SQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_SQ_BASE_H_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_H_REG) {
+		(void)snprintf(buffer, max_sz, "**********");
+		return;
+	}
+
+	(void)snprintf(buffer, max_sz, "0x%08x", hisi_dma_read_queue(hw, qoff));
+}
+
+static void
+hisi_dma_dump_queue(struct hisi_dma_dev *hw, FILE *f)
+{
+#define REG_FMT_LEN	32
+	char buf[REG_FMT_LEN] = { 0 };
+	uint32_t i;
+
+	(void)fprintf(f, "    queue-register:\n");
+	for (i = 0; i < HISI_DMA_QUEUE_REGION_SIZE; ) {
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, "      [%2x]: %s", i, buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s\n", buf);
+		i += sizeof(uint32_t);
+	}
+}
+
+static int
+hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	(void)fprintf(f,
+		"    revision: 0x%x queue_id: %u ring_size: %u\n"
+		"    ridx: %u cridx: %u\n"
+		"    sq_head: %u sq_tail: %u cq_sq_head: %u\n"
+		"    cq_head: %u cqs_completed: %u cqe_vld: %u\n"
+		"    submitted: %" PRIu64 " completed: %" PRIu64 " errors %"
+		PRIu64"\n",
+		hw->revision, hw->queue_id,
+		hw->sq_depth_mask > 0 ? hw->sq_depth_mask + 1 : 0,
+		hw->ridx, hw->cridx,
+		hw->sq_head, hw->sq_tail, hw->cq_sq_head,
+		hw->cq_head, hw->cqs_completed, hw->cqe_vld,
+		hw->submitted, hw->completed, hw->errors);
+	hisi_dma_dump_queue(hw, f);
+	hisi_dma_dump_common(hw, f);
+
+	return 0;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -196,6 +549,33 @@ hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function, queue_id);
 }
 
+/**
+ * Hardware queue state machine:
+ *
+ *   -----------  dmadev_create	  ------------------
+ *   | Unknown | ---------------> |      IDLE      |
+ *   -----------                  ------------------
+ *                                   ^          |
+ *                                   |          |dev_start
+ *                           dev_stop|          |
+ *                                   |          v
+ *                                ------------------
+ *                                |      RUN       |
+ *                                ------------------
+ *
+ */
+static const struct rte_dma_dev_ops hisi_dmadev_ops = {
+	.dev_info_get     = hisi_dma_info_get,
+	.dev_configure    = hisi_dma_configure,
+	.dev_start        = hisi_dma_start,
+	.dev_stop         = hisi_dma_stop,
+	.dev_close        = hisi_dma_close,
+	.vchan_setup      = hisi_dma_vchan_setup,
+	.stats_get        = hisi_dma_stats_get,
+	.stats_reset      = hisi_dma_stats_reset,
+	.dev_dump         = hisi_dma_dump,
+};
+
 static int
 hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 		uint8_t revision)
@@ -216,6 +596,7 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	}
 
 	dev->device = &pci_dev->device;
+	dev->dev_ops = &hisi_dmadev_ops;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
@@ -223,6 +604,10 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->reg_layout = hisi_dma_reg_layout(revision);
 	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
 	hw->queue_id = queue_id;
+	hw->sq_tail_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_SQ_TAIL_REG);
+	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
 	ret = hisi_dma_reset_hw(hw);
 	if (ret) {
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 50aaa38b72..dd0315cd31 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -16,12 +16,17 @@
 #define FIELD_GET(mask, reg) \
 		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
 
+#define lower_32_bits(x) ((uint32_t)(x))
+#define upper_32_bits(x) ((uint32_t)(((x) >> 16) >> 16))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
 #define HISI_DMA_MAX_HW_QUEUES			4
+#define HISI_DMA_MAX_DESC_NUM			8192
+#define HISI_DMA_MIN_DESC_NUM			32
 
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
@@ -110,12 +115,106 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * After scanning the CQ array, the CQ head register needs to be updated.
+ * Updating the register involves write memory barrier operations.
+ * Here use the following method to reduce WMB operations:
+ *   a) malloc more CQEs, which correspond to the macro HISI_DMA_CQ_RESERVED.
+ *   b) update the CQ head register after accumulated number of completed CQs
+ *      is greater than or equal to HISI_DMA_CQ_RESERVED.
+ */
+#define HISI_DMA_CQ_RESERVED		64
+
+struct hisi_dma_sqe {
+	uint32_t dw0;
+#define SQE_FENCE_FLAG	BIT(10)
+#define SQE_OPCODE_M2M	0x4
+	uint32_t dw1;
+	uint32_t dw2;
+	uint32_t length;
+	uint64_t src_addr;
+	uint64_t dst_addr;
+};
+
+struct hisi_dma_cqe {
+	uint64_t rsv;
+	uint64_t misc;
+#define CQE_SQ_HEAD_MASK	GENMASK(15, 0)
+#define CQE_VALID_B		BIT(48)
+#define CQE_STATUS_MASK		GENMASK(63, 49)
+};
+
 struct hisi_dma_dev {
+	struct hisi_dma_sqe *sqe;
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t *status; /* the completion status array of SQEs. */
+
+	volatile void *sq_tail_reg; /**< register address for doorbell. */
+	volatile void *cq_head_reg; /**< register address for answer CQ. */
+
+	uint16_t sq_depth_mask; /**< SQ depth - 1, the SQ depth is power of 2 */
+	uint16_t cq_depth; /* CQ depth */
+
+	uint16_t ridx; /**< ring index which will assign to the next request. */
+	/** ring index which returned by hisi_dmadev_completed APIs. */
+	uint16_t cridx;
+
+	/**
+	 * SQE array management fields:
+	 *
+	 *  -----------------------------------------------------
+	 *  | SQE0 | SQE1 | SQE2 |   ...  | SQEx | ... | SQEn-1 |
+	 *  -----------------------------------------------------
+	 *     ^             ^               ^
+	 *     |             |               |
+	 *   sq_head     cq_sq_head       sq_tail
+	 *
+	 *  sq_head: index to the oldest completed request, this filed was
+	 *           updated by hisi_dmadev_completed* APIs.
+	 *  sq_tail: index of the next new request, this field was updated by
+	 *           hisi_dmadev_copy API.
+	 *  cq_sq_head: next index of index that has been completed by hardware,
+	 *              this filed was updated by hisi_dmadev_completed* APIs.
+	 *
+	 *  [sq_head, cq_sq_head): the SQEs that hardware already completed.
+	 *  [cq_sq_head, sq_tail): the SQEs that hardware processing.
+	 */
+	uint16_t sq_head;
+	uint16_t sq_tail;
+	uint16_t cq_sq_head;
+	/**
+	 * The driver scans the CQE array, if the valid bit changes, the CQE is
+	 * considered valid.
+	 * Note: One CQE is corresponding to one or several SQEs, e.g. app
+	 *       submits two copy requests, the hardware processes the two SQEs,
+	 *       but it may write back only one CQE and the CQE's sq_head field
+	 *       indicates the index of the second copy request in the SQE
+	 *       array.
+	 */
+	uint16_t cq_head; /**< CQ index for next scans. */
+	/** accumulated number of completed CQs
+	 * @see HISI_DMA_CQ_RESERVED
+	 */
+	uint16_t cqs_completed;
+	uint8_t cqe_vld; /**< valid bit for CQE, will change for every round. */
+
+	uint64_t submitted;
+	uint64_t completed;
+	uint64_t errors;
+
+	/**
+	 * The following fields are not accessed in the I/O path, so they are
+	 * placed at the end.
+	 */
 	struct rte_dma_dev_data *data;
 	uint8_t revision; /**< PCI revision. */
 	uint8_t reg_layout; /**< hardware register layout. */
 	void *io_base;
 	uint8_t queue_id; /**< hardware DMA queue index. */
+	const struct rte_memzone *iomz;
+	uint32_t iomz_sz;
+	rte_iova_t sqe_iova;
+	rte_iova_t cqe_iova;
 };
 
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data path functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (2 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
                     ` (2 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add data path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 206 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  16 +++
 2 files changed, 222 insertions(+)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index bcdcf4de4b..d03967cae3 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -529,6 +529,206 @@ hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
 	return 0;
 }
 
+static int
+hisi_dma_copy(void *dev_private, uint16_t vchan,
+		 rte_iova_t src, rte_iova_t dst,
+		 uint32_t length, uint64_t flags)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	struct hisi_dma_sqe *sqe = &hw->sqe[hw->sq_tail];
+
+	RTE_SET_USED(vchan);
+
+	if (((hw->sq_tail + 1) & hw->sq_depth_mask) == hw->sq_head)
+		return -ENOSPC;
+
+	sqe->dw0 = rte_cpu_to_le_32(SQE_OPCODE_M2M);
+	sqe->dw1 = 0;
+	sqe->dw2 = 0;
+	sqe->length = rte_cpu_to_le_32(length);
+	sqe->src_addr = rte_cpu_to_le_64(src);
+	sqe->dst_addr = rte_cpu_to_le_64(dst);
+	hw->sq_tail = (hw->sq_tail + 1) & hw->sq_depth_mask;
+	hw->submitted++;
+
+	if (flags & RTE_DMA_OP_FLAG_FENCE)
+		sqe->dw0 |= rte_cpu_to_le_32(SQE_FENCE_FLAG);
+	if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+		rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return hw->ridx++;
+}
+
+static int
+hisi_dma_submit(void *dev_private, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev_private;
+
+	RTE_SET_USED(vchan);
+	rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return 0;
+}
+
+static inline void
+hisi_dma_scan_cq(struct hisi_dma_dev *hw)
+{
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t csq_head = hw->cq_sq_head;
+	uint16_t cq_head = hw->cq_head;
+	uint16_t count = 0;
+	uint64_t misc;
+
+	while (true) {
+		cqe = &hw->cqe[cq_head];
+		misc = cqe->misc;
+		misc = rte_le_to_cpu_64(misc);
+		if (FIELD_GET(CQE_VALID_B, misc) != hw->cqe_vld)
+			break;
+
+		csq_head = FIELD_GET(CQE_SQ_HEAD_MASK, misc);
+		if (unlikely(misc & CQE_STATUS_MASK))
+			hw->status[csq_head] = FIELD_GET(CQE_STATUS_MASK,
+							 misc);
+
+		count++;
+		cq_head++;
+		if (cq_head == hw->cq_depth) {
+			hw->cqe_vld = !hw->cqe_vld;
+			cq_head = 0;
+		}
+	}
+
+	if (count == 0)
+		return;
+
+	hw->cq_head = cq_head;
+	hw->cq_sq_head = (csq_head + 1) & hw->sq_depth_mask;
+	hw->cqs_completed += count;
+	if (hw->cqs_completed >= HISI_DMA_CQ_RESERVED) {
+		rte_write32(rte_cpu_to_le_32(cq_head), hw->cq_head_reg);
+		hw->cqs_completed = 0;
+	}
+}
+
+static inline uint16_t
+hisi_dma_calc_cpls(struct hisi_dma_dev *hw, const uint16_t nb_cpls)
+{
+	uint16_t cpl_num;
+
+	if (hw->cq_sq_head >= hw->sq_head)
+		cpl_num = hw->cq_sq_head - hw->sq_head;
+	else
+		cpl_num = hw->sq_depth_mask + 1 - hw->sq_head + hw->cq_sq_head;
+
+	if (cpl_num > nb_cpls)
+		cpl_num = nb_cpls;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_completed(void *dev_private,
+		   uint16_t vchan, const uint16_t nb_cpls,
+		   uint16_t *last_idx, bool *has_error)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		if (hw->status[sq_head]) {
+			*has_error = true;
+			break;
+		}
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (i > 0) {
+		hw->cridx += i;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += i;
+
+	return i;
+}
+
+static enum rte_dma_status_code
+hisi_dma_convert_status(uint16_t status)
+{
+	switch (status) {
+	case HISI_DMA_STATUS_SUCCESS:
+		return RTE_DMA_STATUS_SUCCESSFUL;
+	case HISI_DMA_STATUS_INVALID_OPCODE:
+		return RTE_DMA_STATUS_INVALID_OPCODE;
+	case HISI_DMA_STATUS_INVALID_LENGTH:
+		return RTE_DMA_STATUS_INVALID_LENGTH;
+	case HISI_DMA_STATUS_USER_ABORT:
+		return RTE_DMA_STATUS_USER_ABORT;
+	case HISI_DMA_STATUS_REMOTE_READ_ERROR:
+	case HISI_DMA_STATUS_AXI_READ_ERROR:
+		return RTE_DMA_STATUS_BUS_READ_ERROR;
+	case HISI_DMA_STATUS_AXI_WRITE_ERROR:
+		return RTE_DMA_STATUS_BUS_WRITE_ERROR;
+	case HISI_DMA_STATUS_DATA_POISON:
+	case HISI_DMA_STATUS_REMOTE_DATA_POISION:
+		return RTE_DMA_STATUS_DATA_POISION;
+	case HISI_DMA_STATUS_SQE_READ_ERROR:
+	case HISI_DMA_STATUS_SQE_READ_POISION:
+		return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR;
+	case HISI_DMA_STATUS_LINK_DOWN_ERROR:
+		return RTE_DMA_STATUS_DEV_LINK_ERROR;
+	default:
+		return RTE_DMA_STATUS_ERROR_UNKNOWN;
+	}
+}
+
+static uint16_t
+hisi_dma_completed_status(void *dev_private,
+			  uint16_t vchan, const uint16_t nb_cpls,
+			  uint16_t *last_idx, enum rte_dma_status_code *status)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		status[i] = hisi_dma_convert_status(hw->status[sq_head]);
+		hw->errors += !!status[i];
+		hw->status[sq_head] = HISI_DMA_STATUS_SUCCESS;
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (likely(cpl_num > 0)) {
+		hw->cridx += cpl_num;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += cpl_num;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_burst_capacity(const void *dev_private, uint16_t vchan)
+{
+	const struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t sq_tail = hw->sq_tail;
+
+	RTE_SET_USED(vchan);
+
+	return (sq_tail >= sq_head) ? hw->sq_depth_mask - sq_tail + sq_head :
+				      sq_head - 1 - sq_tail;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -597,6 +797,12 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 
 	dev->device = &pci_dev->device;
 	dev->dev_ops = &hisi_dmadev_ops;
+	dev->fp_obj->dev_private = dev->data->dev_private;
+	dev->fp_obj->copy = hisi_dma_copy;
+	dev->fp_obj->submit = hisi_dma_submit;
+	dev->fp_obj->completed = hisi_dma_completed;
+	dev->fp_obj->completed_status = hisi_dma_completed_status;
+	dev->fp_obj->burst_capacity = hisi_dma_burst_capacity;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index dd0315cd31..12e209c86e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -115,6 +115,22 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * Hardware complete status define:
+ */
+#define HISI_DMA_STATUS_SUCCESS			0x0
+#define HISI_DMA_STATUS_INVALID_OPCODE		0x1
+#define HISI_DMA_STATUS_INVALID_LENGTH		0x2
+#define HISI_DMA_STATUS_USER_ABORT		0x4
+#define HISI_DMA_STATUS_REMOTE_READ_ERROR	0x10
+#define HISI_DMA_STATUS_AXI_READ_ERROR		0x20
+#define HISI_DMA_STATUS_AXI_WRITE_ERROR		0x40
+#define HISI_DMA_STATUS_DATA_POISON		0x80
+#define HISI_DMA_STATUS_SQE_READ_ERROR		0x100
+#define HISI_DMA_STATUS_SQE_READ_POISION	0x200
+#define HISI_DMA_STATUS_REMOTE_DATA_POISION	0x400
+#define HISI_DMA_STATUS_LINK_DOWN_ERROR		0x800
+
 /**
  * After scanning the CQ array, the CQ head register needs to be updated.
  * Updating the register involves write memory barrier operations.
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (3 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
  2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add multi-process support for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index d03967cae3..05066b4d0e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -392,8 +392,10 @@ hisi_dma_stop(struct rte_dma_dev *dev)
 static int
 hisi_dma_close(struct rte_dma_dev *dev)
 {
-	/* The dmadev already stopped */
-	hisi_dma_free_iomem(dev->data->dev_private);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		/* The dmadev already stopped */
+		hisi_dma_free_iomem(dev->data->dev_private);
+	}
 	return 0;
 }
 
@@ -815,11 +817,13 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
 						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
-	ret = hisi_dma_reset_hw(hw);
-	if (ret) {
-		HISI_DMA_LOG(ERR, "%s init device fail!", name);
-		(void)rte_dma_pmd_release(name);
-		return -EIO;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = hisi_dma_reset_hw(hw);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s init device fail!", name);
+			(void)rte_dma_pmd_release(name);
+			return -EIO;
+		}
 	}
 
 	dev->state = RTE_DMA_DEV_READY;
@@ -872,7 +876,8 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
-	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
 
 	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
 		ret = hisi_dma_create(pci_dev, i, revision);
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (4 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

add Kunpeng DMA device ID to dmadev category.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 usertools/dpdk-devbind.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index bb00f43702..a74a68ed82 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -68,10 +68,14 @@
 intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
                  'SVendor': None, 'SDevice': None}
 
+hisilicon_dma = {'Class': '08', 'Vendor': '19e5', 'Device': 'a122',
+                 'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx]
+dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
+               hisilicon_dma]
 eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (5 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
@ 2021-11-07 18:59   ` Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Thomas Monjalon @ 2021-11-07 18:59 UTC (permalink / raw)
  To: Chengwen Feng; +Cc: dev

02/11/2021 13:37, Chengwen Feng:
> This patch set add hisilicon DMA driver.
> 
> Chengwen Feng (6):
>   dma/hisilicon: add device probe and remove functions
>   dma/hisilicon: add dmadev instances create and destroy
>   dma/hisilicon: add control path functions
>   dma/hisilicon: add data path functions
>   dma/hisilicon: support multi-process
>   devbind: add Kunpeng DMA to dmadev category

Applied with few minor fixes, thanks.




^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2021-11-07 18:59 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
2021-11-07 17:28     ` Thomas Monjalon
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).