From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 81C63A0C45; Wed, 22 Sep 2021 04:04:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 05D554003F; Wed, 22 Sep 2021 04:04:11 +0200 (CEST) Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by mails.dpdk.org (Postfix) with ESMTP id DE24C4003C for ; Wed, 22 Sep 2021 04:04:08 +0200 (CEST) Received: from dggemv703-chm.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4HDhLz27xrzbmg0; Wed, 22 Sep 2021 09:59:55 +0800 (CST) Received: from dggpeml500024.china.huawei.com (7.185.36.10) by dggemv703-chm.china.huawei.com (10.3.19.46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Wed, 22 Sep 2021 10:04:06 +0800 Received: from [10.40.190.165] (10.40.190.165) by dggpeml500024.china.huawei.com (7.185.36.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Wed, 22 Sep 2021 10:04:06 +0800 To: Kevin Laatz , CC: , , References: <20210827172048.558704-1-kevin.laatz@intel.com> <20210917152437.3270330-1-kevin.laatz@intel.com> <20210917152437.3270330-5-kevin.laatz@intel.com> From: fengchengwen Message-ID: Date: Wed, 22 Sep 2021 10:04:05 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <20210917152437.3270330-5-kevin.laatz@intel.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.40.190.165] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpeml500024.china.huawei.com (7.185.36.10) X-CFilter-Loop: Reflected Subject: Re: [dpdk-dev] [PATCH v5 04/16] dma/idxd: create dmadev instances on bus probe X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 2021/9/17 23:24, Kevin Laatz wrote: > When a suitable device is found during the bus scan/probe, create a dmadev > instance for each HW queue. Internal structures required for device > creation are also added. > > Signed-off-by: Bruce Richardson > Signed-off-by: Kevin Laatz > Reviewed-by: Conor Walsh > > --- > v4: > - fix 'vdev' naming, changed to 'bus' > - rebase changes > --- > drivers/dma/idxd/idxd_bus.c | 19 ++++++++ > drivers/dma/idxd/idxd_common.c | 76 ++++++++++++++++++++++++++++++++ > drivers/dma/idxd/idxd_internal.h | 40 +++++++++++++++++ > drivers/dma/idxd/meson.build | 1 + > 4 files changed, 136 insertions(+) > create mode 100644 drivers/dma/idxd/idxd_common.c > > diff --git a/drivers/dma/idxd/idxd_bus.c b/drivers/dma/idxd/idxd_bus.c > index ef589af30e..b48fa954ed 100644 > --- a/drivers/dma/idxd/idxd_bus.c > +++ b/drivers/dma/idxd/idxd_bus.c > @@ -85,6 +85,18 @@ dsa_get_sysfs_path(void) > return path ? path : DSA_SYSFS_PATH; > } > > +static int > +idxd_dev_close(struct rte_dma_dev *dev) > +{ > + struct idxd_dmadev *idxd = dev->data->dev_private; > + munmap(idxd->portal, 0x1000); > + return 0; > +} > + > +static const struct rte_dma_dev_ops idxd_bus_ops = { > + .dev_close = idxd_dev_close, > +}; > + > static void * > idxd_bus_mmap_wq(struct rte_dsa_device *dev) > { > @@ -206,6 +218,7 @@ idxd_probe_dsa(struct rte_dsa_device *dev) > return -1; > idxd.max_batch_size = ret; > idxd.qid = dev->addr.wq_id; > + idxd.u.bus.dsa_id = dev->addr.device_id; > idxd.sva_support = 1; > > idxd.portal = idxd_bus_mmap_wq(dev); > @@ -214,6 +227,12 @@ idxd_probe_dsa(struct rte_dsa_device *dev) > return -ENOENT; > } > > + ret = idxd_dmadev_create(dev->wq_name, &dev->device, &idxd, &idxd_bus_ops); > + if (ret) { > + IDXD_PMD_ERR("Failed to create rawdev %s", dev->wq_name); > + return ret; > + } > + > return 0; > } > > diff --git a/drivers/dma/idxd/idxd_common.c b/drivers/dma/idxd/idxd_common.c > new file mode 100644 > index 0000000000..8afad637fc > --- /dev/null > +++ b/drivers/dma/idxd/idxd_common.c > @@ -0,0 +1,76 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2021 Intel Corporation > + */ > + > +#include > +#include > +#include > + > +#include "idxd_internal.h" > + > +#define IDXD_PMD_NAME_STR "dmadev_idxd" > + > +int > +idxd_dmadev_create(const char *name, struct rte_device *dev, > + const struct idxd_dmadev *base_idxd, > + const struct rte_dma_dev_ops *ops) > +{ > + struct idxd_dmadev *idxd; > + struct rte_dma_dev *dmadev = NULL; > + int ret = 0; > + > + if (!name) { > + IDXD_PMD_ERR("Invalid name of the device!"); > + ret = -EINVAL; > + goto cleanup; > + } > + > + /* Allocate device structure */ > + dmadev = rte_dma_pmd_allocate(name, dev->numa_node, > + sizeof(dmadev->dev_private)); > + if (dmadev == NULL) { > + IDXD_PMD_ERR("Unable to allocate raw device"); > + ret = -ENOMEM; > + goto cleanup; > + } > + dmadev->dev_ops = ops; > + dmadev->device = dev; > + > + idxd = rte_malloc_socket(NULL, sizeof(struct idxd_dmadev), 0, dev->numa_node); > + if (idxd == NULL) { > + IDXD_PMD_ERR("Unable to allocate memory for device"); > + ret = -ENOMEM; > + goto cleanup; > + } > + dmadev->data->dev_private = idxd; > + dmadev->dev_private = idxd; The dmadev->dev_private and dmadev->data->dev_private already inited by rte_dma_pmd_allocate, and the driver only needs to pass in the correct parameters. Recommended: dmadev = rte_dma_pmd_allocate(name, dev->name, sizeof(struct idxd_dmadev)); > + *idxd = *base_idxd; /* copy over the main fields already passed in */ > + idxd->dmadev = dmadev; > + > + /* allocate batch index ring and completion ring. > + * The +1 is because we can never fully use > + * the ring, otherwise read == write means both full and empty. > + */ > + idxd->batch_comp_ring = rte_zmalloc(NULL, (sizeof(idxd->batch_idx_ring[0]) + > + sizeof(idxd->batch_comp_ring[0])) * (idxd->max_batches + 1), > + sizeof(idxd->batch_comp_ring[0])); > + if (idxd->batch_comp_ring == NULL) { > + IDXD_PMD_ERR("Unable to reserve memory for batch data\n"); > + ret = -ENOMEM; > + goto cleanup; > + } > + idxd->batch_idx_ring = (void *)&idxd->batch_comp_ring[idxd->max_batches+1]; > + idxd->batch_iova = rte_mem_virt2iova(idxd->batch_comp_ring); > + Once init one dmadev successful, driver need changes it's state to READY, like: dmadev->state = RTE_DMA_DEV_READY; This was useful when call rte_dma_pmd_release: if the state is ready, lib will call rte_dma_close() to release the dmadev, else it only clean the struct which lib holds. > + return 0; > + > +cleanup: > + if (dmadev) > + rte_dma_pmd_release(name); > + > + return ret; > +} > + > +int idxd_pmd_logtype; > + > +RTE_LOG_REGISTER_DEFAULT(idxd_pmd_logtype, WARNING); > diff --git a/drivers/dma/idxd/idxd_internal.h b/drivers/dma/idxd/idxd_internal.h > index c6a7dcd72f..fa6f053f72 100644 > --- a/drivers/dma/idxd/idxd_internal.h > +++ b/drivers/dma/idxd/idxd_internal.h > @@ -24,4 +24,44 @@ extern int idxd_pmd_logtype; > #define IDXD_PMD_ERR(fmt, args...) IDXD_PMD_LOG(ERR, fmt, ## args) > #define IDXD_PMD_WARN(fmt, args...) IDXD_PMD_LOG(WARNING, fmt, ## args) > > +struct idxd_dmadev { > + /* counters to track the batches */ > + unsigned short max_batches; > + unsigned short batch_idx_read; > + unsigned short batch_idx_write; > + > + /* track descriptors and handles */ > + unsigned short desc_ring_mask; > + unsigned short ids_avail; /* handles for ops completed */ > + unsigned short ids_returned; /* the read pointer for hdls/desc rings */ > + unsigned short batch_start; /* start+size == write pointer for hdls/desc */ > + unsigned short batch_size; > + > + void *portal; /* address to write the batch descriptor */ > + > + struct idxd_completion *batch_comp_ring; > + unsigned short *batch_idx_ring; /* store where each batch ends */ > + > + struct rte_dma_stats stats; > + > + rte_iova_t batch_iova; /* base address of the batch comp ring */ > + rte_iova_t desc_iova; /* base address of desc ring, needed for completions */ > + > + unsigned short max_batch_size; > + > + struct rte_dma_dev *dmadev; > + struct rte_dma_vchan_conf qcfg; > + uint8_t sva_support; > + uint8_t qid; > + > + union { > + struct { > + unsigned int dsa_id; > + } bus; > + } u; > +}; > + > +int idxd_dmadev_create(const char *name, struct rte_device *dev, > + const struct idxd_dmadev *base_idxd, const struct rte_dma_dev_ops *ops); > + > #endif /* _IDXD_INTERNAL_H_ */ > diff --git a/drivers/dma/idxd/meson.build b/drivers/dma/idxd/meson.build > index c864fce3b3..36dbd3e518 100644 > --- a/drivers/dma/idxd/meson.build > +++ b/drivers/dma/idxd/meson.build > @@ -8,5 +8,6 @@ endif > deps += ['bus_pci'] > sources = files( > 'idxd_bus.c', > + 'idxd_common.c', > 'idxd_pci.c' > ) > \ No newline at end of file >