From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CFB62A0524; Tue, 1 Jun 2021 05:17:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B4D2240E78; Tue, 1 Jun 2021 05:17:52 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id A8FC840E6E for ; Tue, 1 Jun 2021 05:17:51 +0200 (CEST) IronPort-SDR: a91HO7d+UlykVIiecTt19buFTltM+uyjbmSO13zWU+WMU+zAWNOejFUZR6A9NgLZLXvxoDwiFU Bn9xUy1dciEw== X-IronPort-AV: E=McAfee;i="6200,9189,10001"; a="289079291" X-IronPort-AV: E=Sophos;i="5.83,239,1616482800"; d="scan'208";a="289079291" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 May 2021 20:17:50 -0700 IronPort-SDR: 16VRn3utpkvMzx/dsp2k/SWjPAF2FF7I36QUXOdt+IwzgNbT//P69BOPaC1kRWNHTo3yug58Jg xdXL9FwGwWjA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,239,1616482800"; d="scan'208";a="482315443" Received: from npg-dpdk-virtio-xiachenbo-nw.sh.intel.com ([10.67.118.250]) by fmsmga002.fm.intel.com with ESMTP; 31 May 2021 20:17:44 -0700 From: Chenbo Xia To: dev@dpdk.org, thomas@monjalon.net, cunming.liang@intel.com, jingjing.wu@intel.com Cc: anatoly.burakov@intel.com, ferruh.yigit@intel.com, mdr@ashroe.eu, nhorman@tuxdriver.com, bruce.richardson@intel.com, david.marchand@redhat.com, stephen@networkplumber.org, konstantin.ananyev@intel.com, Tiwei Bie Date: Tue, 1 Jun 2021 11:06:41 +0800 Message-Id: <20210601030644.3318-4-chenbo.xia@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20210601030644.3318-1-chenbo.xia@intel.com> References: <20190715075214.16616-6-tiwei.bie@intel.com> <20210601030644.3318-1-chenbo.xia@intel.com> Subject: [dpdk-dev] [RFC v3 3/6] bus/pci: introduce helper for MMIO read and write X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Tiwei Bie The MMIO regions may not be mmap-able for mediated PCI device. In this case, the application should explicitly do read and write to access these regions. Signed-off-by: Tiwei Bie --- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 4 +++ 7 files changed, 188 insertions(+) diff --git a/drivers/bus/pci/bsd/pci.c b/drivers/bus/pci/bsd/pci.c index 20ce979f60..781f65c637 100644 --- a/drivers/bus/pci/bsd/pci.c +++ b/drivers/bus/pci/bsd/pci.c @@ -494,6 +494,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev, return -1; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read(const struct rte_pci_device *dev, int bar, + void *buf, size_t len, off_t offset) +{ + if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || + (uint64_t)offset + len > dev->mem_resource[bar].len) + return -1; + memcpy(buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); + return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write(const struct rte_pci_device *dev, int bar, + const void *buf, size_t len, off_t offset) +{ + if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || + (uint64_t)offset + len > dev->mem_resource[bar].len) + return -1; + memcpy((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); + return len; +} + int rte_pci_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p) diff --git a/drivers/bus/pci/linux/pci.c b/drivers/bus/pci/linux/pci.c index 8f1fddbf20..4805f277c5 100644 --- a/drivers/bus/pci/linux/pci.c +++ b/drivers/bus/pci/linux/pci.c @@ -682,6 +682,52 @@ int rte_pci_write_config(const struct rte_pci_device *device, } } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read(const struct rte_pci_device *device, int bar, + void *buf, size_t len, off_t offset) +{ + char devname[RTE_DEV_NAME_MAX_LEN] = ""; + + switch (device->kdrv) { + case RTE_PCI_KDRV_IGB_UIO: + case RTE_PCI_KDRV_UIO_GENERIC: + return pci_uio_mmio_read(device, bar, buf, len, offset); +#ifdef VFIO_PRESENT + case RTE_PCI_KDRV_VFIO: + return pci_vfio_mmio_read(device, bar, buf, len, offset); +#endif + default: + rte_pci_device_name(&device->addr, devname, + RTE_DEV_NAME_MAX_LEN); + RTE_LOG(ERR, EAL, + "Unknown driver type for %s\n", devname); + return -1; + } +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write(const struct rte_pci_device *device, int bar, + const void *buf, size_t len, off_t offset) +{ + char devname[RTE_DEV_NAME_MAX_LEN] = ""; + + switch (device->kdrv) { + case RTE_PCI_KDRV_IGB_UIO: + case RTE_PCI_KDRV_UIO_GENERIC: + return pci_uio_mmio_write(device, bar, buf, len, offset); +#ifdef VFIO_PRESENT + case RTE_PCI_KDRV_VFIO: + return pci_vfio_mmio_write(device, bar, buf, len, offset); +#endif + default: + rte_pci_device_name(&device->addr, devname, + RTE_DEV_NAME_MAX_LEN); + RTE_LOG(ERR, EAL, + "Unknown driver type for %s\n", devname); + return -1; + } +} + int rte_pci_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p) diff --git a/drivers/bus/pci/linux/pci_init.h b/drivers/bus/pci/linux/pci_init.h index 9f6659ba6e..6853fa88a3 100644 --- a/drivers/bus/pci/linux/pci_init.h +++ b/drivers/bus/pci/linux/pci_init.h @@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle, int pci_uio_write_config(const struct rte_intr_handle *intr_handle, const void *buf, size_t len, off_t offs); +int pci_uio_mmio_read(const struct rte_pci_device *dev, int bar, + void *buf, size_t len, off_t offset); +int pci_uio_mmio_write(const struct rte_pci_device *dev, int bar, + const void *buf, size_t len, off_t offset); + int pci_uio_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p, @@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev, int pci_vfio_write_config(const struct rte_pci_device *dev, const void *buf, size_t len, off_t offs); +int pci_vfio_mmio_read(const struct rte_pci_device *dev, int bar, + void *buf, size_t len, off_t offset); +int pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, + const void *buf, size_t len, off_t offset); + int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar, struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p, diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c index 39ebeac2a0..2482635058 100644 --- a/drivers/bus/pci/linux/pci_uio.c +++ b/drivers/bus/pci/linux/pci_uio.c @@ -45,6 +45,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle, return pwrite(intr_handle->uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read(const struct rte_pci_device *dev, int bar, + void *buf, size_t len, off_t offset) +{ + if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || + (uint64_t)offset + len > dev->mem_resource[bar].len) + return -1; + memcpy(buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); + return len; +} + +int +pci_uio_mmio_write(const struct rte_pci_device *dev, int bar, + const void *buf, size_t len, off_t offset) +{ + if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || + (uint64_t)offset + len > dev->mem_resource[bar].len) + return -1; + memcpy((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); + return len; +} + static int pci_uio_set_bus_master(int dev_fd) { diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index 012e7f72c1..3ecd984215 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -1212,6 +1212,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p) return -1; } +int +pci_vfio_mmio_read(const struct rte_pci_device *dev, int bar, + void *buf, size_t len, off_t offs) +{ + uint64_t size, offset; + int fd; + + fd = dev->intr_handle.vfio_dev_fd; + + if (pci_vfio_get_region(dev, bar, &size, &offset) != 0) + return -1; + + if ((uint64_t)len + offs > size) + return -1; + + return pread64(fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write(const struct rte_pci_device *dev, int bar, + const void *buf, size_t len, off_t offs) +{ + uint64_t size, offset; + int fd; + + fd = dev->intr_handle.vfio_dev_fd; + + if (pci_vfio_get_region(dev, bar, &size, &offset) != 0) + return -1; + + if ((uint64_t)len + offs > size) + return -1; + + return pwrite64(fd, buf, len, offset + offs); +} + int pci_vfio_is_enabled(void) { diff --git a/drivers/bus/pci/rte_bus_pci.h b/drivers/bus/pci/rte_bus_pci.h index 64886b4731..dc26811b0a 100644 --- a/drivers/bus/pci/rte_bus_pci.h +++ b/drivers/bus/pci/rte_bus_pci.h @@ -310,6 +310,54 @@ int rte_pci_read_config(const struct rte_pci_device *device, int rte_pci_write_config(const struct rte_pci_device *device, const void *buf, size_t len, off_t offset); +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Read from a MMIO pci resource. + * + * @param device + * A pointer to a rte_pci_device structure describing the device + * to use + * @param bar + * Index of the io pci resource we want to access. + * @param buf + * A data buffer where the bytes should be read into + * @param len + * The length of the data buffer. + * @param offset + * The offset into MMIO space described by @bar + * @return + * Number of bytes read on success, negative on error. + */ +__rte_experimental +int rte_pci_mmio_read(const struct rte_pci_device *device, int bar, + void *buf, size_t len, off_t offset); + +/** + * @warning + * @b EXPERIMENTAL: this API may change without prior notice + * + * Write to a MMIO pci resource. + * + * @param device + * A pointer to a rte_pci_device structure describing the device + * to use + * @param bar + * Index of the io pci resource we want to access. + * @param buf + * A data buffer containing the bytes should be written + * @param len + * The length of the data buffer. + * @param offset + * The offset into MMIO space described by @bar + * @return + * Number of bytes written on success, negative on error. + */ +__rte_experimental +int rte_pci_mmio_write(const struct rte_pci_device *device, int bar, + const void *buf, size_t len, off_t offset); + /** * A structure used to access io resources for a pci device. * rte_pci_ioport is arch, os, driver specific, and should not be used outside diff --git a/drivers/bus/pci/version.map b/drivers/bus/pci/version.map index f33ed0abd1..02e4219aab 100644 --- a/drivers/bus/pci/version.map +++ b/drivers/bus/pci/version.map @@ -21,4 +21,8 @@ EXPERIMENTAL { global: rte_pci_find_ext_capability; + + # added in 21.08 + rte_pci_mmio_read; + rte_pci_mmio_write; }; -- 2.17.1