* Re: [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding
@ 2014-05-27 3:23 Xu, HuilongX
0 siblings, 0 replies; 3+ messages in thread
From: Xu, HuilongX @ 2014-05-27 3:23 UTC (permalink / raw)
To: dev, Burakov, Anatoly
Add support for binding VFIO devices if RTE_PCI_DRV_NEED_IGB_UIO
is set for this driver. Try VFIO first, if not mapped then try
IGB_UIO too.
Signed-off-by: Anatoly Burakov <anatoly.burakov at intel.com>
Test-by: HuilongX Xu huilongx.xu@intel.com
Compile pass
>>Compile OS: FC20 x86_64
>>Kernel version: 3.13.6-200
>>GCC version: 4.8.2
>>Server: Crownpass
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 55 ++++++++++++++++++++++++++++++++-
1 files changed, 54 insertions(+), 1 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 7256406..953abe6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -401,6 +401,7 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct rte_pci_device *d
{
struct rte_pci_id *id_table;
int ret = 0;
+ int mapped = 0;
for (id_table = dr->id_table ; id_table->vendor_id != 0; id_table++) {
@@ -435,8 +436,17 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct rte_pci_device *d
}
if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
+ /* try mapping the NIC resources using VFIO if it exists */
+#ifdef VFIO_PRESENT
+ if (vfio_cfg.vfio_enabled) {
+ if ((ret = pci_vfio_map_resource(dev)) == 0)
+ mapped = 1;
+ else if (ret < 0)
+ return ret;
+ }
+#endif
/* map resources for devices that use igb_uio */
- if ((ret = pci_uio_map_resource(dev)) != 0)
+ if (!mapped && (ret = pci_uio_map_resource(dev)) != 0)
return ret;
} else if (dr->drv_flags & RTE_PCI_DRV_FORCE_UNBIND &&
rte_eal_process_type() == RTE_PROC_PRIMARY) {
@@ -471,5 +481,48 @@ rte_eal_pci_init(void)
RTE_LOG(ERR, EAL, "%s(): Cannot scan PCI bus\n", __func__);
return -1;
}
+#ifdef VFIO_PRESENT
+ memset(&vfio_cfg, 0, sizeof(vfio_cfg));
+
+ /* initialize group list */
+ int i, ret;
+
+ for (i = 0; i < VFIO_MAX_GROUPS; i++) {
+ vfio_cfg.vfio_groups[i].fd = -1;
+ vfio_cfg.vfio_groups[i].group_no = -1;
+ }
+ vfio_cfg.vfio_container_fd = -1;
+
+ /* check if we have VFIO driver enabled */
+ if (access(VFIO_DIR, F_OK) == 0) {
+ static int socket_fd;
+
+ vfio_cfg.vfio_enabled = 1;
+
+ /* if we are primary process, create a thread to communicate with
+ * secondary processes. the thread will use a socket to wait for
+ * requests from secondary process to send open file descriptors,
+ * because VFIO does not allow multiple open descriptors on a group or
+ * VFIO container.
+ */
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ /* set up local socket */
+ if ((socket_fd = pci_vfio_socket_setup()) < 0) {
+ RTE_LOG(ERR, EAL, "Failed to set up local socket!\n");
+ return -1;
+ }
+ ret = pthread_create(&socket_thread, NULL,
+ pci_vfio_socket_thread, (void*) &socket_fd);
+ if (ret) {
+ RTE_LOG(ERR, EAL,
+ "Failed to create thread for communication with secondary "
+ "processes!\n");
+ return -1;
+ }
+ }
+ }
+ else
+ RTE_LOG(INFO, EAL, "VFIO driver not loaded or wrong permissions\n");
+#endif
return 0;
}
--
1.7.0.7
^ permalink raw reply [flat|nested] 3+ messages in thread* [dpdk-dev] [PATCH 00/16] [RFC] [VFIO] Add VFIO support to DPDK
@ 2014-05-01 11:05 Burakov, Anatoly
2014-05-19 15:51 ` [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding Anatoly Burakov
0 siblings, 1 reply; 3+ messages in thread
From: Burakov, Anatoly @ 2014-05-01 11:05 UTC (permalink / raw)
To: dev
This patchset adds support for using VFIO instead of IGB_UIO to
map the device BARs.
VFIO is a kernel 3.6+ driver allowing secure DMA from userspace
by means of using IOMMU instead of working directly with physical
memory like igb_uio does.
Short summary:
* Adding support for VFIO in EAL PCI code
* Adding new command-line parameter for VFIO interrupt type
* Adding support for VFIO in setup.sh
* Renaming igb_uio_bind to dpdk_nic_bind and adding support for
VFIO there
* Removing PCI ID list from igb_uio, effectively making it another
generic PCI driver similar to pci_stub, vfio-pci et al
* Adding autotest for VFIO interrupt types
* Making igb_uio and VFIO compilation optional
I'm submitting this as an RFC because this patch is based off
current dpdk.org branch with David Marchand's RTE_EAL_UNBIND_PORTS
patchset. IOW, this will *not* apply to the dpdk.org tree *unless* you
also apply David's patches beforehand.
Signed-off by: Anatoly Burakov <anatoly.burakov@intel.com>
Anatoly Burakov (16):
Separate igb_uio mapping into a separate file
Distinguish between legitimate failures and non-fatal errors
Rename RTE_PCI_DRV_NEED_IGB_UIO to RTE_PCI_DRV_NEED_MAPPING
Make igb_uio compilation optional
Moved interrupt type out of igb_uio
Add support for VFIO in Linuxapp targets
Add support for VFIO interrupts, add VFIO header
Add support for mapping devices through VFIO.
Enable VFIO device binding
Added support for selecting VFIO interrupt type from EAL command-line
Make --no-huge use mmap instead of malloc.
Adding unit tests for VFIO EAL command-line parameter
Removed PCI ID table from igb_uio
Renamed igb_uio_bind to dpdk_nic_bind
Added support for VFIO drivers in dpdk_nic_bind.py
Adding support for VFIO to setup.sh
app/test/test_eal_flags.c | 24 +
app/test/test_pci.c | 4 +-
config/defconfig_i686-default-linuxapp-gcc | 2 +
config/defconfig_i686-default-linuxapp-icc | 2 +
config/defconfig_x86_64-default-linuxapp-gcc | 2 +
config/defconfig_x86_64-default-linuxapp-icc | 2 +
lib/librte_eal/bsdapp/eal/eal_pci.c | 2 +-
lib/librte_eal/common/Makefile | 1 +
lib/librte_eal/common/eal_common_pci.c | 17 +-
lib/librte_eal/common/include/rte_pci.h | 7 +-
.../common/include/rte_pci_dev_feature_defs.h | 46 ++
.../common/include/rte_pci_dev_features.h | 42 ++
lib/librte_eal/linuxapp/Makefile | 2 +
lib/librte_eal/linuxapp/eal/Makefile | 6 +-
lib/librte_eal/linuxapp/eal/eal.c | 35 +
lib/librte_eal/linuxapp/eal/eal_interrupts.c | 203 +++++-
lib/librte_eal/linuxapp/eal/eal_memory.c | 8 +-
lib/librte_eal/linuxapp/eal/eal_pci.c | 480 ++------------
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 416 ++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 709 +++++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_vfio_socket.c | 367 +++++++++++
.../linuxapp/eal/include/eal_internal_cfg.h | 3 +
lib/librte_eal/linuxapp/eal/include/eal_pci_init.h | 120 ++++
lib/librte_eal/linuxapp/eal/include/eal_vfio.h | 55 ++
.../linuxapp/eal/include/exec-env/rte_interrupts.h | 7 +-
lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 70 +-
lib/librte_pmd_e1000/em_ethdev.c | 2 +-
lib/librte_pmd_e1000/igb_ethdev.c | 4 +-
lib/librte_pmd_ixgbe/ixgbe_ethdev.c | 4 +-
lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c | 2 +-
tools/dpdk_nic_bind.py | 500 +++++++++++++++
tools/igb_uio_bind.py | 485 --------------
tools/setup.sh | 168 ++++-
33 files changed, 2797 insertions(+), 1000 deletions(-)
create mode 100644 lib/librte_eal/common/include/rte_pci_dev_feature_defs.h
create mode 100644 lib/librte_eal/common/include/rte_pci_dev_features.h
create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_uio.c
create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio_socket.c
create mode 100644 lib/librte_eal/linuxapp/eal/include/eal_pci_init.h
create mode 100644 lib/librte_eal/linuxapp/eal/include/eal_vfio.h
create mode 100755 tools/dpdk_nic_bind.py
delete mode 100755 tools/igb_uio_bind.py
--
1.8.1.4
^ permalink raw reply [flat|nested] 3+ messages in thread
* [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding
2014-05-01 11:05 [dpdk-dev] [PATCH 00/16] [RFC] [VFIO] Add VFIO support to DPDK Burakov, Anatoly
@ 2014-05-19 15:51 ` Anatoly Burakov
2014-05-22 12:03 ` Thomas Monjalon
0 siblings, 1 reply; 3+ messages in thread
From: Anatoly Burakov @ 2014-05-19 15:51 UTC (permalink / raw)
To: dev
Add support for binding VFIO devices if RTE_PCI_DRV_NEED_IGB_UIO
is set for this driver. Try VFIO first, if not mapped then try
IGB_UIO too.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/librte_eal/linuxapp/eal/eal_pci.c | 55 ++++++++++++++++++++++++++++++++-
1 files changed, 54 insertions(+), 1 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index 7256406..953abe6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -401,6 +401,7 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct rte_pci_device *d
{
struct rte_pci_id *id_table;
int ret = 0;
+ int mapped = 0;
for (id_table = dr->id_table ; id_table->vendor_id != 0; id_table++) {
@@ -435,8 +436,17 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct rte_pci_device *d
}
if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
+ /* try mapping the NIC resources using VFIO if it exists */
+#ifdef VFIO_PRESENT
+ if (vfio_cfg.vfio_enabled) {
+ if ((ret = pci_vfio_map_resource(dev)) == 0)
+ mapped = 1;
+ else if (ret < 0)
+ return ret;
+ }
+#endif
/* map resources for devices that use igb_uio */
- if ((ret = pci_uio_map_resource(dev)) != 0)
+ if (!mapped && (ret = pci_uio_map_resource(dev)) != 0)
return ret;
} else if (dr->drv_flags & RTE_PCI_DRV_FORCE_UNBIND &&
rte_eal_process_type() == RTE_PROC_PRIMARY) {
@@ -471,5 +481,48 @@ rte_eal_pci_init(void)
RTE_LOG(ERR, EAL, "%s(): Cannot scan PCI bus\n", __func__);
return -1;
}
+#ifdef VFIO_PRESENT
+ memset(&vfio_cfg, 0, sizeof(vfio_cfg));
+
+ /* initialize group list */
+ int i, ret;
+
+ for (i = 0; i < VFIO_MAX_GROUPS; i++) {
+ vfio_cfg.vfio_groups[i].fd = -1;
+ vfio_cfg.vfio_groups[i].group_no = -1;
+ }
+ vfio_cfg.vfio_container_fd = -1;
+
+ /* check if we have VFIO driver enabled */
+ if (access(VFIO_DIR, F_OK) == 0) {
+ static int socket_fd;
+
+ vfio_cfg.vfio_enabled = 1;
+
+ /* if we are primary process, create a thread to communicate with
+ * secondary processes. the thread will use a socket to wait for
+ * requests from secondary process to send open file descriptors,
+ * because VFIO does not allow multiple open descriptors on a group or
+ * VFIO container.
+ */
+ if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ /* set up local socket */
+ if ((socket_fd = pci_vfio_socket_setup()) < 0) {
+ RTE_LOG(ERR, EAL, "Failed to set up local socket!\n");
+ return -1;
+ }
+ ret = pthread_create(&socket_thread, NULL,
+ pci_vfio_socket_thread, (void*) &socket_fd);
+ if (ret) {
+ RTE_LOG(ERR, EAL,
+ "Failed to create thread for communication with secondary "
+ "processes!\n");
+ return -1;
+ }
+ }
+ }
+ else
+ RTE_LOG(INFO, EAL, "VFIO driver not loaded or wrong permissions\n");
+#endif
return 0;
}
--
1.7.0.7
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding
2014-05-19 15:51 ` [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding Anatoly Burakov
@ 2014-05-22 12:03 ` Thomas Monjalon
0 siblings, 0 replies; 3+ messages in thread
From: Thomas Monjalon @ 2014-05-22 12:03 UTC (permalink / raw)
To: Anatoly Burakov; +Cc: dev
2014-05-19 16:51, Anatoly Burakov:
> Add support for binding VFIO devices if RTE_PCI_DRV_NEED_IGB_UIO
> is set for this driver. Try VFIO first, if not mapped then try
> IGB_UIO too.
You have renamed RTE_PCI_DRV_NEED_IGB_UIO. Please update this log :)
> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
> @@ -401,6 +401,7 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr,
> struct rte_pci_device *d {
> struct rte_pci_id *id_table;
> int ret = 0;
> + int mapped = 0;
>
> for (id_table = dr->id_table ; id_table->vendor_id != 0; id_table++) {
>
> @@ -435,8 +436,17 @@ rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr,
> struct rte_pci_device *d }
>
> if (dr->drv_flags & RTE_PCI_DRV_NEED_MAPPING) {
> + /* try mapping the NIC resources using VFIO if it exists */
> +#ifdef VFIO_PRESENT
> + if (vfio_cfg.vfio_enabled) {
> + if ((ret = pci_vfio_map_resource(dev)) == 0)
> + mapped = 1;
> + else if (ret < 0)
> + return ret;
> + }
> +#endif
> /* map resources for devices that use igb_uio */
> - if ((ret = pci_uio_map_resource(dev)) != 0)
> + if (!mapped && (ret = pci_uio_map_resource(dev)) != 0)
> return ret;
I think creating a function pci_map_resource() could be cleaner (you won't
need variable mapped).
> +#ifdef VFIO_PRESENT
> + memset(&vfio_cfg, 0, sizeof(vfio_cfg));
> +
> + /* initialize group list */
> + int i, ret;
> +
> + for (i = 0; i < VFIO_MAX_GROUPS; i++) {
> + vfio_cfg.vfio_groups[i].fd = -1;
> + vfio_cfg.vfio_groups[i].group_no = -1;
> + }
> + vfio_cfg.vfio_container_fd = -1;
> +
> + /* check if we have VFIO driver enabled */
> + if (access(VFIO_DIR, F_OK) == 0) {
> + static int socket_fd;
> +
> + vfio_cfg.vfio_enabled = 1;
> +
> + /* if we are primary process, create a thread to communicate with
> + * secondary processes. the thread will use a socket to wait for
> + * requests from secondary process to send open file descriptors,
> + * because VFIO does not allow multiple open descriptors on a group
or
> + * VFIO container.
> + */
> + if (internal_config.process_type == RTE_PROC_PRIMARY) {
> + /* set up local socket */
> + if ((socket_fd = pci_vfio_socket_setup()) < 0) {
> + RTE_LOG(ERR, EAL, "Failed to set up local socket!\n");
> + return -1;
> + }
> + ret = pthread_create(&socket_thread, NULL,
> + pci_vfio_socket_thread, (void*) &socket_fd);
> + if (ret) {
> + RTE_LOG(ERR, EAL,
> + "Failed to create thread for communication with secondary
"
> + "processes!\n");
> + return -1;
> + }
> + }
Also here, it could help to have a dedicated function for vfio init.
> + }
> + else
checkpatch.pl reports an error: "else should follow close brace '}'"
> + RTE_LOG(INFO, EAL, "VFIO driver not loaded or wrong permissions\n");
> +#endif
Thanks
--
Thomas
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2014-05-27 3:23 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-05-27 3:23 [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding Xu, HuilongX
-- strict thread matches above, loose matches on Subject: below --
2014-05-01 11:05 [dpdk-dev] [PATCH 00/16] [RFC] [VFIO] Add VFIO support to DPDK Burakov, Anatoly
2014-05-19 15:51 ` [dpdk-dev] [PATCH v2 09/16] Enable VFIO device binding Anatoly Burakov
2014-05-22 12:03 ` Thomas Monjalon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).