* [dpdk-dev] [PATCH v3 01/12] virtio: Introduce config RTE_VIRTIO_INC_VECTOR
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
@ 2016-01-07 16:32 ` Santosh Shukla
2016-01-07 16:32 ` [dpdk-dev] [PATCH v3 02/12] config: i686: set RTE_VIRTIO_INC_VECTOR=n Santosh Shukla
` (10 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:32 UTC (permalink / raw)
To: dev
virtio_recv_pkts_vec and other virtio vector friend apis are written for sse/avx
instructions. For arm64 in particular, virtio vector implementation does not
exist(todo).
So virtio pmd driver wont build for targets like i686, arm64. By making
RTE_VIRTIO_INC_VECTOR=n, Driver can build for non-sse/avx targets and will work
in non-vectored virtio mode.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
config/common_linuxapp | 1 +
drivers/net/virtio/Makefile | 2 +-
drivers/net/virtio/virtio_rxtx.c | 7 +++++++
3 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/config/common_linuxapp b/config/common_linuxapp
index 74bc515..8677697 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -274,6 +274,7 @@ CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n
CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DUMP=n
+CONFIG_RTE_VIRTIO_INC_VECTOR=y
#
# Compile burst-oriented VMXNET3 PMD driver
diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile
index 43835ba..25a842d 100644
--- a/drivers/net/virtio/Makefile
+++ b/drivers/net/virtio/Makefile
@@ -50,7 +50,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtqueue.c
SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_pci.c
SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx.c
SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_ethdev.c
-SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio_rxtx_simple.c
+SRCS-$(CONFIG_RTE_VIRTIO_INC_VECTOR) += virtio_rxtx_simple.c
# this lib depends upon:
DEPDIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += lib/librte_eal lib/librte_ether
diff --git a/drivers/net/virtio/virtio_rxtx.c b/drivers/net/virtio/virtio_rxtx.c
index 74b39ef..23be1ff 100644
--- a/drivers/net/virtio/virtio_rxtx.c
+++ b/drivers/net/virtio/virtio_rxtx.c
@@ -438,7 +438,9 @@ virtio_dev_rx_queue_setup(struct rte_eth_dev *dev,
dev->data->rx_queues[queue_idx] = vq;
+#ifdef RTE_VIRTIO_INC_VECTOR
virtio_rxq_vec_setup(vq);
+#endif
return 0;
}
@@ -464,7 +466,10 @@ virtio_dev_tx_queue_setup(struct rte_eth_dev *dev,
const struct rte_eth_txconf *tx_conf)
{
uint8_t vtpci_queue_idx = 2 * queue_idx + VTNET_SQ_TQ_QUEUE_IDX;
+
+#ifdef RTE_VIRTIO_INC_VECTOR
struct virtio_hw *hw = dev->data->dev_private;
+#endif
struct virtqueue *vq;
uint16_t tx_free_thresh;
int ret;
@@ -477,6 +482,7 @@ virtio_dev_tx_queue_setup(struct rte_eth_dev *dev,
return -EINVAL;
}
+#ifdef RTE_VIRTIO_INC_VECTOR
/* Use simple rx/tx func if single segment and no offloads */
if ((tx_conf->txq_flags & VIRTIO_SIMPLE_FLAGS) == VIRTIO_SIMPLE_FLAGS &&
!vtpci_with_feature(hw, VIRTIO_NET_F_MRG_RXBUF)) {
@@ -485,6 +491,7 @@ virtio_dev_tx_queue_setup(struct rte_eth_dev *dev,
dev->rx_pkt_burst = virtio_recv_pkts_vec;
use_simple_rxtx = 1;
}
+#endif
ret = virtio_dev_queue_setup(dev, VTNET_TQ, queue_idx, vtpci_queue_idx,
nb_desc, socket_id, &vq);
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 02/12] config: i686: set RTE_VIRTIO_INC_VECTOR=n
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
2016-01-07 16:32 ` [dpdk-dev] [PATCH v3 01/12] virtio: Introduce config RTE_VIRTIO_INC_VECTOR Santosh Shukla
@ 2016-01-07 16:32 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init() Santosh Shukla
` (9 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:32 UTC (permalink / raw)
To: dev
i686 target config example:
config/defconfig_i686-native-linuxapp-gcc says "Vectorized PMD is not supported
on 32-bit".
So setting RTE_VIRTIO_INC_VECTOR to 'n'.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
config/defconfig_i686-native-linuxapp-gcc | 1 +
config/defconfig_i686-native-linuxapp-icc | 1 +
2 files changed, 2 insertions(+)
diff --git a/config/defconfig_i686-native-linuxapp-gcc b/config/defconfig_i686-native-linuxapp-gcc
index a90de9b..a4b1c49 100644
--- a/config/defconfig_i686-native-linuxapp-gcc
+++ b/config/defconfig_i686-native-linuxapp-gcc
@@ -49,3 +49,4 @@ CONFIG_RTE_LIBRTE_KNI=n
# Vectorized PMD is not supported on 32-bit
#
CONFIG_RTE_IXGBE_INC_VECTOR=n
+CONFIG_RTE_VIRTIO_INC_VECTOR=n
diff --git a/config/defconfig_i686-native-linuxapp-icc b/config/defconfig_i686-native-linuxapp-icc
index c021321..f8eb6ad 100644
--- a/config/defconfig_i686-native-linuxapp-icc
+++ b/config/defconfig_i686-native-linuxapp-icc
@@ -49,3 +49,4 @@ CONFIG_RTE_LIBRTE_KNI=n
# Vectorized PMD is not supported on 32-bit
#
CONFIG_RTE_IXGBE_INC_VECTOR=n
+CONFIG_RTE_VIRTIO_INC_VECTOR=n
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init()
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
2016-01-07 16:32 ` [dpdk-dev] [PATCH v3 01/12] virtio: Introduce config RTE_VIRTIO_INC_VECTOR Santosh Shukla
2016-01-07 16:32 ` [dpdk-dev] [PATCH v3 02/12] config: i686: set RTE_VIRTIO_INC_VECTOR=n Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 18:14 ` Stephen Hemminger
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region Santosh Shukla
` (8 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
iopl() syscall not supported in linux-arm/arm64 so always return 0 value.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
Acked-by: Jan Viktorin <viktorin@rehivetech.com>
---
lib/librte_eal/linuxapp/eal/eal.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 635ec36..2617037 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -716,6 +716,9 @@ rte_eal_iopl_init(void)
return -1;
return 0;
#else
+#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
+ return 0; /* iopl syscall not supported for ARM/ARM64 */
+#endif
return -1;
#endif
}
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init()
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init() Santosh Shukla
@ 2016-01-07 18:14 ` Stephen Hemminger
2016-01-09 13:18 ` Santosh Shukla
0 siblings, 1 reply; 29+ messages in thread
From: Stephen Hemminger @ 2016-01-07 18:14 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
On Thu, 7 Jan 2016 22:03:00 +0530
Santosh Shukla <sshukla@mvista.com> wrote:
> #else
> +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
> + return 0; /* iopl syscall not supported for ARM/ARM64 */
> +#endif
> return -1;
> #endif
Minor net why not:
#elif defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
return -1
#else
That way you won't generate two return statements and potentially
trigger warnings from static checkers.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init()
2016-01-07 18:14 ` Stephen Hemminger
@ 2016-01-09 13:18 ` Santosh Shukla
0 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-09 13:18 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Thu, Jan 7, 2016 at 11:44 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Thu, 7 Jan 2016 22:03:00 +0530
> Santosh Shukla <sshukla@mvista.com> wrote:
>
>> #else
>> +#if defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
>> + return 0; /* iopl syscall not supported for ARM/ARM64 */
>> +#endif
>> return -1;
>> #endif
>
> Minor net why not:
>
> #elif defined(RTE_ARCH_ARM) || defined(RTE_ARCH_ARM64)
> return -1
> #else
>
> That way you won't generate two return statements and potentially
> trigger warnings from static checkers.
returning -1 would fail for arm/arm64. I guess you meant return 0,
right? if so then would need one more return for non-x86/non-arm case.
Also I am working on another patchset suggested by Jerin [1] on iopl()
in v2 series, That new patchset intended to get rid-off ifdef X_86
clutter for sys/io.h and more iop() definition to arch/platform file.
I don't want to include those changes in v3 series as because it seems
like two different topic.
[1] http://dpdk.org/dev/patchwork/patch/9533/
^ permalink raw reply [flat|nested] 29+ messages in thread
* [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (2 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 03/12] linuxapp: eal: arm: Always return 0 for rte_eal_iopl_init() Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 18:16 ` Stephen Hemminger
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 05/12] virtio_pci.h: build fix for sys/io.h for non-x86 arch Santosh Shukla
` (7 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
vfio_pci_mmap() try to map all pci bars. ioport region ar not mapped in
vfio/kernel so ignore mmaping for ioport.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index 74f91ba..4077eb6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -760,6 +760,26 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
return -1;
}
+ /* chk for io port region */
+ uint32_t ioport_bar;
+ ret = pread64(vfio_dev_fd, &ioport_bar, sizeof(ioport_bar),
+ VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX)
+ + PCI_BASE_ADDRESS_0 + i*4);
+
+ if (ret != sizeof(ioport_bar)) {
+ RTE_LOG(ERR, EAL,
+ "Cannot read command (%x) from PCI config"
+ "space!\n", PCI_BASE_ADDRESS_0 + i*4);
+ return -1;
+ }
+
+ if (ioport_bar & PCI_BASE_ADDRESS_SPACE_IO) {
+ RTE_LOG(INFO, EAL, "\tIgnore mapping since Its a i/o"
+ "port bar (%d) addr : %x\n", i,
+ ioport_bar);
+ continue;
+ }
+
/* skip non-mmapable BARs */
if ((reg.flags & VFIO_REGION_INFO_FLAG_MMAP) == 0)
continue;
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region Santosh Shukla
@ 2016-01-07 18:16 ` Stephen Hemminger
2016-01-07 18:53 ` Santosh Shukla
0 siblings, 1 reply; 29+ messages in thread
From: Stephen Hemminger @ 2016-01-07 18:16 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
This looks like the right thing to do. Minor nits.
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index 74f91ba..4077eb6 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -760,6 +760,26 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
> return -1;
> }
>
> + /* chk for io port region */
> + uint32_t ioport_bar;
In general DPDK has followed the kernel practice of putting declarations
at the start of function/basic block. It is ok by me, but just noting that
the rest of the code doesn't do it.
> + ret = pread64(vfio_dev_fd, &ioport_bar, sizeof(ioport_bar),
> + VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX)
> + + PCI_BASE_ADDRESS_0 + i*4);
> +
> + if (ret != sizeof(ioport_bar)) {
> + RTE_LOG(ERR, EAL,
> + "Cannot read command (%x) from PCI config"
> + "space!\n", PCI_BASE_ADDRESS_0 + i*4);
Please dont split the line of a log message string in mid sentence.
> + return -1;
> + }
> +
> + if (ioport_bar & PCI_BASE_ADDRESS_SPACE_IO) {
> + RTE_LOG(INFO, EAL, "\tIgnore mapping since Its a i/o"
> + "port bar (%d) addr : %x\n", i,
same here
> + ioport_bar);
> + continue;
> + }
> +
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region
2016-01-07 18:16 ` Stephen Hemminger
@ 2016-01-07 18:53 ` Santosh Shukla
2016-01-08 7:29 ` Yuanhan Liu
0 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 18:53 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Thu, Jan 7, 2016 at 11:46 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:
> This looks like the right thing to do. Minor nits.
>
> > diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > index 74f91ba..4077eb6 100644
> > --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> > @@ -760,6 +760,26 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
> > return -1;
> > }
> >
> > + /* chk for io port region */
> > + uint32_t ioport_bar;
>
> In general DPDK has followed the kernel practice of putting declarations
> at the start of function/basic block. It is ok by me, but just noting that
> the rest of the code doesn't do it.
>
>
My bad, Thanks!
> > + ret = pread64(vfio_dev_fd, &ioport_bar, sizeof(ioport_bar),
> > +
> VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX)
> > + + PCI_BASE_ADDRESS_0 + i*4);
> > +
> > + if (ret != sizeof(ioport_bar)) {
> > + RTE_LOG(ERR, EAL,
> > + "Cannot read command (%x) from PCI config"
> > + "space!\n", PCI_BASE_ADDRESS_0 + i*4);
>
> Please dont split the line of a log message string in mid sentence.
>
>
me to don't like splitting, This was deliberate to keep checkpatch happy,
If we are ok with debug message > 80 line warning I guess it will improve
code readability.
> + return -1;
> > + }
> > +
> > + if (ioport_bar & PCI_BASE_ADDRESS_SPACE_IO) {
> > + RTE_LOG(INFO, EAL, "\tIgnore mapping since Its a
> i/o"
> > + "port bar (%d) addr : %x\n", i,
> same here
>
> Agreed.
> > + ioport_bar);
> > + continue;
> > + }
> > +
>
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region
2016-01-07 18:53 ` Santosh Shukla
@ 2016-01-08 7:29 ` Yuanhan Liu
0 siblings, 0 replies; 29+ messages in thread
From: Yuanhan Liu @ 2016-01-08 7:29 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
On Fri, Jan 08, 2016 at 12:23:15AM +0530, Santosh Shukla wrote:
>
>
> On Thu, Jan 7, 2016 at 11:46 PM, Stephen Hemminger <stephen@networkplumber.org>
> wrote:
>
>
> > + ret = pread64(vfio_dev_fd, &ioport_bar, sizeof(ioport_bar),
> > + VFIO_GET_REGION_ADDR
> (VFIO_PCI_CONFIG_REGION_INDEX)
> > + + PCI_BASE_ADDRESS_0 + i*4);
> > +
> > + if (ret != sizeof(ioport_bar)) {
> > + RTE_LOG(ERR, EAL,
> > + "Cannot read command (%x) from PCI config"
> > + "space!\n", PCI_BASE_ADDRESS_0 + i*4);
>
> Please dont split the line of a log message string in mid sentence.
>
First of all, it'd be good if you can fix your email client to reply
emails in a way open source world prefer, such as prefixing last email
with leading '> ', and do not auto fold lines.
> me to don't like splitting, This was deliberate to keep checkpatch happy, If we
> are ok with debug message > 80 line warning I guess it will improve code
> readability.
You may try to shorten your message to "Cannot read cmd %x from config
space!" :)
>
>
> > + return -1;
> > + }
> > +
> > + if (ioport_bar & PCI_BASE_ADDRESS_SPACE_IO) {
> > + RTE_LOG(INFO, EAL, "\tIgnore mapping since Its a i/
> o"
> > + "port bar (%d) addr : %x\n", i,
> same here
>
>
> Agreed.
Ditto, maybe following is better
"Ignore mapping IO port bar %d, addr: %x\n".
--yliu
^ permalink raw reply [flat|nested] 29+ messages in thread
* [dpdk-dev] [PATCH v3 05/12] virtio_pci.h: build fix for sys/io.h for non-x86 arch
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (3 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 04/12] linuxapp/vfio: ignore mapping for ioport region Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space Santosh Shukla
` (6 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
make sure sys/io.h used only for x86 archs. This fixes build error
arm64/arm case.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
drivers/net/virtio/virtio_pci.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index 47f722a..8b5b031 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -40,8 +40,10 @@
#include <sys/types.h>
#include <machine/cpufunc.h>
#else
+#if defined(RTE_ARCH_X86_64) || defined(RTE_ARCH_I686)
#include <sys/io.h>
#endif
+#endif
#include <rte_ethdev.h>
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (4 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 05/12] virtio_pci.h: build fix for sys/io.h for non-x86 arch Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 18:19 ` Stephen Hemminger
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 07/12] virtio: vfio: add api support to rd/wr ioport bar Santosh Shukla
` (5 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
Introducing below api for pci bar space rd/wr. Currently used for
pci iobar rd/wr.
Api's are:
- rte_eal_pci_read_bar
- rte_eal_pci_write_bar
virtio when used for vfio-mode then virtio driver will use these api
to do rd/wr operation on ioport pci bar.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
lib/librte_eal/common/include/rte_pci.h | 38 ++++++++++++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci.c | 28 ++++++++++++++++++++
lib/librte_eal/linuxapp/eal/eal_pci_init.h | 6 +++++
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 28 ++++++++++++++++++++
4 files changed, 100 insertions(+)
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index 334c12e..53437cc 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -471,6 +471,44 @@ int rte_eal_pci_read_config(const struct rte_pci_device *device,
void *buf, size_t len, off_t offset);
/**
+ * Read PCI bar space.
+ *
+ * @param device
+ * A pointer to a rte_pci_device structure describing the device
+ * to use
+ * @param buf
+ * A data buffer where the bytes should be read into
+ * @param len
+ * The length of the data buffer.
+ * @param offset
+ * The offset into PCI bar space
+ * @param bar_idx
+ * The pci bar index (valid range is 0..5)
+ */
+int rte_eal_pci_read_bar(const struct rte_pci_device *device,
+ void *buf, size_t len, off_t offset, int bar_idx);
+
+/**
+ * Write PCI bar space.
+ *
+ * @param device
+ * A pointer to a rte_pci_device structure describing the device
+ * to use
+ * @param buf
+ * A data buffer containing the bytes should be written
+ * @param len
+ * The length of the data buffer.
+ * @param offset
+ * The offset into PCI config space
+ * @param bar_idx
+ * The pci bar index (valid range is 0..5)
+*/
+int rte_eal_pci_write_bar(const struct rte_pci_device *device,
+ const void *buf, size_t len, off_t offset,
+ int bar_idx);
+
+
+/**
* Write PCI config space.
*
* @param device
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c b/lib/librte_eal/linuxapp/eal/eal_pci.c
index bc5b5be..22d3321 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci.c
@@ -621,6 +621,34 @@ int rte_eal_pci_write_config(const struct rte_pci_device *device,
}
}
+int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
+ void *buf __rte_unused,
+ size_t len __rte_unused,
+ off_t offset __rte_unused,
+ int bar_idx __rte_unused)
+{
+#ifdef VFIO_PRESENT
+ const struct rte_intr_handle *intr_handle = &device->intr_handle;
+ return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
+#else
+ return 0; /* UIO's not applicable */
+#endif
+}
+
+int rte_eal_pci_write_bar(const struct rte_pci_device *device __rte_unused,
+ const void *buf __rte_unused,
+ size_t len __rte_unused,
+ off_t offset __rte_unused,
+ int bar_idx __rte_unused)
+{
+#ifdef VFIO_PRESENT
+ const struct rte_intr_handle *intr_handle = &device->intr_handle;
+ return pci_vfio_write_bar(intr_handle, buf, len, offset, bar_idx);
+#else
+ return 0; /* UIO's not applicable */
+#endif
+}
+
/* Init the PCI EAL subsystem */
int
rte_eal_pci_init(void)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/eal_pci_init.h
index a17c708..3bc592b 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_init.h
@@ -68,6 +68,12 @@ int pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
int pci_vfio_write_config(const struct rte_intr_handle *intr_handle,
const void *buf, size_t len, off_t offs);
+int pci_vfio_read_bar(const struct rte_intr_handle *intr_handle,
+ void *buf, size_t len, off_t offs, int bar_idx);
+
+int pci_vfio_write_bar(const struct rte_intr_handle *intr_handle,
+ const void *buf, size_t len, off_t offs, int bar_idx);
+
/* map VFIO resource prototype */
int pci_vfio_map_resource(struct rte_pci_device *dev);
int pci_vfio_get_group_fd(int iommu_group_fd);
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index 4077eb6..7ec203c 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -93,6 +93,34 @@ pci_vfio_write_config(const struct rte_intr_handle *intr_handle,
VFIO_GET_REGION_ADDR(VFIO_PCI_CONFIG_REGION_INDEX) + offs);
}
+int
+pci_vfio_read_bar(const struct rte_intr_handle *intr_handle,
+ void *buf, size_t len, off_t offs, int bar_idx)
+{
+ if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX
+ || bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {
+ RTE_LOG(ERR, EAL, "invalid bar_idx!\n");
+ return -1;
+ }
+
+ return pread64(intr_handle->vfio_dev_fd, buf, len,
+ VFIO_GET_REGION_ADDR(bar_idx) + offs);
+}
+
+int
+pci_vfio_write_bar(const struct rte_intr_handle *intr_handle,
+ const void *buf, size_t len, off_t offs, int bar_idx)
+{
+ if (bar_idx < VFIO_PCI_BAR0_REGION_INDEX
+ || bar_idx > VFIO_PCI_BAR5_REGION_INDEX) {
+ RTE_LOG(ERR, EAL, "invalid bar_idx!\n");
+ return -1;
+ }
+
+ return pwrite64(intr_handle->vfio_dev_fd, buf, len,
+ VFIO_GET_REGION_ADDR(bar_idx) + offs);
+}
+
/* get PCI BAR number where MSI-X interrupts are */
static int
pci_vfio_get_msix_bar(int fd, int *msix_bar, uint32_t *msix_table_offset,
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space Santosh Shukla
@ 2016-01-07 18:19 ` Stephen Hemminger
2016-01-07 18:26 ` Wiles, Keith
` (2 more replies)
0 siblings, 3 replies; 29+ messages in thread
From: Stephen Hemminger @ 2016-01-07 18:19 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
On Thu, 7 Jan 2016 22:03:03 +0530
Santosh Shukla <sshukla@mvista.com> wrote:
>
> +int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
> + void *buf __rte_unused,
> + size_t len __rte_unused,
> + off_t offset __rte_unused,
> + int bar_idx __rte_unused)
> +{
> +#ifdef VFIO_PRESENT
> + const struct rte_intr_handle *intr_handle = &device->intr_handle;
> + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> +#else
> + return 0; /* UIO's not applicable */
> +#endif
> +}
It seems wrong to declare all the parameters as unused but then use them.
Maybe there is a way to have a macro for USED(x) in the #else case
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 18:19 ` Stephen Hemminger
@ 2016-01-07 18:26 ` Wiles, Keith
2016-01-07 18:39 ` Stephen Hemminger
2016-01-07 18:46 ` Santosh Shukla
2016-01-13 14:47 ` Bruce Richardson
2 siblings, 1 reply; 29+ messages in thread
From: Wiles, Keith @ 2016-01-07 18:26 UTC (permalink / raw)
To: Stephen Hemminger, Santosh Shukla; +Cc: dev
On 1/7/16, 12:19 PM, "dev on behalf of Stephen Hemminger" <dev-bounces@dpdk.org on behalf of stephen@networkplumber.org> wrote:
>On Thu, 7 Jan 2016 22:03:03 +0530
>Santosh Shukla <sshukla@mvista.com> wrote:
>
>>
>> +int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
>> + void *buf __rte_unused,
>> + size_t len __rte_unused,
>> + off_t offset __rte_unused,
>> + int bar_idx __rte_unused)
>> +{
>> +#ifdef VFIO_PRESENT
>> + const struct rte_intr_handle *intr_handle = &device->intr_handle;
>> + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
>> +#else
>> + return 0; /* UIO's not applicable */
>> +#endif
>> +}
>
>It seems wrong to declare all the parameters as unused but then use them.
>Maybe there is a way to have a macro for USED(x) in the #else case
I would suggest we create a macro '#define RTE_UNUSED(x) ((void)x)’, unless we have one and I missed it groping though the code.
>
Regards,
Keith
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 18:26 ` Wiles, Keith
@ 2016-01-07 18:39 ` Stephen Hemminger
2016-01-07 18:48 ` Santosh Shukla
0 siblings, 1 reply; 29+ messages in thread
From: Stephen Hemminger @ 2016-01-07 18:39 UTC (permalink / raw)
To: Wiles, Keith; +Cc: dev
On Thu, 7 Jan 2016 18:26:30 +0000
"Wiles, Keith" <keith.wiles@intel.com> wrote:
> On 1/7/16, 12:19 PM, "dev on behalf of Stephen Hemminger" <dev-bounces@dpdk.org on behalf of stephen@networkplumber.org> wrote:
>
> >On Thu, 7 Jan 2016 22:03:03 +0530
> >Santosh Shukla <sshukla@mvista.com> wrote:
> >
> >>
> >> +int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
> >> + void *buf __rte_unused,
> >> + size_t len __rte_unused,
> >> + off_t offset __rte_unused,
> >> + int bar_idx __rte_unused)
> >> +{
> >> +#ifdef VFIO_PRESENT
> >> + const struct rte_intr_handle *intr_handle = &device->intr_handle;
> >> + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> >> +#else
> >> + return 0; /* UIO's not applicable */
> >> +#endif
> >> +}
> >
> >It seems wrong to declare all the parameters as unused but then use them.
> >Maybe there is a way to have a macro for USED(x) in the #else case
>
> I would suggest we create a macro '#define RTE_UNUSED(x) ((void)x)’, unless we have one and I missed it groping though the code.
> >
>
>
> Regards,
> Keith
>
>
>
>
Or just move the #ifdef outside the function and have two versions
#ifdef VFIO_PRESENT
int rte_eal_pci_read_bar(const struct rte_pci_device *device,
void *buf, size_t len,
off_t offset, int bar_idx)
{
const struct rte_intr_handle *intr_handle = &device->intr_handle;
return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
}
}
#else
int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
void *buf __rte_unused,
size_t len __rte_unused,
off_t offset __rte_unused,
int bar_idx __rte_unused)
{
return 0; /* UIO's not applicable */
}
#endif
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 18:39 ` Stephen Hemminger
@ 2016-01-07 18:48 ` Santosh Shukla
0 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 18:48 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Fri, Jan 8, 2016 at 12:09 AM, Stephen Hemminger <
stephen@networkplumber.org> wrote:
> On Thu, 7 Jan 2016 18:26:30 +0000
> "Wiles, Keith" <keith.wiles@intel.com> wrote:
>
> > On 1/7/16, 12:19 PM, "dev on behalf of Stephen Hemminger" <
> dev-bounces@dpdk.org on behalf of stephen@networkplumber.org> wrote:
> >
> > >On Thu, 7 Jan 2016 22:03:03 +0530
> > >Santosh Shukla <sshukla@mvista.com> wrote:
> > >
> > >>
> > >> +int rte_eal_pci_read_bar(const struct rte_pci_device *device
> __rte_unused,
> > >> + void *buf __rte_unused,
> > >> + size_t len __rte_unused,
> > >> + off_t offset __rte_unused,
> > >> + int bar_idx __rte_unused)
> > >> +{
> > >> +#ifdef VFIO_PRESENT
> > >> + const struct rte_intr_handle *intr_handle = &device->intr_handle;
> > >> + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> > >> +#else
> > >> + return 0; /* UIO's not applicable */
> > >> +#endif
> > >> +}
> > >
> > >It seems wrong to declare all the parameters as unused but then use
> them.
> > >Maybe there is a way to have a macro for USED(x) in the #else case
> >
> > I would suggest we create a macro '#define RTE_UNUSED(x) ((void)x)’,
> unless we have one and I missed it groping though the code.
> > >
> >
> >
> > Regards,
> > Keith
> >
> >
> >
> >
>
> Or just move the #ifdef outside the function and have two versions
>
> #ifdef VFIO_PRESENT
> int rte_eal_pci_read_bar(const struct rte_pci_device *device,
> void *buf, size_t len,
> off_t offset, int bar_idx)
> {
> const struct rte_intr_handle *intr_handle = &device->intr_handle;
>
> return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> }
> }
> #else
> int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
> void *buf __rte_unused,
> size_t len __rte_unused,
> off_t offset __rte_unused,
> int bar_idx __rte_unused)
> {
> return 0; /* UIO's not applicable */
> }
> #endif
>
I'll pick this one!, Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 18:19 ` Stephen Hemminger
2016-01-07 18:26 ` Wiles, Keith
@ 2016-01-07 18:46 ` Santosh Shukla
2016-01-13 14:47 ` Bruce Richardson
2 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 18:46 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Thu, Jan 7, 2016 at 11:49 PM, Stephen Hemminger <
stephen@networkplumber.org> wrote:
> On Thu, 7 Jan 2016 22:03:03 +0530
> Santosh Shukla <sshukla@mvista.com> wrote:
>
> >
> > +int rte_eal_pci_read_bar(const struct rte_pci_device *device
> __rte_unused,
> > + void *buf __rte_unused,
> > + size_t len __rte_unused,
> > + off_t offset __rte_unused,
> > + int bar_idx __rte_unused)
> > +{
> > +#ifdef VFIO_PRESENT
> > + const struct rte_intr_handle *intr_handle = &device->intr_handle;
> > + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> > +#else
> > + return 0; /* UIO's not applicable */
> > +#endif
> > +}
>
> It seems wrong to declare all the parameters as unused but then use them.
> Maybe there is a way to have a macro for USED(x) in the #else case
>
Yes, I followed such practice in v2 series but missed on v3, Sorry for that
and we'll take care in v4.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space
2016-01-07 18:19 ` Stephen Hemminger
2016-01-07 18:26 ` Wiles, Keith
2016-01-07 18:46 ` Santosh Shukla
@ 2016-01-13 14:47 ` Bruce Richardson
2 siblings, 0 replies; 29+ messages in thread
From: Bruce Richardson @ 2016-01-13 14:47 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Thu, Jan 07, 2016 at 10:19:25AM -0800, Stephen Hemminger wrote:
> On Thu, 7 Jan 2016 22:03:03 +0530
> Santosh Shukla <sshukla@mvista.com> wrote:
>
> >
> > +int rte_eal_pci_read_bar(const struct rte_pci_device *device __rte_unused,
> > + void *buf __rte_unused,
> > + size_t len __rte_unused,
> > + off_t offset __rte_unused,
> > + int bar_idx __rte_unused)
> > +{
> > +#ifdef VFIO_PRESENT
> > + const struct rte_intr_handle *intr_handle = &device->intr_handle;
> > + return pci_vfio_read_bar(intr_handle, buf, len, offset, bar_idx);
> > +#else
> > + return 0; /* UIO's not applicable */
> > +#endif
> > +}
>
> It seems wrong to declare all the parameters as unused but then use them.
> Maybe there is a way to have a macro for USED(x) in the #else case
There is RTE_SET_USED() in rte_common.h.
I'd prefer that macro used than the option of duplicating the entire function
which really pads out the code.
/Bruce
^ permalink raw reply [flat|nested] 29+ messages in thread
* [dpdk-dev] [PATCH v3 07/12] virtio: vfio: add api support to rd/wr ioport bar
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (5 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 06/12] eal: pci: vfio: add rd/wr func for pci bar space Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 08/12] virtio: Add capability to initialize driver for vfio interface Santosh Shukla
` (4 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev; +Cc: Rakesh Krishnamurthy, Rizwan Ansari
For vfio case - Use pread/pwrite api to access virtio
ioport space.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
Signed-off-by: Rizwan Ansari <ransari@mvista.com>
Signed-off-by: Rakesh Krishnamurthy <rakeshk@mvista.com>
---
drivers/net/virtio/virtio_vfio.h | 146 ++++++++++++++++++++++++++++++++++++++
1 file changed, 146 insertions(+)
create mode 100644 drivers/net/virtio/virtio_vfio.h
diff --git a/drivers/net/virtio/virtio_vfio.h b/drivers/net/virtio/virtio_vfio.h
new file mode 100644
index 0000000..fd94b30
--- /dev/null
+++ b/drivers/net/virtio/virtio_vfio.h
@@ -0,0 +1,146 @@
+/*
+ * BSD LICENSE
+ *
+ * Copyright(c) 2015 Cavium Networks. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#ifndef _VIRTIO_VFIO_H_
+#define _VIRTIO_VFIO_H_
+
+#ifdef RTE_EAL_VFIO
+#include <stdio.h>
+#include <stdint.h>
+#include <rte_pci.h>
+
+#include "virtio_logs.h"
+
+static inline void ioport_inb(const struct rte_pci_device *pci_dev,
+ uint8_t reg, uint8_t *val)
+{
+ if (rte_eal_pci_read_bar(pci_dev, (uint8_t *)val, sizeof(uint8_t), reg,
+ 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+static inline void ioport_inw(const struct rte_pci_device *pci_dev,
+ uint16_t reg, uint16_t *val)
+{
+ if (rte_eal_pci_read_bar(pci_dev, (uint16_t *)val, sizeof(uint16_t),
+ reg, 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+static inline void ioport_inl(const struct rte_pci_device *pci_dev,
+ uint32_t reg, uint32_t *val)
+{
+ if (rte_eal_pci_read_bar(pci_dev, (uint32_t *)val, sizeof(uint32_t),
+ reg, 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+static inline void ioport_outb_p(const struct rte_pci_device *pci_dev,
+ uint8_t reg, uint8_t val)
+{
+ if (rte_eal_pci_write_bar(pci_dev, (uint8_t *)&val, sizeof(uint8_t),
+ reg, 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+
+static inline void ioport_outw_p(const struct rte_pci_device *pci_dev,
+ uint16_t reg, uint16_t val)
+{
+ if (rte_eal_pci_write_bar(pci_dev, (uint16_t *)&val, sizeof(uint16_t),
+ reg, 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+
+static inline void ioport_outl_p(const struct rte_pci_device *pci_dev,
+ uint32_t reg, uint32_t val)
+{
+ if (rte_eal_pci_write_bar(pci_dev, (uint32_t *)&val, sizeof(uint32_t),
+ reg, 0) <= 0) {
+ PMD_DRV_LOG(ERR, "Can't read from PCI bar space");
+ return;
+ }
+}
+
+#define VIRTIO_READ_REG_1(hw, reg) \
+({ \
+ uint8_t ret; \
+ ioport_inb(((hw)->pci_dev), reg, &ret); \
+ ret; \
+})
+
+#define VIRTIO_WRITE_REG_1(hw, reg, val) \
+({ \
+ ioport_outb_p(((hw)->pci_dev), reg, (uint8_t)val); \
+})
+
+#define VIRTIO_READ_REG_2(hw, reg) \
+({ \
+ uint16_t ret; \
+ ioport_inw(((hw)->pci_dev), reg, &ret); \
+ ret; \
+})
+
+#define VIRTIO_WRITE_REG_2(hw, reg, val) \
+({ \
+ ioport_outw_p(((hw)->pci_dev), reg, (uint16_t)(val));\
+})
+
+#define VIRTIO_READ_REG_4(hw, reg) \
+({ \
+ uint32_t ret; \
+ ioport_inl(((hw)->pci_dev), reg, &ret); \
+ ret; \
+})
+
+#define VIRTIO_WRITE_REG_4(hw, reg, val) \
+({ \
+ ioport_outl_p(((hw)->pci_dev), reg, (uint32_t)(val));\
+})
+
+#else /* !RTE_EAL_VFIO */
+
+#error "vfio not supported!!! pl. enable vfio config for your platform"
+#endif /* RTE_EAL_VFIO */
+#endif /* _VIRTIO_VFIO_H_ */
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 08/12] virtio: Add capability to initialize driver for vfio interface
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (6 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 07/12] virtio: vfio: add api support to rd/wr ioport bar Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver Santosh Shukla
` (3 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev; +Cc: Rakesh Krishnamurhty, Rizwan Ansari
- adding pci_dev member in struct virtio_hw{} to track
vfio_dev_fd per virtio interface, required for ioport
rd/wr operation.
- Masked default VIRTIO_READ/WRITE for vfio case.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
Signed-off-by: Rizwan Ansari <ransari@mvista.com>
Signed-off-by: Rakesh Krishnamurhty <rakeshk@mvista.com>
---
drivers/net/virtio/virtio_ethdev.c | 11 ++++++++++-
drivers/net/virtio/virtio_pci.h | 9 +++++++++
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index d928339..9ca99d5 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1287,9 +1287,18 @@ eth_virtio_dev_init(struct rte_eth_dev *eth_dev)
pci_dev = eth_dev->pci_dev;
+#ifdef RTE_EAL_VFIO
+ hw->pci_dev = pci_dev;
+
+ /* For debug use only */
+ const struct rte_intr_handle *intr_handle __rte_unused = &pci_dev->intr_handle;
+ PMD_INIT_LOG(DEBUG, "hw->pci_dev %p intr_handle %p vfio_dev_fd %d\n",
+ hw->pci_dev, intr_handle,
+ intr_handle->vfio_dev_fd);
+#endif
+
if (virtio_resource_init(pci_dev) < 0)
return -1;
-
hw->use_msix = virtio_has_msix(&pci_dev->addr);
hw->io_base = (uint32_t)(uintptr_t)pci_dev->mem_resource[0].addr;
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index 8b5b031..6dabede 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -47,6 +47,10 @@
#include <rte_ethdev.h>
+#ifdef RTE_EAL_VFIO
+#include "virtio_vfio.h"
+#endif
+
struct virtqueue;
/* VirtIO PCI vendor/device ID. */
@@ -176,6 +180,9 @@ struct virtio_hw {
uint8_t use_msix;
uint8_t started;
uint8_t mac_addr[ETHER_ADDR_LEN];
+#ifdef RTE_EAL_VFIO
+ const struct rte_pci_device *pci_dev;
+#endif
};
/*
@@ -228,6 +235,7 @@ outl_p(unsigned int data, unsigned int port)
}
#endif
+#ifndef RTE_EAL_VFIO
#define VIRTIO_PCI_REG_ADDR(hw, reg) \
(unsigned short)((hw)->io_base + (reg))
@@ -245,6 +253,7 @@ outl_p(unsigned int data, unsigned int port)
inl((VIRTIO_PCI_REG_ADDR((hw), (reg))))
#define VIRTIO_WRITE_REG_4(hw, reg, value) \
outl_p((unsigned int)(value), (VIRTIO_PCI_REG_ADDR((hw), (reg))))
+#endif
static inline int
vtpci_with_feature(struct virtio_hw *hw, uint32_t bit)
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (7 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 08/12] virtio: Add capability to initialize driver for vfio interface Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 18:20 ` Stephen Hemminger
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 10/12] config: armv7/v8: Enable RTE_LIBRTE_VIRTIO_PMD Santosh Shukla
` (2 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
Flag required for vfio case, It helps to update vfio_dev_fd for each
virtio net interface.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
drivers/net/virtio/virtio_ethdev.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 9ca99d5..8d55049 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1447,7 +1447,11 @@ static struct eth_driver rte_virtio_pmd = {
.pci_drv = {
.name = "rte_virtio_pmd",
.id_table = pci_id_virtio_map,
+#ifdef RTE_EAL_VFIO
+ .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,
+#else
.drv_flags = RTE_PCI_DRV_DETACHABLE,
+#endif
},
.eth_dev_init = eth_virtio_dev_init,
.eth_dev_uninit = eth_virtio_dev_uninit,
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver Santosh Shukla
@ 2016-01-07 18:20 ` Stephen Hemminger
2016-01-09 12:38 ` Santosh Shukla
0 siblings, 1 reply; 29+ messages in thread
From: Stephen Hemminger @ 2016-01-07 18:20 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
On Thu, 7 Jan 2016 22:03:06 +0530
Santosh Shukla <sshukla@mvista.com> wrote:
> +#ifdef RTE_EAL_VFIO
> + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,
> +#else
> .drv_flags = RTE_PCI_DRV_DETACHABLE,
> +#endif
Since VFIO is determined at runtime not compile time, the flags should
be updated at runtime not compile time.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver
2016-01-07 18:20 ` Stephen Hemminger
@ 2016-01-09 12:38 ` Santosh Shukla
2016-01-12 7:14 ` Yuanhan Liu
0 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-09 12:38 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
On Thu, Jan 7, 2016 at 11:50 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Thu, 7 Jan 2016 22:03:06 +0530
> Santosh Shukla <sshukla@mvista.com> wrote:
>
>> +#ifdef RTE_EAL_VFIO
>> + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,
>> +#else
>> .drv_flags = RTE_PCI_DRV_DETACHABLE,
>> +#endif
>
> Since VFIO is determined at runtime not compile time, the flags should
> be updated at runtime not compile time.
>
>
In general, Yes, Its a wrong approach i..e. Wrapping __need_mapping
flag only for vfio case. I am thinking to add vfio parser routine
something similar to virtio_xxx_xx_uio_xx() / virtio_xx_xx_ioport()
currently exist. This will remove RTE_EAL_VFIO ifdef clutter for this
patch and [08/12] patch and also virtio pmd driver can then initialize
device for vfio mode..
_but_ I still need _MAPPING flag enabled for in virtio driver as
because for vfio case - I want vfio_xx_mmap() routine to create vfio
container/group_id and then create vfio_dev_fd for each virtio-net-pci
interface. Let me know my approach aligned to your suggestion.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver
2016-01-09 12:38 ` Santosh Shukla
@ 2016-01-12 7:14 ` Yuanhan Liu
2016-01-13 12:18 ` Santosh Shukla
0 siblings, 1 reply; 29+ messages in thread
From: Yuanhan Liu @ 2016-01-12 7:14 UTC (permalink / raw)
To: Santosh Shukla; +Cc: dev
On Sat, Jan 09, 2016 at 06:08:46PM +0530, Santosh Shukla wrote:
> On Thu, Jan 7, 2016 at 11:50 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> > On Thu, 7 Jan 2016 22:03:06 +0530
> > Santosh Shukla <sshukla@mvista.com> wrote:
> >
> >> +#ifdef RTE_EAL_VFIO
> >> + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,
> >> +#else
> >> .drv_flags = RTE_PCI_DRV_DETACHABLE,
> >> +#endif
> >
> > Since VFIO is determined at runtime not compile time, the flags should
> > be updated at runtime not compile time.
> >
> >
> In general, Yes, Its a wrong approach i..e. Wrapping __need_mapping
> flag only for vfio case. I am thinking to add vfio parser routine
> something similar to virtio_xxx_xx_uio_xx() / virtio_xx_xx_ioport()
> currently exist. This will remove RTE_EAL_VFIO ifdef clutter for this
> patch and [08/12] patch and also virtio pmd driver can then initialize
> device for vfio mode..
>
> _but_ I still need _MAPPING flag enabled for in virtio driver as
> because for vfio case - I want vfio_xx_mmap() routine to create vfio
> container/group_id and then create vfio_dev_fd for each virtio-net-pci
> interface.
I'm thinking my following patch will help:
http://dpdk.org/dev/patchwork/patch/9814/
--yliu
> Let me know my approach aligned to your suggestion.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver
2016-01-12 7:14 ` Yuanhan Liu
@ 2016-01-13 12:18 ` Santosh Shukla
0 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-13 12:18 UTC (permalink / raw)
To: Yuanhan Liu; +Cc: dev
On Tue, Jan 12, 2016 at 12:44 PM, Yuanhan Liu
<yuanhan.liu@linux.intel.com> wrote:
> On Sat, Jan 09, 2016 at 06:08:46PM +0530, Santosh Shukla wrote:
>> On Thu, Jan 7, 2016 at 11:50 PM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>> > On Thu, 7 Jan 2016 22:03:06 +0530
>> > Santosh Shukla <sshukla@mvista.com> wrote:
>> >
>> >> +#ifdef RTE_EAL_VFIO
>> >> + .drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,
>> >> +#else
>> >> .drv_flags = RTE_PCI_DRV_DETACHABLE,
>> >> +#endif
>> >
>> > Since VFIO is determined at runtime not compile time, the flags should
>> > be updated at runtime not compile time.
>> >
>> >
>> In general, Yes, Its a wrong approach i..e. Wrapping __need_mapping
>> flag only for vfio case. I am thinking to add vfio parser routine
>> something similar to virtio_xxx_xx_uio_xx() / virtio_xx_xx_ioport()
>> currently exist. This will remove RTE_EAL_VFIO ifdef clutter for this
>> patch and [08/12] patch and also virtio pmd driver can then initialize
>> device for vfio mode..
>>
>> _but_ I still need _MAPPING flag enabled for in virtio driver as
>> because for vfio case - I want vfio_xx_mmap() routine to create vfio
>> container/group_id and then create vfio_dev_fd for each virtio-net-pci
>> interface.
>
> I'm thinking my following patch will help:
>
> http://dpdk.org/dev/patchwork/patch/9814/
>
Yes, It works, so wont need NEED_MAPPING flag, Sending v4 patch series
rebased on this patch..
> --yliu
>
>> Let me know my approach aligned to your suggestion.
^ permalink raw reply [flat|nested] 29+ messages in thread
* [dpdk-dev] [PATCH v3 10/12] config: armv7/v8: Enable RTE_LIBRTE_VIRTIO_PMD
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (8 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 09/12] virtio: vfio: Enable RTE_PCI_DRV_NEED_MAPPING flag in driver Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 11/12] vfio: Support for no-IOMMU mode Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 12/12] eal: pci: vfio: fix build error Santosh Shukla
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
Enable RTE_LIBRTE_VIRTIO_PMD for armv7/v8 and setting RTE_VIRTIO_INC_VEC=n.
Builds successfully for armv7/v8.
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
config/defconfig_arm-armv7a-linuxapp-gcc | 6 +++++-
config/defconfig_arm64-armv8a-linuxapp-gcc | 6 +++++-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/config/defconfig_arm-armv7a-linuxapp-gcc b/config/defconfig_arm-armv7a-linuxapp-gcc
index cbebd64..d840dc2 100644
--- a/config/defconfig_arm-armv7a-linuxapp-gcc
+++ b/config/defconfig_arm-armv7a-linuxapp-gcc
@@ -43,6 +43,11 @@ CONFIG_RTE_FORCE_INTRINSICS=y
CONFIG_RTE_TOOLCHAIN="gcc"
CONFIG_RTE_TOOLCHAIN_GCC=y
+# VIRTIO support for ARM
+CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
+# Disable VIRTIO VECTOR support
+CONFIG_RTE_VIRTIO_INC_VECTOR=n
+
# ARM doesn't have support for vmware TSC map
CONFIG_RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT=n
@@ -70,7 +75,6 @@ CONFIG_RTE_LIBRTE_I40E_PMD=n
CONFIG_RTE_LIBRTE_IXGBE_PMD=n
CONFIG_RTE_LIBRTE_MLX4_PMD=n
CONFIG_RTE_LIBRTE_MPIPE_PMD=n
-CONFIG_RTE_LIBRTE_VIRTIO_PMD=n
CONFIG_RTE_LIBRTE_VMXNET3_PMD=n
CONFIG_RTE_LIBRTE_PMD_XENVIRT=n
CONFIG_RTE_LIBRTE_PMD_BNX2X=n
diff --git a/config/defconfig_arm64-armv8a-linuxapp-gcc b/config/defconfig_arm64-armv8a-linuxapp-gcc
index 504f3ed..b3a4b28 100644
--- a/config/defconfig_arm64-armv8a-linuxapp-gcc
+++ b/config/defconfig_arm64-armv8a-linuxapp-gcc
@@ -45,8 +45,12 @@ CONFIG_RTE_TOOLCHAIN_GCC=y
CONFIG_RTE_CACHE_LINE_SIZE=64
+# Enable VIRTIO support for ARM
+CONFIG_RTE_LIBRTE_VIRTIO_PMD=y
+# Disable VIRTIO VECTOR support
+CONFIG_RTE_VIRTIO_INC_VECTOR=n
+
CONFIG_RTE_IXGBE_INC_VECTOR=n
-CONFIG_RTE_LIBRTE_VIRTIO_PMD=n
CONFIG_RTE_LIBRTE_IVSHMEM=n
CONFIG_RTE_LIBRTE_FM10K_PMD=n
CONFIG_RTE_LIBRTE_I40E_PMD=n
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 11/12] vfio: Support for no-IOMMU mode
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (9 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 10/12] config: armv7/v8: Enable RTE_LIBRTE_VIRTIO_PMD Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 12/12] eal: pci: vfio: fix build error Santosh Shukla
11 siblings, 0 replies; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev
From: Anatoly Burakov <anatoly.burakov@intel.com>
This commit is adding a generic mechanism to support multiple IOMMU
types. For now, it's only type 1 (x86 IOMMU) and no-IOMMU (a special
VFIO mode that doesn't use IOMMU at all), but it's easily extended
by adding necessary definitions into eal_pci_init.h and a DMA
mapping function to eal_pci_vfio_dma.c.
Since type 1 IOMMU module is no longer necessary to have VFIO,
we fix the module check to check for vfio-pci instead. It's not
ideal and triggers VFIO checks more often (and thus produces more
error output, which was the reason behind the module check in the
first place), so we compensate for that by providing more verbose
logging, indicating whether VFIO initialization has succeeded or
failed.
Signed-off-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
lib/librte_eal/linuxapp/eal/Makefile | 1 +
lib/librte_eal/linuxapp/eal/eal_pci_init.h | 22 ++++
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 142 +++++++++++++++---------
lib/librte_eal/linuxapp/eal/eal_pci_vfio_dma.c | 84 ++++++++++++++
lib/librte_eal/linuxapp/eal/eal_vfio.h | 5 +
5 files changed, 201 insertions(+), 53 deletions(-)
create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_vfio_dma.c
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 26eced5..5c9e9d9 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_log.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_pci.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_pci_uio.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_pci_vfio.c
+SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_pci_vfio_dma.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_pci_vfio_mp_sync.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_debug.c
SRCS-$(CONFIG_RTE_LIBRTE_EAL_LINUXAPP) += eal_lcore.c
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_init.h b/lib/librte_eal/linuxapp/eal/eal_pci_init.h
index 3bc592b..068800c 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_init.h
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_init.h
@@ -112,6 +112,28 @@ struct vfio_config {
struct vfio_group vfio_groups[VFIO_MAX_GROUPS];
};
+/* function pointer typedef for DMA mapping functions */
+typedef int (*vfio_dma_func_t)(int);
+
+/* Structure to hold supported IOMMU types */
+struct vfio_iommu_type {
+ int type_id;
+ const char *name;
+ vfio_dma_func_t dma_map_func;
+};
+
+/* function prototypes for different IOMMU types */
+int vfio_iommu_type1_dma_map(int container_fd);
+int vfio_iommu_noiommu_dma_map(int container_fd);
+
+/* IOMMU types we support */
+static const struct vfio_iommu_type iommu_types[] = {
+ /* x86 IOMMU, otherwise known as type 1 */
+ { VFIO_TYPE1_IOMMU, "Type 1", &vfio_iommu_type1_dma_map},
+ /* IOMMU-less mode */
+ { VFIO_NOIOMMU_IOMMU, "No-IOMMU", &vfio_iommu_noiommu_dma_map},
+};
+
#endif
#endif /* EAL_PCI_INIT_H_ */
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index 7ec203c..c69050d 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -72,6 +72,7 @@ EAL_REGISTER_TAILQ(rte_vfio_tailq)
#define VFIO_DIR "/dev/vfio"
#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"
#define VFIO_GROUP_FMT "/dev/vfio/%u"
+#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u"
#define VFIO_GET_REGION_ADDR(x) ((uint64_t) x << 40ULL)
/* per-process VFIO config */
@@ -236,42 +237,57 @@ pci_vfio_set_bus_master(int dev_fd)
return 0;
}
-/* set up DMA mappings */
-static int
-pci_vfio_setup_dma_maps(int vfio_container_fd)
-{
- const struct rte_memseg *ms = rte_eal_get_physmem_layout();
- int i, ret;
-
- ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
- VFIO_TYPE1_IOMMU);
- if (ret) {
- RTE_LOG(ERR, EAL, " cannot set IOMMU type, "
- "error %i (%s)\n", errno, strerror(errno));
- return -1;
+/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
+static const struct vfio_iommu_type *
+pci_vfio_set_iommu_type(int vfio_container_fd) {
+ for (unsigned idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_type *t = &iommu_types[idx];
+
+ int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
+ t->type_id);
+ if (!ret) {
+ RTE_LOG(NOTICE, EAL, " using IOMMU type %d (%s)\n",
+ t->type_id, t->name);
+ return t;
+ }
+ /* not an error, there may be more supported IOMMU types */
+ RTE_LOG(DEBUG, EAL, " set IOMMU type %d (%s) failed, "
+ "error %i (%s)\n", t->type_id, t->name, errno,
+ strerror(errno));
}
+ /* if we didn't find a suitable IOMMU type, fail */
+ return NULL;
+}
- /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
- for (i = 0; i < RTE_MAX_MEMSEG; i++) {
- struct vfio_iommu_type1_dma_map dma_map;
-
- if (ms[i].addr == NULL)
- break;
-
- memset(&dma_map, 0, sizeof(dma_map));
- dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
- dma_map.vaddr = ms[i].addr_64;
- dma_map.size = ms[i].len;
- dma_map.iova = ms[i].phys_addr;
- dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
-
- ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+/* check if we have any supported extensions */
+static int
+pci_vfio_has_supported_extensions(int vfio_container_fd) {
+ int ret;
+ unsigned idx, n_extensions = 0;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ const struct vfio_iommu_type *t = &iommu_types[idx];
- if (ret) {
- RTE_LOG(ERR, EAL, " cannot set up DMA remapping, "
- "error %i (%s)\n", errno, strerror(errno));
+ ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION,
+ t->type_id);
+ if (ret < 0) {
+ RTE_LOG(ERR, EAL, " could not get IOMMU type, "
+ "error %i (%s)\n", errno,
+ strerror(errno));
+ close(vfio_container_fd);
return -1;
+ } else if (ret == 1) {
+ /* we found a supported extension */
+ n_extensions++;
}
+ RTE_LOG(DEBUG, EAL, " IOMMU type %d (%s) is %s\n",
+ t->type_id, t->name,
+ ret ? "supported" : "not supported");
+ }
+
+ /* if we didn't find any supported IOMMU types, fail */
+ if (!n_extensions) {
+ close(vfio_container_fd);
+ return -1;
}
return 0;
@@ -400,17 +416,10 @@ pci_vfio_get_container_fd(void)
return -1;
}
- /* check if we support IOMMU type 1 */
- ret = ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU);
- if (ret != 1) {
- if (ret < 0)
- RTE_LOG(ERR, EAL, " could not get IOMMU type, "
- "error %i (%s)\n", errno,
- strerror(errno));
- else
- RTE_LOG(ERR, EAL, " unsupported IOMMU type "
- "detected in VFIO\n");
- close(vfio_container_fd);
+ ret = pci_vfio_has_supported_extensions(vfio_container_fd);
+ if (ret) {
+ RTE_LOG(ERR, EAL, " no supported IOMMU "
+ "extensions found!\n");
return -1;
}
@@ -460,6 +469,7 @@ pci_vfio_get_group_fd(int iommu_group_no)
/* if primary, try to open the group */
if (internal_config.process_type == RTE_PROC_PRIMARY) {
+ /* try regular group format */
snprintf(filename, sizeof(filename),
VFIO_GROUP_FMT, iommu_group_no);
vfio_group_fd = open(filename, O_RDWR);
@@ -470,7 +480,20 @@ pci_vfio_get_group_fd(int iommu_group_no)
strerror(errno));
return -1;
}
- return 0;
+
+ /* special case: try no-IOMMU path as well */
+ snprintf(filename, sizeof(filename),
+ VFIO_NOIOMMU_GROUP_FMT, iommu_group_no);
+ vfio_group_fd = open(filename, O_RDWR);
+ if (vfio_group_fd < 0) {
+ if (errno != ENOENT) {
+ RTE_LOG(ERR, EAL, "Cannot open %s: %s\n", filename,
+ strerror(errno));
+ return -1;
+ }
+ return 0;
+ }
+ /* noiommu group found */
}
/* if the fd is valid, create a new group for it */
@@ -688,14 +711,21 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
}
/*
- * set up DMA mappings for container
+ * pick an IOMMU type and set up DMA mappings for container
*
* needs to be done only once, only when at least one group is assigned to
* a container and only in primary process
*/
if (internal_config.process_type == RTE_PROC_PRIMARY &&
vfio_cfg.vfio_container_has_dma == 0) {
- ret = pci_vfio_setup_dma_maps(vfio_cfg.vfio_container_fd);
+ /* select an IOMMU type which we will be using */
+ const struct vfio_iommu_type *t =
+ pci_vfio_set_iommu_type(vfio_cfg.vfio_container_fd);
+ if (!t) {
+ RTE_LOG(ERR, EAL, " %s failed to select IOMMU type\n", pci_addr);
+ return -1;
+ }
+ ret = t->dma_map_func(vfio_cfg.vfio_container_fd);
if (ret) {
RTE_LOG(ERR, EAL, " %s DMA remapping failed, "
"error %i (%s)\n", pci_addr, errno, strerror(errno));
@@ -935,35 +965,41 @@ pci_vfio_enable(void)
{
/* initialize group list */
int i;
- int module_vfio_type1;
+ int vfio_available;
for (i = 0; i < VFIO_MAX_GROUPS; i++) {
vfio_cfg.vfio_groups[i].fd = -1;
vfio_cfg.vfio_groups[i].group_no = -1;
}
- module_vfio_type1 = rte_eal_check_module("vfio_iommu_type1");
+ /* inform the user that we are probing for VFIO */
+ RTE_LOG(INFO, EAL, "Probing VFIO support...\n");
+
+ /* check if vfio-pci module is loaded */
+ vfio_available = rte_eal_check_module("vfio_pci");
/* return error directly */
- if (module_vfio_type1 == -1) {
+ if (vfio_available == -1) {
RTE_LOG(INFO, EAL, "Could not get loaded module details!\n");
return -1;
}
/* return 0 if VFIO modules not loaded */
- if (module_vfio_type1 == 0) {
- RTE_LOG(INFO, EAL, "VFIO modules not all loaded, "
- "skip VFIO support...\n");
+ if (vfio_available == 0) {
+ RTE_LOG(INFO, EAL, "VFIO modules not loaded, "
+ "skipping VFIO support...\n");
return 0;
}
vfio_cfg.vfio_container_fd = pci_vfio_get_container_fd();
/* check if we have VFIO driver enabled */
- if (vfio_cfg.vfio_container_fd != -1)
+ if (vfio_cfg.vfio_container_fd != -1) {
+ RTE_LOG(NOTICE, EAL, "VFIO support initialized\n");
vfio_cfg.vfio_enabled = 1;
- else
+ } else {
RTE_LOG(NOTICE, EAL, "VFIO support could not be initialized\n");
+ }
return 0;
}
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio_dma.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio_dma.c
new file mode 100644
index 0000000..50d3563
--- /dev/null
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio_dma.c
@@ -0,0 +1,84 @@
+/*-
+ * BSD LICENSE
+ *
+ * Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in
+ * the documentation and/or other materials provided with the
+ * distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ * contributors may be used to endorse or promote products derived
+ * from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#include <string.h>
+#include <sys/ioctl.h>
+
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_eal_memconfig.h>
+
+#include "eal_pci_init.h"
+
+#ifdef VFIO_PRESENT
+
+int
+vfio_iommu_type1_dma_map(int vfio_container_fd)
+{
+ const struct rte_memseg *ms = rte_eal_get_physmem_layout();
+ int i, ret;
+
+ /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */
+ for (i = 0; i < RTE_MAX_MEMSEG; i++) {
+ struct vfio_iommu_type1_dma_map dma_map;
+
+ if (ms[i].addr == NULL)
+ break;
+
+ memset(&dma_map, 0, sizeof(dma_map));
+ dma_map.argsz = sizeof(struct vfio_iommu_type1_dma_map);
+ dma_map.vaddr = ms[i].addr_64;
+ dma_map.size = ms[i].len;
+ dma_map.iova = ms[i].phys_addr;
+ dma_map.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE;
+
+ ret = ioctl(vfio_container_fd, VFIO_IOMMU_MAP_DMA, &dma_map);
+
+ if (ret) {
+ RTE_LOG(ERR, EAL, " cannot set up DMA remapping, "
+ "error %i (%s)\n", errno, strerror(errno));
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+int
+vfio_iommu_noiommu_dma_map(int __rte_unused vfio_container_fd)
+{
+ /* No-IOMMU mode does not need DMA mapping */
+ return 0;
+}
+
+#endif
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index 72ec3f6..638ee31 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -52,6 +52,11 @@
#define RTE_PCI_MSIX_FLAGS_QSIZE PCI_MSIX_FLAGS_QSIZE
#endif
+/* older kernels may not have no-IOMMU mode */
+#ifndef VFIO_NOIOMMU_IOMMU
+#define VFIO_NOIOMMU_IOMMU 8
+#endif
+
#define VFIO_PRESENT
#endif /* kernel version */
#endif /* RTE_EAL_VFIO */
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* [dpdk-dev] [PATCH v3 12/12] eal: pci: vfio: fix build error
2016-01-07 16:32 [dpdk-dev] [PATCH v3 00/12] Add virtio support for arm/arm64 Santosh Shukla
` (10 preceding siblings ...)
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 11/12] vfio: Support for no-IOMMU mode Santosh Shukla
@ 2016-01-07 16:33 ` Santosh Shukla
2016-01-07 17:02 ` Burakov, Anatoly
11 siblings, 1 reply; 29+ messages in thread
From: Santosh Shukla @ 2016-01-07 16:33 UTC (permalink / raw)
To: dev; +Cc: Rizwan Ansari
From: Rizwan Ansari <ransari@mvista.com>
Patch fixes below build error:
/home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c:
In function ‘pci_vfio_set_iommu_type’:
/home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c:243:2:
error: ‘for’ loop initial declarations are only allowed in C99 mode
for (unsigned idx = 0; idx < RTE_DIM(iommu_types); idx++) {
^
/home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c:243:2:
note: use option -std=c99 or -std=gnu99 to compile your code
Signed-off-by: Rizwan Ansari <ransari@mvista.com>
Signed-off-by: Santosh Shukla <sshukla@mvista.com>
---
lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index c69050d..844ef80 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -240,7 +240,8 @@ pci_vfio_set_bus_master(int dev_fd)
/* pick IOMMU type. returns a pointer to vfio_iommu_type or NULL for error */
static const struct vfio_iommu_type *
pci_vfio_set_iommu_type(int vfio_container_fd) {
- for (unsigned idx = 0; idx < RTE_DIM(iommu_types); idx++) {
+ unsigned idx;
+ for (idx = 0; idx < RTE_DIM(iommu_types); idx++) {
const struct vfio_iommu_type *t = &iommu_types[idx];
int ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU,
--
1.7.9.5
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [dpdk-dev] [PATCH v3 12/12] eal: pci: vfio: fix build error
2016-01-07 16:33 ` [dpdk-dev] [PATCH v3 12/12] eal: pci: vfio: fix build error Santosh Shukla
@ 2016-01-07 17:02 ` Burakov, Anatoly
0 siblings, 0 replies; 29+ messages in thread
From: Burakov, Anatoly @ 2016-01-07 17:02 UTC (permalink / raw)
To: Santosh Shukla, dev; +Cc: Rizwan Ansari
Hi Santosh,
> Patch fixes below build error:
> /home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxa
> pp/eal/eal_pci_vfio.c:
> In function ‘pci_vfio_set_iommu_type’:
> /home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxa
> pp/eal/eal_pci_vfio.c:243:2:
> error: ‘for’ loop initial declarations are only allowed in C99 mode
> for (unsigned idx = 0; idx < RTE_DIM(iommu_types); idx++) {
> ^
> /home/mv/work/thunder/santosh/dpdk/santosh/dpdk/lib/librte_eal/linuxa
> pp/eal/eal_pci_vfio.c:243:2:
> note: use option -std=c99 or -std=gnu99 to compile your code
>
> Signed-off-by: Rizwan Ansari <ransari@mvista.com>
> Signed-off-by: Santosh Shukla <sshukla@mvista.com>
Oops, thanks, I'll submit a v2.
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 29+ messages in thread