[dpdk-dev] DPDK on IBM Power9

DPDK patches and discussions
 help / color / mirror / Atom feed

* [dpdk-dev] DPDK on IBM Power9
@ 2019-01-09 22:25 Daniel Waddington
  2019-01-09 22:52 ` Wiles, Keith
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Daniel Waddington @ 2019-01-09 22:25 UTC (permalink / raw)
  To: dev

Hi,
I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power). 
Thx,
Daniel
Below is what I get:

$ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
EAL: Detected 128 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1583 net_i40e
EAL:   using IOMMU type 7 (sPAPR)
EAL:   cannot create new DMA window, error 22 (Invalid argument)
EAL: Could not create new DMA window
EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
EAL: Requested device 0000:01:00.0 cannot be used
Number of NICs: 0
EAL: Error - exiting with code: 1
  Cause: No available NIC ports!

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 22:25 [dpdk-dev] DPDK on IBM Power9 Daniel Waddington
@ 2019-01-09 22:52 ` Wiles, Keith
  2019-01-09 23:22 ` Daniel Waddington
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 9+ messages in thread
From: Wiles, Keith @ 2019-01-09 22:52 UTC (permalink / raw)
  To: Daniel Waddington; +Cc: dev



> On Jan 9, 2019, at 4:25 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
> 
> Hi,
> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power). 
> Thx,
> Daniel
> Below is what I get:
> 
> $ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
> EAL: Detected 128 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1583 net_i40e
> EAL:   using IOMMU type 7 (sPAPR)
> EAL:   cannot create new DMA window, error 22 (Invalid argument)
> EAL: Could not create new DMA window
> EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
> EAL: Requested device 0000:01:00.0 cannot be used
> Number of NICs: 0
> EAL: Error - exiting with code: 1
>  Cause: No available NIC ports!
> 

>From the message above you need support NIC for DPDK is all or you a NIC needs to be bound to something like igb_uio for linux. I have never run DPDK on power before and not sure what is next other then using a supported NIC.

> 
> 

Regards,
Keith

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 22:25 [dpdk-dev] DPDK on IBM Power9 Daniel Waddington
  2019-01-09 22:52 ` Wiles, Keith
@ 2019-01-09 23:22 ` Daniel Waddington
  2019-01-09 23:35   ` Wiles, Keith
  2019-01-09 23:58   ` Daniel Waddington
  2019-01-10  0:07 ` Thomas Monjalon
  2019-01-10  8:51 ` Burakov, Anatoly
  3 siblings, 2 replies; 9+ messages in thread
From: Daniel Waddington @ 2019-01-09 23:22 UTC (permalink / raw)
  To: Wiles, Keith; +Cc: dev

This NIC (000:01:00.0) is supported (Intel XL710). I assume that I can use VFIO instead of UIO. UIO is really bad from a security perspective.

I think its the "cannot create new DMA window, error 22" which is the first indicator of the problem.

Daniel

-----"Wiles, Keith" <keith.wiles@intel.com> wrote: -----
To: Daniel Waddington <Daniel.Waddington@ibm.com>
From: "Wiles, Keith" <keith.wiles@intel.com>
Date: 01/09/2019 02:52PM
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] DPDK on IBM Power9


> On Jan 9, 2019, at 4:25 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
> 
> Hi,
> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power). 
> Thx,
> Daniel
> Below is what I get:
> 
> $ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
> EAL: Detected 128 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1583 net_i40e
> EAL:   using IOMMU type 7 (sPAPR)
> EAL:   cannot create new DMA window, error 22 (Invalid argument)
> EAL: Could not create new DMA window
> EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
> EAL: Requested device 0000:01:00.0 cannot be used
> Number of NICs: 0
> EAL: Error - exiting with code: 1
>  Cause: No available NIC ports!
> 

>From the message above you need support NIC for DPDK is all or you a NIC needs to be bound to something like igb_uio for linux. I have never run DPDK on power before and not sure what is next other then using a supported NIC.

> 
> 

Regards,
Keith

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 23:22 ` Daniel Waddington
@ 2019-01-09 23:35   ` Wiles, Keith
  2019-01-09 23:58   ` Daniel Waddington
  1 sibling, 0 replies; 9+ messages in thread
From: Wiles, Keith @ 2019-01-09 23:35 UTC (permalink / raw)
  To: Daniel Waddington; +Cc: dev



> On Jan 9, 2019, at 5:22 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
> 
> This NIC (000:01:00.0) is supported (Intel XL710). I assume that I can use VFIO instead of UIO. UIO is really bad from a security perspective.
> 
> I think its the "cannot create new DMA window, error 22" which is the first indicator of the problem.
> 
> Daniel
> 
> -----"Wiles, Keith" <keith.wiles@intel.com> wrote: -----
> To: Daniel Waddington <Daniel.Waddington@ibm.com>
> From: "Wiles, Keith" <keith.wiles@intel.com>
> Date: 01/09/2019 02:52PM
> Cc: "dev@dpdk.org" <dev@dpdk.org>
> Subject: Re: [dpdk-dev] DPDK on IBM Power9
> 
> 
>> On Jan 9, 2019, at 4:25 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
>> 
>> Hi,
>> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power). 
>> Thx,
>> Daniel
>> Below is what I get:
>> 
>> $ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
>> EAL: Detected 128 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:1583 net_i40e
>> EAL:   using IOMMU type 7 (sPAPR)
>> EAL:   cannot create new DMA window, error 22 (Invalid argument)
>> EAL: Could not create new DMA window
>> EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
>> EAL: Requested device 0000:01:00.0 cannot be used
>> Number of NICs: 0
>> EAL: Error - exiting with code: 1
>> Cause: No available NIC ports!
>> 
> 
> From the message above you need support NIC for DPDK is all or you a NIC needs to be bound to something like igb_uio for linux. I have never run DPDK on power before and not sure what is next other then using a supported NIC.
> 

Yes you should be able to use VFIO. Did you bind the NIC to VFIO ?

There should be some docs about how that is done.

Beyond this I have not idea?
>> 
>> 
> 
> Regards,
> Keith
> 
> 

Regards,
Keith

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 23:22 ` Daniel Waddington
  2019-01-09 23:35   ` Wiles, Keith
@ 2019-01-09 23:58   ` Daniel Waddington
  1 sibling, 0 replies; 9+ messages in thread
From: Daniel Waddington @ 2019-01-09 23:58 UTC (permalink / raw)
  To: Wiles, Keith; +Cc: dev

Also, this is what the kernel shows.

....
[ 2812.456773] pci 0000:01     : [PE# 00] Enabling 64-bit DMA bypass
[ 2876.395586] pci 0000:01     : [PE# 00] Disabling 64-bit DMA bypass
[ 2876.395593] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 2876.395705] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 2876.395711] pci 0000:01     : [PE# 00] Removing DMA window #1
[ 2876.399657] pci 0000:01     : [PE# 00] Setting up window#0 0..7fffffff pg=1000
[ 2876.399663] pci 0000:01     : [PE# 00] Enabling 64-bit DMA bypass
[ 6519.671051] pci 0000:01     : [PE# 00] Disabling 64-bit DMA bypass
[ 6519.671059] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 6519.671174] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 6519.671180] pci 0000:01     : [PE# 00] Removing DMA window #1
[ 6519.675520] pci 0000:01     : [PE# 00] Setting up window#0 0..7fffffff pg=1000
[ 6519.675526] pci 0000:01     : [PE# 00] Enabling 64-bit DMA bypass
[ 9015.064426] pci 0000:01     : [PE# 00] Disabling 64-bit DMA bypass
[ 9015.064432] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 9015.064527] pci 0000:01     : [PE# 00] Removing DMA window #0
[ 9015.064531] pci 0000:01     : [PE# 00] Removing DMA window #1
[ 9015.067168] pci 0000:01     : [PE# 00] Setting up window#0 0..7fffffff pg=1000
[ 9015.067173] pci 0000:01     : [PE# 00] Enabling 64-bit DMA bypass

Daniel

-----"Wiles, Keith" <keith.wiles@intel.com> wrote: -----
To: Daniel Waddington <Daniel.Waddington@ibm.com>
From: "Wiles, Keith" <keith.wiles@intel.com>
Date: 01/09/2019 03:35PM
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] DPDK on IBM Power9


> On Jan 9, 2019, at 5:22 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
> 
> This NIC (000:01:00.0) is supported (Intel XL710). I assume that I can use VFIO instead of UIO. UIO is really bad from a security perspective.
> 
> I think its the "cannot create new DMA window, error 22" which is the first indicator of the problem.
> 
> Daniel
> 
> -----"Wiles, Keith" <keith.wiles@intel.com> wrote: -----
> To: Daniel Waddington <Daniel.Waddington@ibm.com>
> From: "Wiles, Keith" <keith.wiles@intel.com>
> Date: 01/09/2019 02:52PM
> Cc: "dev@dpdk.org" <dev@dpdk.org>
> Subject: Re: [dpdk-dev] DPDK on IBM Power9
> 
> 
>> On Jan 9, 2019, at 4:25 PM, Daniel Waddington <Daniel.Waddington@ibm.com> wrote:
>> 
>> Hi,
>> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power). 
>> Thx,
>> Daniel
>> Below is what I get:
>> 
>> $ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
>> EAL: Detected 128 lcore(s)
>> EAL: Detected 2 NUMA nodes
>> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
>> EAL: Probing VFIO support...
>> EAL: VFIO support initialized
>> EAL: PCI device 0000:01:00.0 on NUMA socket 0
>> EAL:   probe driver: 8086:1583 net_i40e
>> EAL:   using IOMMU type 7 (sPAPR)
>> EAL:   cannot create new DMA window, error 22 (Invalid argument)
>> EAL: Could not create new DMA window
>> EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
>> EAL: Requested device 0000:01:00.0 cannot be used
>> Number of NICs: 0
>> EAL: Error - exiting with code: 1
>> Cause: No available NIC ports!
>> 
> 
> From the message above you need support NIC for DPDK is all or you a NIC needs to be bound to something like igb_uio for linux. I have never run DPDK on power before and not sure what is next other then using a supported NIC.
> 

Yes you should be able to use VFIO. Did you bind the NIC to VFIO ?

There should be some docs about how that is done.

Beyond this I have not idea?
>> 
>> 
> 
> Regards,
> Keith
> 
> 

Regards,
Keith

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 22:25 [dpdk-dev] DPDK on IBM Power9 Daniel Waddington
  2019-01-09 22:52 ` Wiles, Keith
  2019-01-09 23:22 ` Daniel Waddington
@ 2019-01-10  0:07 ` Thomas Monjalon
  2019-01-10  0:23   ` Wiles, Keith
  2019-01-10  7:40   ` Olga Shern
  2019-01-10  8:51 ` Burakov, Anatoly
  3 siblings, 2 replies; 9+ messages in thread
From: Thomas Monjalon @ 2019-01-10  0:07 UTC (permalink / raw)
  To: Daniel Waddington; +Cc: dev

Hi,

09/01/2019 23:25, Daniel Waddington:
> Hi,
> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power).

As mentioned in the release notes, Power9 is supported:
	http://doc.dpdk.org/guides/rel_notes/release_18_11.html#known-issues

We are missing a strong support from IBM.
Several requests were already sent.
Given your email address, you should be able to forward the need ;)

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-10  0:07 ` Thomas Monjalon
@ 2019-01-10  0:23   ` Wiles, Keith
  2019-01-10  7:40   ` Olga Shern
  1 sibling, 0 replies; 9+ messages in thread
From: Wiles, Keith @ 2019-01-10  0:23 UTC (permalink / raw)
  To: Thomas Monjalon; +Cc: Daniel Waddington, dev



> On Jan 9, 2019, at 6:07 PM, Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> Hi,
> 
> 09/01/2019 23:25, Daniel Waddington:
>> Hi,
>> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power).
> 
> As mentioned in the release notes, Power9 is supported:
> 	http://doc.dpdk.org/guides/rel_notes/release_18_11.html#known-issues
> 
> We are missing a strong support from IBM.
> Several requests were already sent.
> Given your email address, you should be able to forward the need ;)
> 

>From the MAINTAINERS file:

IBM POWER (alpha)
M: Chao Zhu <chaozhu@linux.vnet.ibm.com>
F: lib/librte_eal/common/arch/ppc_64/
F: lib/librte_eal/common/include/arch/ppc_64/
F: drivers/net/i40e/i40e_rxtx_vec_altivec.c
F: examples/l3fwd/*altivec.h

So Chao Zhu IBM is the maintainer of the ppc code. That was the only reference I could find.


Regards,
Keith

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-10  0:07 ` Thomas Monjalon
  2019-01-10  0:23   ` Wiles, Keith
@ 2019-01-10  7:40   ` Olga Shern
  1 sibling, 0 replies; 9+ messages in thread
From: Olga Shern @ 2019-01-10  7:40 UTC (permalink / raw)
  To: Thomas Monjalon, Daniel Waddington; +Cc: dev

Daniel, 

What exactly is not working for you?
Mellanox is testing Power9  for 19.02 release.

Thanks,
Olga


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
Sent: Thursday, January 10, 2019 2:08 AM
To: Daniel Waddington <Daniel.Waddington@ibm.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] DPDK on IBM Power9

Hi,

09/01/2019 23:25, Daniel Waddington:
> Hi,
> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power).

As mentioned in the release notes, Power9 is supported:
	https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc.dpdk.org%2Fguides%2Frel_notes%2Frelease_18_11.html%23known-issues&amp;data=02%7C01%7Colgas%40mellanox.com%7C6c8d43623a0348adae1408d6768fb1b1%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636826756880461285&amp;sdata=WzB%2Fz2zsUUZUawHT1W42BPI2M%2FfDu943xNgDI4w90lU%3D&amp;reserved=0

We are missing a strong support from IBM.
Several requests were already sent.
Given your email address, you should be able to forward the need ;)

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [dpdk-dev] DPDK on IBM Power9
  2019-01-09 22:25 [dpdk-dev] DPDK on IBM Power9 Daniel Waddington
                   ` (2 preceding siblings ...)
  2019-01-10  0:07 ` Thomas Monjalon
@ 2019-01-10  8:51 ` Burakov, Anatoly
  3 siblings, 0 replies; 9+ messages in thread
From: Burakov, Anatoly @ 2019-01-10  8:51 UTC (permalink / raw)
  To: Daniel Waddington, dev

On 09-Jan-19 10:25 PM, Daniel Waddington wrote:
> Hi,
> I'm trying to run DPDK 18.11 on IBM Power9.  Should it run?  (I'n new to DPDK on Power).
> Thx,
> Daniel
> Below is what I get:
>   
> $ sudo ./examples/ethtool/ethtool-app/ppc_64-power8-linuxapp-gcc/app/ethtool --iova-mode=pa -c7 -w 0000:01:00.0
> EAL: Detected 128 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:01:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1583 net_i40e
> EAL:   using IOMMU type 7 (sPAPR)
> EAL:   cannot create new DMA window, error 22 (Invalid argument)
> EAL: Could not create new DMA window
> EAL:   0000:01:00.0 DMA remapping failed, error 22 (Invalid argument)
> EAL: Requested device 0000:01:00.0 cannot be used
> Number of NICs: 0
> EAL: Error - exiting with code: 1
>    Cause: No available NIC ports!
> 

I know _where_ this happens, but as to _why_... You're IBM, you tell me! 
There have been numerous issues with VFIO/memory subsystem being pretty 
much unmaintained as far as IBM platforms support is concerned, but 
there's little i, as a maintainer, can do about it. Thomas is right, one 
more reason to ping the relevant people within IBM :)

-- 
Thanks,
Anatoly

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2019-01-10  8:51 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-09 22:25 [dpdk-dev] DPDK on IBM Power9 Daniel Waddington
2019-01-09 22:52 ` Wiles, Keith
2019-01-09 23:22 ` Daniel Waddington
2019-01-09 23:35   ` Wiles, Keith
2019-01-09 23:58   ` Daniel Waddington
2019-01-10  0:07 ` Thomas Monjalon
2019-01-10  0:23   ` Wiles, Keith
2019-01-10  7:40   ` Olga Shern
2019-01-10  8:51 ` Burakov, Anatoly

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).