OpenWRT Related Question

OpenWRT Related Question

[R T]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]

Hello Folks,I've been writing a small application on Ubuntu 20.04 that uses DPDK.  I started with some of the simpler examples such as l2fwd and packet_ordering and added some other function I wanted to experiment with.  I'm getting good packet throughput and the library is working well.  Overall, DPDK is feature rich, easy to compile, and working as advertised.
As next steps, I would like to move my application to run on top of a very small Linux image that can be loaded off of a very small flash or even over the network.OpenWRT checks all the boxes for being small, having a lot of network utilities, and supporting DPDK.
Looking at the documents, the OpenWRT docs is pretty small and I don't see mmay uch discussions about it on the archives of this mailing list.
I'm asking if people could provide some feedback on writing a DPDK app for use on OpenWRT.
Is the advertised integration working well?How easy is it to cross compile an app to run on OpenWRT (x86 in m y case)?
I would welcome any comment on this topic

[-- Attachment #2: Type: text/html, Size: 1784 bytes --]

Re: OpenWRT Related Question

[Thomas Monjalon]

Hello,

20/07/2022 06:15, R T:
> Hello Folks,I've been writing a small application on Ubuntu 20.04 that uses DPDK.  I started with some of the simpler examples such as l2fwd and packet_ordering and added some other function I wanted to experiment with.  I'm getting good packet throughput and the library is working well.  Overall, DPDK is feature rich, easy to compile, and working as advertised.

Great to read, thanks.

> As next steps, I would like to move my application to run on top of a very small Linux image that can be loaded off of a very small flash or even over the network.OpenWRT checks all the boxes for being small, having a lot of network utilities, and supporting DPDK.
> Looking at the documents, the OpenWRT docs is pretty small and I don't see mmay uch discussions about it on the archives of this mailing list.
> I'm asking if people could provide some feedback on writing a DPDK app for use on OpenWRT.
> Is the advertised integration working well?How easy is it to cross compile an app to run on OpenWRT (x86 in m y case)?
> I would welcome any comment on this topic

We have this howto page about OpenWRT:
https://doc.dpdk.org/guides/howto/openwrt.html
I am not sure how much it is up to date,
please do not hesitate to give feedback if anything can be improved.

Re: OpenWRT Related Question

[R T]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

 

    On Wednesday, July 20, 2022 at 06:03:32 AM EDT, Thomas Monjalon <thomas@monjalon.net> wrote:  

We have this howto page about OpenWRT:
https://doc.dpdk.org/guides/howto/openwrt.html
I am not sure how much it is up to date,
please do not hesitate to give feedback if anything can be improved.


One of the items that I'm not seeing, where the regular mason build env on Ubuntu was pretty clear on, is building any of the example for testing on OpenWRT.
How would I go about building and installing one or more of the DPDK examples on the OpenWRT Image?

  

[-- Attachment #2: Type: text/html, Size: 1622 bytes --]

Re: OpenWRT Related Question

[Thomas Monjalon]

21/07/2022 03:08, R T:
>     On Wednesday, July 20, 2022 at 06:03:32 AM EDT, Thomas Monjalon <thomas@monjalon.net> wrote:  
> 
> > We have this howto page about OpenWRT:
> > https://doc.dpdk.org/guides/howto/openwrt.html
> > I am not sure how much it is up to date,
> > please do not hesitate to give feedback if anything can be improved.
> 
> One of the items that I'm not seeing, where the regular mason build env on Ubuntu was pretty clear on, is building any of the example for testing on OpenWRT.
> How would I go about building and installing one or more of the DPDK examples on the OpenWRT Image?

It should not be different of any other OS.
You can compile examples either by adding -Dexamples=all in meson command,
or by calling make in the example directory.

Re: OpenWRT Related Question

[IraM]

Hi all


I haven’t been successful in getting a DPDK app running on OpenWRT.

running into some issues.  Hope to get some help here.

I am starting with the basics.   
-    Followed https://doc.dpdk.org/guides/howto/openwrt.html and https://github.com/k13132/openwrt-dpdk
-    Ubuntu 20.04 as build environment.  OpenWRT (target is x86_64) is getting cross compiled on Ubuntu.
-    OpenWRT is at version 22.03, DPDK is 20.11

OpenWRT image that is build
-    gets populated with DPDK libraries and tools
-    Boots up on a VM just fine

I couldn't see a way to use mason for building the samples during the build so I created an OpenWRT manifest for building it and including it in the image.
My manifest makes l2fwd show up as a package during "make menuconfig" and the code gets compiled and copied to the image.

Having said this, the main issue is that l2fwd starts up but EAL complains that no ports being found. 

Here are some info from the OpenWRT VM:

Dpdk-devbind.py -s shows:
Network devices using DPDK-compatible driver ======================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv= uio_pci_generic unused=vfio-pci
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv= uio_pci_generic unused=vfio-pci

Network devices using kernel driver
=============================
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=vfio-pci,uio_pci_generic


L2fwd gives the following output:

root@OpenWrt:/# l2fwd -- -p 3
EAL: Detected CPU lcores: 6
EAL Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
MAC updating enabled
EAL: Error - exiting with code 1
   Cause: No Ethernet ports - bye

Best I can tell is l2fwd is looking for vfio drivers, where only uio drivers are loaded.

I tried binding VFIO drivers using
dpdk-devbind.py --bind=vfio-pci 0000:03:00.0 

But it fails as follows:
Vfio-pci: probe of 0000:0b:00.0 failed with error -22
Vfio-pci: probe of 0000:0b:00.0 failed with error -22
Lspci: unable to load libkmod resources: error -12
Error: bind failed for 0000:0b:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device
Error: unbind failed for 0000:0b:00.0 - Cannot open /sysy/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'

So.. although the following kernel options are enabled, per https://doc.dpdk.org/guides/howto/openwrt.html
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
CONFIG_VFIO_NOIOMMU=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_PROC_PAGE_MONITOR=y

I can't load a VFIO driver on OpenWRT.

UIO driver loads fine, but l2fwd isn't finding the UIO ports.

As a side experiment, I built l2fwd on my Ubuntu build machine without cross compiling for OpenWRT, and copied it over to my OpenWRT VM.  Interestingly, that exe runs find and finds the UIO drivers.



Sent from my iPhone

> On Jul 21, 2022, at 03:26, Thomas Monjalon <thomas@monjalon.net> wrote:
> 
> 21/07/2022 03:08, R T:
>>>    On Wednesday, July 20, 2022 at 06:03:32 AM EDT, Thomas Monjalon <thomas@monjalon.net> wrote:  
>>> 
>>> We have this howto page about OpenWRT:
>>> https://doc.dpdk.org/guides/howto/openwrt.html
>>> I am not sure how much it is up to date,
>>> please do not hesitate to give feedback if anything can be improved.
>> 
>> One of the items that I'm not seeing, where the regular mason build env on Ubuntu was pretty clear on, is building any of the example for testing on OpenWRT.
>> How would I go about building and installing one or more of the DPDK examples on the OpenWRT Image?
> 
> It should not be different of any other OS.
> You can compile examples either by adding -Dexamples=all in meson command,
> or by calling make in the example directory.
> 
> 

Re: OpenWRT Related Question

[Stephen Hemminger]

On Thu, 28 Jul 2022 17:06:18 -0400
IraM <iramica@yahoo.com> wrote:

> Best I can tell is l2fwd is looking for vfio drivers, where only uio drivers are loaded.
> 
> I tried binding VFIO drivers using
> dpdk-devbind.py --bind=vfio-pci 0000:03:00.0 
> 
> But it fails as follows:
> Vfio-pci: probe of 0000:0b:00.0 failed with error -22
> Vfio-pci: probe of 0000:0b:00.0 failed with error -22
> Lspci: unable to load libkmod resources: error -12
> Error: bind failed for 0000:0b:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device
> Error: unbind failed for 0000:0b:00.0 - Cannot open /sysy/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'

You need to run as root.
Does VMWare have virtual IOMMU?