Re: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# Getting memory backing issues with qemu parameter passing

[Abhijeet Karve <abhijeet.karve@tcs.com>]

Hi Przemek,

Thank you so much for your quick response. 

The guide(
https://github.com/openstack/networking-ovs-dpdk/blob/stable/kilo/doc/source/getstarted/ubuntu.rst
) which you have suggested that is for openstack vhost user installations 
with devstack. 
Can't we have any reference for including ovs-dpdk mechanisam driver for 
openstack Ubuntu distribution which we are following for 
compute+controller node setup?" 

We are facing below listed issues With the current approach of setting up 
openstack kilo interactively + replacing ovs with ovs-dpdk enabled and 
Instance creation in openstack with
passing that instance id to QEMU command line which further passes the 
vhost-user sockets to instances for enabling the DPDK libraries in it.


1. Created a flavor m1.hugepages which is backed by hugepage memory, 
unable to spawn instance with this flavor – Getting a issue like: No 
matching hugetlbfs for the number of hugepages assigned to the flavor.
2. Passing socket info to instances via qemu manually and instnaces 
created are not persistent.

Now as you suggested, we are looking in enabling ovsdpdk ml2 mechanism 
driver and agent all of that in our openstack ubuntu distribution.

Would be really appriciate if get any help or ref with explanation.

We are using compute + controller node setup and we are using following 
software platform on compute node: 
_____________ 
Openstack: Kilo 
Distribution: Ubuntu 14.04 
OVS Version: 2.4.0 
DPDK 2.0.0 
_____________ 

Thanks,
Abhijeet Karve





From:   "Czesnowicz, Przemyslaw" <przemyslaw.czesnowicz@intel.com>
To:     Abhijeet Karve <abhijeet.karve@tcs.com>
Cc:     "dev@dpdk.org" <dev@dpdk.org>, "discuss@openvswitch.org" 
<discuss@openvswitch.org>, "Gray, Mark D" <mark.d.gray@intel.com>
Date:   12/17/2015 06:32 PM
Subject:        RE: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# 
Successfully setup DPDK OVS with vhostuser



I haven’t tried that approach not sure if that would work, it seems 
clunky.
 
If you enable ovsdpdk ml2 mechanism driver and agent all of that (add 
ports to ovs with the right type, pass the sockets to qemu) would be done 
by OpenStack.
 
Przemek
 
From: Abhijeet Karve [mailto:abhijeet.karve@tcs.com] 
Sent: Thursday, December 17, 2015 12:41 PM
To: Czesnowicz, Przemyslaw
Cc: dev@dpdk.org; discuss@openvswitch.org; Gray, Mark D
Subject: RE: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# 
Successfully setup DPDK OVS with vhostuser
 
Hi Przemek, 

Thank you so much for sharing the ref guide. 

Would be appreciate if clear one doubt. 

At present we are setting up openstack kilo interactively and further 
replacing ovs with ovs-dpdk enabled. 
Once the above setup done, We are creating instance in openstack and 
passing that instance id to QEMU command line which further passes the 
vhost-user sockets to instances, enabling the DPDK libraries in it. 

Isn't this the correct way of integrating ovs-dpdk with openstack? 


Thanks & Regards
Abhijeet Karve




From:        "Czesnowicz, Przemyslaw" <przemyslaw.czesnowicz@intel.com> 
To:        Abhijeet Karve <abhijeet.karve@tcs.com> 
Cc:        "dev@dpdk.org" <dev@dpdk.org>, "discuss@openvswitch.org" <
discuss@openvswitch.org>, "Gray, Mark D" <mark.d.gray@intel.com> 
Date:        12/17/2015 05:27 PM 
Subject:        RE: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# 
Successfully setup DPDK OVS with vhostuser 




HI Abhijeet, 
  
For Kilo you need to use ovsdpdk mechanism driver and a matching agent to 
integrate ovs-dpdk with OpenStack. 
  
The guide you are following only talks about running ovs-dpdk not how it 
should be integrated with OpenStack. 
  
Please follow this guide: 
https://github.com/openstack/networking-ovs-dpdk/blob/stable/kilo/doc/source/getstarted/ubuntu.rst 

  
Best regards 
Przemek 
  
  
From: Abhijeet Karve [mailto:abhijeet.karve@tcs.com] 
Sent: Wednesday, December 16, 2015 9:37 AM
To: Czesnowicz, Przemyslaw
Cc: dev@dpdk.org; discuss@openvswitch.org; Gray, Mark D
Subject: RE: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# 
Successfully setup DPDK OVS with vhostuser 
  
Hi Przemek, 


We have configured the accelerated data path between a physical interface 
to the VM using openvswitch netdev-dpdk with vhost-user support. The VM 
created with this special data path and vhost library, I am calling as 
DPDK instance. 

If assigning ip manually to the newly created Cirros VM instance, We are 
able to make 2 VM's to communicate on the same compute node. Else it's not 
associating any ip through DHCP though DHCP is in compute node only. 

Yes it's a compute + controller node setup and we are using following 
software platform on compute node: 
_____________ 
Openstack: Kilo 
Distribution: Ubuntu 14.04 
OVS Version: 2.4.0 
DPDK 2.0.0 
_____________ 

We are following the intel guide 
https://software.intel.com/en-us/blogs/2015/06/09/building-vhost-user-for-ovs-today-using-dpdk-200 


When doing "ovs-vsctl show" in compute node, it shows below output: 
_____________________________________________ 
ovs-vsctl show 
c2ec29a5-992d-4875-8adc-1265c23e0304 
   Bridge br-ex 
       Port phy-br-ex 
           Interface phy-br-ex 
               type: patch 
               options: {peer=int-br-ex} 
       Port br-ex 
           Interface br-ex 
               type: internal 
   Bridge br-tun 
       fail_mode: secure 
       Port br-tun 
           Interface br-tun 
               type: internal 
       Port patch-int 
           Interface patch-int 
               type: patch 
               options: {peer=patch-tun} 
   Bridge br-int 
       fail_mode: secure 
       Port "qvo0ae19a43-b6" 
           tag: 2 
           Interface "qvo0ae19a43-b6" 
       Port br-int 
           Interface br-int 
               type: internal 
       Port "qvo31c89856-a2" 
           tag: 1 
           Interface "qvo31c89856-a2" 
       Port patch-tun 
           Interface patch-tun 
               type: patch 
               options: {peer=patch-int} 
       Port int-br-ex 
           Interface int-br-ex 
               type: patch 
               options: {peer=phy-br-ex} 
       Port "qvo97fef28a-ec" 
           tag: 2 
           Interface "qvo97fef28a-ec" 
   Bridge br-dpdk 
       Port br-dpdk 
           Interface br-dpdk 
               type: internal 
   Bridge "br0" 
       Port "br0" 
           Interface "br0" 
               type: internal 
       Port "dpdk0" 
           Interface "dpdk0" 
               type: dpdk 
       Port "vhost-user-2" 
           Interface "vhost-user-2" 
               type: dpdkvhostuser 
       Port "vhost-user-0" 
           Interface "vhost-user-0" 
               type: dpdkvhostuser 
       Port "vhost-user-1" 
           Interface "vhost-user-1" 
               type: dpdkvhostuser 
   ovs_version: "2.4.0" 
root@dpdk:~# 
_____________________________________________ 

Open flows output in bridge in compute node are as below: 
_____________________________________________ 
root@dpdk:~# ovs-ofctl dump-flows br-tun 
NXST_FLOW reply (xid=0x4): 
cookie=0x0, duration=71796.741s, table=0, n_packets=519, n_bytes=33794, 
idle_age=19982, hard_age=65534, priority=1,in_port=1 actions=resubmit(,2) 
cookie=0x0, duration=71796.700s, table=0, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=0 actions=drop 
cookie=0x0, duration=71796.649s, table=2, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, 
priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 
actions=resubmit(,20) 
cookie=0x0, duration=71796.610s, table=2, n_packets=519, n_bytes=33794, 
idle_age=19982, hard_age=65534, 
priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 
actions=resubmit(,22) 
cookie=0x0, duration=71794.631s, table=3, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=1,tun_id=0x5c 
actions=mod_vlan_vid:2,resubmit(,10) 
cookie=0x0, duration=71794.316s, table=3, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=1,tun_id=0x57 
actions=mod_vlan_vid:1,resubmit(,10) 
cookie=0x0, duration=71796.565s, table=3, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=0 actions=drop 
cookie=0x0, duration=71796.522s, table=4, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=0 actions=drop 
cookie=0x0, duration=71796.481s, table=10, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=1 
actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1 

cookie=0x0, duration=71796.439s, table=20, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=0 actions=resubmit(,22) 
cookie=0x0, duration=71796.398s, table=22, n_packets=519, n_bytes=33794, 
idle_age=19982, hard_age=65534, priority=0 actions=drop 
root@dpdk:~# 
root@dpdk:~# 
root@dpdk:~# 
root@dpdk:~# ovs-ofctl dump-flows br-tun 
int NXST_FLOW reply (xid=0x4): 
cookie=0x0, duration=71801.275s, table=0, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=2,in_port=10 actions=drop 
cookie=0x0, duration=71801.862s, table=0, n_packets=661, n_bytes=48912, 
idle_age=19981, hard_age=65534, priority=1 actions=NORMAL 
cookie=0x0, duration=71801.817s, table=23, n_packets=0, n_bytes=0, 
idle_age=65534, hard_age=65534, priority=0 actions=drop 
root@dpdk:~# 
_____________________________________________ 


Further we don't know what all the network changes(Packet Flow addition) 
if required for associating IP address through the DHCP. 

Would be really appreciate if have clarity on DHCP flow establishment. 



Thanks & Regards
Abhijeet Karve





From:        "Czesnowicz, Przemyslaw" <przemyslaw.czesnowicz@intel.com> 
To:        Abhijeet Karve <abhijeet.karve@tcs.com>, "Gray, Mark D" <
mark.d.gray@intel.com> 
Cc:        "dev@dpdk.org" <dev@dpdk.org>, "discuss@openvswitch.org" <
discuss@openvswitch.org> 
Date:        12/15/2015 09:13 PM 
Subject:        RE: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved# 
Successfully setup DPDK OVS with vhostuser 





Hi Abhijeet,

If you answer below questions it will help me understand your problem.

What do you mean by DPDK instance?
Are you able to communicate with other VM's on the same compute node?
Can you check if the DHCP requests arrive on the controller node? (I'm 
assuming this is at least compute+ controller setup)

Best regards
Przemek

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Abhijeet Karve
> Sent: Tuesday, December 15, 2015 5:56 AM
> To: Gray, Mark D
> Cc: dev@dpdk.org; discuss@openvswitch.org
> Subject: Re: [dpdk-dev] DPDK OVS on Ubuntu 14.04# Issue's Resolved#
> Successfully setup DPDK OVS with vhostuser
> 
> Dear All,
> 
> After seting up system boot parameters as shown below, the issue is
> resolved now & we are able to successfully setup openvswitch netdev-dpdk
> with vhostuser support.
> 
> __________________________________________________________
> _______________________________________________________
> Setup 2 sets of huge pages with different sizes. One for Vhost and 
another
> for Guest VM.
>          Edit /etc/default/grub.
>             GRUB_CMDLINE_LINUX="iommu=pt intel_iommu=on  hugepagesz=1G
> hugepages=10 hugepagesz=2M hugepages=4096"
>          # update-grub
>        - Mount the huge pages into different directory.
>           # sudo mount -t hugetlbfs nodev /mnt/huge_2M -o pagesize=2M
>           # sudo mount -t hugetlbfs nodev /mnt/huge_1G -o pagesize=1G
> __________________________________________________________
> _______________________________________________________
> 
> At present we are facing an issue in Testing DPDK application on setup. 
In our
> scenario, We have DPDK instance launched on top of the Openstack Kilo
> compute node. Not able to assign DHCP IP from controller.
> 
> 
> Thanks & Regards
> Abhijeet Karve
> 
> =====-----=====-----=====
> Notice: The information contained in this e-mail message and/or
> attachments to it may contain confidential or privileged information. If 
you
> are not the intended recipient, any dissemination, use, review, 
distribution,
> printing or copying of the information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If you have received 
this
> communication in error, please notify us by reply e-mail or telephone 
and
> immediately and permanently delete the message and any attachments.
> Thank you
>