Re: [dpdk-users] Traffic doesn't forward on virtual devices

[Bala Sankaran <bsankara@redhat.com>]

----- Original Message -----
> From: "Ciara Loftus" <ciara.loftus@intel.com>
> To: "Bala Sankaran" <bsankara@redhat.com>
> Cc: "Aaron Conole" <aconole@redhat.com>, "Keith Wiles" <keith.wiles@intel.com>, users@dpdk.org, dev@dpdk.org, "Maxime
> Coquelin" <maxime.coquelin@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>
> Sent: Wednesday, July 11, 2018 4:12:38 AM
> Subject: RE: [dpdk-users] Traffic doesn't forward on virtual devices
> 
> > > >
> > > > Bala Sankaran <bsankara@redhat.com> writes:
> > > >
> > > > > Perfect!
> > > > >
> > > > > Thanks for the help.
> > > > >
> > > > > ----- Original Message -----
> > > > >> From: "Keith Wiles" <keith.wiles@intel.com>
> > > > >> To: "Bala Sankaran" <bsankara@redhat.com>
> > > > >> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
> > > > >> Sent: Thursday, July 5, 2018 11:41:46 AM
> > > > >> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
> > > > >>
> > > > >>
> > > > >>
> > > > >> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com>
> > > > wrote:
> > > > >> >
> > > > >> > Greetings,
> > > > >> >
> > > > >> > I am currently using dpdk version 17.11.2. I see that there are a
> > > > >> > few
> > > > other
> > > > >> > revisions in 17.11.3, followed by the latest stable version of
> > > > >> > 18.02.2.
> > > > >> >
> > > > >> > Based on the issues I have faced so far (see Original
> > > > >> > Message below), would you suggest that  I go for
> > > > >> > another version? If yes, which one? In essence, my question is,
> > would
> > > > >> > resorting to a different version of dpdk solve my current issue of
> > > > >> > virtqueue id being invalid?
> > > > >> >
> > > > >> > Any help is much appreciated.
> > > > >>
> > > > >> From a support perspective using the latest version 18.05 or the
> > > > >> long
> > > > >> term
> > > > >> supported version 17.11.3 is easier for most to help. I would pick
> > > > >> the
> > > > >> latest release 18.05 myself. As for fixing this problem I do not
> > > > >> know.
> > > > >> You
> > > > >> can look into the MAINTAINERS file and find the maintainers of
> > > > >> area(s)
> > > > and
> > > > >> include them in the CC line on your questions as sometimes they miss
> > the
> > > > >> emails as the volume can be high at times.
> > > >
> > > > Thanks Keith.
> > > >
> > > > I took a quick look and it seems like the queues are not setting up
> > > > correctly between OvS and testpmd?  Probably there's a step missing
> > > > somewhere, although nothing in either the netdev-dpdk.c from OvS nor
> > the
> > > > rte_ethdev was obvious to stand out to me.
> > > >
> > > > I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to
> > > > try?
> > >
> > > Hi,
> > >
> > > I think the appropriate driver to use in this test on the test-pmd side
> > > might
> > > be virtio-user.
> > > Follow the same steps just change your vdev test-pmd arguments to:
> > > --vdev='net_virtio_user0,path=/usr/local/var/run/openvswitch/vhu0'
> > >
> > > Thanks,
> > > Ciara
> > >
> > 
> > Thank you for your response.
> > 
> > I tried using virtio-user, but I face an error that says: Failed to prepare
> > memory for vhost-user.
> > The command I ran is as below:
> > 
> > [root@localhost openvswitch]# testpmd --socket-mem=1024 --
> > vdev='net_virtio_user1,path=/usr/local/var/run/openvswitch/vhu1,server=
> > 1'  --vdev='net_tap1,iface=tap1' --file-prefix page1 -- -i
> > EAL: Detected 4 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/page1/mp_socket
> > EAL: Probing VFIO support...
> > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > unreliable clock cycles !
> > rte_pmd_tap_probe(): Initializing pmd_tap for net_tap1 as tap1
> > Interactive-mode selected
> > Warning: NUMA should be configured manually by using --port-numa-config
> > and --ring-numa-config parameters along with --numa.
> > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> > size=2176, socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Configuring Port 0 (socket 0)
> > virtio_user_server_reconnect(): WARNING: Some features 0x1801 are not
> > supported by vhost-user!
> > get_hugepage_file_info(): Exceed maximum of 8
> > prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
> > Port 0: DA:60:01:0C:4B:29
> > Configuring Port 1 (socket 0)
> > Port 1: D2:5A:94:68:AF:B3
> > Checking link statuses...
> > 
> > Port 0: LSC event
> > Done
> > 
> > I tried increasing the socket-memory, I checked /proc/meminfo and found
> > there were over
> > 1280 free hugepages.
> > So my understanding is that this is not an issue where I don't have enough
> > hugepages.
> > 
> > Can you provide leads on what's wrong here?
> 
> Hi,
> 
> The limitations section for Virtio User
> https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html#limitations
> states that:
> 
> " Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8)
> hugepages. If you have more regions (especially when 2MB hugepages are
> used), the option, --single-file-segments, can help to reduce the number of
> shared files."
> 
> Suggest using the --single-file-segments option on the test-pmd command line
> or failing that increasing your hugepage size to 1G
> https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#use-of-hugepages-in-the-linux-environment
> 
> Thanks,
> Ciara

Thanks a ton. 

Looks like I had just missed out on adding the single-file-segments argument.
After addition to my existing command structure, testpmd is setup and further,
I am able to ping the tap devices. 

Thanks
Bala.

> 
> > 
> > > >
> > > > >> >
> > > > >> > Thanks
> > > > >> >
> > > > >> > ----- Original Message -----
> > > > >> >> From: "Bala Sankaran" <bsankara@redhat.com>
> > > > >> >> To: users@dpdk.org
> > > > >> >> Cc: "Aaron Conole" <aconole@redhat.com>
> > > > >> >> Sent: Thursday, June 28, 2018 3:18:13 PM
> > > > >> >> Subject: Traffic doesn't forward on virtual devices
> > > > >> >>
> > > > >> >>
> > > > >> >> Hello team,
> > > > >> >>
> > > > >> >> I am working on a project to do PVP tests on dpdk. As a first
> > > > >> >> step, I
> > > > >> >> would
> > > > >> >> like to get traffic flow between tap devices. I'm in process of
> > > > >> >> setting up
> > > > >> >> the architecture, in which I've used testpmd to forward traffic
> > > > between
> > > > >> >> two
> > > > >> >> virtual devices(tap and vhost users) over a bridge.
> > > > >> >>
> > > > >> >> While I'm at it, I've identified that the internal dev_attached
> > > > >> >> flag
> > > > >> >> never
> > > > >> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to
> > > > >> >> manually
> > > > >> >> set it
> > > > >> >> to 1 in the start routine, but I just see that the queue index
> > > > >> >> being
> > > > >> >> referenced is out of range.
> > > > >> >>
> > > > >> >> I'm not sure how to proceed.  Has anyone had luck using testpmd
> > to
> > > > >> >> communicate with vhost-user devices?  If yes, any hints on a
> > > > workaround?
> > > > >> >>
> > > > >> >> Here's how I configured my setup after installing dpdk and
> > > > openvswitch:
> > > > >> >>
> > > > >> >> 1. To start ovs-ctl:
> > > > >> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
> > > > >> >>
> > > > >> >> 2. Setup hugepages:
> > > > >> >> echo '2048' > /proc/sys/vm/nr_hugepages
> > > > >> >>
> > > > >> >> 3. Add a new network namespace:
> > > > >> >> ip netns add ns1
> > > > >> >>
> > > > >> >> 4. Add and set a bridge:
> > > > >> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0
> > datapath_type=netdev
> > > > >> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
> > > > >> >> ovs-vsctl show
> > > > >> >>
> > > > >> >> 5. Add a vhost user to the bridge created:
> > > > >> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
> > > > >> >> type=dpdkvhostuserclient
> > > > >> >>
> > > > >> >> 6. Execute bash on the network namespace:
> > > > >> >> ip netns exec ns1 bash
> > > > >> >>
> > > > >> >> 7. Use testpmd and connect the namespaces:
> > > > >> >> testpmd --socket-mem=512
> > > > >> >> --
> > > >
> > vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
> > > > >> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
> > > > >> >>
> > > > >> >>
> > > > >> >> I repeated steps 3 - 7 for another network namespace on the same
> > > > bridge.
> > > > >> >> Following this, in fresh terminals, I assigned IP addresses to
> > > > >> >> the
> > > > >> >> tap
> > > > >> >> devices created and tried pinging them. From port statistics,
> > > > >> >> I identified the above mentioned issue with the dev_attached and
> > > > queue
> > > > >> >> statistics.
> > > > >> >>
> > > > >> >> I would greatly appreciate any help from your end.
> > > > >> >>
> > > > >> >> Thanks.
> > > > >> >>
> > > > >> >> -------------------------------------------------
> > > > >> >> Bala Sankaran
> > > > >> >> Networking Services Intern
> > > > >> >> Red Hat Inc .,
> > > > >> >>
> > > > >> > -------------------------------------------------
> > > > >> > Bala Sankaran
> > > > >> > Networking Services Intern
> > > > >>
> > > > >> Regards,
> > > > >> Keith
> > > > >>
> > > > >>
> > > > >
> > > > > --------------------------------------------------
> > > > > Bala Sankaran
> > > > > Networking Services Intern
> > > > > Red Hat Inc .,
> > >
> > 
> > Thanks.
> > Bala.
>