Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

[Aaron Conole]

Bala Sankaran <bsankara@redhat.com> writes:

> Perfect!
>
> Thanks for the help.
>
> ----- Original Message -----
>> From: "Keith Wiles" <keith.wiles@intel.com>
>> To: "Bala Sankaran" <bsankara@redhat.com>
>> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
>> Sent: Thursday, July 5, 2018 11:41:46 AM
>> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
>> 
>> 
>> 
>> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com> wrote:
>> > 
>> > Greetings,
>> > 
>> > I am currently using dpdk version 17.11.2. I see that there are a few other
>> > revisions in 17.11.3, followed by the latest stable version of 18.02.2.
>> > 
>> > Based on the issues I have faced so far (see Original
>> > Message below), would you suggest that  I go for
>> > another version? If yes, which one? In essence, my question is, would
>> > resorting to a different version of dpdk solve my current issue of
>> > virtqueue id being invalid?
>> > 
>> > Any help is much appreciated.
>> 
>> From a support perspective using the latest version 18.05 or the long term
>> supported version 17.11.3 is easier for most to help. I would pick the
>> latest release 18.05 myself. As for fixing this problem I do not know. You
>> can look into the MAINTAINERS file and find the maintainers of area(s) and
>> include them in the CC line on your questions as sometimes they miss the
>> emails as the volume can be high at times.

Thanks Keith.

I took a quick look and it seems like the queues are not setting up
correctly between OvS and testpmd?  Probably there's a step missing
somewhere, although nothing in either the netdev-dpdk.c from OvS nor the
rte_ethdev was obvious to stand out to me.

I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to try?

>> > 
>> > Thanks
>> > 
>> > ----- Original Message -----
>> >> From: "Bala Sankaran" <bsankara@redhat.com>
>> >> To: users@dpdk.org
>> >> Cc: "Aaron Conole" <aconole@redhat.com>
>> >> Sent: Thursday, June 28, 2018 3:18:13 PM
>> >> Subject: Traffic doesn't forward on virtual devices
>> >> 
>> >> 
>> >> Hello team,
>> >> 
>> >> I am working on a project to do PVP tests on dpdk. As a first step, I
>> >> would
>> >> like to get traffic flow between tap devices. I'm in process of setting up
>> >> the architecture, in which I've used testpmd to forward traffic between
>> >> two
>> >> virtual devices(tap and vhost users) over a bridge.
>> >> 
>> >> While I'm at it, I've identified that the internal dev_attached flag never
>> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to manually set it
>> >> to 1 in the start routine, but I just see that the queue index being
>> >> referenced is out of range.
>> >> 
>> >> I'm not sure how to proceed.  Has anyone had luck using testpmd to
>> >> communicate with vhost-user devices?  If yes, any hints on a workaround?
>> >> 
>> >> Here's how I configured my setup after installing dpdk and openvswitch:
>> >> 
>> >> 1. To start ovs-ctl:
>> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
>> >> 
>> >> 2. Setup hugepages:
>> >> echo '2048' > /proc/sys/vm/nr_hugepages
>> >> 
>> >> 3. Add a new network namespace:
>> >> ip netns add ns1
>> >> 
>> >> 4. Add and set a bridge:
>> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0 datapath_type=netdev
>> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
>> >> ovs-vsctl show
>> >> 
>> >> 5. Add a vhost user to the bridge created:
>> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
>> >> type=dpdkvhostuserclient
>> >> 
>> >> 6. Execute bash on the network namespace:
>> >> ip netns exec ns1 bash
>> >> 
>> >> 7. Use testpmd and connect the namespaces:
>> >> testpmd --socket-mem=512
>> >> --vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
>> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
>> >> 
>> >> 
>> >> I repeated steps 3 - 7 for another network namespace on the same bridge.
>> >> Following this, in fresh terminals, I assigned IP addresses to the tap
>> >> devices created and tried pinging them. From port statistics,
>> >> I identified the above mentioned issue with the dev_attached and queue
>> >> statistics.
>> >> 
>> >> I would greatly appreciate any help from your end.
>> >> 
>> >> Thanks.
>> >> 
>> >> -------------------------------------------------
>> >> Bala Sankaran
>> >> Networking Services Intern
>> >> Red Hat Inc .,
>> >> 
>> > -------------------------------------------------
>> > Bala Sankaran
>> > Networking Services Intern
>> 
>> Regards,
>> Keith
>> 
>> 
>
> --------------------------------------------------
> Bala Sankaran
> Networking Services Intern
> Red Hat Inc .,

Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

[Loftus, Ciara]

> 
> Bala Sankaran <bsankara@redhat.com> writes:
> 
> > Perfect!
> >
> > Thanks for the help.
> >
> > ----- Original Message -----
> >> From: "Keith Wiles" <keith.wiles@intel.com>
> >> To: "Bala Sankaran" <bsankara@redhat.com>
> >> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
> >> Sent: Thursday, July 5, 2018 11:41:46 AM
> >> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
> >>
> >>
> >>
> >> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com>
> wrote:
> >> >
> >> > Greetings,
> >> >
> >> > I am currently using dpdk version 17.11.2. I see that there are a few
> other
> >> > revisions in 17.11.3, followed by the latest stable version of 18.02.2.
> >> >
> >> > Based on the issues I have faced so far (see Original
> >> > Message below), would you suggest that  I go for
> >> > another version? If yes, which one? In essence, my question is, would
> >> > resorting to a different version of dpdk solve my current issue of
> >> > virtqueue id being invalid?
> >> >
> >> > Any help is much appreciated.
> >>
> >> From a support perspective using the latest version 18.05 or the long term
> >> supported version 17.11.3 is easier for most to help. I would pick the
> >> latest release 18.05 myself. As for fixing this problem I do not know. You
> >> can look into the MAINTAINERS file and find the maintainers of area(s)
> and
> >> include them in the CC line on your questions as sometimes they miss the
> >> emails as the volume can be high at times.
> 
> Thanks Keith.
> 
> I took a quick look and it seems like the queues are not setting up
> correctly between OvS and testpmd?  Probably there's a step missing
> somewhere, although nothing in either the netdev-dpdk.c from OvS nor the
> rte_ethdev was obvious to stand out to me.
> 
> I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to try?

Hi,

I think the appropriate driver to use in this test on the test-pmd side might be virtio-user.
Follow the same steps just change your vdev test-pmd arguments to:
--vdev='net_virtio_user0,path=/usr/local/var/run/openvswitch/vhu0'

Thanks,
Ciara

> 
> >> >
> >> > Thanks
> >> >
> >> > ----- Original Message -----
> >> >> From: "Bala Sankaran" <bsankara@redhat.com>
> >> >> To: users@dpdk.org
> >> >> Cc: "Aaron Conole" <aconole@redhat.com>
> >> >> Sent: Thursday, June 28, 2018 3:18:13 PM
> >> >> Subject: Traffic doesn't forward on virtual devices
> >> >>
> >> >>
> >> >> Hello team,
> >> >>
> >> >> I am working on a project to do PVP tests on dpdk. As a first step, I
> >> >> would
> >> >> like to get traffic flow between tap devices. I'm in process of setting up
> >> >> the architecture, in which I've used testpmd to forward traffic
> between
> >> >> two
> >> >> virtual devices(tap and vhost users) over a bridge.
> >> >>
> >> >> While I'm at it, I've identified that the internal dev_attached flag never
> >> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to manually set it
> >> >> to 1 in the start routine, but I just see that the queue index being
> >> >> referenced is out of range.
> >> >>
> >> >> I'm not sure how to proceed.  Has anyone had luck using testpmd to
> >> >> communicate with vhost-user devices?  If yes, any hints on a
> workaround?
> >> >>
> >> >> Here's how I configured my setup after installing dpdk and
> openvswitch:
> >> >>
> >> >> 1. To start ovs-ctl:
> >> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
> >> >>
> >> >> 2. Setup hugepages:
> >> >> echo '2048' > /proc/sys/vm/nr_hugepages
> >> >>
> >> >> 3. Add a new network namespace:
> >> >> ip netns add ns1
> >> >>
> >> >> 4. Add and set a bridge:
> >> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0 datapath_type=netdev
> >> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
> >> >> ovs-vsctl show
> >> >>
> >> >> 5. Add a vhost user to the bridge created:
> >> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
> >> >> type=dpdkvhostuserclient
> >> >>
> >> >> 6. Execute bash on the network namespace:
> >> >> ip netns exec ns1 bash
> >> >>
> >> >> 7. Use testpmd and connect the namespaces:
> >> >> testpmd --socket-mem=512
> >> >> --
> vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
> >> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
> >> >>
> >> >>
> >> >> I repeated steps 3 - 7 for another network namespace on the same
> bridge.
> >> >> Following this, in fresh terminals, I assigned IP addresses to the tap
> >> >> devices created and tried pinging them. From port statistics,
> >> >> I identified the above mentioned issue with the dev_attached and
> queue
> >> >> statistics.
> >> >>
> >> >> I would greatly appreciate any help from your end.
> >> >>
> >> >> Thanks.
> >> >>
> >> >> -------------------------------------------------
> >> >> Bala Sankaran
> >> >> Networking Services Intern
> >> >> Red Hat Inc .,
> >> >>
> >> > -------------------------------------------------
> >> > Bala Sankaran
> >> > Networking Services Intern
> >>
> >> Regards,
> >> Keith
> >>
> >>
> >
> > --------------------------------------------------
> > Bala Sankaran
> > Networking Services Intern
> > Red Hat Inc .,

Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

[Bala Sankaran]

----- Original Message -----
> From: "Ciara Loftus" <ciara.loftus@intel.com>
> To: "Aaron Conole" <aconole@redhat.com>, "Bala Sankaran" <bsankara@redhat.com>
> Cc: "Keith Wiles" <keith.wiles@intel.com>, users@dpdk.org, dev@dpdk.org, "Maxime Coquelin"
> <maxime.coquelin@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>
> Sent: Tuesday, July 10, 2018 4:20:31 AM
> Subject: RE: [dpdk-users] Traffic doesn't forward on virtual devices
> 
> > 
> > Bala Sankaran <bsankara@redhat.com> writes:
> > 
> > > Perfect!
> > >
> > > Thanks for the help.
> > >
> > > ----- Original Message -----
> > >> From: "Keith Wiles" <keith.wiles@intel.com>
> > >> To: "Bala Sankaran" <bsankara@redhat.com>
> > >> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
> > >> Sent: Thursday, July 5, 2018 11:41:46 AM
> > >> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
> > >>
> > >>
> > >>
> > >> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com>
> > wrote:
> > >> >
> > >> > Greetings,
> > >> >
> > >> > I am currently using dpdk version 17.11.2. I see that there are a few
> > other
> > >> > revisions in 17.11.3, followed by the latest stable version of
> > >> > 18.02.2.
> > >> >
> > >> > Based on the issues I have faced so far (see Original
> > >> > Message below), would you suggest that  I go for
> > >> > another version? If yes, which one? In essence, my question is, would
> > >> > resorting to a different version of dpdk solve my current issue of
> > >> > virtqueue id being invalid?
> > >> >
> > >> > Any help is much appreciated.
> > >>
> > >> From a support perspective using the latest version 18.05 or the long
> > >> term
> > >> supported version 17.11.3 is easier for most to help. I would pick the
> > >> latest release 18.05 myself. As for fixing this problem I do not know.
> > >> You
> > >> can look into the MAINTAINERS file and find the maintainers of area(s)
> > and
> > >> include them in the CC line on your questions as sometimes they miss the
> > >> emails as the volume can be high at times.
> > 
> > Thanks Keith.
> > 
> > I took a quick look and it seems like the queues are not setting up
> > correctly between OvS and testpmd?  Probably there's a step missing
> > somewhere, although nothing in either the netdev-dpdk.c from OvS nor the
> > rte_ethdev was obvious to stand out to me.
> > 
> > I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to try?
> 
> Hi,
> 
> I think the appropriate driver to use in this test on the test-pmd side might
> be virtio-user.
> Follow the same steps just change your vdev test-pmd arguments to:
> --vdev='net_virtio_user0,path=/usr/local/var/run/openvswitch/vhu0'
> 
> Thanks,
> Ciara
> 

Thank you for your response.

I tried using virtio-user, but I face an error that says: Failed to prepare memory for vhost-user.
The command I ran is as below: 

[root@localhost openvswitch]# testpmd --socket-mem=1024 --vdev='net_virtio_user1,path=/usr/local/var/run/openvswitch/vhu1,server=1'  --vdev='net_tap1,iface=tap1' --file-prefix page1 -- -i
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/page1/mp_socket
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
rte_pmd_tap_probe(): Initializing pmd_tap for net_tap1 as tap1
Interactive-mode selected
Warning: NUMA should be configured manually by using --port-numa-config and --ring-numa-config parameters along with --numa.
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
virtio_user_server_reconnect(): WARNING: Some features 0x1801 are not supported by vhost-user!
get_hugepage_file_info(): Exceed maximum of 8
prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
Port 0: DA:60:01:0C:4B:29
Configuring Port 1 (socket 0)
Port 1: D2:5A:94:68:AF:B3
Checking link statuses...

Port 0: LSC event
Done

I tried increasing the socket-memory, I checked /proc/meminfo and found there were over 
1280 free hugepages. 
So my understanding is that this is not an issue where I don't have enough hugepages.

Can you provide leads on what's wrong here?

> > 
> > >> >
> > >> > Thanks
> > >> >
> > >> > ----- Original Message -----
> > >> >> From: "Bala Sankaran" <bsankara@redhat.com>
> > >> >> To: users@dpdk.org
> > >> >> Cc: "Aaron Conole" <aconole@redhat.com>
> > >> >> Sent: Thursday, June 28, 2018 3:18:13 PM
> > >> >> Subject: Traffic doesn't forward on virtual devices
> > >> >>
> > >> >>
> > >> >> Hello team,
> > >> >>
> > >> >> I am working on a project to do PVP tests on dpdk. As a first step, I
> > >> >> would
> > >> >> like to get traffic flow between tap devices. I'm in process of
> > >> >> setting up
> > >> >> the architecture, in which I've used testpmd to forward traffic
> > between
> > >> >> two
> > >> >> virtual devices(tap and vhost users) over a bridge.
> > >> >>
> > >> >> While I'm at it, I've identified that the internal dev_attached flag
> > >> >> never
> > >> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to manually
> > >> >> set it
> > >> >> to 1 in the start routine, but I just see that the queue index being
> > >> >> referenced is out of range.
> > >> >>
> > >> >> I'm not sure how to proceed.  Has anyone had luck using testpmd to
> > >> >> communicate with vhost-user devices?  If yes, any hints on a
> > workaround?
> > >> >>
> > >> >> Here's how I configured my setup after installing dpdk and
> > openvswitch:
> > >> >>
> > >> >> 1. To start ovs-ctl:
> > >> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
> > >> >>
> > >> >> 2. Setup hugepages:
> > >> >> echo '2048' > /proc/sys/vm/nr_hugepages
> > >> >>
> > >> >> 3. Add a new network namespace:
> > >> >> ip netns add ns1
> > >> >>
> > >> >> 4. Add and set a bridge:
> > >> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0 datapath_type=netdev
> > >> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
> > >> >> ovs-vsctl show
> > >> >>
> > >> >> 5. Add a vhost user to the bridge created:
> > >> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
> > >> >> type=dpdkvhostuserclient
> > >> >>
> > >> >> 6. Execute bash on the network namespace:
> > >> >> ip netns exec ns1 bash
> > >> >>
> > >> >> 7. Use testpmd and connect the namespaces:
> > >> >> testpmd --socket-mem=512
> > >> >> --
> > vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
> > >> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
> > >> >>
> > >> >>
> > >> >> I repeated steps 3 - 7 for another network namespace on the same
> > bridge.
> > >> >> Following this, in fresh terminals, I assigned IP addresses to the
> > >> >> tap
> > >> >> devices created and tried pinging them. From port statistics,
> > >> >> I identified the above mentioned issue with the dev_attached and
> > queue
> > >> >> statistics.
> > >> >>
> > >> >> I would greatly appreciate any help from your end.
> > >> >>
> > >> >> Thanks.
> > >> >>
> > >> >> -------------------------------------------------
> > >> >> Bala Sankaran
> > >> >> Networking Services Intern
> > >> >> Red Hat Inc .,
> > >> >>
> > >> > -------------------------------------------------
> > >> > Bala Sankaran
> > >> > Networking Services Intern
> > >>
> > >> Regards,
> > >> Keith
> > >>
> > >>
> > >
> > > --------------------------------------------------
> > > Bala Sankaran
> > > Networking Services Intern
> > > Red Hat Inc .,
> 

Thanks.
Bala.

Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

[Loftus, Ciara]

> > >
> > > Bala Sankaran <bsankara@redhat.com> writes:
> > >
> > > > Perfect!
> > > >
> > > > Thanks for the help.
> > > >
> > > > ----- Original Message -----
> > > >> From: "Keith Wiles" <keith.wiles@intel.com>
> > > >> To: "Bala Sankaran" <bsankara@redhat.com>
> > > >> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
> > > >> Sent: Thursday, July 5, 2018 11:41:46 AM
> > > >> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
> > > >>
> > > >>
> > > >>
> > > >> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com>
> > > wrote:
> > > >> >
> > > >> > Greetings,
> > > >> >
> > > >> > I am currently using dpdk version 17.11.2. I see that there are a few
> > > other
> > > >> > revisions in 17.11.3, followed by the latest stable version of
> > > >> > 18.02.2.
> > > >> >
> > > >> > Based on the issues I have faced so far (see Original
> > > >> > Message below), would you suggest that  I go for
> > > >> > another version? If yes, which one? In essence, my question is,
> would
> > > >> > resorting to a different version of dpdk solve my current issue of
> > > >> > virtqueue id being invalid?
> > > >> >
> > > >> > Any help is much appreciated.
> > > >>
> > > >> From a support perspective using the latest version 18.05 or the long
> > > >> term
> > > >> supported version 17.11.3 is easier for most to help. I would pick the
> > > >> latest release 18.05 myself. As for fixing this problem I do not know.
> > > >> You
> > > >> can look into the MAINTAINERS file and find the maintainers of area(s)
> > > and
> > > >> include them in the CC line on your questions as sometimes they miss
> the
> > > >> emails as the volume can be high at times.
> > >
> > > Thanks Keith.
> > >
> > > I took a quick look and it seems like the queues are not setting up
> > > correctly between OvS and testpmd?  Probably there's a step missing
> > > somewhere, although nothing in either the netdev-dpdk.c from OvS nor
> the
> > > rte_ethdev was obvious to stand out to me.
> > >
> > > I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to try?
> >
> > Hi,
> >
> > I think the appropriate driver to use in this test on the test-pmd side might
> > be virtio-user.
> > Follow the same steps just change your vdev test-pmd arguments to:
> > --vdev='net_virtio_user0,path=/usr/local/var/run/openvswitch/vhu0'
> >
> > Thanks,
> > Ciara
> >
> 
> Thank you for your response.
> 
> I tried using virtio-user, but I face an error that says: Failed to prepare
> memory for vhost-user.
> The command I ran is as below:
> 
> [root@localhost openvswitch]# testpmd --socket-mem=1024 --
> vdev='net_virtio_user1,path=/usr/local/var/run/openvswitch/vhu1,server=
> 1'  --vdev='net_tap1,iface=tap1' --file-prefix page1 -- -i
> EAL: Detected 4 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/page1/mp_socket
> EAL: Probing VFIO support...
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> unreliable clock cycles !
> rte_pmd_tap_probe(): Initializing pmd_tap for net_tap1 as tap1
> Interactive-mode selected
> Warning: NUMA should be configured manually by using --port-numa-config
> and --ring-numa-config parameters along with --numa.
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Configuring Port 0 (socket 0)
> virtio_user_server_reconnect(): WARNING: Some features 0x1801 are not
> supported by vhost-user!
> get_hugepage_file_info(): Exceed maximum of 8
> prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
> Port 0: DA:60:01:0C:4B:29
> Configuring Port 1 (socket 0)
> Port 1: D2:5A:94:68:AF:B3
> Checking link statuses...
> 
> Port 0: LSC event
> Done
> 
> I tried increasing the socket-memory, I checked /proc/meminfo and found
> there were over
> 1280 free hugepages.
> So my understanding is that this is not an issue where I don't have enough
> hugepages.
> 
> Can you provide leads on what's wrong here?

Hi,

The limitations section for Virtio User https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html#limitations states that:

" Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. If you have more regions (especially when 2MB hugepages are used), the option, --single-file-segments, can help to reduce the number of shared files."

Suggest using the --single-file-segments option on the test-pmd command line or failing that increasing your hugepage size to 1G https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#use-of-hugepages-in-the-linux-environment

Thanks,
Ciara

> 
> > >
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> > ----- Original Message -----
> > > >> >> From: "Bala Sankaran" <bsankara@redhat.com>
> > > >> >> To: users@dpdk.org
> > > >> >> Cc: "Aaron Conole" <aconole@redhat.com>
> > > >> >> Sent: Thursday, June 28, 2018 3:18:13 PM
> > > >> >> Subject: Traffic doesn't forward on virtual devices
> > > >> >>
> > > >> >>
> > > >> >> Hello team,
> > > >> >>
> > > >> >> I am working on a project to do PVP tests on dpdk. As a first step, I
> > > >> >> would
> > > >> >> like to get traffic flow between tap devices. I'm in process of
> > > >> >> setting up
> > > >> >> the architecture, in which I've used testpmd to forward traffic
> > > between
> > > >> >> two
> > > >> >> virtual devices(tap and vhost users) over a bridge.
> > > >> >>
> > > >> >> While I'm at it, I've identified that the internal dev_attached flag
> > > >> >> never
> > > >> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to manually
> > > >> >> set it
> > > >> >> to 1 in the start routine, but I just see that the queue index being
> > > >> >> referenced is out of range.
> > > >> >>
> > > >> >> I'm not sure how to proceed.  Has anyone had luck using testpmd
> to
> > > >> >> communicate with vhost-user devices?  If yes, any hints on a
> > > workaround?
> > > >> >>
> > > >> >> Here's how I configured my setup after installing dpdk and
> > > openvswitch:
> > > >> >>
> > > >> >> 1. To start ovs-ctl:
> > > >> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
> > > >> >>
> > > >> >> 2. Setup hugepages:
> > > >> >> echo '2048' > /proc/sys/vm/nr_hugepages
> > > >> >>
> > > >> >> 3. Add a new network namespace:
> > > >> >> ip netns add ns1
> > > >> >>
> > > >> >> 4. Add and set a bridge:
> > > >> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0
> datapath_type=netdev
> > > >> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
> > > >> >> ovs-vsctl show
> > > >> >>
> > > >> >> 5. Add a vhost user to the bridge created:
> > > >> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
> > > >> >> type=dpdkvhostuserclient
> > > >> >>
> > > >> >> 6. Execute bash on the network namespace:
> > > >> >> ip netns exec ns1 bash
> > > >> >>
> > > >> >> 7. Use testpmd and connect the namespaces:
> > > >> >> testpmd --socket-mem=512
> > > >> >> --
> > >
> vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
> > > >> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
> > > >> >>
> > > >> >>
> > > >> >> I repeated steps 3 - 7 for another network namespace on the same
> > > bridge.
> > > >> >> Following this, in fresh terminals, I assigned IP addresses to the
> > > >> >> tap
> > > >> >> devices created and tried pinging them. From port statistics,
> > > >> >> I identified the above mentioned issue with the dev_attached and
> > > queue
> > > >> >> statistics.
> > > >> >>
> > > >> >> I would greatly appreciate any help from your end.
> > > >> >>
> > > >> >> Thanks.
> > > >> >>
> > > >> >> -------------------------------------------------
> > > >> >> Bala Sankaran
> > > >> >> Networking Services Intern
> > > >> >> Red Hat Inc .,
> > > >> >>
> > > >> > -------------------------------------------------
> > > >> > Bala Sankaran
> > > >> > Networking Services Intern
> > > >>
> > > >> Regards,
> > > >> Keith
> > > >>
> > > >>
> > > >
> > > > --------------------------------------------------
> > > > Bala Sankaran
> > > > Networking Services Intern
> > > > Red Hat Inc .,
> >
> 
> Thanks.
> Bala.

Re: [dpdk-dev] [dpdk-users] Traffic doesn't forward on virtual devices

[Bala Sankaran]

----- Original Message -----
> From: "Ciara Loftus" <ciara.loftus@intel.com>
> To: "Bala Sankaran" <bsankara@redhat.com>
> Cc: "Aaron Conole" <aconole@redhat.com>, "Keith Wiles" <keith.wiles@intel.com>, users@dpdk.org, dev@dpdk.org, "Maxime
> Coquelin" <maxime.coquelin@redhat.com>, "Ian Stokes" <ian.stokes@intel.com>
> Sent: Wednesday, July 11, 2018 4:12:38 AM
> Subject: RE: [dpdk-users] Traffic doesn't forward on virtual devices
> 
> > > >
> > > > Bala Sankaran <bsankara@redhat.com> writes:
> > > >
> > > > > Perfect!
> > > > >
> > > > > Thanks for the help.
> > > > >
> > > > > ----- Original Message -----
> > > > >> From: "Keith Wiles" <keith.wiles@intel.com>
> > > > >> To: "Bala Sankaran" <bsankara@redhat.com>
> > > > >> Cc: users@dpdk.org, "Aaron Conole" <aconole@redhat.com>
> > > > >> Sent: Thursday, July 5, 2018 11:41:46 AM
> > > > >> Subject: Re: [dpdk-users] Traffic doesn't forward on virtual devices
> > > > >>
> > > > >>
> > > > >>
> > > > >> > On Jul 5, 2018, at 9:53 AM, Bala Sankaran <bsankara@redhat.com>
> > > > wrote:
> > > > >> >
> > > > >> > Greetings,
> > > > >> >
> > > > >> > I am currently using dpdk version 17.11.2. I see that there are a
> > > > >> > few
> > > > other
> > > > >> > revisions in 17.11.3, followed by the latest stable version of
> > > > >> > 18.02.2.
> > > > >> >
> > > > >> > Based on the issues I have faced so far (see Original
> > > > >> > Message below), would you suggest that  I go for
> > > > >> > another version? If yes, which one? In essence, my question is,
> > would
> > > > >> > resorting to a different version of dpdk solve my current issue of
> > > > >> > virtqueue id being invalid?
> > > > >> >
> > > > >> > Any help is much appreciated.
> > > > >>
> > > > >> From a support perspective using the latest version 18.05 or the
> > > > >> long
> > > > >> term
> > > > >> supported version 17.11.3 is easier for most to help. I would pick
> > > > >> the
> > > > >> latest release 18.05 myself. As for fixing this problem I do not
> > > > >> know.
> > > > >> You
> > > > >> can look into the MAINTAINERS file and find the maintainers of
> > > > >> area(s)
> > > > and
> > > > >> include them in the CC line on your questions as sometimes they miss
> > the
> > > > >> emails as the volume can be high at times.
> > > >
> > > > Thanks Keith.
> > > >
> > > > I took a quick look and it seems like the queues are not setting up
> > > > correctly between OvS and testpmd?  Probably there's a step missing
> > > > somewhere, although nothing in either the netdev-dpdk.c from OvS nor
> > the
> > > > rte_ethdev was obvious to stand out to me.
> > > >
> > > > I've CC'd Maxime, Ian, and Ciara - maybe they have a better idea to
> > > > try?
> > >
> > > Hi,
> > >
> > > I think the appropriate driver to use in this test on the test-pmd side
> > > might
> > > be virtio-user.
> > > Follow the same steps just change your vdev test-pmd arguments to:
> > > --vdev='net_virtio_user0,path=/usr/local/var/run/openvswitch/vhu0'
> > >
> > > Thanks,
> > > Ciara
> > >
> > 
> > Thank you for your response.
> > 
> > I tried using virtio-user, but I face an error that says: Failed to prepare
> > memory for vhost-user.
> > The command I ran is as below:
> > 
> > [root@localhost openvswitch]# testpmd --socket-mem=1024 --
> > vdev='net_virtio_user1,path=/usr/local/var/run/openvswitch/vhu1,server=
> > 1'  --vdev='net_tap1,iface=tap1' --file-prefix page1 -- -i
> > EAL: Detected 4 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/page1/mp_socket
> > EAL: Probing VFIO support...
> > EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > unreliable clock cycles !
> > rte_pmd_tap_probe(): Initializing pmd_tap for net_tap1 as tap1
> > Interactive-mode selected
> > Warning: NUMA should be configured manually by using --port-numa-config
> > and --ring-numa-config parameters along with --numa.
> > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> > size=2176, socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Configuring Port 0 (socket 0)
> > virtio_user_server_reconnect(): WARNING: Some features 0x1801 are not
> > supported by vhost-user!
> > get_hugepage_file_info(): Exceed maximum of 8
> > prepare_vhost_memory_user(): Failed to prepare memory for vhost-user
> > Port 0: DA:60:01:0C:4B:29
> > Configuring Port 1 (socket 0)
> > Port 1: D2:5A:94:68:AF:B3
> > Checking link statuses...
> > 
> > Port 0: LSC event
> > Done
> > 
> > I tried increasing the socket-memory, I checked /proc/meminfo and found
> > there were over
> > 1280 free hugepages.
> > So my understanding is that this is not an issue where I don't have enough
> > hugepages.
> > 
> > Can you provide leads on what's wrong here?
> 
> Hi,
> 
> The limitations section for Virtio User
> https://doc.dpdk.org/guides/howto/virtio_user_for_container_networking.html#limitations
> states that:
> 
> " Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8)
> hugepages. If you have more regions (especially when 2MB hugepages are
> used), the option, --single-file-segments, can help to reduce the number of
> shared files."
> 
> Suggest using the --single-file-segments option on the test-pmd command line
> or failing that increasing your hugepage size to 1G
> https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#use-of-hugepages-in-the-linux-environment
> 
> Thanks,
> Ciara

Thanks a ton. 

Looks like I had just missed out on adding the single-file-segments argument.
After addition to my existing command structure, testpmd is setup and further,
I am able to ping the tap devices. 

Thanks
Bala.

> 
> > 
> > > >
> > > > >> >
> > > > >> > Thanks
> > > > >> >
> > > > >> > ----- Original Message -----
> > > > >> >> From: "Bala Sankaran" <bsankara@redhat.com>
> > > > >> >> To: users@dpdk.org
> > > > >> >> Cc: "Aaron Conole" <aconole@redhat.com>
> > > > >> >> Sent: Thursday, June 28, 2018 3:18:13 PM
> > > > >> >> Subject: Traffic doesn't forward on virtual devices
> > > > >> >>
> > > > >> >>
> > > > >> >> Hello team,
> > > > >> >>
> > > > >> >> I am working on a project to do PVP tests on dpdk. As a first
> > > > >> >> step, I
> > > > >> >> would
> > > > >> >> like to get traffic flow between tap devices. I'm in process of
> > > > >> >> setting up
> > > > >> >> the architecture, in which I've used testpmd to forward traffic
> > > > between
> > > > >> >> two
> > > > >> >> virtual devices(tap and vhost users) over a bridge.
> > > > >> >>
> > > > >> >> While I'm at it, I've identified that the internal dev_attached
> > > > >> >> flag
> > > > >> >> never
> > > > >> >> gets set to 1 from the rte_eth_vhost.c file. I've tried to
> > > > >> >> manually
> > > > >> >> set it
> > > > >> >> to 1 in the start routine, but I just see that the queue index
> > > > >> >> being
> > > > >> >> referenced is out of range.
> > > > >> >>
> > > > >> >> I'm not sure how to proceed.  Has anyone had luck using testpmd
> > to
> > > > >> >> communicate with vhost-user devices?  If yes, any hints on a
> > > > workaround?
> > > > >> >>
> > > > >> >> Here's how I configured my setup after installing dpdk and
> > > > openvswitch:
> > > > >> >>
> > > > >> >> 1. To start ovs-ctl:
> > > > >> >> /usr/local/share/openvswitch/scripts/ovs-ctl start
> > > > >> >>
> > > > >> >> 2. Setup hugepages:
> > > > >> >> echo '2048' > /proc/sys/vm/nr_hugepages
> > > > >> >>
> > > > >> >> 3. Add a new network namespace:
> > > > >> >> ip netns add ns1
> > > > >> >>
> > > > >> >> 4. Add and set a bridge:
> > > > >> >> ovs-vsctl add-br dpdkbr0 -- set Bridge dpdkbr0
> > datapath_type=netdev
> > > > >> >> options:vhost-server-path=/usr/local/var/run/openvswitch/vhu0
> > > > >> >> ovs-vsctl show
> > > > >> >>
> > > > >> >> 5. Add a vhost user to the bridge created:
> > > > >> >> ovs-vsctl add-port dpdkbr0 vhu0 -- set Interface vhu0
> > > > >> >> type=dpdkvhostuserclient
> > > > >> >>
> > > > >> >> 6. Execute bash on the network namespace:
> > > > >> >> ip netns exec ns1 bash
> > > > >> >>
> > > > >> >> 7. Use testpmd and connect the namespaces:
> > > > >> >> testpmd --socket-mem=512
> > > > >> >> --
> > > >
> > vdev='eth_vhost0,iface=/usr/local/var/run/openvswitch/vhu0,queues=1'
> > > > >> >> --vdev='net_tap0,iface=tap0' --file-prefix page0 -- -i
> > > > >> >>
> > > > >> >>
> > > > >> >> I repeated steps 3 - 7 for another network namespace on the same
> > > > bridge.
> > > > >> >> Following this, in fresh terminals, I assigned IP addresses to
> > > > >> >> the
> > > > >> >> tap
> > > > >> >> devices created and tried pinging them. From port statistics,
> > > > >> >> I identified the above mentioned issue with the dev_attached and
> > > > queue
> > > > >> >> statistics.
> > > > >> >>
> > > > >> >> I would greatly appreciate any help from your end.
> > > > >> >>
> > > > >> >> Thanks.
> > > > >> >>
> > > > >> >> -------------------------------------------------
> > > > >> >> Bala Sankaran
> > > > >> >> Networking Services Intern
> > > > >> >> Red Hat Inc .,
> > > > >> >>
> > > > >> > -------------------------------------------------
> > > > >> > Bala Sankaran
> > > > >> > Networking Services Intern
> > > > >>
> > > > >> Regards,
> > > > >> Keith
> > > > >>
> > > > >>
> > > > >
> > > > > --------------------------------------------------
> > > > > Bala Sankaran
> > > > > Networking Services Intern
> > > > > Red Hat Inc .,
> > >
> > 
> > Thanks.
> > Bala.
>