[dpdk-users] Virtio-user exceptional path performance

[dpdk-users] Virtio-user exceptional path performance

[Eric Tremblay]

Hello,

I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).

Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.

Regards,

Eric

Re: [dpdk-users] Virtio-user exceptional path performance

[Tiwei Bie]

On Tue, Sep 04, 2018 at 05:54:20PM +0000, Eric Tremblay wrote:
> Hello,
> 
> I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).

Somehow below fix isn't backported to 17.11 while the
commit it fixed was backported.

https://git.dpdk.org/dpdk/commit/?id=bce7e9050f9b
commit bce7e9050f9b ("net/virtio-user: fix start with kernel vhost")

Please apply above fix locally and see if it works.

Thanks

> 
> Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.
> 
> Regards,
> 
> Eric
> 

Re: [dpdk-users] Virtio-user exceptional path performance

[Eric Tremblay]

Thanks for the reply.

With the patch, the tap interface is created in the kernel even when specifying the --enable-lro and --enable-rx-cksum options. However, the behaviour is the same as without these options. That is, ethtool shows LRO and rx-checksum as off (and [fixed]) and TCP traffic gets dropped unless tx-checksum is turned off. The difference is that testpmd now says that LRO and RX checksum are on (even though they don't seem to be).

Eric 

-----Original Message-----
From: Tiwei Bie <tiwei.bie@intel.com> 
Sent: September 18, 2018 1:30 AM
To: Eric Tremblay <ertr@kaloom.com>; users@dpdk.org
Cc: zhihong.wang@intel.com; maxime.coquelin@redhat.com
Subject: Re: [dpdk-users] Virtio-user exceptional path performance

On Tue, Sep 04, 2018 at 05:54:20PM +0000, Eric Tremblay wrote:
> Hello,
> 
> I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).

Somehow below fix isn't backported to 17.11 while the commit it fixed was backported.

https://git.dpdk.org/dpdk/commit/?id=bce7e9050f9b
commit bce7e9050f9b ("net/virtio-user: fix start with kernel vhost")

Please apply above fix locally and see if it works.

Thanks

> 
> Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.
> 
> Regards,
> 
> Eric
> 

Re: [dpdk-users] Virtio-user exceptional path performance

[Tiwei Bie]

On Tue, Sep 18, 2018 at 02:45:24PM +0000, Eric Tremblay wrote:
> Thanks for the reply.
> 
> With the patch, the tap interface is created in the kernel even when specifying the --enable-lro and --enable-rx-cksum options. However, the behaviour is the same as without these options. That is, ethtool shows LRO and rx-checksum as off (and [fixed]) and TCP traffic gets dropped unless tx-checksum is turned off. The difference is that testpmd now says that LRO and RX checksum are on (even though they don't seem to be).

Are you using csum fwd in testpmd? You also need to
enable the hw csum offload. You can do it via:

testpmd> csum set tcp hw 0
testpmd> csum set tcp hw 1

And make sure you get below output:

TCP checksum offload is hw

> 
> Eric 
> 
> -----Original Message-----
> From: Tiwei Bie <tiwei.bie@intel.com> 
> Sent: September 18, 2018 1:30 AM
> To: Eric Tremblay <ertr@kaloom.com>; users@dpdk.org
> Cc: zhihong.wang@intel.com; maxime.coquelin@redhat.com
> Subject: Re: [dpdk-users] Virtio-user exceptional path performance
> 
> On Tue, Sep 04, 2018 at 05:54:20PM +0000, Eric Tremblay wrote:
> > Hello,
> > 
> > I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).
> 
> Somehow below fix isn't backported to 17.11 while the commit it fixed was backported.
> 
> https://git.dpdk.org/dpdk/commit/?id=bce7e9050f9b
> commit bce7e9050f9b ("net/virtio-user: fix start with kernel vhost")
> 
> Please apply above fix locally and see if it works.
> 
> Thanks
> 
> > 
> > Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.
> > 
> > Regards,
> > 
> > Eric
> > 

Re: [dpdk-users] Virtio-user exceptional path performance

[Eric Tremblay]

It does not work even if I do the following:

testpmd> set fwd csum
Set csum packet forwarding mode
testpmd> csum set tcp hw 0
Parse tunnel is off
IP checksum offload is sw
UDP checksum offload is sw
TCP checksum offload is hw
SCTP checksum offload is sw
Outer-Ip checksum offload is sw
testpmd> csum set tcp hw 1
Parse tunnel is off
IP checksum offload is sw
UDP checksum offload is sw
TCP checksum offload is hw
SCTP checksum offload is sw
Outer-Ip checksum offload is sw
testpmd>

I also tried using a NIC but it doesn't work either. Packets sent through the virtio-user tap interface arrive on the other server with an incorrect checksum. Is the checksum offloading broken for this type of virtual device?

Eric 

-----Original Message-----
From: Tiwei Bie <tiwei.bie@intel.com> 
Sent: September 19, 2018 1:08 AM
To: Eric Tremblay <ertr@kaloom.com>
Cc: users@dpdk.org; zhihong.wang@intel.com; maxime.coquelin@redhat.com
Subject: Re: [dpdk-users] Virtio-user exceptional path performance

On Tue, Sep 18, 2018 at 02:45:24PM +0000, Eric Tremblay wrote:
> Thanks for the reply.
> 
> With the patch, the tap interface is created in the kernel even when specifying the --enable-lro and --enable-rx-cksum options. However, the behaviour is the same as without these options. That is, ethtool shows LRO and rx-checksum as off (and [fixed]) and TCP traffic gets dropped unless tx-checksum is turned off. The difference is that testpmd now says that LRO and RX checksum are on (even though they don't seem to be).

Are you using csum fwd in testpmd? You also need to enable the hw csum offload. You can do it via:

testpmd> csum set tcp hw 0
testpmd> csum set tcp hw 1

And make sure you get below output:

TCP checksum offload is hw

> 
> Eric
> 
> -----Original Message-----
> From: Tiwei Bie <tiwei.bie@intel.com>
> Sent: September 18, 2018 1:30 AM
> To: Eric Tremblay <ertr@kaloom.com>; users@dpdk.org
> Cc: zhihong.wang@intel.com; maxime.coquelin@redhat.com
> Subject: Re: [dpdk-users] Virtio-user exceptional path performance
> 
> On Tue, Sep 04, 2018 at 05:54:20PM +0000, Eric Tremblay wrote:
> > Hello,
> > 
> > I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).
> 
> Somehow below fix isn't backported to 17.11 while the commit it fixed was backported.
> 
> https://git.dpdk.org/dpdk/commit/?id=bce7e9050f9b
> commit bce7e9050f9b ("net/virtio-user: fix start with kernel vhost")
> 
> Please apply above fix locally and see if it works.
> 
> Thanks
> 
> > 
> > Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.
> > 
> > Regards,
> > 
> > Eric
> > 

Re: [dpdk-users] Virtio-user exceptional path performance

[Tiwei Bie]

On Mon, Sep 24, 2018 at 03:25:20PM +0000, Eric Tremblay wrote:
> It does not work even if I do the following:
> 
> testpmd> set fwd csum
> Set csum packet forwarding mode
> testpmd> csum set tcp hw 0
> Parse tunnel is off
> IP checksum offload is sw
> UDP checksum offload is sw
> TCP checksum offload is hw
> SCTP checksum offload is sw
> Outer-Ip checksum offload is sw
> testpmd> csum set tcp hw 1
> Parse tunnel is off
> IP checksum offload is sw
> UDP checksum offload is sw
> TCP checksum offload is hw
> SCTP checksum offload is sw
> Outer-Ip checksum offload is sw
> testpmd>
> 
> I also tried using a NIC but it doesn't work either. Packets sent through the virtio-user tap interface arrive on the other server with an incorrect checksum. Is the checksum offloading broken for this type of virtual device?

The Rx and Tx checksum offloading works in virtio-user.

Below are the steps I used on my machine:

# Launch the testpmd with two virtio-user ports
$ ./x86_64-native-linuxapp-gcc/app/testpmd \
	-l 0,1 \
	--file-prefix=vhost-kernel \
	--vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024,iface=virtio0 \
	--vdev=virtio_user1,path=/dev/vhost-net,queue_size=1024,iface=virtio1 \
	--no-pci \
	-- \
	-i \
	--disable-hw-vlan \
	--enable-lro \
	--enable-rx-cksum \
	--eth-peer=0,00:01:02:03:04:01 \
	--eth-peer=1,00:01:02:03:04:02

# Configure and start testpmd
testpmd> set fwd csum
testpmd> csum set tcp hw 0
testpmd> csum set tcp hw 1
testpmd> start

# Configure the first tap interface
$ ifconfig virtio0 hw ether 00:01:02:03:04:01
$ ifconfig virtio0 1.1.1.1 up

# Create a network namespace
$ ip netns add ns1

# Configure the second tap interface
$ ip link set virtio1 netns ns1
$ ip netns exec ns1 ifconfig virtio1 hw ether 00:01:02:03:04:02
$ ip netns exec ns1 ifconfig virtio1 1.1.1.2 up

# Start the iperf server
$ ip netns exec ns1 iperf -s -i 1

# Start the iperf client
$ iperf -c 1.1.1.2 -i 1 -t 60


> 
> Eric 
> 
> -----Original Message-----
> From: Tiwei Bie <tiwei.bie@intel.com> 
> Sent: September 19, 2018 1:08 AM
> To: Eric Tremblay <ertr@kaloom.com>
> Cc: users@dpdk.org; zhihong.wang@intel.com; maxime.coquelin@redhat.com
> Subject: Re: [dpdk-users] Virtio-user exceptional path performance
> 
> On Tue, Sep 18, 2018 at 02:45:24PM +0000, Eric Tremblay wrote:
> > Thanks for the reply.
> > 
> > With the patch, the tap interface is created in the kernel even when specifying the --enable-lro and --enable-rx-cksum options. However, the behaviour is the same as without these options. That is, ethtool shows LRO and rx-checksum as off (and [fixed]) and TCP traffic gets dropped unless tx-checksum is turned off. The difference is that testpmd now says that LRO and RX checksum are on (even though they don't seem to be).
> 
> Are you using csum fwd in testpmd? You also need to enable the hw csum offload. You can do it via:
> 
> testpmd> csum set tcp hw 0
> testpmd> csum set tcp hw 1
> 
> And make sure you get below output:
> 
> TCP checksum offload is hw
> 
> > 
> > Eric
> > 
> > -----Original Message-----
> > From: Tiwei Bie <tiwei.bie@intel.com>
> > Sent: September 18, 2018 1:30 AM
> > To: Eric Tremblay <ertr@kaloom.com>; users@dpdk.org
> > Cc: zhihong.wang@intel.com; maxime.coquelin@redhat.com
> > Subject: Re: [dpdk-users] Virtio-user exceptional path performance
> > 
> > On Tue, Sep 04, 2018 at 05:54:20PM +0000, Eric Tremblay wrote:
> > > Hello,
> > > 
> > > I am testing the performance capabilities of the virtio-user exceptional path for use in container networking scenarios. I am trying to reproduce the results obtained in Test Case 1 described in the paper "VIRTIO-USER: A New Versatile Channel for Kernel-Bypass Networks".  That is, I am creating two virtio-user vdevs in testpmd, putting each one in separate network namespaces and running iperf3 between the two. However, I am unable to obtain the near 25Gbps throughput shown in the paper. Instead, I obtained a mere 3Gbps. Has anyone else been able to reproduce these results? My understanding is that this type of virtual device takes advantage of both checksum offloading and LRO in order to get such high throughput. However, for some reason, I am unable to use RX checksum offloading nor LRO in my setup. If I specify the --enable-lro and --enable-rx-cksum options in testpmd (as shown here https://doc.dpdk.org/guides-17.11/howto/virtio_user_as_exceptional_path.html), the tap device is never created in the kernel. If I do not specify these options, the tap is created correctly but the rx-checksum and LRO are off and cannot be turned on using ethtool (they are marked as [fixed]).
> > 
> > Somehow below fix isn't backported to 17.11 while the commit it fixed was backported.
> > 
> > https://git.dpdk.org/dpdk/commit/?id=bce7e9050f9b
> > commit bce7e9050f9b ("net/virtio-user: fix start with kernel vhost")
> > 
> > Please apply above fix locally and see if it works.
> > 
> > Thanks
> > 
> > > 
> > > Does anyone have any idea what may be preventing me from taking advantage of checksum offloading and LRO? Also, I am I right in assuming that this is the reason why the performance is so poor or could there be another problem with my setup? I am using DPDK 17.11 on CentOS 7.5. Any help would be appreciated.
> > > 
> > > Regards,
> > > 
> > > Eric
> > >