[dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

[dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

[Abhishek Mahajan]

Hi,

With pktgen-dpdk inside VM, it is observed that packet TX rate (526Kpps with 64byte size) is very low. How can higher packet rates be achieved ?

On enabling logs of virtio pmd in DPDK, getting following logs in syslogs:
PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit

Two PC setup having configuration as follow:
-> Xeon Processor : Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz  having 6 cores.
-> RAM : 16GB
->One PC is generating traffic other is receiving the packets using pktgen tool. The two PC are connected directly with 10G Ethernet interface.

Generation side PC - Host
-> 2 cores dedicated to OVS-DPDK
-> 1 core is free for Linux kernel
-> OS: Ubuntu 14.04
-> Linux kernel : 3.13.0-85-generic
-> DPDK version : 2.2.0
-> OVS version : 2.5.0
-> VFIO driver for port binding

VM - Total 3 cores
-> 2 cores for pktgen
-> 1 core for Linux kernel
-> OS: Ubuntu 15.04
-> Linux kernel : 3.19.8
-> DPDK version : 16.04
-> Pktgen version : 3.0.00
-> UIO driver for port binding

DPDK, OVS and pktgen are build using - x86_64-ivshmen-linuxapp-gcc option.

VM launched through following Qemu command :

*         qemu-system-x86_64 --enable-kvm -smp 3 -cpu host -m 2048 -chardev socket,id=charnet0,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet0,chardev=charnet0,vhostforce=on -device virtio-net-pci,netdev=hostnet0,mac=16:fa:13:26:31:1b -object memory-backend-file,id=mem,size=2048M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -hda /home/aricent/ubuntu_15_04.img

Pktgen is launched using following command:
                Pktgen -c -0x3 -n 4 -m 512 -b 000:00:04.0 - p 0x3 -t -m "[1:1].0"

OVS command used:

*         /usr/sbin/ovs-vswitchd --dpdk -c 0xc -n 4 -m 2048 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach -monitor

Thanks & Regards
Abhishek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Re: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

[Wiles, Keith]

-----Original Message-----
From: Abhishek Mahajan <abhishek.mahajan@aricent.com>
Date: Tuesday, July 5, 2016 at 1:49 AM
To: Keith Wiles <keith.wiles@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "Addepalli, Srinivasa R" <srinivasa.r.addepalli@intel.com>, "Shivastava, RakeshX" <rakeshx.shivastava@intel.com>
Subject: RE: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

Hi Keith,

I tried your core mask suggestion, still I am getting same performance values.

I just enabled logs for virtio tx in DPDK to find the problem, I am getting "PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit".
This seems to be virtio issue, is there any tuning parameters for increasing the virtio performance for using with pktgen-dpdk ?

KW: Pktgen has its own mbufs for sending this message is from the virtio driver and you need to look at the configuration options for that driver. Normally the TX mbufs are used directly from the application (Pktgen), but in this case it must have its own set of mbufs.

Regards
Abhishek

-----Original Message-----
From: Wiles, Keith [mailto:keith.wiles@intel.com]
Sent: Monday, July 04, 2016 8:03 PM
To: Abhishek Mahajan <abhishek.mahajan@aricent.com>; dev@dpdk.org
Cc: Addepalli, Srinivasa R <srinivasa.r.addepalli@intel.com>; RakeshX Contact <rakeshx.shivastava@intel.com>
Subject: Re: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

-----Original Message-----
From: dev <dev-bounces@dpdk.org> on behalf of Abhishek Mahajan <abhishek.mahajan@aricent.com>
Date: Monday, July 4, 2016 at 7:27 AM
To: "dev@dpdk.org" <dev@dpdk.org>
Cc: "Addepalli, Srinivasa R" <srinivasa.r.addepalli@intel.com>, "Shivastava, RakeshX" <rakeshx.shivastava@intel.com>
Subject: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

Hi,

With pktgen-dpdk inside VM, it is observed that packet TX rate (526Kpps with 64byte size) is very low. How can higher packet rates be achieved ?

On enabling logs of virtio pmd in DPDK, getting following logs in syslogs:
PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit

Two PC setup having configuration as follow:
-> Xeon Processor : Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz  having 6 cores.
-> RAM : 16GB
->One PC is generating traffic other is receiving the packets using pktgen tool. The two PC are connected directly with 10G Ethernet interface.

Generation side PC - Host
-> 2 cores dedicated to OVS-DPDK
-> 1 core is free for Linux kernel
-> OS: Ubuntu 14.04
-> Linux kernel : 3.13.0-85-generic
-> DPDK version : 2.2.0
-> OVS version : 2.5.0
-> VFIO driver for port binding

VM - Total 3 cores
-> 2 cores for pktgen
-> 1 core for Linux kernel
-> OS: Ubuntu 15.04
-> Linux kernel : 3.19.8
-> DPDK version : 16.04
-> Pktgen version : 3.0.00
-> UIO driver for port binding

DPDK, OVS and pktgen are build using - x86_64-ivshmen-linuxapp-gcc option.

VM launched through following Qemu command :

*         qemu-system-x86_64 --enable-kvm -smp 3 -cpu host -m 2048 -chardev socket,id=charnet0,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet0,chardev=charnet0,vhostforce=on -device virtio-net-pci,netdev=hostnet0,mac=16:fa:13:26:31:1b -object memory-backend-file,id=mem,size=2048M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -hda /home/aricent/ubuntu_15_04.img

Pktgen is launched using following command:
                Pktgen -c -0x3 -n 4 -m 512 -b 000:00:04.0 - p 0x3 -t -m "[1:1].0"

KW: I do not think this is Pktgen problem, but please try pktgen –c 0x6 –n 4 –m 512 –b 0000:00:04.0 -- -T –m”[2:2].0”
Core 0 needs to be left alone for Linux and core 1 is used for Pktgen display and core 2 is used for Rx/Tx on port 0.

OVS command used:

*         /usr/sbin/ovs-vswitchd --dpdk -c 0xc -n 4 -m 2048 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach -monitor

Thanks & Regards
Abhishek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."





"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Re: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

[Abhishek Mahajan]

Hi Keith,

I tried your core mask suggestion, still I am getting same performance values.

I just enabled logs for virtio tx in DPDK to find the problem, I am getting "PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit".
This seems to be virtio issue, is there any tuning parameters for increasing the virtio performance for using with pktgen-dpdk ?

Regards
Abhishek

-----Original Message-----
From: Wiles, Keith [mailto:keith.wiles@intel.com]
Sent: Monday, July 04, 2016 8:03 PM
To: Abhishek Mahajan <abhishek.mahajan@aricent.com>; dev@dpdk.org
Cc: Addepalli, Srinivasa R <srinivasa.r.addepalli@intel.com>; RakeshX Contact <rakeshx.shivastava@intel.com>
Subject: Re: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

-----Original Message-----
From: dev <dev-bounces@dpdk.org> on behalf of Abhishek Mahajan <abhishek.mahajan@aricent.com>
Date: Monday, July 4, 2016 at 7:27 AM
To: "dev@dpdk.org" <dev@dpdk.org>
Cc: "Addepalli, Srinivasa R" <srinivasa.r.addepalli@intel.com>, "Shivastava, RakeshX" <rakeshx.shivastava@intel.com>
Subject: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

Hi,

With pktgen-dpdk inside VM, it is observed that packet TX rate (526Kpps with 64byte size) is very low. How can higher packet rates be achieved ?

On enabling logs of virtio pmd in DPDK, getting following logs in syslogs:
PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit

Two PC setup having configuration as follow:
-> Xeon Processor : Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz  having 6 cores.
-> RAM : 16GB
->One PC is generating traffic other is receiving the packets using pktgen tool. The two PC are connected directly with 10G Ethernet interface.

Generation side PC - Host
-> 2 cores dedicated to OVS-DPDK
-> 1 core is free for Linux kernel
-> OS: Ubuntu 14.04
-> Linux kernel : 3.13.0-85-generic
-> DPDK version : 2.2.0
-> OVS version : 2.5.0
-> VFIO driver for port binding

VM - Total 3 cores
-> 2 cores for pktgen
-> 1 core for Linux kernel
-> OS: Ubuntu 15.04
-> Linux kernel : 3.19.8
-> DPDK version : 16.04
-> Pktgen version : 3.0.00
-> UIO driver for port binding

DPDK, OVS and pktgen are build using - x86_64-ivshmen-linuxapp-gcc option.

VM launched through following Qemu command :

*         qemu-system-x86_64 --enable-kvm -smp 3 -cpu host -m 2048 -chardev socket,id=charnet0,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet0,chardev=charnet0,vhostforce=on -device virtio-net-pci,netdev=hostnet0,mac=16:fa:13:26:31:1b -object memory-backend-file,id=mem,size=2048M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -hda /home/aricent/ubuntu_15_04.img

Pktgen is launched using following command:
                Pktgen -c -0x3 -n 4 -m 512 -b 000:00:04.0 - p 0x3 -t -m "[1:1].0"

KW: I do not think this is Pktgen problem, but please try pktgen –c 0x6 –n 4 –m 512 –b 0000:00:04.0 -- -T –m”[2:2].0”
Core 0 needs to be left alone for Linux and core 1 is used for Pktgen display and core 2 is used for Rx/Tx on port 0.

OVS command used:

*         /usr/sbin/ovs-vswitchd --dpdk -c 0xc -n 4 -m 2048 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach -monitor

Thanks & Regards
Abhishek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."





"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Re: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

[Wiles, Keith]

-----Original Message-----
From: dev <dev-bounces@dpdk.org> on behalf of Abhishek Mahajan <abhishek.mahajan@aricent.com>
Date: Monday, July 4, 2016 at 7:27 AM
To: "dev@dpdk.org" <dev@dpdk.org>
Cc: "Addepalli, Srinivasa R" <srinivasa.r.addepalli@intel.com>, "Shivastava, RakeshX" <rakeshx.shivastava@intel.com>
Subject: [dpdk-dev] Low packet generation rate of 526kpps using pktgen-dpdk from inside VM

Hi,

With pktgen-dpdk inside VM, it is observed that packet TX rate (526Kpps with 64byte size) is very low. How can higher packet rates be achieved ?

On enabling logs of virtio pmd in DPDK, getting following logs in syslogs:
PMD: virtio_xmit_pkts() tx: No free tx descriptors to transmit

Two PC setup having configuration as follow:
-> Xeon Processor : Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz  having 6 cores.
-> RAM : 16GB
->One PC is generating traffic other is receiving the packets using pktgen tool. The two PC are connected directly with 10G Ethernet interface.

Generation side PC - Host
-> 2 cores dedicated to OVS-DPDK
-> 1 core is free for Linux kernel
-> OS: Ubuntu 14.04
-> Linux kernel : 3.13.0-85-generic
-> DPDK version : 2.2.0
-> OVS version : 2.5.0
-> VFIO driver for port binding

VM - Total 3 cores
-> 2 cores for pktgen
-> 1 core for Linux kernel
-> OS: Ubuntu 15.04
-> Linux kernel : 3.19.8
-> DPDK version : 16.04
-> Pktgen version : 3.0.00
-> UIO driver for port binding

DPDK, OVS and pktgen are build using - x86_64-ivshmen-linuxapp-gcc option.

VM launched through following Qemu command :

*         qemu-system-x86_64 --enable-kvm -smp 3 -cpu host -m 2048 -chardev socket,id=charnet0,path=/var/run/openvswitch/vhost-user1 -netdev type=vhost-user,id=hostnet0,chardev=charnet0,vhostforce=on -device virtio-net-pci,netdev=hostnet0,mac=16:fa:13:26:31:1b -object memory-backend-file,id=mem,size=2048M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -hda /home/aricent/ubuntu_15_04.img

Pktgen is launched using following command:
                Pktgen -c -0x3 -n 4 -m 512 -b 000:00:04.0 - p 0x3 -t -m "[1:1].0"

KW: I do not think this is Pktgen problem, but please try pktgen –c 0x6 –n 4 –m 512 –b 0000:00:04.0 -- -T –m”[2:2].0”
Core 0 needs to be left alone for Linux and core 1 is used for Pktgen display and core 2 is used for Rx/Tx on port 0.

OVS command used:

*         /usr/sbin/ovs-vswitchd --dpdk -c 0xc -n 4 -m 2048 -- unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach -monitor

Thanks & Regards
Abhishek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."