From: "Wiles, Keith" <keith.wiles@intel.com>
To: Xia Rui <xiarui_work@163.com>
Cc: dpdk dev community <dev@dpdk.org>
Subject: Re: [dpdk-dev] pktgen-dpdk send too big packet, and stop sending packets after few seconds
Date: Sun, 23 Jun 2019 13:26:15 +0000 [thread overview]
Message-ID: <56DE7D0F-2523-4BBF-B05C-36D26D8FAD40@intel.com> (raw)
In-Reply-To: <2d935d71.4455.16b8359bea1.Coremail.xiarui_work@163.com>
> On Jun 23, 2019, at 3:00 AM, Xia Rui <xiarui_work@163.com> wrote:
>
> Hello, everyone.
> I am using pktgen-dpdk and testpmd to test the functionality of ovs-dpdk. The network topology is :
> +-------------+----------------------+ host(OVS-DPDK) +-----------------------+-----------------+
> | | vhost-user port 1 |<----------------------------------->| vhost-user port 3 | |
> | +----------------------+ +-----------------------+ |
> | container | pktgen | | testpmd | container |
> | +----------------------+ +-------------------+ |
> | | vhost-user port 2 |<------------------------------------>| vhost-user port 4 | |
> +--------------+---------------------+ +----------------------+----------------+
>
> The version of my platform:
> 1. host OS: ubuntu 16.04.5 LTS
> 2. host linux kernel: 4.15.0-15
> 3. host OVS: 2.8.0
> 4. host DPDK : 17.05.2
> 5. container pktgen-dpdk: 3.4.9 + DPDK 17.05.2
> 6. container DPDK (testpmd): 17.05.2
At one point virtio drivers were changing the length of the tx packet in the mbuf for some type of meta data being shared between the two virtio points. Pktgen expected the length to remain the same when the mbuf was returned to pktgen mempool. This explains the non-64 byte frames. The fix was added to virtio in later releases to restore the length in the mbuf, plus I may’ve added code to pktgen in later release to fix the length back to what I expected when sending the packets. I looked in the latests version of pktgen and found the code in pktgen_setup_cb() which reset the length in the packet. The code should call rte_pktmbuf_reset() routine to restore the mbuf to expected state.
If OVS can work with a later version of DPDK then I would upgrade to the latest release, if not then I would look at and compare the two versions of virtio PMD and see if you can find that fix. If not then you can look in pktgen and repair the length in the mbuf before it is sent plus you may have to fix the location of the read/write offsets in the mbuf as well. This change will effect pktgen performance, but for virtio that should not be a problem. You can also connect the Maintainers of virtio and see if they remember the fix.
For the traffic stopping, I do not remember if this fix solved that problem.
>
> There are two docker containers. One is running pktgen-dpdk with:
> ############################pktgen-dpdk start script############################
> ./app/x86_64-native-linuxapp-gcc/pktgen -c 0x70 --master-lcore 4 -n 1 --file-prefix pktgen --no-pci \
> --vdev 'net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1' \
> --vdev 'net_virtio_user2,mac=00:00:00:00:00:02,path=/var/run/openvswitch/vhost-user2' \
> -- -T -P -m "5.0,6.1"
> ############################pktgen-dpdk start script END############################
>
> The other is running testpmd with:
> ############################testpmd start script############################
> testpmd -c 0xE0 -n 1 --socket-mem=1024,0 --file-prefix testpmd --no-pci \
> --vdev 'net_virtio_user3,mac=00:00:00:00:00:03,path=/var/run/openvswitch/vhost-user3' \
> --vdev 'net_virtio_user4,mac=00:00:00:00:00:04,path=/var/run/openvswitch/vhost-user4' \
> -- -i --burst=64 --disable-hw-vlan --txd=2048 --rxd=2048 --auto-start --coremask=0xc0
> ############################testpmd start script END############################
>
> The two containers are connected using ovs-docker in the host. I create four vhost-user ports, two of which are for pktgen, the other of which are for testpmd. I connect the vhost-user ports by adding the
> routes between them. The start script of ovs is:
>
> ############################ovs-dpdk start script############################
> sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
> --private-key=db:Open_vSwitch,SSL,private_key \
> --certificate=db:Open_vSwitch,SSL,certificate \
> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
> --pidfile --detach
>
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
>
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x02
>
> sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x04
>
> sudo ovs-vswitchd --pidfile --detach --log-file=/var/log/openvswitch/vhost-ovs-vswitchd.log
>
> sudo /usr/local/share/openvswitch/scripts/ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
> ############################ovs-dpdk start script END############################
>
> When I start the pktgen-dpdk to send packets, there are something wrong.
>
> First, I set the packet size to 64Bytes, there are some big packets (more than 64Bytes). I set the rate to 10%, 300 packets, and get:
>
> ###########################CMD shot#######################
> Ports 0-1 of 2 <Main Page> Copyright (c) <2010-2017>, Intel Corporation
> Flags:Port : P--------------:0 P--------------:1
> Link State : <UP-10000-FD> <UP-10000-FD> ----TotalRate----
> Pkts/s Max/Rx : 256/0 300/0 556/0
> Max/Tx : 300/0 300/0 600/0
> MBits/s Rx/Tx : 0/0 0/0 0/0
> Broadcast : 0 0
> Multicast : 0 0
> 64 Bytes : 1104 1148
> 65-127 : 64 152
> 128-255 : 0 0
> 256-511 : 0 0
> 512-1023 : 0 0
> 1024-1518 : 0 0
> Runts/Jumbos : 0/0 0/0
> Errors Rx/Tx : 0/0 0/0
> Total Rx Pkts : 1168 1300
> Tx Pkts : 1300 1300
> Rx MBs : 0 0
> Tx MBs : 1 1
> ARP/ICMP Pkts : 0/0 0/0
> :
> Pattern Type : abcd... abcd...
> Tx Count/% Rate : 300 /10% 300 /10%
> PktSize/Tx Burst : 64 / 64 64 / 64
> Src/Dest Port : 1234 / 5678 1234 / 5678
> Pkt Type:VLAN ID : IPv4 / TCP:0001 IPv4 / TCP:0001
> 802.1p CoS : 0 0
> ToS Value: : 0 0
> - DSCP value : 0 0
> - IPP value : 0 0
> Dst IP Address : 192.168.1.1 192.168.0.1
> Src IP Address : 192.168.0.1/24 192.168.1.1/24
> Dst MAC Address : 00:00:00:00:00:02 00:00:00:00:00:01
> Src MAC Address : 00:00:00:00:00:01 00:00:00:00:00:02
> VendID/PCI Addr : 0000:0000/00:00.0 0000:0000/00:00.0
>
> -- Pktgen Ver: 3.4.9 (DPDK 17.05.2) Powered by DPDK --------------------------
> ###########################CMD shot END#######################
>
> There ought to be no packets greater than 64Bytes, but there exist.
>
> Second, I reset the configuration ("rst") and try to start send packets continuously. However, the pktgen works few seconds and stop sending packets, with output:
>
>
> ###########################CMD shot#######################
> Ports 0-1 of 2 <Main Page> Copyright (c) <2010-2017>, Intel Corporation
> Flags:Port : P--------------:0 P--------------:1
> Link State : <UP-10000-FD> <UP-10000-FD> ----TotalRate----
> Pkts/s Max/Rx : 176288/0 146016/0 308224/0
> Max/Tx : 1344832/0 767520/0 1535040/0
> MBits/s Rx/Tx : 0/0 0/0 0/0
> Broadcast : 0 0
> Multicast : 0 0
> 64 Bytes : 15872 15104
> 65-127 : 50368 61248
> 128-255 : 44096 62848
> 256-511 : 51840 93216
> 512-1023 : 63264 151456
> 1024-1518 : 51936 126240
> Runts/Jumbos : 0/0 0/0
> Errors Rx/Tx : 0/0 0/0
> Total Rx Pkts : 277376 510112
> Tx Pkts : 4529248 1162368
> Rx MBs : 1215 2701
> Tx MBs : 665276 57380
> ARP/ICMP Pkts : 0/0 0/0
> :
> Pattern Type : abcd... abcd...
> Tx Count/% Rate : Forever /100% Forever /100%
> PktSize/Tx Burst : 64 / 64 64 / 64
> Src/Dest Port : 1234 / 5678 1234 / 5678
> Pkt Type:VLAN ID : IPv4 / TCP:0001 IPv4 / TCP:0001
> 802.1p CoS : 0 0
> ToS Value: : 0 0
> - DSCP value : 0 0
> - IPP value : 0 0
> Dst IP Address : 192.168.1.1 192.168.0.1
> Src IP Address : 192.168.0.1/24 192.168.1.1/24
> Dst MAC Address : 00:00:00:00:00:02 00:00:00:00:00:01
> Src MAC Address : 00:00:00:00:00:01 00:00:00:00:00:02
> VendID/PCI Addr : 0000:0000/00:00.0 0000:0000/00:00.0
>
> -- Pktgen Ver: 3.4.9 (DPDK 17.05.2) Powered by DPDK --------------------------
> ###########################CMD shot END#######################
>
> Pktgen is stuck at this setting. There are a lot of large packets!
>
> I check the log of ovs-dpdk and get:
>
> #############################ovs-dpdk log#############################
> 2019-06-23T07:51:39.349Z|00022|netdev_dpdk(pmd8)|WARN|Dropped 5803564 log messages in last 102581 seconds (most recently, 102576 seconds ago) due to excessive rate
> 2019-06-23T07:51:39.349Z|00023|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00024|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00025|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00026|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00027|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00028|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00029|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00030|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00031|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00032|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00033|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00034|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00035|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00036|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00037|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00038|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00039|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00040|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00041|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> 2019-06-23T07:51:39.349Z|00042|netdev_dpdk(pmd8)|WARN|vhost-user3: Too big size 1524 max_packet_len 1518
> #############################ovs-dpdk log END#############################
> Thank you for share your ideas.
>
>
> Best wishes,
> Xia Rui
Regards,
Keith
next prev parent reply other threads:[~2019-06-23 13:26 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-06-23 8:00 Xia Rui
2019-06-23 13:26 ` Wiles, Keith [this message]
2019-06-24 9:22 ` Xia Rui
2019-06-24 13:17 ` Wiles, Keith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56DE7D0F-2523-4BBF-B05C-36D26D8FAD40@intel.com \
--to=keith.wiles@intel.com \
--cc=dev@dpdk.org \
--cc=xiarui_work@163.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).