From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f176.google.com (mail-ig0-f176.google.com [209.85.213.176]) by dpdk.org (Postfix) with ESMTP id BDE9868AC for ; Mon, 1 Dec 2014 20:22:18 +0100 (CET) Received: by mail-ig0-f176.google.com with SMTP id l13so14569068iga.3 for ; Mon, 01 Dec 2014 11:22:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=z5AOf0RlS1IYQofGu1WRN0/L17mfXR7KWTwTZUHy8ko=; b=ZGpEFa7MibOEoF6KM7eheK6JRkteHSi+mHtu/wKFz7Vs+xEf28XPBx3tMgTbbssrh1 SzRR9jdcVP+RVuSpKFv1hYaC9YaB0Boz3VKYjQOUFn7Zi5KmkV7O7We2eJ6MneKRpqkp uLx+Bh0xXp+ovY9ZLkkZX18ELbaLENUVbA87b5OKmvhULELHkGlx47hT8zZjH0gMhxJe SD6OGPewMP52IG3MbpSKRGFr2tFJF0SNBvhBuf07EpGRci6JPlqUTEOe/Kk0ujcgA+ds 4c0zS1E1mVbdaK59VWPEm4JJTqlcYciLFGjmAfP3wIUIFZ7KitnBQr9lQFeVkhFOed58 BCAw== MIME-Version: 1.0 X-Received: by 10.42.64.143 with SMTP id g15mr46633866ici.59.1417461738054; Mon, 01 Dec 2014 11:22:18 -0800 (PST) Received: by 10.64.17.65 with HTTP; Mon, 1 Dec 2014 11:22:18 -0800 (PST) Date: Mon, 1 Dec 2014 11:22:18 -0800 Message-ID: From: Suresh Vinapamula To: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Testpmd with SR-IOV. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Dec 2014 19:22:19 -0000 Hi I am trying to run latest DPDK on SR-IOV interface in VM environment. Interfaces are detected well in VM environment. When I started traffic through testpmd, it indicates both tx and rx are zero. I tried debugging. Wondering if you could point me some steps which I might be missing? Thanks in advance. Here are the steps that I followed. SR-IOV interfaces are detected in VM environment. Lspci indicate below interfaces in VM environment and ethtool indicates links are up and link speeds are fine. 00:08.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 00:09.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) root@regress-KVM:/dev/abc/dpdk/tools# ethtool eth2 Settings for eth2: Supported ports: [ ] Supported link modes: 10000baseT/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: Other PHYAD: 0 Transceiver: Unknown! Auto-negotiation: off Current message level: 0x00000007 (7) drv probe link Link detected: yes root@regress-KVM:/dev/abc/dpdk/tools# ethtool eth3 Settings for eth3: Supported ports: [ ] Supported link modes: 10000baseT/Full Supported pause frame use: No Supports auto-negotiation: No Advertised link modes: Not reported Advertised pause frame use: No Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: Other PHYAD: 0 Transceiver: Unknown! Auto-negotiation: off Current message level: 0x00000007 (7) drv probe link Link detected: yes Loaded uio module. root@regress-KVM:/dev/abc/dpdk/tools# modprobe uio root@regress-KVM:/dev/abc/dpdk/tools# insmod ../x86_64-native-linuxapp-gcc/kmod/igb_uio.ko Bound interfaces to igb_uio. root@regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --bind=igb_uio 00:09.0 root@regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --bind=igb_uio 00:08.0 Verified that interfaces are bound to igb_uio. root@regress-KVM:/dev/abc/dpdk/tools# python dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:00:08.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=vfio-pci 0000:00:09.0 '82599 Ethernet Controller Virtual Function' drv=igb_uio unused=vfio-pci Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio,vfio-pci 0000:00:07.0 '82599 Ethernet Controller Virtual Function' if=eth1 drv=ixgbevf unused=igb_uio,vfio-pci Other network devices ===================== root@regress-KVM:/dev/abc/dpdk/tools# Built test-pmd and executed and links detected. root@regress-KVM:/dev/abc/dpdk/app/test-pmd# ./testpmd -c 0xf -n 4 -- -i --nb-cores=2 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Detected lcore 2 as core 0 on socket 0 EAL: Detected lcore 3 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 4 lcore(s) EAL: unsupported IOMMU type! EAL: VFIO support could not be initialized EAL: Setting up memory... EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd8e1200000 (size = 0x200000) EAL: Ask a virtual area of 0x7f000000 bytes EAL: Virtual area found at 0x7fd862000000 (size = 0x7f000000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd861c00000 (size = 0x200000) EAL: Ask a virtual area of 0x400000 bytes EAL: Virtual area found at 0x7fd861600000 (size = 0x400000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd861200000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860e00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860a00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fd860600000 (size = 0x200000) EAL: Requesting 1024 pages of size 2MB from socket 0 EAL: TSC frequency is ~1999999 KHz EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: Master core 0 is ready (tid=e281a840) EAL: Core 3 is ready (tid=5e9f0700) EAL: Core 2 is ready (tid=5f1f1700) EAL: Core 1 is ready (tid=5f9f2700) EAL: PCI device 0000:00:03.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: 0000:00:03.0 not managed by UIO driver, skipping EAL: PCI device 0000:00:07.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: 0000:00:07.0 not managed by UIO driver, skipping EAL: PCI device 0000:00:08.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: PCI memory mapped at 0x7fd8e2828000 EAL: PCI memory mapped at 0x7fd8e2824000 EAL: PCI device 0000:00:09.0 on NUMA socket -1 EAL: probe driver: 8086:10ed rte_ixgbevf_pmd EAL: PCI memory mapped at 0x7fd8e2820000 EAL: PCI memory mapped at 0x7fd8e27e0000 Interactive-mode selected Configuring Port 0 (socket 0) Port 0: 52:54:00:E2:90:42 Configuring Port 1 (socket 0) Port 1: 52:54:00:CA:0D:17 Checking link statuses... Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed 10000 Mbps - full-duplex Done testpmd> When I start and stop traffic, stats show that zero traffic xmitted. testpmd> start io packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=2 - nb forwarding ports=2 RX queues=1 - RX desc=128 - RX free threshold=0 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=1 - TX desc=512 - TX free threshold=0 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=0 - TXQ flags=0x0 testpmd> stop Telling cores to stop... Waiting for lcores to finish... ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 0 TX-dropped: 0 TX-total: 0 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Done. testpmd> I debugged this to some extent and below are my observations. For some reason I am not able to compile old dpdk code from the website. Can you point me to your code? When I start traffic in testpmd, I see two thread taking 100%cpu utilization. Ideally I would expect one thread to be rx and another to be tx. One thread does the tx and while other thread does rx, is my understanding correct? But what I see is both the treads are in rx path!? Is that expected? Am I missing something? I believe you must have automated these steps in scripts for vmx. Can you please point me to them? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8496 root 20 0 2291m 5844 5532 R 99.9 0.0 2:27.13 testpmd 8495 root 20 0 2291m 5844 5532 R 99.8 0.0 2:27.12 testpmd 8896 root 20 0 24972 1712 1124 R 0.3 0.0 0:05.60 top 1 root 20 0 27352 3200 1416 S 0.0 0.0 0:01.42 init (gdb) info threads Id Target Id Frame 5 Thread 0x7f7d029f3700 (LWP 8494) "testpmd" 0x00007f7d84234083 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 * 4 Thread 0x7f7d021f2700 (LWP 8495) "testpmd" rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 3 Thread 0x7f7d019f1700 (LWP 8496) "testpmd" rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2 Thread 0x7f7d011f0700 (LWP 8497) "testpmd" 0x00007f7d8450fc9d in read () at ../sysdeps/unix/syscall-template.S:81 1 Thread 0x7f7d85037840 (LWP 8493) "testpmd" 0x00007f7d8450fc9d in read () at ../sysdeps/unix/syscall-template.S:81 (gdb) thread 4 [Switching to thread 4 (Thread 0x7f7d021f2700 (LWP 8495))] #0 rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2304} (gdb) bt #0 rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f7d021f0d10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 #1 0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6200) at iofwd.c:97 #2 0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7dc0, pkt_fwd=0x43841f ) at testpmd.c:952 #3 0x00000000004275df in start_pkt_forward_on_core (fwd_arg=0x7f7d052f7dc0) at testpmd.c:959 #4 0x0000000000498759 in eal_thread_loop (arg=) at /dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224 #5 0x00007f7d84508f6e in start_thread (arg=0x7f7d021f2700) at pthread_create.c:311 #6 0x00007f7d842339cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) thread 3 [Switching to thread 3 (Thread 0x7f7d019f1700 (LWP 8496))] #0 rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 2304} (gdb) bt #0 rte_eth_rx_burst (port_id=2 '\002', queue_id=0, rx_pkts=0x7f7d019efd10, nb_pkts=32) at /dev/abc/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:2304 #1 0x0000000000438464 in pkt_burst_io_forward (fs=0x7f7d052f6100) at iofwd.c:97 #2 0x0000000000427596 in run_pkt_fwd_on_lcore (fc=0x7f7d052f7d40, pkt_fwd=0x43841f ) at testpmd.c:952 #3 0x00000000004275df in start_pkt_forward_on_core (fwd_arg=0x7f7d052f7d40) at testpmd.c:959 #4 0x0000000000498759 in eal_thread_loop (arg=) at /dev/abc/dpdk/lib/librte_eal/linuxapp/eal/eal_thread.c:224 #5 0x00007f7d84508f6e in start_thread (arg=0x7f7d019f1700) at pthread_create.c:311 #6 0x00007f7d842339cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb)