Re: [dpdk-dev] [PATCH v6 0/8] Interrupt mode PMD

["Liu, Yong" <yong.liu@intel.com>]

Tested-by: Yong Liu <yong.liu@intel.com>

- Tested Commit: 00c685634b8a43e4594e26949a6c4f1cf5b67047
- OS: Fedora20 3.15.8-200.fc20.x86_64
- GCC: gcc version 4.8.3 20140911 (Red Hat 4.8.3-7) (GCC)
- CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
- NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
- Default x86_64-native-linuxapp-gcc configuration
- Total 4 cases, 4 passed, 0 failed

- Case: interrupt pmd on PF with single queue
  Description: Check interrupt pmd work with single queue
  Command / instruction:
    Bind ports to vfio-pci.
      modprobe vfio
      modprobe vfio-pci
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 0000:08:00.0 0000:08:00.1
    Start l3fwd-power with one queue per port.
      l3fwd-power -c 7 -n 4 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Send one packet to Port0 and Port1, check that thread on core1 and core2 
    waked up.
      L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0
      L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq0
  Expected test result:
    l3fwd-power can forward packets normally and thread on core1 and core2 
    will sleep when there's no packet received.

- Case: interrupt pmd on PF with multi queue
  Description: Check interrupt pmd work with multiple queues
  Command / instruction:
    Start l3fwd-power with two queues per port.
      l3fwd-power -c 1f -n 4 -- -p 0x3 \
      --config="(0,0,1),(0,1,2)(1,0,3),(1,1,4)"
    Send packet with increased dest IP to Port0 and Port1, check that thread 
    on core1,core2,core3,core4 waked up.
      L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0
      L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq1
      L3FWD_POWER: lcore 3 is waked up from rx interrupt on port1,rxq0
      L3FWD_POWER: lcore 4 is waked up from rx interrupt on port1,rxq1
  Expected test result:
    l3fwd-power can forward packets normally and thread on core1-core4 will 
	sleep when there's no packet received.

- Case: interrupt pmd on PF with max Rx queues
  Description: Check interrupt pmd work with maximum queues
  Command / instruction:
    Start l3fwd-power with 32 queues per port.
    l3fwd-power -c ffffffff -n 4 -- -p 0x3 -P --config="(0,0,0),(0,1,1),\
      (0,2,2),(0,3,3),(0,4,4),(0,5,5),(0,6,6),(0,7,7),(0,8,8),
      (0,9,9),(0,10,10),(0,11,11),(0,12,12),(0,13,13),(0,14,14),\
      (0,15,15),\
      (1,0,16),(1,1,17),(1,2,18),(1,3,19),(1,4,20),(1,5,21),(1,6,22),\
      (1,7,23),(1,8,24),(1,9,25),(1,10,26),(1,11,27),(1,12,28),\
      (1,13,29),(1,14,30),\(1,15,31)"
    Send packet with increased dest IP to Port0 and Port1, check that all 
    threads waked up.
  Expected test result:
    l3fwd-power can forward packets normally and thread on core1-core31
    will sleep when there's no packet received.
		
- Case: interrupt pmd on VF with single queue
  Description: Check interrupt pmd work on VF device
  Command / instruction:
    Bind ports to back to ixgbe driver.
      ./tools/dpdk_nic_bind.py --bind=ixgbe 0000:08:00.0 0000:08:00.1
    Create one VF per Port in host and make sure PF interface up
	  echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
      echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs
      ifconfig p786p1 up
      ifconfig p786p2 up	  
    Bind VF device to vfio-pci.
      ./tools/dpdk_nic_bind.py --bind=vfio-pci 0000:08:10.0 0000:08:10.1
    Start l3fwd-power on host with one queue per port.
	  l3fwd-power -c 1f -n 4 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
    Send one packet to Port0 and Port1, check that thread on core1 and core2 
    waked up.
      L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0
      L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq0
  Expected test result:
    l3fwd-power can forward packets normally on VF and thread on core1 and 
    core2 will sleep when there's no packet received.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Cunming Liang
> Sent: Friday, February 27, 2015 12:56 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] [PATCH v6 0/8] Interrupt mode PMD
> 
> v6 changes
>  - split rte_intr_wait_rx_pkt into two APIs 'wait' and 'set'.
>  - rewrite rte_intr_rx_wait/rte_intr_rx_set.
>  - using vector number instead of queue_id as interrupt API params.
>  - patch reorder and split.
> 
> v5 changes
>  - Rebase the patchset onto the HEAD
>  - Isolate ethdev from EAL for new-added wait-for-rx interrupt function
>  - Export wait-for-rx interrupt function for shared libraries
>  - Split-off a new patch file for changed struct rte_intr_handle that
>    other patches depend on, to avoid breaking git bisect
>  - Change sample applicaiton to accomodate EAL function spec change
>    accordingly
> 
> v4 changes
>  - Export interrupt enable/disable functions for shared libraries
>  - Adjust position of new-added structure fields and functions to
>    avoid breaking ABI
> 
> v3 changes
>  - Add return value for interrupt enable/disable functions
>  - Move spinlok from PMD to L3fwd-power
>  - Remove unnecessary variables in e1000_mac_info
>  - Fix miscelleous review comments
> 
> v2 changes
>  - Fix compilation issue in Makefile for missed header file.
>  - Consolidate internal and community review comments of v1 patch set.
> 
> The patch series introduce low-latency one-shot rx interrupt into DPDK
> with
> polling and interrupt mode switch control example.
> 
> DPDK userspace interrupt notification and handling mechanism is based on
> UIO
> with below limitation:
> 1) It is designed to handle LSC interrupt only with inefficient suspended
>    pthread wakeup procedure (e.g. UIO wakes up LSC interrupt handling
> thread
>    which then wakes up DPDK polling thread). In this way, it introduces
>    non-deterministic wakeup latency for DPDK polling thread as well as
> packet
>    latency if it is used to handle Rx interrupt.
> 2) UIO only supports a single interrupt vector which has to been shared by
>    LSC interrupt and interrupts assigned to dedicated rx queues.
> 
> This patchset includes below features:
> 1) Enable one-shot rx queue interrupt in ixgbe PMD(PF & VF) and igb PMD(PF
> only).
> 2) Build on top of the VFIO mechanism instead of UIO, so it could support
>    up to 64 interrupt vectors for rx queue interrupts.
> 3) Have 1 DPDK polling thread handle per Rx queue interrupt with a
> dedicated
>    VFIO eventfd, which eliminates non-deterministic pthread wakeup latency
> in
>    user space.
> 4) Demonstrate interrupts control APIs and userspace NAIP-like
> polling/interrupt
>    switch algorithms in L3fwd-power example.
> 
> Known limitations:
> 1) It does not work for UIO due to a single interrupt eventfd shared by
> LSC
>    and rx queue interrupt handlers causes a mess.
> 2) LSC interrupt is not supported by VF driver, so it is by default
> disabled
>    in L3fwd-power now. Feel free to turn in on if you want to support both
> LSC
>    and rx queue interrupts on a PF.
> 
> Cunming Liang (5):
>   eal: declare new interrupt api
>   eal/linux: add rx queue interrupt FDs to intr handle struct
>   eal/bsd: dummy for new intr definition
>   eal/linux: add per rx queue interrupt handling based on VFIO
>   ethdev: add rx interrupt enable/disable functions
> 
> Zhou, Danny (3):
>   ixgbe: enable rx queue interrupts for both PF and VF
>   igb: enable rx queue interrupts for PF
>   l3fwd-power: enable one-shot rx interrupt and polling/interrupt mode
>     switch
> 
>  examples/l3fwd-power/main.c                        | 194 ++++++++---
>  lib/librte_eal/bsdapp/eal/eal_interrupts.c         |  15 +
>  .../bsdapp/eal/include/exec-env/rte_interrupts.h   |   4 +
>  lib/librte_eal/bsdapp/eal/rte_eal_version.map      |   2 +
>  lib/librte_eal/common/include/rte_interrupts.h     |  38 +++
>  lib/librte_eal/linuxapp/eal/eal_interrupts.c       | 224 +++++++++---
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c         |  23 +-
>  .../linuxapp/eal/include/exec-env/rte_interrupts.h |   9 +
>  lib/librte_eal/linuxapp/eal/rte_eal_version.map    |   2 +
>  lib/librte_ether/rte_ethdev.c                      |  66 ++++
>  lib/librte_ether/rte_ethdev.h                      |  77 +++++
>  lib/librte_ether/rte_ether_version.map             |   3 +
>  lib/librte_pmd_e1000/e1000_ethdev.h                |   3 +
>  lib/librte_pmd_e1000/igb_ethdev.c                  | 231 +++++++++++--
>  lib/librte_pmd_ixgbe/ixgbe_ethdev.c                | 377
> ++++++++++++++++++++-
>  lib/librte_pmd_ixgbe/ixgbe_ethdev.h                |   7 +
>  16 files changed, 1156 insertions(+), 119 deletions(-)
> 
> --
> 1.8.1.4