From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 459285910 for ; Tue, 25 Nov 2014 15:04:10 +0100 (CET) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 25 Nov 2014 06:11:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,455,1413270000"; d="scan'208";a="643224983" Received: from shvmail01.sh.intel.com ([10.239.29.42]) by orsmga002.jf.intel.com with ESMTP; 25 Nov 2014 06:11:32 -0800 Received: from shecgisg004.sh.intel.com (shecgisg004.sh.intel.com [10.239.29.89]) by shvmail01.sh.intel.com with ESMTP id sAPEBTqS022613; Tue, 25 Nov 2014 22:11:29 +0800 Received: from shecgisg004.sh.intel.com (localhost [127.0.0.1]) by shecgisg004.sh.intel.com (8.13.6/8.13.6/SuSE Linux 0.8) with ESMTP id sAPEBRx3024214; Tue, 25 Nov 2014 22:11:29 +0800 Received: (from cliang18@localhost) by shecgisg004.sh.intel.com (8.13.6/8.13.6/Submit) id sAPEBQ7B024210; Tue, 25 Nov 2014 22:11:26 +0800 From: Cunming Liang To: dev@dpdk.org Date: Tue, 25 Nov 2014 22:11:16 +0800 Message-Id: <1416924682-24170-1-git-send-email-cunming.liang@intel.com> X-Mailer: git-send-email 1.7.4.1 Subject: [dpdk-dev] [RFC PATCH 0/6] DPDK support to bifurcated driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2014 14:04:11 -0000 This is a RFC patch set to support "bifurcated driver" in DPDK. What is "bifurcated driver"? =========================== The "bifurcated driver" stands for the kernel NIC driver that supports: 1. on-demand rx/tx queue pairs split-off and assignment to user space 2. direct NIC resource(e.g. rx/tx queue registers) access from user space 3. distributing packets to kernel or user space rx queues by NIC's flow director according to the filter rules Here's the kernel patch set to support. http://comments.gmane.org/gmane.linux.network/333615 Usage scenario ================= It's well accepted by industry to use DPDK to process fast path packets in user space in a high performance fashion, meanwhile processing slow path control packets in kernel space is still needed as those packets usually rely on in_kernel TCP/IP stacks and/or socket programming interface. KNI(Kernel NIC Interface) mechanism in DPDK is designed to meet this requirement, with below limitation: 1) Software classifies packets and distributes them to kernel via DPDK software rings, at the cost of significant CPU cycles and memory bandwidth. 2) Memory copy packets between kernel' socket buffer and mbuf brings significant negative performance impact to KNI performance. The bifurcated driver provides a alternative approach that not only offloads flow classification and distribution to NIC but also support packets zero_copy. User can use standard ethtool to add filter rules to the NIC in order to distribute specific flows to the queues only accessed by kernel driver and stack, and add other rules to distribute packets to the queues assigned to user-space. For those rx/tx queue pairs that directly accessed from user space, DPDK takes over the packets rx/tx as well as corresponding DMA operation for high performance packet I/O. What's the impact and change to DPDK ====================================== DPDK usually binds PCIe NIC devices by leveraging kernel' user space driver mechanism UIO or VFIO to map entire NIC' PCIe I/O space of NIC to user space. The bifurcated driver PMD talks to a NIC interface using raw socket APIs and only mmap() limited I/O space (e.g. certain 4K pages) for accessing involved rx/tx queue pairs. So the impact and changes mainly comes with below: - netdev DPDK needs to create a af_packet socket and bind it to a bifurcated netdev. The socket fd will be used to request 'queue pairs info', 'split/return queue pairs' and etc. The PCIe device ID, netdev MAC address, numa info are also from the netdev response. - PCIe device scan and driver probe netdev provides the PCIe device ID information. Refer to the device ID, the correct driver should be used. And for such netdev device, the creation of PCIe device is no longer from scan but the on-demand assignment. - PCIe BAR mapping "bifurcated driver" maps several pages for the queue pairs. Others BAR register space maps to a fake page. The BAR mapping go through mmap on sockfd. Which is a little different from what UIO/VFIO does. - PMD The PMD will no longer really initialize and configure NIC. Instead, it only takes care the queue pair setup, rx_burst and tx_burst. The patch uses eal '--vdev' parameter to assign netdev iface name and number of queue pairs. Here's a example about how to configure the bifurcated driver and run DPDK testpmd with bifurcated PMD. 1. Set promisc mode > ifconfig eth0 promisc 2. Turn on fdir > ethtool -K eth0 ntuple on 3. Setup a flow director rule to distribute packets with source ip 0.0.0.0 to rxq No.0 > ethtool -N eth0 flow-type udp4 src-ip 0.0.0.0 action 0 4. Run testpmd on netdev 'eth0' with 1 queue pair. > ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 \ > --vdev=rte_bifurc,iface=eth0,qpairs=1 -- \ > -i --rxfreet=32 --txfreet=32 --txrst=32 Note: iface and qpairs arguments above specify the netdev interface name and number of qpairs that user space request from the "bifurcated driver" respectively. 5. Setup a flow director rule to distribute packets with source ip 1.1.1.1 to rxq No.32. This needs to be done after testpmd starts. > ethtool -N eth0 flow-type udp4 src-ip 1.1.1.1 action 32 Below illustrates the detailed changes in this patch set. eal -------- The first two patches are all about the eal API declaration and Linux version definition to support af_packet socket and verbs of bifurcated netdev. Those APIs include the verbs like open, bind, (un)map, split/retturn, map_umem. And other APIs like set_pci, get_ifinfo and get/put_devargs which help to generate pci device from bifurcated netdev and get basic netdev info. The third patch is used to allow probing driver on the PCIe VDEV created from a NIC interface driven by "bifurcated driver". It defines a new flag 'RTE_PCI_DRV_BIFURC' used for direct ring access PMD. librte_bifurc --------------- The library is used as a VDEV bus driver to scan '--vdev=rte_bifurc' VDEV from eal command-line. It generates the PCIe VDEV device ready for further driver probe. It maintains the bifurcated device information include sockfd, hwaddr, mtu, qpairs, iface_name. It's used for other direct ring access PMD to apply for bifurcated device info. direct ring access PMD ------------------------- The patch provides direct ring access PMD for ixgbe. Comparing to the normal PMD ixgbe, it uses 'RTE_PCI_DRV_BIFURC' flag during self registration. It mostly reuses the existing PMD ops to avoid re-implementing everything from scratch. And it also modifies the rx/tx_queue_setup to allow queue setup from any queue offset. Supported NIC driver ======================== The "bifurcated driver" kernel patch only supports "ixgbe" driver at the moment, so this RFC patch also provides "ixgbe" PMD via direct-mapped rings as sample. The support for 40GE(i40e) will be added in the future. In addition, for those multi-queues enabled NIC with flow director capability to do perform packet classification and distribution, there's no special technical gap to provide bifurcated driver approach support. Limitation ============ By using "bifurcated driver", user space only takes over the DMA operation. For those NIC configure setting, it's out of control from user space PMD. All the NIC setting including add/del filter rules need to be done by standard Linux network tools(e.g. ethtool). So the feature support really depend on how much are supported by ethtool. Any questions, comments and feedback are welcome. -END- Signed-off-by: Cunming Liang Signed-off-by: Danny Zhou *** BLURB HERE *** Cunming Liang (6): eal: common direct ring access API eal: direct ring access support by linux af_packet pci: allow VDEV as pci device during device driver probe bifurc: add driver to scan bifurcated netdev ixgbe: rx/tx queue stop bug fix ixgbe: PMD for bifurc ixgbe net device config/common_linuxapp | 5 + lib/Makefile | 1 + lib/librte_bifurc/Makefile | 58 +++++ lib/librte_bifurc/rte_bifurc.c | 284 +++++++++++++++++++++ lib/librte_bifurc/rte_bifurc.h | 90 +++++++ lib/librte_eal/common/Makefile | 5 + lib/librte_eal/common/include/rte_pci.h | 4 + lib/librte_eal/common/include/rte_pci_bifurc.h | 186 ++++++++++++++ lib/librte_eal/linuxapp/eal/Makefile | 1 + lib/librte_eal/linuxapp/eal/eal_pci.c | 42 ++-- lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c | 336 +++++++++++++++++++++++++ lib/librte_ether/rte_ethdev.c | 3 +- lib/librte_pmd_ixgbe/Makefile | 13 +- lib/librte_pmd_ixgbe/ixgbe_bifurcate.c | 303 ++++++++++++++++++++++ lib/librte_pmd_ixgbe/ixgbe_bifurcate.h | 57 +++++ lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 44 +++- lib/librte_pmd_ixgbe/ixgbe_rxtx.h | 10 + mk/rte.app.mk | 6 + 18 files changed, 1421 insertions(+), 27 deletions(-) create mode 100644 lib/librte_bifurc/Makefile create mode 100644 lib/librte_bifurc/rte_bifurc.c create mode 100644 lib/librte_bifurc/rte_bifurc.h create mode 100644 lib/librte_eal/common/include/rte_pci_bifurc.h create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.c create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.h -- 1.8.1.4