From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) by dpdk.org (Postfix) with ESMTP id 39A815921 for ; Thu, 9 Apr 2015 05:43:27 +0200 (CEST) Received: by igblo3 with SMTP id lo3so55324313igb.0 for ; Wed, 08 Apr 2015 20:43:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=pBF3wK94ZujBtuJZisYM8SHBK/Abhwe5L1FO/bQDYHA=; b=Q5njPosWfz5NZuBUI1mnIUvO+TCH+jfO8YOgFP7oUtHa5SezUw2fAf5TTo6VW9HZEF rWzQXD/n9Sh3ZRDrJS6Mgjb/xFnx9QU/5jiPwrl2P9Q3dYIiWE2l9NoPHWGibByBioIK C2ye9nMnzCp+aVVbIlNvW0MwAhxEu1m7h+wx2edPrTbDkRX9vsamjnlw1kR9PdjM0idU RdnNMudGkgoFm49W2YH34JQ5PCWbSOFWbvkb+vpcIT6r9sQU/uP1TxtHHzqWemvKkh0h BLNs/0BW7GgkQaELkoHXH2WFAkDnjEEY2jFQtAgeULacY34IC0wiNMQKuNQJhDfatOX3 66qQ== MIME-Version: 1.0 X-Received: by 10.107.137.218 with SMTP id t87mr44769397ioi.3.1428551006624; Wed, 08 Apr 2015 20:43:26 -0700 (PDT) Sender: letaotor@gmail.com Received: by 10.50.169.42 with HTTP; Wed, 8 Apr 2015 20:43:26 -0700 (PDT) In-Reply-To: <1416924682-24170-1-git-send-email-cunming.liang@intel.com> References: <1416924682-24170-1-git-send-email-cunming.liang@intel.com> Date: Thu, 9 Apr 2015 11:43:26 +0800 X-Google-Sender-Auth: iHA_217wx2eNWxgWPjjjGtffc90 Message-ID: From: =?UTF-8?B?6LS+5a2m5rab?= To: Cunming Liang , dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [RFC PATCH 0/6] DPDK support to bifurcated driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: jiaxt@sinogrid.com List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2015 03:43:27 -0000 Hi Cunming, I applyed bifurc dirver patches and tested it follow your example. But I can't received packets with testpmd and l2fwd. Kernel stack can receive packets from 10.0.0.2 before "ethtool -N XGE4.1 flow-type ip4 src-ip 10.0.0.2 action 12". After "thtool -N XGE4.1 flow-type ip4 src-ip 10.0.0.2 action 12", kernel stack can't receive packets from 10.0.0.2, but testpmd and l2fwd cannot receive any packets too. queue 0-11 used by kernel and queue 12 used by bifurc dirver. How can I make it work? 2014-11-25 22:11 GMT+08:00 Cunming Liang : > > This is a RFC patch set to support "bifurcated driver" in DPDK. > > > What is "bifurcated driver"? > =========================== > > The "bifurcated driver" stands for the kernel NIC driver that supports: > > 1. on-demand rx/tx queue pairs split-off and assignment to user space > > 2. direct NIC resource(e.g. rx/tx queue registers) access from user space > > 3. distributing packets to kernel or user space rx queues by > NIC's flow director according to the filter rules > > Here's the kernel patch set to support. > http://comments.gmane.org/gmane.linux.network/333615 > > > Usage scenario > ================= > > It's well accepted by industry to use DPDK to process fast path packets in > user space in a high performance fashion, meanwhile processing slow path > control packets in kernel space is still needed as those packets usually > rely on in_kernel TCP/IP stacks and/or socket programming interface. > > KNI(Kernel NIC Interface) mechanism in DPDK is designed to meet this > requirement, with below limitation: > > 1) Software classifies packets and distributes them to kernel via DPDK > software rings, at the cost of significant CPU cycles and memory > bandwidth. > > 2) Memory copy packets between kernel' socket buffer and mbuf brings > significant negative performance impact to KNI performance. > > The bifurcated driver provides a alternative approach that not only > offloads > flow classification and distribution to NIC but also support packets > zero_copy. > > User can use standard ethtool to add filter rules to the NIC in order to > distribute specific flows to the queues only accessed by kernel driver and > stack, and add other rules to distribute packets to the queues assigned to > user-space. > > For those rx/tx queue pairs that directly accessed from user space, > DPDK takes over the packets rx/tx as well as corresponding DMA operation > for high performance packet I/O. > > > What's the impact and change to DPDK > ====================================== > > DPDK usually binds PCIe NIC devices by leveraging kernel' user space driver > mechanism UIO or VFIO to map entire NIC' PCIe I/O space of NIC to user > space. > The bifurcated driver PMD talks to a NIC interface using raw socket APIs > and > only mmap() limited I/O space (e.g. certain 4K pages) for accessing > involved > rx/tx queue pairs. So the impact and changes mainly comes with below: > > - netdev > DPDK needs to create a af_packet socket and bind it to a bifurcated > netdev. > The socket fd will be used to request 'queue pairs info', > 'split/return queue pairs' and etc. The PCIe device ID, netdev MAC > address, > numa info are also from the netdev response. > > - PCIe device scan and driver probe > netdev provides the PCIe device ID information. Refer to the device ID, > the correct driver should be used. And for such netdev device, the > creation > of PCIe device is no longer from scan but the on-demand assignment. > > - PCIe BAR mapping > "bifurcated driver" maps several pages for the queue pairs. > Others BAR register space maps to a fake page. The BAR mapping go > through > mmap on sockfd. Which is a little different from what UIO/VFIO does. > > - PMD > The PMD will no longer really initialize and configure NIC. > Instead, it only takes care the queue pair setup, rx_burst and > tx_burst. > > The patch uses eal '--vdev' parameter to assign netdev iface name and > number of > queue pairs. Here's a example about how to configure the bifurcated driver > and > run DPDK testpmd with bifurcated PMD. > > 1. Set promisc mode > > ifconfig eth0 promisc > > 2. Turn on fdir > > ethtool -K eth0 ntuple on > > 3. Setup a flow director rule to distribute packets with source ip > 0.0.0.0 to rxq No.0 > > ethtool -N eth0 flow-type udp4 src-ip 0.0.0.0 action 0 > > 4. Run testpmd on netdev 'eth0' with 1 queue pair. > > ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 \ > > --vdev=rte_bifurc,iface=eth0,qpairs=1 -- \ > > -i --rxfreet=32 --txfreet=32 --txrst=32 > Note: > iface and qpairs arguments above specify the netdev interface name and > number of qpairs that user space request from the "bifurcated driver" > respectively. > > 5. Setup a flow director rule to distribute packets with source ip > 1.1.1.1 to rxq No.32. This needs to be done after testpmd starts. > > ethtool -N eth0 flow-type udp4 src-ip 1.1.1.1 action 32 > > Below illustrates the detailed changes in this patch set. > > eal > -------- > The first two patches are all about the eal API declaration and Linux > version > definition to support af_packet socket and verbs of bifurcated netdev. > Those APIs include the verbs like open, bind, (un)map, split/retturn, > map_umem. > And other APIs like set_pci, get_ifinfo and get/put_devargs which help to > generate pci device from bifurcated netdev and get basic netdev info. > > The third patch is used to allow probing driver on the PCIe VDEV created > from > a NIC interface driven by "bifurcated driver". It defines a new flag > 'RTE_PCI_DRV_BIFURC' used for direct ring access PMD. > > librte_bifurc > --------------- > The library is used as a VDEV bus driver to scan '--vdev=rte_bifurc' VDEV > from eal command-line. It generates the PCIe VDEV device ready for further > driver probe. It maintains the bifurcated device information include > sockfd, > hwaddr, mtu, qpairs, iface_name. It's used for other direct ring access PMD > to apply for bifurcated device info. > > direct ring access PMD > ------------------------- > The patch provides direct ring access PMD for ixgbe. Comparing to the > normal > PMD ixgbe, it uses 'RTE_PCI_DRV_BIFURC' flag during self registration. > It mostly reuses the existing PMD ops to avoid re-implementing everything > from scratch. And it also modifies the rx/tx_queue_setup to allow queue > setup from any queue offset. > > Supported NIC driver > ======================== > > The "bifurcated driver" kernel patch only supports "ixgbe" driver at the > moment, > so this RFC patch also provides "ixgbe" PMD via direct-mapped rings as > sample. > The support for 40GE(i40e) will be added in the future. > > In addition, for those multi-queues enabled NIC with flow director > capability > to do perform packet classification and distribution, there's no special > technical gap to provide bifurcated driver approach support. > > Limitation > ============ > > By using "bifurcated driver", user space only takes over the DMA operation. > For those NIC configure setting, it's out of control from user space PMD. > All the NIC setting including add/del filter rules need to be done by > standard Linux network tools(e.g. ethtool). > So the feature support really depend on how much are supported by ethtool. > > > Any questions, comments and feedback are welcome. > > > -END- > > Signed-off-by: Cunming Liang > Signed-off-by: Danny Zhou > > *** BLURB HERE *** > > Cunming Liang (6): > eal: common direct ring access API > eal: direct ring access support by linux af_packet > pci: allow VDEV as pci device during device driver probe > bifurc: add driver to scan bifurcated netdev > ixgbe: rx/tx queue stop bug fix > ixgbe: PMD for bifurc ixgbe net device > > config/common_linuxapp | 5 + > lib/Makefile | 1 + > lib/librte_bifurc/Makefile | 58 +++++ > lib/librte_bifurc/rte_bifurc.c | 284 +++++++++++++++++++++ > lib/librte_bifurc/rte_bifurc.h | 90 +++++++ > lib/librte_eal/common/Makefile | 5 + > lib/librte_eal/common/include/rte_pci.h | 4 + > lib/librte_eal/common/include/rte_pci_bifurc.h | 186 ++++++++++++++ > lib/librte_eal/linuxapp/eal/Makefile | 1 + > lib/librte_eal/linuxapp/eal/eal_pci.c | 42 ++-- > lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c | 336 > +++++++++++++++++++++++++ > lib/librte_ether/rte_ethdev.c | 3 +- > lib/librte_pmd_ixgbe/Makefile | 13 +- > lib/librte_pmd_ixgbe/ixgbe_bifurcate.c | 303 > ++++++++++++++++++++++ > lib/librte_pmd_ixgbe/ixgbe_bifurcate.h | 57 +++++ > lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 44 +++- > lib/librte_pmd_ixgbe/ixgbe_rxtx.h | 10 + > mk/rte.app.mk | 6 + > 18 files changed, 1421 insertions(+), 27 deletions(-) > create mode 100644 lib/librte_bifurc/Makefile > create mode 100644 lib/librte_bifurc/rte_bifurc.c > create mode 100644 lib/librte_bifurc/rte_bifurc.h > create mode 100644 lib/librte_eal/common/include/rte_pci_bifurc.h > create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c > create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.c > create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.h > > -- > 1.8.1.4 > >