From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from APAC01-SG1-obe.outbound.protection.outlook.com (mail-sg1on0116.outbound.protection.outlook.com [134.170.132.116]) by dpdk.org (Postfix) with ESMTP id 2EC796A87 for ; Mon, 20 Apr 2015 12:01:11 +0200 (CEST) Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none; Received: from mail-ig0-f180.google.com (209.85.213.180) by HKNPR04MB146.apcprd04.prod.outlook.com (10.242.103.147) with Microsoft SMTP Server (TLS) id 15.1.136.25; Mon, 20 Apr 2015 10:01:07 +0000 Received: by igbhj9 with SMTP id hj9so56044815igb.1 for ; Mon, 20 Apr 2015 03:01:00 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.50.61.226 with SMTP id t2mr6866955igr.19.1429523599477; Mon, 20 Apr 2015 02:53:19 -0700 (PDT) Received: by 10.50.37.232 with HTTP; Mon, 20 Apr 2015 02:53:19 -0700 (PDT) In-Reply-To: References: <1416924682-24170-1-git-send-email-cunming.liang@intel.com> Date: Mon, 20 Apr 2015 17:53:19 +0800 Message-ID: From: Shelton Chia To: Cunming Liang , X-Originating-IP: [209.85.213.180] X-ClientProxiedBy: HKNPR04CA003.apcprd04.prod.outlook.com (10.242.116.33) To HKNPR04MB146.apcprd04.prod.outlook.com (10.242.103.147) X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HKNPR04MB146; X-Forefront-Antispam-Report: BMV:1; SFV:NSPM; SFS:(10019020)(377424004)(51704005)(450100001)(40100003)(62966003)(5001770100001)(55446002)(54356999)(63696999)(53806999)(15975445007)(2950100001)(77156002)(98316002)(84326002)(66066001)(59536001)(42186005)(16601075003)(107886001)(76176999)(86362001)(122856001)(61726006)(122386002)(46102003)(87976001)(43066003)(512874002)(19580405001)(50986999)(19617315012)(19580395003)(61266001)(42262002)(217873001); DIR:OUT; SFP:1102; SCL:1; SRVR:HKNPR04MB146; H:mail-ig0-f180.google.com; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5002010)(5005006); SRVR:HKNPR04MB146; BCL:0; PCL:0; RULEID:; SRVR:HKNPR04MB146; X-Forefront-PRVS: 05529C6FDB X-OriginatorOrg: sinogrid.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Apr 2015 10:01:07.1558 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HKNPR04MB146 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [RFC PATCH 0/6] DPDK support to bifurcated driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: jiaxt@sinogrid.com List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Apr 2015 10:01:13 -0000 Hi, I can receive packets when I mmaped all pci memory not only rx and tx desc. 2015-04-09 11:43 GMT+08:00 =E8=B4=BE=E5=AD=A6=E6=B6=9B = : > Hi Cunming, > I applyed bifurc dirver patches and tested it follow your example. > But I can't received packets with testpmd and l2fwd. > Kernel stack can receive packets from 10.0.0.2 before "ethtool -N > XGE4.1 flow-type ip4 src-ip 10.0.0.2 action 12". After "thtool -N XGE4.1 > flow-type ip4 src-ip 10.0.0.2 action 12", kernel stack can't receive > packets from 10.0.0.2, but testpmd and l2fwd cannot receive any packets > too. > queue 0-11 used by kernel and queue 12 used by bifurc dirver. > How can I make it work? > > 2014-11-25 22:11 GMT+08:00 Cunming Liang : > >> >> This is a RFC patch set to support "bifurcated driver" in DPDK. >> >> >> What is "bifurcated driver"? >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D >> >> The "bifurcated driver" stands for the kernel NIC driver that supports: >> >> 1. on-demand rx/tx queue pairs split-off and assignment to user space >> >> 2. direct NIC resource(e.g. rx/tx queue registers) access from user spac= e >> >> 3. distributing packets to kernel or user space rx queues by >> NIC's flow director according to the filter rules >> >> Here's the kernel patch set to support. >> http://comments.gmane.org/gmane.linux.network/333615 >> >> >> Usage scenario >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> >> It's well accepted by industry to use DPDK to process fast path packets = in >> user space in a high performance fashion, meanwhile processing slow path >> control packets in kernel space is still needed as those packets usually >> rely on in_kernel TCP/IP stacks and/or socket programming interface. >> >> KNI(Kernel NIC Interface) mechanism in DPDK is designed to meet this >> requirement, with below limitation: >> >> 1) Software classifies packets and distributes them to kernel via DPDK >> software rings, at the cost of significant CPU cycles and memory >> bandwidth. >> >> 2) Memory copy packets between kernel' socket buffer and mbuf brings >> significant negative performance impact to KNI performance. >> >> The bifurcated driver provides a alternative approach that not only >> offloads >> flow classification and distribution to NIC but also support packets >> zero_copy. >> >> User can use standard ethtool to add filter rules to the NIC in order to >> distribute specific flows to the queues only accessed by kernel driver a= nd >> stack, and add other rules to distribute packets to the queues assigned = to >> user-space. >> >> For those rx/tx queue pairs that directly accessed from user space, >> DPDK takes over the packets rx/tx as well as corresponding DMA operation >> for high performance packet I/O. >> >> >> What's the impact and change to DPDK >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> >> DPDK usually binds PCIe NIC devices by leveraging kernel' user space >> driver >> mechanism UIO or VFIO to map entire NIC' PCIe I/O space of NIC to user >> space. >> The bifurcated driver PMD talks to a NIC interface using raw socket APIs >> and >> only mmap() limited I/O space (e.g. certain 4K pages) for accessing >> involved >> rx/tx queue pairs. So the impact and changes mainly comes with below: >> >> - netdev >> DPDK needs to create a af_packet socket and bind it to a bifurcated >> netdev. >> The socket fd will be used to request 'queue pairs info', >> 'split/return queue pairs' and etc. The PCIe device ID, netdev MAC >> address, >> numa info are also from the netdev response. >> >> - PCIe device scan and driver probe >> netdev provides the PCIe device ID information. Refer to the device >> ID, >> the correct driver should be used. And for such netdev device, the >> creation >> of PCIe device is no longer from scan but the on-demand assignment. >> >> - PCIe BAR mapping >> "bifurcated driver" maps several pages for the queue pairs. >> Others BAR register space maps to a fake page. The BAR mapping go >> through >> mmap on sockfd. Which is a little different from what UIO/VFIO does. >> >> - PMD >> The PMD will no longer really initialize and configure NIC. >> Instead, it only takes care the queue pair setup, rx_burst and >> tx_burst. >> >> The patch uses eal '--vdev' parameter to assign netdev iface name and >> number of >> queue pairs. Here's a example about how to configure the bifurcated >> driver and >> run DPDK testpmd with bifurcated PMD. >> >> 1. Set promisc mode >> > ifconfig eth0 promisc >> >> 2. Turn on fdir >> > ethtool -K eth0 ntuple on >> >> 3. Setup a flow director rule to distribute packets with source ip >> 0.0.0.0 to rxq No.0 >> > ethtool -N eth0 flow-type udp4 src-ip 0.0.0.0 action 0 >> >> 4. Run testpmd on netdev 'eth0' with 1 queue pair. >> > ./x86_64-native-linuxapp-gcc/app/testpmd -c 0x3 -n 4 \ >> > --vdev=3Drte_bifurc,iface=3Deth0,qpairs=3D1 -- \ >> > -i --rxfreet=3D32 --txfreet=3D32 --txrst=3D32 >> Note: >> iface and qpairs arguments above specify the netdev interface name a= nd >> number of qpairs that user space request from the "bifurcated driver= " >> respectively. >> >> 5. Setup a flow director rule to distribute packets with source ip >> 1.1.1.1 to rxq No.32. This needs to be done after testpmd starts. >> > ethtool -N eth0 flow-type udp4 src-ip 1.1.1.1 action 32 >> >> Below illustrates the detailed changes in this patch set. >> >> eal >> -------- >> The first two patches are all about the eal API declaration and Linux >> version >> definition to support af_packet socket and verbs of bifurcated netdev. >> Those APIs include the verbs like open, bind, (un)map, split/retturn, >> map_umem. >> And other APIs like set_pci, get_ifinfo and get/put_devargs which help t= o >> generate pci device from bifurcated netdev and get basic netdev info. >> >> The third patch is used to allow probing driver on the PCIe VDEV created >> from >> a NIC interface driven by "bifurcated driver". It defines a new flag >> 'RTE_PCI_DRV_BIFURC' used for direct ring access PMD. >> >> librte_bifurc >> --------------- >> The library is used as a VDEV bus driver to scan '--vdev=3Drte_bifurc' V= DEV >> from eal command-line. It generates the PCIe VDEV device ready for furth= er >> driver probe. It maintains the bifurcated device information include >> sockfd, >> hwaddr, mtu, qpairs, iface_name. It's used for other direct ring access >> PMD >> to apply for bifurcated device info. >> >> direct ring access PMD >> ------------------------- >> The patch provides direct ring access PMD for ixgbe. Comparing to the >> normal >> PMD ixgbe, it uses 'RTE_PCI_DRV_BIFURC' flag during self registration. >> It mostly reuses the existing PMD ops to avoid re-implementing everythin= g >> from scratch. And it also modifies the rx/tx_queue_setup to allow queue >> setup from any queue offset. >> >> Supported NIC driver >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> >> The "bifurcated driver" kernel patch only supports "ixgbe" driver at the >> moment, >> so this RFC patch also provides "ixgbe" PMD via direct-mapped rings as >> sample. >> The support for 40GE(i40e) will be added in the future. >> >> In addition, for those multi-queues enabled NIC with flow director >> capability >> to do perform packet classification and distribution, there's no special >> technical gap to provide bifurcated driver approach support. >> >> Limitation >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> >> By using "bifurcated driver", user space only takes over the DMA >> operation. >> For those NIC configure setting, it's out of control from user space PMD= . >> All the NIC setting including add/del filter rules need to be done by >> standard Linux network tools(e.g. ethtool). >> So the feature support really depend on how much are supported by ethtoo= l. >> >> >> Any questions, comments and feedback are welcome. >> >> >> -END- >> >> Signed-off-by: Cunming Liang >> Signed-off-by: Danny Zhou >> >> *** BLURB HERE *** >> >> Cunming Liang (6): >> eal: common direct ring access API >> eal: direct ring access support by linux af_packet >> pci: allow VDEV as pci device during device driver probe >> bifurc: add driver to scan bifurcated netdev >> ixgbe: rx/tx queue stop bug fix >> ixgbe: PMD for bifurc ixgbe net device >> >> config/common_linuxapp | 5 + >> lib/Makefile | 1 + >> lib/librte_bifurc/Makefile | 58 +++++ >> lib/librte_bifurc/rte_bifurc.c | 284 >> +++++++++++++++++++++ >> lib/librte_bifurc/rte_bifurc.h | 90 +++++++ >> lib/librte_eal/common/Makefile | 5 + >> lib/librte_eal/common/include/rte_pci.h | 4 + >> lib/librte_eal/common/include/rte_pci_bifurc.h | 186 ++++++++++++++ >> lib/librte_eal/linuxapp/eal/Makefile | 1 + >> lib/librte_eal/linuxapp/eal/eal_pci.c | 42 ++-- >> lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c | 336 >> +++++++++++++++++++++++++ >> lib/librte_ether/rte_ethdev.c | 3 +- >> lib/librte_pmd_ixgbe/Makefile | 13 +- >> lib/librte_pmd_ixgbe/ixgbe_bifurcate.c | 303 >> ++++++++++++++++++++++ >> lib/librte_pmd_ixgbe/ixgbe_bifurcate.h | 57 +++++ >> lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 44 +++- >> lib/librte_pmd_ixgbe/ixgbe_rxtx.h | 10 + >> mk/rte.app.mk | 6 + >> 18 files changed, 1421 insertions(+), 27 deletions(-) >> create mode 100644 lib/librte_bifurc/Makefile >> create mode 100644 lib/librte_bifurc/rte_bifurc.c >> create mode 100644 lib/librte_bifurc/rte_bifurc.h >> create mode 100644 lib/librte_eal/common/include/rte_pci_bifurc.h >> create mode 100644 lib/librte_eal/linuxapp/eal/eal_pci_bifurc.c >> create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.c >> create mode 100644 lib/librte_pmd_ixgbe/ixgbe_bifurcate.h >> >> -- >> 1.8.1.4 >> >> >