From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 3F13911A2 for ; Fri, 27 Feb 2015 09:00:18 +0100 (CET) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP; 26 Feb 2015 23:54:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.09,658,1418112000"; d="scan'208";a="657987044" Received: from pgsmsx104.gar.corp.intel.com ([10.221.44.91]) by orsmga001.jf.intel.com with ESMTP; 27 Feb 2015 00:00:16 -0800 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by PGSMSX104.gar.corp.intel.com (10.221.44.91) with Microsoft SMTP Server (TLS) id 14.3.195.1; Fri, 27 Feb 2015 16:00:13 +0800 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.197]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.209]) with mapi id 14.03.0195.001; Fri, 27 Feb 2015 16:00:11 +0800 From: "Liu, Yong" To: "Liang, Cunming" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v6 0/8] Interrupt mode PMD Thread-Index: AQHQUknK2vgP0C8/K0+OOxezTFUqVp0EIemA Date: Fri, 27 Feb 2015 08:00:10 +0000 Message-ID: <86228AFD5BCD8E4EBFD2B90117B5E81E10D8247D@SHSMSX103.ccr.corp.intel.com> References: <1424710542-14637-1-git-send-email-danny.zhou@intel.com> <1425012976-10173-1-git-send-email-cunming.liang@intel.com> In-Reply-To: <1425012976-10173-1-git-send-email-cunming.liang@intel.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v6 0/8] Interrupt mode PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Feb 2015 08:00:19 -0000 Tested-by: Yong Liu - Tested Commit: 00c685634b8a43e4594e26949a6c4f1cf5b67047 - OS: Fedora20 3.15.8-200.fc20.x86_64 - GCC: gcc version 4.8.3 20140911 (Red Hat 4.8.3-7) (GCC) - CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz - NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection - Default x86_64-native-linuxapp-gcc configuration - Total 4 cases, 4 passed, 0 failed - Case: interrupt pmd on PF with single queue Description: Check interrupt pmd work with single queue Command / instruction: Bind ports to vfio-pci. modprobe vfio modprobe vfio-pci ./tools/dpdk_nic_bind.py --bind=3Dvfio-pci 0000:08:00.0 0000:08:00.1 Start l3fwd-power with one queue per port. l3fwd-power -c 7 -n 4 -- -p 0x3 -P --config=3D"(0,0,1),(1,0,2)" Send one packet to Port0 and Port1, check that thread on core1 and core= 2=20 waked up. L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0 L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq0 Expected test result: l3fwd-power can forward packets normally and thread on core1 and core2= =20 will sleep when there's no packet received. - Case: interrupt pmd on PF with multi queue Description: Check interrupt pmd work with multiple queues Command / instruction: Start l3fwd-power with two queues per port. l3fwd-power -c 1f -n 4 -- -p 0x3 \ --config=3D"(0,0,1),(0,1,2)(1,0,3),(1,1,4)" Send packet with increased dest IP to Port0 and Port1, check that threa= d=20 on core1,core2,core3,core4 waked up. L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0 L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq1 L3FWD_POWER: lcore 3 is waked up from rx interrupt on port1,rxq0 L3FWD_POWER: lcore 4 is waked up from rx interrupt on port1,rxq1 Expected test result: l3fwd-power can forward packets normally and thread on core1-core4 will= =20 sleep when there's no packet received. - Case: interrupt pmd on PF with max Rx queues Description: Check interrupt pmd work with maximum queues Command / instruction: Start l3fwd-power with 32 queues per port. l3fwd-power -c ffffffff -n 4 -- -p 0x3 -P --config=3D"(0,0,0),(0,1,1),\ (0,2,2),(0,3,3),(0,4,4),(0,5,5),(0,6,6),(0,7,7),(0,8,8), (0,9,9),(0,10,10),(0,11,11),(0,12,12),(0,13,13),(0,14,14),\ (0,15,15),\ (1,0,16),(1,1,17),(1,2,18),(1,3,19),(1,4,20),(1,5,21),(1,6,22),\ (1,7,23),(1,8,24),(1,9,25),(1,10,26),(1,11,27),(1,12,28),\ (1,13,29),(1,14,30),\(1,15,31)" Send packet with increased dest IP to Port0 and Port1, check that all=20 threads waked up. Expected test result: l3fwd-power can forward packets normally and thread on core1-core31 will sleep when there's no packet received. =09 - Case: interrupt pmd on VF with single queue Description: Check interrupt pmd work on VF device Command / instruction: Bind ports to back to ixgbe driver. ./tools/dpdk_nic_bind.py --bind=3Dixgbe 0000:08:00.0 0000:08:00.1 Create one VF per Port in host and make sure PF interface up echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs ifconfig p786p1 up ifconfig p786p2 up =20 Bind VF device to vfio-pci. ./tools/dpdk_nic_bind.py --bind=3Dvfio-pci 0000:08:10.0 0000:08:10.1 Start l3fwd-power on host with one queue per port. l3fwd-power -c 1f -n 4 -- -p 0x3 -P --config=3D"(0,0,1),(1,0,2)" Send one packet to Port0 and Port1, check that thread on core1 and core= 2=20 waked up. L3FWD_POWER: lcore 1 is waked up from rx interrupt on port1,rxq0 L3FWD_POWER: lcore 2 is waked up from rx interrupt on port1,rxq0 Expected test result: l3fwd-power can forward packets normally on VF and thread on core1 and= =20 core2 will sleep when there's no packet received. > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Cunming Liang > Sent: Friday, February 27, 2015 12:56 PM > To: dev@dpdk.org > Subject: [dpdk-dev] [PATCH v6 0/8] Interrupt mode PMD >=20 > v6 changes > - split rte_intr_wait_rx_pkt into two APIs 'wait' and 'set'. > - rewrite rte_intr_rx_wait/rte_intr_rx_set. > - using vector number instead of queue_id as interrupt API params. > - patch reorder and split. >=20 > v5 changes > - Rebase the patchset onto the HEAD > - Isolate ethdev from EAL for new-added wait-for-rx interrupt function > - Export wait-for-rx interrupt function for shared libraries > - Split-off a new patch file for changed struct rte_intr_handle that > other patches depend on, to avoid breaking git bisect > - Change sample applicaiton to accomodate EAL function spec change > accordingly >=20 > v4 changes > - Export interrupt enable/disable functions for shared libraries > - Adjust position of new-added structure fields and functions to > avoid breaking ABI >=20 > v3 changes > - Add return value for interrupt enable/disable functions > - Move spinlok from PMD to L3fwd-power > - Remove unnecessary variables in e1000_mac_info > - Fix miscelleous review comments >=20 > v2 changes > - Fix compilation issue in Makefile for missed header file. > - Consolidate internal and community review comments of v1 patch set. >=20 > The patch series introduce low-latency one-shot rx interrupt into DPDK > with > polling and interrupt mode switch control example. >=20 > DPDK userspace interrupt notification and handling mechanism is based on > UIO > with below limitation: > 1) It is designed to handle LSC interrupt only with inefficient suspended > pthread wakeup procedure (e.g. UIO wakes up LSC interrupt handling > thread > which then wakes up DPDK polling thread). In this way, it introduces > non-deterministic wakeup latency for DPDK polling thread as well as > packet > latency if it is used to handle Rx interrupt. > 2) UIO only supports a single interrupt vector which has to been shared b= y > LSC interrupt and interrupts assigned to dedicated rx queues. >=20 > This patchset includes below features: > 1) Enable one-shot rx queue interrupt in ixgbe PMD(PF & VF) and igb PMD(P= F > only). > 2) Build on top of the VFIO mechanism instead of UIO, so it could support > up to 64 interrupt vectors for rx queue interrupts. > 3) Have 1 DPDK polling thread handle per Rx queue interrupt with a > dedicated > VFIO eventfd, which eliminates non-deterministic pthread wakeup latenc= y > in > user space. > 4) Demonstrate interrupts control APIs and userspace NAIP-like > polling/interrupt > switch algorithms in L3fwd-power example. >=20 > Known limitations: > 1) It does not work for UIO due to a single interrupt eventfd shared by > LSC > and rx queue interrupt handlers causes a mess. > 2) LSC interrupt is not supported by VF driver, so it is by default > disabled > in L3fwd-power now. Feel free to turn in on if you want to support bot= h > LSC > and rx queue interrupts on a PF. >=20 > Cunming Liang (5): > eal: declare new interrupt api > eal/linux: add rx queue interrupt FDs to intr handle struct > eal/bsd: dummy for new intr definition > eal/linux: add per rx queue interrupt handling based on VFIO > ethdev: add rx interrupt enable/disable functions >=20 > Zhou, Danny (3): > ixgbe: enable rx queue interrupts for both PF and VF > igb: enable rx queue interrupts for PF > l3fwd-power: enable one-shot rx interrupt and polling/interrupt mode > switch >=20 > examples/l3fwd-power/main.c | 194 ++++++++--- > lib/librte_eal/bsdapp/eal/eal_interrupts.c | 15 + > .../bsdapp/eal/include/exec-env/rte_interrupts.h | 4 + > lib/librte_eal/bsdapp/eal/rte_eal_version.map | 2 + > lib/librte_eal/common/include/rte_interrupts.h | 38 +++ > lib/librte_eal/linuxapp/eal/eal_interrupts.c | 224 +++++++++--- > lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 23 +- > .../linuxapp/eal/include/exec-env/rte_interrupts.h | 9 + > lib/librte_eal/linuxapp/eal/rte_eal_version.map | 2 + > lib/librte_ether/rte_ethdev.c | 66 ++++ > lib/librte_ether/rte_ethdev.h | 77 +++++ > lib/librte_ether/rte_ether_version.map | 3 + > lib/librte_pmd_e1000/e1000_ethdev.h | 3 + > lib/librte_pmd_e1000/igb_ethdev.c | 231 +++++++++++-- > lib/librte_pmd_ixgbe/ixgbe_ethdev.c | 377 > ++++++++++++++++++++- > lib/librte_pmd_ixgbe/ixgbe_ethdev.h | 7 + > 16 files changed, 1156 insertions(+), 119 deletions(-) >=20 > -- > 1.8.1.4