* [dpdk-dev] add support for HTM lock elision for x86
@ 2015-06-02 13:17 Dementiev, Roman
0 siblings, 0 replies; 6+ messages in thread
From: Dementiev, Roman @ 2015-06-02 13:17 UTC (permalink / raw)
To: dev
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
From rolette@infiniteio.com Tue Jun 2 15:21:24 2015
Return-Path: <rolette@infiniteio.com>
Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com
[209.85.212.182]) by dpdk.org (Postfix) with ESMTP id B03FBBDC2
for <dev@dpdk.org>; Tue, 2 Jun 2015 15:21:24 +0200 (CEST)
Received: by wizo1 with SMTP id o1so144374926wiz.1
for <dev@dpdk.org>; Tue, 02 Jun 2015 06:21:24 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d\x1e100.net; s 130820;
h=x-gm-message-state:mime-version:in-reply-to:references:date
:message-id:subject:from:to:cc:content-type;
bh=EZ5V09Uf0HqR5pepp3xmM2K+ESwvO6YmUxphj1p/FH4=;
b=KSmNxWms+vaLWfI43mVrAVKs9hjo5OOHZj8i/fWcPZBQdx7OF1WZqsohu1JQL/p5c2
4wxywVCESWJefU+OtJ4aluXRwJMbKZaX6KGPS9yNIEkXffuZhxri5VkEzyig3Cz6FzZg
vMEvsEiVcO3b40xPZ/KtVENZ1AXE48EeeXwvlcpKNzbSittMT2qxJvWqIFtWIhHyZz2G
LXZ7WdpwSYhyobXxFX/kMMe2n/qDqlAvgo+pIgf0jZdJVBTqz/2Mx8MdJQlCz2BQbzpv
KyylO/bE46ktW7D8eQVKtBMtykrW1svCYCebSYLEiWD9PA/Mlq+g59HV7uOBjpR0LRag
JHDw=X-Gm-Message-State: ALoCoQnTMgOKUua0wm5br0wzyd2vyF1dbh6JeCzRCh1zTNb0jX61HFls8jGfZRV8ah0sjrcDwWll
MIME-Version: 1.0
X-Received: by 10.195.18.8 with SMTP id gi8mr11293724wjd.153.1433251284551;
Tue, 02 Jun 2015 06:21:24 -0700 (PDT)
Received: by 10.194.36.193 with HTTP; Tue, 2 Jun 2015 06:21:24 -0700 (PDT)
In-Reply-To: <1433250693-23644-1-git-send-email-roman.dementiev@intel.com>
References: <1433250693-23644-1-git-send-email-roman.dementiev@intel.com>
Date: Tue, 2 Jun 2015 08:21:24 -0500
Message-ID: <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
From: Jay Rolette <rolette@infiniteio.com>
To: Roman Dementiev <roman.dementiev@intel.com>
Content-Type: text/plain; charset=UTF-8
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Cc: DPDK <dev@dpdk.org>
Subject: Re: [dpdk-dev] add support for HTM lock elision for x86
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 02 Jun 2015 13:21:24 -0000
On Tue, Jun 2, 2015 at 8:11 AM, Roman Dementiev <roman.dementiev@intel.com>
wrote:
>
> This series of patches adds methods that use hardware memory transactions
> (HTM)
> on fast-path for DPDK locks (a.k.a. lock elision). Here the methods are
> implemented
> for x86 using Restricted Transactional Memory instructions (Intel(r)
> Transactional
> Synchronization Extensions). The implementation fall-backs to the normal
> DPDK lock
> if HTM is not available or memory transactions fail.
>
This is very interesting. Do you have any summary you can give us of what
the performance implications are from your test data?
> This is not a replacement for lock usages since not all critical sections
> protected
> by locks are friendly to HTM.
>
Any pointers to material that describe where HTM in its current incarnation
on x86 is approprate?
>
> Roman Dementiev (3):
> spinlock: add support for HTM lock elision for x86
> rwlock: add support for HTM lock elision for x86
> test scaling of HTM lock elision protecting rte_hash
>
> app/test/Makefile | 1 +
> app/test/test_hash_scaling.c | 223
> +++++++++++++++++++++
> lib/librte_eal/common/Makefile | 4 +-
> .../common/include/arch/ppc_64/rte_rwlock.h | 38 ++++
> .../common/include/arch/ppc_64/rte_spinlock.h | 41 ++++
> lib/librte_eal/common/include/arch/x86/rte_rtm.h | 73 +++++++
> .../common/include/arch/x86/rte_rwlock.h | 82 ++++++++
> .../common/include/arch/x86/rte_spinlock.h | 107 ++++++++++
> lib/librte_eal/common/include/generic/rte_rwlock.h | 194
> ++++++++++++++++++
> .../common/include/generic/rte_spinlock.h | 75 +++++++
> lib/librte_eal/common/include/rte_rwlock.h | 158 ---------------
> 11 files changed, 836 insertions(+), 160 deletions(-)
>
Thanks!
Jay
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] add support for HTM lock elision for x86
2015-06-03 18:40 ` Stephen Hemminger
@ 2015-06-05 15:12 ` Roman Dementiev
0 siblings, 0 replies; 6+ messages in thread
From: Roman Dementiev @ 2015-06-05 15:12 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev
Hello Stephen,
Wednesday, June 3, 2015, 8:40:14 PM, you wrote:
> On Tue, 2 Jun 2015 15:11:30 +0200
> Roman Dementiev <roman.dementiev@intel.com> wrote:
>>
>> This series of patches adds methods that use hardware memory transactions (HTM)
>> on fast-path for DPDK locks (a.k.a. lock elision). Here the methods are implemented
>> for x86 using Restricted Transactional Memory instructions (Intel(r) Transactional
>> Synchronization Extensions). The implementation fall-backs to the normal DPDK lock
>> if HTM is not available or memory transactions fail.
>> This is not a replacement for all lock usages since not all critical sections protected
>> by locks are friendly to HTM.
>>
> You probably want to put a caveat around this, it won't work for people
> that expect to use spinlocks to protect I/O operations on hardware.
> Since I/O operations aren't like memory.
yes, I/O can not be rolled back by the CPU if the transaction should fail. Thus
the HTM transaction protecting I/O operations are always aborted by
CPU. In Intel TSX the I/O operations (MMIO, outp, etc) are TSX-unfriendly
causing immediate abort.
--
Best regards,
Roman mailto:roman.dementiev@intel.com
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] add support for HTM lock elision for x86
2015-06-02 13:11 Roman Dementiev
[not found] ` <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
@ 2015-06-03 18:40 ` Stephen Hemminger
2015-06-05 15:12 ` Roman Dementiev
1 sibling, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2015-06-03 18:40 UTC (permalink / raw)
To: Roman Dementiev; +Cc: dev
On Tue, 2 Jun 2015 15:11:30 +0200
Roman Dementiev <roman.dementiev@intel.com> wrote:
>
> This series of patches adds methods that use hardware memory transactions (HTM)
> on fast-path for DPDK locks (a.k.a. lock elision). Here the methods are implemented
> for x86 using Restricted Transactional Memory instructions (Intel(r) Transactional
> Synchronization Extensions). The implementation fall-backs to the normal DPDK lock
> if HTM is not available or memory transactions fail.
> This is not a replacement for lock usages since not all critical sections protected
> by locks are friendly to HTM.
>
> Roman Dementiev (3):
> spinlock: add support for HTM lock elision for x86
> rwlock: add support for HTM lock elision for x86
> test scaling of HTM lock elision protecting rte_hash
>
> app/test/Makefile | 1 +
> app/test/test_hash_scaling.c | 223 +++++++++++++++++++++
> lib/librte_eal/common/Makefile | 4 +-
> .../common/include/arch/ppc_64/rte_rwlock.h | 38 ++++
> .../common/include/arch/ppc_64/rte_spinlock.h | 41 ++++
> lib/librte_eal/common/include/arch/x86/rte_rtm.h | 73 +++++++
> .../common/include/arch/x86/rte_rwlock.h | 82 ++++++++
> .../common/include/arch/x86/rte_spinlock.h | 107 ++++++++++
> lib/librte_eal/common/include/generic/rte_rwlock.h | 194 ++++++++++++++++++
> .../common/include/generic/rte_spinlock.h | 75 +++++++
> lib/librte_eal/common/include/rte_rwlock.h | 158 ---------------
> 11 files changed, 836 insertions(+), 160 deletions(-)
>
> Intel GmbH
> Dornacher Strasse 1
> 85622 Feldkirchen/Muenchen, Deutschland
> Sitz der Gesellschaft: Feldkirchen bei Muenchen
> Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
> Registergericht: Muenchen HRB 47456
> Ust.-IdNr./VAT Registration No.: DE129385895
> Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
>
You probably want to put a caveat around this, it won't work for people
that expect to use spinlocks to protect I/O operations on hardware.
Since I/O operations aren't like memory.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] add support for HTM lock elision for x86
[not found] ` <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
2015-06-02 13:39 ` Dementiev, Roman
@ 2015-06-02 14:55 ` Roman Dementiev
1 sibling, 0 replies; 6+ messages in thread
From: Roman Dementiev @ 2015-06-02 14:55 UTC (permalink / raw)
To: Jay Rolette; +Cc: DPDK
Hello Jay,
Tuesday, June 2, 2015, 3:21:24 PM, you wrote:
> On Tue, Jun 2, 2015 at 8:11 AM, Roman Dementiev <roman.dementiev@intel.com>wrote:
> This series of patches adds methods that use hardware memory transactions (HTM)
> on fast-path for DPDK locks (a.k.a. lock elision). Here the methods are implemented
> for x86 using Restricted Transactional Memory instructions (Intel(r) Transactional
> Synchronization Extensions). The implementation fall-backs to the normal DPDK lock
> if HTM is not available or memory transactions fail.
It provides a very good scaling given the protected data structure is friendly
to HTM. One example is rte_hash which is benchmarked in the unit test I have
provided as the last patch.
There are some papers showing additional test results with similar lock
elision implementations: www.intel.com/software/tsx
> This is very interesting. Do you have any summary you can give us
> of what the performance implications are from your test data?
>
> This is not a replacement for lock usages since not all critical sections protected
> by locks are friendly to HTM.
> Any pointers to material that describe where HTM in its current incarnation on x86 is approprate?
>
>
I meant to say: not a replacement to *ALL* lock usages. You can check the material here: www.intel.com/software/tsx
Please let me know if you need additional information.
> Roman Dementiev (3):
> spinlock: add support for HTM lock elision for x86
> rwlock: add support for HTM lock elision for x86
> test scaling of HTM lock elision protecting rte_hash
>
> app/test/Makefile | 1 +
> app/test/test_hash_scaling.c | 223 +++++++++++++++++++++
> lib/librte_eal/common/Makefile | 4 +-
> .../common/include/arch/ppc_64/rte_rwlock.h | 38 ++++
> .../common/include/arch/ppc_64/rte_spinlock.h | 41 ++++
> lib/librte_eal/common/include/arch/x86/rte_rtm.h | 73 +++++++
> .../common/include/arch/x86/rte_rwlock.h | 82 ++++++++
> .../common/include/arch/x86/rte_spinlock.h | 107 ++++++++++
> lib/librte_eal/common/include/generic/rte_rwlock.h | 194 ++++++++++++++++++
> .../common/include/generic/rte_spinlock.h | 75 +++++++
> lib/librte_eal/common/include/rte_rwlock.h | 158 ---------------
> 11 files changed, 836 insertions(+), 160 deletions(-)
> Thanks!
> Jay
--
Best regards,
Roman mailto:roman.dementiev@intel.com
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] add support for HTM lock elision for x86
[not found] ` <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
@ 2015-06-02 13:39 ` Dementiev, Roman
2015-06-02 14:55 ` Roman Dementiev
1 sibling, 0 replies; 6+ messages in thread
From: Dementiev, Roman @ 2015-06-02 13:39 UTC (permalink / raw)
To: Jay Rolette; +Cc: DPDK
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
From yong.liu@intel.com Tue Jun 2 16:09:11 2015
Return-Path: <yong.liu@intel.com>
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
by dpdk.org (Postfix) with ESMTP id 23E59C334
for <dev@dpdk.org>; Tue, 2 Jun 2015 16:09:09 +0200 (CEST)
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
by fmsmga101.fm.intel.com with ESMTP; 02 Jun 2015 07:08:47 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.13,540,1427785200"; d="scan'208";a="735637182"
Received: from kmsmsx152.gar.corp.intel.com ([172.21.73.87])
by fmsmga002.fm.intel.com with ESMTP; 02 Jun 2015 07:08:45 -0700
Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by
KMSMSX152.gar.corp.intel.com (172.21.73.87) with Microsoft SMTP Server (TLS)
id 14.3.224.2; Tue, 2 Jun 2015 22:08:44 +0800
Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.23]) by
shsmsx102.ccr.corp.intel.com ([169.254.2.109]) with mapi id 14.03.0224.002;
Tue, 2 Jun 2015 22:08:43 +0800
From: "Liu, Yong" <yong.liu@intel.com>
To: "Liang, Cunming" <cunming.liang@intel.com>, "dev@dpdk.org" <dev@dpdk.org>
Thread-Topic: [PATCH v10 00/13] Interrupt mode PMD
Thread-Index: AQHQnQDjRUxaR9oYIk2Xx85CWpKmRJ2ZQSHw
Date: Tue, 2 Jun 2015 14:08:43 +0000
Message-ID: <86228AFD5BCD8E4EBFD2B90117B5E81E10E37490@SHSMSX103.ccr.corp.intel.com>
References: <1432889125-20255-1-git-send-email-cunming.liang@intel.com>
<1433228006-24661-1-git-send-email-cunming.liang@intel.com>
In-Reply-To: <1433228006-24661-1-git-send-email-cunming.liang@intel.com>
Accept-Language: zh-CN, en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.239.127.40]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [dpdk-dev] [PATCH v10 00/13] Interrupt mode PMD
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 02 Jun 2015 14:09:11 -0000
Tested-by: Yong Liu <yong.liu@intel.com>
- Tested Commit: 7c4c66bf666b8059ed0ad2f2478ef349b3272f51
- OS: Fedora20 3.15.5
- GCC: gcc version 4.8.3 20140911
- CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- NIC: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ [8086:10fb]
- NIC: Intel Corporation I350 Gigabit Network Connection [8086:1521]
- Default x86_64-native-linuxapp-gcc configuration
- Prerequisites: vfio related case request vt-d enable in bios and IOMMU enable in kernel
- Total 17 cases, 17 passed, 0 failed
- Case: pf_lsc_igbuio_legacy
Description: check when pf bound to igb_uio with legacy mode, link status change interrupt can be normally handled
Command / instruction:
Insmod igb_uio driver with legacy interrupt mode
insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=legacy
Change port config to lsc enable and rxq disable in l3fwd-power/main.c
Build l3fwd-power and start l3fwd-power with 2 ports
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Change tester port0 link down and verify link down detected on dut port0
Port 0: link down
Change tester port0 link up and verify link up detected on dut port0
Port 0: link up
Change tester port1 link down and verify link down detected on dut port1
Port 1: link down
Change tester port1 link up and verify link up detected on dut port1
Port 1: link up
Change port config to lsc enable and rxq enable in l3fwd-power/main.c
Build l3fwd-power and start l3fwd-power with 2 ports
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify lsc disabled for can't enable lsc and rxq in the same time when pf bound to igb_uio
lsc won't enable because of no intr multiplex
- Case: pf_lsc_igbuio_msix
Description: check when pf bound to igb_uio with msix mode, link status change interrupt can be normally handled
Command / instruction:
Insmod igb_uio driver with msix interrupt mode
insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=msix
Verify link status can be normally handled like previous case pf_lsc_igbuio_legacy.
- Case: pf_lsc_vfio_legacy
Description: check when pf bound to vfio with legacy mode, link status change interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Change port config to lsc enable and rxq disable in l3fwd-power/main.c
Start l3fwd-power with vfio legacy mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Check link status change interrupt can be normally handled like previous case.
Change port config to lsc enable and rxq enable in l3fwd-power/main.c
Start l3fwd-power with vfio legacy mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify lsc disabled for can't enable lsc and rxq in the same time with legacy mode.
- Case: pf_lsc_vfio_msi
Description: check when pf bound to vfio with msi mode, link status change interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Change port config to lsc enable and rxq disable in l3fwd-power/main.c
Start l3fwd-power with vfio msi mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Check link status change interrupt can be normally handled like previous case.
Change port config to lsc enable and rxq enable in l3fwd-power/main.c
Start l3fwd-power with vfio msi mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify lsc disabled for can't enable lsc and rxq in the same time with legacy mode.
- Case: pf_lsc_vfio_msix
Description: check when pf bound to vfio with msix mode, link status change interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Change port config to lsc enable and rxq disable in l3fwd-power/main.c
Start l3fwd-power with vfio msix mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=msix -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Check link status change interrupt can be normally handled like previous case.
Change port config to lsc enable and rxq enable in l3fwd-power/main.c
Start l3fwd-power with vfio msix mode
l3fwd-power -c 0x6 -n 3 --vfio-intr=msix -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Check link status change interrupt can be normally handled like previous case.
- Case: pf_rxq_on_vfio_msix
Description: check when pf bound to vfio with default msix mode, receive packet interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Send packet from tester port0 and verify dut core1 wakeup and then sleep.
lcore 1 is waked up from rx interrupt on port 0 queue 0
lcore 1 sleeps until interrupt triggers
Send packet from tester port1 and verify dut core2 wakeup and then sleep.
lcore 2 is waked up from rx interrupt on port 1 queue 0
lcore 2 sleeps until interrupt triggers
- Case: pf_rxq_on_vfio_msi
Description: check when pf bound to vfio with msi mode, receive packet interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 --vfio-intr=msi -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_rxq_on_vfio_legacy
Description: check when pf bound to vfio with legacy mode, receive packet interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 --vfio-intr=legacy -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_onecore_on_vfio
Description: check when all pf devices bound to one core, receive packet interrupt can be normally handled
Command / instruction:
Do prerequisites for vfio driver then bind device to vfio-driver
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 1 cores.
l3fwd-power -c 0x2 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,1)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_multiqueue_on_vfio
Description: check when pf device has mulit queues, receive packet interrupt can be normally handled
Command / instruction:
Start l3fwd-power with 2 ports and 4 cores.
l3fwd-power -c 0x100000e -n 3 -- -p 0x3 -P --config="(0,0,1),(0,1,2),(1,0,3),(1,1,24)"
Send enough packets with different destination ip address.
sendp([Ether()/IP(dst="127.0.0.X")/UDP()/Raw('0'*18)], iface="p786p1")
Verify all cores wakeup and then sleep as expected.
- Case: pf_maxqueue_on_vfio
Description: check when pf device has maximum queues, receive packet interrupt can be normally handled
Command / instruction:
Start l3fwd-power with 2 ports and 32 cores [only for niantic], different nic has different maximum rx queues
l3fwd-power -c 0x3fdfe3fdfe -n 3 -- -p 0x3 -P --config="(0,0,1),(0,1,21),(0,2,2),(0,3,22),\
(0,4,3),(0,5,23),(0,6,24),(0,7,4),(0,8,25),(0,9,5),(0,10,26),(0,11,6),(0,12,27),(0,13,7),\
(0,14,8),(0,15,28),(1,0,10),(1,1,30),(1,2,11),(1,3,31),(1,4,32),(1,5,12),(1,6,33),(1,7,13),\
(1,8,34),(1,9,14),(1,10,35),(1,11,15),(1,12,16),(1,13,36),(1,14,17),(1,15,37),"
Send enough packets with different destination ip address.
sendp([Ether()/IP(dst="127.0.0.X")/UDP()/Raw('0'*18)], iface="p786p1")
Verify all cores wakeup and then sleep as expected.
- Case: pf_rxq_on_igbuio_legacy
Description: check when pf bound to igb_uio with legacy mode, receive packet interrupt can be normally handled
Command / instruction:
Insmod igb_uio driver with legacy interrupt mode
insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=legacy
./tools/dpdk_nic_bind.py --bind=igb_uio 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_rxq_on_igbuio_msix
Description: check when pf bound to igb_uio with msix mode, receive packet interrupt can be normally handled
Command / instruction:
Insmod igb_uio driver with msix interrupt mode
insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko intr_mode=msix
./tools/dpdk_nic_bind.py --bind=igb_uio 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_rxq_on_uiopcigeneric
Description: check when pf bound to uio_pci_generic, receive packet interrupt can be normally handled
Command / instruction:
Insmod uio_pci_generic driver and bind pf device on it.
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify packet interrupt can be normally handled like previous case pf_rxq_on_vfio_msix.
- Case: pf_lsc_on_uiopcigeneric
Description: check when pf bound to uio_pci_generic, link status changed interrupt can be normally handled
Command / instruction:
Insmod uio_pci_generic driver and bind pf device on it.
./tools/dpdk_nic_bind.py --bind=uio_pci_generic 08:00.0 08:00.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Change tester port0 link down and verify link down detected on dut port0
Port 0: link down
Change tester port0 link up and verify link up detected on dut port0
Port 0: link up
Change tester port1 link down and verify link down detected on dut port1
Port 1: link down
Change tester port1 link up and verify link up detected on dut port1
Port 1: link up
Change port config to lsc enable and rxq enable in l3fwd-power/main.c
Build l3fwd-power and start l3fwd-power with 2 ports
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Verify lsc disabled for can't enable lsc and rxq in the same time when pf bound to uio_pci_generic
lsc won't enable because of no intr multiplex
- Case: vf_in_vm_rxq
Description: check when vf bound to igb_uio in virtual machine, receive packet interrupt can be normally handled
Only support niantic by now.
Command / instruction:
Create vf devices and bound into virtual machine
echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs
virsh
virsh # nodedev-dettach pci_0000_08_10_0
virsh # nodedev-dettach pci_0000_08_10_1
Start virtual machine and bind vf devices to driver igb_uio.
./tools/dpdk_nic_bind.py --bind=igb_uio eth1 eth2
Change port config to lsc disable and rxq enable in l3fwd-power/main.c
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Send packet from tester port0 with promisc mac and verify vm core1 wakeup and then sleep.
lcore 1 is waked up from rx interrupt on port 0 queue 0
lcore 1 sleeps until interrupt triggers
Send packet from tester port1 with promisc mac and verify vm core2 wakeup and then sleep.
lcore 2 is waked up from rx interrupt on port 1 queue 0
lcore 2 sleeps until interrupt triggers
- Case: vf_in_host_rxq
Description: check when vf bound to vfio with msix mode, receive packet interrupt can be normally handled
Only support niantic by now.
Command / instruction:
Create vf devices and bound to vfio
echo 1 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs
echo 1 > /sys/bus/pci/devices/0000\:08\:00.1/sriov_numvfs
modprobe vfio
modprobe vfio-pci
./tools/dpdk_nic_bind.py --bind=vfio-pci 08:10.0 08:10.1
Start l3fwd-power with 2 ports and 2 cores.
l3fwd-power -c 0x6 -n 3 -- -p 0x3 -P --config="(0,0,1),(1,0,2)"
Send packet from tester port0 with promisc mac and verify dut core1 wakeup and then sleep.
lcore 1 is waked up from rx interrupt on port 0 queue 0
lcore 1 sleeps until interrupt triggers
Send packet from tester port1 with promisc mac and verify dut core2 wakeup and then sleep.
lcore 2 is waked up from rx interrupt on port 1 queue 0
lcore 2 sleeps until interrupt triggers
> -----Original Message-----
> From: Liang, Cunming
> Sent: Tuesday, June 02, 2015 2:53 PM
> To: dev@dpdk.org
> Cc: shemming@brocade.com; david.marchand@6wind.com;
> thomas.monjalon@6wind.com; Zhou, Danny; Wang, Liang-min; Richardson, Bruce;
> Liu, Yong; nhorman@tuxdriver.com; Liang, Cunming
> Subject: [PATCH v10 00/13] Interrupt mode PMD
>
> v10 changes
> - code rework to return actual error code
> - bug fix for lsc when using uio_pci_generic
>
> v9 changes
> - code rework to fix open comment
> - bug fix for igb lsc when both lsc and rxq are enabled in vfio-msix
> - new patch to turn off the feature by defalut so as to avoid v2.1 abi
> broken
>
> v8 changes
> - remove condition check for only vfio-msix
> - add multiplex intr support when only one intr vector allowed
> - lsc and rxq interrupt runtime enable decision
> - add safe event delete while the event wakeup execution happens
>
> v7 changes
> - decouple epoll event and intr operation
> - add condition check in the case intr vector is disabled
> - renaming some APIs
>
> v6 changes
> - split rte_intr_wait_rx_pkt into two APIs 'wait' and 'set'.
> - rewrite rte_intr_rx_wait/rte_intr_rx_set.
> - using vector number instead of queue_id as interrupt API params.
> - patch reorder and split.
>
> v5 changes
> - Rebase the patchset onto the HEAD
> - Isolate ethdev from EAL for new-added wait-for-rx interrupt function
> - Export wait-for-rx interrupt function for shared libraries
> - Split-off a new patch file for changed struct rte_intr_handle that
> other patches depend on, to avoid breaking git bisect
> - Change sample applicaiton to accomodate EAL function spec change
> accordingly
>
> v4 changes
> - Export interrupt enable/disable functions for shared libraries
> - Adjust position of new-added structure fields and functions to
> avoid breaking ABI
>
> v3 changes
> - Add return value for interrupt enable/disable functions
> - Move spinlok from PMD to L3fwd-power
> - Remove unnecessary variables in e1000_mac_info
> - Fix miscelleous review comments
>
> v2 changes
> - Fix compilation issue in Makefile for missed header file.
> - Consolidate internal and community review comments of v1 patch set.
>
> The patch series introduce low-latency one-shot rx interrupt into DPDK
> with
> polling and interrupt mode switch control example.
>
> DPDK userspace interrupt notification and handling mechanism is based on
> UIO
> with below limitation:
> 1) It is designed to handle LSC interrupt only with inefficient suspended
> pthread wakeup procedure (e.g. UIO wakes up LSC interrupt handling
> thread
> which then wakes up DPDK polling thread). In this way, it introduces
> non-deterministic wakeup latency for DPDK polling thread as well as
> packet
> latency if it is used to handle Rx interrupt.
> 2) UIO only supports a single interrupt vector which has to been shared by
> LSC interrupt and interrupts assigned to dedicated rx queues.
>
> This patchset includes below features:
> 1) Enable one-shot rx queue interrupt in ixgbe PMD(PF & VF) and igb PMD(PF
> only).
> 2) Build on top of the VFIO mechanism instead of UIO, so it could support
> up to 64 interrupt vectors for rx queue interrupts.
> 3) Have 1 DPDK polling thread handle per Rx queue interrupt with a
> dedicated
> VFIO eventfd, which eliminates non-deterministic pthread wakeup latency
> in
> user space.
> 4) Demonstrate interrupts control APIs and userspace NAIP-like
> polling/interrupt
> switch algorithms in L3fwd-power example.
>
> Known limitations:
> 1) It does not work for UIO due to a single interrupt eventfd shared by
> LSC
> and rx queue interrupt handlers causes a mess. [FIXED]
> 2) LSC interrupt is not supported by VF driver, so it is by default
> disabled
> in L3fwd-power now. Feel free to turn in on if you want to support both
> LSC
> and rx queue interrupts on a PF.
>
> Cunming Liang (13):
> eal/linux: add interrupt vectors support in intr_handle
> eal/linux: add rte_epoll_wait/ctl support
> eal/linux: add API to set rx interrupt event monitor
> eal/linux: fix comments typo on vfio msi
> eal/linux: add interrupt vectors handling on VFIO
> eal/linux: standalone intr event fd create support
> eal/linux: fix lsc read error in uio_pci_generic
> eal/bsd: dummy for new intr definition
> ethdev: add rx intr enable, disable and ctl functions
> ixgbe: enable rx queue interrupts for both PF and VF
> igb: enable rx queue interrupts for PF
> l3fwd-power: enable one-shot rx interrupt and polling/interrupt mode
> switch
> abi: fix v2.1 abi broken issue
>
> drivers/net/e1000/igb_ethdev.c | 311 ++++++++++--
> drivers/net/ixgbe/ixgbe_ethdev.c | 519
> ++++++++++++++++++++-
> drivers/net/ixgbe/ixgbe_ethdev.h | 4 +
> examples/l3fwd-power/main.c | 206 ++++++--
> lib/librte_eal/bsdapp/eal/eal_interrupts.c | 19 +
> .../bsdapp/eal/include/exec-env/rte_interrupts.h | 81 ++++
> lib/librte_eal/bsdapp/eal/rte_eal_version.map | 5 +
> lib/librte_eal/linuxapp/eal/eal_interrupts.c | 360 ++++++++++++--
> .../linuxapp/eal/include/exec-env/rte_interrupts.h | 219 +++++++++
> lib/librte_eal/linuxapp/eal/rte_eal_version.map | 8 +
> lib/librte_ether/rte_ethdev.c | 109 +++++
> lib/librte_ether/rte_ethdev.h | 132 ++++++
> lib/librte_ether/rte_ether_version.map | 4 +
> 13 files changed, 1852 insertions(+), 125 deletions(-)
>
> --
> 1.8.1.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* [dpdk-dev] add support for HTM lock elision for x86
@ 2015-06-02 13:11 Roman Dementiev
[not found] ` <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
2015-06-03 18:40 ` Stephen Hemminger
0 siblings, 2 replies; 6+ messages in thread
From: Roman Dementiev @ 2015-06-02 13:11 UTC (permalink / raw)
To: dev
This series of patches adds methods that use hardware memory transactions (HTM)
on fast-path for DPDK locks (a.k.a. lock elision). Here the methods are implemented
for x86 using Restricted Transactional Memory instructions (Intel(r) Transactional
Synchronization Extensions). The implementation fall-backs to the normal DPDK lock
if HTM is not available or memory transactions fail.
This is not a replacement for lock usages since not all critical sections protected
by locks are friendly to HTM.
Roman Dementiev (3):
spinlock: add support for HTM lock elision for x86
rwlock: add support for HTM lock elision for x86
test scaling of HTM lock elision protecting rte_hash
app/test/Makefile | 1 +
app/test/test_hash_scaling.c | 223 +++++++++++++++++++++
lib/librte_eal/common/Makefile | 4 +-
.../common/include/arch/ppc_64/rte_rwlock.h | 38 ++++
.../common/include/arch/ppc_64/rte_spinlock.h | 41 ++++
lib/librte_eal/common/include/arch/x86/rte_rtm.h | 73 +++++++
.../common/include/arch/x86/rte_rwlock.h | 82 ++++++++
.../common/include/arch/x86/rte_spinlock.h | 107 ++++++++++
lib/librte_eal/common/include/generic/rte_rwlock.h | 194 ++++++++++++++++++
.../common/include/generic/rte_spinlock.h | 75 +++++++
lib/librte_eal/common/include/rte_rwlock.h | 158 ---------------
11 files changed, 836 insertions(+), 160 deletions(-)
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen, Deutschland
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk
Registergericht: Muenchen HRB 47456
Ust.-IdNr./VAT Registration No.: DE129385895
Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-06-05 15:12 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-02 13:17 [dpdk-dev] add support for HTM lock elision for x86 Dementiev, Roman
-- strict thread matches above, loose matches on Subject: below --
2015-06-02 13:11 Roman Dementiev
[not found] ` <CADNuJVpeKa9-R7WHkoCzw82vpYd=3XmhOoz2JfGsFLzDW+F5UQ@mail.gmail.com>
2015-06-02 13:39 ` Dementiev, Roman
2015-06-02 14:55 ` Roman Dementiev
2015-06-03 18:40 ` Stephen Hemminger
2015-06-05 15:12 ` Roman Dementiev
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).