From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 002B2A2E1B for ; Tue, 3 Sep 2019 13:00:28 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1B40F1EAC9; Tue, 3 Sep 2019 12:59:56 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id B80A81EAAE for ; Tue, 3 Sep 2019 12:59:54 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x83Axbc0028707; Tue, 3 Sep 2019 03:59:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0818; bh=Ftl/T2o6tpaXrycBCCZvO3rJJxgOct4UO/MCDxHNol8=; b=SIxWNmj2kiyYHPFqH47DfhEHIAW7vHqankMVnupB+dOSPfPLh7UHwlj42y1YYmQ/bOI2 0u/LPtF7LKPecITpaeY/qpuBfpk1LM857L78reItGLYVTg/Ih/1ZQLbTxyJTp2CwtuJJ Bl9B23pJ3+WQsKBeQH7ArqmZy1aeJ64+Ap7/fDl7e42UBIKLIcNO3cTA09QtYiGmVL44 IskOscXRRzBVij4qDmmvD9WF1BgnrvG+fzTFs0IEK16/nQryraQGTOuySQhwb5bvl5GM PZAFlO2bPMXfBjO16l1aj2PA1Tp8u7jvZbTHH21icsLDnAfBwXlxnT+2aWpFmmiuMxQZ yw== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0b-0016f401.pphosted.com with ESMTP id 2uqrdm958g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 03 Sep 2019 03:59:50 -0700 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Tue, 3 Sep 2019 03:59:48 -0700 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Tue, 3 Sep 2019 03:59:47 -0700 Received: from jerin-lab.marvell.com (jerin-lab.marvell.com [10.28.34.14]) by maili.marvell.com (Postfix) with ESMTP id 8CBCA3F7041; Tue, 3 Sep 2019 03:59:43 -0700 (PDT) From: To: CC: , , , , Jerin Jacob Date: Tue, 3 Sep 2019 16:29:37 +0530 Message-ID: <20190903105938.33231-8-jerinj@marvell.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190903105938.33231-1-jerinj@marvell.com> References: <20190903105938.33231-1-jerinj@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-09-03_01:2019-09-03,2019-09-03 signatures=0 Subject: [dpdk-dev] [PATCH 7/8] bpf/arm64: add atomic-exchange-and-add operation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Jerin Jacob Implement XADD eBPF instruction using STADD arm64 instruction. If the given platform does not have atomics support, use LDXR and STXR pair for critical section instead of STADD. Signed-off-by: Jerin Jacob --- lib/librte_bpf/bpf_jit_arm64.c | 85 +++++++++++++++++++++++++++++++++- 1 file changed, 84 insertions(+), 1 deletion(-) diff --git a/lib/librte_bpf/bpf_jit_arm64.c b/lib/librte_bpf/bpf_jit_arm64.c index c797c9c62..62fa6a505 100644 --- a/lib/librte_bpf/bpf_jit_arm64.c +++ b/lib/librte_bpf/bpf_jit_arm64.c @@ -837,6 +837,83 @@ emit_return_zero_if_src_zero(struct a64_jit_ctx *ctx, bool is64, uint8_t src) emit_b(ctx, jump_to_epilogue); } +static void +emit_stadd(struct a64_jit_ctx *ctx, bool is64, uint8_t rs, uint8_t rn) +{ + uint32_t insn; + + insn = 0xb820001f; + insn |= (!!is64) << 30; + insn |= rs << 16; + insn |= rn << 5; + + emit_insn(ctx, insn, check_reg(rs) || check_reg(rn)); +} + +static void +emit_ldxr(struct a64_jit_ctx *ctx, bool is64, uint8_t rt, uint8_t rn) +{ + uint32_t insn; + + insn = 0x885f7c00; + insn |= (!!is64) << 30; + insn |= rn << 5; + insn |= rt; + + emit_insn(ctx, insn, check_reg(rt) || check_reg(rn)); +} + +static void +emit_stxr(struct a64_jit_ctx *ctx, bool is64, uint8_t rs, uint8_t rt, + uint8_t rn) +{ + uint32_t insn; + + insn = 0x88007c00; + insn |= (!!is64) << 30; + insn |= rs << 16; + insn |= rn << 5; + insn |= rt; + + emit_insn(ctx, insn, check_reg(rs) || check_reg(rt) || check_reg(rn)); +} + +static int +has_atomics(void) +{ + int rc = 0; + +#if defined(__ARM_FEATURE_ATOMICS) || defined(RTE_ARM_FEATURE_ATOMICS) + rc = 1; +#endif + return rc; +} + +static void +emit_xadd(struct a64_jit_ctx *ctx, uint8_t op, uint8_t tmp1, uint8_t tmp2, + uint8_t tmp3, uint8_t dst, int16_t off, uint8_t src) +{ + bool is64 = (BPF_SIZE(op) == EBPF_DW); + uint8_t rn; + + if (off) { + emit_mov_imm(ctx, 1, tmp1, off); + emit_add(ctx, 1, tmp1, dst); + rn = tmp1; + } else { + rn = dst; + } + + if (has_atomics()) { + emit_stadd(ctx, is64, src, rn); + } else { + emit_ldxr(ctx, is64, tmp2, rn); + emit_add(ctx, is64, tmp2, src); + emit_stxr(ctx, is64, tmp3, tmp2, rn); + emit_cbnz(ctx, is64, tmp3, -3); + } +} + static void check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) { @@ -863,7 +940,7 @@ check_program_has_call(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) static int emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) { - uint8_t op, dst, src, tmp1, tmp2; + uint8_t op, dst, src, tmp1, tmp2, tmp3; const struct ebpf_insn *ins; uint64_t u64; int16_t off; @@ -878,6 +955,7 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) ctx->stack_sz = RTE_ALIGN_MUL_CEIL(bpf->stack_sz, 16); tmp1 = ebpf_to_a64_reg(ctx, TMP_REG_1); tmp2 = ebpf_to_a64_reg(ctx, TMP_REG_2); + tmp3 = ebpf_to_a64_reg(ctx, TMP_REG_3); emit_prologue(ctx); @@ -1067,6 +1145,11 @@ emit(struct a64_jit_ctx *ctx, struct rte_bpf *bpf) emit_mov_imm(ctx, 1, tmp2, off); emit_str(ctx, BPF_SIZE(op), tmp1, dst, tmp2); break; + /* STX XADD: lock *(size *)(dst + off) += src */ + case (BPF_STX | EBPF_XADD | BPF_W): + case (BPF_STX | EBPF_XADD | EBPF_DW): + emit_xadd(ctx, op, tmp1, tmp2, tmp3, dst, off, src); + break; /* Return r0 */ case (BPF_JMP | EBPF_EXIT): emit_epilogue(ctx); -- 2.23.0