* [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions
@ 2020-05-18 15:52 Konstantin Ananyev
2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev
` (5 more replies)
0 siblings, 6 replies; 17+ messages in thread
From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw)
To: dev; +Cc: stephen, jerinj, Konstantin Ananyev
To fill the gap with linux kernel eBPF implementation,
add support for two non-generic instructions:
(BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD)
which are used to access packet data.
Make necessary changes in BPF verifier, interpreter and x86 JIT code.
Konstantin Ananyev (5):
test/bpf: fix few small issues
bpf: fix add/sub min/max estimations
bpf: add support for packet data load instructions
test/bpf: add new test cases for mbuf load instructions
bpf: x86 JIT support for packet data load instructions
app/test/test_bpf.c | 504 ++++++++++++++++++++++++++++--
doc/guides/prog_guide/bpf_lib.rst | 30 +-
lib/librte_bpf/bpf_exec.c | 57 ++++
lib/librte_bpf/bpf_jit_x86.c | 181 +++++++++++
lib/librte_bpf/bpf_validate.c | 104 +++++-
5 files changed, 843 insertions(+), 33 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 17+ messages in thread* [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev @ 2020-05-18 15:52 ` Konstantin Ananyev 2020-06-24 21:33 ` [dpdk-dev] [dpdk-stable] " Thomas Monjalon 2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev ` (4 subsequent siblings) 5 siblings, 1 reply; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw) To: dev; +Cc: stephen, jerinj, Konstantin Ananyev, stable Address for few small issues: - unreachable return statement - failed test-case can finish with 'success' status Also use unified cmp_res() function to check return value. Fixes: a9de470cc7c0 ("test: move to app directory") Cc: stable@dpdk.org Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- app/test/test_bpf.c | 41 +++++++++++++++-------------------------- 1 file changed, 15 insertions(+), 26 deletions(-) diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c index ee534687a..4a61a7d7c 100644 --- a/app/test/test_bpf.c +++ b/app/test/test_bpf.c @@ -1797,13 +1797,6 @@ test_call1_check(uint64_t rc, const void *arg) dummy_func1(arg, &v32, &v64); v64 += v32; - if (v64 != rc) { - printf("%s@%d: invalid return value " - "expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n", - __func__, __LINE__, v64, rc); - return -1; - } - return 0; return cmp_res(__func__, v64, rc, dv, dv, sizeof(*dv)); } @@ -1934,13 +1927,7 @@ test_call2_check(uint64_t rc, const void *arg) dummy_func2(&a, &b); v = a.u64 + a.u32 + b.u16 + b.u8; - if (v != rc) { - printf("%s@%d: invalid return value " - "expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n", - __func__, __LINE__, v, rc); - return -1; - } - return 0; + return cmp_res(__func__, v, rc, arg, arg, 0); } static const struct rte_bpf_xsym test_call2_xsym[] = { @@ -2429,7 +2416,6 @@ test_call5_check(uint64_t rc, const void *arg) v = 0; fail: - return cmp_res(__func__, v, rc, &v, &rc, sizeof(v)); } @@ -2458,6 +2444,7 @@ static const struct rte_bpf_xsym test_call5_xsym[] = { }, }; +/* all bpf test cases */ static const struct bpf_test tests[] = { { .name = "test_store1", @@ -2738,7 +2725,6 @@ run_test(const struct bpf_test *tst) } tst->prepare(tbuf); - rc = rte_bpf_exec(bpf, tbuf); ret = tst->check_result(rc, tbuf); if (ret != 0) { @@ -2746,17 +2732,20 @@ run_test(const struct bpf_test *tst) __func__, __LINE__, tst->name, ret, strerror(ret)); } + /* repeat the same test with jit, when possible */ rte_bpf_get_jit(bpf, &jit); - if (jit.func == NULL) - return 0; - - tst->prepare(tbuf); - rc = jit.func(tbuf); - rv = tst->check_result(rc, tbuf); - ret |= rv; - if (rv != 0) { - printf("%s@%d: check_result(%s) failed, error: %d(%s);\n", - __func__, __LINE__, tst->name, rv, strerror(ret)); + if (jit.func != NULL) { + + tst->prepare(tbuf); + rc = jit.func(tbuf); + rv = tst->check_result(rc, tbuf); + ret |= rv; + if (rv != 0) { + printf("%s@%d: check_result(%s) failed, " + "error: %d(%s);\n", + __func__, __LINE__, tst->name, + rv, strerror(ret)); + } } rte_bpf_destroy(bpf); -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [dpdk-stable] [PATCH 1/5] test/bpf: fix few small issues 2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev @ 2020-06-24 21:33 ` Thomas Monjalon 0 siblings, 0 replies; 17+ messages in thread From: Thomas Monjalon @ 2020-06-24 21:33 UTC (permalink / raw) To: Konstantin Ananyev; +Cc: dev, stable, stephen, jerinj 18/05/2020 17:52, Konstantin Ananyev: > Address for few small issues: > - unreachable return statement > - failed test-case can finish with 'success' status > > Also use unified cmp_res() function to check return value. > > Fixes: a9de470cc7c0 ("test: move to app directory") Looks like you don't want to track the original commits. In this case, better to not add any fake reference. > Cc: stable@dpdk.org > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev @ 2020-05-18 15:52 ` Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev ` (3 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw) To: dev; +Cc: stephen, jerinj, Konstantin Ananyev, stable eval_add()/eval_sub() not always correctly estimate minimum and maximum possible values of add/sub operations. Fixes: 8021917293d0 ("bpf: add extra validation for input BPF program") Cc: stable@dpdk.org Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- lib/librte_bpf/bpf_validate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index 6bd6f78e9..80d21fabb 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -226,7 +226,7 @@ eval_add(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk) struct bpf_reg_val rv; rv.u.min = (rd->u.min + rs->u.min) & msk; - rv.u.max = (rd->u.min + rs->u.max) & msk; + rv.u.max = (rd->u.max + rs->u.max) & msk; rv.s.min = (rd->s.min + rs->s.min) & msk; rv.s.max = (rd->s.max + rs->s.max) & msk; @@ -254,10 +254,10 @@ eval_sub(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk) { struct bpf_reg_val rv; - rv.u.min = (rd->u.min - rs->u.min) & msk; - rv.u.max = (rd->u.min - rs->u.max) & msk; - rv.s.min = (rd->s.min - rs->s.min) & msk; - rv.s.max = (rd->s.max - rs->s.max) & msk; + rv.u.min = (rd->u.min - rs->u.max) & msk; + rv.u.max = (rd->u.max - rs->u.min) & msk; + rv.s.min = (rd->s.min - rs->s.max) & msk; + rv.s.max = (rd->s.max - rs->s.min) & msk; /* * if at least one of the operands is not constant, -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev @ 2020-05-18 15:52 ` Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev ` (2 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw) To: dev; +Cc: stephen, jerinj, Konstantin Ananyev To fill the gap with linux kernel eBPF implementation, add support for two non-generic instructions: (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) which are used to access packet data. These instructions can only be used when BPF context is a pointer to 'struct rte_mbuf' (i.e: RTE_BPF_ARG_PTR_MBUF type). Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- doc/guides/prog_guide/bpf_lib.rst | 30 +++++++++++- lib/librte_bpf/bpf_exec.c | 57 +++++++++++++++++++++++ lib/librte_bpf/bpf_validate.c | 77 +++++++++++++++++++++++++++++++ 3 files changed, 163 insertions(+), 1 deletion(-) diff --git a/doc/guides/prog_guide/bpf_lib.rst b/doc/guides/prog_guide/bpf_lib.rst index 9c728da7b..1feb7734a 100644 --- a/doc/guides/prog_guide/bpf_lib.rst +++ b/doc/guides/prog_guide/bpf_lib.rst @@ -27,6 +27,35 @@ The library API provides the following basic operations: * Load BPF program from the ELF file and install callback to execute it on given ethdev port/queue. +Packet data load instructions +----------------------------- + +DPDK supports two non-generic instructions: ``(BPF_ABS | size | BPF_LD)`` +and ``(BPF_IND | size | BPF_LD)`` which are used to access packet data. +These instructions can only be used when execution context is a pointer to +``struct rte_mbuf`` and have seven implicit operands. +Register ``R6`` is an implicit input that must contain pointer to ``rte_mbuf``. +Register ``R0`` is an implicit output which contains the data fetched from the +packet. Registers ``R1-R5`` are scratch registers +and must not be used to store the data across these instructions. +These instructions have implicit program exit condition as well. When +eBPF program is trying to access the data beyond the packet boundary, +the interpreter will abort the execution of the program. JIT compilers +therefore must preserve this property. ``src_reg`` and ``imm32`` fields are +explicit inputs to these instructions. +For example, ``(BPF_IND | BPF_W | BPF_LD)`` means: + +.. code-block:: c + + uint32_t tmp; + R0 = rte_pktmbuf_read((const struct rte_mbuf *)R6, src_reg + imm32, + sizeof(tmp), &tmp); + if (R0 == NULL) return FAILED; + R0 = ntohl(*(uint32_t *)R0); + +and ``R1-R5`` were scratched. + + Not currently supported eBPF features ------------------------------------- @@ -34,5 +63,4 @@ Not currently supported eBPF features - cBPF - tail-pointer call - eBPF MAP - - skb - external function calls for 32-bit platforms diff --git a/lib/librte_bpf/bpf_exec.c b/lib/librte_bpf/bpf_exec.c index 1bb226643..b921112fe 100644 --- a/lib/librte_bpf/bpf_exec.c +++ b/lib/librte_bpf/bpf_exec.c @@ -74,6 +74,26 @@ (uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \ reg[ins->src_reg])) +/* BPF_LD | BPF_ABS/BPF_IND */ + +#define NOP(x) (x) + +#define BPF_LD_ABS(bpf, reg, ins, type, op) do { \ + const type *p = bpf_ld_mbuf(bpf, reg, ins, (ins)->imm, sizeof(type)); \ + if (p == NULL) \ + return 0; \ + reg[EBPF_REG_0] = op(p[0]); \ +} while (0) + +#define BPF_LD_IND(bpf, reg, ins, type, op) do { \ + uint32_t ofs = reg[ins->src_reg] + (ins)->imm; \ + const type *p = bpf_ld_mbuf(bpf, reg, ins, ofs, sizeof(type)); \ + if (p == NULL) \ + return 0; \ + reg[EBPF_REG_0] = op(p[0]); \ +} while (0) + + static inline void bpf_alu_be(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins) { @@ -112,6 +132,23 @@ bpf_alu_le(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins) } } +static inline const void * +bpf_ld_mbuf(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM], + const struct ebpf_insn *ins, uint32_t off, uint32_t len) +{ + const struct rte_mbuf *mb; + const void *p; + + mb = (const struct rte_mbuf *)(uintptr_t)reg[EBPF_REG_6]; + p = rte_pktmbuf_read(mb, off, len, reg + EBPF_REG_0); + if (p == NULL) + RTE_BPF_LOG(DEBUG, "%s(bpf=%p, mbuf=%p, ofs=%u, len=%u): " + "load beyond packet boundary at pc: %#zx;\n", + __func__, bpf, mb, off, len, + (uintptr_t)(ins) - (uintptr_t)(bpf)->prm.ins); + return p; +} + static inline uint64_t bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM]) { @@ -296,6 +333,26 @@ bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM]) (uint64_t)(uint32_t)ins[1].imm << 32; ins++; break; + /* load absolute instructions */ + case (BPF_LD | BPF_ABS | BPF_B): + BPF_LD_ABS(bpf, reg, ins, uint8_t, NOP); + break; + case (BPF_LD | BPF_ABS | BPF_H): + BPF_LD_ABS(bpf, reg, ins, uint16_t, rte_be_to_cpu_16); + break; + case (BPF_LD | BPF_ABS | BPF_W): + BPF_LD_ABS(bpf, reg, ins, uint32_t, rte_be_to_cpu_32); + break; + /* load indirect instructions */ + case (BPF_LD | BPF_IND | BPF_B): + BPF_LD_IND(bpf, reg, ins, uint8_t, NOP); + break; + case (BPF_LD | BPF_IND | BPF_H): + BPF_LD_IND(bpf, reg, ins, uint16_t, rte_be_to_cpu_16); + break; + case (BPF_LD | BPF_IND | BPF_W): + BPF_LD_IND(bpf, reg, ins, uint32_t, rte_be_to_cpu_32); + break; /* store instructions */ case (BPF_STX | BPF_MEM | BPF_B): BPF_ST_REG(reg, ins, uint8_t); diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index 80d21fabb..fecdda0e1 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -102,6 +102,9 @@ struct bpf_ins_check { #define WRT_REGS RTE_LEN2MASK(EBPF_REG_10, uint16_t) #define ZERO_REG RTE_LEN2MASK(EBPF_REG_1, uint16_t) +/* For LD_IND R6 is an implicit CTX register. */ +#define IND_SRC_REGS (WRT_REGS ^ 1 << EBPF_REG_6) + /* * check and evaluate functions for particular instruction types. */ @@ -580,6 +583,42 @@ eval_neg(struct bpf_reg_val *rd, size_t opsz, uint64_t msk) rd->s.min = RTE_MIN(sx, sy); } +static const char * +eval_ld_mbuf(struct bpf_verifier *bvf, const struct ebpf_insn *ins) +{ + uint32_t i, mode; + struct bpf_reg_val *rv, ri, rs; + + mode = BPF_MODE(ins->code); + + /* R6 is an implicit input that must contain pointer to mbuf */ + if (bvf->evst->rv[EBPF_REG_6].v.type != RTE_BPF_ARG_PTR_MBUF) + return "invalid type for implicit ctx register"; + + if (mode == BPF_IND) { + rs = bvf->evst->rv[ins->src_reg]; + if (rs.v.type != RTE_BPF_ARG_RAW) + return "unexpected type for src register"; + + eval_fill_imm(&ri, UINT64_MAX, ins->imm); + eval_add(&rs, &ri, UINT64_MAX); + + if (rs.s.max < 0 || rs.u.min > UINT32_MAX) + return "mbuf boundary violation"; + } + + /* R1-R5 scratch registers */ + for (i = EBPF_REG_1; i != EBPF_REG_6; i++) + bvf->evst->rv[i].v.type = RTE_BPF_ARG_UNDEF; + + /* R0 is an implicit output, contains data fetched from the packet */ + rv = bvf->evst->rv + EBPF_REG_0; + rv->v.size = bpf_size(BPF_SIZE(ins->code)); + eval_fill_max_bound(rv, RTE_LEN2MASK(rv->v.size * CHAR_BIT, uint64_t)); + + return NULL; +} + /* * check that destination and source operand are in defined state. */ @@ -1425,6 +1464,44 @@ static const struct bpf_ins_check ins_chk[UINT8_MAX + 1] = { .imm = { .min = 0, .max = UINT32_MAX}, .eval = eval_ld_imm64, }, + /* load absolute instructions */ + [(BPF_LD | BPF_ABS | BPF_B)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_ABS | BPF_H)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_ABS | BPF_W)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + /* load indirect instructions */ + [(BPF_LD | BPF_IND | BPF_B)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_IND | BPF_H)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_IND | BPF_W)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, /* store REG instructions */ [(BPF_STX | BPF_MEM | BPF_B)] = { .mask = { .dreg = ALL_REGS, .sreg = ALL_REGS}, -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf load instructions 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (2 preceding siblings ...) 2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev @ 2020-05-18 15:52 ` Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw) To: dev; +Cc: stephen, jerinj, Konstantin Ananyev Add new test-cases for BPF_ABS/BPF_IND load instructions. Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- app/test/test_bpf.c | 463 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 463 insertions(+) diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c index 4a61a7d7c..7c3de96c6 100644 --- a/app/test/test_bpf.c +++ b/app/test/test_bpf.c @@ -43,6 +43,14 @@ struct dummy_net { struct rte_ipv4_hdr ip_hdr; }; +#define DUMMY_MBUF_NUM 2 + +/* first mbuf in the packet, should always be at offset 0 */ +struct dummy_mbuf { + struct rte_mbuf mb[DUMMY_MBUF_NUM]; + uint8_t buf[DUMMY_MBUF_NUM][RTE_MBUF_DEFAULT_BUF_SIZE]; +}; + #define TEST_FILL_1 0xDEADBEEF #define TEST_MUL_1 21 @@ -2444,6 +2452,413 @@ static const struct rte_bpf_xsym test_call5_xsym[] = { }, }; +/* load mbuf (BPF_ABS/BPF_IND) test-cases */ +static const struct ebpf_insn test_ld_mbuf1_prog[] = { + + /* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_6, + .src_reg = EBPF_REG_1, + }, + /* load IPv4 version and IHL */ + { + .code = (BPF_LD | BPF_ABS | BPF_B), + .imm = offsetof(struct rte_ipv4_hdr, version_ihl), + }, + /* check IP version */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_2, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = 0xf0, + }, + { + .code = (BPF_JMP | BPF_JEQ | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = IPVERSION << 4, + .off = 2, + }, + /* invalid IP version, return 0 */ + { + .code = (EBPF_ALU64 | BPF_XOR | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, + /* load 3-rd byte of IP data */ + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = RTE_IPV4_HDR_IHL_MASK, + }, + { + .code = (BPF_ALU | BPF_LSH | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = 2, + }, + { + .code = (BPF_LD | BPF_IND | BPF_B), + .src_reg = EBPF_REG_0, + .imm = 3, + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load IPv4 src addr */ + { + .code = (BPF_LD | BPF_ABS | BPF_W), + .imm = offsetof(struct rte_ipv4_hdr, src_addr), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load IPv4 total length */ + { + .code = (BPF_LD | BPF_ABS | BPF_H), + .imm = offsetof(struct rte_ipv4_hdr, total_length), + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_8, + .src_reg = EBPF_REG_0, + }, + /* load last 4 bytes of IP data */ + { + .code = (BPF_LD | BPF_IND | BPF_W), + .src_reg = EBPF_REG_8, + .imm = -(int32_t)sizeof(uint32_t), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load 2 bytes from the middle of IP data */ + { + .code = (EBPF_ALU64 | BPF_RSH | BPF_K), + .dst_reg = EBPF_REG_8, + .imm = 1, + }, + { + .code = (BPF_LD | BPF_IND | BPF_H), + .src_reg = EBPF_REG_8, + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_7, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, +}; + +static void +dummy_mbuf_prep(struct rte_mbuf *mb, uint8_t buf[], uint32_t buf_len, + uint32_t data_len) +{ + uint32_t i; + uint8_t *db; + + mb->buf_addr = buf; + mb->buf_iova = (uintptr_t)buf; + mb->buf_len = buf_len; + rte_mbuf_refcnt_set(mb, 1); + + /* set pool pointer to dummy value, test doesn't use it */ + mb->pool = (void *)buf; + + rte_pktmbuf_reset(mb); + db = (uint8_t *)rte_pktmbuf_append(mb, data_len); + + for (i = 0; i != data_len; i++) + db[i] = i; +} + +static void +test_ld_mbuf1_prepare(void *arg) +{ + struct dummy_mbuf *dm; + struct rte_ipv4_hdr *ph; + + const uint32_t plen = 400; + const struct rte_ipv4_hdr iph = { + .version_ihl = RTE_IPV4_VHL_DEF, + .total_length = rte_cpu_to_be_16(plen), + .time_to_live = IPDEFTTL, + .next_proto_id = IPPROTO_RAW, + .src_addr = rte_cpu_to_be_32(RTE_IPV4_LOOPBACK), + .dst_addr = rte_cpu_to_be_32(RTE_IPV4_BROADCAST), + }; + + dm = arg; + memset(dm, 0, sizeof(*dm)); + + dummy_mbuf_prep(&dm->mb[0], dm->buf[0], sizeof(dm->buf[0]), + plen / 2 + 1); + dummy_mbuf_prep(&dm->mb[1], dm->buf[1], sizeof(dm->buf[0]), + plen / 2 - 1); + + rte_pktmbuf_chain(&dm->mb[0], &dm->mb[1]); + + ph = rte_pktmbuf_mtod(dm->mb, typeof(ph)); + memcpy(ph, &iph, sizeof(iph)); +} + +static uint64_t +test_ld_mbuf1(const struct rte_mbuf *pkt) +{ + uint64_t n, v; + const uint8_t *p8; + const uint16_t *p16; + const uint32_t *p32; + struct dummy_offset dof; + + /* load IPv4 version and IHL */ + p8 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, version_ihl), sizeof(*p8), + &dof); + if (p8 == NULL) + return 0; + + /* check IP version */ + if ((p8[0] & 0xf0) != IPVERSION << 4) + return 0; + + n = (p8[0] & RTE_IPV4_HDR_IHL_MASK) * RTE_IPV4_IHL_MULTIPLIER; + + /* load 3-rd byte of IP data */ + p8 = rte_pktmbuf_read(pkt, n + 3, sizeof(*p8), &dof); + if (p8 == NULL) + return 0; + + v = p8[0]; + + /* load IPv4 src addr */ + p32 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, src_addr), sizeof(*p32), + &dof); + if (p32 == NULL) + return 0; + + v += rte_be_to_cpu_32(p32[0]); + + /* load IPv4 total length */ + p16 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, total_length), sizeof(*p16), + &dof); + if (p16 == NULL) + return 0; + + n = rte_be_to_cpu_16(p16[0]); + + /* load last 4 bytes of IP data */ + p32 = rte_pktmbuf_read(pkt, n - sizeof(*p32), sizeof(*p32), &dof); + if (p32 == NULL) + return 0; + + v += rte_be_to_cpu_32(p32[0]); + + /* load 2 bytes from the middle of IP data */ + p16 = rte_pktmbuf_read(pkt, n / 2, sizeof(*p16), &dof); + if (p16 == NULL) + return 0; + + v += rte_be_to_cpu_16(p16[0]); + return v; +} + +static int +test_ld_mbuf1_check(uint64_t rc, const void *arg) +{ + const struct dummy_mbuf *dm; + uint64_t v; + + dm = arg; + v = test_ld_mbuf1(dm->mb); + return cmp_res(__func__, v, rc, arg, arg, 0); +} + +/* + * same as ld_mbuf1, but then trancate the mbuf by 1B, + * so load of last 4B fail. + */ +static void +test_ld_mbuf2_prepare(void *arg) +{ + struct dummy_mbuf *dm; + + test_ld_mbuf1_prepare(arg); + dm = arg; + rte_pktmbuf_trim(dm->mb, 1); +} + +static int +test_ld_mbuf2_check(uint64_t rc, const void *arg) +{ + return cmp_res(__func__, 0, rc, arg, arg, 0); +} + +/* same as test_ld_mbuf1, but now store intermediate results on the stack */ +static const struct ebpf_insn test_ld_mbuf3_prog[] = { + + /* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_6, + .src_reg = EBPF_REG_1, + }, + /* load IPv4 version and IHL */ + { + .code = (BPF_LD | BPF_ABS | BPF_B), + .imm = offsetof(struct rte_ipv4_hdr, version_ihl), + }, + /* check IP version */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_2, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = 0xf0, + }, + { + .code = (BPF_JMP | BPF_JEQ | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = IPVERSION << 4, + .off = 2, + }, + /* invalid IP version, return 0 */ + { + .code = (EBPF_ALU64 | BPF_XOR | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, + /* load 3-rd byte of IP data */ + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = RTE_IPV4_HDR_IHL_MASK, + }, + { + .code = (BPF_ALU | BPF_LSH | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = 2, + }, + { + .code = (BPF_LD | BPF_IND | BPF_B), + .src_reg = EBPF_REG_0, + .imm = 3, + }, + { + .code = (BPF_STX | BPF_MEM | BPF_B), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u8) - + sizeof(struct dummy_offset)), + }, + /* load IPv4 src addr */ + { + .code = (BPF_LD | BPF_ABS | BPF_W), + .imm = offsetof(struct rte_ipv4_hdr, src_addr), + }, + { + .code = (BPF_STX | BPF_MEM | BPF_W), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u32) - + sizeof(struct dummy_offset)), + }, + /* load IPv4 total length */ + { + .code = (BPF_LD | BPF_ABS | BPF_H), + .imm = offsetof(struct rte_ipv4_hdr, total_length), + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_8, + .src_reg = EBPF_REG_0, + }, + /* load last 4 bytes of IP data */ + { + .code = (BPF_LD | BPF_IND | BPF_W), + .src_reg = EBPF_REG_8, + .imm = -(int32_t)sizeof(uint32_t), + }, + { + .code = (BPF_STX | BPF_MEM | EBPF_DW), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u64) - + sizeof(struct dummy_offset)), + }, + /* load 2 bytes from the middle of IP data */ + { + .code = (EBPF_ALU64 | BPF_RSH | BPF_K), + .dst_reg = EBPF_REG_8, + .imm = 1, + }, + { + .code = (BPF_LD | BPF_IND | BPF_H), + .src_reg = EBPF_REG_8, + }, + { + .code = (BPF_LDX | BPF_MEM | EBPF_DW), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u64) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_LDX | BPF_MEM | BPF_W), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u32) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_LDX | BPF_MEM | BPF_B), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u8) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, +}; + /* all bpf test cases */ static const struct bpf_test tests[] = { { @@ -2704,6 +3119,54 @@ static const struct bpf_test tests[] = { /* for now don't support function calls on 32 bit platform */ .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), }, + { + .name = "test_ld_mbuf1", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf1_prog, + .nb_ins = RTE_DIM(test_ld_mbuf1_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf1_prepare, + .check_result = test_ld_mbuf1_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, + { + .name = "test_ld_mbuf2", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf1_prog, + .nb_ins = RTE_DIM(test_ld_mbuf1_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf2_prepare, + .check_result = test_ld_mbuf2_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, + { + .name = "test_ld_mbuf3", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf3_prog, + .nb_ins = RTE_DIM(test_ld_mbuf3_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf1_prepare, + .check_result = test_ld_mbuf1_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, }; static int -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data load instructions 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (3 preceding siblings ...) 2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev @ 2020-05-18 15:52 ` Konstantin Ananyev 2020-05-24 13:37 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 5 siblings, 1 reply; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-18 15:52 UTC (permalink / raw) To: dev; +Cc: stephen, jerinj, Konstantin Ananyev Make x86 JIT to generate native code for (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) instructions. Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- lib/librte_bpf/bpf_jit_x86.c | 181 ++++++++++++++++++++++++++++++++++ lib/librte_bpf/bpf_validate.c | 17 +++- 2 files changed, 197 insertions(+), 1 deletion(-) diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c index f70cd6be5..8a7ad45b3 100644 --- a/lib/librte_bpf/bpf_jit_x86.c +++ b/lib/librte_bpf/bpf_jit_x86.c @@ -87,6 +87,14 @@ enum { REG_TMP1 = R10, }; +/* LD_ABS/LD_IMM offsets */ +enum { + LDMB_FSP_OFS, /* fast-path */ + LDMB_SLP_OFS, /* slow-path */ + LDMB_FIN_OFS, /* final part */ + LDMB_OFS_NUM +}; + /* * callee saved registers list. * keep RBP as the last one. @@ -100,6 +108,9 @@ struct bpf_jit_state { uint32_t num; int32_t off; } exit; + struct { + uint32_t stack_ofs; + } ldmb; uint32_t reguse; int32_t *off; uint8_t *ins; @@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t dreg, emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX); } +/* + * helper function, used by emit_ld_mbuf(). + * generates code for 'fast_path': + * calculate load offset and check is it inside first packet segment. + */ +static void +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7], + uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm, + const int32_t ofs[LDMB_OFS_NUM]) +{ + /* make R2 contain *off* value */ + + if (sreg != rg[EBPF_REG_2]) { + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, + rg[EBPF_REG_2], imm); + if (mode == BPF_IND) + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + sreg, rg[EBPF_REG_2]); + } else + /* BPF_IND with sreg == R2 */ + emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K, + rg[EBPF_REG_2], imm); + + /* R3 = mbuf->data_len */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, + rg[EBPF_REG_6], rg[EBPF_REG_3], + offsetof(struct rte_mbuf, data_len)); + + /* R3 = R3 - R2 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X, + rg[EBPF_REG_2], rg[EBPF_REG_3]); + + /* JSLE R3, <sz> <slow_path> */ + emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz); + emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]); + + /* R3 = mbuf->data_off */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, + rg[EBPF_REG_6], rg[EBPF_REG_3], + offsetof(struct rte_mbuf, data_off)); + + /* R0 = mbuf->buf_addr */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW, + rg[EBPF_REG_6], rg[EBPF_REG_0], + offsetof(struct rte_mbuf, buf_addr)); + + /* R0 = R0 + R3 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + rg[EBPF_REG_3], rg[EBPF_REG_0]); + + /* R0 = R0 + R2 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + rg[EBPF_REG_2], rg[EBPF_REG_0]); + + /* JMP <fin_part> */ + emit_abs_jmp(st, ofs[LDMB_FIN_OFS]); +} + +/* + * helper function, used by emit_ld_mbuf(). + * generates code for 'slow_path': + * call __rte_pktmbuf_read() and check return value. + */ +static void +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7], + uint32_t sz) +{ + /* make R3 contain *len* value (1/2/4) */ + + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz); + + /* make R4 contain (RBP - ldmb.stack_ofs) */ + + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]); + emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4], + st->ldmb.stack_ofs); + + /* make R1 contain mbuf ptr */ + + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, + rg[EBPF_REG_6], rg[EBPF_REG_1]); + + /* call rte_pktmbuf_read */ + emit_call(st, (uintptr_t)__rte_pktmbuf_read); + + /* check that return value (R0) is not zero */ + emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]); + emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off); +} + +/* + * helper function, used by emit_ld_mbuf(). + * generates final part of code for BPF_ABS/BPF_IND load: + * perform data load and endianness conversion. + * expects dreg to contain valid data pointer. + */ +static void +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz, + uint32_t sz) +{ + emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0); + if (sz != sizeof(uint8_t)) + emit_be2le(st, dreg, sz * CHAR_BIT); +} + +/* + * emit code for BPF_ABS/BPF_IND load. + * generates the following construction: + * fast_path: + * off = ins->sreg + ins->imm + * if (off + ins->opsz < mbuf->data_len) + * goto slow_path; + * ptr = mbuf->buf_addr + bmf->data_off + off; + * goto fin_part; + * slow_path: + * typeof(ins->opsz) buf; //allocate space on the stack + * ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf); + * if (ptr == NULL) + * goto exit_label; + * fin_part: + * res = *(typeof(ins->opsz))ptr; + * res = bswap(res); + */ +static void +emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t imm) +{ + uint32_t i, mode, opsz, sz; + uint32_t rg[EBPF_REG_7]; + int32_t ofs[LDMB_OFS_NUM]; + + mode = BPF_MODE(op); + opsz = BPF_SIZE(op); + sz = bpf_size(opsz); + + for (i = 0; i != RTE_DIM(rg); i++) + rg[i] = ebpf2x86[i]; + + /* fill with fake offsets */ + for (i = 0; i != RTE_DIM(ofs); i++) + ofs[i] = st->sz + INT8_MAX; + + /* dry run first to calculate jump offsets */ + + ofs[LDMB_FSP_OFS] = st->sz; + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); + ofs[LDMB_SLP_OFS] = st->sz; + emit_ldmb_slow_path(st, rg, sz); + ofs[LDMB_FIN_OFS] = st->sz; + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); + + RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX); + + /* reset dry-run code and do a proper run */ + + st->sz = ofs[LDMB_FSP_OFS]; + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); + emit_ldmb_slow_path(st, rg, sz); + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); +} + static void emit_prolog(struct bpf_jit_state *st, int32_t stack_size) { @@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf) /* reset state fields */ st->sz = 0; st->exit.num = 0; + st->ldmb.stack_ofs = bpf->stack_sz; emit_prolog(st, bpf->stack_sz); @@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf) emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm); i++; break; + /* load absolute/indirect instructions */ + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): + emit_ld_mbuf(st, op, sr, ins->imm); + break; /* store instructions */ case (BPF_STX | BPF_MEM | BPF_B): case (BPF_STX | BPF_MEM | BPF_H): diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index fecdda0e1..9214f1503 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -70,6 +70,7 @@ struct bpf_verifier { uint64_t stack_sz; uint32_t nb_nodes; uint32_t nb_jcc_nodes; + uint32_t nb_ldmb_nodes; uint32_t node_colour[MAX_NODE_COLOUR]; uint32_t edge_type[MAX_EDGE_TYPE]; struct bpf_eval_state *evst; @@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf) rc |= add_edge(bvf, node, i + 2); i++; break; + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): + bvf->nb_ldmb_nodes++; + /* fallthrough */ default: rc |= add_edge(bvf, node, i + 1); break; @@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf) free(bvf.in); /* copy collected info */ - if (rc == 0) + if (rc == 0) { bpf->stack_sz = bvf.stack_sz; + /* for LD_ABS/LD_IND, we'll need extra space on the stack */ + if (bvf.nb_ldmb_nodes != 0) + bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz + + sizeof(uint64_t), sizeof(uint64_t)); + } + return rc; } -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions 2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev @ 2020-05-24 13:37 ` Morten Brørup 2020-05-24 23:08 ` Ananyev, Konstantin 0 siblings, 1 reply; 17+ messages in thread From: Morten Brørup @ 2020-05-24 13:37 UTC (permalink / raw) To: Konstantin Ananyev, dev; +Cc: stephen, jerinj > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Konstantin Ananyev > Sent: Monday, May 18, 2020 5:53 PM > > Make x86 JIT to generate native code for > (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) > instructions. > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> > --- > lib/librte_bpf/bpf_jit_x86.c | 181 ++++++++++++++++++++++++++++++++++ > lib/librte_bpf/bpf_validate.c | 17 +++- > 2 files changed, 197 insertions(+), 1 deletion(-) > > diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c > index f70cd6be5..8a7ad45b3 100644 > --- a/lib/librte_bpf/bpf_jit_x86.c > +++ b/lib/librte_bpf/bpf_jit_x86.c > @@ -87,6 +87,14 @@ enum { > REG_TMP1 = R10, > }; > > +/* LD_ABS/LD_IMM offsets */ > +enum { > + LDMB_FSP_OFS, /* fast-path */ > + LDMB_SLP_OFS, /* slow-path */ > + LDMB_FIN_OFS, /* final part */ > + LDMB_OFS_NUM > +}; > + > /* > * callee saved registers list. > * keep RBP as the last one. > @@ -100,6 +108,9 @@ struct bpf_jit_state { > uint32_t num; > int32_t off; > } exit; > + struct { > + uint32_t stack_ofs; > + } ldmb; > uint32_t reguse; > int32_t *off; > uint8_t *ins; > @@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op, > uint32_t sreg, uint32_t dreg, > emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX); > } > > +/* > + * helper function, used by emit_ld_mbuf(). > + * generates code for 'fast_path': > + * calculate load offset and check is it inside first packet segment. > + */ > +static void > +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t > rg[EBPF_REG_7], > + uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm, > + const int32_t ofs[LDMB_OFS_NUM]) > +{ > + /* make R2 contain *off* value */ > + > + if (sreg != rg[EBPF_REG_2]) { > + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, > + rg[EBPF_REG_2], imm); > + if (mode == BPF_IND) > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > + sreg, rg[EBPF_REG_2]); > + } else > + /* BPF_IND with sreg == R2 */ > + emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K, > + rg[EBPF_REG_2], imm); > + > + /* R3 = mbuf->data_len */ > + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, > + rg[EBPF_REG_6], rg[EBPF_REG_3], > + offsetof(struct rte_mbuf, data_len)); > + > + /* R3 = R3 - R2 */ > + emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X, > + rg[EBPF_REG_2], rg[EBPF_REG_3]); > + > + /* JSLE R3, <sz> <slow_path> */ > + emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz); > + emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]); > + > + /* R3 = mbuf->data_off */ > + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, > + rg[EBPF_REG_6], rg[EBPF_REG_3], > + offsetof(struct rte_mbuf, data_off)); > + > + /* R0 = mbuf->buf_addr */ > + emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW, > + rg[EBPF_REG_6], rg[EBPF_REG_0], > + offsetof(struct rte_mbuf, buf_addr)); > + > + /* R0 = R0 + R3 */ > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > + rg[EBPF_REG_3], rg[EBPF_REG_0]); > + > + /* R0 = R0 + R2 */ > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > + rg[EBPF_REG_2], rg[EBPF_REG_0]); > + > + /* JMP <fin_part> */ > + emit_abs_jmp(st, ofs[LDMB_FIN_OFS]); > +} > + > +/* > + * helper function, used by emit_ld_mbuf(). > + * generates code for 'slow_path': > + * call __rte_pktmbuf_read() and check return value. > + */ > +static void > +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t > rg[EBPF_REG_7], > + uint32_t sz) > +{ > + /* make R3 contain *len* value (1/2/4) */ > + > + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz); > + > + /* make R4 contain (RBP - ldmb.stack_ofs) */ > + > + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]); > + emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4], > + st->ldmb.stack_ofs); > + > + /* make R1 contain mbuf ptr */ > + > + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, > + rg[EBPF_REG_6], rg[EBPF_REG_1]); > + > + /* call rte_pktmbuf_read */ > + emit_call(st, (uintptr_t)__rte_pktmbuf_read); > + > + /* check that return value (R0) is not zero */ > + emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]); > + emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off); > +} > + > +/* > + * helper function, used by emit_ld_mbuf(). > + * generates final part of code for BPF_ABS/BPF_IND load: > + * perform data load and endianness conversion. > + * expects dreg to contain valid data pointer. > + */ > +static void > +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz, > + uint32_t sz) > +{ > + emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0); > + if (sz != sizeof(uint8_t)) > + emit_be2le(st, dreg, sz * CHAR_BIT); > +} > + > +/* > + * emit code for BPF_ABS/BPF_IND load. > + * generates the following construction: > + * fast_path: > + * off = ins->sreg + ins->imm > + * if (off + ins->opsz < mbuf->data_len) > + * goto slow_path; I am not an eBPF expert, but I am not sure this is correct. I think it should be > instead of <. Also, it looks like you actually emit: if (mbuf->data_len - off <= ins->opsz) goto slow_path; Could you please double check that both the comment and the emitted code has the comparison turning the correct way. > + * ptr = mbuf->buf_addr + bmf->data_off + off; > + * goto fin_part; > + * slow_path: > + * typeof(ins->opsz) buf; //allocate space on the stack > + * ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf); > + * if (ptr == NULL) > + * goto exit_label; > + * fin_part: > + * res = *(typeof(ins->opsz))ptr; > + * res = bswap(res); > + */ > +static void > +emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, > uint32_t imm) > +{ > + uint32_t i, mode, opsz, sz; > + uint32_t rg[EBPF_REG_7]; > + int32_t ofs[LDMB_OFS_NUM]; > + > + mode = BPF_MODE(op); > + opsz = BPF_SIZE(op); > + sz = bpf_size(opsz); > + > + for (i = 0; i != RTE_DIM(rg); i++) > + rg[i] = ebpf2x86[i]; > + > + /* fill with fake offsets */ > + for (i = 0; i != RTE_DIM(ofs); i++) > + ofs[i] = st->sz + INT8_MAX; > + > + /* dry run first to calculate jump offsets */ > + > + ofs[LDMB_FSP_OFS] = st->sz; > + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); > + ofs[LDMB_SLP_OFS] = st->sz; > + emit_ldmb_slow_path(st, rg, sz); > + ofs[LDMB_FIN_OFS] = st->sz; > + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); > + > + RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX); > + > + /* reset dry-run code and do a proper run */ > + > + st->sz = ofs[LDMB_FSP_OFS]; > + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); > + emit_ldmb_slow_path(st, rg, sz); > + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); > +} > + > static void > emit_prolog(struct bpf_jit_state *st, int32_t stack_size) > { > @@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf > *bpf) > /* reset state fields */ > st->sz = 0; > st->exit.num = 0; > + st->ldmb.stack_ofs = bpf->stack_sz; > > emit_prolog(st, bpf->stack_sz); > > @@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf > *bpf) > emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm); > i++; > break; > + /* load absolute/indirect instructions */ > + case (BPF_LD | BPF_ABS | BPF_B): > + case (BPF_LD | BPF_ABS | BPF_H): > + case (BPF_LD | BPF_ABS | BPF_W): > + case (BPF_LD | BPF_IND | BPF_B): > + case (BPF_LD | BPF_IND | BPF_H): > + case (BPF_LD | BPF_IND | BPF_W): > + emit_ld_mbuf(st, op, sr, ins->imm); > + break; > /* store instructions */ > case (BPF_STX | BPF_MEM | BPF_B): > case (BPF_STX | BPF_MEM | BPF_H): > diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c > index fecdda0e1..9214f1503 100644 > --- a/lib/librte_bpf/bpf_validate.c > +++ b/lib/librte_bpf/bpf_validate.c > @@ -70,6 +70,7 @@ struct bpf_verifier { > uint64_t stack_sz; > uint32_t nb_nodes; > uint32_t nb_jcc_nodes; > + uint32_t nb_ldmb_nodes; > uint32_t node_colour[MAX_NODE_COLOUR]; > uint32_t edge_type[MAX_EDGE_TYPE]; > struct bpf_eval_state *evst; > @@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf) > rc |= add_edge(bvf, node, i + 2); > i++; > break; > + case (BPF_LD | BPF_ABS | BPF_B): > + case (BPF_LD | BPF_ABS | BPF_H): > + case (BPF_LD | BPF_ABS | BPF_W): > + case (BPF_LD | BPF_IND | BPF_B): > + case (BPF_LD | BPF_IND | BPF_H): > + case (BPF_LD | BPF_IND | BPF_W): > + bvf->nb_ldmb_nodes++; > + /* fallthrough */ > default: > rc |= add_edge(bvf, node, i + 1); > break; > @@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf) > free(bvf.in); > > /* copy collected info */ > - if (rc == 0) > + if (rc == 0) { > bpf->stack_sz = bvf.stack_sz; > > + /* for LD_ABS/LD_IND, we'll need extra space on the stack */ > + if (bvf.nb_ldmb_nodes != 0) > + bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz + > + sizeof(uint64_t), sizeof(uint64_t)); > + } > + > return rc; > } > -- > 2.17.1 > ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions 2020-05-24 13:37 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup @ 2020-05-24 23:08 ` Ananyev, Konstantin 0 siblings, 0 replies; 17+ messages in thread From: Ananyev, Konstantin @ 2020-05-24 23:08 UTC (permalink / raw) To: Morten Brørup, dev; +Cc: stephen, jerinj > > > > +/* > > + * helper function, used by emit_ld_mbuf(). > > + * generates code for 'fast_path': > > + * calculate load offset and check is it inside first packet segment. > > + */ > > +static void > > +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t > > rg[EBPF_REG_7], > > + uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm, > > + const int32_t ofs[LDMB_OFS_NUM]) > > +{ > > + /* make R2 contain *off* value */ > > + > > + if (sreg != rg[EBPF_REG_2]) { > > + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, > > + rg[EBPF_REG_2], imm); > > + if (mode == BPF_IND) > > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > > + sreg, rg[EBPF_REG_2]); > > + } else > > + /* BPF_IND with sreg == R2 */ > > + emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K, > > + rg[EBPF_REG_2], imm); > > + > > + /* R3 = mbuf->data_len */ > > + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, > > + rg[EBPF_REG_6], rg[EBPF_REG_3], > > + offsetof(struct rte_mbuf, data_len)); > > + > > + /* R3 = R3 - R2 */ > > + emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X, > > + rg[EBPF_REG_2], rg[EBPF_REG_3]); > > + > > + /* JSLE R3, <sz> <slow_path> */ > > + emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz); > > + emit_abs_jcc(st, BPF_JMP | EBPF_JSLE | BPF_K, ofs[LDMB_SLP_OFS]); > > + > > + /* R3 = mbuf->data_off */ > > + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, > > + rg[EBPF_REG_6], rg[EBPF_REG_3], > > + offsetof(struct rte_mbuf, data_off)); > > + > > + /* R0 = mbuf->buf_addr */ > > + emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW, > > + rg[EBPF_REG_6], rg[EBPF_REG_0], > > + offsetof(struct rte_mbuf, buf_addr)); > > + > > + /* R0 = R0 + R3 */ > > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > > + rg[EBPF_REG_3], rg[EBPF_REG_0]); > > + > > + /* R0 = R0 + R2 */ > > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > > + rg[EBPF_REG_2], rg[EBPF_REG_0]); > > + > > + /* JMP <fin_part> */ > > + emit_abs_jmp(st, ofs[LDMB_FIN_OFS]); > > +} > > + > > +/* > > + * helper function, used by emit_ld_mbuf(). > > + * generates code for 'slow_path': > > + * call __rte_pktmbuf_read() and check return value. > > + */ > > +static void > > +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t > > rg[EBPF_REG_7], > > + uint32_t sz) > > +{ > > + /* make R3 contain *len* value (1/2/4) */ > > + > > + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz); > > + > > + /* make R4 contain (RBP - ldmb.stack_ofs) */ > > + > > + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]); > > + emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4], > > + st->ldmb.stack_ofs); > > + > > + /* make R1 contain mbuf ptr */ > > + > > + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, > > + rg[EBPF_REG_6], rg[EBPF_REG_1]); > > + > > + /* call rte_pktmbuf_read */ > > + emit_call(st, (uintptr_t)__rte_pktmbuf_read); > > + > > + /* check that return value (R0) is not zero */ > > + emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]); > > + emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off); > > +} > > + > > +/* > > + * helper function, used by emit_ld_mbuf(). > > + * generates final part of code for BPF_ABS/BPF_IND load: > > + * perform data load and endianness conversion. > > + * expects dreg to contain valid data pointer. > > + */ > > +static void > > +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz, > > + uint32_t sz) > > +{ > > + emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0); > > + if (sz != sizeof(uint8_t)) > > + emit_be2le(st, dreg, sz * CHAR_BIT); > > +} > > + > > +/* > > + * emit code for BPF_ABS/BPF_IND load. > > + * generates the following construction: > > + * fast_path: > > + * off = ins->sreg + ins->imm > > + * if (off + ins->opsz < mbuf->data_len) > > + * goto slow_path; > > I am not an eBPF expert, but I am not sure this is correct. > > I think it should be > instead of <. Also, it looks like you actually emit: > if (mbuf->data_len - off <= ins->opsz) > goto slow_path; > > Could you please double check that both the comment and the emitted code has the comparison turning the correct way. Ack, should be: if (mbuf->data_len - off < ins->opsz) Will send v2. ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (4 preceding siblings ...) 2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev ` (5 more replies) 5 siblings, 6 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev To fill the gap with linux kernel eBPF implementation, add support for two non-generic instructions: (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) which are used to access packet data in a safe manner. Make necessary changes in BPF verifier, interpreter and x86 JIT code. Konstantin Ananyev (5): test/bpf: fix few small issues bpf: fix add/sub min/max estimations bpf: add support for packet data load instructions test/bpf: add new test cases for mbuf load instructions bpf: x86 JIT support for packet data load instructions app/test/test_bpf.c | 504 +++++++++++++++++++++++-- doc/guides/prog_guide/bpf_lib.rst | 30 +- doc/guides/rel_notes/release_20_08.rst | 7 + lib/librte_bpf/bpf_exec.c | 57 +++ lib/librte_bpf/bpf_jit_x86.c | 181 +++++++++ lib/librte_bpf/bpf_validate.c | 104 ++++- 6 files changed, 850 insertions(+), 33 deletions(-) -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev ` (4 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev, stable Address for few small issues: - unreachable return statement - failed test-case can finish with 'success' status Also use unified cmp_res() function to check return value. Fixes: a9de470cc7c0 ("test: move to app directory") Cc: stable@dpdk.org Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- app/test/test_bpf.c | 41 +++++++++++++++-------------------------- 1 file changed, 15 insertions(+), 26 deletions(-) diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c index ee534687a..4a61a7d7c 100644 --- a/app/test/test_bpf.c +++ b/app/test/test_bpf.c @@ -1797,13 +1797,6 @@ test_call1_check(uint64_t rc, const void *arg) dummy_func1(arg, &v32, &v64); v64 += v32; - if (v64 != rc) { - printf("%s@%d: invalid return value " - "expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n", - __func__, __LINE__, v64, rc); - return -1; - } - return 0; return cmp_res(__func__, v64, rc, dv, dv, sizeof(*dv)); } @@ -1934,13 +1927,7 @@ test_call2_check(uint64_t rc, const void *arg) dummy_func2(&a, &b); v = a.u64 + a.u32 + b.u16 + b.u8; - if (v != rc) { - printf("%s@%d: invalid return value " - "expected=0x%" PRIx64 ", actual=0x%" PRIx64 "\n", - __func__, __LINE__, v, rc); - return -1; - } - return 0; + return cmp_res(__func__, v, rc, arg, arg, 0); } static const struct rte_bpf_xsym test_call2_xsym[] = { @@ -2429,7 +2416,6 @@ test_call5_check(uint64_t rc, const void *arg) v = 0; fail: - return cmp_res(__func__, v, rc, &v, &rc, sizeof(v)); } @@ -2458,6 +2444,7 @@ static const struct rte_bpf_xsym test_call5_xsym[] = { }, }; +/* all bpf test cases */ static const struct bpf_test tests[] = { { .name = "test_store1", @@ -2738,7 +2725,6 @@ run_test(const struct bpf_test *tst) } tst->prepare(tbuf); - rc = rte_bpf_exec(bpf, tbuf); ret = tst->check_result(rc, tbuf); if (ret != 0) { @@ -2746,17 +2732,20 @@ run_test(const struct bpf_test *tst) __func__, __LINE__, tst->name, ret, strerror(ret)); } + /* repeat the same test with jit, when possible */ rte_bpf_get_jit(bpf, &jit); - if (jit.func == NULL) - return 0; - - tst->prepare(tbuf); - rc = jit.func(tbuf); - rv = tst->check_result(rc, tbuf); - ret |= rv; - if (rv != 0) { - printf("%s@%d: check_result(%s) failed, error: %d(%s);\n", - __func__, __LINE__, tst->name, rv, strerror(ret)); + if (jit.func != NULL) { + + tst->prepare(tbuf); + rc = jit.func(tbuf); + rv = tst->check_result(rc, tbuf); + ret |= rv; + if (rv != 0) { + printf("%s@%d: check_result(%s) failed, " + "error: %d(%s);\n", + __func__, __LINE__, tst->name, + rv, strerror(ret)); + } } rte_bpf_destroy(bpf); -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev ` (3 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev, stable eval_add()/eval_sub() not always correctly estimate minimum and maximum possible values of add/sub operations. Fixes: 8021917293d0 ("bpf: add extra validation for input BPF program") Cc: stable@dpdk.org Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- lib/librte_bpf/bpf_validate.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index 6bd6f78e9..80d21fabb 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -226,7 +226,7 @@ eval_add(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk) struct bpf_reg_val rv; rv.u.min = (rd->u.min + rs->u.min) & msk; - rv.u.max = (rd->u.min + rs->u.max) & msk; + rv.u.max = (rd->u.max + rs->u.max) & msk; rv.s.min = (rd->s.min + rs->s.min) & msk; rv.s.max = (rd->s.max + rs->s.max) & msk; @@ -254,10 +254,10 @@ eval_sub(struct bpf_reg_val *rd, const struct bpf_reg_val *rs, uint64_t msk) { struct bpf_reg_val rv; - rv.u.min = (rd->u.min - rs->u.min) & msk; - rv.u.max = (rd->u.min - rs->u.max) & msk; - rv.s.min = (rd->s.min - rs->s.min) & msk; - rv.s.max = (rd->s.max - rs->s.max) & msk; + rv.u.min = (rd->u.min - rs->u.max) & msk; + rv.u.max = (rd->u.max - rs->u.min) & msk; + rv.s.min = (rd->s.min - rs->s.max) & msk; + rv.s.max = (rd->s.max - rs->s.min) & msk; /* * if at least one of the operands is not constant, -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev ` (2 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev To fill the gap with linux kernel eBPF implementation, add support for two non-generic instructions: (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) which are used to access packet data. These instructions can only be used when BPF context is a pointer to 'struct rte_mbuf' (i.e: RTE_BPF_ARG_PTR_MBUF type). Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- doc/guides/prog_guide/bpf_lib.rst | 30 +++++++++- doc/guides/rel_notes/release_20_08.rst | 7 +++ lib/librte_bpf/bpf_exec.c | 57 +++++++++++++++++++ lib/librte_bpf/bpf_validate.c | 77 ++++++++++++++++++++++++++ 4 files changed, 170 insertions(+), 1 deletion(-) diff --git a/doc/guides/prog_guide/bpf_lib.rst b/doc/guides/prog_guide/bpf_lib.rst index 9c728da7b..1feb7734a 100644 --- a/doc/guides/prog_guide/bpf_lib.rst +++ b/doc/guides/prog_guide/bpf_lib.rst @@ -27,6 +27,35 @@ The library API provides the following basic operations: * Load BPF program from the ELF file and install callback to execute it on given ethdev port/queue. +Packet data load instructions +----------------------------- + +DPDK supports two non-generic instructions: ``(BPF_ABS | size | BPF_LD)`` +and ``(BPF_IND | size | BPF_LD)`` which are used to access packet data. +These instructions can only be used when execution context is a pointer to +``struct rte_mbuf`` and have seven implicit operands. +Register ``R6`` is an implicit input that must contain pointer to ``rte_mbuf``. +Register ``R0`` is an implicit output which contains the data fetched from the +packet. Registers ``R1-R5`` are scratch registers +and must not be used to store the data across these instructions. +These instructions have implicit program exit condition as well. When +eBPF program is trying to access the data beyond the packet boundary, +the interpreter will abort the execution of the program. JIT compilers +therefore must preserve this property. ``src_reg`` and ``imm32`` fields are +explicit inputs to these instructions. +For example, ``(BPF_IND | BPF_W | BPF_LD)`` means: + +.. code-block:: c + + uint32_t tmp; + R0 = rte_pktmbuf_read((const struct rte_mbuf *)R6, src_reg + imm32, + sizeof(tmp), &tmp); + if (R0 == NULL) return FAILED; + R0 = ntohl(*(uint32_t *)R0); + +and ``R1-R5`` were scratched. + + Not currently supported eBPF features ------------------------------------- @@ -34,5 +63,4 @@ Not currently supported eBPF features - cBPF - tail-pointer call - eBPF MAP - - skb - external function calls for 32-bit platforms diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst index 39064afbe..3c0824616 100644 --- a/doc/guides/rel_notes/release_20_08.rst +++ b/doc/guides/rel_notes/release_20_08.rst @@ -56,6 +56,13 @@ New Features Also, make sure to start the actual text at the margin. ========================================================= +* **Added support for BPF_ABS/BPF_IND load instructions.** + + Added support for two BPF non-generic instructions: + ``(BPF_ABS | <size> | BPF_LD)`` and ``(BPF_IND | <size> | BPF_LD)`` + which are used to access packet data in a safe manner. Currently JIT support + for these instructions is implemented for x86 only. + Removed Items ------------- diff --git a/lib/librte_bpf/bpf_exec.c b/lib/librte_bpf/bpf_exec.c index 1bb226643..b921112fe 100644 --- a/lib/librte_bpf/bpf_exec.c +++ b/lib/librte_bpf/bpf_exec.c @@ -74,6 +74,26 @@ (uintptr_t)((reg)[(ins)->dst_reg] + (ins)->off), \ reg[ins->src_reg])) +/* BPF_LD | BPF_ABS/BPF_IND */ + +#define NOP(x) (x) + +#define BPF_LD_ABS(bpf, reg, ins, type, op) do { \ + const type *p = bpf_ld_mbuf(bpf, reg, ins, (ins)->imm, sizeof(type)); \ + if (p == NULL) \ + return 0; \ + reg[EBPF_REG_0] = op(p[0]); \ +} while (0) + +#define BPF_LD_IND(bpf, reg, ins, type, op) do { \ + uint32_t ofs = reg[ins->src_reg] + (ins)->imm; \ + const type *p = bpf_ld_mbuf(bpf, reg, ins, ofs, sizeof(type)); \ + if (p == NULL) \ + return 0; \ + reg[EBPF_REG_0] = op(p[0]); \ +} while (0) + + static inline void bpf_alu_be(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins) { @@ -112,6 +132,23 @@ bpf_alu_le(uint64_t reg[EBPF_REG_NUM], const struct ebpf_insn *ins) } } +static inline const void * +bpf_ld_mbuf(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM], + const struct ebpf_insn *ins, uint32_t off, uint32_t len) +{ + const struct rte_mbuf *mb; + const void *p; + + mb = (const struct rte_mbuf *)(uintptr_t)reg[EBPF_REG_6]; + p = rte_pktmbuf_read(mb, off, len, reg + EBPF_REG_0); + if (p == NULL) + RTE_BPF_LOG(DEBUG, "%s(bpf=%p, mbuf=%p, ofs=%u, len=%u): " + "load beyond packet boundary at pc: %#zx;\n", + __func__, bpf, mb, off, len, + (uintptr_t)(ins) - (uintptr_t)(bpf)->prm.ins); + return p; +} + static inline uint64_t bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM]) { @@ -296,6 +333,26 @@ bpf_exec(const struct rte_bpf *bpf, uint64_t reg[EBPF_REG_NUM]) (uint64_t)(uint32_t)ins[1].imm << 32; ins++; break; + /* load absolute instructions */ + case (BPF_LD | BPF_ABS | BPF_B): + BPF_LD_ABS(bpf, reg, ins, uint8_t, NOP); + break; + case (BPF_LD | BPF_ABS | BPF_H): + BPF_LD_ABS(bpf, reg, ins, uint16_t, rte_be_to_cpu_16); + break; + case (BPF_LD | BPF_ABS | BPF_W): + BPF_LD_ABS(bpf, reg, ins, uint32_t, rte_be_to_cpu_32); + break; + /* load indirect instructions */ + case (BPF_LD | BPF_IND | BPF_B): + BPF_LD_IND(bpf, reg, ins, uint8_t, NOP); + break; + case (BPF_LD | BPF_IND | BPF_H): + BPF_LD_IND(bpf, reg, ins, uint16_t, rte_be_to_cpu_16); + break; + case (BPF_LD | BPF_IND | BPF_W): + BPF_LD_IND(bpf, reg, ins, uint32_t, rte_be_to_cpu_32); + break; /* store instructions */ case (BPF_STX | BPF_MEM | BPF_B): BPF_ST_REG(reg, ins, uint8_t); diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index 80d21fabb..fecdda0e1 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -102,6 +102,9 @@ struct bpf_ins_check { #define WRT_REGS RTE_LEN2MASK(EBPF_REG_10, uint16_t) #define ZERO_REG RTE_LEN2MASK(EBPF_REG_1, uint16_t) +/* For LD_IND R6 is an implicit CTX register. */ +#define IND_SRC_REGS (WRT_REGS ^ 1 << EBPF_REG_6) + /* * check and evaluate functions for particular instruction types. */ @@ -580,6 +583,42 @@ eval_neg(struct bpf_reg_val *rd, size_t opsz, uint64_t msk) rd->s.min = RTE_MIN(sx, sy); } +static const char * +eval_ld_mbuf(struct bpf_verifier *bvf, const struct ebpf_insn *ins) +{ + uint32_t i, mode; + struct bpf_reg_val *rv, ri, rs; + + mode = BPF_MODE(ins->code); + + /* R6 is an implicit input that must contain pointer to mbuf */ + if (bvf->evst->rv[EBPF_REG_6].v.type != RTE_BPF_ARG_PTR_MBUF) + return "invalid type for implicit ctx register"; + + if (mode == BPF_IND) { + rs = bvf->evst->rv[ins->src_reg]; + if (rs.v.type != RTE_BPF_ARG_RAW) + return "unexpected type for src register"; + + eval_fill_imm(&ri, UINT64_MAX, ins->imm); + eval_add(&rs, &ri, UINT64_MAX); + + if (rs.s.max < 0 || rs.u.min > UINT32_MAX) + return "mbuf boundary violation"; + } + + /* R1-R5 scratch registers */ + for (i = EBPF_REG_1; i != EBPF_REG_6; i++) + bvf->evst->rv[i].v.type = RTE_BPF_ARG_UNDEF; + + /* R0 is an implicit output, contains data fetched from the packet */ + rv = bvf->evst->rv + EBPF_REG_0; + rv->v.size = bpf_size(BPF_SIZE(ins->code)); + eval_fill_max_bound(rv, RTE_LEN2MASK(rv->v.size * CHAR_BIT, uint64_t)); + + return NULL; +} + /* * check that destination and source operand are in defined state. */ @@ -1425,6 +1464,44 @@ static const struct bpf_ins_check ins_chk[UINT8_MAX + 1] = { .imm = { .min = 0, .max = UINT32_MAX}, .eval = eval_ld_imm64, }, + /* load absolute instructions */ + [(BPF_LD | BPF_ABS | BPF_B)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_ABS | BPF_H)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_ABS | BPF_W)] = { + .mask = {. dreg = ZERO_REG, .sreg = ZERO_REG}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = INT32_MAX}, + .eval = eval_ld_mbuf, + }, + /* load indirect instructions */ + [(BPF_LD | BPF_IND | BPF_B)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_IND | BPF_H)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, + [(BPF_LD | BPF_IND | BPF_W)] = { + .mask = {. dreg = ZERO_REG, .sreg = IND_SRC_REGS}, + .off = { .min = 0, .max = 0}, + .imm = { .min = 0, .max = UINT32_MAX}, + .eval = eval_ld_mbuf, + }, /* store REG instructions */ [(BPF_STX | BPF_MEM | BPF_B)] = { .mask = { .dreg = ALL_REGS, .sreg = ALL_REGS}, -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf load instructions 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (2 preceding siblings ...) 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev 2020-06-24 21:43 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon 5 siblings, 0 replies; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev Add new test-cases for BPF_ABS/BPF_IND load instructions. Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- app/test/test_bpf.c | 463 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 463 insertions(+) diff --git a/app/test/test_bpf.c b/app/test/test_bpf.c index 4a61a7d7c..7c3de96c6 100644 --- a/app/test/test_bpf.c +++ b/app/test/test_bpf.c @@ -43,6 +43,14 @@ struct dummy_net { struct rte_ipv4_hdr ip_hdr; }; +#define DUMMY_MBUF_NUM 2 + +/* first mbuf in the packet, should always be at offset 0 */ +struct dummy_mbuf { + struct rte_mbuf mb[DUMMY_MBUF_NUM]; + uint8_t buf[DUMMY_MBUF_NUM][RTE_MBUF_DEFAULT_BUF_SIZE]; +}; + #define TEST_FILL_1 0xDEADBEEF #define TEST_MUL_1 21 @@ -2444,6 +2452,413 @@ static const struct rte_bpf_xsym test_call5_xsym[] = { }, }; +/* load mbuf (BPF_ABS/BPF_IND) test-cases */ +static const struct ebpf_insn test_ld_mbuf1_prog[] = { + + /* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_6, + .src_reg = EBPF_REG_1, + }, + /* load IPv4 version and IHL */ + { + .code = (BPF_LD | BPF_ABS | BPF_B), + .imm = offsetof(struct rte_ipv4_hdr, version_ihl), + }, + /* check IP version */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_2, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = 0xf0, + }, + { + .code = (BPF_JMP | BPF_JEQ | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = IPVERSION << 4, + .off = 2, + }, + /* invalid IP version, return 0 */ + { + .code = (EBPF_ALU64 | BPF_XOR | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, + /* load 3-rd byte of IP data */ + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = RTE_IPV4_HDR_IHL_MASK, + }, + { + .code = (BPF_ALU | BPF_LSH | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = 2, + }, + { + .code = (BPF_LD | BPF_IND | BPF_B), + .src_reg = EBPF_REG_0, + .imm = 3, + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load IPv4 src addr */ + { + .code = (BPF_LD | BPF_ABS | BPF_W), + .imm = offsetof(struct rte_ipv4_hdr, src_addr), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load IPv4 total length */ + { + .code = (BPF_LD | BPF_ABS | BPF_H), + .imm = offsetof(struct rte_ipv4_hdr, total_length), + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_8, + .src_reg = EBPF_REG_0, + }, + /* load last 4 bytes of IP data */ + { + .code = (BPF_LD | BPF_IND | BPF_W), + .src_reg = EBPF_REG_8, + .imm = -(int32_t)sizeof(uint32_t), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_7, + .src_reg = EBPF_REG_0, + }, + /* load 2 bytes from the middle of IP data */ + { + .code = (EBPF_ALU64 | BPF_RSH | BPF_K), + .dst_reg = EBPF_REG_8, + .imm = 1, + }, + { + .code = (BPF_LD | BPF_IND | BPF_H), + .src_reg = EBPF_REG_8, + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_7, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, +}; + +static void +dummy_mbuf_prep(struct rte_mbuf *mb, uint8_t buf[], uint32_t buf_len, + uint32_t data_len) +{ + uint32_t i; + uint8_t *db; + + mb->buf_addr = buf; + mb->buf_iova = (uintptr_t)buf; + mb->buf_len = buf_len; + rte_mbuf_refcnt_set(mb, 1); + + /* set pool pointer to dummy value, test doesn't use it */ + mb->pool = (void *)buf; + + rte_pktmbuf_reset(mb); + db = (uint8_t *)rte_pktmbuf_append(mb, data_len); + + for (i = 0; i != data_len; i++) + db[i] = i; +} + +static void +test_ld_mbuf1_prepare(void *arg) +{ + struct dummy_mbuf *dm; + struct rte_ipv4_hdr *ph; + + const uint32_t plen = 400; + const struct rte_ipv4_hdr iph = { + .version_ihl = RTE_IPV4_VHL_DEF, + .total_length = rte_cpu_to_be_16(plen), + .time_to_live = IPDEFTTL, + .next_proto_id = IPPROTO_RAW, + .src_addr = rte_cpu_to_be_32(RTE_IPV4_LOOPBACK), + .dst_addr = rte_cpu_to_be_32(RTE_IPV4_BROADCAST), + }; + + dm = arg; + memset(dm, 0, sizeof(*dm)); + + dummy_mbuf_prep(&dm->mb[0], dm->buf[0], sizeof(dm->buf[0]), + plen / 2 + 1); + dummy_mbuf_prep(&dm->mb[1], dm->buf[1], sizeof(dm->buf[0]), + plen / 2 - 1); + + rte_pktmbuf_chain(&dm->mb[0], &dm->mb[1]); + + ph = rte_pktmbuf_mtod(dm->mb, typeof(ph)); + memcpy(ph, &iph, sizeof(iph)); +} + +static uint64_t +test_ld_mbuf1(const struct rte_mbuf *pkt) +{ + uint64_t n, v; + const uint8_t *p8; + const uint16_t *p16; + const uint32_t *p32; + struct dummy_offset dof; + + /* load IPv4 version and IHL */ + p8 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, version_ihl), sizeof(*p8), + &dof); + if (p8 == NULL) + return 0; + + /* check IP version */ + if ((p8[0] & 0xf0) != IPVERSION << 4) + return 0; + + n = (p8[0] & RTE_IPV4_HDR_IHL_MASK) * RTE_IPV4_IHL_MULTIPLIER; + + /* load 3-rd byte of IP data */ + p8 = rte_pktmbuf_read(pkt, n + 3, sizeof(*p8), &dof); + if (p8 == NULL) + return 0; + + v = p8[0]; + + /* load IPv4 src addr */ + p32 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, src_addr), sizeof(*p32), + &dof); + if (p32 == NULL) + return 0; + + v += rte_be_to_cpu_32(p32[0]); + + /* load IPv4 total length */ + p16 = rte_pktmbuf_read(pkt, + offsetof(struct rte_ipv4_hdr, total_length), sizeof(*p16), + &dof); + if (p16 == NULL) + return 0; + + n = rte_be_to_cpu_16(p16[0]); + + /* load last 4 bytes of IP data */ + p32 = rte_pktmbuf_read(pkt, n - sizeof(*p32), sizeof(*p32), &dof); + if (p32 == NULL) + return 0; + + v += rte_be_to_cpu_32(p32[0]); + + /* load 2 bytes from the middle of IP data */ + p16 = rte_pktmbuf_read(pkt, n / 2, sizeof(*p16), &dof); + if (p16 == NULL) + return 0; + + v += rte_be_to_cpu_16(p16[0]); + return v; +} + +static int +test_ld_mbuf1_check(uint64_t rc, const void *arg) +{ + const struct dummy_mbuf *dm; + uint64_t v; + + dm = arg; + v = test_ld_mbuf1(dm->mb); + return cmp_res(__func__, v, rc, arg, arg, 0); +} + +/* + * same as ld_mbuf1, but then trancate the mbuf by 1B, + * so load of last 4B fail. + */ +static void +test_ld_mbuf2_prepare(void *arg) +{ + struct dummy_mbuf *dm; + + test_ld_mbuf1_prepare(arg); + dm = arg; + rte_pktmbuf_trim(dm->mb, 1); +} + +static int +test_ld_mbuf2_check(uint64_t rc, const void *arg) +{ + return cmp_res(__func__, 0, rc, arg, arg, 0); +} + +/* same as test_ld_mbuf1, but now store intermediate results on the stack */ +static const struct ebpf_insn test_ld_mbuf3_prog[] = { + + /* BPF_ABS/BPF_IND implicitly expect mbuf ptr in R6 */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_6, + .src_reg = EBPF_REG_1, + }, + /* load IPv4 version and IHL */ + { + .code = (BPF_LD | BPF_ABS | BPF_B), + .imm = offsetof(struct rte_ipv4_hdr, version_ihl), + }, + /* check IP version */ + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_2, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = 0xf0, + }, + { + .code = (BPF_JMP | BPF_JEQ | BPF_K), + .dst_reg = EBPF_REG_2, + .imm = IPVERSION << 4, + .off = 2, + }, + /* invalid IP version, return 0 */ + { + .code = (EBPF_ALU64 | BPF_XOR | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_0, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, + /* load 3-rd byte of IP data */ + { + .code = (BPF_ALU | BPF_AND | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = RTE_IPV4_HDR_IHL_MASK, + }, + { + .code = (BPF_ALU | BPF_LSH | BPF_K), + .dst_reg = EBPF_REG_0, + .imm = 2, + }, + { + .code = (BPF_LD | BPF_IND | BPF_B), + .src_reg = EBPF_REG_0, + .imm = 3, + }, + { + .code = (BPF_STX | BPF_MEM | BPF_B), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u8) - + sizeof(struct dummy_offset)), + }, + /* load IPv4 src addr */ + { + .code = (BPF_LD | BPF_ABS | BPF_W), + .imm = offsetof(struct rte_ipv4_hdr, src_addr), + }, + { + .code = (BPF_STX | BPF_MEM | BPF_W), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u32) - + sizeof(struct dummy_offset)), + }, + /* load IPv4 total length */ + { + .code = (BPF_LD | BPF_ABS | BPF_H), + .imm = offsetof(struct rte_ipv4_hdr, total_length), + }, + { + .code = (EBPF_ALU64 | EBPF_MOV | BPF_X), + .dst_reg = EBPF_REG_8, + .src_reg = EBPF_REG_0, + }, + /* load last 4 bytes of IP data */ + { + .code = (BPF_LD | BPF_IND | BPF_W), + .src_reg = EBPF_REG_8, + .imm = -(int32_t)sizeof(uint32_t), + }, + { + .code = (BPF_STX | BPF_MEM | EBPF_DW), + .dst_reg = EBPF_REG_10, + .src_reg = EBPF_REG_0, + .off = (int16_t)(offsetof(struct dummy_offset, u64) - + sizeof(struct dummy_offset)), + }, + /* load 2 bytes from the middle of IP data */ + { + .code = (EBPF_ALU64 | BPF_RSH | BPF_K), + .dst_reg = EBPF_REG_8, + .imm = 1, + }, + { + .code = (BPF_LD | BPF_IND | BPF_H), + .src_reg = EBPF_REG_8, + }, + { + .code = (BPF_LDX | BPF_MEM | EBPF_DW), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u64) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_LDX | BPF_MEM | BPF_W), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u32) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_LDX | BPF_MEM | BPF_B), + .dst_reg = EBPF_REG_1, + .src_reg = EBPF_REG_10, + .off = (int16_t)(offsetof(struct dummy_offset, u8) - + sizeof(struct dummy_offset)), + }, + { + .code = (EBPF_ALU64 | BPF_ADD | BPF_X), + .dst_reg = EBPF_REG_0, + .src_reg = EBPF_REG_1, + }, + { + .code = (BPF_JMP | EBPF_EXIT), + }, +}; + /* all bpf test cases */ static const struct bpf_test tests[] = { { @@ -2704,6 +3119,54 @@ static const struct bpf_test tests[] = { /* for now don't support function calls on 32 bit platform */ .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), }, + { + .name = "test_ld_mbuf1", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf1_prog, + .nb_ins = RTE_DIM(test_ld_mbuf1_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf1_prepare, + .check_result = test_ld_mbuf1_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, + { + .name = "test_ld_mbuf2", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf1_prog, + .nb_ins = RTE_DIM(test_ld_mbuf1_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf2_prepare, + .check_result = test_ld_mbuf2_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, + { + .name = "test_ld_mbuf3", + .arg_sz = sizeof(struct dummy_mbuf), + .prm = { + .ins = test_ld_mbuf3_prog, + .nb_ins = RTE_DIM(test_ld_mbuf3_prog), + .prog_arg = { + .type = RTE_BPF_ARG_PTR_MBUF, + .buf_size = sizeof(struct dummy_mbuf), + }, + }, + .prepare = test_ld_mbuf1_prepare, + .check_result = test_ld_mbuf1_check, + /* mbuf as input argument is not supported on 32 bit platform */ + .allow_fail = (sizeof(uint64_t) != sizeof(uintptr_t)), + }, }; static int -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data load instructions 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (3 preceding siblings ...) 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev @ 2020-05-27 14:16 ` Konstantin Ananyev 2020-05-28 8:50 ` Morten Brørup 2020-06-24 21:43 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon 5 siblings, 1 reply; 17+ messages in thread From: Konstantin Ananyev @ 2020-05-27 14:16 UTC (permalink / raw) To: dev; +Cc: jerinj, stephen, mb, Konstantin Ananyev Make x86 JIT to generate native code for (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) instructions. Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> --- lib/librte_bpf/bpf_jit_x86.c | 181 ++++++++++++++++++++++++++++++++++ lib/librte_bpf/bpf_validate.c | 17 +++- 2 files changed, 197 insertions(+), 1 deletion(-) diff --git a/lib/librte_bpf/bpf_jit_x86.c b/lib/librte_bpf/bpf_jit_x86.c index f70cd6be5..aa22ea78a 100644 --- a/lib/librte_bpf/bpf_jit_x86.c +++ b/lib/librte_bpf/bpf_jit_x86.c @@ -87,6 +87,14 @@ enum { REG_TMP1 = R10, }; +/* LD_ABS/LD_IMM offsets */ +enum { + LDMB_FSP_OFS, /* fast-path */ + LDMB_SLP_OFS, /* slow-path */ + LDMB_FIN_OFS, /* final part */ + LDMB_OFS_NUM +}; + /* * callee saved registers list. * keep RBP as the last one. @@ -100,6 +108,9 @@ struct bpf_jit_state { uint32_t num; int32_t off; } exit; + struct { + uint32_t stack_ofs; + } ldmb; uint32_t reguse; int32_t *off; uint8_t *ins; @@ -1024,6 +1035,166 @@ emit_div(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t dreg, emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, REG_TMP1, RDX); } +/* + * helper function, used by emit_ld_mbuf(). + * generates code for 'fast_path': + * calculate load offset and check is it inside first packet segment. + */ +static void +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7], + uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm, + const int32_t ofs[LDMB_OFS_NUM]) +{ + /* make R2 contain *off* value */ + + if (sreg != rg[EBPF_REG_2]) { + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, + rg[EBPF_REG_2], imm); + if (mode == BPF_IND) + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + sreg, rg[EBPF_REG_2]); + } else + /* BPF_IND with sreg == R2 */ + emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K, + rg[EBPF_REG_2], imm); + + /* R3 = mbuf->data_len */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, + rg[EBPF_REG_6], rg[EBPF_REG_3], + offsetof(struct rte_mbuf, data_len)); + + /* R3 = R3 - R2 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X, + rg[EBPF_REG_2], rg[EBPF_REG_3]); + + /* JSLT R3, <sz> <slow_path> */ + emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz); + emit_abs_jcc(st, BPF_JMP | EBPF_JSLT | BPF_K, ofs[LDMB_SLP_OFS]); + + /* R3 = mbuf->data_off */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, + rg[EBPF_REG_6], rg[EBPF_REG_3], + offsetof(struct rte_mbuf, data_off)); + + /* R0 = mbuf->buf_addr */ + emit_ld_reg(st, BPF_LDX | BPF_MEM | EBPF_DW, + rg[EBPF_REG_6], rg[EBPF_REG_0], + offsetof(struct rte_mbuf, buf_addr)); + + /* R0 = R0 + R3 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + rg[EBPF_REG_3], rg[EBPF_REG_0]); + + /* R0 = R0 + R2 */ + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, + rg[EBPF_REG_2], rg[EBPF_REG_0]); + + /* JMP <fin_part> */ + emit_abs_jmp(st, ofs[LDMB_FIN_OFS]); +} + +/* + * helper function, used by emit_ld_mbuf(). + * generates code for 'slow_path': + * call __rte_pktmbuf_read() and check return value. + */ +static void +emit_ldmb_slow_path(struct bpf_jit_state *st, const uint32_t rg[EBPF_REG_7], + uint32_t sz) +{ + /* make R3 contain *len* value (1/2/4) */ + + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, rg[EBPF_REG_3], sz); + + /* make R4 contain (RBP - ldmb.stack_ofs) */ + + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, RBP, rg[EBPF_REG_4]); + emit_alu_imm(st, EBPF_ALU64 | BPF_SUB | BPF_K, rg[EBPF_REG_4], + st->ldmb.stack_ofs); + + /* make R1 contain mbuf ptr */ + + emit_mov_reg(st, EBPF_ALU64 | EBPF_MOV | BPF_X, + rg[EBPF_REG_6], rg[EBPF_REG_1]); + + /* call rte_pktmbuf_read */ + emit_call(st, (uintptr_t)__rte_pktmbuf_read); + + /* check that return value (R0) is not zero */ + emit_tst_reg(st, EBPF_ALU64, rg[EBPF_REG_0], rg[EBPF_REG_0]); + emit_abs_jcc(st, BPF_JMP | BPF_JEQ | BPF_K, st->exit.off); +} + +/* + * helper function, used by emit_ld_mbuf(). + * generates final part of code for BPF_ABS/BPF_IND load: + * perform data load and endianness conversion. + * expects dreg to contain valid data pointer. + */ +static void +emit_ldmb_fin(struct bpf_jit_state *st, uint32_t dreg, uint32_t opsz, + uint32_t sz) +{ + emit_ld_reg(st, BPF_LDX | BPF_MEM | opsz, dreg, dreg, 0); + if (sz != sizeof(uint8_t)) + emit_be2le(st, dreg, sz * CHAR_BIT); +} + +/* + * emit code for BPF_ABS/BPF_IND load. + * generates the following construction: + * fast_path: + * off = ins->sreg + ins->imm + * if (mbuf->data_len - off < ins->opsz) + * goto slow_path; + * ptr = mbuf->buf_addr + mbuf->data_off + off; + * goto fin_part; + * slow_path: + * typeof(ins->opsz) buf; //allocate space on the stack + * ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf); + * if (ptr == NULL) + * goto exit_label; + * fin_part: + * res = *(typeof(ins->opsz))ptr; + * res = bswap(res); + */ +static void +emit_ld_mbuf(struct bpf_jit_state *st, uint32_t op, uint32_t sreg, uint32_t imm) +{ + uint32_t i, mode, opsz, sz; + uint32_t rg[EBPF_REG_7]; + int32_t ofs[LDMB_OFS_NUM]; + + mode = BPF_MODE(op); + opsz = BPF_SIZE(op); + sz = bpf_size(opsz); + + for (i = 0; i != RTE_DIM(rg); i++) + rg[i] = ebpf2x86[i]; + + /* fill with fake offsets */ + for (i = 0; i != RTE_DIM(ofs); i++) + ofs[i] = st->sz + INT8_MAX; + + /* dry run first to calculate jump offsets */ + + ofs[LDMB_FSP_OFS] = st->sz; + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); + ofs[LDMB_SLP_OFS] = st->sz; + emit_ldmb_slow_path(st, rg, sz); + ofs[LDMB_FIN_OFS] = st->sz; + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); + + RTE_VERIFY(ofs[LDMB_FIN_OFS] - ofs[LDMB_FSP_OFS] <= INT8_MAX); + + /* reset dry-run code and do a proper run */ + + st->sz = ofs[LDMB_FSP_OFS]; + emit_ldmb_fast_path(st, rg, sreg, mode, sz, imm, ofs); + emit_ldmb_slow_path(st, rg, sz); + emit_ldmb_fin(st, rg[EBPF_REG_0], opsz, sz); +} + static void emit_prolog(struct bpf_jit_state *st, int32_t stack_size) { @@ -1121,6 +1292,7 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf) /* reset state fields */ st->sz = 0; st->exit.num = 0; + st->ldmb.stack_ofs = bpf->stack_sz; emit_prolog(st, bpf->stack_sz); @@ -1240,6 +1412,15 @@ emit(struct bpf_jit_state *st, const struct rte_bpf *bpf) emit_ld_imm64(st, dr, ins[0].imm, ins[1].imm); i++; break; + /* load absolute/indirect instructions */ + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): + emit_ld_mbuf(st, op, sr, ins->imm); + break; /* store instructions */ case (BPF_STX | BPF_MEM | BPF_B): case (BPF_STX | BPF_MEM | BPF_H): diff --git a/lib/librte_bpf/bpf_validate.c b/lib/librte_bpf/bpf_validate.c index fecdda0e1..9214f1503 100644 --- a/lib/librte_bpf/bpf_validate.c +++ b/lib/librte_bpf/bpf_validate.c @@ -70,6 +70,7 @@ struct bpf_verifier { uint64_t stack_sz; uint32_t nb_nodes; uint32_t nb_jcc_nodes; + uint32_t nb_ldmb_nodes; uint32_t node_colour[MAX_NODE_COLOUR]; uint32_t edge_type[MAX_EDGE_TYPE]; struct bpf_eval_state *evst; @@ -2020,6 +2021,14 @@ validate(struct bpf_verifier *bvf) rc |= add_edge(bvf, node, i + 2); i++; break; + case (BPF_LD | BPF_ABS | BPF_B): + case (BPF_LD | BPF_ABS | BPF_H): + case (BPF_LD | BPF_ABS | BPF_W): + case (BPF_LD | BPF_IND | BPF_B): + case (BPF_LD | BPF_IND | BPF_H): + case (BPF_LD | BPF_IND | BPF_W): + bvf->nb_ldmb_nodes++; + /* fallthrough */ default: rc |= add_edge(bvf, node, i + 1); break; @@ -2320,8 +2329,14 @@ bpf_validate(struct rte_bpf *bpf) free(bvf.in); /* copy collected info */ - if (rc == 0) + if (rc == 0) { bpf->stack_sz = bvf.stack_sz; + /* for LD_ABS/LD_IND, we'll need extra space on the stack */ + if (bvf.nb_ldmb_nodes != 0) + bpf->stack_sz = RTE_ALIGN_CEIL(bpf->stack_sz + + sizeof(uint64_t), sizeof(uint64_t)); + } + return rc; } -- 2.17.1 ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data load instructions 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev @ 2020-05-28 8:50 ` Morten Brørup 0 siblings, 0 replies; 17+ messages in thread From: Morten Brørup @ 2020-05-28 8:50 UTC (permalink / raw) To: Konstantin Ananyev, dev; +Cc: jerinj, stephen > From: Konstantin Ananyev [mailto:konstantin.ananyev@intel.com] > Sent: Wednesday, May 27, 2020 4:17 PM > > Make x86 JIT to generate native code for > (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) > instructions. > > Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com> > --- [...] > +/* > + * helper function, used by emit_ld_mbuf(). > + * generates code for 'fast_path': > + * calculate load offset and check is it inside first packet segment. > + */ > +static void > +emit_ldmb_fast_path(struct bpf_jit_state *st, const uint32_t > rg[EBPF_REG_7], > + uint32_t sreg, uint32_t mode, uint32_t sz, uint32_t imm, > + const int32_t ofs[LDMB_OFS_NUM]) > +{ > + /* make R2 contain *off* value */ > + > + if (sreg != rg[EBPF_REG_2]) { > + emit_mov_imm(st, EBPF_ALU64 | EBPF_MOV | BPF_K, > + rg[EBPF_REG_2], imm); > + if (mode == BPF_IND) > + emit_alu_reg(st, EBPF_ALU64 | BPF_ADD | BPF_X, > + sreg, rg[EBPF_REG_2]); > + } else > + /* BPF_IND with sreg == R2 */ > + emit_alu_imm(st, EBPF_ALU64 | BPF_ADD | BPF_K, > + rg[EBPF_REG_2], imm); > + > + /* R3 = mbuf->data_len */ > + emit_ld_reg(st, BPF_LDX | BPF_MEM | BPF_H, > + rg[EBPF_REG_6], rg[EBPF_REG_3], > + offsetof(struct rte_mbuf, data_len)); > + > + /* R3 = R3 - R2 */ > + emit_alu_reg(st, EBPF_ALU64 | BPF_SUB | BPF_X, > + rg[EBPF_REG_2], rg[EBPF_REG_3]); > + > + /* JSLT R3, <sz> <slow_path> */ > + emit_cmp_imm(st, EBPF_ALU64, rg[EBPF_REG_3], sz); > + emit_abs_jcc(st, BPF_JMP | EBPF_JSLT | BPF_K, ofs[LDMB_SLP_OFS]); > + [...] > + > +/* > + * emit code for BPF_ABS/BPF_IND load. > + * generates the following construction: > + * fast_path: > + * off = ins->sreg + ins->imm > + * if (mbuf->data_len - off < ins->opsz) > + * goto slow_path; > + * ptr = mbuf->buf_addr + mbuf->data_off + off; > + * goto fin_part; > + * slow_path: > + * typeof(ins->opsz) buf; //allocate space on the stack > + * ptr = __rte_pktmbuf_read(mbuf, off, ins->opsz, &buf); > + * if (ptr == NULL) > + * goto exit_label; > + * fin_part: > + * res = *(typeof(ins->opsz))ptr; > + * res = bswap(res); > + */ [...] The comparison for jumping to the slow path looks correct now. I haven't reviewed it all in depth, but it certainly deserves an: Acked-by: Morten Brørup <mb@smartsharesystems.com> ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev ` (4 preceding siblings ...) 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev @ 2020-06-24 21:43 ` Thomas Monjalon 5 siblings, 0 replies; 17+ messages in thread From: Thomas Monjalon @ 2020-06-24 21:43 UTC (permalink / raw) To: Konstantin Ananyev; +Cc: dev, jerinj, stephen, mb 27/05/2020 16:16, Konstantin Ananyev: > To fill the gap with linux kernel eBPF implementation, > add support for two non-generic instructions: > (BPF_ABS | <size> | BPF_LD) and (BPF_IND | <size> | BPF_LD) > which are used to access packet data in a safe manner. > Make necessary changes in BPF verifier, interpreter and x86 JIT code. > > Konstantin Ananyev (5): > test/bpf: fix few small issues > bpf: fix add/sub min/max estimations > bpf: add support for packet data load instructions > test/bpf: add new test cases for mbuf load instructions > bpf: x86 JIT support for packet data load instructions Applied, thanks ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-06-24 21:43 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-05-18 15:52 [dpdk-dev] [PATCH 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 1/5] test/bpf: fix few small issues Konstantin Ananyev 2020-06-24 21:33 ` [dpdk-dev] [dpdk-stable] " Thomas Monjalon 2020-05-18 15:52 ` [dpdk-dev] [PATCH 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 3/5] bpf: add support for packet data load instructions Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev 2020-05-18 15:52 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev 2020-05-24 13:37 ` [dpdk-dev] [PATCH 5/5] bpf: x86 JIT support for packet data loadinstructions Morten Brørup 2020-05-24 23:08 ` Ananyev, Konstantin 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 1/5] test/bpf: fix few small issues Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 2/5] bpf: fix add/sub min/max estimations Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 3/5] bpf: add support for packet data load instructions Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 4/5] test/bpf: add new test cases for mbuf " Konstantin Ananyev 2020-05-27 14:16 ` [dpdk-dev] [PATCH v2 5/5] bpf: x86 JIT support for packet data " Konstantin Ananyev 2020-05-28 8:50 ` Morten Brørup 2020-06-24 21:43 ` [dpdk-dev] [PATCH v2 0/5] bpf: add support for BPF_ABS/BPF_IND instructions Thomas Monjalon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).