From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0D3CB42CC4; Thu, 15 Jun 2023 10:07:07 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EACEB410D0; Thu, 15 Jun 2023 10:07:06 +0200 (CEST) Received: from mail-ua1-f48.google.com (mail-ua1-f48.google.com [209.85.222.48]) by mails.dpdk.org (Postfix) with ESMTP id D383240A84 for ; Thu, 15 Jun 2023 10:07:04 +0200 (CEST) Received: by mail-ua1-f48.google.com with SMTP id a1e0cc1a2514c-78a1e095508so1111759241.0 for ; Thu, 15 Jun 2023 01:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1686816424; x=1689408424; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=k849m19mQ9rxfgrADe+4cqGl7FPfS4iCabDwhxubTS8=; b=E/hLoonaZWhVb303wb+jTVwufRW36Wur7Mg64x65iM+xzG9248tZlaQkxRSsD1GvDR dwpFtsAr4Ak/jlb6CXdup3/SVTiO3jVRkz1b/cMQ6H7LvPr+u+YQscG1+G/RrbPDNtTh LLd57lS2b/Fl4ymzQDFwfsqRnbGKXGOLS/pYVGTQ2vFGJH8J3N3NoDAC/CAnxKyDPiu5 wghf93GQTiUU3AQGCx8S3eOqQZHFzCECDaLxThaqbisySRoSliQMhXFqHkhUPm79j5A/ LxQ4t8UZ4KOSw1O2G6eiudKiF7fVvgfslrHqGtUpO/nKnOgyow0MtKxyhtf9lYV52K3k WsaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686816424; x=1689408424; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k849m19mQ9rxfgrADe+4cqGl7FPfS4iCabDwhxubTS8=; b=AtJzqWgqDPlQ/XKU8TrYBWg75pOpE9yl3Qhc0pB7HFGmQNkP+TdMVhf0SMbg7iIdMi g4fKG3/8KSITkTHujpJU+/HC4p3Af+awObn0CzLK7wHn3N8WdwiawmSmKKrN/9ZeVXaE sRo87Vy8cHRoVi0bX8s526rM9NoZfqG3Jealch+N4J6S9EtRJub33qn1BepiosTZLgVu DNioUGIuV9GvBZ2TpfMVa/tue2xuVvg/iUghCpqhdQCnQI+be0w2R7hP97JxGFeXsRyt wb1RxRJSXTkG7zOw1ZQK01sNM1qbuBk8ChJk2SfJ2rk+R77rtlD0eiu6R921hgTboYwb z+7Q== X-Gm-Message-State: AC+VfDyiYcAdxwzNeTDImPViO2ily6IVAi7QN2eB624eb1HG4E1sYzFx hqug2X2pYUTxK7rmy5/Rp3UhDNKJEw2nnsD0vv5nbYN5Ww/jbyLp X-Google-Smtp-Source: ACHHUZ5XGYHu8M155zoQis1dCktuVzPG624WikAxYoAzhe1SotF/wIqXxuC1mc3PCLv9781AQYI33qZP986jSqABiMA= X-Received: by 2002:a1f:bd0e:0:b0:46e:9c55:e800 with SMTP id n14-20020a1fbd0e000000b0046e9c55e800mr1910484vkf.16.1686816423488; Thu, 15 Jun 2023 01:07:03 -0700 (PDT) MIME-Version: 1.0 References: <20230613150904.765725-1-ndabilpuram@marvell.com> In-Reply-To: <20230613150904.765725-1-ndabilpuram@marvell.com> From: Jerin Jacob Date: Thu, 15 Jun 2023 13:36:36 +0530 Message-ID: Subject: Re: [PATCH v3] net/cnxk: add support for reassembly of multi-seg pkts To: Nithin Dabilpuram Cc: Pavan Nikhilesh , Shijith Thotton , Kiran Kumar K , Sunil Kumar Kori , Satha Rao , jerinj@marvell.com, dev@dpdk.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Jun 13, 2023 at 8:39=E2=80=AFPM Nithin Dabilpuram wrote: > > Add support for HW reassembly of multi-seg pkts. > Also optimize the code for normal reassembly path. > > Signed-off-by: Nithin Dabilpuram Updated the release notes as folllowing diff --git a/doc/guides/rel_notes/release_23_07.rst b/doc/guides/rel_notes/release_23_07.rst index d6af8c25a7..e0b706b540 100644 --- a/doc/guides/rel_notes/release_23_07.rst +++ b/doc/guides/rel_notes/release_23_07.rst @@ -144,6 +144,7 @@ New Features * Extended ``RTE_FLOW_ACTION_TYPE_PORT_ID`` to redirect traffic across PF ports. * Added support for Inline MACsec processing using rte_security framewor= k for CN103 platform. + * Added support for reassembly of multi-seg packets. Updated the git commit as follows and applied to dpdk-next-net-mrvl/for-next-net. Thanks net/cnxk: support reassembly of multi-seg packets Add support for HW reassembly of multi-seg packets. Also optimize the code for normal reassembly path. Signed-off-by: Nithin Dabilpuram > --- > v3: > - No changes, resend after rebase > > v2: > - Removed change id > > drivers/event/cnxk/cn10k_worker.h | 21 +- > drivers/net/cnxk/cn10k_rx.h | 714 +++++++++++++++--------------- > 2 files changed, 375 insertions(+), 360 deletions(-) > > diff --git a/drivers/event/cnxk/cn10k_worker.h b/drivers/event/cnxk/cn10k= _worker.h > index a01894ae10..f5d7d879a7 100644 > --- a/drivers/event/cnxk/cn10k_worker.h > +++ b/drivers/event/cnxk/cn10k_worker.h > @@ -16,7 +16,7 @@ > static __rte_always_inline void > cn10k_wqe_to_mbuf(uint64_t wqe, const uint64_t __mbuf, uint8_t port_id, > const uint32_t tag, const uint32_t flags, > - const void *const lookup_mem) > + const void *const lookup_mem, uintptr_t cpth, uintptr_t= sa_base) > { > const uint64_t mbuf_init =3D 0x100010000ULL | RTE_PKTMBUF_HEADROO= M | > (flags & NIX_RX_OFFLOAD_TSTAMP_F ? 8 := 0); > @@ -24,7 +24,7 @@ cn10k_wqe_to_mbuf(uint64_t wqe, const uint64_t __mbuf, = uint8_t port_id, > > cn10k_nix_cqe_to_mbuf((struct nix_cqe_hdr_s *)wqe, tag, > (struct rte_mbuf *)mbuf, lookup_mem, > - mbuf_init | ((uint64_t)port_id) << 48, flag= s); > + mbuf_init | ((uint64_t)port_id) << 48, cpth= , sa_base, flags); > } > > static void > @@ -59,6 +59,7 @@ cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id, co= nst uint32_t flags, struc > uint16_t lmt_id, d_off; > struct rte_mbuf **wqe; > struct rte_mbuf *mbuf; > + uintptr_t cpth =3D 0; > uint8_t loff =3D 0; > uint64_t sa_base; > int i; > @@ -122,13 +123,15 @@ cn10k_process_vwqe(uintptr_t vwqe, uint16_t port_id= , const uint32_t flags, struc > const uint64_t cq_w1 =3D *((const uint64_t *)cqe = + 1); > const uint64_t cq_w5 =3D *((const uint64_t *)cqe = + 5); > > + cpth =3D ((uintptr_t)mbuf + (uint16_t)d_off); > + > mbuf =3D nix_sec_meta_to_mbuf_sc(cq_w1, cq_w5, sa= _base, laddr, > &loff, mbuf, d_off= , > flags, mbuf_init); > } > > cn10k_nix_cqe_to_mbuf(cqe, cqe->tag, mbuf, lookup_mem, > - mbuf_init, flags); > + mbuf_init, cpth, sa_base, flags); > > if (flags & NIX_RX_OFFLOAD_TSTAMP_F) > cn10k_sso_process_tstamp((uint64_t)wqe[0], > @@ -149,6 +152,8 @@ static __rte_always_inline void > cn10k_sso_hws_post_process(struct cn10k_sso_hws *ws, uint64_t *u64, > const uint32_t flags) > { > + uintptr_t sa_base =3D 0; > + > u64[0] =3D (u64[0] & (0x3ull << 32)) << 6 | > (u64[0] & (0x3FFull << 36)) << 4 | (u64[0] & 0xffffffff)= ; > if (CNXK_EVENT_TYPE_FROM_TAG(u64[0]) =3D=3D RTE_EVENT_TYPE_CRYPTO= DEV) { > @@ -157,6 +162,7 @@ cn10k_sso_hws_post_process(struct cn10k_sso_hws *ws, = uint64_t *u64, > u64[1] =3D cn10k_cpt_crypto_adapter_vector_dequeue(u64[1]= ); > } else if (CNXK_EVENT_TYPE_FROM_TAG(u64[0]) =3D=3D RTE_EVENT_TYPE= _ETHDEV) { > uint8_t port =3D CNXK_SUB_EVENT_FROM_TAG(u64[0]); > + uintptr_t cpth =3D 0; > uint64_t mbuf; > > mbuf =3D u64[1] - sizeof(struct rte_mbuf); > @@ -174,7 +180,6 @@ cn10k_sso_hws_post_process(struct cn10k_sso_hws *ws, = uint64_t *u64, > 0x100010000ULL | RTE_PKTMBUF_HEADROOM | > (flags & NIX_RX_OFFLOAD_TSTAMP_F ? 8 : 0)= ; > struct rte_mbuf *m; > - uintptr_t sa_base; > uint64_t iova =3D 0; > uint8_t loff =3D 0; > uint16_t d_off; > @@ -190,12 +195,14 @@ cn10k_sso_hws_post_process(struct cn10k_sso_hws *ws= , uint64_t *u64, > sa_base =3D cnxk_nix_sa_base_get(port, ws->lookup= _mem); > sa_base &=3D ~(ROC_NIX_INL_SA_BASE_ALIGN - 1); > > + cpth =3D ((uintptr_t)mbuf + (uint16_t)d_off); > + mp =3D (struct rte_mempool *)cnxk_nix_inl_metapoo= l_get(port, lookup_mem); > + meta_aura =3D mp ? mp->pool_id : m->pool->pool_id= ; > + > mbuf =3D (uint64_t)nix_sec_meta_to_mbuf_sc( > cq_w1, cq_w5, sa_base, (uintptr_t)&iova, = &loff, > (struct rte_mbuf *)mbuf, d_off, flags, > mbuf_init | ((uint64_t)port) << 48); > - mp =3D (struct rte_mempool *)cnxk_nix_inl_metapoo= l_get(port, lookup_mem); > - meta_aura =3D mp ? mp->pool_id : m->pool->pool_id= ; > > if (loff) > roc_npa_aura_op_free(meta_aura, 0, iova); > @@ -203,7 +210,7 @@ cn10k_sso_hws_post_process(struct cn10k_sso_hws *ws, = uint64_t *u64, > > u64[0] =3D CNXK_CLR_SUB_EVENT(u64[0]); > cn10k_wqe_to_mbuf(u64[1], mbuf, port, u64[0] & 0xFFFFF, f= lags, > - ws->lookup_mem); > + ws->lookup_mem, cpth, sa_base); > if (flags & NIX_RX_OFFLOAD_TSTAMP_F) > cn10k_sso_process_tstamp(u64[1], mbuf, > ws->tstamp[port]); > diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h > index 68c8c73790..b7074906a6 100644 > --- a/drivers/net/cnxk/cn10k_rx.h > +++ b/drivers/net/cnxk/cn10k_rx.h > @@ -144,18 +144,126 @@ nix_sec_flush_meta(uintptr_t laddr, uint16_t lmt_i= d, uint8_t loff, > roc_lmt_submit_steorl(lmt_id, pa); > } > > +#if defined(RTE_ARCH_ARM64) > +static __rte_always_inline uint64_t > +nix_sec_reass_frags_get(const struct cpt_parse_hdr_s *hdr, struct rte_mb= uf **next_mbufs) > +{ > + const struct cpt_frag_info_s *finfo; > + uint32_t offset =3D hdr->w2.fi_offset; > + const uint64_t *frag_ptr; > + uint64x2_t frags23; > + uint16x4_t fsz_w1; > + > + /* offset of 0 implies 256B, otherwise it implies offset*8B */ > + offset =3D (((offset - 1) & 0x1f) + 1) * 8; > + finfo =3D RTE_PTR_ADD(hdr, offset); > + frag_ptr =3D (const uint64_t *)(finfo + 1); > + frags23 =3D vrev64q_u8(vld1q_u64(frag_ptr)); > + > + next_mbufs[0] =3D ((struct rte_mbuf *)rte_be_to_cpu_64(hdr->frag1= _wqe_ptr) - 1); > + next_mbufs[1] =3D ((struct rte_mbuf *)vgetq_lane_u64(frags23, 0) = - 1); > + next_mbufs[2] =3D ((struct rte_mbuf *)vgetq_lane_u64(frags23, 1) = - 1); > + > + fsz_w1 =3D vdup_n_u64(finfo->w1.u64); > + fsz_w1 =3D vrev16_u8(fsz_w1); > + return vget_lane_u64(fsz_w1, 0); > +} > + > +static __rte_always_inline void > +nix_sec_reass_first_frag_update(struct rte_mbuf *head, const uint8_t *m_= ipptr, > + uint64_t fsz, uint64_t cq_w1, uint16_t *i= hl) > +{ > + union nix_rx_parse_u *rx =3D (union nix_rx_parse_u *)((uintptr_t)= (head + 1) + 8); > + uint16_t fragx_sum =3D vaddv_u16(vdup_n_u64(fsz)); > + uint8_t lcptr =3D rx->lcptr; > + uint16_t tot_len; > + uint32_t cksum; > + uint8_t *ipptr; > + > + ipptr =3D (uint8_t *)head->buf_addr + head->data_off + lcptr; > + /* Find the L3 header length and update inner pkt based on meta l= c type */ > + if (((cq_w1 >> 40) & 0xF) =3D=3D NPC_LT_LC_IP) { > + const struct rte_ipv4_hdr *m_hdr =3D (const struct rte_ip= v4_hdr *)m_ipptr; > + struct rte_ipv4_hdr *hdr =3D (struct rte_ipv4_hdr *)ipptr= ; > + > + *ihl =3D (m_hdr->version_ihl & 0xf) << 2; > + > + hdr->fragment_offset =3D 0; > + tot_len =3D rte_cpu_to_be_16(fragx_sum + *ihl); > + hdr->total_length =3D tot_len; > + /* Perform incremental checksum based on meta pkt ip hdr = */ > + cksum =3D m_hdr->hdr_checksum; > + cksum +=3D m_hdr->fragment_offset; > + cksum +=3D 0xFFFF; > + cksum +=3D m_hdr->total_length; > + cksum +=3D (uint16_t)(~tot_len); > + cksum =3D (cksum & 0xFFFF) + ((cksum & 0xFFFF0000) >> 16)= ; > + hdr->hdr_checksum =3D cksum; > + > + head->pkt_len =3D lcptr + *ihl + fragx_sum; > + } else { > + struct rte_ipv6_hdr *hdr =3D (struct rte_ipv6_hdr *)ipptr= ; > + size_t ext_len =3D sizeof(struct rte_ipv6_hdr); > + uint8_t *nxt_hdr =3D (uint8_t *)hdr; > + int nh =3D hdr->proto; > + > + *ihl =3D 0; > + while (nh !=3D -EINVAL) { > + nxt_hdr +=3D ext_len; > + *ihl +=3D ext_len; > + nh =3D rte_ipv6_get_next_ext(nxt_hdr, nh, &ext_le= n); > + } > + > + /* Remove the frag header by moving header 8 bytes forwar= d */ > + hdr->payload_len =3D rte_cpu_to_be_16(fragx_sum + *ihl - > + 8 - sizeof(struct rte_ipv6_hdr)); > + > + rte_memcpy(rte_pktmbuf_mtod_offset(head, void *, 8), > + rte_pktmbuf_mtod(head, void *), > + lcptr + sizeof(struct rte_ipv6_hdr)); > + > + head->data_len -=3D 8; > + head->data_off +=3D 8; > + head->pkt_len =3D lcptr + *ihl - 8 + fragx_sum; > + } > +} > + > +#else > +static __rte_always_inline uint64_t > +nix_sec_reass_frags_get(const struct cpt_parse_hdr_s *hdr, struct rte_mb= uf **next_mbufs) > +{ > + RTE_SET_USED(hdr); > + next_mbufs[0] =3D NULL; > + next_mbufs[1] =3D NULL; > + next_mbufs[2] =3D NULL; > + return 0; > +} > + > +static __rte_always_inline void > +nix_sec_reass_first_frag_update(struct rte_mbuf *head, const uint8_t *m_= ipptr, > + uint64_t fsz, uint64_t cq_w1, uint16_t *i= hl) > +{ > + RTE_SET_USED(head); > + RTE_SET_USED(m_ipptr); > + RTE_SET_USED(fsz); > + RTE_SET_USED(cq_w1); > + *ihl =3D 0; > +} > +#endif > + > static struct rte_mbuf * > nix_sec_attach_frags(const struct cpt_parse_hdr_s *hdr, > + struct rte_mbuf *head, > struct cn10k_inb_priv_data *inb_priv, > const uint64_t mbuf_init) > { > - struct rte_mbuf *head, *mbuf, *mbuf_prev; > - uint32_t offset =3D hdr->w2.fi_offset; > + uint8_t num_frags =3D hdr->w0.num_frags; > + struct rte_mbuf *next_mbufs[3]; > union nix_rx_parse_u *frag_rx; > - struct cpt_frag_info_s *finfo; > - uint64_t *frag_ptr =3D NULL; > + struct rte_mbuf *mbuf; > uint64_t ol_flags; > uint16_t frag_size; > + uint8_t frag_i =3D 0; > uint16_t rlen; > uint64_t *wqe; > int off; > @@ -164,48 +272,37 @@ nix_sec_attach_frags(const struct cpt_parse_hdr_s *= hdr, > ol_flags =3D BIT_ULL(inb_priv->reass_dynflag_bit); > ol_flags |=3D RTE_MBUF_F_RX_SEC_OFFLOAD; > > - /* offset of 0 implies 256B, otherwise it implies offset*8B */ > - offset =3D (((offset - 1) & 0x1f) + 1) * 8; > - finfo =3D RTE_PTR_ADD(hdr, offset); > + /* Get frags list */ > + nix_sec_reass_frags_get(hdr, next_mbufs); > > /* Frag-0: */ > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(hdr->wqe_ptr)); > + wqe =3D (uint64_t *)(head + 1); > rlen =3D ((*(wqe + 10)) >> 16) & 0xFFFF; > > frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > - frag_size =3D rlen + frag_rx->lcptr - frag_rx->laptr; > - frag_rx->pkt_lenm1 =3D frag_size - 1; > > - mbuf =3D (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_m= buf)); > - *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init; > - mbuf->data_len =3D frag_size; > - mbuf->pkt_len =3D frag_size; > - mbuf->ol_flags =3D ol_flags; > - mbuf->next =3D NULL; > - head =3D mbuf; > - mbuf_prev =3D mbuf; > + head->ol_flags =3D ol_flags; > /* Update dynamic field with userdata */ > - *rte_security_dynfield(mbuf) =3D (uint64_t)inb_priv->userdata; > + *rte_security_dynfield(head) =3D (uint64_t)inb_priv->userdata; > > - cnxk_ip_reassembly_dynfield(head, off)->nb_frags =3D hdr->w0.num_= frags - 1; > - cnxk_ip_reassembly_dynfield(head, off)->next_frag =3D NULL; > + num_frags--; > + mbuf =3D head; > > - /* Frag-1: */ > - if (hdr->w0.num_frags > 1) { > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(hdr->frag1_wqe_ptr)= ); > + /* Frag-1+: */ > + while (num_frags) { > + cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag =3D nex= t_mbufs[frag_i]; > + cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags =3D num_= frags; > + mbuf =3D next_mbufs[frag_i]; > + wqe =3D (uint64_t *)(mbuf + 1); > rlen =3D ((*(wqe + 10)) >> 16) & 0xFFFF; > > frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > frag_size =3D rlen + frag_rx->lcptr - frag_rx->laptr; > - frag_rx->pkt_lenm1 =3D frag_size - 1; > > - mbuf =3D (struct rte_mbuf *)((uintptr_t)wqe - > - sizeof(struct rte_mbuf)); > *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init; > mbuf->data_len =3D frag_size; > mbuf->pkt_len =3D frag_size; > mbuf->ol_flags =3D ol_flags; > - mbuf->next =3D NULL; > > /* Update dynamic field with userdata */ > *rte_security_dynfield(mbuf) =3D (uint64_t)inb_priv->user= data; > @@ -213,218 +310,95 @@ nix_sec_attach_frags(const struct cpt_parse_hdr_s = *hdr, > /* Mark frag as get */ > RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > > - cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags =3D > - hdr->w0.num_frags - 2; > - cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag =3D NUL= L; > - cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = =3D mbuf; > - mbuf_prev =3D mbuf; > + num_frags--; > + frag_i++; > } > + cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags =3D 0; > + cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag =3D NULL; > > - /* Frag-2: */ > - if (hdr->w0.num_frags > 2) { > - frag_ptr =3D (uint64_t *)(finfo + 1); > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(*frag_ptr)); > - rlen =3D ((*(wqe + 10)) >> 16) & 0xFFFF; > - > - frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > - frag_size =3D rlen + frag_rx->lcptr - frag_rx->laptr; > - frag_rx->pkt_lenm1 =3D frag_size - 1; > - > - mbuf =3D (struct rte_mbuf *)((uintptr_t)wqe - > - sizeof(struct rte_mbuf)); > - *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init; > - mbuf->data_len =3D frag_size; > - mbuf->pkt_len =3D frag_size; > - mbuf->ol_flags =3D ol_flags; > - mbuf->next =3D NULL; > - > - /* Update dynamic field with userdata */ > - *rte_security_dynfield(mbuf) =3D (uint64_t)inb_priv->user= data; > - > - /* Mark frag as get */ > - RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > - > - cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags =3D > - hdr->w0.num_frags - 3; > - cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag =3D NUL= L; > - cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = =3D mbuf; > - mbuf_prev =3D mbuf; > - } > - > - /* Frag-3: */ > - if (hdr->w0.num_frags > 3) { > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(*(frag_ptr + 1))); > - rlen =3D ((*(wqe + 10)) >> 16) & 0xFFFF; > - > - frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > - frag_size =3D rlen + frag_rx->lcptr - frag_rx->laptr; > - frag_rx->pkt_lenm1 =3D frag_size - 1; > - > - mbuf =3D (struct rte_mbuf *)((uintptr_t)wqe - > - sizeof(struct rte_mbuf)); > - *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init; > - mbuf->data_len =3D frag_size; > - mbuf->pkt_len =3D frag_size; > - mbuf->ol_flags =3D ol_flags; > - mbuf->next =3D NULL; > - > - /* Mark frag as get */ > - RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > - > - /* Update dynamic field with userdata */ > - *rte_security_dynfield(mbuf) =3D (uint64_t)inb_priv->user= data; > - > - cnxk_ip_reassembly_dynfield(mbuf, off)->nb_frags =3D > - hdr->w0.num_frags - 4; > - cnxk_ip_reassembly_dynfield(mbuf, off)->next_frag =3D NUL= L; > - cnxk_ip_reassembly_dynfield(mbuf_prev, off)->next_frag = =3D mbuf; > - } > return head; > } > > -static struct rte_mbuf * > -nix_sec_reassemble_frags(const struct cpt_parse_hdr_s *hdr, uint64_t cq_= w1, > - uint64_t cq_w5, uint64_t mbuf_init) > +static __rte_always_inline struct rte_mbuf * > +nix_sec_reassemble_frags(const struct cpt_parse_hdr_s *hdr, struct rte_m= buf *head, > + uint64_t cq_w1, uint64_t cq_w5, uint64_t mbuf_in= it) > { > - uint32_t fragx_sum, pkt_hdr_len, l3_hdr_size; > - uint32_t offset =3D hdr->w2.fi_offset; > - union nix_rx_parse_u *inner_rx; > - uint16_t rlen, data_off, b_off; > + uint8_t num_frags =3D hdr->w0.num_frags; > union nix_rx_parse_u *frag_rx; > - struct cpt_frag_info_s *finfo; > - struct rte_mbuf *head, *mbuf; > - uint64_t *frag_ptr =3D NULL; > - rte_iova_t *inner_iova; > + struct rte_mbuf *next_mbufs[3]; > + uint16_t data_off, b_off; > + const uint8_t *m_ipptr; > + uint16_t l3_hdr_size; > + struct rte_mbuf *mbuf; > uint16_t frag_size; > + uint64_t fsz_w1; > uint64_t *wqe; > > /* Base data offset */ > b_off =3D mbuf_init & 0xFFFFUL; > mbuf_init &=3D ~0xFFFFUL; > > - /* offset of 0 implies 256B, otherwise it implies offset*8B */ > - offset =3D (((offset - 1) & 0x1f) + 1) * 8; > - finfo =3D RTE_PTR_ADD(hdr, offset); > + /* Get list of all fragments and frag sizes */ > + fsz_w1 =3D nix_sec_reass_frags_get(hdr, next_mbufs); > > /* Frag-0: */ > - wqe =3D (uint64_t *)rte_be_to_cpu_64(hdr->wqe_ptr); > - inner_rx =3D (union nix_rx_parse_u *)(wqe + 1); > - inner_iova =3D (rte_iova_t *)*(wqe + 9); > + wqe =3D (uint64_t *)(head + 1); > > - /* Update only the upper 28-bits from meta pkt parse info */ > - *((uint64_t *)inner_rx) =3D ((*((uint64_t *)inner_rx) & ((1ULL <<= 36) - 1)) | > - (cq_w1 & ~((1ULL << 36) - 1))); > - > - rlen =3D ((*(wqe + 10)) >> 16) & 0xFFFF; > - frag_size =3D rlen + ((cq_w5 >> 16) & 0xFF) - (cq_w5 & 0xFF); > - fragx_sum =3D rte_be_to_cpu_16(finfo->w1.frag_size0); > - pkt_hdr_len =3D frag_size - fragx_sum; > - > - mbuf =3D (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct rte_m= buf)); > - *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init | b_off; > - mbuf->data_len =3D frag_size; > - head =3D mbuf; > - > - if (inner_rx->lctype =3D=3D NPC_LT_LC_IP) { > - struct rte_ipv4_hdr *hdr =3D (struct rte_ipv4_hdr *) > - RTE_PTR_ADD(inner_iova, inner_rx->lcptr); > - > - l3_hdr_size =3D (hdr->version_ihl & 0xf) << 2; > - } else { > - struct rte_ipv6_hdr *hdr =3D (struct rte_ipv6_hdr *) > - RTE_PTR_ADD(inner_iova, inner_rx->lcptr); > - size_t ext_len =3D sizeof(struct rte_ipv6_hdr); > - uint8_t *nxt_hdr =3D (uint8_t *)hdr; > - int nh =3D hdr->proto; > - > - l3_hdr_size =3D 0; > - while (nh !=3D -EINVAL) { > - nxt_hdr +=3D ext_len; > - l3_hdr_size +=3D ext_len; > - nh =3D rte_ipv6_get_next_ext(nxt_hdr, nh, &ext_le= n); > - } > - } > + /* First fragment data len is already update by caller */ > + m_ipptr =3D ((const uint8_t *)hdr + ((cq_w5 >> 16) & 0xFF)); > + nix_sec_reass_first_frag_update(head, m_ipptr, fsz_w1, cq_w1, &l3= _hdr_size); > + fsz_w1 >>=3D 16; > > /* Frag-1: */ > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(hdr->frag1_wqe_ptr)); > - frag_size =3D rte_be_to_cpu_16(finfo->w1.frag_size1); > + head->next =3D next_mbufs[0]; > + mbuf =3D next_mbufs[0]; > + wqe =3D (uint64_t *)(mbuf + 1); > frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > + frag_size =3D fsz_w1 & 0xFFFF; > + fsz_w1 >>=3D 16; > > - mbuf->next =3D (struct rte_mbuf *)((uintptr_t)wqe - sizeof(struct= rte_mbuf)); > - mbuf =3D mbuf->next; > data_off =3D b_off + frag_rx->lcptr + l3_hdr_size; > *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init | data_off; > mbuf->data_len =3D frag_size; > - fragx_sum +=3D frag_size; > > /* Mark frag as get */ > RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, 1); > > /* Frag-2: */ > - if (hdr->w0.num_frags > 2) { > - frag_ptr =3D (uint64_t *)(finfo + 1); > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(*frag_ptr)); > - frag_size =3D rte_be_to_cpu_16(finfo->w1.frag_size2); > + if (num_frags > 2) { > + mbuf->next =3D next_mbufs[1]; > + mbuf =3D next_mbufs[1]; > + wqe =3D (uint64_t *)(mbuf + 1); > frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > + frag_size =3D fsz_w1 & 0xFFFF; > + fsz_w1 >>=3D 16; > > - mbuf->next =3D (struct rte_mbuf *)((uintptr_t)wqe - sizeo= f(struct rte_mbuf)); > - mbuf =3D mbuf->next; > data_off =3D b_off + frag_rx->lcptr + l3_hdr_size; > *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init | data_off= ; > mbuf->data_len =3D frag_size; > - fragx_sum +=3D frag_size; > > /* Mark frag as get */ > RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > } > > /* Frag-3: */ > - if (hdr->w0.num_frags > 3) { > - wqe =3D (uint64_t *)(rte_be_to_cpu_64(*(frag_ptr + 1))); > - frag_size =3D rte_be_to_cpu_16(finfo->w1.frag_size3); > + if (num_frags > 3) { > + mbuf->next =3D next_mbufs[2]; > + mbuf =3D next_mbufs[2]; > + wqe =3D (uint64_t *)(mbuf + 1); > frag_rx =3D (union nix_rx_parse_u *)(wqe + 1); > + frag_size =3D fsz_w1 & 0xFFFF; > + fsz_w1 >>=3D 16; > > - mbuf->next =3D (struct rte_mbuf *)((uintptr_t)wqe - sizeo= f(struct rte_mbuf)); > - mbuf =3D mbuf->next; > data_off =3D b_off + frag_rx->lcptr + l3_hdr_size; > *(uint64_t *)(&mbuf->rearm_data) =3D mbuf_init | data_off= ; > mbuf->data_len =3D frag_size; > - fragx_sum +=3D frag_size; > > /* Mark frag as get */ > RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > } > > - if (inner_rx->lctype =3D=3D NPC_LT_LC_IP) { > - struct rte_ipv4_hdr *hdr =3D (struct rte_ipv4_hdr *) > - RTE_PTR_ADD(inner_iova, inner_rx->lcptr); > - > - hdr->fragment_offset =3D 0; > - hdr->total_length =3D rte_cpu_to_be_16(fragx_sum + l3_hdr= _size); > - hdr->hdr_checksum =3D 0; > - hdr->hdr_checksum =3D rte_ipv4_cksum(hdr); > - > - inner_rx->pkt_lenm1 =3D pkt_hdr_len + fragx_sum - 1; > - } else { > - /* Remove the frag header by moving header 8 bytes forwar= d */ > - struct rte_ipv6_hdr *hdr =3D (struct rte_ipv6_hdr *) > - RTE_PTR_ADD(inner_iova, inner_rx->lcptr); > - > - hdr->payload_len =3D rte_cpu_to_be_16(fragx_sum + l3_hdr_= size - > - 8 - sizeof(struct rte_ipv6_hdr)); > - > - rte_memcpy(rte_pktmbuf_mtod_offset(head, void *, 8), > - rte_pktmbuf_mtod(head, void *), > - inner_rx->lcptr + sizeof(struct rte_ipv6_hdr))= ; > - > - inner_rx->pkt_lenm1 =3D pkt_hdr_len + fragx_sum - 8 - 1; > - head->data_len -=3D 8; > - head->data_off +=3D 8; > - } > - mbuf->next =3D NULL; > - head->pkt_len =3D inner_rx->pkt_lenm1 + 1; > - head->nb_segs =3D hdr->w0.num_frags; > - > + head->nb_segs =3D num_frags; > return head; > } > > @@ -432,7 +406,7 @@ static __rte_always_inline struct rte_mbuf * > nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, uint64_t cq_w5, const uint64_t s= a_base, > uintptr_t laddr, uint8_t *loff, struct rte_mbuf *= mbuf, > uint16_t data_off, const uint16_t flags, > - const uint64_t mbuf_init) > + uint64_t mbuf_init) > { > const void *__p =3D (void *)((uintptr_t)mbuf + (uint16_t)data_off= ); > const struct cpt_parse_hdr_s *hdr =3D (const struct cpt_parse_hdr= _s *)__p; > @@ -445,60 +419,70 @@ nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, uint64_t cq= _w5, const uint64_t sa_base, > void *inb_sa; > uint64_t w0; > > - if ((flags & NIX_RX_REAS_F) && (cq_w1 & BIT(11))) { > - /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ > - w0 =3D hdr->w0.u64; > - sa_idx =3D w0 >> 32; > - > - inb_sa =3D roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); > - inb_priv =3D roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); > - > - if (!hdr->w0.num_frags) { > - /* No Reassembly or inbound error */ > - inner =3D (struct rte_mbuf *) > - (rte_be_to_cpu_64(hdr->wqe_ptr) - > - sizeof(struct rte_mbuf)); > - > - /* Update dynamic field with userdata */ > - *rte_security_dynfield(inner) =3D > - (uint64_t)inb_priv->userdata; > - > - /* Get ucc from cpt parse header */ > - ucc =3D hdr->w3.hw_ccode; > - > - /* Calculate inner packet length as > - * IP total len + l2 len > - */ > - ip =3D (uintptr_t)hdr + ((cq_w5 >> 16) & 0xFF); > - ip +=3D ((cq_w1 >> 40) & 0x6); > - len =3D rte_be_to_cpu_16(*(uint16_t *)ip); > - len +=3D ((cq_w5 >> 16) & 0xFF) - (cq_w5 & 0xFF); > - len +=3D (cq_w1 & BIT(42)) ? 40 : 0; > - > - inner->pkt_len =3D len; > - inner->data_len =3D len; > - *(uint64_t *)(&inner->rearm_data) =3D mbuf_init; > - > - inner->ol_flags =3D ((CPT_COMP_HWGOOD_MASK & (1U = << ucc)) ? > - RTE_MBUF_F_RX_SEC_OFFLOAD : > - (RTE_MBUF_F_RX_SEC_OFFLOAD | > - RTE_MBUF_F_RX_SEC_OFFLOAD_FAI= LED)); > - > - ucc =3D hdr->w3.uc_ccode; > - > - if (ucc && ucc < 0xED) { > - inner->ol_flags |=3D RTE_MBUF_F_RX_SEC_OF= FLOAD_FAILED; > - } else { > - ucc +=3D 3; /* To make codes in 0xFx seri= es except 0 */ > - inner->ol_flags |=3D ((ucc & 0xF0) =3D=3D= 0xF0) ? > - ((NIX_RX_SEC_UCC_CONST= >> ((ucc & 0xF) << 3)) > - & 0xFF) << 1 : RTE_MB= UF_F_RX_IP_CKSUM_GOOD; > - } > - } else if ((!(hdr->w0.err_sum) || roc_ie_ot_ucc_is_succes= s(hdr->w3.uc_ccode)) && > - !(hdr->w0.reas_sts)) { > + if (!(cq_w1 & BIT(11))) > + return mbuf; > + > + inner =3D (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_ptr) - > + sizeof(struct rte_mbuf)); > + > + /* Store meta in lmtline to free > + * Assume all meta's from same aura. > + */ > + *(uint64_t *)(laddr + (*loff << 3)) =3D (uint64_t)mbuf; > + *loff =3D *loff + 1; > + > + /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ > + w0 =3D hdr->w0.u64; > + sa_idx =3D w0 >> 32; > + > + inb_sa =3D roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); > + inb_priv =3D roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); > + > + /* Update dynamic field with userdata */ > + *rte_security_dynfield(inner) =3D (uint64_t)inb_priv->userdata; > + > + /* Get ucc from cpt parse header */ > + ucc =3D hdr->w3.hw_ccode; > + > + /* Calculate inner packet length as IP total len + l2 len */ > + ip =3D (uintptr_t)hdr + ((cq_w5 >> 16) & 0xFF); > + ip +=3D ((cq_w1 >> 40) & 0x6); > + len =3D rte_be_to_cpu_16(*(uint16_t *)ip); > + len +=3D ((cq_w5 >> 16) & 0xFF) - (cq_w5 & 0xFF); > + len +=3D (cq_w1 & BIT(42)) ? 40 : 0; > + > + inner->pkt_len =3D len; > + inner->data_len =3D len; > + *(uint64_t *)(&inner->rearm_data) =3D mbuf_init; > + > + inner->ol_flags =3D ((CPT_COMP_HWGOOD_MASK & (1U << ucc)) ? > + RTE_MBUF_F_RX_SEC_OFFLOAD : > + (RTE_MBUF_F_RX_SEC_OFFLOAD | > + RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); > + > + ucc =3D hdr->w3.uc_ccode; > + > + if (ucc && ucc < 0xED) { > + inner->ol_flags |=3D RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED; > + } else { > + ucc +=3D 3; /* To make codes in 0xFx series except 0 */ > + inner->ol_flags |=3D ((ucc & 0xF0) =3D=3D 0xF0) ? > + ((NIX_RX_SEC_UCC_CONST >> ((ucc & 0xF) << 3)) > + & 0xFF) << 1 : RTE_MBUF_F_RX_IP_CKSUM_GOOD; > + } > + > + /* Mark meta mbuf as put */ > + RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, 0); > + > + /* Mark inner mbuf as get */ > + RTE_MEMPOOL_CHECK_COOKIES(inner->pool, (void **)&inner, 1, 1); > + > + /* Skip reassembly processing when multi-seg is enabled */ > + if (!(flags & NIX_RX_MULTI_SEG_F) && (flags & NIX_RX_REAS_F) && h= dr->w0.num_frags) { > + if ((!(hdr->w0.err_sum) || roc_ie_ot_ucc_is_success(hdr->= w3.uc_ccode)) && > + !(hdr->w0.reas_sts)) { > /* Reassembly success */ > - inner =3D nix_sec_reassemble_frags(hdr, cq_w1, cq= _w5, > - mbuf_init); > + nix_sec_reassemble_frags(hdr, inner, cq_w1, cq_w5= , mbuf_init); > > /* Update dynamic field with userdata */ > *rte_security_dynfield(inner) =3D > @@ -508,82 +492,10 @@ nix_sec_meta_to_mbuf_sc(uint64_t cq_w1, uint64_t cq= _w5, const uint64_t sa_base, > inner->ol_flags =3D RTE_MBUF_F_RX_SEC_OFFLOAD; > } else { > /* Reassembly failure */ > - inner =3D nix_sec_attach_frags(hdr, inb_priv, mbu= f_init); > + nix_sec_attach_frags(hdr, inner, inb_priv, mbuf_i= nit); > } > - > - /* Store meta in lmtline to free > - * Assume all meta's from same aura. > - */ > - *(uint64_t *)(laddr + (*loff << 3)) =3D (uint64_t)mbuf; > - *loff =3D *loff + 1; > - > - /* Mark meta mbuf as put */ > - RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 0); > - > - /* Mark inner mbuf as get */ > - RTE_MEMPOOL_CHECK_COOKIES(inner->pool, (void **)&inner, 1= , 1); > - > - return inner; > - } else if (cq_w1 & BIT(11)) { > - inner =3D (struct rte_mbuf *)(rte_be_to_cpu_64(hdr->wqe_p= tr) - > - sizeof(struct rte_mbuf)); > - > - /* Get SPI from CPT_PARSE_S's cookie(already swapped) */ > - w0 =3D hdr->w0.u64; > - sa_idx =3D w0 >> 32; > - > - inb_sa =3D roc_nix_inl_ot_ipsec_inb_sa(sa_base, sa_idx); > - inb_priv =3D roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(inb_sa); > - > - /* Update dynamic field with userdata */ > - *rte_security_dynfield(inner) =3D (uint64_t)inb_priv->use= rdata; > - > - /* Get ucc from cpt parse header */ > - ucc =3D hdr->w3.hw_ccode; > - > - /* Calculate inner packet length as IP total len + l2 len= */ > - ip =3D (uintptr_t)hdr + ((cq_w5 >> 16) & 0xFF); > - ip +=3D ((cq_w1 >> 40) & 0x6); > - len =3D rte_be_to_cpu_16(*(uint16_t *)ip); > - len +=3D ((cq_w5 >> 16) & 0xFF) - (cq_w5 & 0xFF); > - len +=3D (cq_w1 & BIT(42)) ? 40 : 0; > - > - inner->pkt_len =3D len; > - inner->data_len =3D len; > - *(uint64_t *)(&inner->rearm_data) =3D mbuf_init; > - > - inner->ol_flags =3D ((CPT_COMP_HWGOOD_MASK & (1U << ucc))= ? > - RTE_MBUF_F_RX_SEC_OFFLOAD : > - (RTE_MBUF_F_RX_SEC_OFFLOAD | > - RTE_MBUF_F_RX_SEC_OFFLOAD_FAILED)); > - > - ucc =3D hdr->w3.uc_ccode; > - > - if (ucc && ucc < 0xED) { > - inner->ol_flags |=3D RTE_MBUF_F_RX_SEC_OFFLOAD_FA= ILED; > - } else { > - ucc +=3D 3; /* To make codes in 0xFx series excep= t 0 */ > - inner->ol_flags |=3D ((ucc & 0xF0) =3D=3D 0xF0) ? > - ((NIX_RX_SEC_UCC_CONST >> ((uc= c & 0xF) << 3)) > - & 0xFF) << 1 : RTE_MBUF_F_RX_= IP_CKSUM_GOOD; > - } > - > - /* Store meta in lmtline to free > - * Assume all meta's from same aura. > - */ > - *(uint64_t *)(laddr + (*loff << 3)) =3D (uint64_t)mbuf; > - *loff =3D *loff + 1; > - > - /* Mark meta mbuf as put */ > - RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 0); > - > - /* Mark inner mbuf as get */ > - RTE_MEMPOOL_CHECK_COOKIES(inner->pool, (void **)&inner, 1= , 1); > - > - return inner; > } > - > - return mbuf; > + return inner; > } > > #if defined(RTE_ARCH_ARM64) > @@ -612,11 +524,16 @@ nix_sec_meta_to_mbuf(uint64_t cq_w1, uint64_t cq_w5= , uintptr_t inb_sa, > /* Mark inner mbuf as get */ > RTE_MEMPOOL_CHECK_COOKIES(inner->pool, (void **)&inner, 1, 1); > > - if (flags & NIX_RX_REAS_F && hdr->w0.num_frags) { > + if (!(flags & NIX_RX_MULTI_SEG_F) && flags & NIX_RX_REAS_F && hdr= ->w0.num_frags) { > if ((!(hdr->w0.err_sum) || roc_ie_ot_ucc_is_success(hdr->= w3.uc_ccode)) && > !(hdr->w0.reas_sts)) { > + /* First frag len */ > + inner->pkt_len =3D vgetq_lane_u16(*rx_desc_field1= , 2); > + inner->data_len =3D vgetq_lane_u16(*rx_desc_field= 1, 4); > + *(uint64_t *)(&inner->rearm_data) =3D mbuf_init; > + > /* Reassembly success */ > - nix_sec_reassemble_frags(hdr, cq_w1, cq_w5, mbuf_= init); > + nix_sec_reassemble_frags(hdr, inner, cq_w1, cq_w5= , mbuf_init); > > /* Assume success */ > *ol_flags |=3D RTE_MBUF_F_RX_SEC_OFFLOAD; > @@ -632,14 +549,8 @@ nix_sec_meta_to_mbuf(uint64_t cq_w1, uint64_t cq_w5,= uintptr_t inb_sa, > *rearm =3D vsetq_lane_u64(mbuf_init, *rearm, 0); > } else { > /* Reassembly failure */ > - nix_sec_attach_frags(hdr, inb_priv, mbuf_init); > + nix_sec_attach_frags(hdr, inner, inb_priv, mbuf_i= nit); > *ol_flags |=3D inner->ol_flags; > - > - /* Update pkt_len and data_len */ > - *rx_desc_field1 =3D vsetq_lane_u16(inner->pkt_len= , > - *rx_desc_field1,= 2); > - *rx_desc_field1 =3D vsetq_lane_u16(inner->data_le= n, > - *rx_desc_field1,= 4); > } > } > } > @@ -697,18 +608,30 @@ nix_update_match_id(const uint16_t match_id, uint64= _t ol_flags, > > static __rte_always_inline void > nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx, struct rte_mbuf *mbu= f, > - uint64_t rearm, const uint16_t flags) > + uint64_t rearm, uintptr_t cpth, uintptr_t sa_base, co= nst uint16_t flags) > { > + const struct cpt_parse_hdr_s *hdr =3D (const struct cpt_parse_hdr= _s *)cpth; > + struct cn10k_inb_priv_data *inb_priv =3D NULL; > + uint8_t num_frags =3D 0, frag_i =3D 0; > + struct rte_mbuf *next_mbufs[3]; > const rte_iova_t *iova_list; > + bool reas_success =3D false; > uint16_t later_skip =3D 0; > struct rte_mbuf *head; > const rte_iova_t *eol; > + uint64_t cq_w5 =3D 0; > + uint16_t ihl =3D 0; > + uint64_t fsz =3D 0; > + int dyn_off =3D 0; > uint8_t nb_segs; > + uint16_t sg_len; > uint64_t cq_w1; > int64_t len; > uint64_t sg; > > cq_w1 =3D *(const uint64_t *)rx; > + if (flags & NIX_RX_REAS_F) > + cq_w5 =3D *((const uint64_t *)rx + 4); > /* Use inner rx parse for meta pkts sg list */ > if (cq_w1 & BIT(11) && flags & NIX_RX_OFFLOAD_SECURITY_F) { > const uint64_t *wqe =3D (const uint64_t *)(mbuf + 1); > @@ -718,40 +641,69 @@ nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx,= struct rte_mbuf *mbuf, > sg =3D *(const uint64_t *)(rx + 1); > nb_segs =3D (sg >> 48) & 0x3; > > - if (nb_segs =3D=3D 1) > + if (nb_segs =3D=3D 1 && !(flags & NIX_RX_REAS_F)) > return; > > /* For security we have already updated right pkt_len */ > - if (cq_w1 & BIT(11) && flags & NIX_RX_OFFLOAD_SECURITY_F) > + if (cq_w1 & BIT(11) && flags & NIX_RX_OFFLOAD_SECURITY_F) { > len =3D mbuf->pkt_len; > - else > + > + /* Handle reassembly with multi segs */ > + if (flags & NIX_RX_REAS_F && hdr->w0.num_frags) { > + void *inb_sa; > + > + num_frags =3D hdr->w0.num_frags; > + inb_sa =3D roc_nix_inl_ot_ipsec_inb_sa(sa_base, h= dr->w0.u64 >> 32); > + inb_priv =3D roc_nix_inl_ot_ipsec_inb_sa_sw_rsvd(= inb_sa); > + ihl =3D 0; > + > + dyn_off =3D inb_priv->reass_dynfield_off; > + fsz =3D nix_sec_reass_frags_get(hdr, next_mbufs); > + num_frags -=3D 1; > + > + if (!(hdr->w0.reas_sts) && > + (!(hdr->w0.err_sum) || > + roc_ie_ot_ucc_is_success(hdr->w3.uc_ccode))) > + reas_success =3D true; > + } > + } else { > len =3D rx->pkt_lenm1 + 1; > + } > + > mbuf->pkt_len =3D len - (flags & NIX_RX_OFFLOAD_TSTAMP_F ? CNXK_N= IX_TIMESYNC_RX_OFFSET : 0); > + mbuf->nb_segs =3D nb_segs; > + head =3D mbuf; > mbuf->data_len =3D > (sg & 0xFFFF) - (flags & NIX_RX_OFFLOAD_TSTAMP_F ? CNXK_N= IX_TIMESYNC_RX_OFFSET : 0); > + eol =3D ((const rte_iova_t *)(rx + 1) + ((rx->desc_sizem1 + 1) <<= 1)); > +again: > len -=3D mbuf->data_len; > - mbuf->nb_segs =3D nb_segs; > sg =3D sg >> 16; > - > - eol =3D ((const rte_iova_t *)(rx + 1) + ((rx->desc_sizem1 + 1) <<= 1)); > /* Skip SG_S and first IOVA*/ > iova_list =3D ((const rte_iova_t *)(rx + 1)) + 2; > nb_segs--; > > - rearm =3D rearm & ~0xFFFF; > later_skip =3D (uintptr_t)mbuf->buf_addr - (uintptr_t)mbuf; > > - head =3D mbuf; > while (nb_segs) { > mbuf->next =3D (struct rte_mbuf *)(*iova_list - later_ski= p); > mbuf =3D mbuf->next; > > RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > > - mbuf->data_len =3D sg & 0xFFFF; > - len -=3D sg & 0XFFFF; > + sg_len =3D sg & 0XFFFF; > + if (flags & NIX_RX_OFFLOAD_SECURITY_F) { > + /* Adjust last mbuf data length with negative off= set for > + * security pkts if needed. > + */ > + len -=3D sg_len; > + sg_len =3D (len > 0) ? sg_len : (sg_len + len); > + len =3D (len > 0) ? len : 0; > + } > + > + mbuf->data_len =3D sg_len; > sg =3D sg >> 16; > - *(uint64_t *)(&mbuf->rearm_data) =3D rearm; > + *(uint64_t *)(&mbuf->rearm_data) =3D rearm & ~0xFFFF; > nb_segs--; > iova_list++; > > @@ -763,15 +715,66 @@ nix_cqe_xtract_mseg(const union nix_rx_parse_u *rx,= struct rte_mbuf *mbuf, > } > } > > - /* Adjust last mbuf data length with negative offset for security= pkts if needed */ > - if (cq_w1 & BIT(11) && flags & NIX_RX_OFFLOAD_SECURITY_F && len <= 0) > - mbuf->data_len +=3D len; > + if ((flags & NIX_RX_REAS_F) && (cq_w1 & BIT(11)) && num_frags) { > + struct rte_mbuf *next_frag =3D next_mbufs[frag_i]; > + uint16_t lcptr, ldptr =3D 0; > + > + rx =3D (const union nix_rx_parse_u *)((uintptr_t)(next_fr= ag + 1) + 8); > + lcptr =3D (*((const uint64_t *)rx + 4) >> 16) & 0xFF; > + eol =3D ((const rte_iova_t *)(rx + 1) + ((rx->desc_sizem1= + 1) << 1)); > + sg =3D *(const uint64_t *)(rx + 1); > + nb_segs =3D (sg >> 48) & 0x3; > + > + if (reas_success) { > + /* Update first fragment info */ > + if (!frag_i) { > + const uint8_t *ipptr; > + > + ipptr =3D ((const uint8_t *)hdr + ((cq_w5= >> 16) & 0xFF)); > + nix_sec_reass_first_frag_update(head, ipp= tr, fsz, cq_w1, &ihl); > + fsz >>=3D 16; > + } > + mbuf->next =3D next_frag; > + head->nb_segs +=3D nb_segs; > + len =3D fsz & 0xFFFF; > + fsz >>=3D 16; > + ldptr =3D ihl + lcptr; > + } else { > + len =3D ((eol[0] >> 16) & 0xFFFF) + lcptr; > + head->ol_flags |=3D BIT_ULL(inb_priv->reass_dynfl= ag_bit) | > + RTE_MBUF_F_RX_SEC_OFFLOAD; > + cnxk_ip_reassembly_dynfield(head, dyn_off)->next_= frag =3D next_frag; > + cnxk_ip_reassembly_dynfield(head, dyn_off)->nb_fr= ags =3D num_frags; > + /* Update dynamic field with userdata from prev h= ead */ > + *rte_security_dynfield(next_frag) =3D *rte_securi= ty_dynfield(head); > + head =3D next_frag; > + head->pkt_len =3D len - (flags & NIX_RX_OFFLOAD_T= STAMP_F ? > + CNXK_NIX_TIMESYNC_RX_OFFSE= T : 0); > + head->nb_segs =3D nb_segs; > + } > + mbuf =3D next_frag; > + *(uint64_t *)(&mbuf->rearm_data) =3D rearm + ldptr; > + mbuf->data_len =3D (sg & 0xFFFF) - ldptr - > + (flags & NIX_RX_OFFLOAD_TSTAMP_F ? > + CNXK_NIX_TIMESYNC_RX_OFFSET : 0); > + RTE_MEMPOOL_CHECK_COOKIES(mbuf->pool, (void **)&mbuf, 1, = 1); > + num_frags--; > + frag_i++; > + goto again; > + } > + > + /* Update for last failure fragment */ > + if ((flags & NIX_RX_REAS_F) && frag_i && !reas_success) { > + cnxk_ip_reassembly_dynfield(head, dyn_off)->next_frag =3D= NULL; > + cnxk_ip_reassembly_dynfield(head, dyn_off)->nb_frags =3D = 0; > + } > } > > static __rte_always_inline void > cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq, const uint32_t tag= , > struct rte_mbuf *mbuf, const void *lookup_mem, > - const uint64_t val, const uint16_t flag) > + const uint64_t val, const uintptr_t cpth, const uin= tptr_t sa_base, > + const uint16_t flag) > { > const union nix_rx_parse_u *rx =3D > (const union nix_rx_parse_u *)((const uint64_t *)cq + 1); > @@ -825,7 +828,7 @@ cn10k_nix_cqe_to_mbuf(const struct nix_cqe_hdr_s *cq,= const uint32_t tag, > * timestamp data process. > * Hence, timestamp flag argument is not required. > */ > - nix_cqe_xtract_mseg(rx, mbuf, val, flag & ~NIX_RX_OFFLOAD= _TSTAMP_F); > + nix_cqe_xtract_mseg(rx, mbuf, val, cpth, sa_base, flag & = ~NIX_RX_OFFLOAD_TSTAMP_F); > } > > static inline uint16_t > @@ -906,6 +909,7 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf *= *rx_pkts, uint16_t pkts, > struct nix_cqe_hdr_s *cq; > struct rte_mbuf *mbuf; > uint64_t aura_handle; > + uintptr_t cpth =3D 0; > uint64_t sa_base; > uint16_t lmt_id; > uint64_t laddr; > @@ -937,13 +941,15 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf= **rx_pkts, uint16_t pkts, > const uint64_t cq_w1 =3D *((const uint64_t *)cq += 1); > const uint64_t cq_w5 =3D *((const uint64_t *)cq += 5); > > + cpth =3D ((uintptr_t)mbuf + (uint16_t)data_off); > + > mbuf =3D nix_sec_meta_to_mbuf_sc(cq_w1, cq_w5, sa= _base, laddr, > &loff, mbuf, data_= off, > flags, mbuf_init); > } > > cn10k_nix_cqe_to_mbuf(cq, cq->tag, mbuf, lookup_mem, mbuf= _init, > - flags); > + cpth, sa_base, flags); > cn10k_nix_mbuf_to_tstamp(mbuf, rxq->tstamp, > (flags & NIX_RX_OFFLOAD_TSTAMP_F)= , > (uint64_t *)((uint8_t *)mbuf > @@ -1033,6 +1039,7 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_m= buf **mbufs, uint16_t pkts, > const uint64_t wdata =3D flags & NIX_RX_VWQE_F ? 0 : rxq->wdata; > const uintptr_t desc =3D flags & NIX_RX_VWQE_F ? 0 : rxq->desc; > uint64x2_t cq0_w8, cq1_w8, cq2_w8, cq3_w8, mbuf01, mbuf23; > + uintptr_t cpth0 =3D 0, cpth1 =3D 0, cpth2 =3D 0, cpth3 =3D 0; > uint64_t ol_flags0, ol_flags1, ol_flags2, ol_flags3; > uint64x2_t rearm0 =3D vdupq_n_u64(mbuf_initializer); > uint64x2_t rearm1 =3D vdupq_n_u64(mbuf_initializer); > @@ -1333,10 +1340,6 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_= mbuf **mbufs, uint16_t pkts, > uint64_t cq1_w5 =3D *CQE_PTR_OFF(cq0, 1, 40, flag= s); > uint64_t cq2_w5 =3D *CQE_PTR_OFF(cq0, 2, 40, flag= s); > uint64_t cq3_w5 =3D *CQE_PTR_OFF(cq0, 3, 40, flag= s); > - uintptr_t cpth0 =3D (uintptr_t)mbuf0 + d_off; > - uintptr_t cpth1 =3D (uintptr_t)mbuf1 + d_off; > - uintptr_t cpth2 =3D (uintptr_t)mbuf2 + d_off; > - uintptr_t cpth3 =3D (uintptr_t)mbuf3 + d_off; > uint8_t code; > > uint64x2_t inner0, inner1, inner2, inner3; > @@ -1344,6 +1347,11 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte_= mbuf **mbufs, uint16_t pkts, > uint16x4_t lens, l2lens, ltypes; > uint8x8_t ucc; > > + cpth0 =3D (uintptr_t)mbuf0 + d_off; > + cpth1 =3D (uintptr_t)mbuf1 + d_off; > + cpth2 =3D (uintptr_t)mbuf2 + d_off; > + cpth3 =3D (uintptr_t)mbuf3 + d_off; > + > inner0 =3D vld1q_u64((const uint64_t *)cpth0); > inner1 =3D vld1q_u64((const uint64_t *)cpth1); > inner2 =3D vld1q_u64((const uint64_t *)cpth2); > @@ -1729,16 +1737,16 @@ cn10k_nix_recv_pkts_vector(void *args, struct rte= _mbuf **mbufs, uint16_t pkts, > */ > nix_cqe_xtract_mseg((union nix_rx_parse_u *) > (CQE_PTR_OFF(cq0, 0, 8, flags= )), > - mbuf0, mbuf_initializer, flag= s); > + mbuf0, mbuf_initializer, cpth= 0, sa_base, flags); > nix_cqe_xtract_mseg((union nix_rx_parse_u *) > (CQE_PTR_OFF(cq0, 1, 8, flags= )), > - mbuf1, mbuf_initializer, flag= s); > + mbuf1, mbuf_initializer, cpth= 1, sa_base, flags); > nix_cqe_xtract_mseg((union nix_rx_parse_u *) > (CQE_PTR_OFF(cq0, 2, 8, flags= )), > - mbuf2, mbuf_initializer, flag= s); > + mbuf2, mbuf_initializer, cpth= 2, sa_base, flags); > nix_cqe_xtract_mseg((union nix_rx_parse_u *) > (CQE_PTR_OFF(cq0, 3, 8, flags= )), > - mbuf3, mbuf_initializer, flag= s); > + mbuf3, mbuf_initializer, cpth= 3, sa_base, flags); > } > > /* Store the mbufs to rx_pkts */ > -- > 2.25.1 >