From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C9A32A04B1; Sun, 22 Nov 2020 22:18:22 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 882C3FFA; Sun, 22 Nov 2020 22:18:19 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 5E5E22AB for ; Sun, 22 Nov 2020 22:18:16 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0AMLGa3L019866 for ; Sun, 22 Nov 2020 13:18:15 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding : content-type; s=pfpt0220; bh=yNp2dd3pRsQ4aWSVhXjPenRJLt3/2kCHaJH3KRDpefw=; b=MWCpQyo+9nZ400jd2J5c+9CGP1kVRHrQLt8eYrQcLqQSzkBMbU/azbtj+FwAd90Yhj8m JScKOANQ5NCNov2vvLpYZzu9aXvXkMLJfbE4bQMJcu2RnvkdEFctzR+uIzFfXoh0V3gx tCGrb8yqLd7AJbSZ5ulYQlEDBVBrM53w5BQRga6Hg+eIwlelPR1dNtv3nHZiX6apQE7p EZg/Zk+cpjvnd6R0FrOahQqpwLw2uhkFXxBbaBmcEL10Yqjfj+4X97PNxFADrb71ofCG EUVpFAP16Ne0ic/MFoxmf15Utk/W6TSDRvqADYqRpAiyW+J9AEiB9t+u0xFxRWHSUUOH 0w== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0b-0016f401.pphosted.com with ESMTP id 34y39r3889-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Sun, 22 Nov 2020 13:18:15 -0800 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Sun, 22 Nov 2020 13:18:13 -0800 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Sun, 22 Nov 2020 13:18:13 -0800 Received: from BG-LT7430.marvell.com (BG-LT7430.marvell.com [10.28.170.178]) by maili.marvell.com (Postfix) with ESMTP id 455063F703F; Sun, 22 Nov 2020 13:18:10 -0800 (PST) From: To: , Pavan Nikhilesh , "Anoob Joseph" , Nithin Dabilpuram , Kiran Kumar K CC: Date: Mon, 23 Nov 2020 02:48:07 +0530 Message-ID: <20201122211807.2401-1-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-22_15:2020-11-20, 2020-11-22 signatures=0 Subject: [dpdk-dev] [PATCH] event/octeontx2: enhance Tx path cache locality X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pavan Nikhilesh Enhance Tx path cache locality, remove current tag type and group stores from datapath to conserve store buffers. Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx2/otx2_evdev.c | 15 ++---- drivers/event/octeontx2/otx2_evdev.h | 24 ++++----- drivers/event/octeontx2/otx2_worker.c | 42 +++++---------- drivers/event/octeontx2/otx2_worker.h | 32 +++++------ drivers/event/octeontx2/otx2_worker_dual.c | 63 ++++++++++------------ drivers/event/octeontx2/otx2_worker_dual.h | 2 - drivers/net/octeontx2/otx2_ethdev_sec_tx.h | 9 ++-- 7 files changed, 74 insertions(+), 113 deletions(-) diff --git a/drivers/event/octeontx2/otx2_evdev.c b/drivers/event/octeontx2/otx2_evdev.c index 0fe014c24..14f16a68f 100644 --- a/drivers/event/octeontx2/otx2_evdev.c +++ b/drivers/event/octeontx2/otx2_evdev.c @@ -833,10 +833,12 @@ sso_configure_dual_ports(const struct rte_eventdev *event_dev) ws->port = i; base = dev->bar2 + (RVU_BLOCK_ADDR_SSOW << 20 | vws << 12); sso_set_port_ops((struct otx2_ssogws *)&ws->ws_state[0], base); + ws->base[0] = base; vws++; base = dev->bar2 + (RVU_BLOCK_ADDR_SSOW << 20 | vws << 12); sso_set_port_ops((struct otx2_ssogws *)&ws->ws_state[1], base); + ws->base[1] = base; vws++; gws_cookie = ssogws_get_cookie(ws); @@ -909,6 +911,7 @@ sso_configure_ports(const struct rte_eventdev *event_dev) ws->port = i; base = dev->bar2 + (RVU_BLOCK_ADDR_SSOW << 20 | i << 12); sso_set_port_ops(ws, base); + ws->base = base; gws_cookie = ssogws_get_cookie(ws); gws_cookie->event_dev = event_dev; @@ -1447,20 +1450,12 @@ sso_cleanup(struct rte_eventdev *event_dev, uint8_t enable) ws = event_dev->data->ports[i]; ssogws_reset((struct otx2_ssogws *)&ws->ws_state[0]); ssogws_reset((struct otx2_ssogws *)&ws->ws_state[1]); - ws->swtag_req = 0; ws->vws = 0; - ws->ws_state[0].cur_grp = 0; - ws->ws_state[0].cur_tt = SSO_SYNC_EMPTY; - ws->ws_state[1].cur_grp = 0; - ws->ws_state[1].cur_tt = SSO_SYNC_EMPTY; } else { struct otx2_ssogws *ws; ws = event_dev->data->ports[i]; ssogws_reset(ws); - ws->swtag_req = 0; - ws->cur_grp = 0; - ws->cur_tt = SSO_SYNC_EMPTY; } } @@ -1479,8 +1474,6 @@ sso_cleanup(struct rte_eventdev *event_dev, uint8_t enable) otx2_write64(enable, ws->grps_base[i] + SSO_LF_GGRP_QCTL); } - ws->ws_state[0].cur_grp = 0; - ws->ws_state[0].cur_tt = SSO_SYNC_EMPTY; } else { struct otx2_ssogws *ws = event_dev->data->ports[0]; @@ -1492,8 +1485,6 @@ sso_cleanup(struct rte_eventdev *event_dev, uint8_t enable) otx2_write64(enable, ws->grps_base[i] + SSO_LF_GGRP_QCTL); } - ws->cur_grp = 0; - ws->cur_tt = SSO_SYNC_EMPTY; } /* reset SSO GWS cache */ diff --git a/drivers/event/octeontx2/otx2_evdev.h b/drivers/event/octeontx2/otx2_evdev.h index 0513cb81c..e381b9e52 100644 --- a/drivers/event/octeontx2/otx2_evdev.h +++ b/drivers/event/octeontx2/otx2_evdev.h @@ -80,6 +80,8 @@ #define OTX2_SSOW_GET_BASE_ADDR(_GW) ((_GW) - SSOW_LF_GWS_OP_GET_WORK) #define OTX2_SSOW_TT_FROM_TAG(x) (((x) >> 32) & SSO_TT_EMPTY) +#define OTX2_SSOW_GRP_FROM_TAG(x) (((x) >> 36) & 0x3ff) +#define OTX2_SSOW_SWTAG_PEND(x) ((x) & BIT_ULL(62)) #define NSEC2USEC(__ns) ((__ns) / 1E3) #define USEC2NSEC(__us) ((__us) * 1E3) @@ -169,25 +171,23 @@ struct otx2_sso_evdev { uintptr_t wqp_op; \ uintptr_t swtag_flush_op; \ uintptr_t swtag_norm_op; \ - uintptr_t swtag_desched_op; \ - uint8_t cur_tt; \ - uint8_t cur_grp + uintptr_t swtag_desched_op; /* Event port aka GWS */ struct otx2_ssogws { /* Get Work Fastpath data */ OTX2_SSOGWS_OPS; - uint8_t swtag_req; + /* PTP timestamp */ + struct otx2_timesync_info *tstamp; void *lookup_mem; uint8_t port; /* Add Work Fastpath data */ uint64_t xaq_lmt __rte_cache_aligned; uint64_t *fc_mem; uintptr_t grps_base[OTX2_SSO_MAX_VHGRP]; - /* PTP timestamp */ - struct otx2_timesync_info *tstamp; /* Tx Fastpath data */ - uint8_t tx_adptr_data[] __rte_cache_aligned; + uint64_t base __rte_cache_aligned; + uint8_t tx_adptr_data[]; } __rte_cache_aligned; struct otx2_ssogws_state { @@ -197,18 +197,18 @@ struct otx2_ssogws_state { struct otx2_ssogws_dual { /* Get Work Fastpath data */ struct otx2_ssogws_state ws_state[2]; /* Ping and Pong */ - uint8_t swtag_req; - uint8_t vws; /* Ping pong bit */ + /* PTP timestamp */ + struct otx2_timesync_info *tstamp; void *lookup_mem; + uint8_t vws; /* Ping pong bit */ uint8_t port; /* Add Work Fastpath data */ uint64_t xaq_lmt __rte_cache_aligned; uint64_t *fc_mem; uintptr_t grps_base[OTX2_SSO_MAX_VHGRP]; - /* PTP timestamp */ - struct otx2_timesync_info *tstamp; /* Tx Fastpath data */ - uint8_t tx_adptr_data[] __rte_cache_aligned; + uint64_t base[2] __rte_cache_aligned; + uint8_t tx_adptr_data[]; } __rte_cache_aligned; static inline struct otx2_sso_evdev * diff --git a/drivers/event/octeontx2/otx2_worker.c b/drivers/event/octeontx2/otx2_worker.c index b098407e0..7ed836c1e 100644 --- a/drivers/event/octeontx2/otx2_worker.c +++ b/drivers/event/octeontx2/otx2_worker.c @@ -25,7 +25,7 @@ otx2_ssogws_fwd_swtag(struct otx2_ssogws *ws, const struct rte_event *ev) { const uint32_t tag = (uint32_t)ev->event; const uint8_t new_tt = ev->sched_type; - const uint8_t cur_tt = ws->cur_tt; + const uint8_t cur_tt = OTX2_SSOW_TT_FROM_TAG(otx2_read64(ws->tag_op)); /* 96XX model * cur_tt/new_tt SSO_SYNC_ORDERED SSO_SYNC_ATOMIC SSO_SYNC_UNTAGGED @@ -41,8 +41,6 @@ otx2_ssogws_fwd_swtag(struct otx2_ssogws *ws, const struct rte_event *ev) } else { otx2_ssogws_swtag_norm(ws, tag, new_tt); } - - ws->swtag_req = 1; } static __rte_always_inline void @@ -64,7 +62,7 @@ otx2_ssogws_forward_event(struct otx2_ssogws *ws, const struct rte_event *ev) const uint8_t grp = ev->queue_id; /* Group hasn't changed, Use SWTAG to forward the event */ - if (ws->cur_grp == grp) + if (OTX2_SSOW_GRP_FROM_TAG(otx2_read64(ws->tag_op)) == grp) otx2_ssogws_fwd_swtag(ws, ev); else /* @@ -75,12 +73,6 @@ otx2_ssogws_forward_event(struct otx2_ssogws *ws, const struct rte_event *ev) otx2_ssogws_fwd_group(ws, ev, grp); } -static __rte_always_inline void -otx2_ssogws_release_event(struct otx2_ssogws *ws) -{ - otx2_ssogws_swtag_flush(ws); -} - #define R(name, f6, f5, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_hot \ otx2_ssogws_deq_ ##name(void *port, struct rte_event *ev, \ @@ -90,8 +82,7 @@ otx2_ssogws_deq_ ##name(void *port, struct rte_event *ev, \ \ RTE_SET_USED(timeout_ticks); \ \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64(ws->tag_op))) { \ otx2_ssogws_swtag_wait(ws); \ return 1; \ } \ @@ -117,8 +108,7 @@ otx2_ssogws_deq_timeout_ ##name(void *port, struct rte_event *ev, \ uint16_t ret = 1; \ uint64_t iter; \ \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64(ws->tag_op))) { \ otx2_ssogws_swtag_wait(ws); \ return ret; \ } \ @@ -149,8 +139,7 @@ otx2_ssogws_deq_seg_ ##name(void *port, struct rte_event *ev, \ \ RTE_SET_USED(timeout_ticks); \ \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64(ws->tag_op))) { \ otx2_ssogws_swtag_wait(ws); \ return 1; \ } \ @@ -177,8 +166,7 @@ otx2_ssogws_deq_seg_timeout_ ##name(void *port, struct rte_event *ev, \ uint16_t ret = 1; \ uint64_t iter; \ \ - if (ws->swtag_req) { \ - ws->swtag_req = 0; \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64(ws->tag_op))) { \ otx2_ssogws_swtag_wait(ws); \ return ret; \ } \ @@ -221,7 +209,7 @@ otx2_ssogws_enq(void *port, const struct rte_event *ev) otx2_ssogws_forward_event(ws, ev); break; case RTE_EVENT_OP_RELEASE: - otx2_ssogws_release_event(ws); + otx2_ssogws_swtag_flush(ws->tag_op, ws->swtag_flush_op); break; default: return 0; @@ -274,14 +262,13 @@ otx2_ssogws_tx_adptr_enq_ ## name(void *port, struct rte_event ev[], \ { \ struct otx2_ssogws *ws = port; \ uint64_t cmd[sz]; \ - int i; \ \ - for (i = 0; i < nb_events; i++) \ - otx2_ssogws_event_tx(ws, &ev[i], cmd, (const uint64_t \ + RTE_SET_USED(nb_events); \ + return otx2_ssogws_event_tx(ws->base, &ev[0], cmd, \ + (const uint64_t \ (*)[RTE_MAX_QUEUES_PER_PORT]) \ &ws->tx_adptr_data, \ flags); \ - return nb_events; \ } SSO_TX_ADPTR_ENQ_FASTPATH_FUNC #undef T @@ -293,14 +280,13 @@ otx2_ssogws_tx_adptr_enq_seg_ ## name(void *port, struct rte_event ev[],\ { \ uint64_t cmd[(sz) + NIX_TX_MSEG_SG_DWORDS - 2]; \ struct otx2_ssogws *ws = port; \ - int i; \ \ - for (i = 0; i < nb_events; i++) \ - otx2_ssogws_event_tx(ws, &ev[i], cmd, (const uint64_t \ + RTE_SET_USED(nb_events); \ + return otx2_ssogws_event_tx(ws->base, &ev[0], cmd, \ + (const uint64_t \ (*)[RTE_MAX_QUEUES_PER_PORT]) \ &ws->tx_adptr_data, \ (flags) | NIX_TX_MULTI_SEG_F); \ - return nb_events; \ } SSO_TX_ADPTR_ENQ_FASTPATH_FUNC #undef T @@ -335,7 +321,7 @@ ssogws_flush_events(struct otx2_ssogws *ws, uint8_t queue_id, uintptr_t base, if (fn != NULL && ev.u64 != 0) fn(arg, ev); if (ev.sched_type != SSO_TT_EMPTY) - otx2_ssogws_swtag_flush(ws); + otx2_ssogws_swtag_flush(ws->tag_op, ws->swtag_flush_op); rte_mb(); aq_cnt = otx2_read64(base + SSO_LF_GGRP_AQ_CNT); ds_cnt = otx2_read64(base + SSO_LF_GGRP_MISC_CNT); diff --git a/drivers/event/octeontx2/otx2_worker.h b/drivers/event/octeontx2/otx2_worker.h index 0a7d6671c..2b716c042 100644 --- a/drivers/event/octeontx2/otx2_worker.h +++ b/drivers/event/octeontx2/otx2_worker.h @@ -64,8 +64,6 @@ otx2_ssogws_get_work(struct otx2_ssogws *ws, struct rte_event *ev, event.get_work0 = (event.get_work0 & (0x3ull << 32)) << 6 | (event.get_work0 & (0x3FFull << 36)) << 4 | (event.get_work0 & 0xffffffff); - ws->cur_tt = event.sched_type; - ws->cur_grp = event.queue_id; if (event.sched_type != SSO_TT_EMPTY) { if ((flags & NIX_RX_OFFLOAD_SECURITY_F) && @@ -136,8 +134,6 @@ otx2_ssogws_get_work_empty(struct otx2_ssogws *ws, struct rte_event *ev, event.get_work0 = (event.get_work0 & (0x3ull << 32)) << 6 | (event.get_work0 & (0x3FFull << 36)) << 4 | (event.get_work0 & 0xffffffff); - ws->cur_tt = event.sched_type; - ws->cur_grp = event.queue_id; if (event.sched_type != SSO_TT_EMPTY && event.event_type == RTE_EVENT_TYPE_ETHDEV) { @@ -192,18 +188,14 @@ otx2_ssogws_swtag_untag(struct otx2_ssogws *ws) { otx2_write64(0, OTX2_SSOW_GET_BASE_ADDR(ws->getwrk_op) + SSOW_LF_GWS_OP_SWTAG_UNTAG); - ws->cur_tt = SSO_SYNC_UNTAGGED; } static __rte_always_inline void -otx2_ssogws_swtag_flush(struct otx2_ssogws *ws) +otx2_ssogws_swtag_flush(uint64_t tag_op, uint64_t flush_op) { - if (OTX2_SSOW_TT_FROM_TAG(otx2_read64(ws->tag_op)) == SSO_TT_EMPTY) { - ws->cur_tt = SSO_SYNC_EMPTY; + if (OTX2_SSOW_TT_FROM_TAG(otx2_read64(tag_op)) == SSO_TT_EMPTY) return; - } - otx2_write64(0, ws->swtag_flush_op); - ws->cur_tt = SSO_SYNC_EMPTY; + otx2_write64(0, flush_op); } static __rte_always_inline void @@ -236,7 +228,7 @@ otx2_ssogws_swtag_wait(struct otx2_ssogws *ws) } static __rte_always_inline void -otx2_ssogws_head_wait(struct otx2_ssogws *ws) +otx2_ssogws_head_wait(uint64_t tag_op) { #ifdef RTE_ARCH_ARM64 uint64_t tag; @@ -250,11 +242,11 @@ otx2_ssogws_head_wait(struct otx2_ssogws *ws) " tbz %[tag], 35, rty%= \n" "done%=: \n" : [tag] "=&r" (tag) - : [tag_op] "r" (ws->tag_op) + : [tag_op] "r" (tag_op) ); #else /* Wait for the HEAD to be set */ - while (!(otx2_read64(ws->tag_op) & BIT_ULL(35))) + while (!(otx2_read64(tag_op) & BIT_ULL(35))) ; #endif } @@ -276,8 +268,7 @@ otx2_ssogws_prepare_pkt(const struct otx2_eth_txq *txq, struct rte_mbuf *m, } static __rte_always_inline uint16_t -otx2_ssogws_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, - uint64_t *cmd, +otx2_ssogws_event_tx(uint64_t base, struct rte_event *ev, uint64_t *cmd, const uint64_t txq_data[][RTE_MAX_QUEUES_PER_PORT], const uint32_t flags) { @@ -288,7 +279,7 @@ otx2_ssogws_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, if ((flags & NIX_TX_OFFLOAD_SECURITY_F) && (m->ol_flags & PKT_TX_SEC_OFFLOAD)) { txq = otx2_ssogws_xtract_meta(m, txq_data); - return otx2_sec_event_tx(ws, ev, m, txq, flags); + return otx2_sec_event_tx(base, ev, m, txq, flags); } /* Perform header writes before barrier for TSO */ @@ -309,7 +300,7 @@ otx2_ssogws_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, m->ol_flags, segdw, flags); if (!ev->sched_type) { otx2_nix_xmit_mseg_prep_lmt(cmd, txq->lmt_addr, segdw); - otx2_ssogws_head_wait(ws); + otx2_ssogws_head_wait(base + SSOW_LF_GWS_TAG); if (otx2_nix_xmit_submit_lmt(txq->io_addr) == 0) otx2_nix_xmit_mseg_one(cmd, txq->lmt_addr, txq->io_addr, segdw); @@ -324,7 +315,7 @@ otx2_ssogws_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, if (!ev->sched_type) { otx2_nix_xmit_prep_lmt(cmd, txq->lmt_addr, flags); - otx2_ssogws_head_wait(ws); + otx2_ssogws_head_wait(base + SSOW_LF_GWS_TAG); if (otx2_nix_xmit_submit_lmt(txq->io_addr) == 0) otx2_nix_xmit_one(cmd, txq->lmt_addr, txq->io_addr, flags); @@ -339,7 +330,8 @@ otx2_ssogws_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, return 1; } - otx2_ssogws_swtag_flush(ws); + otx2_ssogws_swtag_flush(base + SSOW_LF_GWS_TAG, + base + SSOW_LF_GWS_OP_SWTAG_FLUSH); return 1; } diff --git a/drivers/event/octeontx2/otx2_worker_dual.c b/drivers/event/octeontx2/otx2_worker_dual.c index 946488eab..820455788 100644 --- a/drivers/event/octeontx2/otx2_worker_dual.c +++ b/drivers/event/octeontx2/otx2_worker_dual.c @@ -26,9 +26,9 @@ static __rte_always_inline void otx2_ssogws_dual_fwd_swtag(struct otx2_ssogws_state *ws, const struct rte_event *ev) { + const uint8_t cur_tt = OTX2_SSOW_TT_FROM_TAG(otx2_read64(ws->tag_op)); const uint32_t tag = (uint32_t)ev->event; const uint8_t new_tt = ev->sched_type; - const uint8_t cur_tt = ws->cur_tt; /* 96XX model * cur_tt/new_tt SSO_SYNC_ORDERED SSO_SYNC_ATOMIC SSO_SYNC_UNTAGGED @@ -59,22 +59,20 @@ otx2_ssogws_dual_fwd_group(struct otx2_ssogws_state *ws, } static __rte_always_inline void -otx2_ssogws_dual_forward_event(struct otx2_ssogws_dual *ws, - struct otx2_ssogws_state *vws, +otx2_ssogws_dual_forward_event(struct otx2_ssogws_state *vws, const struct rte_event *ev) { const uint8_t grp = ev->queue_id; /* Group hasn't changed, Use SWTAG to forward the event */ - if (vws->cur_grp == grp) { + if (OTX2_SSOW_GRP_FROM_TAG(otx2_read64(vws->tag_op)) == grp) { otx2_ssogws_dual_fwd_swtag(vws, ev); - ws->swtag_req = 1; } else { - /* - * Group has been changed for group based work pipelining, - * Use deschedule/add_work operation to transfer the event to - * new group/core - */ + /* + * Group has been changed for group based work pipelining, + * Use deschedule/add_work operation to transfer the event to + * new group/core + */ otx2_ssogws_dual_fwd_group(vws, ev, grp); } } @@ -90,10 +88,10 @@ otx2_ssogws_dual_enq(void *port, const struct rte_event *ev) rte_smp_mb(); return otx2_ssogws_dual_new_event(ws, ev); case RTE_EVENT_OP_FORWARD: - otx2_ssogws_dual_forward_event(ws, vws, ev); + otx2_ssogws_dual_forward_event(vws, ev); break; case RTE_EVENT_OP_RELEASE: - otx2_ssogws_swtag_flush((struct otx2_ssogws *)vws); + otx2_ssogws_swtag_flush(vws->tag_op, vws->swtag_flush_op); break; default: return 0; @@ -135,7 +133,7 @@ otx2_ssogws_dual_enq_fwd_burst(void *port, const struct rte_event ev[], struct otx2_ssogws_state *vws = &ws->ws_state[!ws->vws]; RTE_SET_USED(nb_events); - otx2_ssogws_dual_forward_event(ws, vws, ev); + otx2_ssogws_dual_forward_event(vws, ev); return 1; } @@ -150,10 +148,10 @@ otx2_ssogws_dual_deq_ ##name(void *port, struct rte_event *ev, \ \ rte_prefetch_non_temporal(ws); \ RTE_SET_USED(timeout_ticks); \ - if (ws->swtag_req) { \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64( \ + ws->ws_state[!ws->vws].tag_op))) { \ otx2_ssogws_swtag_wait((struct otx2_ssogws *) \ &ws->ws_state[!ws->vws]); \ - ws->swtag_req = 0; \ return 1; \ } \ \ @@ -184,10 +182,10 @@ otx2_ssogws_dual_deq_timeout_ ##name(void *port, struct rte_event *ev, \ uint64_t iter; \ uint8_t gw; \ \ - if (ws->swtag_req) { \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64( \ + ws->ws_state[!ws->vws].tag_op))) { \ otx2_ssogws_swtag_wait((struct otx2_ssogws *) \ &ws->ws_state[!ws->vws]); \ - ws->swtag_req = 0; \ return 1; \ } \ \ @@ -228,10 +226,10 @@ otx2_ssogws_dual_deq_seg_ ##name(void *port, struct rte_event *ev, \ uint8_t gw; \ \ RTE_SET_USED(timeout_ticks); \ - if (ws->swtag_req) { \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64( \ + ws->ws_state[!ws->vws].tag_op))) { \ otx2_ssogws_swtag_wait((struct otx2_ssogws *) \ &ws->ws_state[!ws->vws]); \ - ws->swtag_req = 0; \ return 1; \ } \ \ @@ -266,10 +264,10 @@ otx2_ssogws_dual_deq_seg_timeout_ ##name(void *port, \ uint64_t iter; \ uint8_t gw; \ \ - if (ws->swtag_req) { \ + if (OTX2_SSOW_SWTAG_PEND(otx2_read64( \ + ws->ws_state[!ws->vws].tag_op))) { \ otx2_ssogws_swtag_wait((struct otx2_ssogws *) \ &ws->ws_state[!ws->vws]); \ - ws->swtag_req = 0; \ return 1; \ } \ \ @@ -314,15 +312,13 @@ otx2_ssogws_dual_tx_adptr_enq_ ## name(void *port, \ uint16_t nb_events) \ { \ struct otx2_ssogws_dual *ws = port; \ - struct otx2_ssogws *vws = \ - (struct otx2_ssogws *)&ws->ws_state[!ws->vws]; \ uint64_t cmd[sz]; \ \ RTE_SET_USED(nb_events); \ - return otx2_ssogws_event_tx(vws, ev, cmd, (const uint64_t \ - (*)[RTE_MAX_QUEUES_PER_PORT]) \ - ws->tx_adptr_data, \ - flags); \ + return otx2_ssogws_event_tx(ws->base[!ws->vws], &ev[0], \ + cmd, (const uint64_t \ + (*)[RTE_MAX_QUEUES_PER_PORT]) \ + &ws->tx_adptr_data, flags); \ } SSO_TX_ADPTR_ENQ_FASTPATH_FUNC #undef T @@ -333,16 +329,15 @@ otx2_ssogws_dual_tx_adptr_enq_seg_ ## name(void *port, \ struct rte_event ev[], \ uint16_t nb_events) \ { \ - struct otx2_ssogws_dual *ws = port; \ - struct otx2_ssogws *vws = \ - (struct otx2_ssogws *)&ws->ws_state[!ws->vws]; \ uint64_t cmd[(sz) + NIX_TX_MSEG_SG_DWORDS - 2]; \ + struct otx2_ssogws_dual *ws = port; \ \ RTE_SET_USED(nb_events); \ - return otx2_ssogws_event_tx(vws, ev, cmd, (const uint64_t \ - (*)[RTE_MAX_QUEUES_PER_PORT]) \ - ws->tx_adptr_data, \ - (flags) | NIX_TX_MULTI_SEG_F); \ + return otx2_ssogws_event_tx(ws->base[!ws->vws], &ev[0], \ + cmd, (const uint64_t \ + (*)[RTE_MAX_QUEUES_PER_PORT]) \ + &ws->tx_adptr_data, \ + (flags) | NIX_TX_MULTI_SEG_F);\ } SSO_TX_ADPTR_ENQ_FASTPATH_FUNC #undef T diff --git a/drivers/event/octeontx2/otx2_worker_dual.h b/drivers/event/octeontx2/otx2_worker_dual.h index 6e6061821..72b616439 100644 --- a/drivers/event/octeontx2/otx2_worker_dual.h +++ b/drivers/event/octeontx2/otx2_worker_dual.h @@ -61,8 +61,6 @@ otx2_ssogws_dual_get_work(struct otx2_ssogws_state *ws, event.get_work0 = (event.get_work0 & (0x3ull << 32)) << 6 | (event.get_work0 & (0x3FFull << 36)) << 4 | (event.get_work0 & 0xffffffff); - ws->cur_tt = event.sched_type; - ws->cur_grp = event.queue_id; if (event.sched_type != SSO_TT_EMPTY) { if ((flags & NIX_RX_OFFLOAD_SECURITY_F) && diff --git a/drivers/net/octeontx2/otx2_ethdev_sec_tx.h b/drivers/net/octeontx2/otx2_ethdev_sec_tx.h index 284bcd536..c8eae3d62 100644 --- a/drivers/net/octeontx2/otx2_ethdev_sec_tx.h +++ b/drivers/net/octeontx2/otx2_ethdev_sec_tx.h @@ -30,12 +30,11 @@ otx2_ipsec_fp_out_rlen_get(struct otx2_sec_session_ipsec_ip *sess, } static __rte_always_inline void -otx2_ssogws_head_wait(struct otx2_ssogws *ws); +otx2_ssogws_head_wait(uint64_t base); static __rte_always_inline int -otx2_sec_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, - struct rte_mbuf *m, const struct otx2_eth_txq *txq, - const uint32_t offload_flags) +otx2_sec_event_tx(uint64_t base, struct rte_event *ev, struct rte_mbuf *m, + const struct otx2_eth_txq *txq, const uint32_t offload_flags) { uint32_t dlen, rlen, desc_headroom, extend_head, extend_tail; struct otx2_sec_session_ipsec_ip *sess; @@ -149,7 +148,7 @@ otx2_sec_event_tx(struct otx2_ssogws *ws, struct rte_event *ev, __mempool_check_cookies(m->pool, (void **)&m, 1, 0); if (!ev->sched_type) - otx2_ssogws_head_wait(ws); + otx2_ssogws_head_wait(base + SSOW_LF_GWS_TAG); inst.param1 = sess->esn_hi >> 16; inst.param2 = sess->esn_hi & 0xffff; -- 2.17.1