From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E02F6A0C46; Fri, 18 Jun 2021 12:45:44 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BC91541175; Fri, 18 Jun 2021 12:41:50 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 7C4E34121C for ; Fri, 18 Jun 2021 12:41:49 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15IAZdve003612 for ; Fri, 18 Jun 2021 03:41:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=dfjiGWCJHwoWnijHaLhk8ONDil7iSTGyTnJaUrIFNRg=; b=gJHEHva8CA1AicF/5Yeel7P1H0i4bnkxZhf70ShpBB4HUgKEsxya9KDTbCidrJY3XOO9 aPseklaTnrsiUSqvPZuT1Rz0nLLU5Cph+obadesmgBsfPnjhay5yPB2ys2e+BFTIUGkD u3arFgJVljT+S1mz0PFJoQ2iydG45yIM00PCeAf2CJf5GS2GSgmosHg/LafZgR6Mt0Ml TBggaFdXGkwf53F4ZjL4gVsxeajI43AwXsuQfwxqdnPGP1xreEXIjeuuMf9GAfn79eXE DpWjlUtzY9o6ZkvKMP2RCQS/IrGYh5HjeDdggEJAsxs6cQCx4nrueNI2TFcPiEyU3RF8 sg== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 397udry7p7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Fri, 18 Jun 2021 03:41:48 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 18 Jun 2021 03:41:45 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Fri, 18 Jun 2021 03:41:46 -0700 Received: from hyd1588t430.marvell.com (unknown [10.29.52.204]) by maili.marvell.com (Postfix) with ESMTP id 87CAB5B697F; Fri, 18 Jun 2021 03:41:24 -0700 (PDT) From: Nithin Dabilpuram To: CC: , , , , , , , Date: Fri, 18 Jun 2021 16:07:34 +0530 Message-ID: <20210618103741.26526-56-ndabilpuram@marvell.com> X-Mailer: git-send-email 2.8.4 In-Reply-To: <20210618103741.26526-1-ndabilpuram@marvell.com> References: <20210306153404.10781-1-ndabilpuram@marvell.com> <20210618103741.26526-1-ndabilpuram@marvell.com> MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-ORIG-GUID: RTNpTV--L6-cg7Jvd8iw99-EsIhN0opT X-Proofpoint-GUID: RTNpTV--L6-cg7Jvd8iw99-EsIhN0opT X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-06-18_04:2021-06-18, 2021-06-18 signatures=0 Subject: [dpdk-dev] [PATCH v3 55/62] net/cnxk: add base PTP timesync support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Sunil Kumar Kori Base PTP timesync support is added for cn9k and cn10k platforms. Signed-off-by: Sunil Kumar Kori --- drivers/net/cnxk/cn10k_ethdev.c | 31 +++++ drivers/net/cnxk/cn10k_ethdev.h | 1 + drivers/net/cnxk/cn10k_rx.c | 32 ++--- drivers/net/cnxk/cn10k_rx.h | 61 +++++++--- drivers/net/cnxk/cn10k_rx_mseg.c | 2 +- drivers/net/cnxk/cn10k_rx_vec.c | 5 +- drivers/net/cnxk/cn10k_tx.c | 28 +++-- drivers/net/cnxk/cn10k_tx.h | 193 +++++++++++++++++++++++------ drivers/net/cnxk/cn10k_tx_mseg.c | 2 +- drivers/net/cnxk/cn10k_tx_vec.c | 3 +- drivers/net/cnxk/cn9k_ethdev.c | 34 +++++- drivers/net/cnxk/cn9k_ethdev.h | 1 + drivers/net/cnxk/cn9k_rx.c | 32 ++--- drivers/net/cnxk/cn9k_rx.h | 61 +++++++--- drivers/net/cnxk/cn9k_rx_mseg.c | 2 +- drivers/net/cnxk/cn9k_rx_vec.c | 5 +- drivers/net/cnxk/cn9k_tx.c | 28 +++-- drivers/net/cnxk/cn9k_tx.h | 253 ++++++++++++++++++++++++++++----------- drivers/net/cnxk/cn9k_tx_mseg.c | 2 +- drivers/net/cnxk/cn9k_tx_vec.c | 3 +- drivers/net/cnxk/cnxk_ethdev.c | 36 +++++- drivers/net/cnxk/cnxk_ethdev.h | 64 +++++++++- drivers/net/cnxk/cnxk_ptp.c | 169 ++++++++++++++++++++++++++ drivers/net/cnxk/meson.build | 1 + 24 files changed, 834 insertions(+), 215 deletions(-) create mode 100644 drivers/net/cnxk/cnxk_ptp.c diff --git a/drivers/net/cnxk/cn10k_ethdev.c b/drivers/net/cnxk/cn10k_ethdev.c index bddb7fb..5e0de13 100644 --- a/drivers/net/cnxk/cn10k_ethdev.c +++ b/drivers/net/cnxk/cn10k_ethdev.c @@ -30,6 +30,9 @@ nix_rx_offload_flags(struct rte_eth_dev *eth_dev) if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) flags |= NIX_RX_MULTI_SEG_F; + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) + flags |= NIX_RX_OFFLOAD_TSTAMP_F; + if (!dev->ptype_disable) flags |= NIX_RX_OFFLOAD_PTYPE_F; @@ -95,6 +98,9 @@ nix_tx_offload_flags(struct rte_eth_dev *eth_dev) flags |= (NIX_TX_OFFLOAD_TSO_F | NIX_TX_OFFLOAD_OL3_OL4_CSUM_F | NIX_TX_OFFLOAD_L3_L4_CSUM_F); + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) + flags |= NIX_TX_OFFLOAD_TSTAMP_F; + return flags; } @@ -121,6 +127,7 @@ nix_form_default_desc(struct cnxk_eth_dev *dev, struct cn10k_eth_txq *txq, { struct nix_send_ext_s *send_hdr_ext; union nix_send_hdr_w0_u send_hdr_w0; + struct nix_send_mem_s *send_mem; union nix_send_sg_s sg_w0; RTE_SET_USED(dev); @@ -136,6 +143,22 @@ nix_form_default_desc(struct cnxk_eth_dev *dev, struct cn10k_eth_txq *txq, send_hdr_ext = (struct nix_send_ext_s *)&txq->cmd[0]; send_hdr_ext->w0.subdc = NIX_SUBDC_EXT; + if (dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F) { + /* Default: one seg packet would have: + * 2(HDR) + 2(EXT) + 1(SG) + 1(IOVA) + 2(MEM) + * => 8/2 - 1 = 3 + */ + send_hdr_w0.sizem1 = 3; + send_hdr_ext->w0.tstmp = 1; + + /* To calculate the offset for send_mem, + * send_hdr->w0.sizem1 * 2 + */ + send_mem = (struct nix_send_mem_s *)(txq->cmd + 2); + send_mem->w0.subdc = NIX_SUBDC_MEM; + send_mem->w0.alg = NIX_SENDMEMALG_SETTSTMP; + send_mem->addr = dev->tstamp.tx_tstamp_iova; + } } else { /* 2(HDR) + 1(SG) + 1(IOVA) = 4/2 - 1 = 1 */ send_hdr_w0.sizem1 = 1; @@ -221,6 +244,7 @@ cn10k_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rxq->wdata = cq->wdata; rxq->head = cq->head; rxq->qmask = cq->qmask; + rxq->tstamp = &dev->tstamp; /* Data offset from data to start of mbuf is first_skip */ rxq->data_off = rq->first_skip; @@ -342,6 +366,7 @@ static int cn10k_nix_dev_start(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_nix *nix = &dev->nix; int rc; /* Common eth dev start */ @@ -349,6 +374,12 @@ cn10k_nix_dev_start(struct rte_eth_dev *eth_dev) if (rc) return rc; + /* Update VF about data off shifted by 8 bytes if PTP already + * enabled in PF owning this VF + */ + if (dev->ptp_en && (!roc_nix_is_pf(nix) && (!roc_nix_is_sdp(nix)))) + nix_ptp_enable_vf(eth_dev); + /* Setting up the rx[tx]_offload_flags due to change * in rx[tx]_offloads. */ diff --git a/drivers/net/cnxk/cn10k_ethdev.h b/drivers/net/cnxk/cn10k_ethdev.h index d39ca31..8b6e0f2 100644 --- a/drivers/net/cnxk/cn10k_ethdev.h +++ b/drivers/net/cnxk/cn10k_ethdev.h @@ -31,6 +31,7 @@ struct cn10k_eth_rxq { uint32_t available; uint16_t data_off; uint16_t rq; + struct cnxk_timesync_info *tstamp; } __plt_cache_aligned; /* Rx and Tx routines */ diff --git a/drivers/net/cnxk/cn10k_rx.c b/drivers/net/cnxk/cn10k_rx.c index 0598111..c9744e2 100644 --- a/drivers/net/cnxk/cn10k_rx.c +++ b/drivers/net/cnxk/cn10k_rx.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ @@ -17,12 +17,13 @@ NIX_RX_FASTPATH_MODES static inline void pick_rx_func(struct rte_eth_dev *eth_dev, - const eth_rx_burst_t rx_burst[2][2][2][2]) + const eth_rx_burst_t rx_burst[2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - /* [MARK] [CKSUM] [PTYPE] [RSS] */ + /* [TSP] [MARK] [CKSUM] [PTYPE] [RSS] */ eth_dev->rx_pkt_burst = rx_burst + [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_TSTAMP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_MARK_UPDATE_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_CHECKSUM_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_PTYPE_F)] @@ -34,31 +35,34 @@ cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, + const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, + const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_mseg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn10k_nix_recv_pkts_vec_##name, NIX_RX_FASTPATH_MODES #undef R }; - if (dev->scalar_ena) + /* For PTP enabled, scalar rx function should be chosen as most of the + * PTP apps are implemented to rx burst 1 pkt. + */ + if (dev->scalar_ena || dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP) pick_rx_func(eth_dev, nix_eth_rx_burst); else pick_rx_func(eth_dev, nix_eth_rx_vec_burst); @@ -69,6 +73,6 @@ cn10k_eth_set_rx_function(struct rte_eth_dev *eth_dev) /* Copy multi seg version with no offload for tear down sequence */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) dev->rx_pkt_burst_no_offload = - nix_eth_rx_burst_mseg[0][0][0][0]; + nix_eth_rx_burst_mseg[0][0][0][0][0]; rte_mb(); } diff --git a/drivers/net/cnxk/cn10k_rx.h b/drivers/net/cnxk/cn10k_rx.h index 29ee0ac..c09ccdf 100644 --- a/drivers/net/cnxk/cn10k_rx.h +++ b/drivers/net/cnxk/cn10k_rx.h @@ -7,6 +7,8 @@ #include #include +#include + #define NIX_RX_OFFLOAD_NONE (0) #define NIX_RX_OFFLOAD_RSS_F BIT(0) #define NIX_RX_OFFLOAD_PTYPE_F BIT(1) @@ -250,6 +252,10 @@ cn10k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, cn10k_nix_cqe_to_mbuf(cq, cq->tag, mbuf, lookup_mem, mbuf_init, flags); + cnxk_nix_mbuf_to_tstamp(mbuf, rxq->tstamp, + (flags & NIX_RX_OFFLOAD_TSTAMP_F), + (uint64_t *)((uint8_t *)mbuf + data_off) + ); rx_pkts[packets++] = mbuf; roc_prefetch_store_keep(mbuf); head++; @@ -487,27 +493,44 @@ cn10k_nix_recv_pkts_vector(void *rx_queue, struct rte_mbuf **rx_pkts, #define PTYPE_F NIX_RX_OFFLOAD_PTYPE_F #define CKSUM_F NIX_RX_OFFLOAD_CHECKSUM_F #define MARK_F NIX_RX_OFFLOAD_MARK_UPDATE_F +#define TS_F NIX_RX_OFFLOAD_TSTAMP_F -/* [MARK] [CKSUM] [PTYPE] [RSS] */ -#define NIX_RX_FASTPATH_MODES \ -R(no_offload, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ -R(rss, 0, 0, 0, 1, RSS_F) \ -R(ptype, 0, 0, 1, 0, PTYPE_F) \ -R(ptype_rss, 0, 0, 1, 1, PTYPE_F | RSS_F) \ -R(cksum, 0, 1, 0, 0, CKSUM_F) \ -R(cksum_rss, 0, 1, 0, 1, CKSUM_F | RSS_F) \ -R(cksum_ptype, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ -R(cksum_ptype_rss, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ -R(mark, 1, 0, 0, 0, MARK_F) \ -R(mark_rss, 1, 0, 0, 1, MARK_F | RSS_F) \ -R(mark_ptype, 1, 0, 1, 0, MARK_F | PTYPE_F) \ -R(mark_ptype_rss, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ -R(mark_cksum, 1, 1, 0, 0, MARK_F | CKSUM_F) \ -R(mark_cksum_rss, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ -R(mark_cksum_ptype, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F)\ -R(mark_cksum_ptype_rss, 1, 1, 1, 1, MARK_F | CKSUM_F | PTYPE_F | RSS_F) +/* [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ +#define NIX_RX_FASTPATH_MODES \ +R(no_offload, 0, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ +R(rss, 0, 0, 0, 0, 1, RSS_F) \ +R(ptype, 0, 0, 0, 1, 0, PTYPE_F) \ +R(ptype_rss, 0, 0, 0, 1, 1, PTYPE_F | RSS_F) \ +R(cksum, 0, 0, 1, 0, 0, CKSUM_F) \ +R(cksum_rss, 0, 0, 1, 0, 1, CKSUM_F | RSS_F) \ +R(cksum_ptype, 0, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ +R(cksum_ptype_rss, 0, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ +R(mark, 0, 1, 0, 0, 0, MARK_F) \ +R(mark_rss, 0, 1, 0, 0, 1, MARK_F | RSS_F) \ +R(mark_ptype, 0, 1, 0, 1, 0, MARK_F | PTYPE_F) \ +R(mark_ptype_rss, 0, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ +R(mark_cksum, 0, 1, 1, 0, 0, MARK_F | CKSUM_F) \ +R(mark_cksum_rss, 0, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ +R(mark_cksum_ptype, 0, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F) \ +R(mark_cksum_ptype_rss, 0, 1, 1, 1, 1, MARK_F | CKSUM_F | PTYPE_F | RSS_F)\ +R(ts, 1, 0, 0, 0, 0, TS_F) \ +R(ts_rss, 1, 0, 0, 0, 1, TS_F | RSS_F) \ +R(ts_ptype, 1, 0, 0, 1, 0, TS_F | PTYPE_F) \ +R(ts_ptype_rss, 1, 0, 0, 1, 1, TS_F | PTYPE_F | RSS_F) \ +R(ts_cksum, 1, 0, 1, 0, 0, TS_F | CKSUM_F) \ +R(ts_cksum_rss, 1, 0, 1, 0, 1, TS_F | CKSUM_F | RSS_F) \ +R(ts_cksum_ptype, 1, 0, 1, 1, 0, TS_F | CKSUM_F | PTYPE_F) \ +R(ts_cksum_ptype_rss, 1, 0, 1, 1, 1, TS_F | CKSUM_F | PTYPE_F | RSS_F)\ +R(ts_mark, 1, 1, 0, 0, 0, TS_F | MARK_F) \ +R(ts_mark_rss, 1, 1, 0, 0, 1, TS_F | MARK_F | RSS_F) \ +R(ts_mark_ptype, 1, 1, 0, 1, 0, TS_F | MARK_F | PTYPE_F) \ +R(ts_mark_ptype_rss, 1, 1, 0, 1, 1, TS_F | MARK_F | PTYPE_F | RSS_F)\ +R(ts_mark_cksum, 1, 1, 1, 0, 0, TS_F | MARK_F | CKSUM_F) \ +R(ts_mark_cksum_rss, 1, 1, 1, 0, 1, TS_F | MARK_F | CKSUM_F | RSS_F)\ +R(ts_mark_cksum_ptype, 1, 1, 1, 1, 0, TS_F | MARK_F | CKSUM_F | PTYPE_F)\ +R(ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn10k_rx_mseg.c b/drivers/net/cnxk/cn10k_rx_mseg.c index 9d283f7..b67d21f 100644 --- a/drivers/net/cnxk/cn10k_rx_mseg.c +++ b/drivers/net/cnxk/cn10k_rx_mseg.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_recv_pkts_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cn10k_rx_vec.c b/drivers/net/cnxk/cn10k_rx_vec.c index 0fa079c..1330235 100644 --- a/drivers/net/cnxk/cn10k_rx_vec.c +++ b/drivers/net/cnxk/cn10k_rx_vec.c @@ -5,12 +5,15 @@ #include "cn10k_ethdev.h" #include "cn10k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_recv_pkts_vec_##name(void *rx_queue, \ struct rte_mbuf **rx_pkts, \ uint16_t pkts) \ { \ + /* TSTMP is not supported by vector */ \ + if ((flags) & NIX_RX_OFFLOAD_TSTAMP_F) \ + return 0; \ return cn10k_nix_recv_pkts_vector(rx_queue, rx_pkts, pkts, \ (flags)); \ } diff --git a/drivers/net/cnxk/cn10k_tx.c b/drivers/net/cnxk/cn10k_tx.c index e6eb101..18694dc 100644 --- a/drivers/net/cnxk/cn10k_tx.c +++ b/drivers/net/cnxk/cn10k_tx.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ @@ -24,12 +24,13 @@ NIX_TX_FASTPATH_MODES static inline void pick_tx_func(struct rte_eth_dev *eth_dev, - const eth_tx_burst_t tx_burst[2][2][2][2][2]) + const eth_tx_burst_t tx_burst[2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - /* [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ + /* [TSP] [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ eth_dev->tx_pkt_burst = tx_burst + [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSO_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_VLAN_QINQ_F)] @@ -42,25 +43,25 @@ cn10k_eth_set_tx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_##name, + const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_mseg_##name, + const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_mseg_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn10k_nix_xmit_pkts_vec_##name, NIX_TX_FASTPATH_MODES #undef T @@ -68,7 +69,8 @@ cn10k_eth_set_tx_function(struct rte_eth_dev *eth_dev) if (dev->scalar_ena || (dev->tx_offload_flags & - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F))) + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F | + NIX_TX_OFFLOAD_TSO_F))) pick_tx_func(eth_dev, nix_eth_tx_burst); else pick_tx_func(eth_dev, nix_eth_tx_vec_burst); diff --git a/drivers/net/cnxk/cn10k_tx.h b/drivers/net/cnxk/cn10k_tx.h index b74df10..8b1446f 100644 --- a/drivers/net/cnxk/cn10k_tx.h +++ b/drivers/net/cnxk/cn10k_tx.h @@ -12,6 +12,7 @@ #define NIX_TX_OFFLOAD_VLAN_QINQ_F BIT(2) #define NIX_TX_OFFLOAD_MBUF_NOFF_F BIT(3) #define NIX_TX_OFFLOAD_TSO_F BIT(4) +#define NIX_TX_OFFLOAD_TSTAMP_F BIT(5) /* Flags to control xmit_prepare function. * Defining it from backwards to denote its been @@ -24,7 +25,8 @@ NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F) #define NIX_TX_NEED_EXT_HDR \ - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F) + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F | \ + NIX_TX_OFFLOAD_TSO_F) #define NIX_XMIT_FC_OR_RETURN(txq, pkts) \ do { \ @@ -49,8 +51,12 @@ static __rte_always_inline int cn10k_nix_tx_ext_subs(const uint16_t flags) { - return (flags & - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) ? 1 : 0; + return (flags & NIX_TX_OFFLOAD_TSTAMP_F) + ? 2 + : ((flags & + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) + ? 1 + : 0); } static __rte_always_inline uint8_t @@ -380,6 +386,45 @@ cn10k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, uintptr_t lmt_addr, *(rte_iova_t *)(lmt_addr + 8) = *(rte_iova_t *)(sg + 1); } +static __rte_always_inline void +cn10k_nix_xmit_prepare_tstamp(uintptr_t lmt_addr, const uint64_t *cmd, + const uint64_t ol_flags, const uint16_t no_segdw, + const uint16_t flags) +{ + if (flags & NIX_TX_OFFLOAD_TSTAMP_F) { + const uint8_t is_ol_tstamp = !(ol_flags & PKT_TX_IEEE1588_TMST); + struct nix_send_ext_s *send_hdr_ext = + (struct nix_send_ext_s *)lmt_addr + 16; + uint64_t *lmt = (uint64_t *)lmt_addr; + uint16_t off = (no_segdw - 1) << 1; + struct nix_send_mem_s *send_mem; + + send_mem = (struct nix_send_mem_s *)(lmt + off); + send_hdr_ext->w0.subdc = NIX_SUBDC_EXT; + send_hdr_ext->w0.tstmp = 1; + if (flags & NIX_TX_MULTI_SEG_F) { + /* Retrieving the default desc values */ + lmt[off] = cmd[2]; + + /* Using compiler barier to avoid voilation of C + * aliasing rules. + */ + rte_compiler_barrier(); + } + + /* Packets for which PKT_TX_IEEE1588_TMST is not set, tx tstamp + * should not be recorded, hence changing the alg type to + * NIX_SENDMEMALG_SET and also changing send mem addr field to + * next 8 bytes as it corrpt the actual tx tstamp registered + * address. + */ + send_mem->w0.subdc = NIX_SUBDC_MEM; + send_mem->w0.alg = NIX_SENDMEMALG_SETTSTMP - (is_ol_tstamp); + send_mem->addr = + (rte_iova_t)(((uint64_t *)cmd[3]) + is_ol_tstamp); + } +} + static __rte_always_inline uint16_t cn10k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) { @@ -445,7 +490,7 @@ cn10k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) /* Roundup extra dwords to multiple of 2 */ segdw = (segdw >> 1) + (segdw & 0x1); /* Default dwords */ - segdw += (off >> 1) + 1; + segdw += (off >> 1) + 1 + !!(flags & NIX_TX_OFFLOAD_TSTAMP_F); send_hdr->w0.sizem1 = segdw - 1; return segdw; @@ -487,6 +532,8 @@ cn10k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, cn10k_nix_xmit_prepare(tx_pkts[i], cmd, lmt_addr, flags, lso_tun_fmt); + cn10k_nix_xmit_prepare_tstamp(lmt_addr, &txq->cmd[0], + tx_pkts[i]->ol_flags, 4, flags); lmt_addr += (1ULL << ROC_LMT_LINE_SIZE_LOG2); } @@ -576,6 +623,9 @@ cn10k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, /* Store sg list directly on lmt line */ segdw = cn10k_nix_prepare_mseg(tx_pkts[i], (uint64_t *)lmt_addr, flags); + cn10k_nix_xmit_prepare_tstamp(lmt_addr, &txq->cmd[0], + tx_pkts[i]->ol_flags, segdw, + flags); lmt_addr += (1ULL << ROC_LMT_LINE_SIZE_LOG2); data128 |= (((__uint128_t)(segdw - 1)) << shft); shft += 3; @@ -1406,75 +1456,140 @@ cn10k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, #define VLAN_F NIX_TX_OFFLOAD_VLAN_QINQ_F #define NOFF_F NIX_TX_OFFLOAD_MBUF_NOFF_F #define TSO_F NIX_TX_OFFLOAD_TSO_F +#define TSP_F NIX_TX_OFFLOAD_TSTAMP_F -/* [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ +/* [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ #define NIX_TX_FASTPATH_MODES \ -T(no_offload, 0, 0, 0, 0, 0, 4, \ +T(no_offload, 0, 0, 0, 0, 0, 0, 4, \ NIX_TX_OFFLOAD_NONE) \ -T(l3l4csum, 0, 0, 0, 0, 1, 4, \ +T(l3l4csum, 0, 0, 0, 0, 0, 1, 4, \ L3L4CSUM_F) \ -T(ol3ol4csum, 0, 0, 0, 1, 0, 4, \ +T(ol3ol4csum, 0, 0, 0, 0, 1, 0, 4, \ OL3OL4CSUM_F) \ -T(ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 4, \ +T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 4, \ OL3OL4CSUM_F | L3L4CSUM_F) \ -T(vlan, 0, 0, 1, 0, 0, 6, \ +T(vlan, 0, 0, 0, 1, 0, 0, 6, \ VLAN_F) \ -T(vlan_l3l4csum, 0, 0, 1, 0, 1, 6, \ +T(vlan_l3l4csum, 0, 0, 0, 1, 0, 1, 6, \ VLAN_F | L3L4CSUM_F) \ -T(vlan_ol3ol4csum, 0, 0, 1, 1, 0, 6, \ +T(vlan_ol3ol4csum, 0, 0, 0, 1, 1, 0, 6, \ VLAN_F | OL3OL4CSUM_F) \ -T(vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 6, \ +T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 6, \ VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff, 0, 1, 0, 0, 0, 4, \ +T(noff, 0, 0, 1, 0, 0, 0, 4, \ NOFF_F) \ -T(noff_l3l4csum, 0, 1, 0, 0, 1, 4, \ +T(noff_l3l4csum, 0, 0, 1, 0, 0, 1, 4, \ NOFF_F | L3L4CSUM_F) \ -T(noff_ol3ol4csum, 0, 1, 0, 1, 0, 4, \ +T(noff_ol3ol4csum, 0, 0, 1, 0, 1, 0, 4, \ NOFF_F | OL3OL4CSUM_F) \ -T(noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 4, \ +T(noff_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 4, \ NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff_vlan, 0, 1, 1, 0, 0, 6, \ +T(noff_vlan, 0, 0, 1, 1, 0, 0, 6, \ NOFF_F | VLAN_F) \ -T(noff_vlan_l3l4csum, 0, 1, 1, 0, 1, 6, \ +T(noff_vlan_l3l4csum, 0, 0, 1, 1, 0, 1, 6, \ NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(noff_vlan_ol3ol4csum, 0, 1, 1, 1, 0, 6, \ +T(noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 0, 6, \ NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 6, \ +T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 6, \ NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso, 1, 0, 0, 0, 0, 6, \ +T(tso, 0, 1, 0, 0, 0, 0, 6, \ TSO_F) \ -T(tso_l3l4csum, 1, 0, 0, 0, 1, 6, \ +T(tso_l3l4csum, 0, 1, 0, 0, 0, 1, 6, \ TSO_F | L3L4CSUM_F) \ -T(tso_ol3ol4csum, 1, 0, 0, 1, 0, 6, \ +T(tso_ol3ol4csum, 0, 1, 0, 0, 1, 0, 6, \ TSO_F | OL3OL4CSUM_F) \ -T(tso_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 6, \ +T(tso_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 6, \ TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_vlan, 1, 0, 1, 0, 0, 6, \ +T(tso_vlan, 0, 1, 0, 1, 0, 0, 6, \ TSO_F | VLAN_F) \ -T(tso_vlan_l3l4csum, 1, 0, 1, 0, 1, 6, \ +T(tso_vlan_l3l4csum, 0, 1, 0, 1, 0, 1, 6, \ TSO_F | VLAN_F | L3L4CSUM_F) \ -T(tso_vlan_ol3ol4csum, 1, 0, 1, 1, 0, 6, \ +T(tso_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 0, 6, \ TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 6, \ +T(tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 6, \ TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff, 1, 1, 0, 0, 0, 6, \ +T(tso_noff, 0, 1, 1, 0, 0, 0, 6, \ TSO_F | NOFF_F) \ -T(tso_noff_l3l4csum, 1, 1, 0, 0, 1, 6, \ +T(tso_noff_l3l4csum, 0, 1, 1, 0, 0, 1, 6, \ TSO_F | NOFF_F | L3L4CSUM_F) \ -T(tso_noff_ol3ol4csum, 1, 1, 0, 1, 0, 6, \ +T(tso_noff_ol3ol4csum, 0, 1, 1, 0, 1, 0, 6, \ TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(tso_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 6, \ +T(tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 6, \ TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff_vlan, 1, 1, 1, 0, 0, 6, \ +T(tso_noff_vlan, 0, 1, 1, 1, 0, 0, 6, \ TSO_F | NOFF_F | VLAN_F) \ -T(tso_noff_vlan_l3l4csum, 1, 1, 1, 0, 1, 6, \ +T(tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 0, 1, 6, \ TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 0, 6, \ +T(tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 0, 6, \ TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 6, \ - TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) +T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 6, \ + TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts, 1, 0, 0, 0, 0, 0, 8, \ + TSP_F) \ +T(ts_l3l4csum, 1, 0, 0, 0, 0, 1, 8, \ + TSP_F | L3L4CSUM_F) \ +T(ts_ol3ol4csum, 1, 0, 0, 0, 1, 0, 8, \ + TSP_F | OL3OL4CSUM_F) \ +T(ts_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 8, \ + TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_vlan, 1, 0, 0, 1, 0, 0, 8, \ + TSP_F | VLAN_F) \ +T(ts_vlan_l3l4csum, 1, 0, 0, 1, 0, 1, 8, \ + TSP_F | VLAN_F | L3L4CSUM_F) \ +T(ts_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 0, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff, 1, 0, 1, 0, 0, 0, 8, \ + TSP_F | NOFF_F) \ +T(ts_noff_l3l4csum, 1, 0, 1, 0, 0, 1, 8, \ + TSP_F | NOFF_F | L3L4CSUM_F) \ +T(ts_noff_ol3ol4csum, 1, 0, 1, 0, 1, 0, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff_vlan, 1, 0, 1, 1, 0, 0, 8, \ + TSP_F | NOFF_F | VLAN_F) \ +T(ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 0, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 0, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso, 1, 1, 0, 0, 0, 0, 8, \ + TSP_F | TSO_F) \ +T(ts_tso_l3l4csum, 1, 1, 0, 0, 0, 1, 8, \ + TSP_F | TSO_F | L3L4CSUM_F) \ +T(ts_tso_ol3ol4csum, 1, 1, 0, 0, 1, 0, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F) \ +T(ts_tso_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_vlan, 1, 1, 0, 1, 0, 0, 8, \ + TSP_F | TSO_F | VLAN_F) \ +T(ts_tso_vlan_l3l4csum, 1, 1, 0, 1, 0, 1, 8, \ + TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 0, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff, 1, 1, 1, 0, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F) \ +T(ts_tso_noff_l3l4csum, 1, 1, 1, 0, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum, 1, 1, 1, 0, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan, 1, 1, 1, 1, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F) \ +T(ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn10k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn10k_tx_mseg.c b/drivers/net/cnxk/cn10k_tx_mseg.c index 6ae6907..33f6754 100644 --- a/drivers/net/cnxk/cn10k_tx_mseg.c +++ b/drivers/net/cnxk/cn10k_tx_mseg.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_xmit_pkts_mseg_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn10k_tx_vec.c b/drivers/net/cnxk/cn10k_tx_vec.c index 42baeb5..7453f3b 100644 --- a/drivers/net/cnxk/cn10k_tx_vec.c +++ b/drivers/net/cnxk/cn10k_tx_vec.c @@ -5,7 +5,7 @@ #include "cn10k_ethdev.h" #include "cn10k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn10k_nix_xmit_pkts_vec_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ @@ -15,6 +15,7 @@ \ /* VLAN, TSTMP, TSO is not supported by vec */ \ if ((flags) & NIX_TX_OFFLOAD_VLAN_QINQ_F || \ + (flags) & NIX_TX_OFFLOAD_TSTAMP_F || \ (flags) & NIX_TX_OFFLOAD_TSO_F) \ return 0; \ return cn10k_nix_xmit_pkts_vector(tx_queue, tx_pkts, pkts, cmd,\ diff --git a/drivers/net/cnxk/cn9k_ethdev.c b/drivers/net/cnxk/cn9k_ethdev.c index fc3054d..924b747 100644 --- a/drivers/net/cnxk/cn9k_ethdev.c +++ b/drivers/net/cnxk/cn9k_ethdev.c @@ -30,6 +30,9 @@ nix_rx_offload_flags(struct rte_eth_dev *eth_dev) if (dev->rx_offloads & DEV_RX_OFFLOAD_SCATTER) flags |= NIX_RX_MULTI_SEG_F; + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) + flags |= NIX_RX_OFFLOAD_TSTAMP_F; + if (!dev->ptype_disable) flags |= NIX_RX_OFFLOAD_PTYPE_F; @@ -95,6 +98,9 @@ nix_tx_offload_flags(struct rte_eth_dev *eth_dev) flags |= (NIX_TX_OFFLOAD_TSO_F | NIX_TX_OFFLOAD_OL3_OL4_CSUM_F | NIX_TX_OFFLOAD_L3_L4_CSUM_F); + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP)) + flags |= NIX_TX_OFFLOAD_TSTAMP_F; + return flags; } @@ -121,10 +127,9 @@ nix_form_default_desc(struct cnxk_eth_dev *dev, struct cn9k_eth_txq *txq, { struct nix_send_ext_s *send_hdr_ext; struct nix_send_hdr_s *send_hdr; + struct nix_send_mem_s *send_mem; union nix_send_sg_s *sg; - RTE_SET_USED(dev); - /* Initialize the fields based on basic single segment packet */ memset(&txq->cmd, 0, sizeof(txq->cmd)); @@ -135,6 +140,23 @@ nix_form_default_desc(struct cnxk_eth_dev *dev, struct cn9k_eth_txq *txq, send_hdr_ext = (struct nix_send_ext_s *)&txq->cmd[2]; send_hdr_ext->w0.subdc = NIX_SUBDC_EXT; + if (dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F) { + /* Default: one seg packet would have: + * 2(HDR) + 2(EXT) + 1(SG) + 1(IOVA) + 2(MEM) + * => 8/2 - 1 = 3 + */ + send_hdr->w0.sizem1 = 3; + send_hdr_ext->w0.tstmp = 1; + + /* To calculate the offset for send_mem, + * send_hdr->w0.sizem1 * 2 + */ + send_mem = (struct nix_send_mem_s *) + (txq->cmd + (send_hdr->w0.sizem1 << 1)); + send_mem->w0.cn9k.subdc = NIX_SUBDC_MEM; + send_mem->w0.cn9k.alg = NIX_SENDMEMALG_SETTSTMP; + send_mem->addr = dev->tstamp.tx_tstamp_iova; + } sg = (union nix_send_sg_s *)&txq->cmd[4]; } else { send_hdr = (struct nix_send_hdr_s *)&txq->cmd[0]; @@ -219,6 +241,7 @@ cn9k_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, rxq->wdata = cq->wdata; rxq->head = cq->head; rxq->qmask = cq->qmask; + rxq->tstamp = &dev->tstamp; /* Data offset from data to start of mbuf is first_skip */ rxq->data_off = rq->first_skip; @@ -351,6 +374,7 @@ static int cn9k_nix_dev_start(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct roc_nix *nix = &dev->nix; int rc; /* Common eth dev start */ @@ -358,6 +382,12 @@ cn9k_nix_dev_start(struct rte_eth_dev *eth_dev) if (rc) return rc; + /* Update VF about data off shifted by 8 bytes if PTP already + * enabled in PF owning this VF + */ + if (dev->ptp_en && (!roc_nix_is_pf(nix) && (!roc_nix_is_sdp(nix)))) + nix_ptp_enable_vf(eth_dev); + /* Setting up the rx[tx]_offload_flags due to change * in rx[tx]_offloads. */ diff --git a/drivers/net/cnxk/cn9k_ethdev.h b/drivers/net/cnxk/cn9k_ethdev.h index b92f3fc..f481b3a 100644 --- a/drivers/net/cnxk/cn9k_ethdev.h +++ b/drivers/net/cnxk/cn9k_ethdev.h @@ -29,6 +29,7 @@ struct cn9k_eth_rxq { uint32_t qmask; uint32_t available; uint16_t rq; + struct cnxk_timesync_info *tstamp; } __plt_cache_aligned; /* Rx and Tx routines */ diff --git a/drivers/net/cnxk/cn9k_rx.c b/drivers/net/cnxk/cn9k_rx.c index 01eb21f..a15428d 100644 --- a/drivers/net/cnxk/cn9k_rx.c +++ b/drivers/net/cnxk/cn9k_rx.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ @@ -17,12 +17,13 @@ NIX_RX_FASTPATH_MODES static inline void pick_rx_func(struct rte_eth_dev *eth_dev, - const eth_rx_burst_t rx_burst[2][2][2][2]) + const eth_rx_burst_t rx_burst[2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - /* [MARK] [CKSUM] [PTYPE] [RSS] */ + /* [TSP] [MARK] [CKSUM] [PTYPE] [RSS] */ eth_dev->rx_pkt_burst = rx_burst + [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_TSTAMP_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_MARK_UPDATE_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_CHECKSUM_F)] [!!(dev->rx_offload_flags & NIX_RX_OFFLOAD_PTYPE_F)] @@ -34,31 +35,34 @@ cn9k_eth_set_rx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn9k_nix_recv_pkts_##name, + const eth_rx_burst_t nix_eth_rx_burst[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn9k_nix_recv_pkts_mseg_##name, + const eth_rx_burst_t nix_eth_rx_burst_mseg[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_mseg_##name, NIX_RX_FASTPATH_MODES #undef R }; - const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2] = { -#define R(name, f3, f2, f1, f0, flags) \ - [f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_##name, + const eth_rx_burst_t nix_eth_rx_vec_burst[2][2][2][2][2] = { +#define R(name, f4, f3, f2, f1, f0, flags) \ + [f4][f3][f2][f1][f0] = cn9k_nix_recv_pkts_vec_##name, NIX_RX_FASTPATH_MODES #undef R }; - if (dev->scalar_ena) + /* For PTP enabled, scalar rx function should be chosen as most of the + * PTP apps are implemented to rx burst 1 pkt. + */ + if (dev->scalar_ena || dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP) pick_rx_func(eth_dev, nix_eth_rx_burst); else pick_rx_func(eth_dev, nix_eth_rx_vec_burst); @@ -69,6 +73,6 @@ cn9k_eth_set_rx_function(struct rte_eth_dev *eth_dev) /* Copy multi seg version with no offload for tear down sequence */ if (rte_eal_process_type() == RTE_PROC_PRIMARY) dev->rx_pkt_burst_no_offload = - nix_eth_rx_burst_mseg[0][0][0][0]; + nix_eth_rx_burst_mseg[0][0][0][0][0]; rte_mb(); } diff --git a/drivers/net/cnxk/cn9k_rx.h b/drivers/net/cnxk/cn9k_rx.h index f4b3282..c5ad5db 100644 --- a/drivers/net/cnxk/cn9k_rx.h +++ b/drivers/net/cnxk/cn9k_rx.h @@ -8,6 +8,8 @@ #include #include +#include + #define NIX_RX_OFFLOAD_NONE (0) #define NIX_RX_OFFLOAD_RSS_F BIT(0) #define NIX_RX_OFFLOAD_PTYPE_F BIT(1) @@ -253,6 +255,10 @@ cn9k_nix_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts, cn9k_nix_cqe_to_mbuf(cq, cq->tag, mbuf, lookup_mem, mbuf_init, flags); + cnxk_nix_mbuf_to_tstamp(mbuf, rxq->tstamp, + (flags & NIX_RX_OFFLOAD_TSTAMP_F), + (uint64_t *)((uint8_t *)mbuf + data_off) + ); rx_pkts[packets++] = mbuf; roc_prefetch_store_keep(mbuf); head++; @@ -489,27 +495,44 @@ cn9k_nix_recv_pkts_vector(void *rx_queue, struct rte_mbuf **rx_pkts, #define PTYPE_F NIX_RX_OFFLOAD_PTYPE_F #define CKSUM_F NIX_RX_OFFLOAD_CHECKSUM_F #define MARK_F NIX_RX_OFFLOAD_MARK_UPDATE_F +#define TS_F NIX_RX_OFFLOAD_TSTAMP_F -/* [MARK] [CKSUM] [PTYPE] [RSS] */ -#define NIX_RX_FASTPATH_MODES \ -R(no_offload, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ -R(rss, 0, 0, 0, 1, RSS_F) \ -R(ptype, 0, 0, 1, 0, PTYPE_F) \ -R(ptype_rss, 0, 0, 1, 1, PTYPE_F | RSS_F) \ -R(cksum, 0, 1, 0, 0, CKSUM_F) \ -R(cksum_rss, 0, 1, 0, 1, CKSUM_F | RSS_F) \ -R(cksum_ptype, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ -R(cksum_ptype_rss, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ -R(mark, 1, 0, 0, 0, MARK_F) \ -R(mark_rss, 1, 0, 0, 1, MARK_F | RSS_F) \ -R(mark_ptype, 1, 0, 1, 0, MARK_F | PTYPE_F) \ -R(mark_ptype_rss, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ -R(mark_cksum, 1, 1, 0, 0, MARK_F | CKSUM_F) \ -R(mark_cksum_rss, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ -R(mark_cksum_ptype, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F)\ -R(mark_cksum_ptype_rss, 1, 1, 1, 1, MARK_F | CKSUM_F | PTYPE_F | RSS_F) +/* [TS] [MARK] [CKSUM] [PTYPE] [RSS] */ +#define NIX_RX_FASTPATH_MODES \ +R(no_offload, 0, 0, 0, 0, 0, NIX_RX_OFFLOAD_NONE) \ +R(rss, 0, 0, 0, 0, 1, RSS_F) \ +R(ptype, 0, 0, 0, 1, 0, PTYPE_F) \ +R(ptype_rss, 0, 0, 0, 1, 1, PTYPE_F | RSS_F) \ +R(cksum, 0, 0, 1, 0, 0, CKSUM_F) \ +R(cksum_rss, 0, 0, 1, 0, 1, CKSUM_F | RSS_F) \ +R(cksum_ptype, 0, 0, 1, 1, 0, CKSUM_F | PTYPE_F) \ +R(cksum_ptype_rss, 0, 0, 1, 1, 1, CKSUM_F | PTYPE_F | RSS_F) \ +R(mark, 0, 1, 0, 0, 0, MARK_F) \ +R(mark_rss, 0, 1, 0, 0, 1, MARK_F | RSS_F) \ +R(mark_ptype, 0, 1, 0, 1, 0, MARK_F | PTYPE_F) \ +R(mark_ptype_rss, 0, 1, 0, 1, 1, MARK_F | PTYPE_F | RSS_F) \ +R(mark_cksum, 0, 1, 1, 0, 0, MARK_F | CKSUM_F) \ +R(mark_cksum_rss, 0, 1, 1, 0, 1, MARK_F | CKSUM_F | RSS_F) \ +R(mark_cksum_ptype, 0, 1, 1, 1, 0, MARK_F | CKSUM_F | PTYPE_F) \ +R(mark_cksum_ptype_rss, 0, 1, 1, 1, 1, MARK_F | CKSUM_F | PTYPE_F | RSS_F)\ +R(ts, 1, 0, 0, 0, 0, TS_F) \ +R(ts_rss, 1, 0, 0, 0, 1, TS_F | RSS_F) \ +R(ts_ptype, 1, 0, 0, 1, 0, TS_F | PTYPE_F) \ +R(ts_ptype_rss, 1, 0, 0, 1, 1, TS_F | PTYPE_F | RSS_F) \ +R(ts_cksum, 1, 0, 1, 0, 0, TS_F | CKSUM_F) \ +R(ts_cksum_rss, 1, 0, 1, 0, 1, TS_F | CKSUM_F | RSS_F) \ +R(ts_cksum_ptype, 1, 0, 1, 1, 0, TS_F | CKSUM_F | PTYPE_F) \ +R(ts_cksum_ptype_rss, 1, 0, 1, 1, 1, TS_F | CKSUM_F | PTYPE_F | RSS_F)\ +R(ts_mark, 1, 1, 0, 0, 0, TS_F | MARK_F) \ +R(ts_mark_rss, 1, 1, 0, 0, 1, TS_F | MARK_F | RSS_F) \ +R(ts_mark_ptype, 1, 1, 0, 1, 0, TS_F | MARK_F | PTYPE_F) \ +R(ts_mark_ptype_rss, 1, 1, 0, 1, 1, TS_F | MARK_F | PTYPE_F | RSS_F)\ +R(ts_mark_cksum, 1, 1, 1, 0, 0, TS_F | MARK_F | CKSUM_F) \ +R(ts_mark_cksum_rss, 1, 1, 1, 0, 1, TS_F | MARK_F | CKSUM_F | RSS_F)\ +R(ts_mark_cksum_ptype, 1, 1, 1, 1, 0, TS_F | MARK_F | CKSUM_F | PTYPE_F)\ +R(ts_mark_cksum_ptype_rss, 1, 1, 1, 1, 1, TS_F | MARK_F | CKSUM_F | PTYPE_F | RSS_F) -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn9k_rx_mseg.c b/drivers/net/cnxk/cn9k_rx_mseg.c index 6ad8c1d..3b26962 100644 --- a/drivers/net/cnxk/cn9k_rx_mseg.c +++ b/drivers/net/cnxk/cn9k_rx_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_mseg_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ diff --git a/drivers/net/cnxk/cn9k_rx_vec.c b/drivers/net/cnxk/cn9k_rx_vec.c index 997177f..b19c7f3 100644 --- a/drivers/net/cnxk/cn9k_rx_vec.c +++ b/drivers/net/cnxk/cn9k_rx_vec.c @@ -5,10 +5,13 @@ #include "cn9k_ethdev.h" #include "cn9k_rx.h" -#define R(name, f3, f2, f1, f0, flags) \ +#define R(name, f4, f3, f2, f1, f0, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_recv_pkts_vec_##name( \ void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t pkts) \ { \ + /* TSTMP is not supported by vector */ \ + if ((flags) & NIX_RX_OFFLOAD_TSTAMP_F) \ + return 0; \ return cn9k_nix_recv_pkts_vector(rx_queue, rx_pkts, pkts, \ (flags)); \ } diff --git a/drivers/net/cnxk/cn9k_tx.c b/drivers/net/cnxk/cn9k_tx.c index 2ff9720..b802606 100644 --- a/drivers/net/cnxk/cn9k_tx.c +++ b/drivers/net/cnxk/cn9k_tx.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts) \ { \ @@ -23,12 +23,13 @@ NIX_TX_FASTPATH_MODES static inline void pick_tx_func(struct rte_eth_dev *eth_dev, - const eth_tx_burst_t tx_burst[2][2][2][2][2]) + const eth_tx_burst_t tx_burst[2][2][2][2][2][2]) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - /* [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ + /* [TS] [TSO] [NOFF] [VLAN] [OL3_OL4_CSUM] [IL3_IL4_CSUM] */ eth_dev->tx_pkt_burst = tx_burst + [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSTAMP_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_TSO_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_MBUF_NOFF_F)] [!!(dev->tx_offload_flags & NIX_TX_OFFLOAD_VLAN_QINQ_F)] @@ -41,25 +42,25 @@ cn9k_eth_set_tx_function(struct rte_eth_dev *eth_dev) { struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); - const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_##name, + const eth_tx_burst_t nix_eth_tx_burst[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_mseg_##name, + const eth_tx_burst_t nix_eth_tx_burst_mseg[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_mseg_##name, NIX_TX_FASTPATH_MODES #undef T }; - const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2] = { -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ - [f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_##name, + const eth_tx_burst_t nix_eth_tx_vec_burst[2][2][2][2][2][2] = { +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ + [f5][f4][f3][f2][f1][f0] = cn9k_nix_xmit_pkts_vec_##name, NIX_TX_FASTPATH_MODES #undef T @@ -67,7 +68,8 @@ cn9k_eth_set_tx_function(struct rte_eth_dev *eth_dev) if (dev->scalar_ena || (dev->tx_offload_flags & - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F))) + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F | + NIX_TX_OFFLOAD_TSO_F))) pick_tx_func(eth_dev, nix_eth_tx_burst); else pick_tx_func(eth_dev, nix_eth_tx_vec_burst); diff --git a/drivers/net/cnxk/cn9k_tx.h b/drivers/net/cnxk/cn9k_tx.h index 7b0d536..1899d66 100644 --- a/drivers/net/cnxk/cn9k_tx.h +++ b/drivers/net/cnxk/cn9k_tx.h @@ -12,6 +12,7 @@ #define NIX_TX_OFFLOAD_VLAN_QINQ_F BIT(2) #define NIX_TX_OFFLOAD_MBUF_NOFF_F BIT(3) #define NIX_TX_OFFLOAD_TSO_F BIT(4) +#define NIX_TX_OFFLOAD_TSTAMP_F BIT(5) /* Flags to control xmit_prepare function. * Defining it from backwards to denote its been @@ -24,7 +25,8 @@ NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F) #define NIX_TX_NEED_EXT_HDR \ - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F) + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSTAMP_F | \ + NIX_TX_OFFLOAD_TSO_F) #define NIX_XMIT_FC_OR_RETURN(txq, pkts) \ do { \ @@ -46,8 +48,12 @@ static __rte_always_inline int cn9k_nix_tx_ext_subs(const uint16_t flags) { - return (flags & - (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) ? 1 : 0; + return (flags & NIX_TX_OFFLOAD_TSTAMP_F) + ? 2 + : ((flags & + (NIX_TX_OFFLOAD_VLAN_QINQ_F | NIX_TX_OFFLOAD_TSO_F)) + ? 1 + : 0); } static __rte_always_inline void @@ -283,6 +289,41 @@ cn9k_nix_xmit_prepare(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags, } static __rte_always_inline void +cn9k_nix_xmit_prepare_tstamp(uint64_t *cmd, const uint64_t *send_mem_desc, + const uint64_t ol_flags, const uint16_t no_segdw, + const uint16_t flags) +{ + if (flags & NIX_TX_OFFLOAD_TSTAMP_F) { + struct nix_send_mem_s *send_mem; + uint16_t off = (no_segdw - 1) << 1; + const uint8_t is_ol_tstamp = !(ol_flags & PKT_TX_IEEE1588_TMST); + + send_mem = (struct nix_send_mem_s *)(cmd + off); + if (flags & NIX_TX_MULTI_SEG_F) { + /* Retrieving the default desc values */ + cmd[off] = send_mem_desc[6]; + + /* Using compiler barier to avoid voilation of C + * aliasing rules. + */ + rte_compiler_barrier(); + } + + /* Packets for which PKT_TX_IEEE1588_TMST is not set, tx tstamp + * should not be recorded, hence changing the alg type to + * NIX_SENDMEMALG_SET and also changing send mem addr field to + * next 8 bytes as it corrpt the actual tx tstamp registered + * address. + */ + send_mem->w0.cn9k.alg = + NIX_SENDMEMALG_SETTSTMP - (is_ol_tstamp); + + send_mem->addr = (rte_iova_t)((uint64_t *)send_mem_desc[7] + + (is_ol_tstamp)); + } +} + +static __rte_always_inline void cn9k_nix_xmit_one(uint64_t *cmd, void *lmt_addr, const rte_iova_t io_addr, const uint32_t flags) { @@ -380,7 +421,7 @@ cn9k_nix_prepare_mseg(struct rte_mbuf *m, uint64_t *cmd, const uint16_t flags) /* Roundup extra dwords to multiple of 2 */ segdw = (segdw >> 1) + (segdw & 0x1); /* Default dwords */ - segdw += (off >> 1) + 1; + segdw += (off >> 1) + 1 + !!(flags & NIX_TX_OFFLOAD_TSTAMP_F); send_hdr->w0.sizem1 = segdw - 1; return segdw; @@ -447,6 +488,8 @@ cn9k_nix_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts, for (i = 0; i < pkts; i++) { cn9k_nix_xmit_prepare(tx_pkts[i], cmd, flags, lso_tun_fmt); + cn9k_nix_xmit_prepare_tstamp(cmd, &txq->cmd[0], + tx_pkts[i]->ol_flags, 4, flags); cn9k_nix_xmit_one(cmd, lmt_addr, io_addr, flags); } @@ -488,6 +531,9 @@ cn9k_nix_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, for (i = 0; i < pkts; i++) { cn9k_nix_xmit_prepare(tx_pkts[i], cmd, flags, lso_tun_fmt); segdw = cn9k_nix_prepare_mseg(tx_pkts[i], cmd, flags); + cn9k_nix_xmit_prepare_tstamp(cmd, &txq->cmd[0], + tx_pkts[i]->ol_flags, segdw, + flags); cn9k_nix_xmit_mseg_one(cmd, lmt_addr, io_addr, segdw); } @@ -1241,75 +1287,140 @@ cn9k_nix_xmit_pkts_vector(void *tx_queue, struct rte_mbuf **tx_pkts, #define VLAN_F NIX_TX_OFFLOAD_VLAN_QINQ_F #define NOFF_F NIX_TX_OFFLOAD_MBUF_NOFF_F #define TSO_F NIX_TX_OFFLOAD_TSO_F +#define TSP_F NIX_TX_OFFLOAD_TSTAMP_F -/* [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ -#define NIX_TX_FASTPATH_MODES \ -T(no_offload, 0, 0, 0, 0, 0, 4, \ - NIX_TX_OFFLOAD_NONE) \ -T(l3l4csum, 0, 0, 0, 0, 1, 4, \ - L3L4CSUM_F) \ -T(ol3ol4csum, 0, 0, 0, 1, 0, 4, \ - OL3OL4CSUM_F) \ -T(ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 4, \ - OL3OL4CSUM_F | L3L4CSUM_F) \ -T(vlan, 0, 0, 1, 0, 0, 6, \ - VLAN_F) \ -T(vlan_l3l4csum, 0, 0, 1, 0, 1, 6, \ - VLAN_F | L3L4CSUM_F) \ -T(vlan_ol3ol4csum, 0, 0, 1, 1, 0, 6, \ - VLAN_F | OL3OL4CSUM_F) \ -T(vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 6, \ - VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff, 0, 1, 0, 0, 0, 4, \ - NOFF_F) \ -T(noff_l3l4csum, 0, 1, 0, 0, 1, 4, \ - NOFF_F | L3L4CSUM_F) \ -T(noff_ol3ol4csum, 0, 1, 0, 1, 0, 4, \ - NOFF_F | OL3OL4CSUM_F) \ -T(noff_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 4, \ - NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(noff_vlan, 0, 1, 1, 0, 0, 6, \ - NOFF_F | VLAN_F) \ -T(noff_vlan_l3l4csum, 0, 1, 1, 0, 1, 6, \ - NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(noff_vlan_ol3ol4csum, 0, 1, 1, 1, 0, 6, \ - NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 6, \ - NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso, 1, 0, 0, 0, 0, 6, \ - TSO_F) \ -T(tso_l3l4csum, 1, 0, 0, 0, 1, 6, \ - TSO_F | L3L4CSUM_F) \ -T(tso_ol3ol4csum, 1, 0, 0, 1, 0, 6, \ - TSO_F | OL3OL4CSUM_F) \ -T(tso_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 6, \ - TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_vlan, 1, 0, 1, 0, 0, 6, \ - TSO_F | VLAN_F) \ -T(tso_vlan_l3l4csum, 1, 0, 1, 0, 1, 6, \ - TSO_F | VLAN_F | L3L4CSUM_F) \ -T(tso_vlan_ol3ol4csum, 1, 0, 1, 1, 0, 6, \ - TSO_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 6, \ - TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff, 1, 1, 0, 0, 0, 6, \ - TSO_F | NOFF_F) \ -T(tso_noff_l3l4csum, 1, 1, 0, 0, 1, 6, \ - TSO_F | NOFF_F | L3L4CSUM_F) \ -T(tso_noff_ol3ol4csum, 1, 1, 0, 1, 0, 6, \ - TSO_F | NOFF_F | OL3OL4CSUM_F) \ -T(tso_noff_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 6, \ - TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ -T(tso_noff_vlan, 1, 1, 1, 0, 0, 6, \ - TSO_F | NOFF_F | VLAN_F) \ -T(tso_noff_vlan_l3l4csum, 1, 1, 1, 0, 1, 6, \ - TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 0, 6, \ - TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ -T(tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 6, \ - TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) +/* [TSP] [TSO] [NOFF] [VLAN] [OL3OL4CSUM] [L3L4CSUM] */ +#define NIX_TX_FASTPATH_MODES \ +T(no_offload, 0, 0, 0, 0, 0, 0, 4, \ + NIX_TX_OFFLOAD_NONE) \ +T(l3l4csum, 0, 0, 0, 0, 0, 1, 4, \ + L3L4CSUM_F) \ +T(ol3ol4csum, 0, 0, 0, 0, 1, 0, 4, \ + OL3OL4CSUM_F) \ +T(ol3ol4csum_l3l4csum, 0, 0, 0, 0, 1, 1, 4, \ + OL3OL4CSUM_F | L3L4CSUM_F) \ +T(vlan, 0, 0, 0, 1, 0, 0, 6, \ + VLAN_F) \ +T(vlan_l3l4csum, 0, 0, 0, 1, 0, 1, 6, \ + VLAN_F | L3L4CSUM_F) \ +T(vlan_ol3ol4csum, 0, 0, 0, 1, 1, 0, 6, \ + VLAN_F | OL3OL4CSUM_F) \ +T(vlan_ol3ol4csum_l3l4csum, 0, 0, 0, 1, 1, 1, 6, \ + VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(noff, 0, 0, 1, 0, 0, 0, 4, \ + NOFF_F) \ +T(noff_l3l4csum, 0, 0, 1, 0, 0, 1, 4, \ + NOFF_F | L3L4CSUM_F) \ +T(noff_ol3ol4csum, 0, 0, 1, 0, 1, 0, 4, \ + NOFF_F | OL3OL4CSUM_F) \ +T(noff_ol3ol4csum_l3l4csum, 0, 0, 1, 0, 1, 1, 4, \ + NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(noff_vlan, 0, 0, 1, 1, 0, 0, 6, \ + NOFF_F | VLAN_F) \ +T(noff_vlan_l3l4csum, 0, 0, 1, 1, 0, 1, 6, \ + NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(noff_vlan_ol3ol4csum, 0, 0, 1, 1, 1, 0, 6, \ + NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(noff_vlan_ol3ol4csum_l3l4csum, 0, 0, 1, 1, 1, 1, 6, \ + NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso, 0, 1, 0, 0, 0, 0, 6, \ + TSO_F) \ +T(tso_l3l4csum, 0, 1, 0, 0, 0, 1, 6, \ + TSO_F | L3L4CSUM_F) \ +T(tso_ol3ol4csum, 0, 1, 0, 0, 1, 0, 6, \ + TSO_F | OL3OL4CSUM_F) \ +T(tso_ol3ol4csum_l3l4csum, 0, 1, 0, 0, 1, 1, 6, \ + TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_vlan, 0, 1, 0, 1, 0, 0, 6, \ + TSO_F | VLAN_F) \ +T(tso_vlan_l3l4csum, 0, 1, 0, 1, 0, 1, 6, \ + TSO_F | VLAN_F | L3L4CSUM_F) \ +T(tso_vlan_ol3ol4csum, 0, 1, 0, 1, 1, 0, 6, \ + TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(tso_vlan_ol3ol4csum_l3l4csum, 0, 1, 0, 1, 1, 1, 6, \ + TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_noff, 0, 1, 1, 0, 0, 0, 6, \ + TSO_F | NOFF_F) \ +T(tso_noff_l3l4csum, 0, 1, 1, 0, 0, 1, 6, \ + TSO_F | NOFF_F | L3L4CSUM_F) \ +T(tso_noff_ol3ol4csum, 0, 1, 1, 0, 1, 0, 6, \ + TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(tso_noff_ol3ol4csum_l3l4csum, 0, 1, 1, 0, 1, 1, 6, \ + TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(tso_noff_vlan, 0, 1, 1, 1, 0, 0, 6, \ + TSO_F | NOFF_F | VLAN_F) \ +T(tso_noff_vlan_l3l4csum, 0, 1, 1, 1, 0, 1, 6, \ + TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(tso_noff_vlan_ol3ol4csum, 0, 1, 1, 1, 1, 0, 6, \ + TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(tso_noff_vlan_ol3ol4csum_l3l4csum, 0, 1, 1, 1, 1, 1, 6, \ + TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts, 1, 0, 0, 0, 0, 0, 8, \ + TSP_F) \ +T(ts_l3l4csum, 1, 0, 0, 0, 0, 1, 8, \ + TSP_F | L3L4CSUM_F) \ +T(ts_ol3ol4csum, 1, 0, 0, 0, 1, 0, 8, \ + TSP_F | OL3OL4CSUM_F) \ +T(ts_ol3ol4csum_l3l4csum, 1, 0, 0, 0, 1, 1, 8, \ + TSP_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_vlan, 1, 0, 0, 1, 0, 0, 8, \ + TSP_F | VLAN_F) \ +T(ts_vlan_l3l4csum, 1, 0, 0, 1, 0, 1, 8, \ + TSP_F | VLAN_F | L3L4CSUM_F) \ +T(ts_vlan_ol3ol4csum, 1, 0, 0, 1, 1, 0, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_vlan_ol3ol4csum_l3l4csum, 1, 0, 0, 1, 1, 1, 8, \ + TSP_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff, 1, 0, 1, 0, 0, 0, 8, \ + TSP_F | NOFF_F) \ +T(ts_noff_l3l4csum, 1, 0, 1, 0, 0, 1, 8, \ + TSP_F | NOFF_F | L3L4CSUM_F) \ +T(ts_noff_ol3ol4csum, 1, 0, 1, 0, 1, 0, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_noff_ol3ol4csum_l3l4csum, 1, 0, 1, 0, 1, 1, 8, \ + TSP_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_noff_vlan, 1, 0, 1, 1, 0, 0, 8, \ + TSP_F | NOFF_F | VLAN_F) \ +T(ts_noff_vlan_l3l4csum, 1, 0, 1, 1, 0, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum, 1, 0, 1, 1, 1, 0, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_noff_vlan_ol3ol4csum_l3l4csum, 1, 0, 1, 1, 1, 1, 8, \ + TSP_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso, 1, 1, 0, 0, 0, 0, 8, \ + TSP_F | TSO_F) \ +T(ts_tso_l3l4csum, 1, 1, 0, 0, 0, 1, 8, \ + TSP_F | TSO_F | L3L4CSUM_F) \ +T(ts_tso_ol3ol4csum, 1, 1, 0, 0, 1, 0, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F) \ +T(ts_tso_ol3ol4csum_l3l4csum, 1, 1, 0, 0, 1, 1, 8, \ + TSP_F | TSO_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_vlan, 1, 1, 0, 1, 0, 0, 8, \ + TSP_F | TSO_F | VLAN_F) \ +T(ts_tso_vlan_l3l4csum, 1, 1, 0, 1, 0, 1, 8, \ + TSP_F | TSO_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum, 1, 1, 0, 1, 1, 0, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_vlan_ol3ol4csum_l3l4csum, 1, 1, 0, 1, 1, 1, 8, \ + TSP_F | TSO_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff, 1, 1, 1, 0, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F) \ +T(ts_tso_noff_l3l4csum, 1, 1, 1, 0, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | L3L4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum, 1, 1, 1, 0, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_ol3ol4csum_l3l4csum, 1, 1, 1, 0, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | OL3OL4CSUM_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan, 1, 1, 1, 1, 0, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F) \ +T(ts_tso_noff_vlan_l3l4csum, 1, 1, 1, 1, 0, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | L3L4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum, 1, 1, 1, 1, 1, 0, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F) \ +T(ts_tso_noff_vlan_ol3ol4csum_l3l4csum, 1, 1, 1, 1, 1, 1, 8, \ + TSP_F | TSO_F | NOFF_F | VLAN_F | OL3OL4CSUM_F | L3L4CSUM_F) -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot cn9k_nix_xmit_pkts_##name( \ void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t pkts); \ \ diff --git a/drivers/net/cnxk/cn9k_tx_mseg.c b/drivers/net/cnxk/cn9k_tx_mseg.c index 65c5f36..f3c427c 100644 --- a/drivers/net/cnxk/cn9k_tx_mseg.c +++ b/drivers/net/cnxk/cn9k_tx_mseg.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn9k_nix_xmit_pkts_mseg_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ diff --git a/drivers/net/cnxk/cn9k_tx_vec.c b/drivers/net/cnxk/cn9k_tx_vec.c index 21ffc2c..a6e7c9e 100644 --- a/drivers/net/cnxk/cn9k_tx_vec.c +++ b/drivers/net/cnxk/cn9k_tx_vec.c @@ -5,7 +5,7 @@ #include "cn9k_ethdev.h" #include "cn9k_tx.h" -#define T(name, f4, f3, f2, f1, f0, sz, flags) \ +#define T(name, f5, f4, f3, f2, f1, f0, sz, flags) \ uint16_t __rte_noinline __rte_hot \ cn9k_nix_xmit_pkts_vec_##name(void *tx_queue, \ struct rte_mbuf **tx_pkts, \ @@ -15,6 +15,7 @@ \ /* VLAN, TSTMP, TSO is not supported by vec */ \ if ((flags) & NIX_TX_OFFLOAD_VLAN_QINQ_F || \ + (flags) & NIX_TX_OFFLOAD_TSTAMP_F || \ (flags) & NIX_TX_OFFLOAD_TSO_F) \ return 0; \ return cn9k_nix_xmit_pkts_vector(tx_queue, tx_pkts, pkts, cmd, \ diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c index 094f1cb..71d567a 100644 --- a/drivers/net/cnxk/cnxk_ethdev.c +++ b/drivers/net/cnxk/cnxk_ethdev.c @@ -150,7 +150,8 @@ cnxk_nix_rxq_mbuf_setup(struct cnxk_eth_dev *dev) offsetof(struct rte_mbuf, data_off) != 6); mb_def.nb_segs = 1; - mb_def.data_off = RTE_PKTMBUF_HEADROOM; + mb_def.data_off = RTE_PKTMBUF_HEADROOM + + (dev->ptp_en * CNXK_NIX_TIMESYNC_RX_OFFSET); mb_def.port = port_id; rte_mbuf_refcnt_set(&mb_def, 1); @@ -356,6 +357,18 @@ cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, eth_dev->data->rx_queues[qid] = rxq_sp + 1; eth_dev->data->rx_queue_state[qid] = RTE_ETH_QUEUE_STATE_STOPPED; + /* Calculating delta and freq mult between PTP HI clock and tsc. + * These are needed in deriving raw clock value from tsc counter. + * read_clock eth op returns raw clock value. + */ + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP) || dev->ptp_en) { + rc = cnxk_nix_tsc_convert(dev); + if (rc) { + plt_err("Failed to calculate delta and freq mult"); + goto rq_fini; + } + } + return 0; rq_fini: rc |= roc_nix_rq_fini(rq); @@ -1112,7 +1125,7 @@ cnxk_nix_dev_start(struct rte_eth_dev *eth_dev) struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); int rc, i; - if (eth_dev->data->nb_rx_queues != 0) { + if (eth_dev->data->nb_rx_queues != 0 && !dev->ptp_en) { rc = nix_recalc_mtu(eth_dev); if (rc) return rc; @@ -1157,6 +1170,25 @@ cnxk_nix_dev_start(struct rte_eth_dev *eth_dev) } } + /* Enable PTP if it is requested by the user or already + * enabled on PF owning this VF + */ + memset(&dev->tstamp, 0, sizeof(struct cnxk_timesync_info)); + if ((dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP) || dev->ptp_en) + cnxk_eth_dev_ops.timesync_enable(eth_dev); + else + cnxk_eth_dev_ops.timesync_disable(eth_dev); + + if (dev->rx_offloads & DEV_RX_OFFLOAD_TIMESTAMP) { + rc = rte_mbuf_dyn_rx_timestamp_register + (&dev->tstamp.tstamp_dynfield_offset, + &dev->tstamp.rx_tstamp_dynflag); + if (rc != 0) { + plt_err("Failed to register Rx timestamp field/flag"); + goto rx_disable; + } + } + cnxk_nix_toggle_flag_link_cfg(dev, false); return 0; diff --git a/drivers/net/cnxk/cnxk_ethdev.h b/drivers/net/cnxk/cnxk_ethdev.h index 1c41dcb..de6d533 100644 --- a/drivers/net/cnxk/cnxk_ethdev.h +++ b/drivers/net/cnxk/cnxk_ethdev.h @@ -13,6 +13,7 @@ #include #include #include +#include #include "roc_api.h" @@ -75,7 +76,7 @@ (DEV_RX_OFFLOAD_CHECKSUM | DEV_RX_OFFLOAD_SCTP_CKSUM | \ DEV_RX_OFFLOAD_OUTER_IPV4_CKSUM | DEV_RX_OFFLOAD_SCATTER | \ DEV_RX_OFFLOAD_JUMBO_FRAME | DEV_RX_OFFLOAD_OUTER_UDP_CKSUM | \ - DEV_RX_OFFLOAD_RSS_HASH) + DEV_RX_OFFLOAD_RSS_HASH | DEV_RX_OFFLOAD_TIMESTAMP) #define RSS_IPV4_ENABLE \ (ETH_RSS_IPV4 | ETH_RSS_FRAG_IPV4 | ETH_RSS_NONFRAG_IPV4_UDP | \ @@ -100,7 +101,10 @@ /* Default mark value used when none is provided. */ #define CNXK_FLOW_ACTION_FLAG_DEFAULT 0xffff +/* Default cycle counter mask */ +#define CNXK_CYCLECOUNTER_MASK 0xffffffffffffffffULL #define CNXK_NIX_TIMESYNC_RX_OFFSET 8 + #define PTYPE_NON_TUNNEL_WIDTH 16 #define PTYPE_TUNNEL_WIDTH 12 #define PTYPE_NON_TUNNEL_ARRAY_SZ BIT(PTYPE_NON_TUNNEL_WIDTH) @@ -130,6 +134,16 @@ struct cnxk_eth_qconf { uint8_t valid; }; +struct cnxk_timesync_info { + uint64_t rx_tstamp_dynflag; + rte_iova_t tx_tstamp_iova; + uint64_t *tx_tstamp; + uint64_t rx_tstamp; + int tstamp_dynfield_offset; + uint8_t tx_ready; + uint8_t rx_ready; +} __plt_cache_aligned; + struct cnxk_eth_dev { /* ROC NIX */ struct roc_nix nix; @@ -188,6 +202,14 @@ struct cnxk_eth_dev { /* Flow control configuration */ struct cnxk_fc_cfg fc_cfg; + /* PTP Counters */ + struct cnxk_timesync_info tstamp; + struct rte_timecounter systime_tc; + struct rte_timecounter rx_tstamp_tc; + struct rte_timecounter tx_tstamp_tc; + double clk_freq_mult; + uint64_t clk_delta; + /* Rx burst for cleanup(Only Primary) */ eth_rx_burst_t rx_pkt_burst_no_offload; @@ -288,6 +310,9 @@ int cnxk_nix_rx_queue_setup(struct rte_eth_dev *eth_dev, uint16_t qid, int cnxk_nix_tx_queue_start(struct rte_eth_dev *eth_dev, uint16_t qid); int cnxk_nix_tx_queue_stop(struct rte_eth_dev *eth_dev, uint16_t qid); int cnxk_nix_dev_start(struct rte_eth_dev *eth_dev); +int cnxk_nix_timesync_enable(struct rte_eth_dev *eth_dev); +int cnxk_nix_timesync_disable(struct rte_eth_dev *eth_dev); +int cnxk_nix_tsc_convert(struct cnxk_eth_dev *dev); uint64_t cnxk_nix_rxq_mbuf_setup(struct cnxk_eth_dev *dev); @@ -404,4 +429,41 @@ cnxk_nix_prefree_seg(struct rte_mbuf *m) return 1; } +static inline rte_mbuf_timestamp_t * +cnxk_nix_timestamp_dynfield(struct rte_mbuf *mbuf, + struct cnxk_timesync_info *info) +{ + return RTE_MBUF_DYNFIELD(mbuf, info->tstamp_dynfield_offset, + rte_mbuf_timestamp_t *); +} + +static __rte_always_inline void +cnxk_nix_mbuf_to_tstamp(struct rte_mbuf *mbuf, + struct cnxk_timesync_info *tstamp, bool ts_enable, + uint64_t *tstamp_ptr) +{ + if (ts_enable && + (mbuf->data_off == + RTE_PKTMBUF_HEADROOM + CNXK_NIX_TIMESYNC_RX_OFFSET)) { + mbuf->pkt_len -= CNXK_NIX_TIMESYNC_RX_OFFSET; + + /* Reading the rx timestamp inserted by CGX, viz at + * starting of the packet data. + */ + *cnxk_nix_timestamp_dynfield(mbuf, tstamp) = + rte_be_to_cpu_64(*tstamp_ptr); + /* PKT_RX_IEEE1588_TMST flag needs to be set only in case + * PTP packets are received. + */ + if (mbuf->packet_type == RTE_PTYPE_L2_ETHER_TIMESYNC) { + tstamp->rx_tstamp = + *cnxk_nix_timestamp_dynfield(mbuf, tstamp); + tstamp->rx_ready = 1; + mbuf->ol_flags |= PKT_RX_IEEE1588_PTP | + PKT_RX_IEEE1588_TMST | + tstamp->rx_tstamp_dynflag; + } + } +} + #endif /* __CNXK_ETHDEV_H__ */ diff --git a/drivers/net/cnxk/cnxk_ptp.c b/drivers/net/cnxk/cnxk_ptp.c new file mode 100644 index 0000000..fc317965 --- /dev/null +++ b/drivers/net/cnxk/cnxk_ptp.c @@ -0,0 +1,169 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2021 Marvell. + */ + +#include "cnxk_ethdev.h" + +/* This function calculates two parameters "clk_freq_mult" and + * "clk_delta" which is useful in deriving PTP HI clock from + * timestamp counter (tsc) value. + */ +int +cnxk_nix_tsc_convert(struct cnxk_eth_dev *dev) +{ + uint64_t ticks_base = 0, ticks = 0, tsc = 0, t_freq; + struct roc_nix *nix = &dev->nix; + int rc, val; + + /* Calculating the frequency at which PTP HI clock is running */ + rc = roc_nix_ptp_clock_read(nix, &ticks_base, &tsc, false); + if (rc) { + plt_err("Failed to read the raw clock value: %d", rc); + goto fail; + } + + rte_delay_ms(100); + + rc = roc_nix_ptp_clock_read(nix, &ticks, &tsc, false); + if (rc) { + plt_err("Failed to read the raw clock value: %d", rc); + goto fail; + } + + t_freq = (ticks - ticks_base) * 10; + + /* Calculating the freq multiplier viz the ratio between the + * frequency at which PTP HI clock works and tsc clock runs + */ + dev->clk_freq_mult = + (double)pow(10, floor(log10(t_freq))) / rte_get_timer_hz(); + + val = false; +#ifdef RTE_ARM_EAL_RDTSC_USE_PMU + val = true; +#endif + rc = roc_nix_ptp_clock_read(nix, &ticks, &tsc, val); + if (rc) { + plt_err("Failed to read the raw clock value: %d", rc); + goto fail; + } + + /* Calculating delta between PTP HI clock and tsc */ + dev->clk_delta = ((uint64_t)(ticks / dev->clk_freq_mult) - tsc); + +fail: + return rc; +} + +int +cnxk_nix_timesync_enable(struct rte_eth_dev *eth_dev) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + struct cnxk_timesync_info *tstamp = &dev->tstamp; + struct roc_nix *nix = &dev->nix; + const struct rte_memzone *ts; + int rc = 0; + + /* If we are VF/SDP/LBK, ptp cannot not be enabled */ + if (roc_nix_is_vf_or_sdp(nix) || roc_nix_is_lbk(nix)) { + plt_err("PTP cannot be enabled for VF/SDP/LBK"); + return -EINVAL; + } + + if (dev->ptp_en) + return rc; + + if (dev->ptype_disable) { + plt_err("Ptype offload is disabled, it should be enabled"); + return -EINVAL; + } + + if (dev->npc.switch_header_type == ROC_PRIV_FLAGS_HIGIG) { + plt_err("Both PTP and switch header cannot be enabled"); + return -EINVAL; + } + + /* Allocating a iova address for tx tstamp */ + ts = rte_eth_dma_zone_reserve(eth_dev, "cnxk_ts", 0, 128, 128, 0); + if (ts == NULL) { + plt_err("Failed to allocate mem for tx tstamp addr"); + return -ENOMEM; + } + + tstamp->tx_tstamp_iova = ts->iova; + tstamp->tx_tstamp = ts->addr; + + rc = rte_mbuf_dyn_rx_timestamp_register(&tstamp->tstamp_dynfield_offset, + &tstamp->rx_tstamp_dynflag); + if (rc) { + plt_err("Failed to register Rx timestamp field/flag"); + goto error; + } + + /* System time should be already on by default */ + memset(&dev->systime_tc, 0, sizeof(struct rte_timecounter)); + memset(&dev->rx_tstamp_tc, 0, sizeof(struct rte_timecounter)); + memset(&dev->tx_tstamp_tc, 0, sizeof(struct rte_timecounter)); + + dev->systime_tc.cc_mask = CNXK_CYCLECOUNTER_MASK; + dev->rx_tstamp_tc.cc_mask = CNXK_CYCLECOUNTER_MASK; + dev->tx_tstamp_tc.cc_mask = CNXK_CYCLECOUNTER_MASK; + + dev->rx_offloads |= DEV_RX_OFFLOAD_TIMESTAMP; + + rc = roc_nix_ptp_rx_ena_dis(nix, true); + if (!rc) { + rc = roc_nix_ptp_tx_ena_dis(nix, true); + if (rc) { + roc_nix_ptp_rx_ena_dis(nix, false); + goto error; + } + } + + rc = nix_recalc_mtu(eth_dev); + if (rc) { + plt_err("Failed to set MTU size for ptp"); + goto error; + } + + return rc; + +error: + rte_eth_dma_zone_free(eth_dev, "cnxk_ts", 0); + dev->tstamp.tx_tstamp_iova = 0; + dev->tstamp.tx_tstamp = NULL; + return rc; +} + +int +cnxk_nix_timesync_disable(struct rte_eth_dev *eth_dev) +{ + struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev); + uint64_t rx_offloads = DEV_RX_OFFLOAD_TIMESTAMP; + struct roc_nix *nix = &dev->nix; + int rc = 0; + + /* If we are VF/SDP/LBK, ptp cannot not be disabled */ + if (roc_nix_is_vf_or_sdp(nix) || roc_nix_is_lbk(nix)) + return -EINVAL; + + if (!dev->ptp_en) + return rc; + + dev->rx_offloads &= ~rx_offloads; + + rc = roc_nix_ptp_rx_ena_dis(nix, false); + if (!rc) { + rc = roc_nix_ptp_tx_ena_dis(nix, false); + if (rc) { + roc_nix_ptp_rx_ena_dis(nix, true); + return rc; + } + } + + rc = nix_recalc_mtu(eth_dev); + if (rc) + plt_err("Failed to set MTU size for ptp"); + + return rc; +} diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build index df953fd..2071d0d 100644 --- a/drivers/net/cnxk/meson.build +++ b/drivers/net/cnxk/meson.build @@ -13,6 +13,7 @@ sources = files('cnxk_ethdev.c', 'cnxk_ethdev_devargs.c', 'cnxk_link.c', 'cnxk_lookup.c', + 'cnxk_ptp.c', 'cnxk_rte_flow.c', 'cnxk_stats.c') -- 2.8.4