From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9BBACA0559; Mon, 16 Mar 2020 10:34:44 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6B5941C08C; Mon, 16 Mar 2020 10:34:34 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 345441AFF for ; Mon, 16 Mar 2020 10:34:33 +0100 (CET) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 02G9UM1i015008; Mon, 16 Mar 2020 02:34:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=pfpt0818; bh=+WvXHQyq13pwWVPtE+zveeuPtQwkNaU5zcI67YYWRqs=; b=iF2GGH4nj7tYiTLjp6yJ2Hn5xY9Sgv4eyb9snYPHB1J8DHocuMiCWLhJBUJSyxOqRz7R ZnyzIGW6fzzEudMzex6+LoQH/6j+WePcpmBE7DjVwrzpwknhmPbjoptz+SG+xsN8DKYR T+C9XX9UD6CidKbEef8hhIbfgHmdHEG9ufaexm72rrqRRs2BB+EtN477Fh7RIkJew2In cGMWU2kseM8I9gS87qdQlsxH0ZUPAKvmBMVGsCHc6d8UcjHk0mO6sBHxzNbVkmUU4+7I eUkhCprXn+0kSJzhr+TVJnfeVOd11kjcU771XHIupfiPsYIrIu3KxofzW4D/FFSaH0EO Rg== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0b-0016f401.pphosted.com with ESMTP id 2yrxsme4yv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 16 Mar 2020 02:34:32 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Mar 2020 02:34:30 -0700 Received: from SC-EXCH03.marvell.com (10.93.176.83) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Mar 2020 02:34:29 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.176) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Mon, 16 Mar 2020 02:34:29 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B8ohZsTszO9HDcc9Jwq47w6ZcHaHTpwHwP/uFaDyJUdMYIJIPn79oEJiC8yV00qUL6p8kSTu04oJflHSzwg9Y7e37tKMTBAdCwYzLUgOMfAuxk82i3cg2DBeCq8LC7sXh/F8emYGKz0aw/XkXoe4c6NOUj2rWnI2Xueb+2qjc8O7gdLQr7NasIx/T2yaqY8Bab0K7l/SRcq5KKgTm/HN87/Wj0PP5+uWJVkeD4EFHJyjkbEM9JDBXguP27At7F6ElYXFLzyk4RSy192a2xNtBzmPNBUNSP0dy+vKKQ09F8PxnuBLqfdhlT8ZuOXZEo0m0ilab+Toil1kkGsvIPm3eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WvXHQyq13pwWVPtE+zveeuPtQwkNaU5zcI67YYWRqs=; b=P1IQAhOemihgKZy5YaHzZIjH4g3fhWM1mm1Ol2xF0Me5MT91uEcloeVHpJW5YUTIY3KKvu9uehP0dI/uYSO3oKig+WqClLKkZCwMwdfaQj7ZJ5r3ZMW2ipjB3IEvP+x/vroWIguM58aaFcIBF98cK6F+4g46upV2UeR93LVGmEBLdzO9QVVShpl/V/yffWD7jPJ3GzXnIurTUALCTjLG5WXTnr8ZZ+cRoLHODmDZBhNbaBp69n3ZM4yaJJ/9tUy18oX64blO/uOQZWyJ1HbApUr8guM40+RU6o9g0wPIdLs9bpA7eGucO/gBc/+44+JvTgRBLlYAaskWthcHvKvVlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WvXHQyq13pwWVPtE+zveeuPtQwkNaU5zcI67YYWRqs=; b=eRq/VJdX9y4v40fObP7QbpCBf1vqcddE6nx2oDKpNxPDzOX+Dx77lUYvDiJKK/jTrvXASRAgEcA3efHJbgSwF8+UeFWocLKgGBwTdQY+QaPmFth0x9Ba6uITGi9ZQQ4/hqafUrOdSIoEwpepXIT5BsZVpCC2xdz/p7wcMW/3Gz0= Received: from MN2PR18MB2848.namprd18.prod.outlook.com (2603:10b6:208:38::19) by MN2PR18MB3117.namprd18.prod.outlook.com (2603:10b6:208:162::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2814.14; Mon, 16 Mar 2020 09:34:28 +0000 Received: from MN2PR18MB2848.namprd18.prod.outlook.com ([fe80::f829:3e55:94f6:4efb]) by MN2PR18MB2848.namprd18.prod.outlook.com ([fe80::f829:3e55:94f6:4efb%5]) with mapi id 15.20.2814.021; Mon, 16 Mar 2020 09:34:28 +0000 From: Harman Kalra To: , , CC: , , Harman Kalra Date: Mon, 16 Mar 2020 15:03:38 +0530 Message-ID: <1584351224-23500-3-git-send-email-hkalra@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1584351224-23500-1-git-send-email-hkalra@marvell.com> References: <1584351224-23500-1-git-send-email-hkalra@marvell.com> Content-Type: text/plain X-ClientProxiedBy: BMXPR01CA0006.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:d::16) To MN2PR18MB2848.namprd18.prod.outlook.com (2603:10b6:208:38::19) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from hkarlara-OptiPlex-3046.marvell.com (115.113.156.2) by BMXPR01CA0006.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:d::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.2814.13 via Frontend Transport; Mon, 16 Mar 2020 09:34:26 +0000 X-Mailer: git-send-email 2.7.4 X-Originating-IP: [115.113.156.2] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5bb475c4-4653-46cd-810f-08d7c98d38e6 X-MS-TrafficTypeDiagnostic: MN2PR18MB3117: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2201; X-Forefront-PRVS: 03449D5DD1 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(136003)(376002)(39860400002)(346002)(396003)(199004)(52116002)(7696005)(478600001)(4326008)(8676002)(8936002)(107886003)(26005)(66556008)(30864003)(2906002)(86362001)(5660300002)(36756003)(55236004)(16526019)(2616005)(956004)(66946007)(6486002)(66476007)(81156014)(81166006)(6666004)(186003)(316002); DIR:OUT; SFP:1101; SCL:1; SRVR:MN2PR18MB3117; H:MN2PR18MB2848.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; Received-SPF: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8MvtzjHAL32A9TJUNpkCzm3hVLtRnx3ZEaaYPYhozCVmFxTPSEV4NbmgIF9RGKY0CFliWy5eC53/0b1vn4h60Nbg1zWR4foEBAgoG6jQvZu4Ul/OIQbE7z3PZId6B3KQnojySeW1eEUzFItzf09qndnb5ytGdMeVybM78YKGDJ9eNZNewm0YMnXDdl/dLurdTuz9rtomjQC0LgjaPEcXuvNSxcqPIkhS+3uo8xE2oGqtmw57E3+fiABxF7ZhJ1RWnFFUJAkPq0Mbiw8dtpFUt8w9/gG4FtnBBgphjug+CZ8bBLGKzh63JD2jX7ZGxA9J7zyYytlrQdCwLysb1FCSo6Em2HOizk4Qo8MBGGljJUXPsY4WGHpKvkwT1jAkvgc3iPThdvhJN9JtoGVbwWknq+NAI2CMebqYB+ptZTQxv7ulGSU2oNktpIP6fxUIYChP X-MS-Exchange-AntiSpam-MessageData: zoB6sPYjiWTuGPjHMflMRJ6Nl7Wd0P0Uqg7+jJnqomqrbmpCRAN1CV/JidnsKA6z4BedLhl/l2sfOW3DstCV51Pe0HCvC57CETEnU5aqZPZS0qfdYPyJOYpFLSu3BtXzIdjwaEkAH42UUkCd6AgKUg== X-MS-Exchange-CrossTenant-Network-Message-Id: 5bb475c4-4653-46cd-810f-08d7c98d38e6 X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2020 09:34:28.0005 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nsOoAv2EuOlG6qEDBJh/edOscy3/SoldKO57TbMb8An4LFn3t8JV4nMI5IRawFyRmdJ0334uFfRQ6CJf+sZ31w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR18MB3117 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-03-16_02:2020-03-12, 2020-03-16 signatures=0 Subject: [dpdk-dev] [PATCH 2/8] net/octeontx: add framework for Rx/Tx offloads X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Adding macro based framework to hook rx/tx burst function pointers to the appropriate function based on rx/tx offloads. Signed-off-by: Harman Kalra --- drivers/event/octeontx/ssovf_worker.c | 9 +- drivers/net/octeontx/octeontx_ethdev.c | 27 ++-- drivers/net/octeontx/octeontx_ethdev.h | 12 +- drivers/net/octeontx/octeontx_rxtx.c | 81 +++++------- drivers/net/octeontx/octeontx_rxtx.h | 173 +++++++++++++++---------- 5 files changed, 151 insertions(+), 151 deletions(-) diff --git a/drivers/event/octeontx/ssovf_worker.c b/drivers/event/octeontx/ssovf_worker.c index f11b9d8c4..169939d1a 100644 --- a/drivers/event/octeontx/ssovf_worker.c +++ b/drivers/event/octeontx/ssovf_worker.c @@ -272,7 +272,7 @@ sso_event_tx_adapter_enqueue(void *port, struct rte_eth_dev *ethdev; struct ssows *ws = port; struct octeontx_txq *txq; - octeontx_dq_t *dq; + uint64_t cmd[4]; RTE_SET_USED(nb_events); switch (ev->sched_type) { @@ -297,11 +297,6 @@ sso_event_tx_adapter_enqueue(void *port, queue_id = rte_event_eth_tx_adapter_txq_get(m); ethdev = &rte_eth_devices[port_id]; txq = ethdev->data->tx_queues[queue_id]; - dq = &txq->dq; - if (__octeontx_xmit_pkts(dq->lmtline_va, dq->ioreg_va, dq->fc_status_va, - m, OCCTX_TX_OFFLOAD_NONE) < 0) - return 0; - - return 1; + return __octeontx_xmit_pkts(txq, &m, 1, cmd, OCCTX_TX_OFFLOAD_NONE); } diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c index 24c4e83a9..d6adbbc93 100644 --- a/drivers/net/octeontx/octeontx_ethdev.c +++ b/drivers/net/octeontx/octeontx_ethdev.c @@ -436,27 +436,20 @@ octeontx_recheck_rx_offloads(struct octeontx_rxq *rxq) return 0; } -static void -octeontx_set_tx_function(struct rte_eth_dev *dev) -{ - struct octeontx_nic *nic = octeontx_pmd_priv(dev); - - const eth_tx_burst_t tx_burst_func[2] = { - [0] = octeontx_xmit_pkts, - [1] = octeontx_xmit_pkts_mseg, - }; - - dev->tx_pkt_burst = - tx_burst_func[!!(nic->tx_offloads & DEV_TX_OFFLOAD_MULTI_SEGS)]; -} - static int octeontx_dev_start(struct rte_eth_dev *dev) { struct octeontx_nic *nic = octeontx_pmd_priv(dev); - int ret; + struct octeontx_rxq *rxq; + int ret = 0, i; - ret = 0; + /* Rechecking if any new offload set to update + * rx/tx burst function pointer accordingly. + */ + for (i = 0; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; + octeontx_recheck_rx_offloads(rxq); + } PMD_INIT_FUNC_TRACE(); /* @@ -1159,7 +1152,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev, eth_dev->dev_ops = &octeontx_dev_ops; eth_dev->device = &dev->device; - eth_dev->tx_pkt_burst = octeontx_xmit_pkts; + octeontx_set_tx_function(eth_dev); eth_dev->rx_pkt_burst = octeontx_recv_pkts; rte_eth_dev_probing_finish(eth_dev); return 0; diff --git a/drivers/net/octeontx/octeontx_ethdev.h b/drivers/net/octeontx/octeontx_ethdev.h index 10da6a2a0..06223e6e7 100644 --- a/drivers/net/octeontx/octeontx_ethdev.h +++ b/drivers/net/octeontx/octeontx_ethdev.h @@ -29,12 +29,12 @@ #define OCTEONTX_MAX_BGX_PORTS 4 #define OCTEONTX_MAX_LMAC_PER_BGX 4 -#define OCTEONTX_RX_OFFLOADS (DEV_RX_OFFLOAD_CHECKSUM | \ - DEV_RX_OFFLOAD_SCATTER | \ - DEV_RX_OFFLOAD_JUMBO_FRAME) +#define OCTEONTX_RX_OFFLOADS (DEV_RX_OFFLOAD_CHECKSUM | \ + DEV_RX_OFFLOAD_SCATTER | \ + DEV_RX_OFFLOAD_JUMBO_FRAME) -#define OCTEONTX_TX_OFFLOADS (DEV_TX_OFFLOAD_MT_LOCKFREE | \ - DEV_TX_OFFLOAD_MULTI_SEGS) +#define OCTEONTX_TX_OFFLOADS (DEV_TX_OFFLOAD_MT_LOCKFREE | \ + DEV_TX_OFFLOAD_MULTI_SEGS) static inline struct octeontx_nic * octeontx_pmd_priv(struct rte_eth_dev *dev) @@ -99,4 +99,6 @@ struct octeontx_rxq { struct rte_mempool *pool; } __rte_cache_aligned; +void +octeontx_set_tx_function(struct rte_eth_dev *dev); #endif /* __OCTEONTX_ETHDEV_H__ */ diff --git a/drivers/net/octeontx/octeontx_rxtx.c b/drivers/net/octeontx/octeontx_rxtx.c index 8f6d14b5f..3de88e187 100644 --- a/drivers/net/octeontx/octeontx_rxtx.c +++ b/drivers/net/octeontx/octeontx_rxtx.c @@ -19,56 +19,6 @@ #include "octeontx_rxtx.h" #include "octeontx_logs.h" -uint16_t __hot -octeontx_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) -{ - int count; - struct octeontx_txq *txq = tx_queue; - octeontx_dq_t *dq = &txq->dq; - int res; - - count = 0; - - rte_cio_wmb(); - while (count < nb_pkts) { - res = __octeontx_xmit_pkts(dq->lmtline_va, dq->ioreg_va, - dq->fc_status_va, tx_pkts[count], - OCCTX_TX_OFFLOAD_NONE); - if (res < 0) - break; - - count++; - } - - return count; /* return number of pkts transmitted */ -} - -uint16_t __hot -octeontx_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts) -{ - int count; - struct octeontx_txq *txq = tx_queue; - octeontx_dq_t *dq = &txq->dq; - int res; - - count = 0; - - rte_cio_wmb(); - while (count < nb_pkts) { - res = __octeontx_xmit_pkts(dq->lmtline_va, dq->ioreg_va, - dq->fc_status_va, tx_pkts[count], - OCCTX_TX_OFFLOAD_NONE | - OCCTX_TX_MULTI_SEG_F); - if (res < 0) - break; - - count++; - } - - return count; /* return number of pkts transmitted */ -} - uint16_t __hot octeontx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) { @@ -90,3 +40,34 @@ octeontx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) return count; /* return number of pkts received */ } + +#define T(name, f1, sz, flags) \ +static uint16_t __rte_noinline __hot \ +octeontx_xmit_pkts_ ##name(void *tx_queue, \ + struct rte_mbuf **tx_pkts, uint16_t pkts) \ +{ \ + uint64_t cmd[(sz)]; \ + \ + return __octeontx_xmit_pkts(tx_queue, tx_pkts, pkts, cmd, \ + flags); \ +} + +OCCTX_TX_FASTPATH_MODES +#undef T + +void __hot +octeontx_set_tx_function(struct rte_eth_dev *dev) +{ + struct octeontx_nic *nic = octeontx_pmd_priv(dev); + + const eth_tx_burst_t tx_burst_func[2] = { +#define T(name, f0, sz, flags) \ + [f0] = octeontx_xmit_pkts_ ##name, + +OCCTX_TX_FASTPATH_MODES +#undef T + }; + + dev->tx_pkt_burst = tx_burst_func + [!!(nic->tx_offload_flags & OCCTX_TX_MULTI_SEG_F)]; +} diff --git a/drivers/net/octeontx/octeontx_rxtx.h b/drivers/net/octeontx/octeontx_rxtx.h index 562268f16..6182cd8ee 100644 --- a/drivers/net/octeontx/octeontx_rxtx.h +++ b/drivers/net/octeontx/octeontx_rxtx.h @@ -113,90 +113,119 @@ ptype_table[PTYPE_SIZE][PTYPE_SIZE][PTYPE_SIZE] = { }; -static __rte_always_inline int -__octeontx_xmit_pkts(void *lmtline_va, void *ioreg_va, int64_t *fc_status_va, - struct rte_mbuf *tx_pkt, const uint16_t flag) + +static __rte_always_inline uint16_t +__octeontx_xmit_prepare(struct rte_mbuf *tx_pkt, uint64_t *cmd_buf, + const uint16_t flag __rte_unused) +{ + uint16_t gaura_id, nb_desc = 0; + + /* Setup PKO_SEND_HDR_S */ + cmd_buf[nb_desc++] = tx_pkt->data_len & 0xffff; + cmd_buf[nb_desc++] = 0x0; + + /* Mark mempool object as "put" since it is freed by PKO */ + if (!(cmd_buf[0] & (1ULL << 58))) + __mempool_check_cookies(tx_pkt->pool, (void **)&tx_pkt, + 1, 0); + /* Get the gaura Id */ + gaura_id = octeontx_fpa_bufpool_gpool((uintptr_t) + tx_pkt->pool->pool_id); + + /* Setup PKO_SEND_BUFLINK_S */ + cmd_buf[nb_desc++] = PKO_SEND_BUFLINK_SUBDC | + PKO_SEND_BUFLINK_LDTYPE(0x1ull) | + PKO_SEND_BUFLINK_GAUAR((long)gaura_id) | + tx_pkt->data_len; + cmd_buf[nb_desc++] = rte_mbuf_data_iova(tx_pkt); + + return nb_desc; +} + +static __rte_always_inline uint16_t +__octeontx_xmit_mseg_prepare(struct rte_mbuf *tx_pkt, uint64_t *cmd_buf, + const uint16_t flag __rte_unused) { - uint8_t sz = (4 + (!!(flag & OCCTX_TX_MULTI_SEG_F) * 10)); - /* Max size of PKO SEND desc is 112 bytes*/ - uint64_t cmd_buf[sz] __rte_cache_aligned; - uint8_t nb_segs, nb_desc = 0; + uint16_t nb_segs, nb_desc = 0; uint16_t gaura_id, len = 0; struct rte_mbuf *m_next = NULL; - if (unlikely(*((volatile int64_t *)fc_status_va) < 0)) - return -ENOSPC; - - - if (flag & OCCTX_TX_MULTI_SEG_F) { - nb_segs = tx_pkt->nb_segs; - /* Setup PKO_SEND_HDR_S */ - cmd_buf[nb_desc++] = tx_pkt->pkt_len & 0xffff; - cmd_buf[nb_desc++] = 0x0; - - do { - m_next = tx_pkt->next; - /* To handle case where mbufs belong to diff pools, like - * fragmentation - */ - gaura_id = octeontx_fpa_bufpool_gpool((uintptr_t) - tx_pkt->pool->pool_id); - - /* Setup PKO_SEND_GATHER_S */ - cmd_buf[nb_desc] = PKO_SEND_GATHER_SUBDC | - PKO_SEND_GATHER_LDTYPE(0x1ull) | - PKO_SEND_GATHER_GAUAR((long) - gaura_id) | - tx_pkt->data_len; - /* Mark mempool object as "put" since it is freed by - * PKO. - */ - if (!(cmd_buf[nb_desc] & (1ULL << 57))) { - tx_pkt->next = NULL; - __mempool_check_cookies(tx_pkt->pool, - (void **)&tx_pkt, 1, 0); - } - nb_desc++; - - cmd_buf[nb_desc++] = rte_mbuf_data_iova(tx_pkt); - - nb_segs--; - len += tx_pkt->data_len; - tx_pkt = m_next; - } while (nb_segs); - } else { - /* Setup PKO_SEND_HDR_S */ - cmd_buf[nb_desc++] = tx_pkt->data_len & 0xffff; - cmd_buf[nb_desc++] = 0x0; - - /* Mark mempool object as "put" since it is freed by PKO */ - if (!(cmd_buf[0] & (1ULL << 58))) - __mempool_check_cookies(tx_pkt->pool, (void **)&tx_pkt, - 1, 0); - /* Get the gaura Id */ + nb_segs = tx_pkt->nb_segs; + /* Setup PKO_SEND_HDR_S */ + cmd_buf[nb_desc++] = tx_pkt->pkt_len & 0xffff; + cmd_buf[nb_desc++] = 0x0; + + do { + m_next = tx_pkt->next; + /* To handle case where mbufs belong to diff pools, like + * fragmentation + */ gaura_id = octeontx_fpa_bufpool_gpool((uintptr_t) tx_pkt->pool->pool_id); - /* Setup PKO_SEND_BUFLINK_S */ - cmd_buf[nb_desc++] = PKO_SEND_BUFLINK_SUBDC | - PKO_SEND_BUFLINK_LDTYPE(0x1ull) | - PKO_SEND_BUFLINK_GAUAR((long)gaura_id) | - tx_pkt->data_len; + /* Setup PKO_SEND_GATHER_S */ + cmd_buf[nb_desc] = PKO_SEND_GATHER_SUBDC | + PKO_SEND_GATHER_LDTYPE(0x1ull) | + PKO_SEND_GATHER_GAUAR((long)gaura_id) | + tx_pkt->data_len; + + /* Mark mempool object as "put" since it is freed by + * PKO. + */ + if (!(cmd_buf[nb_desc] & (1ULL << 57))) { + tx_pkt->next = NULL; + __mempool_check_cookies(tx_pkt->pool, + (void **)&tx_pkt, 1, 0); + } + nb_desc++; + cmd_buf[nb_desc++] = rte_mbuf_data_iova(tx_pkt); - } - octeontx_reg_lmtst(lmtline_va, ioreg_va, cmd_buf, nb_desc); - return 0; -} + nb_segs--; + len += tx_pkt->data_len; + tx_pkt = m_next; + } while (nb_segs); -uint16_t -octeontx_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts); + return nb_desc; +} -uint16_t -octeontx_xmit_pkts_mseg(void *tx_queue, struct rte_mbuf **tx_pkts, - uint16_t nb_pkts); +static __rte_always_inline uint16_t +__octeontx_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts, uint64_t *cmd_buf, + const uint16_t flags) +{ + struct octeontx_txq *txq = tx_queue; + octeontx_dq_t *dq = &txq->dq; + uint16_t count = 0, nb_desc; + rte_cio_wmb(); + + while (count < nb_pkts) { + if (unlikely(*((volatile int64_t *)dq->fc_status_va) < 0)) + break; + + if (flags & OCCTX_TX_MULTI_SEG_F) { + nb_desc = __octeontx_xmit_mseg_prepare(tx_pkts[count], + cmd_buf, flags); + } else { + nb_desc = __octeontx_xmit_prepare(tx_pkts[count], + cmd_buf, flags); + } + + octeontx_reg_lmtst(dq->lmtline_va, dq->ioreg_va, cmd_buf, + nb_desc); + + count++; + } + return count; +} uint16_t octeontx_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); -#endif /* __OCTEONTX_RXTX_H__ */ +#define MULT_F OCCTX_TX_MULTI_SEG_F +/* [NOFF] [MULTI_SEG] */ +#define OCCTX_TX_FASTPATH_MODES \ +T(no_offload, 0, 4, OCCTX_TX_OFFLOAD_NONE) \ +T(mseg, 1, 14, MULT_F) \ + + #endif /* __OCTEONTX_RXTX_H__ */ -- 2.18.0