From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0083.outbound.protection.outlook.com [104.47.38.83]) by dpdk.org (Postfix) with ESMTP id 2AE7C1B5AD for ; Fri, 12 Oct 2018 13:09:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=AQUANTIA1COM.onmicrosoft.com; s=selector1-aquantia-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eeMUay/QIVhlnmzBOCzccgCmYnwyaOGunUigTthUmhs=; b=OhlxQCK3pcYqEbTLsmNEkiI/CvFWYMa7UydUuNzoDSP1i6H5rV6vBTXGrmEmPqTETTpUQuyy/cvOIe2Fey+ujanfh0qgqiLL6Z1sDk/2ec0a+8fgo3uZuwUuepbNOH+Pvr0KqFxqZcvQ0tidgoYctwWHFlPerH8BBm/vK24VYNo= Received: from BY1PR0701MB1660.namprd07.prod.outlook.com (10.162.110.22) by BY1PR0701MB1627.namprd07.prod.outlook.com (10.162.110.156) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1228.22; Fri, 12 Oct 2018 11:09:27 +0000 Received: from BY1PR0701MB1660.namprd07.prod.outlook.com ([fe80::346d:e756:e70e:17]) by BY1PR0701MB1660.namprd07.prod.outlook.com ([fe80::346d:e756:e70e:17%3]) with mapi id 15.20.1228.020; Fri, 12 Oct 2018 11:09:27 +0000 From: Igor Russkikh To: "dev@dpdk.org" CC: Pavel Belous , Igor Russkikh , "ferruh.yigit@intel.com" Thread-Topic: [PATCH v6 10/22] net/atlantic: implement Tx path Thread-Index: AQHUYhwKGhPlLvxpsk2EUCBTBWKZQQ== Date: Fri, 12 Oct 2018 11:09:27 +0000 Message-ID: <4a77b5e45cc3c2a741804c0efd62436df9885ed5.1539338074.git.igor.russkikh@aquantia.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: AM5PR0701CA0008.eurprd07.prod.outlook.com (2603:10a6:203:51::18) To BY1PR0701MB1660.namprd07.prod.outlook.com (2a01:111:e400:522a::22) authentication-results: spf=none (sender IP is ) smtp.mailfrom=Igor.Russkikh@aquantia.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [95.79.108.179] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BY1PR0701MB1627; 6:nFvj6Cy3Kgv6xVIWww/PsnEj3SI4K/LzWi0fDBgREePZCtY/L+TfqKdR+dKPtLrkRsxlAK3pZsWTdP+KjvqI6iuWYDCvj9tLOpNistJtauPd4xLtVcRusZ9uowXXFosSVL32iHn8GgxkGlw3n3v5fuMaT6qflCyvhPG5eX9pEvMHDNIVlwUZiNjALrkhQXYmeMUr3rMsI9v7yTnlxMNaotFmZL2PgxPXSsKfH5ZGZtBVvjEnJ4NyrA3TQF7V6Tgt+shOtiJN5b6GrpS3XwJ40+ZqZG962rsY3mIT3ghCB3Jh0pfeRmlGmOw+XtAV6LrgpTHngzvQ7uX2bxpk+HosbS8CWovIZLTTG2O8njviYy2X3Tu5pxE+fW8OGg5aE6m3x1VdSMY2YCRI5KxvweqeBpzJpE7FAWiJaDHBzIi32t2CcBFouCE3Y+0SNfO42erRMNyughowVMruqU5cEKuwSw==; 5:8duct0IE9SyQgk4SL3iE0txgAMegFSo+ZivW4T+HggwCSmEMo5Nwj0QHOxU4QST/Xf+rWuEMmkywBkZMkUJbtYzWyXEzcn3xW58ZrMeSsWTDiIHm6WOzUiZgZp5C6V8KAx297mDMfxfkBt9WNKpR7Lp9w2gmGHV+0UPAqcJb25c=; 7:ZwOnFN+C2DjwDXT5A84hmmDo0diQ3yOXaJ0+JfH1hCm7sM/RfpfG4C2FUbXXdtRgxMK+0STesa1h+nBvTZMYqtNejIelx1FPJn2XPPw6w9C2+6zB1WWZaG+oScCO69dZBQI0fxduOAN1wFIavdzYn0X7MKyL/2w4Us52I9h+QkWDhyPx0ES+c5wVjNf8XP1VaYKrDRXwG1JI8+l/f8QPN94a3eQIaXbvih2NnWch1f6NsyIIjLFN5A23y3853+VZ x-ms-office365-filtering-correlation-id: eeb37344-0231-4b11-75c1-08d630332c7e x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:BY1PR0701MB1627; x-ms-traffictypediagnostic: BY1PR0701MB1627: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(3231355)(944501410)(52105095)(3002001)(93006095)(93001095)(10201501046)(149066)(150057)(6041310)(20161123560045)(20161123562045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(201708071742011)(7699051)(76991067); SRVR:BY1PR0701MB1627; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1627; x-forefront-prvs: 0823A5777B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(366004)(376002)(136003)(39850400004)(396003)(189003)(199004)(186003)(6116002)(486006)(5640700003)(446003)(6436002)(6512007)(53936002)(71200400001)(71190400001)(14454004)(72206003)(316002)(2351001)(44832011)(3846002)(7736002)(305945005)(2906002)(4326008)(6486002)(97736004)(26005)(2501003)(102836004)(53946003)(5250100002)(11346002)(86362001)(25786009)(4744004)(476003)(2616005)(106356001)(54906003)(105586002)(256004)(14444005)(5660300001)(66066001)(81166006)(81156014)(1730700003)(8676002)(478600001)(36756003)(68736007)(6916009)(8936002)(99286004)(386003)(6506007)(118296001)(2900100001)(52116002)(76176011)(309714004); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1627; H:BY1PR0701MB1660.namprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: aquantia.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: k48DKjN0ykEXtrLfJSOIViOskTuqrEtrjhayBh0AZ4A7DmAN5n+mxILI6DKehirBSsRhJ28WBgL7FhPgG/CyDMG+DGKJCniaEtcDhCpQdqE3T91376ZqZ2swnPUkBF29ItXMUnghkSJW/FaCT2+aLvm6XwcCnliIWMI0LfB0An1GwUbT+UWh8LyUV8+TsSL6hJfbnecVZmCxYeGYQWv5QopLavmo4nQKYy7yDbE8k8D6UMLRg2oTrjQCh8PrYU4nOpY2rDr8a2hztdLdCISRUG1NcNdk2wDflZxnt8LPtHisX5MZtfl2IZK45gZrNv8lENSKrJ8YUDViaAPKaSBSJAwwv84EFF9ExwwP9+6qgp0= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: aquantia.com X-MS-Exchange-CrossTenant-Network-Message-Id: eeb37344-0231-4b11-75c1-08d630332c7e X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Oct 2018 11:09:27.2068 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 83e2e134-991c-4ede-8ced-34d47e38e6b1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1627 Subject: [dpdk-dev] [PATCH v6 10/22] net/atlantic: implement Tx path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Oct 2018 11:09:30 -0000 From: Pavel Belous Add implementation for TX datapath. Signed-off-by: Igor Russkikh Signed-off-by: Pavel Belous --- drivers/net/atlantic/atl_ethdev.c | 28 ++ drivers/net/atlantic/atl_ethdev.h | 7 + drivers/net/atlantic/atl_rxtx.c | 546 ++++++++++++++++++++++++++++++++++= +++- 3 files changed, 572 insertions(+), 9 deletions(-) diff --git a/drivers/net/atlantic/atl_ethdev.c b/drivers/net/atlantic/atl_e= thdev.c index 521376d1398c..156f1be99cdf 100644 --- a/drivers/net/atlantic/atl_ethdev.c +++ b/drivers/net/atlantic/atl_ethdev.c @@ -82,12 +82,27 @@ static struct rte_pci_driver rte_atl_pmd =3D { | DEV_RX_OFFLOAD_TCP_CKSUM \ | DEV_RX_OFFLOAD_JUMBO_FRAME) =20 +#define ATL_TX_OFFLOADS (DEV_TX_OFFLOAD_VLAN_INSERT \ + | DEV_TX_OFFLOAD_IPV4_CKSUM \ + | DEV_TX_OFFLOAD_UDP_CKSUM \ + | DEV_TX_OFFLOAD_TCP_CKSUM \ + | DEV_TX_OFFLOAD_TCP_TSO \ + | DEV_TX_OFFLOAD_MULTI_SEGS) + static const struct rte_eth_desc_lim rx_desc_lim =3D { .nb_max =3D ATL_MAX_RING_DESC, .nb_min =3D ATL_MIN_RING_DESC, .nb_align =3D ATL_RXD_ALIGN, }; =20 +static const struct rte_eth_desc_lim tx_desc_lim =3D { + .nb_max =3D ATL_MAX_RING_DESC, + .nb_min =3D ATL_MIN_RING_DESC, + .nb_align =3D ATL_TXD_ALIGN, + .nb_seg_max =3D ATL_TX_MAX_SEG, + .nb_mtu_seg_max =3D ATL_TX_MAX_SEG, +}; + static const struct eth_dev_ops atl_eth_dev_ops =3D { .dev_configure =3D atl_dev_configure, .dev_start =3D atl_dev_start, @@ -104,6 +119,11 @@ static const struct eth_dev_ops atl_eth_dev_ops =3D { .rx_queue_stop =3D atl_rx_queue_stop, .rx_queue_setup =3D atl_rx_queue_setup, .rx_queue_release =3D atl_rx_queue_release, + + .tx_queue_start =3D atl_tx_queue_start, + .tx_queue_stop =3D atl_tx_queue_stop, + .tx_queue_setup =3D atl_tx_queue_setup, + .tx_queue_release =3D atl_tx_queue_release, }; =20 static inline int32_t @@ -350,11 +370,19 @@ atl_dev_info_get(struct rte_eth_dev *dev, struct rte_= eth_dev_info *dev_info) =20 dev_info->rx_offload_capa =3D ATL_RX_OFFLOADS; =20 + dev_info->tx_offload_capa =3D ATL_TX_OFFLOADS; + + dev_info->default_rxconf =3D (struct rte_eth_rxconf) { .rx_free_thresh =3D ATL_DEFAULT_RX_FREE_THRESH, }; =20 + dev_info->default_txconf =3D (struct rte_eth_txconf) { + .tx_free_thresh =3D ATL_DEFAULT_TX_FREE_THRESH, + }; + dev_info->rx_desc_lim =3D rx_desc_lim; + dev_info->tx_desc_lim =3D tx_desc_lim; } =20 static const uint32_t * diff --git a/drivers/net/atlantic/atl_ethdev.h b/drivers/net/atlantic/atl_e= thdev.h index a9a9fc8fe7ff..92bb302ffe55 100644 --- a/drivers/net/atlantic/atl_ethdev.h +++ b/drivers/net/atlantic/atl_ethdev.h @@ -29,12 +29,17 @@ struct atl_adapter { * RX/TX function prototypes */ void atl_rx_queue_release(void *rxq); +void atl_tx_queue_release(void *txq); =20 int atl_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id, uint16_t nb_rx_desc, unsigned int socket_id, const struct rte_eth_rxconf *rx_conf, struct rte_mempool *mb_pool); =20 +int atl_tx_queue_setup(struct rte_eth_dev *dev, uint16_t tx_queue_id, + uint16_t nb_tx_desc, unsigned int socket_id, + const struct rte_eth_txconf *tx_conf); + int atl_rx_init(struct rte_eth_dev *dev); int atl_tx_init(struct rte_eth_dev *dev); =20 @@ -45,6 +50,8 @@ void atl_free_queues(struct rte_eth_dev *dev); int atl_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id); int atl_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id); =20 +int atl_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id); +int atl_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id); =20 uint16_t atl_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); diff --git a/drivers/net/atlantic/atl_rxtx.c b/drivers/net/atlantic/atl_rxt= x.c index 61dbb83b4572..1d4c8099325d 100644 --- a/drivers/net/atlantic/atl_rxtx.c +++ b/drivers/net/atlantic/atl_rxtx.c @@ -4,6 +4,7 @@ =20 #include #include +#include =20 #include "atl_ethdev.h" #include "atl_hw_regs.h" @@ -13,6 +14,20 @@ #include "hw_atl/hw_atl_b0.h" #include "hw_atl/hw_atl_b0_internal.h" =20 +#define ATL_TX_CKSUM_OFFLOAD_MASK ( \ + PKT_TX_IP_CKSUM | \ + PKT_TX_L4_MASK | \ + PKT_TX_TCP_SEG) + +#define ATL_TX_OFFLOAD_MASK ( \ + PKT_TX_VLAN | \ + PKT_TX_IP_CKSUM | \ + PKT_TX_L4_MASK | \ + PKT_TX_TCP_SEG) + +#define ATL_TX_OFFLOAD_NOTSUP_MASK \ + (PKT_TX_OFFLOAD_MASK ^ ATL_TX_OFFLOAD_MASK) + /** * Structure associated with each descriptor of the RX ring of a RX queue. */ @@ -21,6 +36,15 @@ struct atl_rx_entry { }; =20 /** + * Structure associated with each descriptor of the TX ring of a TX queue. + */ +struct atl_tx_entry { + struct rte_mbuf *mbuf; + uint16_t next_id; + uint16_t last_id; +}; + +/** * Structure associated with each RX queue. */ struct atl_rx_queue { @@ -39,6 +63,22 @@ struct atl_rx_queue { bool l4_csum_enabled; }; =20 +/** + * Structure associated with each TX queue. + */ +struct atl_tx_queue { + struct hw_atl_txd_s *hw_ring; + uint64_t hw_ring_phys_addr; + struct atl_tx_entry *sw_ring; + uint16_t nb_tx_desc; + uint16_t tx_tail; + uint16_t tx_head; + uint16_t queue_id; + uint16_t port_id; + uint16_t tx_free_thresh; + uint16_t tx_free; +}; + static inline void atl_reset_rx_queue(struct atl_rx_queue *rxq) { @@ -147,13 +187,152 @@ atl_rx_queue_setup(struct rte_eth_dev *dev, uint16_t= rx_queue_id, return 0; } =20 +static inline void +atl_reset_tx_queue(struct atl_tx_queue *txq) +{ + struct atl_tx_entry *tx_entry; + union hw_atl_txc_s *txc; + uint16_t i; + + PMD_INIT_FUNC_TRACE(); + + if (!txq) { + PMD_DRV_LOG(ERR, "Pointer to txq is NULL"); + return; + } + + tx_entry =3D txq->sw_ring; + + for (i =3D 0; i < txq->nb_tx_desc; i++) { + txc =3D (union hw_atl_txc_s *)&txq->hw_ring[i]; + txc->flags1 =3D 0; + txc->flags2 =3D 2; + } + + for (i =3D 0; i < txq->nb_tx_desc; i++) { + txq->hw_ring[i].dd =3D 1; + tx_entry[i].mbuf =3D NULL; + } + + txq->tx_tail =3D 0; + txq->tx_head =3D 0; + txq->tx_free =3D txq->nb_tx_desc - 1; +} + int -atl_tx_init(struct rte_eth_dev *eth_dev __rte_unused) +atl_tx_queue_setup(struct rte_eth_dev *dev, uint16_t tx_queue_id, + uint16_t nb_tx_desc, unsigned int socket_id, + const struct rte_eth_txconf *tx_conf) { + struct atl_tx_queue *txq; + const struct rte_memzone *mz; + + PMD_INIT_FUNC_TRACE(); + + /* make sure a valid number of descriptors have been requested */ + if (nb_tx_desc < AQ_HW_MIN_TX_RING_SIZE || + nb_tx_desc > AQ_HW_MAX_TX_RING_SIZE) { + PMD_INIT_LOG(ERR, "Number of Tx descriptors must be " + "less than or equal to %d, " + "greater than or equal to %d", AQ_HW_MAX_TX_RING_SIZE, + AQ_HW_MIN_TX_RING_SIZE); + return -EINVAL; + } + + /* + * if this queue existed already, free the associated memory. The + * queue cannot be reused in case we need to allocate memory on + * different socket than was previously used. + */ + if (dev->data->tx_queues[tx_queue_id] !=3D NULL) { + atl_tx_queue_release(dev->data->tx_queues[tx_queue_id]); + dev->data->tx_queues[tx_queue_id] =3D NULL; + } + + /* allocate memory for the queue structure */ + txq =3D rte_zmalloc_socket("atlantic Tx queue", sizeof(*txq), + RTE_CACHE_LINE_SIZE, socket_id); + if (txq =3D=3D NULL) { + PMD_INIT_LOG(ERR, "Cannot allocate queue structure"); + return -ENOMEM; + } + + /* setup queue */ + txq->nb_tx_desc =3D nb_tx_desc; + txq->port_id =3D dev->data->port_id; + txq->queue_id =3D tx_queue_id; + txq->tx_free_thresh =3D tx_conf->tx_free_thresh; + + + /* allocate memory for the software ring */ + txq->sw_ring =3D rte_zmalloc_socket("atlantic sw tx ring", + nb_tx_desc * sizeof(struct atl_tx_entry), + RTE_CACHE_LINE_SIZE, socket_id); + if (txq->sw_ring =3D=3D NULL) { + PMD_INIT_LOG(ERR, + "Port %d: Cannot allocate software ring for queue %d", + txq->port_id, txq->queue_id); + rte_free(txq); + return -ENOMEM; + } + + /* + * allocate memory for the hardware descriptor ring. A memzone large + * enough to hold the maximum ring size is requested to allow for + * resizing in later calls to the queue setup function. + */ + mz =3D rte_eth_dma_zone_reserve(dev, "tx hw_ring", tx_queue_id, + HW_ATL_B0_MAX_TXD * sizeof(struct hw_atl_txd_s), + 128, socket_id); + if (mz =3D=3D NULL) { + PMD_INIT_LOG(ERR, + "Port %d: Cannot allocate hardware ring for queue %d", + txq->port_id, txq->queue_id); + rte_free(txq->sw_ring); + rte_free(txq); + return -ENOMEM; + } + txq->hw_ring =3D mz->addr; + txq->hw_ring_phys_addr =3D mz->iova; + + atl_reset_tx_queue(txq); + + dev->data->tx_queues[tx_queue_id] =3D txq; return 0; } =20 int +atl_tx_init(struct rte_eth_dev *eth_dev) +{ + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private); + struct atl_tx_queue *txq; + uint64_t base_addr =3D 0; + int i =3D 0; + int err =3D 0; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < eth_dev->data->nb_tx_queues; i++) { + txq =3D eth_dev->data->tx_queues[i]; + base_addr =3D txq->hw_ring_phys_addr; + + err =3D hw_atl_b0_hw_ring_tx_init(hw, base_addr, + txq->queue_id, + txq->nb_tx_desc, 0, + txq->port_id); + + if (err) { + PMD_INIT_LOG(ERR, + "Port %d: Cannot init TX queue %d", + txq->port_id, txq->queue_id); + break; + } + } + + return err; +} + +int atl_rx_init(struct rte_eth_dev *eth_dev) { struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private); @@ -325,15 +504,78 @@ atl_rx_queue_release(void *rx_queue) } } =20 -uint16_t -atl_prep_pkts(void *tx_queue __rte_unused, - struct rte_mbuf **tx_pkts __rte_unused, - uint16_t nb_pkts __rte_unused) +static void +atl_tx_queue_release_mbufs(struct atl_tx_queue *txq) +{ + int i; + + PMD_INIT_FUNC_TRACE(); + + if (txq->sw_ring !=3D NULL) { + for (i =3D 0; i < txq->nb_tx_desc; i++) { + if (txq->sw_ring[i].mbuf !=3D NULL) { + rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf); + txq->sw_ring[i].mbuf =3D NULL; + } + } + } +} + +int +atl_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id) +{ + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private); + + PMD_INIT_FUNC_TRACE(); + + if (tx_queue_id < dev->data->nb_tx_queues) { + hw_atl_b0_hw_ring_tx_start(hw, tx_queue_id); + + rte_wmb(); + hw_atl_b0_hw_tx_ring_tail_update(hw, 0, tx_queue_id); + dev->data->tx_queue_state[tx_queue_id] =3D + RTE_ETH_QUEUE_STATE_STARTED; + } else { + return -1; + } + + return 0; +} + +int +atl_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id) { + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct atl_tx_queue *txq; + + PMD_INIT_FUNC_TRACE(); + + txq =3D dev->data->tx_queues[tx_queue_id]; + + hw_atl_b0_hw_ring_tx_stop(hw, tx_queue_id); + + atl_tx_queue_release_mbufs(txq); + atl_reset_tx_queue(txq); + dev->data->tx_queue_state[tx_queue_id] =3D RTE_ETH_QUEUE_STATE_STOPPED; + return 0; } =20 void +atl_tx_queue_release(void *tx_queue) +{ + PMD_INIT_FUNC_TRACE(); + + if (tx_queue !=3D NULL) { + struct atl_tx_queue *txq =3D (struct atl_tx_queue *)tx_queue; + + atl_tx_queue_release_mbufs(txq); + rte_free(txq->sw_ring); + rte_free(txq); + } +} + +void atl_free_queues(struct rte_eth_dev *dev) { unsigned int i; @@ -345,6 +587,12 @@ atl_free_queues(struct rte_eth_dev *dev) dev->data->rx_queues[i] =3D 0; } dev->data->nb_rx_queues =3D 0; + + for (i =3D 0; i < dev->data->nb_tx_queues; i++) { + atl_tx_queue_release(dev->data->tx_queues[i]); + dev->data->tx_queues[i] =3D 0; + } + dev->data->nb_tx_queues =3D 0; } =20 int @@ -354,6 +602,15 @@ atl_start_queues(struct rte_eth_dev *dev) =20 PMD_INIT_FUNC_TRACE(); =20 + for (i =3D 0; i < dev->data->nb_tx_queues; i++) { + if (atl_tx_queue_start(dev, i) !=3D 0) { + PMD_DRV_LOG(ERR, + "Port %d: Start Tx queue %d failed", + dev->data->port_id, i); + return -1; + } + } + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { if (atl_rx_queue_start(dev, i) !=3D 0) { PMD_DRV_LOG(ERR, @@ -373,6 +630,15 @@ atl_stop_queues(struct rte_eth_dev *dev) =20 PMD_INIT_FUNC_TRACE(); =20 + for (i =3D 0; i < dev->data->nb_tx_queues; i++) { + if (atl_tx_queue_stop(dev, i) !=3D 0) { + PMD_DRV_LOG(ERR, + "Port %d: Stop Tx queue %d failed", + dev->data->port_id, i); + return -1; + } + } + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { if (atl_rx_queue_stop(dev, i) !=3D 0) { PMD_DRV_LOG(ERR, @@ -385,6 +651,47 @@ atl_stop_queues(struct rte_eth_dev *dev) return 0; } =20 +uint16_t +atl_prep_pkts(__rte_unused void *tx_queue, struct rte_mbuf **tx_pkts, + uint16_t nb_pkts) +{ + int i, ret; + uint64_t ol_flags; + struct rte_mbuf *m; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < nb_pkts; i++) { + m =3D tx_pkts[i]; + ol_flags =3D m->ol_flags; + + if (m->nb_segs > AQ_HW_MAX_SEGS_SIZE) { + rte_errno =3D -EINVAL; + return i; + } + + if (ol_flags & ATL_TX_OFFLOAD_NOTSUP_MASK) { + rte_errno =3D -ENOTSUP; + return i; + } + +#ifdef RTE_LIBRTE_ETHDEV_DEBUG + ret =3D rte_validate_tx_offload(m); + if (ret !=3D 0) { + rte_errno =3D ret; + return i; + } +#endif + ret =3D rte_net_intel_cksum_prepare(m); + if (ret !=3D 0) { + rte_errno =3D ret; + return i; + } + } + + return i; +} + static uint64_t atl_desc_to_offload_flags(struct atl_rx_queue *rxq, struct hw_atl_rxd_wb_s *rxd_wb) @@ -653,12 +960,233 @@ atl_recv_pkts(void *rx_queue, struct rte_mbuf **rx_p= kts, uint16_t nb_pkts) return nb_rx; } =20 +static void +atl_xmit_cleanup(struct atl_tx_queue *txq) +{ + struct atl_tx_entry *sw_ring; + struct hw_atl_txd_s *txd; + int to_clean =3D 0; + + PMD_INIT_FUNC_TRACE(); + + if (txq !=3D NULL) { + sw_ring =3D txq->sw_ring; + int head =3D txq->tx_head; + int cnt; + int i; + + for (i =3D 0, cnt =3D head; ; i++) { + txd =3D &txq->hw_ring[cnt]; + + if (txd->dd) + to_clean++; + + cnt =3D (cnt + 1) % txq->nb_tx_desc; + if (cnt =3D=3D txq->tx_tail) + break; + } + + if (to_clean =3D=3D 0) + return; + + while (to_clean) { + txd =3D &txq->hw_ring[head]; + + struct atl_tx_entry *rx_entry =3D &sw_ring[head]; + + if (rx_entry->mbuf) { + rte_pktmbuf_free_seg(rx_entry->mbuf); + rx_entry->mbuf =3D NULL; + } + + if (txd->dd) + to_clean--; + + txd->buf_addr =3D 0; + txd->flags =3D 0; + + head =3D (head + 1) % txq->nb_tx_desc; + txq->tx_free++; + } + + txq->tx_head =3D head; + } +} + +static int +atl_tso_setup(struct rte_mbuf *tx_pkt, union hw_atl_txc_s *txc) +{ + uint32_t tx_cmd =3D 0; + uint64_t ol_flags =3D tx_pkt->ol_flags; + + PMD_INIT_FUNC_TRACE(); + + if (ol_flags & PKT_TX_TCP_SEG) { + PMD_DRV_LOG(DEBUG, "xmit TSO pkt"); + + tx_cmd |=3D tx_desc_cmd_lso | tx_desc_cmd_l4cs; + + txc->cmd =3D 0x4; + + if (ol_flags & PKT_TX_IPV6) + txc->cmd |=3D 0x2; + + txc->l2_len =3D tx_pkt->l2_len; + txc->l3_len =3D tx_pkt->l3_len; + txc->l4_len =3D tx_pkt->l4_len; + + txc->mss_len =3D tx_pkt->tso_segsz; + } + + if (ol_flags & PKT_TX_VLAN) { + tx_cmd |=3D tx_desc_cmd_vlan; + txc->vlan_tag =3D tx_pkt->vlan_tci; + } + + if (tx_cmd) { + txc->type =3D tx_desc_type_ctx; + txc->idx =3D 0; + } + + return tx_cmd; +} + +static inline void +atl_setup_csum_offload(struct rte_mbuf *mbuf, struct hw_atl_txd_s *txd, + uint32_t tx_cmd) +{ + txd->cmd |=3D tx_desc_cmd_fcs; + txd->cmd |=3D (mbuf->ol_flags & PKT_TX_IP_CKSUM) ? tx_desc_cmd_ipv4 : 0; + /* L4 csum requested */ + txd->cmd |=3D (mbuf->ol_flags & PKT_TX_L4_MASK) ? tx_desc_cmd_l4cs : 0; + txd->cmd |=3D tx_cmd; +} + +static inline void +atl_xmit_pkt(struct aq_hw_s *hw, struct atl_tx_queue *txq, + struct rte_mbuf *tx_pkt) +{ + uint32_t pay_len =3D 0; + int tail =3D 0; + struct atl_tx_entry *tx_entry; + uint64_t buf_dma_addr; + struct rte_mbuf *m_seg; + union hw_atl_txc_s *txc =3D NULL; + struct hw_atl_txd_s *txd =3D NULL; + u32 tx_cmd =3D 0U; + int desc_count =3D 0; + + PMD_INIT_FUNC_TRACE(); + + tail =3D txq->tx_tail; + + txc =3D (union hw_atl_txc_s *)&txq->hw_ring[tail]; + + txc->flags1 =3D 0U; + txc->flags2 =3D 0U; + + tx_cmd =3D atl_tso_setup(tx_pkt, txc); + + if (tx_cmd) { + /* We've consumed the first desc, adjust counters */ + tail =3D (tail + 1) % txq->nb_tx_desc; + txq->tx_tail =3D tail; + txq->tx_free -=3D 1; + + txd =3D &txq->hw_ring[tail]; + txd->flags =3D 0U; + } else { + txd =3D (struct hw_atl_txd_s *)txc; + } + + txd->ct_en =3D !!tx_cmd; + + txd->type =3D tx_desc_type_desc; + + atl_setup_csum_offload(tx_pkt, txd, tx_cmd); + + if (tx_cmd) + txd->ct_idx =3D 0; + + pay_len =3D tx_pkt->pkt_len; + + txd->pay_len =3D pay_len; + + for (m_seg =3D tx_pkt; m_seg; m_seg =3D m_seg->next) { + if (desc_count > 0) { + txd =3D &txq->hw_ring[tail]; + txd->flags =3D 0U; + } + + buf_dma_addr =3D rte_mbuf_data_iova(m_seg); + txd->buf_addr =3D rte_cpu_to_le_64(buf_dma_addr); + + txd->type =3D tx_desc_type_desc; + txd->len =3D m_seg->data_len; + txd->pay_len =3D pay_len; + + /* Store mbuf for freeing later */ + tx_entry =3D &txq->sw_ring[tail]; + + if (tx_entry->mbuf) + rte_pktmbuf_free_seg(tx_entry->mbuf); + tx_entry->mbuf =3D m_seg; + + tail =3D (tail + 1) % txq->nb_tx_desc; + + desc_count++; + } + + // Last descriptor requires EOP and WB + txd->eop =3D 1U; + txd->cmd |=3D tx_desc_cmd_wb; + + hw_atl_b0_hw_tx_ring_tail_update(hw, tail, txq->queue_id); + + txq->tx_tail =3D tail; + + txq->tx_free -=3D desc_count; +} =20 uint16_t -atl_xmit_pkts(void *tx_queue __rte_unused, - struct rte_mbuf **tx_pkts __rte_unused, - uint16_t nb_pkts __rte_unused) +atl_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { - return 0; + struct rte_eth_dev *dev =3D NULL; + struct aq_hw_s *hw =3D NULL; + struct atl_tx_queue *txq =3D tx_queue; + struct rte_mbuf *tx_pkt; + uint16_t nb_tx; + + dev =3D &rte_eth_devices[txq->port_id]; + hw =3D ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private); + + PMD_TX_LOG(DEBUG, + "port %d txq %d pkts: %d tx_free=3D%d tx_tail=3D%d tx_head=3D%d", + txq->port_id, txq->queue_id, nb_pkts, txq->tx_free, + txq->tx_tail, txq->tx_head); + + for (nb_tx =3D 0; nb_tx < nb_pkts; nb_tx++) { + tx_pkt =3D *tx_pkts++; + + /* Clean Tx queue if needed */ + if (txq->tx_free < txq->tx_free_thresh) + atl_xmit_cleanup(txq); + + /* Check if we have enough free descriptors */ + if (txq->tx_free < tx_pkt->nb_segs) + break; + + /* check mbuf is valid */ + if ((tx_pkt->nb_segs =3D=3D 0) || + ((tx_pkt->nb_segs > 1) && (tx_pkt->next =3D=3D NULL))) + break; + + /* Send the packet */ + atl_xmit_pkt(hw, txq, tx_pkt); + } + + PMD_TX_LOG(DEBUG, "atl_xmit_pkts %d transmitted", nb_tx); + + return nb_tx; } =20 --=20 2.7.4