From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM05-CO1-obe.outbound.protection.outlook.com (mail-eopbgr720042.outbound.protection.outlook.com [40.107.72.42]) by dpdk.org (Postfix) with ESMTP id 915F51B5A2 for ; Fri, 12 Oct 2018 13:09:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=AQUANTIA1COM.onmicrosoft.com; s=selector1-aquantia-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8fkWLzvEWCl2L/FQmmGBLOE3qj4jx+lnbtGyA5NolLY=; b=jaELDJebDfzv/6XZuGkvItSVc7dxR0oTLjReNDA5mKFmfS48sKJaUvJQZ8mhyUmPyw6bugO3xhiCi3372dabzQCKwnDx0FgvaEmN5BHJ7bZewEpKULQcO/P7JawwMCmrLe81vAy75xUg81HaTlGdsRav8/N9QVqm8DkdiEoMOL4= Received: from BY1PR0701MB1660.namprd07.prod.outlook.com (10.162.110.22) by BY1PR0701MB1879.namprd07.prod.outlook.com (10.162.143.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1207.26; Fri, 12 Oct 2018 11:09:25 +0000 Received: from BY1PR0701MB1660.namprd07.prod.outlook.com ([fe80::346d:e756:e70e:17]) by BY1PR0701MB1660.namprd07.prod.outlook.com ([fe80::346d:e756:e70e:17%3]) with mapi id 15.20.1228.020; Fri, 12 Oct 2018 11:09:25 +0000 From: Igor Russkikh To: "dev@dpdk.org" CC: Pavel Belous , Igor Russkikh , "ferruh.yigit@intel.com" Thread-Topic: [PATCH v6 09/22] net/atlantic: implement Rx path Thread-Index: AQHUYhwIH/WQy00dhkiO4U249x0UGw== Date: Fri, 12 Oct 2018 11:09:25 +0000 Message-ID: <81d854b0821545151f493db80650250887ba1ed4.1539338074.git.igor.russkikh@aquantia.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: AM5PR0701CA0008.eurprd07.prod.outlook.com (2603:10a6:203:51::18) To BY1PR0701MB1660.namprd07.prod.outlook.com (2a01:111:e400:522a::22) authentication-results: spf=none (sender IP is ) smtp.mailfrom=Igor.Russkikh@aquantia.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [95.79.108.179] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BY1PR0701MB1879; 6:3CdOFq9BoWrrNAGZe9jKb8xhvraKDrCnylUIkHiDnvkpnhw0610kh62jqMteP6tkpzt8DpV3LFhIHIG6WVn/DYx3umg6oDo4JL16hHtN9FbU02cj3YGBzlkUqcKdz3753s/fgjBrFc58jJRqYb0N7IPrilEZqdHtTCl/3DEcYyyZTCeFYIg0GlvWebW8fFuqs7bGNxaFDF5veBfJFHczXxGxu6NEizEo91PlBAHHt4DiBLR9WNI0slsmcIJWP/vogpcbeg1+QZGD760Ybz5cHSqKOSOGPE5IsDgwpG8VrvoAatetxZnA+uMI3VlyY1yZooc/0/zTjCnY2RIlt2dbatX9D4xBaBcZQaDFrSyNKI9jJHP3R2E/WoDUplqlhJ+IQngV75AvvLG1yU4YRK3FcZM0IM7Ox2M6j71x9GPx3borKzOwbVS618fMpew8LBbmJaCCbAFpA9EF3PVj64fOHxpHI3rpFMpPX1i8WrDWzQM=; 5:NLnZUZCuODrqcq3kzvwJuFTLKLBCOqglfTmIvGf8omOIS0xTh1QBe4+p52LzL8yWz6M+JqKyjHtaPL+ddBnRRbUx0eNlhzSWt3w8NHPtg0WW55BvP9tS1Y9CXY2RRVz6DO4AHrERIKfmAhY8OmPxu2F9ByMwz9+e+DAtvl/zBbA=; 7:XzqR2IryLJ6HfAVpajbFlRhjRyzCZtnfZcsxxwk3tBRJXy05crMx59x/jSJz64UBC+OD2Ln7zOyT/o3MOCGdFNi2LMFKAq8PbQjL1rTCTrsET/FjUUU4YjMzXIvaV7okvBYXNlbUGzQb/Ki6P0G1VfwJ1nGqOFtZx/c/H6U4KvF+ADIgHOHLSrg603Wi/bEikOimnflB9Sz+V3zw1cFRhu/VAl6J2x7elW8ExaKZps/UOkx2XAjCW/WjJrJNn6nW x-ms-office365-filtering-correlation-id: 35cf0ffd-46d3-4fe7-59fa-08d630332b18 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:BY1PR0701MB1879; x-ms-traffictypediagnostic: BY1PR0701MB1879: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231355)(944501410)(52105095)(149066)(150057)(6041310)(20161123558120)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123560045)(201708071742011)(7699051)(76991067); SRVR:BY1PR0701MB1879; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1879; x-forefront-prvs: 0823A5777B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(1496009)(346002)(376002)(366004)(136003)(39850400004)(396003)(189003)(199004)(6436002)(5640700003)(3846002)(68736007)(2900100001)(478600001)(6486002)(14454004)(2906002)(72206003)(71190400001)(71200400001)(26005)(2616005)(44832011)(102836004)(4744004)(446003)(486006)(256004)(118296001)(186003)(105586002)(2351001)(6116002)(106356001)(6512007)(97736004)(6916009)(5660300001)(54906003)(6506007)(386003)(53946003)(11346002)(476003)(14444005)(53936002)(5250100002)(2501003)(25786009)(8936002)(8676002)(4326008)(86362001)(305945005)(7736002)(99286004)(81166006)(1730700003)(81156014)(66066001)(52116002)(36756003)(316002)(76176011)(579004); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1879; H:BY1PR0701MB1660.namprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: aquantia.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: a7uOMeZvsV6hGexJw7v6xP4iolWCGEXF+23CO9HnRDS5VVINcPxJSEs4nkhX1aJ7+qfVL89zv4agEQ8LIrX1h8yfMO46A9M6p3bDkqY4vfLO0KTb7nXe7MxdKP6xAjw7d8VZSSFSkfe9Le5ctUxw2ijh2J3X6+CxE7K3sFYLAqsju/GeOgNEYiUUBOxiRZGVDkIYqzpZbkvbIhK/CS+7xpryjr6zqnIk6W4sQ9DUHDuyz0QWMTKzMHz8vum9ByP6g/qIGn+f9Fn8aEwNP2z948FHMKYBuSkfxFrcmtGeKiksx7BpeB05uCpH0dtmhjxhyiIwEHmC4tMTKkZIGJsunYk7lhbFZEhVVS6KYjxuXHw= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: aquantia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 35cf0ffd-46d3-4fe7-59fa-08d630332b18 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Oct 2018 11:09:25.0600 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 83e2e134-991c-4ede-8ced-34d47e38e6b1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1879 Subject: [dpdk-dev] [PATCH v6 09/22] net/atlantic: implement Rx path X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Oct 2018 11:09:28 -0000 Add implementation for RX datapath. Signed-off-by: Igor Russkikh Signed-off-by: Pavel Belous --- doc/guides/nics/atlantic.rst | 6 + doc/guides/nics/features/atlantic.ini | 5 + drivers/net/atlantic/Makefile | 2 +- drivers/net/atlantic/atl_ethdev.c | 79 ++++- drivers/net/atlantic/atl_ethdev.h | 19 + drivers/net/atlantic/atl_rxtx.c | 632 ++++++++++++++++++++++++++++++= +++- 6 files changed, 733 insertions(+), 10 deletions(-) diff --git a/doc/guides/nics/atlantic.rst b/doc/guides/nics/atlantic.rst index 21e52bb26625..a003ef5110c9 100644 --- a/doc/guides/nics/atlantic.rst +++ b/doc/guides/nics/atlantic.rst @@ -12,6 +12,10 @@ More information can be found at `Aquantia Official Webs= ite Supported features ^^^^^^^^^^^^^^^^^^ =20 +- Base L2 features +- Checksum offload +- Jumbo Frame upto 16K + Configuration Information ^^^^^^^^^^^^^^^^^^^^^^^^^ =20 @@ -29,6 +33,8 @@ Statistics MTU setting ~~~~~~~~~~~ =20 +Atlantic NIC supports up to 16K jumbo frame size + Supported Chipsets and NICs --------------------------- =20 diff --git a/doc/guides/nics/features/atlantic.ini b/doc/guides/nics/featur= es/atlantic.ini index c603b7bfcd42..374cf9fb1061 100644 --- a/doc/guides/nics/features/atlantic.ini +++ b/doc/guides/nics/features/atlantic.ini @@ -5,6 +5,11 @@ ; [Features] Queue start/stop =3D Y +Jumbo frame =3D Y +CRC offload =3D Y +L3 checksum offload =3D Y +L4 checksum offload =3D Y +Packet type parsing =3D Y FW version =3D Y Linux UIO =3D Y ARMv8 =3D Y diff --git a/drivers/net/atlantic/Makefile b/drivers/net/atlantic/Makefile index b88da362146d..62dcdbffa69c 100644 --- a/drivers/net/atlantic/Makefile +++ b/drivers/net/atlantic/Makefile @@ -15,7 +15,7 @@ EXPORT_MAP :=3D rte_pmd_atlantic_version.map =20 LIBABIVER :=3D 1 =20 -LDLIBS +=3D -lrte_eal +LDLIBS +=3D -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS +=3D -lrte_ethdev -lrte_net LDLIBS +=3D -lrte_bus_pci =20 diff --git a/drivers/net/atlantic/atl_ethdev.c b/drivers/net/atlantic/atl_e= thdev.c index 870eafba3e13..521376d1398c 100644 --- a/drivers/net/atlantic/atl_ethdev.c +++ b/drivers/net/atlantic/atl_ethdev.c @@ -27,6 +27,7 @@ static int atl_fw_version_get(struct rte_eth_dev *dev, ch= ar *fw_version, static void atl_dev_info_get(struct rte_eth_dev *dev, struct rte_eth_dev_info *dev_info); =20 +static const uint32_t *atl_dev_supported_ptypes_get(struct rte_eth_dev *de= v); =20 static int eth_atl_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct rte_pci_device *pci_dev); @@ -75,6 +76,18 @@ static struct rte_pci_driver rte_atl_pmd =3D { .remove =3D eth_atl_pci_remove, }; =20 +#define ATL_RX_OFFLOADS (DEV_RX_OFFLOAD_VLAN_STRIP \ + | DEV_RX_OFFLOAD_IPV4_CKSUM \ + | DEV_RX_OFFLOAD_UDP_CKSUM \ + | DEV_RX_OFFLOAD_TCP_CKSUM \ + | DEV_RX_OFFLOAD_JUMBO_FRAME) + +static const struct rte_eth_desc_lim rx_desc_lim =3D { + .nb_max =3D ATL_MAX_RING_DESC, + .nb_min =3D ATL_MIN_RING_DESC, + .nb_align =3D ATL_RXD_ALIGN, +}; + static const struct eth_dev_ops atl_eth_dev_ops =3D { .dev_configure =3D atl_dev_configure, .dev_start =3D atl_dev_start, @@ -84,6 +97,13 @@ static const struct eth_dev_ops atl_eth_dev_ops =3D { =20 .fw_version_get =3D atl_fw_version_get, .dev_infos_get =3D atl_dev_info_get, + .dev_supported_ptypes_get =3D atl_dev_supported_ptypes_get, + + /* Queue Control */ + .rx_queue_start =3D atl_rx_queue_start, + .rx_queue_stop =3D atl_rx_queue_stop, + .rx_queue_setup =3D atl_rx_queue_setup, + .rx_queue_release =3D atl_rx_queue_release, }; =20 static inline int32_t @@ -220,7 +240,7 @@ atl_dev_start(struct rte_eth_dev *dev) err =3D atl_rx_init(dev); if (err) { PMD_INIT_LOG(ERR, "Unable to initialize RX hardware"); - return -EIO; + goto error; } =20 PMD_INIT_LOG(DEBUG, "FW version: %u.%u.%u", @@ -229,7 +249,17 @@ atl_dev_start(struct rte_eth_dev *dev) hw->fw_ver_actual & 0xFFFF); PMD_INIT_LOG(DEBUG, "Driver version: %s", ATL_PMD_DRIVER_VERSION); =20 - return err; + err =3D atl_start_queues(dev); + if (err < 0) { + PMD_INIT_LOG(ERR, "Unable to start rxtx queues"); + goto error; + } + + return 0; + +error: + atl_stop_queues(dev); + return -EIO; } =20 /* @@ -244,6 +274,12 @@ atl_dev_stop(struct rte_eth_dev *dev) /* reset the NIC */ atl_reset_hw(hw); hw->adapter_stopped =3D 1; + + atl_stop_queues(dev); + + /* Clear stored conf */ + dev->data->scattered_rx =3D 0; + dev->data->lro =3D 0; } =20 /* @@ -255,6 +291,8 @@ atl_dev_close(struct rte_eth_dev *dev) PMD_INIT_FUNC_TRACE(); =20 atl_dev_stop(dev); + + atl_free_queues(dev); } =20 static int @@ -298,14 +336,47 @@ atl_dev_info_get(struct rte_eth_dev *dev, struct rte_= eth_dev_info *dev_info) { struct rte_pci_device *pci_dev =3D RTE_ETH_DEV_TO_PCI(dev); =20 - dev_info->max_rx_queues =3D 0; - dev_info->max_rx_queues =3D 0; + dev_info->max_rx_queues =3D AQ_HW_MAX_RX_QUEUES; + dev_info->max_tx_queues =3D AQ_HW_MAX_TX_QUEUES; =20 + dev_info->min_rx_bufsize =3D 1024; + dev_info->max_rx_pktlen =3D HW_ATL_B0_MTU_JUMBO; + dev_info->max_mac_addrs =3D HW_ATL_B0_MAC_MAX; dev_info->max_vfs =3D pci_dev->max_vfs; =20 dev_info->max_hash_mac_addrs =3D 0; dev_info->max_vmdq_pools =3D 0; dev_info->vmdq_queue_num =3D 0; + + dev_info->rx_offload_capa =3D ATL_RX_OFFLOADS; + + dev_info->default_rxconf =3D (struct rte_eth_rxconf) { + .rx_free_thresh =3D ATL_DEFAULT_RX_FREE_THRESH, + }; + + dev_info->rx_desc_lim =3D rx_desc_lim; +} + +static const uint32_t * +atl_dev_supported_ptypes_get(struct rte_eth_dev *dev) +{ + static const uint32_t ptypes[] =3D { + RTE_PTYPE_L2_ETHER, + RTE_PTYPE_L2_ETHER_ARP, + RTE_PTYPE_L2_ETHER_VLAN, + RTE_PTYPE_L3_IPV4, + RTE_PTYPE_L3_IPV6, + RTE_PTYPE_L4_TCP, + RTE_PTYPE_L4_UDP, + RTE_PTYPE_L4_SCTP, + RTE_PTYPE_L4_ICMP, + RTE_PTYPE_UNKNOWN + }; + + if (dev->rx_pkt_burst =3D=3D atl_recv_pkts) + return ptypes; + + return NULL; } =20 RTE_PMD_REGISTER_PCI(net_atlantic, rte_atl_pmd); diff --git a/drivers/net/atlantic/atl_ethdev.h b/drivers/net/atlantic/atl_e= thdev.h index 990e8e4e9978..a9a9fc8fe7ff 100644 --- a/drivers/net/atlantic/atl_ethdev.h +++ b/drivers/net/atlantic/atl_ethdev.h @@ -13,6 +13,10 @@ #define ATL_DEV_PRIVATE_TO_HW(adapter) \ (&((struct atl_adapter *)adapter)->hw) =20 +#define ATL_DEV_TO_ADAPTER(dev) \ + ((struct atl_adapter *)(dev)->data->dev_private) + + /* * Structure to store private data for each driver instance (for each port= ). */ @@ -24,9 +28,24 @@ struct atl_adapter { /* * RX/TX function prototypes */ +void atl_rx_queue_release(void *rxq); + +int atl_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id, + uint16_t nb_rx_desc, unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf, + struct rte_mempool *mb_pool); + int atl_rx_init(struct rte_eth_dev *dev); int atl_tx_init(struct rte_eth_dev *dev); =20 +int atl_start_queues(struct rte_eth_dev *dev); +int atl_stop_queues(struct rte_eth_dev *dev); +void atl_free_queues(struct rte_eth_dev *dev); + +int atl_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id); +int atl_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id); + + uint16_t atl_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); =20 diff --git a/drivers/net/atlantic/atl_rxtx.c b/drivers/net/atlantic/atl_rxt= x.c index 0fbd93038075..61dbb83b4572 100644 --- a/drivers/net/atlantic/atl_rxtx.c +++ b/drivers/net/atlantic/atl_rxtx.c @@ -2,7 +2,150 @@ * Copyright(c) 2018 Aquantia Corporation */ =20 +#include +#include + #include "atl_ethdev.h" +#include "atl_hw_regs.h" + +#include "atl_logs.h" +#include "hw_atl/hw_atl_llh.h" +#include "hw_atl/hw_atl_b0.h" +#include "hw_atl/hw_atl_b0_internal.h" + +/** + * Structure associated with each descriptor of the RX ring of a RX queue. + */ +struct atl_rx_entry { + struct rte_mbuf *mbuf; +}; + +/** + * Structure associated with each RX queue. + */ +struct atl_rx_queue { + struct rte_mempool *mb_pool; + struct hw_atl_rxd_s *hw_ring; + uint64_t hw_ring_phys_addr; + struct atl_rx_entry *sw_ring; + uint16_t nb_rx_desc; + uint16_t rx_tail; + uint16_t nb_rx_hold; + uint16_t rx_free_thresh; + uint16_t queue_id; + uint16_t port_id; + uint16_t buff_size; + bool l3_csum_enabled; + bool l4_csum_enabled; +}; + +static inline void +atl_reset_rx_queue(struct atl_rx_queue *rxq) +{ + struct hw_atl_rxd_s *rxd =3D NULL; + int i; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < rxq->nb_rx_desc; i++) { + rxd =3D (struct hw_atl_rxd_s *)&rxq->hw_ring[i]; + rxd->buf_addr =3D 0; + rxd->hdr_addr =3D 0; + } + + rxq->rx_tail =3D 0; +} + +int +atl_rx_queue_setup(struct rte_eth_dev *dev, uint16_t rx_queue_id, + uint16_t nb_rx_desc, unsigned int socket_id, + const struct rte_eth_rxconf *rx_conf, + struct rte_mempool *mb_pool) +{ + struct atl_rx_queue *rxq; + const struct rte_memzone *mz; + + PMD_INIT_FUNC_TRACE(); + + /* make sure a valid number of descriptors have been requested */ + if (nb_rx_desc < AQ_HW_MIN_RX_RING_SIZE || + nb_rx_desc > AQ_HW_MAX_RX_RING_SIZE) { + PMD_INIT_LOG(ERR, "Number of Rx descriptors must be " + "less than or equal to %d, " + "greater than or equal to %d", AQ_HW_MAX_RX_RING_SIZE, + AQ_HW_MIN_RX_RING_SIZE); + return -EINVAL; + } + + /* + * if this queue existed already, free the associated memory. The + * queue cannot be reused in case we need to allocate memory on + * different socket than was previously used. + */ + if (dev->data->rx_queues[rx_queue_id] !=3D NULL) { + atl_rx_queue_release(dev->data->rx_queues[rx_queue_id]); + dev->data->rx_queues[rx_queue_id] =3D NULL; + } + + /* allocate memory for the queue structure */ + rxq =3D rte_zmalloc_socket("atlantic Rx queue", sizeof(*rxq), + RTE_CACHE_LINE_SIZE, socket_id); + if (rxq =3D=3D NULL) { + PMD_INIT_LOG(ERR, "Cannot allocate queue structure"); + return -ENOMEM; + } + + /* setup queue */ + rxq->mb_pool =3D mb_pool; + rxq->nb_rx_desc =3D nb_rx_desc; + rxq->port_id =3D dev->data->port_id; + rxq->queue_id =3D rx_queue_id; + rxq->rx_free_thresh =3D rx_conf->rx_free_thresh; + + rxq->l3_csum_enabled =3D dev->data->dev_conf.rxmode.offloads & + DEV_RX_OFFLOAD_IPV4_CKSUM; + rxq->l4_csum_enabled =3D dev->data->dev_conf.rxmode.offloads & + (DEV_RX_OFFLOAD_UDP_CKSUM | DEV_RX_OFFLOAD_TCP_CKSUM); + if (dev->data->dev_conf.rxmode.offloads & DEV_RX_OFFLOAD_KEEP_CRC) + PMD_DRV_LOG(ERR, "PMD does not support KEEP_CRC offload"); + + /* allocate memory for the software ring */ + rxq->sw_ring =3D rte_zmalloc_socket("atlantic sw rx ring", + nb_rx_desc * sizeof(struct atl_rx_entry), + RTE_CACHE_LINE_SIZE, socket_id); + if (rxq->sw_ring =3D=3D NULL) { + PMD_INIT_LOG(ERR, + "Port %d: Cannot allocate software ring for queue %d", + rxq->port_id, rxq->queue_id); + rte_free(rxq); + return -ENOMEM; + } + + /* + * allocate memory for the hardware descriptor ring. A memzone large + * enough to hold the maximum ring size is requested to allow for + * resizing in later calls to the queue setup function. + */ + mz =3D rte_eth_dma_zone_reserve(dev, "rx hw_ring", rx_queue_id, + HW_ATL_B0_MAX_RXD * + sizeof(struct hw_atl_rxd_s), + 128, socket_id); + if (mz =3D=3D NULL) { + PMD_INIT_LOG(ERR, + "Port %d: Cannot allocate hardware ring for queue %d", + rxq->port_id, rxq->queue_id); + rte_free(rxq->sw_ring); + rte_free(rxq); + return -ENOMEM; + } + rxq->hw_ring =3D mz->addr; + rxq->hw_ring_phys_addr =3D mz->iova; + + atl_reset_rx_queue(rxq); + + dev->data->rx_queues[rx_queue_id] =3D rxq; + return 0; +} =20 int atl_tx_init(struct rte_eth_dev *eth_dev __rte_unused) @@ -11,11 +154,177 @@ atl_tx_init(struct rte_eth_dev *eth_dev __rte_unused) } =20 int -atl_rx_init(struct rte_eth_dev *eth_dev __rte_unused) +atl_rx_init(struct rte_eth_dev *eth_dev) { + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private); + struct atl_rx_queue *rxq; + uint64_t base_addr =3D 0; + int i =3D 0; + int err =3D 0; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < eth_dev->data->nb_rx_queues; i++) { + rxq =3D eth_dev->data->rx_queues[i]; + base_addr =3D rxq->hw_ring_phys_addr; + + /* Take requested pool mbuf size and adapt + * descriptor buffer to best fit + */ + int buff_size =3D rte_pktmbuf_data_room_size(rxq->mb_pool) - + RTE_PKTMBUF_HEADROOM; + + buff_size =3D RTE_ALIGN_FLOOR(buff_size, 1024); + if (buff_size > HW_ATL_B0_RXD_BUF_SIZE_MAX) { + PMD_INIT_LOG(WARNING, + "Port %d queue %d: mem pool buff size is too big\n", + rxq->port_id, rxq->queue_id); + buff_size =3D HW_ATL_B0_RXD_BUF_SIZE_MAX; + } + if (buff_size < 1024) { + PMD_INIT_LOG(ERR, + "Port %d queue %d: mem pool buff size is too small\n", + rxq->port_id, rxq->queue_id); + return -EINVAL; + } + rxq->buff_size =3D buff_size; + + err =3D hw_atl_b0_hw_ring_rx_init(hw, base_addr, rxq->queue_id, + rxq->nb_rx_desc, buff_size, 0, + rxq->port_id); + + if (err) { + PMD_INIT_LOG(ERR, "Port %d: Cannot init RX queue %d", + rxq->port_id, rxq->queue_id); + break; + } + } + + return err; +} + +static int +atl_alloc_rx_queue_mbufs(struct atl_rx_queue *rxq) +{ + struct atl_rx_entry *rx_entry =3D rxq->sw_ring; + struct hw_atl_rxd_s *rxd; + uint64_t dma_addr =3D 0; + uint32_t i =3D 0; + + PMD_INIT_FUNC_TRACE(); + + /* fill Rx ring */ + for (i =3D 0; i < rxq->nb_rx_desc; i++) { + struct rte_mbuf *mbuf =3D rte_mbuf_raw_alloc(rxq->mb_pool); + + if (mbuf =3D=3D NULL) { + PMD_INIT_LOG(ERR, + "Port %d: mbuf alloc failed for rx queue %d", + rxq->port_id, rxq->queue_id); + return -ENOMEM; + } + + mbuf->data_off =3D RTE_PKTMBUF_HEADROOM; + mbuf->port =3D rxq->port_id; + + dma_addr =3D rte_cpu_to_le_64(rte_mbuf_data_iova_default(mbuf)); + rxd =3D (struct hw_atl_rxd_s *)&rxq->hw_ring[i]; + rxd->buf_addr =3D dma_addr; + rxd->hdr_addr =3D 0; + rx_entry[i].mbuf =3D mbuf; + } + return 0; } =20 +static void +atl_rx_queue_release_mbufs(struct atl_rx_queue *rxq) +{ + int i; + + PMD_INIT_FUNC_TRACE(); + + if (rxq->sw_ring !=3D NULL) { + for (i =3D 0; i < rxq->nb_rx_desc; i++) { + if (rxq->sw_ring[i].mbuf !=3D NULL) { + rte_pktmbuf_free_seg(rxq->sw_ring[i].mbuf); + rxq->sw_ring[i].mbuf =3D NULL; + } + } + } +} + +int +atl_rx_queue_start(struct rte_eth_dev *dev, uint16_t rx_queue_id) +{ + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct atl_rx_queue *rxq =3D NULL; + + PMD_INIT_FUNC_TRACE(); + + if (rx_queue_id < dev->data->nb_rx_queues) { + rxq =3D dev->data->rx_queues[rx_queue_id]; + + if (atl_alloc_rx_queue_mbufs(rxq) !=3D 0) { + PMD_INIT_LOG(ERR, + "Port %d: Allocate mbufs for queue %d failed", + rxq->port_id, rxq->queue_id); + return -1; + } + + hw_atl_b0_hw_ring_rx_start(hw, rx_queue_id); + + rte_wmb(); + hw_atl_reg_rx_dma_desc_tail_ptr_set(hw, rxq->nb_rx_desc - 1, + rx_queue_id); + dev->data->rx_queue_state[rx_queue_id] =3D + RTE_ETH_QUEUE_STATE_STARTED; + } else { + return -1; + } + + return 0; +} + +int +atl_rx_queue_stop(struct rte_eth_dev *dev, uint16_t rx_queue_id) +{ + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct atl_rx_queue *rxq =3D NULL; + + PMD_INIT_FUNC_TRACE(); + + if (rx_queue_id < dev->data->nb_rx_queues) { + rxq =3D dev->data->rx_queues[rx_queue_id]; + + hw_atl_b0_hw_ring_rx_stop(hw, rx_queue_id); + + atl_rx_queue_release_mbufs(rxq); + atl_reset_rx_queue(rxq); + + dev->data->rx_queue_state[rx_queue_id] =3D + RTE_ETH_QUEUE_STATE_STOPPED; + } else { + return -1; + } + + return 0; +} + +void +atl_rx_queue_release(void *rx_queue) +{ + PMD_INIT_FUNC_TRACE(); + + if (rx_queue !=3D NULL) { + struct atl_rx_queue *rxq =3D (struct atl_rx_queue *)rx_queue; + + atl_rx_queue_release_mbufs(rxq); + rte_free(rxq->sw_ring); + rte_free(rxq); + } +} + uint16_t atl_prep_pkts(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts __rte_unused, @@ -24,14 +333,327 @@ atl_prep_pkts(void *tx_queue __rte_unused, return 0; } =20 -uint16_t -atl_recv_pkts(void *rx_queue __rte_unused, - struct rte_mbuf **rx_pkts __rte_unused, - uint16_t nb_pkts __rte_unused) +void +atl_free_queues(struct rte_eth_dev *dev) +{ + unsigned int i; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { + atl_rx_queue_release(dev->data->rx_queues[i]); + dev->data->rx_queues[i] =3D 0; + } + dev->data->nb_rx_queues =3D 0; +} + +int +atl_start_queues(struct rte_eth_dev *dev) { + int i; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { + if (atl_rx_queue_start(dev, i) !=3D 0) { + PMD_DRV_LOG(ERR, + "Port %d: Start Rx queue %d failed", + dev->data->port_id, i); + return -1; + } + } + return 0; } =20 +int +atl_stop_queues(struct rte_eth_dev *dev) +{ + int i; + + PMD_INIT_FUNC_TRACE(); + + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { + if (atl_rx_queue_stop(dev, i) !=3D 0) { + PMD_DRV_LOG(ERR, + "Port %d: Stop Rx queue %d failed", + dev->data->port_id, i); + return -1; + } + } + + return 0; +} + +static uint64_t +atl_desc_to_offload_flags(struct atl_rx_queue *rxq, + struct hw_atl_rxd_wb_s *rxd_wb) +{ + uint64_t mbuf_flags =3D 0; + + PMD_INIT_FUNC_TRACE(); + + /* IPv4 ? */ + if (rxq->l3_csum_enabled && ((rxd_wb->pkt_type & 0x3) =3D=3D 0)) { + /* IPv4 csum error ? */ + if (rxd_wb->rx_stat & BIT(1)) + mbuf_flags |=3D PKT_RX_IP_CKSUM_BAD; + else + mbuf_flags |=3D PKT_RX_IP_CKSUM_GOOD; + } else { + mbuf_flags |=3D PKT_RX_IP_CKSUM_UNKNOWN; + } + + /* CSUM calculated ? */ + if (rxq->l4_csum_enabled && (rxd_wb->rx_stat & BIT(3))) { + if (rxd_wb->rx_stat & BIT(2)) + mbuf_flags |=3D PKT_RX_L4_CKSUM_BAD; + else + mbuf_flags |=3D PKT_RX_L4_CKSUM_GOOD; + } else { + mbuf_flags |=3D PKT_RX_L4_CKSUM_UNKNOWN; + } + + return mbuf_flags; +} + +static uint32_t +atl_desc_to_pkt_type(struct hw_atl_rxd_wb_s *rxd_wb) +{ + uint32_t type =3D RTE_PTYPE_UNKNOWN; + uint16_t l2_l3_type =3D rxd_wb->pkt_type & 0x3; + uint16_t l4_type =3D (rxd_wb->pkt_type & 0x1C) >> 2; + + switch (l2_l3_type) { + case 0: + type =3D RTE_PTYPE_L3_IPV4; + break; + case 1: + type =3D RTE_PTYPE_L3_IPV6; + break; + case 2: + type =3D RTE_PTYPE_L2_ETHER; + break; + case 3: + type =3D RTE_PTYPE_L2_ETHER_ARP; + break; + } + + switch (l4_type) { + case 0: + type |=3D RTE_PTYPE_L4_TCP; + break; + case 1: + type |=3D RTE_PTYPE_L4_UDP; + break; + case 2: + type |=3D RTE_PTYPE_L4_SCTP; + break; + case 3: + type |=3D RTE_PTYPE_L4_ICMP; + break; + } + + if (rxd_wb->pkt_type & BIT(5)) + type |=3D RTE_PTYPE_L2_ETHER_VLAN; + + return type; +} + +uint16_t +atl_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) +{ + struct atl_rx_queue *rxq =3D (struct atl_rx_queue *)rx_queue; + struct rte_eth_dev *dev =3D &rte_eth_devices[rxq->port_id]; + struct atl_adapter *adapter =3D + ATL_DEV_TO_ADAPTER(&rte_eth_devices[rxq->port_id]); + struct aq_hw_s *hw =3D ATL_DEV_PRIVATE_TO_HW(adapter); + struct atl_rx_entry *sw_ring =3D rxq->sw_ring; + + struct rte_mbuf *new_mbuf; + struct rte_mbuf *rx_mbuf, *rx_mbuf_prev, *rx_mbuf_first; + struct atl_rx_entry *rx_entry; + uint16_t nb_rx =3D 0; + uint16_t nb_hold =3D 0; + struct hw_atl_rxd_wb_s rxd_wb; + struct hw_atl_rxd_s *rxd =3D NULL; + uint16_t tail =3D rxq->rx_tail; + uint64_t dma_addr; + uint16_t pkt_len =3D 0; + + while (nb_rx < nb_pkts) { + uint16_t eop_tail =3D tail; + + rxd =3D (struct hw_atl_rxd_s *)&rxq->hw_ring[tail]; + rxd_wb =3D *(struct hw_atl_rxd_wb_s *)rxd; + + if (!rxd_wb.dd) { /* RxD is not done */ + break; + } + + PMD_RX_LOG(ERR, "port_id=3D%u queue_id=3D%u tail=3D%u " + "eop=3D0x%x pkt_len=3D%u hash=3D0x%x hash_type=3D0x%x", + (unsigned int)rxq->port_id, + (unsigned int)rxq->queue_id, + (unsigned int)tail, (unsigned int)rxd_wb.eop, + (unsigned int)rte_le_to_cpu_16(rxd_wb.pkt_len), + rxd_wb.rss_hash, rxd_wb.rss_type); + + /* RxD is not done */ + if (!rxd_wb.eop) { + while (true) { + struct hw_atl_rxd_wb_s *eop_rxwbd; + + eop_tail =3D (eop_tail + 1) % rxq->nb_rx_desc; + eop_rxwbd =3D (struct hw_atl_rxd_wb_s *) + &rxq->hw_ring[eop_tail]; + if (!eop_rxwbd->dd) { + /* no EOP received yet */ + eop_tail =3D tail; + break; + } + if (eop_rxwbd->dd && eop_rxwbd->eop) + break; + } + /* No EOP in ring */ + if (eop_tail =3D=3D tail) + break; + } + rx_mbuf_prev =3D NULL; + rx_mbuf_first =3D NULL; + + /* Run through packet segments */ + while (true) { + new_mbuf =3D rte_mbuf_raw_alloc(rxq->mb_pool); + if (new_mbuf =3D=3D NULL) { + PMD_RX_LOG(ERR, + "RX mbuf alloc failed port_id=3D%u " + "queue_id=3D%u", (unsigned int)rxq->port_id, + (unsigned int)rxq->queue_id); + dev->data->rx_mbuf_alloc_failed++; + goto err_stop; + } + + nb_hold++; + rx_entry =3D &sw_ring[tail]; + + rx_mbuf =3D rx_entry->mbuf; + rx_entry->mbuf =3D new_mbuf; + dma_addr =3D rte_cpu_to_le_64( + rte_mbuf_data_iova_default(new_mbuf)); + + /* setup RX descriptor */ + rxd->hdr_addr =3D 0; + rxd->buf_addr =3D dma_addr; + + /* + * Initialize the returned mbuf. + * 1) setup generic mbuf fields: + * - number of segments, + * - next segment, + * - packet length, + * - RX port identifier. + * 2) integrate hardware offload data, if any: + * < - RSS flag & hash, + * - IP checksum flag, + * - VLAN TCI, if any, + * - error flags. + */ + pkt_len =3D (uint16_t)rte_le_to_cpu_16(rxd_wb.pkt_len); + rx_mbuf->data_off =3D RTE_PKTMBUF_HEADROOM; + rte_prefetch1((char *)rx_mbuf->buf_addr + + rx_mbuf->data_off); + rx_mbuf->nb_segs =3D 0; + rx_mbuf->next =3D NULL; + rx_mbuf->pkt_len =3D pkt_len; + rx_mbuf->data_len =3D pkt_len; + if (rxd_wb.eop) { + u16 remainder_len =3D pkt_len % rxq->buff_size; + if (!remainder_len) + remainder_len =3D rxq->buff_size; + rx_mbuf->data_len =3D remainder_len; + } else { + rx_mbuf->data_len =3D pkt_len > rxq->buff_size ? + rxq->buff_size : pkt_len; + } + rx_mbuf->port =3D rxq->port_id; + + rx_mbuf->hash.rss =3D rxd_wb.rss_hash; + + rx_mbuf->vlan_tci =3D rxd_wb.vlan; + + rx_mbuf->ol_flags =3D + atl_desc_to_offload_flags(rxq, &rxd_wb); + rx_mbuf->packet_type =3D atl_desc_to_pkt_type(&rxd_wb); + + if (!rx_mbuf_first) + rx_mbuf_first =3D rx_mbuf; + rx_mbuf_first->nb_segs++; + + if (rx_mbuf_prev) + rx_mbuf_prev->next =3D rx_mbuf; + rx_mbuf_prev =3D rx_mbuf; + + tail =3D (tail + 1) % rxq->nb_rx_desc; + /* Prefetch next mbufs */ + rte_prefetch0(sw_ring[tail].mbuf); + if ((tail & 0x3) =3D=3D 0) { + rte_prefetch0(&sw_ring[tail]); + rte_prefetch0(&sw_ring[tail]); + } + + /* filled mbuf_first */ + if (rxd_wb.eop) + break; + rxd =3D (struct hw_atl_rxd_s *)&rxq->hw_ring[tail]; + rxd_wb =3D *(struct hw_atl_rxd_wb_s *)rxd; + }; + + /* + * Store the mbuf address into the next entry of the array + * of returned packets. + */ + rx_pkts[nb_rx++] =3D rx_mbuf_first; + + PMD_RX_LOG(ERR, "add mbuf segs=3D%d pkt_len=3D%d", + rx_mbuf_first->nb_segs, + rx_mbuf_first->pkt_len); + } + +err_stop: + + rxq->rx_tail =3D tail; + + /* + * If the number of free RX descriptors is greater than the RX free + * threshold of the queue, advance the Receive Descriptor Tail (RDT) + * register. + * Update the RDT with the value of the last processed RX descriptor + * minus 1, to guarantee that the RDT register is never equal to the + * RDH register, which creates a "full" ring situtation from the + * hardware point of view... + */ + nb_hold =3D (uint16_t)(nb_hold + rxq->nb_rx_hold); + if (nb_hold > rxq->rx_free_thresh) { + PMD_RX_LOG(ERR, "port_id=3D%u queue_id=3D%u rx_tail=3D%u " + "nb_hold=3D%u nb_rx=3D%u", + (unsigned int)rxq->port_id, (unsigned int)rxq->queue_id, + (unsigned int)tail, (unsigned int)nb_hold, + (unsigned int)nb_rx); + tail =3D (uint16_t)((tail =3D=3D 0) ? + (rxq->nb_rx_desc - 1) : (tail - 1)); + + hw_atl_reg_rx_dma_desc_tail_ptr_set(hw, tail, rxq->queue_id); + + nb_hold =3D 0; + } + + rxq->nb_rx_hold =3D nb_hold; + + return nb_rx; +} + + uint16_t atl_xmit_pkts(void *tx_queue __rte_unused, struct rte_mbuf **tx_pkts __rte_unused, --=20 2.7.4