Re: [dpdk-dev] [PATCH v2 09/62] net/cnxk: add build infra and common probe

[Nithin Dabilpuram <nithind1988@gmail.com>]

On Wed, Jun 09, 2021 at 09:38:04AM +0800, Huisong Li wrote:
> 
> 在 2021/6/8 1:58, Nithin Dabilpuram 写道:
> > Add build infrastructure and common probe and remove for cnxk driver
> > which is used by both CN10K and CN9K SoC.
> > 
> > Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> > ---
> >   MAINTAINERS                           |   3 +
> >   doc/guides/nics/cnxk.rst              |  29 +++++
> >   doc/guides/nics/features/cnxk.ini     |   9 ++
> >   doc/guides/nics/features/cnxk_vec.ini |   9 ++
> >   doc/guides/nics/features/cnxk_vf.ini  |   9 ++
> >   doc/guides/nics/index.rst             |   1 +
> >   doc/guides/platform/cnxk.rst          |   3 +
> >   drivers/net/cnxk/cnxk_ethdev.c        | 219 ++++++++++++++++++++++++++++++++++
> >   drivers/net/cnxk/cnxk_ethdev.h        |  57 +++++++++
> >   drivers/net/cnxk/meson.build          |  21 ++++
> >   drivers/net/cnxk/version.map          |   3 +
> >   drivers/net/meson.build               |   1 +
> >   12 files changed, 364 insertions(+)
> >   create mode 100644 doc/guides/nics/cnxk.rst
> >   create mode 100644 doc/guides/nics/features/cnxk.ini
> >   create mode 100644 doc/guides/nics/features/cnxk_vec.ini
> >   create mode 100644 doc/guides/nics/features/cnxk_vf.ini
> >   create mode 100644 drivers/net/cnxk/cnxk_ethdev.c
> >   create mode 100644 drivers/net/cnxk/cnxk_ethdev.h
> >   create mode 100644 drivers/net/cnxk/meson.build
> >   create mode 100644 drivers/net/cnxk/version.map
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 5877a16..2be220e 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -746,6 +746,9 @@ M: Sunil Kumar Kori <skori@marvell.com>
> >   M: Satha Rao <skoteshwar@marvell.com>
> >   T: git://dpdk.org/next/dpdk-next-net-mrvl
> >   F: drivers/common/cnxk/
> > +F: drivers/net/cnxk/
> > +F: doc/guides/nics/cnxk.rst
> > +F: doc/guides/nics/features/cnxk*.ini
> >   F: doc/guides/platform/cnxk.rst
> >   Marvell mvpp2
> > diff --git a/doc/guides/nics/cnxk.rst b/doc/guides/nics/cnxk.rst
> > new file mode 100644
> > index 0000000..ca21842
> > --- /dev/null
> > +++ b/doc/guides/nics/cnxk.rst
> > @@ -0,0 +1,29 @@
> > +..  SPDX-License-Identifier: BSD-3-Clause
> > +    Copyright(C) 2021 Marvell.
> > +
> > +CNXK Poll Mode driver
> > +=====================
> > +
> > +The CNXK ETHDEV PMD (**librte_net_cnxk**) provides poll mode ethdev driver
> > +support for the inbuilt network device found in **Marvell OCTEON CN9K/CN10K**
> > +SoC family as well as for their virtual functions (VF) in SR-IOV context.
> > +
> > +More information can be found at `Marvell Official Website
> > +<https://www.marvell.com/embedded-processors/infrastructure-processors>`_.
> > +
> > +Features
> > +--------
> > +
> > +Features of the CNXK Ethdev PMD are:
> > +
> > +Prerequisites
> > +-------------
> > +
> > +See :doc:`../platform/cnxk` for setup information.
> > +
> > +
> > +Driver compilation and testing
> > +------------------------------
> > +
> > +Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>`
> > +for details.
> > diff --git a/doc/guides/nics/features/cnxk.ini b/doc/guides/nics/features/cnxk.ini
> > new file mode 100644
> > index 0000000..2c23464
> > --- /dev/null
> > +++ b/doc/guides/nics/features/cnxk.ini
> > @@ -0,0 +1,9 @@
> > +;
> > +; Supported features of the 'cnxk' network poll mode driver.
> > +;
> > +; Refer to default.ini for the full list of available PMD features.
> > +;
> > +[Features]
> > +Linux                = Y
> > +ARMv8                = Y
> > +Usage doc            = Y
> > diff --git a/doc/guides/nics/features/cnxk_vec.ini b/doc/guides/nics/features/cnxk_vec.ini
> > new file mode 100644
> > index 0000000..de78516
> > --- /dev/null
> > +++ b/doc/guides/nics/features/cnxk_vec.ini
> > @@ -0,0 +1,9 @@
> > +;
> > +; Supported features of the 'cnxk_vec' network poll mode driver.
> > +;
> > +; Refer to default.ini for the full list of available PMD features.
> > +;
> > +[Features]
> > +Linux                = Y
> > +ARMv8                = Y
> > +Usage doc            = Y
> > diff --git a/doc/guides/nics/features/cnxk_vf.ini b/doc/guides/nics/features/cnxk_vf.ini
> > new file mode 100644
> > index 0000000..9c96351
> > --- /dev/null
> > +++ b/doc/guides/nics/features/cnxk_vf.ini
> > @@ -0,0 +1,9 @@
> > +;
> > +; Supported features of the 'cnxk_vf' network poll mode driver.
> > +;
> > +; Refer to default.ini for the full list of available PMD features.
> > +;
> > +[Features]
> > +Linux                = Y
> > +ARMv8                = Y
> > +Usage doc            = Y
> > diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst
> > index 799697c..c1a04d9 100644
> > --- a/doc/guides/nics/index.rst
> > +++ b/doc/guides/nics/index.rst
> > @@ -19,6 +19,7 @@ Network Interface Controller Drivers
> >       axgbe
> >       bnx2x
> >       bnxt
> > +    cnxk
> >       cxgbe
> >       dpaa
> >       dpaa2
> > diff --git a/doc/guides/platform/cnxk.rst b/doc/guides/platform/cnxk.rst
> > index cebb3d0..b506c11 100644
> > --- a/doc/guides/platform/cnxk.rst
> > +++ b/doc/guides/platform/cnxk.rst
> > @@ -142,6 +142,9 @@ HW Offload Drivers
> >   This section lists dataplane H/W block(s) available in cnxk SoC.
> > +#. **Ethdev Driver**
> > +   See :doc:`../nics/cnxk` for NIX Ethdev driver information.
> > +
> >   #. **Mempool Driver**
> >      See :doc:`../mempool/cnxk` for NPA mempool driver information.
> > diff --git a/drivers/net/cnxk/cnxk_ethdev.c b/drivers/net/cnxk/cnxk_ethdev.c
> > new file mode 100644
> > index 0000000..6717410
> > --- /dev/null
> > +++ b/drivers/net/cnxk/cnxk_ethdev.c
> > @@ -0,0 +1,219 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(C) 2021 Marvell.
> > + */
> > +#include <cnxk_ethdev.h>
> > +
> > +/* CNXK platform independent eth dev ops */
> > +struct eth_dev_ops cnxk_eth_dev_ops;
> > +
> > +static int
> > +cnxk_eth_dev_init(struct rte_eth_dev *eth_dev)
> > +{
> > +	struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev);
> > +	struct roc_nix *nix = &dev->nix;
> > +	struct rte_pci_device *pci_dev;
> > +	int rc, max_entries;
> > +
> > +	eth_dev->dev_ops = &cnxk_eth_dev_ops;
> > +
> > +	/* For secondary processes, the primary has done all the work */
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return 0;
> > +
> > +	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
> > +	rte_eth_copy_pci_info(eth_dev, pci_dev);
> > +	eth_dev->data->dev_flags |= RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS;
> Hi,
> 
> It is recommended not to use this flag when add new driver. Queue stats will
> be moved to xstats and be filled by PMDs.
> 
> Please check the following patch:
> 
> commit f30e69b41f949cd4a9afb6ff39de196e661708e2
> Author: Ferruh Yigit <ferruh.yigit@intel.com>
> Date:   Wed Oct 14 03:26:47 2020 +0100
> 
>     ethdev: add device flag to bypass auto-filled queue xstats
> 
>     Queue stats are stored in 'struct rte_eth_stats' as array and array size
>     is defined by 'RTE_ETHDEV_QUEUE_STAT_CNTRS' compile time flag.
> 
>     As a result of technical board discussion, decided to remove the queue
>     statistics from 'struct rte_eth_stats' in the long term.
> 
>     Instead PMDs should represent the queue statistics via xstats, this
>     gives more flexibility on the number of the queues supported.
> 
>     Currently queue stats in the xstats are filled by ethdev layer, using
>     some basic stats, when queue stats removed from basic stats the
>     responsibility to fill the relevant xstats will be pushed to the PMDs.
> 
>     During the switch period, temporary 'RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS'
>     device flag is created. Initially all PMDs using xstats set this flag.
>     The PMDs implemented queue stats in the xstats should clear the flag.
> 
>     When all PMDs switch to the xstats for the queue stats, queue stats
>     related fields from 'struct rte_eth_stats' will be removed, as well as
>     'RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS' flag.
>     Later 'RTE_ETHDEV_QUEUE_STAT_CNTRS' compile time flag also can be
>     removed.
> 
>     Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>     Acked-by: Haiyue Wang <haiyue.wang@intel.com>
>     Acked-by: Xiao Wang <xiao.w.wang@intel.com>
>     Acked-by: Thomas Monjalon <thomas@monjalon.net>

Ack, will fix it in V3.
> 
> > +
> > +	/* Initialize base roc nix */
> > +	nix->pci_dev = pci_dev;
> > +	rc = roc_nix_dev_init(nix);
> > +	if (rc) {
> > +		plt_err("Failed to initialize roc nix rc=%d", rc);
> > +		goto error;
> > +	}
> > +
> > +	dev->eth_dev = eth_dev;
> > +
> > +	/* For vfs, returned max_entries will be 0. but to keep default mac
> > +	 * address, one entry must be allocated. so setting up to 1.
> > +	 */
> > +	if (roc_nix_is_vf_or_sdp(nix))
> > +		max_entries = 1;
> > +	else
> > +		max_entries = roc_nix_mac_max_entries_get(nix);
> > +
> > +	if (max_entries <= 0) {
> > +		plt_err("Failed to get max entries for mac addr");
> > +		rc = -ENOTSUP;
> > +		goto dev_fini;
> > +	}
> > +
> > +	eth_dev->data->mac_addrs =
> > +		rte_zmalloc("mac_addr", max_entries * RTE_ETHER_ADDR_LEN, 0);
> > +	if (eth_dev->data->mac_addrs == NULL) {
> > +		plt_err("Failed to allocate memory for mac addr");
> > +		rc = -ENOMEM;
> > +		goto dev_fini;
> > +	}
> > +
> > +	dev->max_mac_entries = max_entries;
> > +
> > +	/* Get mac address */
> > +	rc = roc_nix_npc_mac_addr_get(nix, dev->mac_addr);
> > +	if (rc) {
> > +		plt_err("Failed to get mac addr, rc=%d", rc);
> > +		goto free_mac_addrs;
> > +	}
> > +
> > +	/* Update the mac address */
> > +	memcpy(eth_dev->data->mac_addrs, dev->mac_addr, RTE_ETHER_ADDR_LEN);
> > +
> > +	if (!roc_nix_is_vf_or_sdp(nix)) {
> > +		/* Sync same MAC address to CGX/RPM table */
> > +		rc = roc_nix_mac_addr_set(nix, dev->mac_addr);
> > +		if (rc) {
> > +			plt_err("Failed to set mac addr, rc=%d", rc);
> > +			goto free_mac_addrs;
> > +		}
> > +	}
> > +
> > +	/* Initialize roc npc */
> > +	plt_nix_dbg("Port=%d pf=%d vf=%d ver=%s hwcap=0x%" PRIx64
> > +		    " rxoffload_capa=0x%" PRIx64 " txoffload_capa=0x%" PRIx64,
> > +		    eth_dev->data->port_id, roc_nix_get_pf(nix),
> > +		    roc_nix_get_vf(nix), CNXK_ETH_DEV_PMD_VERSION, dev->hwcap,
> > +		    dev->rx_offload_capa, dev->tx_offload_capa);
> > +	return 0;
> > +
> > +free_mac_addrs:
> > +	rte_free(eth_dev->data->mac_addrs);
> > +dev_fini:
> > +	roc_nix_dev_fini(nix);
> > +error:
> > +	plt_err("Failed to init nix eth_dev rc=%d", rc);
> > +	return rc;
> > +}
> > +
> > +static int
> > +cnxk_eth_dev_uninit(struct rte_eth_dev *eth_dev, bool mbox_close)
> > +{
> > +	struct cnxk_eth_dev *dev = cnxk_eth_pmd_priv(eth_dev);
> > +	const struct eth_dev_ops *dev_ops = eth_dev->dev_ops;
> > +	struct roc_nix *nix = &dev->nix;
> > +	int rc, i;
> > +
> > +	/* Nothing to be done for secondary processes */
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return 0;
> > +
> > +	roc_nix_npc_rx_ena_dis(nix, false);
> > +
> > +	/* Free up SQs */
> > +	for (i = 0; i < eth_dev->data->nb_tx_queues; i++) {
> > +		dev_ops->tx_queue_release(eth_dev->data->tx_queues[i]);
> > +		eth_dev->data->tx_queues[i] = NULL;
> > +	}
> > +	eth_dev->data->nb_tx_queues = 0;
> > +
> > +	/* Free up RQ's and CQ's */
> > +	for (i = 0; i < eth_dev->data->nb_rx_queues; i++) {
> > +		dev_ops->rx_queue_release(eth_dev->data->rx_queues[i]);
> > +		eth_dev->data->rx_queues[i] = NULL;
> > +	}
> > +	eth_dev->data->nb_rx_queues = 0;
> > +
> > +	/* Free tm resources */
> > +	roc_nix_tm_fini(nix);
> > +
> > +	/* Unregister queue irqs */
> > +	roc_nix_unregister_queue_irqs(nix);
> > +
> > +	/* Unregister cq irqs */
> > +	if (eth_dev->data->dev_conf.intr_conf.rxq)
> > +		roc_nix_unregister_cq_irqs(nix);
> > +
> > +	/* Free nix lf resources */
> > +	rc = roc_nix_lf_free(nix);
> > +	if (rc)
> > +		plt_err("Failed to free nix lf, rc=%d", rc);
> > +
> > +	rte_free(eth_dev->data->mac_addrs);
> > +	eth_dev->data->mac_addrs = NULL;
> > +
> > +	/* Check if mbox close is needed */
> > +	if (!mbox_close)
> > +		return 0;
> > +
> > +	rc = roc_nix_dev_fini(nix);
> > +	/* Can be freed later by PMD if NPA LF is in use */
> > +	if (rc == -EAGAIN) {
> > +		eth_dev->data->dev_private = NULL;
> > +		return 0;
> > +	} else if (rc) {
> > +		plt_err("Failed in nix dev fini, rc=%d", rc);
> > +	}
> > +
> > +	return rc;
> > +}
> > +
> > +int
> > +cnxk_nix_remove(struct rte_pci_device *pci_dev)
> > +{
> > +	struct rte_eth_dev *eth_dev;
> > +	struct roc_nix *nix;
> > +	int rc = -EINVAL;
> > +
> > +	eth_dev = rte_eth_dev_allocated(pci_dev->device.name);
> > +	if (eth_dev) {
> > +		/* Cleanup eth dev */
> > +		rc = cnxk_eth_dev_uninit(eth_dev, true);
> > +		if (rc)
> > +			return rc;
> > +
> > +		rte_eth_dev_release_port(eth_dev);
> > +	}
> > +
> > +	/* Nothing to be done for secondary processes */
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return 0;
> > +
> > +	/* Check if this device is hosting common resource */
> > +	nix = roc_idev_npa_nix_get();
> > +	if (nix->pci_dev != pci_dev)
> > +		return 0;
> > +
> > +	/* Try nix fini now */
> > +	rc = roc_nix_dev_fini(nix);
> > +	if (rc == -EAGAIN) {
> > +		plt_info("%s: common resource in use by other devices",
> > +			 pci_dev->name);
> > +		goto exit;
> > +	} else if (rc) {
> > +		plt_err("Failed in nix dev fini, rc=%d", rc);
> > +		goto exit;
> > +	}
> > +
> > +	/* Free device pointer as rte_ethdev does not have it anymore */
> > +	rte_free(nix);
> > +exit:
> > +	return rc;
> > +}
> > +
> > +int
> > +cnxk_nix_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)
> > +{
> > +	int rc;
> > +
> > +	RTE_SET_USED(pci_drv);
> > +
> > +	rc = rte_eth_dev_pci_generic_probe(pci_dev, sizeof(struct cnxk_eth_dev),
> > +					   cnxk_eth_dev_init);
> > +
> > +	/* On error on secondary, recheck if port exists in primary or
> > +	 * in mid of detach state.
> > +	 */
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY && rc)
> > +		if (!rte_eth_dev_allocated(pci_dev->device.name))
> > +			return 0;
> > +	return rc;
> > +}
> > diff --git a/drivers/net/cnxk/cnxk_ethdev.h b/drivers/net/cnxk/cnxk_ethdev.h
> > new file mode 100644
> > index 0000000..0460d1e
> > --- /dev/null
> > +++ b/drivers/net/cnxk/cnxk_ethdev.h
> > @@ -0,0 +1,57 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(C) 2021 Marvell.
> > + */
> > +#ifndef __CNXK_ETHDEV_H__
> > +#define __CNXK_ETHDEV_H__
> > +
> > +#include <math.h>
> > +#include <stdint.h>
> > +
> > +#include <ethdev_driver.h>
> > +#include <ethdev_pci.h>
> > +
> > +#include "roc_api.h"
> > +
> > +#define CNXK_ETH_DEV_PMD_VERSION "1.0"
> > +
> > +struct cnxk_eth_dev {
> > +	/* ROC NIX */
> > +	struct roc_nix nix;
> > +
> > +	/* Max macfilter entries */
> > +	uint8_t max_mac_entries;
> > +
> > +	uint16_t flags;
> > +
> > +	/* Pointer back to rte */
> > +	struct rte_eth_dev *eth_dev;
> > +
> > +	/* HW capabilities / Limitations */
> > +	union {
> > +		uint64_t hwcap;
> > +	};
> > +
> > +	/* Rx and Tx offload capabilities */
> > +	uint64_t rx_offload_capa;
> > +	uint64_t tx_offload_capa;
> > +	uint32_t speed_capa;
> > +
> > +	/* Default mac address */
> > +	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
> > +};
> > +
> > +static inline struct cnxk_eth_dev *
> > +cnxk_eth_pmd_priv(struct rte_eth_dev *eth_dev)
> > +{
> > +	return eth_dev->data->dev_private;
> > +}
> > +
> > +/* Common ethdev ops */
> > +extern struct eth_dev_ops cnxk_eth_dev_ops;
> > +
> > +/* Ops */
> > +int cnxk_nix_probe(struct rte_pci_driver *pci_drv,
> > +		   struct rte_pci_device *pci_dev);
> > +int cnxk_nix_remove(struct rte_pci_device *pci_dev);
> > +
> > +#endif /* __CNXK_ETHDEV_H__ */
> > diff --git a/drivers/net/cnxk/meson.build b/drivers/net/cnxk/meson.build
> > new file mode 100644
> > index 0000000..77b2f18
> > --- /dev/null
> > +++ b/drivers/net/cnxk/meson.build
> > @@ -0,0 +1,21 @@
> > +# SPDX-License-Identifier: BSD-3-Clause
> > +# Copyright(C) 2021 Marvell.
> > +#
> > +
> > +if not dpdk_conf.get('RTE_ARCH_64')
> > +	build = false
> > +	reason = 'only supported on 64-bit'
> > +	subdir_done()
> > +endif
> > +
> > +sources = files('cnxk_ethdev.c')
> > +
> > +deps += ['bus_pci', 'cryptodev', 'eventdev', 'security']
> > +deps += ['common_cnxk', 'mempool_cnxk']
> > +
> > +extra_flags = ['-flax-vector-conversions', '-Wno-strict-aliasing']
> > +foreach flag: extra_flags
> > +	if cc.has_argument(flag)
> > +		cflags += flag
> > +	endif
> > +endforeach
> > diff --git a/drivers/net/cnxk/version.map b/drivers/net/cnxk/version.map
> > new file mode 100644
> > index 0000000..ee80c51
> > --- /dev/null
> > +++ b/drivers/net/cnxk/version.map
> > @@ -0,0 +1,3 @@
> > +INTERNAL {
> > +	local: *;
> > +};
> > diff --git a/drivers/net/meson.build b/drivers/net/meson.build
> > index c8b5ce2..5b066fd 100644
> > --- a/drivers/net/meson.build
> > +++ b/drivers/net/meson.build
> > @@ -12,6 +12,7 @@ drivers = [
> >           'bnx2x',
> >           'bnxt',
> >           'bonding',
> > +	'cnxk',
> >           'cxgbe',
> >           'dpaa',
> >           'dpaa2',