[dpdk-dev] [PATCH 00/16] nfp: add pf support

[dpdk-dev] [PATCH 00/16] nfp: add pf support

[Alejandro Lucero]

NFP PMD has just had support for SRIOV VFs until now. This patchset adds
support for the PF, but just for being used as another DPDK port. No VF
management is added by now.

NFP is a programmable device and it supports virtual NICs (vNICs) through
firmware implementation. Different firmware applications implement vNICs
for PF devices and VF devices, being number of vNICs dependent on the
firmware and the NFP card available. PF vNIC (virtual) BARs are a subset
of PF PCI device BARs while VF vNIC BARs are same than VF PCI BARs. 

Working with VF vNICs requires a PF driver uploading the firmware and 
doing some NFP configuration. This needs can be only done with the kernel
NFP PF netdev driver by now.

Working with PF vNIC requires the PMD doing the NFP configuration and for
accessing the NFP a specific user space interface is created. NFP Service 
Processor Userspace (NSPU) interface allows to create specific PCI BAR
windows for accessing different parts of the NFP device, including the
Network Service Processor (NSP) itself. The NSPU interface is implemented
as the base for working with the PF.

Alejandro Lucero (16):
  nfp: add nsp user space interface
  nfp: add specific pf probe function
  nfp: add support for new pci id
  nfp: add nsp support for commands
  nfp: add nsp fw upload command
  nfp: add nsp symbol resolution command
  nfp: add fw upload logic
  nfp: add support for vnic config bar mapping
  nfp: add support for vNIC rx/tx bar mappings
  nfp: support pf devices inside pmd initialization
  nfp: allocate eth_dev from pf probe function
  nfp: support pf multiport
  nfp: add nsp support for hw link configuration
  nfp: add support for hw port link configuration
  nfp: read pf port mac addr using nsp
  doc: update nfp with pf support information

 doc/guides/nics/nfp.rst        |  71 +++--
 drivers/net/nfp/Makefile       |   2 +
 drivers/net/nfp/nfp_net.c      | 377 +++++++++++++++++++++++--
 drivers/net/nfp/nfp_net_ctrl.h |   3 +
 drivers/net/nfp/nfp_net_eth.h  |  82 ++++++
 drivers/net/nfp/nfp_net_pmd.h  |   8 +
 drivers/net/nfp/nfp_nfpu.c     | 103 +++++++
 drivers/net/nfp/nfp_nfpu.h     |  55 ++++
 drivers/net/nfp/nfp_nspu.c     | 623 +++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h     |  83 ++++++
 10 files changed, 1369 insertions(+), 38 deletions(-)
 create mode 100644 drivers/net/nfp/nfp_net_eth.h
 create mode 100644 drivers/net/nfp/nfp_nfpu.c
 create mode 100644 drivers/net/nfp/nfp_nfpu.h
 create mode 100644 drivers/net/nfp/nfp_nspu.c
 create mode 100644 drivers/net/nfp/nfp_nspu.h

-- 
1.9.1

[dpdk-dev] [PATCH 01/16] nfp: add nsp user space interface

[Alejandro Lucero]

Working with the PF requires access to the NFP for basic configuration.
NSP is the NFP Service Processor helping with hardware and firmware
configuration. NSPU is the NSP user space interface for working with the
NSP.

Configuration through NSPU allows to create PCI BAR windows for accessing
different NFP hardware units, including the BAR window for the NSPU
interface access itself. NFP expansion bar registers are used for creating
those PCI BAR windows. NSPU uses a specific expansion bar which is
reprogrammed for accessing/doing different things.

Other expansion bars will be configured later for configuring the PF vNIC
bars, a subset of PF PCI BARs.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/Makefile   |   2 +
 drivers/net/nfp/nfp_nfpu.c | 103 ++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nfpu.h |  55 +++++++++++++++++++
 drivers/net/nfp/nfp_nspu.c | 129 +++++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h |  71 +++++++++++++++++++++++++
 5 files changed, 360 insertions(+)
 create mode 100644 drivers/net/nfp/nfp_nfpu.c
 create mode 100644 drivers/net/nfp/nfp_nfpu.h
 create mode 100644 drivers/net/nfp/nfp_nspu.c
 create mode 100644 drivers/net/nfp/nfp_nspu.h

diff --git a/drivers/net/nfp/Makefile b/drivers/net/nfp/Makefile
index 4ee2c2d..3e4c6f4 100644
--- a/drivers/net/nfp/Makefile
+++ b/drivers/net/nfp/Makefile
@@ -49,5 +49,7 @@ LIBABIVER := 1
 # all source are stored in SRCS-y
 #
 SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_net.c
+SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_nfpu.c
+SRCS-$(CONFIG_RTE_LIBRTE_NFP_PMD) += nfp_nspu.c
 
 include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/net/nfp/nfp_nfpu.c b/drivers/net/nfp/nfp_nfpu.c
new file mode 100644
index 0000000..556ded3
--- /dev/null
+++ b/drivers/net/nfp/nfp_nfpu.c
@@ -0,0 +1,103 @@
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <sys/file.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <sys/types.h>
+
+#include <rte_pci.h>
+#include <rte_malloc.h>
+
+#include "nfp_nfpu.h"
+
+/* PF BAR and expansion BAR for the NSP interface */
+#define NFP_CFG_PCIE_BAR        0
+#define NFP_CFG_EXP_BAR         7
+
+#define NFP_CFG_EXP_BAR_CFG_BASE	0x30000
+
+/* There could be other NFP userspace tools using the NSP interface.
+ * Make sure there is no other process using it and locking the access for
+ * avoiding problems.
+ */
+static int
+nspv_aquire_process_lock(nfpu_desc_t *desc)
+{
+	int rc;
+	struct flock lock;
+	char lockname[30];
+
+	memset(&lock, 0, sizeof(lock));
+
+	snprintf(lockname, sizeof(lockname), "/var/lock/nfp%d", desc->nfp);
+
+	/* Using S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH */
+	desc->lock = open(lockname, O_RDWR | O_CREAT, 0666);
+
+	if (desc->lock < 0)
+		return desc->lock;
+
+	lock.l_type = F_WRLCK;
+	lock.l_whence = SEEK_SET;
+	rc = -1;
+	while (rc != 0) {
+		rc = fcntl(desc->lock, F_SETLK, &lock);
+		if (rc < 0) {
+			if ((errno != EAGAIN) && (errno != EACCES)) {
+				close(desc->lock);
+				return rc;
+			}
+		}
+	}
+
+	return 0;
+}
+
+int
+nfpu_open(struct rte_pci_device *pci_dev, nfpu_desc_t *desc, int nfp)
+{
+	void *cfg_base, *mem_base;
+	size_t barsz;
+	int ret = 0;
+	int i = 0;
+
+	desc->nfp = nfp;
+
+	ret = nspv_aquire_process_lock(desc);
+	if (ret)
+		return -1;
+
+	barsz = pci_dev->mem_resource[0].len;
+
+	/* barsz in log2 */
+	while (barsz >>= 1)
+		i++;
+	barsz = i;
+
+	/* Getting address for NFP expansion BAR registers */
+	cfg_base = pci_dev->mem_resource[0].addr;
+	cfg_base = (uint8_t *)cfg_base + NFP_CFG_EXP_BAR_CFG_BASE;
+
+	/* Getting address for NFP NSP interface registers */
+	mem_base = pci_dev->mem_resource[0].addr;
+	mem_base = (uint8_t *)mem_base + (NFP_CFG_EXP_BAR << (barsz - 3));
+
+
+	desc->nspu = rte_malloc("nfp nspu", sizeof(nspu_desc_t), 0);
+	nfp_nspu_init(desc->nspu, desc->nfp, NFP_CFG_PCIE_BAR, barsz,
+		      NFP_CFG_EXP_BAR, cfg_base, mem_base);
+
+	return ret;
+}
+
+int
+nfpu_close(nfpu_desc_t *desc)
+{
+	rte_free(desc->nspu);
+	close(desc->lock);
+	unlink("/var/lock/nfp0");
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nfpu.h b/drivers/net/nfp/nfp_nfpu.h
new file mode 100644
index 0000000..31511b3
--- /dev/null
+++ b/drivers/net/nfp/nfp_nfpu.h
@@ -0,0 +1,55 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_nfpu.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+/*
+ * NFP User interface creates a window for talking with NFP NSP processor
+ */
+
+
+#include <rte_pci.h>
+#include "nfp_nspu.h"
+
+typedef struct {
+	int nfp;
+	int lock;
+	nspu_desc_t *nspu;
+} nfpu_desc_t;
+
+int nfpu_open(struct rte_pci_device *pci_dev, nfpu_desc_t *desc, int nfp);
+int nfpu_close(nfpu_desc_t *desc);
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
new file mode 100644
index 0000000..a157915
--- /dev/null
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -0,0 +1,129 @@
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/types.h>
+
+#include <rte_log.h>
+
+#include "nfp_nfpu.h"
+
+#define CFG_EXP_BAR_ADDR_SZ     1
+#define CFG_EXP_BAR_MAP_TYPE	1
+
+#define EXP_BAR_TARGET_SHIFT     23
+#define EXP_BAR_LENGTH_SHIFT     27 /* 0=32, 1=64 bit increment */
+#define EXP_BAR_MAP_TYPE_SHIFT   29 /* Bulk BAR map */
+
+/* NFP target for NSP access */
+#define NFP_NSP_TARGET   7
+
+/*
+ * This is an NFP internal address used for configuring properly an NFP
+ * expansion BAR.
+ */
+#define MEM_CMD_BASE_ADDR       0x8100000000
+
+/* NSP interface registers */
+#define NSP_BASE                (MEM_CMD_BASE_ADDR + 0x22100)
+#define NSP_STATUS              0x00
+#define NSP_COMMAND             0x08
+#define NSP_BUFFER		0x10
+#define NSP_DEFAULT_BUFFER      0x18
+#define NSP_DEFAULT_BUFFER_CFG  0x20
+
+#define NSP_MAGIC                0xab10
+#define NSP_STATUS_MAGIC(x)      (((x) >> 48) & 0xffff)
+#define NSP_STATUS_MAJOR(x)      (int)(((x) >> 44) & 0xf)
+#define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
+
+#define NSP_REG_ADDR(d, off, reg) ((uint8_t *)(d)->mem_base + (off) + (reg))
+#define NSP_REG_VAL(p) (*(uint64_t *)(p))
+
+/*
+ * An NFP expansion BAR is configured for allowing access to a specific NFP
+ * target:
+ *
+ *  IN:
+ *	desc: struct with basic NSP addresses to work with
+ *	expbar: NFP PF expansion BAR index to configure
+ *	tgt: NFP target to configure access
+ *	addr: NFP target address
+ *
+ *  OUT:
+ *	pcie_offset: NFP PCI BAR offset to work with
+ */
+static void
+nfp_nspu_mem_bar_cfg(nspu_desc_t *desc, int expbar, int tgt,
+		     uint64_t addr, uint64_t *pcie_offset)
+{
+	uint64_t x, y, barsz;
+	uint32_t *expbar_ptr;
+
+	barsz = desc->barsz;
+
+	/*
+	 * NFP CPP address to configure. This comes from NFP 6000
+	 * datasheet document based on Bulk mapping.
+	 */
+	x = (addr >> (barsz - 3)) << (21 - (40 - (barsz - 3)));
+	x |= CFG_EXP_BAR_MAP_TYPE << EXP_BAR_MAP_TYPE_SHIFT;
+	x |= CFG_EXP_BAR_ADDR_SZ << EXP_BAR_LENGTH_SHIFT;
+	x |= tgt << EXP_BAR_TARGET_SHIFT;
+
+	/* Getting expansion bar configuration register address */
+	expbar_ptr = (uint32_t *)desc->cfg_base;
+	/* Each physical PCI BAR has 8 NFP expansion BARs */
+	expbar_ptr += (desc->pcie_bar * 8) + expbar;
+
+	/* Writing to the expansion BAR register */
+	*expbar_ptr = (uint32_t)x;
+
+	/* Getting the pcie offset to work with from userspace */
+	y = addr & ((uint64_t)(1 << (barsz - 3)) - 1);
+	*pcie_offset = y;
+}
+
+/*
+ * Configuring an expansion bar for accessing NSP userspace interface. This
+ * function configures always the same expansion bar, which implies access to
+ * previously configured NFP target is lost.
+ */
+static void
+nspu_xlate(nspu_desc_t *desc, uint64_t addr, uint64_t *pcie_offset)
+{
+	nfp_nspu_mem_bar_cfg(desc, desc->exp_bar, NFP_NSP_TARGET, addr,
+			     pcie_offset);
+}
+
+int
+nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor)
+{
+	uint64_t pcie_offset;
+	uint64_t nsp_reg;
+
+	nspu_xlate(desc, NSP_BASE, &pcie_offset);
+	nsp_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, pcie_offset, NSP_STATUS));
+
+	if (NSP_STATUS_MAGIC(nsp_reg) != NSP_MAGIC)
+		return -1;
+
+	*major = NSP_STATUS_MAJOR(nsp_reg);
+	*minor = NSP_STATUS_MINOR(nsp_reg);
+
+	return 0;
+}
+
+int
+nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
+	      int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap)
+{
+	desc->nfp = nfp;
+	desc->pcie_bar = pcie_bar;
+	desc->exp_bar = exp_bar;
+	desc->barsz = pcie_barsz;
+	desc->cfg_base = exp_bar_cfg_base;
+	desc->mem_base = exp_bar_mmap;
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
new file mode 100644
index 0000000..7a1ac91
--- /dev/null
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -0,0 +1,71 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_nspu.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+/*
+ * NSP is the NFP Service Processor. NSPU is NSP Userspace interface.
+ *
+ * NFP NSP helps with firmware/hardware configuration. NSP is another component
+ * in NFP programmable processor and accessing it from host requires to firstly
+ * configure a specific NFP PCI expansion BAR.
+ *
+ * Once access is ready, configuration can be done reading and writing
+ * from/to a specific PF PCI BAR window. This same interface will allow to
+ * create other PCI BAR windows for accessing other NFP components.
+ *
+ * This file includes low-level functions, using the NSPU interface, and high
+ * level functions, invoked by the PMD for using NSP services. This allows
+ * firmware upload, vNIC PCI BARs mapping and other low-level configurations
+ * like link setup.
+ *
+ * NSP access is done during initialization and it is not involved at all with
+ * the fast path.
+ */
+
+typedef struct {
+	int nfp;        /* NFP device */
+	int pcie_bar;   /* PF PCI BAR to work with */
+	int exp_bar;    /* Expansion BAR number used by NSPU */
+	int barsz;      /* PCIE BAR log2 size */
+	void *cfg_base; /* Expansion BARs address */
+	void *mem_base; /* NSP interface */
+} nspu_desc_t;
+
+int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
+		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
+int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
-- 
1.9.1

[dpdk-dev] [PATCH 02/16] nfp: add specific pf probe function

[Alejandro Lucero]

Configuring the NFP PMD for using the PF requires access through the
NSPU interface for device configuration. This patch adds a specific probe
function for the PF which uses the NSPU interface. Just basic NSPU access
is done by now reading the NSPU ABI version.

No ethernet port is created yet.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 75 +++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 69 insertions(+), 6 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index a3bf5e1..e2fe83a 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -59,6 +59,8 @@
 #include "nfp_net_logs.h"
 #include "nfp_net_ctrl.h"
 
+#include "nfp_nfpu.h"
+
 /* Prototypes */
 static void nfp_net_close(struct rte_eth_dev *dev);
 static int nfp_net_configure(struct rte_eth_dev *dev);
@@ -2632,12 +2634,63 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	return 0;
 }
 
-static const struct rte_pci_id pci_id_nfp_net_map[] = {
+static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+			    struct rte_pci_device *dev)
+{
+	nfpu_desc_t *nfpu_desc;
+	nspu_desc_t *nspu_desc;
+	int major, minor;
+
+	if (!dev)
+		return -ENODEV;
+
+	nfpu_desc = rte_malloc("nfp nfpu", sizeof(nfpu_desc_t), 0);
+	if (!nfpu_desc)
+		return -ENOMEM;
+
+	if (nfpu_open(dev, nfpu_desc, 0) < 0) {
+		RTE_LOG(ERR, PMD,
+			"nfpu_open failed\n");
+		goto nfpu_error;
+	}
+
+	nspu_desc = nfpu_desc->nspu;
+
+
+	/* Check NSP ABI version */
+	if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
+		RTE_LOG(INFO, PMD, "NFP NSP not present\n");
+		goto no_abi;
+	}
+	PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
+
+	if (minor < 20) {
+		RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required 0.20 or higher\n");
+		goto no_abi;
+	}
+
+	/* No port is created yet */
+
+no_abi:
+	nfpu_close(nfpu_desc);
+nfpu_error:
+	rte_free(nfpu_desc);
+
+	return -ENODEV;
+}
+
+static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_PF_NIC)
 	},
 	{
+		.vendor_id = 0,
+	},
+};
+
+static const struct rte_pci_id pci_id_nfp_vf_net_map[] = {
+	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_VF_NIC)
 	},
@@ -2658,16 +2711,26 @@ static int eth_nfp_pci_remove(struct rte_pci_device *pci_dev)
 	return rte_eth_dev_pci_generic_remove(pci_dev, NULL);
 }
 
-static struct rte_pci_driver rte_nfp_net_pmd = {
-	.id_table = pci_id_nfp_net_map,
+static struct rte_pci_driver rte_nfp_net_pf_pmd = {
+	.id_table = pci_id_nfp_pf_net_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
+	.probe = nfp_pf_pci_probe,
+	.remove = eth_nfp_pci_remove,
+};
+
+static struct rte_pci_driver rte_nfp_net_vf_pmd = {
+	.id_table = pci_id_nfp_vf_net_map,
 	.drv_flags = RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC,
 	.probe = eth_nfp_pci_probe,
 	.remove = eth_nfp_pci_remove,
 };
 
-RTE_PMD_REGISTER_PCI(net_nfp, rte_nfp_net_pmd);
-RTE_PMD_REGISTER_PCI_TABLE(net_nfp, pci_id_nfp_net_map);
-RTE_PMD_REGISTER_KMOD_DEP(net_nfp, "* igb_uio | uio_pci_generic | vfio-pci");
+RTE_PMD_REGISTER_PCI(net_nfp_pf, rte_nfp_net_pf_pmd);
+RTE_PMD_REGISTER_PCI(net_nfp_vf, rte_nfp_net_vf_pmd);
+RTE_PMD_REGISTER_PCI_TABLE(net_nfp_pf, pci_id_nfp_pf_net_map);
+RTE_PMD_REGISTER_PCI_TABLE(net_nfp_vf, pci_id_nfp_vf_net_map);
+RTE_PMD_REGISTER_KMOD_DEP(net_nfp_pf, "* igb_uio | uio_pci_generic | vfio");
+RTE_PMD_REGISTER_KMOD_DEP(net_nfp_vf, "* igb_uio | uio_pci_generic | vfio");
 
 /*
  * Local variables:
-- 
1.9.1

Re: [dpdk-dev] [PATCH 02/16] nfp: add specific pf probe function

[Ferruh Yigit]

On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> Configuring the NFP PMD for using the PF requires access through the
> NSPU interface for device configuration. This patch adds a specific probe
> function for the PF which uses the NSPU interface. Just basic NSPU access
> is done by now reading the NSPU ABI version.
> 
> No ethernet port is created yet.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

<...>

> +	/* Check NSP ABI version */
> +	if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
> +		RTE_LOG(INFO, PMD, "NFP NSP not present\n");
> +		goto no_abi;
> +	}
> +	PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
> +
> +	if (minor < 20) {
> +		RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required 0.20 or higher\n");

I believe it worth documenting this detail in commit log and documentation.

<...>

>  
> -RTE_PMD_REGISTER_PCI(net_nfp, rte_nfp_net_pmd);
> -RTE_PMD_REGISTER_PCI_TABLE(net_nfp, pci_id_nfp_net_map);
> -RTE_PMD_REGISTER_KMOD_DEP(net_nfp, "* igb_uio | uio_pci_generic | vfio-pci");
> +RTE_PMD_REGISTER_PCI(net_nfp_pf, rte_nfp_net_pf_pmd);
> +RTE_PMD_REGISTER_PCI(net_nfp_vf, rte_nfp_net_vf_pmd);

Now pf and vf drivers are separated. For existing drivers this has been
documented in features file as another file (another column in table),
but we are looking for better representation for this.

What do you think, does two drivers has significant enough differences
to be documented as two different drivers?

> +RTE_PMD_REGISTER_PCI_TABLE(net_nfp_pf, pci_id_nfp_pf_net_map);
> +RTE_PMD_REGISTER_PCI_TABLE(net_nfp_vf, pci_id_nfp_vf_net_map);
> +RTE_PMD_REGISTER_KMOD_DEP(net_nfp_pf, "* igb_uio | uio_pci_generic | vfio");
> +RTE_PMD_REGISTER_KMOD_DEP(net_nfp_vf, "* igb_uio | uio_pci_generic | vfio");
>  
>  /*
>   * Local variables:
> 

Re: [dpdk-dev] [PATCH 02/16] nfp: add specific pf probe function

[Alejandro Lucero]

On Mon, Aug 28, 2017 at 5:42 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> > Configuring the NFP PMD for using the PF requires access through the
> > NSPU interface for device configuration. This patch adds a specific probe
> > function for the PF which uses the NSPU interface. Just basic NSPU access
> > is done by now reading the NSPU ABI version.
> >
> > No ethernet port is created yet.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
> <...>
>
> > +     /* Check NSP ABI version */
> > +     if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
> > +             RTE_LOG(INFO, PMD, "NFP NSP not present\n");
> > +             goto no_abi;
> > +     }
> > +     PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
> > +
> > +     if (minor < 20) {
> > +             RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required
> 0.20 or higher\n");
>
> I believe it worth documenting this detail in commit log and documentation.
>

Ok.


>
> <...>
>
> >
> > -RTE_PMD_REGISTER_PCI(net_nfp, rte_nfp_net_pmd);
> > -RTE_PMD_REGISTER_PCI_TABLE(net_nfp, pci_id_nfp_net_map);
> > -RTE_PMD_REGISTER_KMOD_DEP(net_nfp, "* igb_uio | uio_pci_generic |
> vfio-pci");
> > +RTE_PMD_REGISTER_PCI(net_nfp_pf, rte_nfp_net_pf_pmd);
> > +RTE_PMD_REGISTER_PCI(net_nfp_vf, rte_nfp_net_vf_pmd);
>
> Now pf and vf drivers are separated. For existing drivers this has been
> documented in features file as another file (another column in table),
> but we are looking for better representation for this.
>
> What do you think, does two drivers has significant enough differences
> to be documented as two different drivers?
>
>
At this point PF and VF PMDs are exactly the same except for how
initialization is done. But, this will likely change in the near future.

I did not think about splitting out the features file, but I think it makes
sense. The existing one, just for VFs, has a problem with SRIOV. Obviously
VF support implies SRIOV, but I think the original idea of such a feature
was drivers being able to manage SRIOV, this is, creating and destroying
VFs. Also, firmware upload is just available with the PF, although such a
feature is not in the current features description list.

So, yes, I think I should have a file for the PF PMD and another one for
the VF.

I will add this in next patch set version.

Thanks



> > +RTE_PMD_REGISTER_PCI_TABLE(net_nfp_pf, pci_id_nfp_pf_net_map);
> > +RTE_PMD_REGISTER_PCI_TABLE(net_nfp_vf, pci_id_nfp_vf_net_map);
> > +RTE_PMD_REGISTER_KMOD_DEP(net_nfp_pf, "* igb_uio | uio_pci_generic |
> vfio");
> > +RTE_PMD_REGISTER_KMOD_DEP(net_nfp_vf, "* igb_uio | uio_pci_generic |
> vfio");
> >
> >  /*
> >   * Local variables:
> >
>
>

[dpdk-dev] [PATCH 03/16] nfp: add support for new pci id

[Alejandro Lucero]

A NFP PF PCI devices can have PCI ID 4000 or 6000.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 4 ++++
 drivers/net/nfp/nfp_net_pmd.h | 1 +
 2 files changed, 5 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index e2fe83a..1890a4a 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
 	{
 		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
+			       PCI_DEVICE_ID_NFP4000_PF_NIC)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
 			       PCI_DEVICE_ID_NFP6000_PF_NIC)
 	},
 	{
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index c6bddaa..3818130 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -42,6 +42,7 @@
 
 #define NFP_NET_PMD_VERSION "0.1"
 #define PCI_VENDOR_ID_NETRONOME         0x19ee
+#define PCI_DEVICE_ID_NFP4000_PF_NIC    0x4000
 #define PCI_DEVICE_ID_NFP6000_PF_NIC    0x6000
 #define PCI_DEVICE_ID_NFP6000_VF_NIC    0x6003
 
-- 
1.9.1

Re: [dpdk-dev] [PATCH 03/16] nfp: add support for new pci id

[Ferruh Yigit]

On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> A NFP PF PCI devices can have PCI ID 4000 or 6000.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

<...>

> @@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>  static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
>  	{
>  		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
> +			       PCI_DEVICE_ID_NFP4000_PF_NIC)

I have seen nfp documentation updated with this new device, and I
believe it also worth updating release notes to mention new device
support (doc/guides/rel_notes/release_17_11.rst)

Also supported nics web page (http://dpdk.org/doc/nics), needs updating
(http://dpdk.org/browse/tools/dpdk-web/)

> +	},
> +	{
> +		RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
>  			       PCI_DEVICE_ID_NFP6000_PF_NIC)
<...>

Re: [dpdk-dev] [PATCH 03/16] nfp: add support for new pci id

[Alejandro Lucero]

On Mon, Aug 28, 2017 at 5:43 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> > A NFP PF PCI devices can have PCI ID 4000 or 6000.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
> <...>
>
> > @@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct rte_pci_driver
> *pci_drv __rte_unused,
> >  static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
> >       {
> >               RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
> > +                            PCI_DEVICE_ID_NFP4000_PF_NIC)
>
> I have seen nfp documentation updated with this new device, and I
> believe it also worth updating release notes to mention new device
> support (doc/guides/rel_notes/release_17_11.rst)
>
>
Yes, I agree. I should add this as well.


> Also supported nics web page (http://dpdk.org/doc/nics), needs updating
> (http://dpdk.org/browse/tools/dpdk-web/)
>
>
Not sure about this one. I could not find any file in the repo for changing
this. How should I manage it?


> > +     },
> > +     {
> > +             RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
> >                              PCI_DEVICE_ID_NFP6000_PF_NIC)
> <...>
>
>

Re: [dpdk-dev] [PATCH 03/16] nfp: add support for new pci id

[Ferruh Yigit]

On 8/31/2017 10:08 AM, Alejandro Lucero wrote:
> 
> 
> On Mon, Aug 28, 2017 at 5:43 PM, Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>> wrote:
> 
>     On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
>     > A NFP PF PCI devices can have PCI ID 4000 or 6000.
>     >
>     > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com <mailto:alejandro.lucero@netronome.com>>
> 
>     <...>
> 
>     > @@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>     >  static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
>     >       {
>     >               RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
>     > +                            PCI_DEVICE_ID_NFP4000_PF_NIC)
> 
>     I have seen nfp documentation updated with this new device, and I
>     believe it also worth updating release notes to mention new device
>     support (doc/guides/rel_notes/release_17_11.rst)
> 
> 
> Yes, I agree. I should add this as well.
>  
> 
>     Also supported nics web page (http://dpdk.org/doc/nics), needs updating
>     (http://dpdk.org/browse/tools/dpdk-web/
>     <http://dpdk.org/browse/tools/dpdk-web/>)
> 
> 
> Not sure about this one. I could not find any file in the repo for
> changing this. How should I manage it?

Please check [1], currently it lists NFP-6xxx only.

[1]
http://dpdk.org/browse/tools/dpdk-web/tree/doc/nics.html#n93

>  
> 
>     > +     },
>     > +     {
>     > +             RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
>     >                              PCI_DEVICE_ID_NFP6000_PF_NIC)
>     <...>
> 
> 

Re: [dpdk-dev] [PATCH 03/16] nfp: add support for new pci id

[Alejandro Lucero]

On Thu, Aug 31, 2017 at 10:13 AM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/31/2017 10:08 AM, Alejandro Lucero wrote:
> >
> >
> > On Mon, Aug 28, 2017 at 5:43 PM, Ferruh Yigit <ferruh.yigit@intel.com
> > <mailto:ferruh.yigit@intel.com>> wrote:
> >
> >     On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> >     > A NFP PF PCI devices can have PCI ID 4000 or 6000.
> >     >
> >     > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com
> <mailto:alejandro.lucero@netronome.com>>
> >
> >     <...>
> >
> >     > @@ -2682,6 +2682,10 @@ static int nfp_pf_pci_probe(struct
> rte_pci_driver *pci_drv __rte_unused,
> >     >  static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
> >     >       {
> >     >               RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
> >     > +                            PCI_DEVICE_ID_NFP4000_PF_NIC)
> >
> >     I have seen nfp documentation updated with this new device, and I
> >     believe it also worth updating release notes to mention new device
> >     support (doc/guides/rel_notes/release_17_11.rst)
> >
> >
> > Yes, I agree. I should add this as well.
> >
> >
> >     Also supported nics web page (http://dpdk.org/doc/nics), needs
> updating
> >     (http://dpdk.org/browse/tools/dpdk-web/
> >     <http://dpdk.org/browse/tools/dpdk-web/>)
> >
> >
> > Not sure about this one. I could not find any file in the repo for
> > changing this. How should I manage it?
>
> Please check [1], currently it lists NFP-6xxx only.
>
> [1]
> http://dpdk.org/browse/tools/dpdk-web/tree/doc/nics.html#n93
>
>
I did not realize there is a repo for this.

Thanks


> >
> >
> >     > +     },
> >     > +     {
> >     > +             RTE_PCI_DEVICE(PCI_VENDOR_ID_NETRONOME,
> >     >                              PCI_DEVICE_ID_NFP6000_PF_NIC)
> >     <...>
> >
> >
>
>

[dpdk-dev] [PATCH 04/16] nfp: add nsp support for commands

[Alejandro Lucero]

NSPU interface declares a buffer controlled by the NFP NSP service
processor. It is possible to send commands to the NSP using the NSPU
and this buffer for data related to the command. A command can imply
buffer read, buffer write, both or none.

Initial command for resetting the firmware is added as well which
does not require the buffer at all.

Commands will allow firmware upload, symbol resolution and ethernet
link configuration. Future commands will allow specific offloads like
flow offloads and eBPF offload.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 181 ++++++++++++++++++++++++++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |   4 +
 2 files changed, 183 insertions(+), 2 deletions(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index a157915..dbb5305 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -29,14 +29,19 @@
 #define NSP_STATUS              0x00
 #define NSP_COMMAND             0x08
 #define NSP_BUFFER		0x10
-#define NSP_DEFAULT_BUFFER      0x18
-#define NSP_DEFAULT_BUFFER_CFG  0x20
+#define NSP_DEFAULT_BUF         0x18
+#define NSP_DEFAULT_BUF_CFG  0x20
 
 #define NSP_MAGIC                0xab10
 #define NSP_STATUS_MAGIC(x)      (((x) >> 48) & 0xffff)
 #define NSP_STATUS_MAJOR(x)      (int)(((x) >> 44) & 0xf)
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
+/* NSP commands */
+#define NSP_CMD_RESET			1
+
+#define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
+
 #define NSP_REG_ADDR(d, off, reg) ((uint8_t *)(d)->mem_base + (off) + (reg))
 #define NSP_REG_VAL(p) (*(uint64_t *)(p))
 
@@ -118,12 +123,184 @@
 nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 	      int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap)
 {
+	uint64_t offset, buffaddr;
+	uint64_t nsp_reg;
+
 	desc->nfp = nfp;
 	desc->pcie_bar = pcie_bar;
 	desc->exp_bar = exp_bar;
 	desc->barsz = pcie_barsz;
+	desc->windowsz = 1 << (desc->barsz - 3);
 	desc->cfg_base = exp_bar_cfg_base;
 	desc->mem_base = exp_bar_mmap;
 
+	nspu_xlate(desc, NSP_BASE, &offset);
+
+	/*
+	 * Other NSPU clients can use other buffers. Let's tell NSPU we use the
+	 * default buffer.
+	 */
+	buffaddr = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_DEFAULT_BUF));
+	NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_BUFFER)) = buffaddr;
+
+	/* NFP internal addresses are 40 bits. Clean all other bits here */
+	buffaddr = buffaddr & (((uint64_t)1 << 40) - 1);
+	desc->bufaddr = buffaddr;
+
+	/* Lets get information about the buffer */
+	nsp_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_DEFAULT_BUF_CFG));
+
+	/* Buffer size comes in MBs. Coversion to bytes */
+	desc->buf_size = ((size_t)nsp_reg & NSP_BUFFER_CFG_SIZE_MASK) << 20;
+
 	return 0;
 }
+
+#define NSPU_NFP_BUF(addr, base, off) \
+	(*(uint64_t *)((uint8_t *)(addr)->mem_base + ((base) | (off))))
+
+#define NSPU_HOST_BUF(base, off) (*(uint64_t *)((uint8_t *)(base) + (off)))
+
+static int
+nspu_buff_write(nspu_desc_t *desc, void *buffer, size_t size)
+{
+	uint64_t pcie_offset, pcie_window_base, pcie_window_offset;
+	uint64_t windowsz = desc->windowsz;
+	uint64_t buffaddr, j, i = 0;
+	int ret = 0;
+
+	if (size > desc->buf_size)
+		return -1;
+
+	buffaddr = desc->bufaddr;
+	windowsz = desc->windowsz;
+
+	while (i < size) {
+		/* Expansion bar reconfiguration per window size */
+		nspu_xlate(desc, buffaddr + i, &pcie_offset);
+		pcie_window_base = pcie_offset & (~(windowsz - 1));
+		pcie_window_offset = pcie_offset & (windowsz - 1);
+		for (j = pcie_window_offset; ((j < windowsz) && (i < size));
+		     j += 8) {
+			NSPU_NFP_BUF(desc, pcie_window_base, j) =
+				NSPU_HOST_BUF(buffer, i);
+			i += 8;
+		}
+	}
+
+	return ret;
+}
+
+static int
+nspu_buff_read(nspu_desc_t *desc, void *buffer, size_t size)
+{
+	uint64_t pcie_offset, pcie_window_base, pcie_window_offset;
+	uint64_t windowsz, i = 0, j;
+	uint64_t buffaddr;
+	int ret = 0;
+
+	if (size > desc->buf_size)
+		return -1;
+
+	buffaddr = desc->bufaddr;
+	windowsz = desc->windowsz;
+
+	while (i < size) {
+		/* Expansion bar reconfiguration per window size */
+		nspu_xlate(desc, buffaddr + i, &pcie_offset);
+		pcie_window_base = pcie_offset & (~(windowsz - 1));
+		pcie_window_offset = pcie_offset & (windowsz - 1);
+		for (j = pcie_window_offset; ((j < windowsz) && (i < size));
+		     j += 8) {
+			NSPU_HOST_BUF(buffer, i) =
+				NSPU_NFP_BUF(desc, pcie_window_base, j);
+			i += 8;
+		}
+	}
+
+	return ret;
+}
+
+static int
+nspu_command(nspu_desc_t *desc, uint16_t cmd, int read, int write,
+		 void *buffer, size_t rsize, size_t wsize)
+{
+	uint64_t status, cmd_reg;
+	uint64_t offset;
+	int retry = 0;
+	int retries = 120;
+	int ret = 0;
+
+	/* Same expansion BAR is used for different things */
+	nspu_xlate(desc, NSP_BASE, &offset);
+
+	status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+
+	while ((status & 0x1) && (retry < retries)) {
+		status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+		retry++;
+		sleep(1);
+	}
+
+	if (retry == retries)
+		return -1;
+
+	if (write) {
+		ret = nspu_buff_write(desc, buffer, wsize);
+		if (ret)
+			return ret;
+
+		/* Expansion BAR changes when writing the buffer */
+		nspu_xlate(desc, NSP_BASE, &offset);
+	}
+
+	NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND)) =
+		(uint64_t)wsize << 32 | (uint64_t)cmd << 16 | 1;
+
+	retry = 0;
+
+	cmd_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND));
+	while ((cmd_reg & 0x1) && (retry < retries)) {
+		cmd_reg = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_COMMAND));
+		retry++;
+		sleep(1);
+	}
+	if (retry == retries)
+		return -1;
+
+	retry = 0;
+	status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+	while ((status & 0x1) && (retry < retries)) {
+		status = NSP_REG_VAL(NSP_REG_ADDR(desc, offset, NSP_STATUS));
+		retry++;
+		sleep(1);
+	}
+
+	if (retry == retries)
+		return -1;
+
+	ret = status & (0xff << 8);
+	if (ret)
+		return ret;
+
+	if (read) {
+		ret = nspu_buff_read(desc, buffer, rsize);
+		if (ret)
+			return ret;
+	}
+
+	return ret;
+}
+
+int
+nfp_fw_reset(nspu_desc_t *nspu_desc)
+{
+	int res;
+
+	res = nspu_command(nspu_desc, NSP_CMD_RESET, 0, 0, 0, 0, 0);
+
+	if (res < 0)
+		RTE_LOG(INFO, PMD, "fw reset failed: error %d", res);
+
+	return res;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 7a1ac91..a142eb3 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -62,6 +62,9 @@
 	int pcie_bar;   /* PF PCI BAR to work with */
 	int exp_bar;    /* Expansion BAR number used by NSPU */
 	int barsz;      /* PCIE BAR log2 size */
+	uint64_t bufaddr;  /* commands buffer address */
+	size_t buf_size;   /* commands buffer size */
+	uint64_t windowsz; /* NSPU BAR window size */
 	void *cfg_base; /* Expansion BARs address */
 	void *mem_base; /* NSP interface */
 } nspu_desc_t;
@@ -69,3 +72,4 @@
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
+int nfp_fw_reset(nspu_desc_t *nspu_desc);
-- 
1.9.1

[dpdk-dev] [PATCH 05/16] nfp: add nsp fw upload command

[Alejandro Lucero]

Using NSPU interface for fw upload. Firmware file needs to be
installed in specific path inside system firmware directory.

NSPU buffer is used for writing the firmware before sending the
command.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 66 +++++++++++++++++++++++++++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index dbb5305..57ee45f 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -3,6 +3,9 @@
 #include <string.h>
 #include <unistd.h>
 #include <sys/types.h>
+#include <sys/file.h>
+#include <sys/stat.h>
+#include <fcntl.h>
 
 #include <rte_log.h>
 
@@ -38,7 +41,8 @@
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
 /* NSP commands */
-#define NSP_CMD_RESET			1
+#define NSP_CMD_RESET          1
+#define NSP_CMD_FW_LOAD        6
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -304,3 +308,63 @@
 
 	return res;
 }
+
+#define DEFAULT_FW_PATH       "/lib/firmware/netronome"
+#define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"
+
+int
+nfp_fw_upload(nspu_desc_t *nspu_desc)
+{
+	int fw_f;
+	char *fw_buf;
+	char filename[100];
+	struct stat file_stat;
+	off_t fsize, bytes;
+	ssize_t size;
+	int ret;
+
+	size = nspu_desc->buf_size;
+
+	sprintf(filename, "%s/%s", DEFAULT_FW_PATH, DEFAULT_FW_FILENAME);
+	fw_f = open(filename, O_RDONLY);
+	if (fw_f < 0) {
+		RTE_LOG(INFO, PMD, "Firmware file %s/%s not found.",
+			DEFAULT_FW_PATH, DEFAULT_FW_FILENAME);
+		return -ENOENT;
+	}
+
+	fstat(fw_f, &file_stat);
+
+	fsize = file_stat.st_size;
+	RTE_LOG(DEBUG, PMD, "Firmware file with size: %" PRIu64 "\n",
+			    (uint64_t)fsize);
+
+	if (fsize > (off_t)size) {
+		RTE_LOG(INFO, PMD, "fw file too big: %" PRIu64
+				   " bytes (%" PRIu64 " max)",
+				  (uint64_t)fsize, (uint64_t)size);
+		return -EINVAL;
+	}
+
+	fw_buf = malloc((size_t)size);
+	if (!fw_buf) {
+		RTE_LOG(INFO, PMD, "malloc failed for fw buffer");
+		return -ENOMEM;
+	}
+	memset(fw_buf, 0, size);
+
+	bytes = read(fw_f, fw_buf, fsize);
+	if (bytes != fsize) {
+		RTE_LOG(INFO, PMD, "Reading fw to buffer failed.\n"
+				   "Just %" PRIu64 " of %" PRIu64 " bytes read.",
+				   (uint64_t)bytes, (uint64_t)fsize);
+		free(fw_buf);
+		return -EIO;
+	}
+
+	ret = nspu_command(nspu_desc, NSP_CMD_FW_LOAD, 0, 1, fw_buf, 0, bytes);
+
+	free(fw_buf);
+
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index a142eb3..6e1c25f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -73,3 +73,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_fw_reset(nspu_desc_t *nspu_desc);
+int nfp_fw_upload(nspu_desc_t *nspu_desc);
-- 
1.9.1

Re: [dpdk-dev] [PATCH 05/16] nfp: add nsp fw upload command

[Ferruh Yigit]

On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> Using NSPU interface for fw upload. Firmware file needs to be
> installed in specific path inside system firmware directory.
> 
> NSPU buffer is used for writing the firmware before sending the
> command.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

<...>

> +
> +#define DEFAULT_FW_PATH       "/lib/firmware/netronome"
> +#define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"

This can be good to mention about this in documentation.

And FW upload support in release notes.

Re: [dpdk-dev] [PATCH 05/16] nfp: add nsp fw upload command

[Alejandro Lucero]

On Mon, Aug 28, 2017 at 5:42 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> > Using NSPU interface for fw upload. Firmware file needs to be
> > installed in specific path inside system firmware directory.
> >
> > NSPU buffer is used for writing the firmware before sending the
> > command.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
> <...>
>
> > +
> > +#define DEFAULT_FW_PATH       "/lib/firmware/netronome"
> > +#define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"
>
> This can be good to mention about this in documentation.
>
> And FW upload support in release notes.
>

I'll add this in next patchset version.

Thanks

[dpdk-dev] [PATCH 06/16] nfp: add nsp symbol resolution command

[Alejandro Lucero]

Firmware has symbols helping to configure things like number of
PF ports, vNIC BARs addresses inside NFP memories, or ethernet
link state. Different firmware apps have different things to map
and likely different internal NFP addresses to use.

Host drivers can use the NSPU interface for getting symbol data
regarding different hardware configurations. Once the driver has
the information about a specific object, a mapping is required
configuring an NFP expansion bar creating a device PCI bar window.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 86 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.h |  3 ++
 2 files changed, 89 insertions(+)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 57ee45f..4296be1 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -43,6 +43,7 @@
 /* NSP commands */
 #define NSP_CMD_RESET          1
 #define NSP_CMD_FW_LOAD        6
+#define NSP_CMD_GET_SYMBOL     14
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -368,3 +369,88 @@
 
 	return ret;
 }
+
+/* Firmware symbol descriptor size */
+#define NFP_SYM_DESC_LEN 40
+
+#define SYMBOL_DATA(b, off)     (*(int64_t *)((b) + (off)))
+#define SYMBOL_UDATA(b, off)     (*(uint64_t *)((b) + (off)))
+
+/* Firmware symbols contain information about how to access what they
+ * represent. It can be as simple as an numeric variable declared at a
+ * specific NFP memory, but it can also be more complex structures and
+ * related to specific hardware functionalities or components. Target,
+ * domain and address allow to create the BAR window for accessing such
+ * hw object and size defines the length to map.
+ *
+ * A vNIC is a network interface implemented inside the NFP and using a
+ * subset of device PCI BARs. Specific firmware symbols allow to map those
+ * vNIC bars by host drivers like the NFP PMD.
+ *
+ * Accessing what the symbol represents implies to map the access through
+ * a PCI BAR window. NFP expansion BARs are used in this regard through
+ * the NSPU interface.
+ */
+int
+nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
+			    uint32_t expbar, uint64_t *pcie_offset,
+			    ssize_t *size)
+{
+	int64_t type;
+	int64_t target;
+	int64_t domain;
+	uint64_t addr;
+	char *sym_buf;
+	int ret = 0;
+
+	sym_buf = malloc(desc->buf_size);
+	strncpy(sym_buf, symbl, strlen(symbl));
+	ret = nspu_command(desc, NSP_CMD_GET_SYMBOL, 1, 1, sym_buf,
+			   NFP_SYM_DESC_LEN, strlen(symbl));
+	if (ret) {
+		RTE_LOG(DEBUG, PMD, "symbol resolution (%s) failed\n", symbl);
+		goto clean;
+	}
+
+	/* Reading symbol information */
+	type = SYMBOL_DATA(sym_buf, 0);
+	target = SYMBOL_DATA(sym_buf, 8);
+	domain =  SYMBOL_DATA(sym_buf, 16);
+	addr = SYMBOL_UDATA(sym_buf, 24);
+	*size = (ssize_t)SYMBOL_UDATA(sym_buf, 32);
+
+	if (type != 1) {
+		RTE_LOG(INFO, PMD, "wrong symbol type\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+	if (!(target == 7 || target == -7)) {
+		RTE_LOG(INFO, PMD, "wrong symbol target\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+	if (domain == 8 || domain == 9) {
+		RTE_LOG(INFO, PMD, "wrong symbol domain\n");
+		ret = -EINVAL;
+		goto clean;
+	}
+
+	/* Adjusting address based on symbol location */
+	if (domain >= 24 && domain << 28 && target == 7) {
+		addr = 1ULL << 37 | addr | ((uint64_t)domain & 0x3) << 35;
+	} else {
+		addr = 1ULL << 39 | addr | ((uint64_t)domain & 0x3f) << 32;
+		if (target == -7)
+			target = 7;
+	}
+
+	/* Configuring NFP expansion bar for mapping specific PCI BAR window */
+	nfp_nspu_mem_bar_cfg(desc, expbar, target, addr, pcie_offset);
+
+	/* This is the PCI BAR offset to use by the host */
+	*pcie_offset |= ((expbar & 0x7) << (desc->barsz - 3));
+
+clean:
+	free(sym_buf);
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 6e1c25f..7734b4f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -74,3 +74,6 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_fw_reset(nspu_desc_t *nspu_desc);
 int nfp_fw_upload(nspu_desc_t *nspu_desc);
+int nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
+				uint32_t expbar, uint64_t *pcie_offset,
+				ssize_t *size);
-- 
1.9.1

Re: [dpdk-dev] [PATCH 06/16] nfp: add nsp symbol resolution command

[Ferruh Yigit]

On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> Firmware has symbols helping to configure things like number of
> PF ports, vNIC BARs addresses inside NFP memories, or ethernet
> link state. Different firmware apps have different things to map
> and likely different internal NFP addresses to use.
> 
> Host drivers can use the NSPU interface for getting symbol data
> regarding different hardware configurations. Once the driver has
> the information about a specific object, a mapping is required
> configuring an NFP expansion bar creating a device PCI bar window.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

<...>

> +
> +	/* Adjusting address based on symbol location */
> +	if (domain >= 24 && domain << 28 && target == 7) {

gcc is giving following compiler warning [1]. Most probably intention is
the compare, but it is not clear, can you please check?

[1]
.../drivers/net/nfp/nfp_nspu.c: In function ‘nfp_nspu_set_bar_from_symbl’:
.../drivers/net/nfp/nfp_nspu.c:446:29: error: ‘<<’ in boolean context,
did you mean ‘<’ ? [-Werror=int-in-bool-context]
  if (domain >= 24 && domain << 28 && target == 7) {
                      ~~~~~~~^~~~~

> +		addr = 1ULL << 37 | addr | ((uint64_t)domain & 0x3) << 35;
> +	} else {
> +		addr = 1ULL << 39 | addr | ((uint64_t)domain & 0x3f) << 32;
> +		if (target == -7)
> +			target = 7;
> +	}

Re: [dpdk-dev] [PATCH 06/16] nfp: add nsp symbol resolution command

[Alejandro Lucero]

On Mon, Aug 28, 2017 at 5:42 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> > Firmware has symbols helping to configure things like number of
> > PF ports, vNIC BARs addresses inside NFP memories, or ethernet
> > link state. Different firmware apps have different things to map
> > and likely different internal NFP addresses to use.
> >
> > Host drivers can use the NSPU interface for getting symbol data
> > regarding different hardware configurations. Once the driver has
> > the information about a specific object, a mapping is required
> > configuring an NFP expansion bar creating a device PCI bar window.
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
> <...>
>
> > +
> > +     /* Adjusting address based on symbol location */
> > +     if (domain >= 24 && domain << 28 && target == 7) {
>
> gcc is giving following compiler warning [1]. Most probably intention is
> the compare, but it is not clear, can you please check?
>

I have not seen this warning but I will add parenthesis for avoiding
ambiguity.

Thanks


>
> [1]
> .../drivers/net/nfp/nfp_nspu.c: In function ‘nfp_nspu_set_bar_from_symbl’:
> .../drivers/net/nfp/nfp_nspu.c:446:29: error: ‘<<’ in boolean context,
> did you mean ‘<’ ? [-Werror=int-in-bool-context]
>   if (domain >= 24 && domain << 28 && target == 7) {
>                       ~~~~~~~^~~~~
>
> > +             addr = 1ULL << 37 | addr | ((uint64_t)domain & 0x3) << 35;
> > +     } else {
> > +             addr = 1ULL << 39 | addr | ((uint64_t)domain & 0x3f) << 32;
> > +             if (target == -7)
> > +                     target = 7;
> > +     }
>

[dpdk-dev] [PATCH 07/16] nfp: add fw upload logic

[Alejandro Lucero]

PMD will use this function for uploading the firmware. First, a
symbol resolution is done for finding out if there is a firmware
already there. If not, a NFP reset is called before using NSPU
fw upload code.

PMD PF probe function is now using this logic.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c  |  3 +++
 drivers/net/nfp/nfp_nspu.c | 48 +++++++++++++++++++++++++++++++++++++++++++---
 drivers/net/nfp/nfp_nspu.h |  6 +-----
 3 files changed, 49 insertions(+), 8 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 1890a4a..c0d5f58 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2639,6 +2639,7 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
+	uint64_t offset_symbol;
 	int major, minor;
 
 	if (!dev)
@@ -2669,6 +2670,8 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		goto no_abi;
 	}
 
+	nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports", &offset_symbol);
+
 	/* No port is created yet */
 
 no_abi:
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 4296be1..f4fee71 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -21,6 +21,9 @@
 /* NFP target for NSP access */
 #define NFP_NSP_TARGET   7
 
+/* Expansion BARs for mapping PF vnic BARs */
+#define NFP_NET_PF_CFG_EXP_BAR    6
+
 /*
  * This is an NFP internal address used for configuring properly an NFP
  * expansion BAR.
@@ -297,7 +300,7 @@
 	return ret;
 }
 
-int
+static int
 nfp_fw_reset(nspu_desc_t *nspu_desc)
 {
 	int res;
@@ -313,7 +316,7 @@
 #define DEFAULT_FW_PATH       "/lib/firmware/netronome"
 #define DEFAULT_FW_FILENAME   "nic_dpdk_default.nffw"
 
-int
+static int
 nfp_fw_upload(nspu_desc_t *nspu_desc)
 {
 	int fw_f;
@@ -391,7 +394,7 @@
  * a PCI BAR window. NFP expansion BARs are used in this regard through
  * the NSPU interface.
  */
-int
+static int
 nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
 			    uint32_t expbar, uint64_t *pcie_offset,
 			    ssize_t *size)
@@ -454,3 +457,42 @@
 	free(sym_buf);
 	return ret;
 }
+
+int
+nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset)
+{
+	ssize_t bar0_sym_size;
+
+	/* If the symbol resolution works, it implies a firmware app
+	 * is already there.
+	 */
+	if (!nfp_nspu_set_bar_from_symbl(desc, sym, NFP_NET_PF_CFG_EXP_BAR,
+					 pcie_offset, &bar0_sym_size))
+		return 0;
+
+	/* No firmware app detected or not the right one */
+	RTE_LOG(INFO, PMD, "No firmware detected. Resetting NFP...\n");
+	if (nfp_fw_reset(desc) < 0) {
+		RTE_LOG(ERR, PMD, "nfp fw reset failed\n");
+		return -ENODEV;
+	}
+
+	RTE_LOG(INFO, PMD, "Reset done.\n");
+	RTE_LOG(INFO, PMD, "Uploading firmware...\n");
+
+	if (nfp_fw_upload(desc) < 0) {
+		RTE_LOG(ERR, PMD, "nfp fw upload failed\n");
+		return -ENODEV;
+	}
+
+	RTE_LOG(INFO, PMD, "Done.\n");
+
+	/* Now the symbol should be there */
+	if (nfp_nspu_set_bar_from_symbl(desc, sym, NFP_NET_PF_CFG_EXP_BAR,
+					pcie_offset, &bar0_sym_size)) {
+		RTE_LOG(ERR, PMD, "nfp PF BAR symbol resolution failed\n");
+		return -ENODEV;
+	}
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 7734b4f..c439700 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -72,8 +72,4 @@
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
-int nfp_fw_reset(nspu_desc_t *nspu_desc);
-int nfp_fw_upload(nspu_desc_t *nspu_desc);
-int nfp_nspu_set_bar_from_symbl(nspu_desc_t *desc, const char *symbl,
-				uint32_t expbar, uint64_t *pcie_offset,
-				ssize_t *size);
+int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
-- 
1.9.1

[dpdk-dev] [PATCH 08/16] nfp: add support for vnic config bar mapping

[Alejandro Lucero]

NFP vNICs use a subset of PCI device BARs. vNIC config bar depends on
firmware symbol defining how to map it through a NFP expansion bar.

This patch adds a NSPU API function for getting a vNIC config bar
mapped through a expansion bar giving a firmware symbol. The PMD will
use the PCI bar offset returned for accessing the vNIC bar.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 13 +++++++++++++
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index f4fee71..f68bae6 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -496,3 +496,16 @@
 
 	return 0;
 }
+
+int
+nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset)
+{
+	ssize_t bar0_sym_size;
+
+	if (nfp_nspu_set_bar_from_symbl(desc, "_pf0_net_bar0",
+					NFP_NET_PF_CFG_EXP_BAR,
+					pcie_offset, &bar0_sym_size))
+		return -ENODEV;
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index c439700..8211f92 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -73,3 +73,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 		  int exp_bar, void *exp_bar_cfg_base, void *exp_bar_mmap);
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
+int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
-- 
1.9.1

[dpdk-dev] [PATCH 09/16] nfp: add support for vNIC rx/tx bar mappings

[Alejandro Lucero]

NFP vNICs use a subset of PCI device BARs. vNIC rx/tx bars point to
NFP hardware queues unit. Unline vNIC config bar, the NFP address is
always the same so the NFP expansion bar configuration always uses
the same hardcoded physical address.

This patch adds a NSPU API function for getting vNIC rx/tx bars
mapped through a expansion bar using that specific physical address.

The PMD will use the PCI bar offset returned for mapping the vNIC
rx/tx bars.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_nspu.c | 21 ++++++++++++++++++++-
 drivers/net/nfp/nfp_nspu.h |  1 +
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index f68bae6..df5af33 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -22,7 +22,8 @@
 #define NFP_NSP_TARGET   7
 
 /* Expansion BARs for mapping PF vnic BARs */
-#define NFP_NET_PF_CFG_EXP_BAR    6
+#define NFP_NET_PF_CFG_EXP_BAR          6
+#define NFP_NET_PF_HW_QUEUES_EXP_BAR    5
 
 /*
  * This is an NFP internal address used for configuring properly an NFP
@@ -509,3 +510,21 @@
 
 	return 0;
 }
+
+/*
+ * This is a hardcoded fixed NFP internal CPP bus address for the hw queues unit
+ * inside the PCIE island.
+ */
+#define NFP_CPP_PCIE_QUEUES ((uint64_t)(1ULL << 39) |  0x80000 | \
+			     ((uint64_t)0x4 & 0x3f) << 32)
+
+/* Configure a specific NFP expansion bar for accessing the vNIC rx/tx BARs */
+void
+nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset)
+{
+	nfp_nspu_mem_bar_cfg(desc, NFP_NET_PF_HW_QUEUES_EXP_BAR, 0,
+			     NFP_CPP_PCIE_QUEUES, pcie_offset);
+
+	/* This is the pcie offset to use by the host */
+	*pcie_offset |= ((NFP_NET_PF_HW_QUEUES_EXP_BAR & 0x7) << (27 - 3));
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 8211f92..4b09d4f 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -74,3 +74,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_get_abi_version(nspu_desc_t *desc, int *major, int *minor);
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
+void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
-- 
1.9.1

[dpdk-dev] [PATCH 10/16] nfp: support pf devices inside pmd initialization

[Alejandro Lucero]

nfp_net_init is where a dpdk port related to a eth_dev is initialized.
NFP VF vNICs use VF PCI BARs as they come after SRIOV is enabled. But for
NFP PF vNIC just a subset of PF PCI BARs are used.

This patch adds support for mapping the right PCI BAR subsets for the PF
vNIC. It uses the NSPU API functions introduced previously for configuring
NFP expansion bars.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 56 ++++++++++++++++++++++++++++++++++++-------
 drivers/net/nfp/nfp_net_pmd.h |  4 ++++
 2 files changed, 52 insertions(+), 8 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index c0d5f58..7c23b7a 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -55,12 +55,11 @@
 #include <rte_alarm.h>
 #include <rte_spinlock.h>
 
+#include "nfp_nfpu.h"
 #include "nfp_net_pmd.h"
 #include "nfp_net_logs.h"
 #include "nfp_net_ctrl.h"
 
-#include "nfp_nfpu.h"
-
 /* Prototypes */
 static void nfp_net_close(struct rte_eth_dev *dev);
 static int nfp_net_configure(struct rte_eth_dev *dev);
@@ -101,7 +100,7 @@ static uint16_t nfp_net_xmit_pkts(void *tx_queue, struct rte_mbuf **tx_pkts,
  * happen to be at the same offset on the NFP6000 and the NFP3200 so
  * we use a single macro here.
  */
-#define NFP_PCIE_QUEUE(_q)	(0x80000 + (0x800 * ((_q) & 0xff)))
+#define NFP_PCIE_QUEUE(_q)	(0x800 * ((_q) & 0xff))
 
 /* Maximum value which can be added to a queue with one transaction */
 #define NFP_QCP_MAX_ADD	0x7f
@@ -2496,10 +2495,13 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	struct rte_pci_device *pci_dev;
 	struct nfp_net_hw *hw;
 
-	uint32_t tx_bar_off, rx_bar_off;
+	uint64_t tx_bar_off = 0, rx_bar_off = 0;
 	uint32_t start_q;
 	int stride = 4;
 
+	nspu_desc_t *nspu_desc = NULL;
+	uint64_t bar_offset;
+
 	PMD_INIT_FUNC_TRACE();
 
 	hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
@@ -2532,11 +2534,33 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 			"hw->ctrl_bar is NULL. BAR0 not configured\n");
 		return -ENODEV;
 	}
+
+	/* Is this a PF device? */
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		nspu_desc = hw->nspu_desc;
+
+		if (nfp_nsp_map_ctrl_bar(nspu_desc, &bar_offset) != 0) {
+			/*
+			 * A firmware should be there after PF probe so this
+			 * should not happen.
+			 */
+			RTE_LOG(ERR, PMD, "PF BAR symbol resolution failed\n");
+			return -ENODEV;
+		}
+
+		/* vNIC PF control BAR is a subset of PF PCI device BAR */
+		hw->ctrl_bar += bar_offset;
+		PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
+	}
+
 	hw->max_rx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_RXRINGS);
 	hw->max_tx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_TXRINGS);
 
 	/* Work out where in the BAR the queues start. */
 	switch (pci_dev->id.device_id) {
+	case PCI_DEVICE_ID_NFP4000_PF_NIC:
+	case PCI_DEVICE_ID_NFP6000_PF_NIC:
 	case PCI_DEVICE_ID_NFP6000_VF_NIC:
 		start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
 		tx_bar_off = NFP_PCIE_QUEUE(start_q);
@@ -2548,11 +2572,27 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENODEV;
 	}
 
-	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%08x", tx_bar_off);
-	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%08x", rx_bar_off);
+	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%" PRIx64 "\n", tx_bar_off);
+	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%" PRIx64 "\n", rx_bar_off);
 
-	hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
-	hw->rx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + rx_bar_off;
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		/* configure access to tx/rx vNIC BARs */
+		nfp_nsp_map_queues_bar(nspu_desc, &bar_offset);
+		PMD_INIT_LOG(DEBUG, "tx/rx bar_offset: %" PRIx64 "\n",
+				    bar_offset);
+		hw->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
+
+		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
+		hw->hw_queues += bar_offset;
+		hw->tx_bar = hw->hw_queues + tx_bar_off;
+		hw->rx_bar = hw->hw_queues + rx_bar_off;
+	} else {
+		hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
+			     tx_bar_off;
+		hw->rx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
+			     rx_bar_off;
+	}
 
 	PMD_INIT_LOG(DEBUG, "ctrl_bar: %p, tx_bar: %p, rx_bar: %p",
 		     hw->ctrl_bar, hw->tx_bar, hw->rx_bar);
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index 3818130..0f902fc 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -437,6 +437,10 @@ struct nfp_net_hw {
 	struct nfp_cpp_area *rx_area;
 	struct nfp_cpp_area *msix_area;
 #endif
+	uint8_t *hw_queues;
+	uint8_t is_pf;
+	nspu_desc_t *nspu_desc;
+	nfpu_desc_t *nfpu_desc;
 };
 
 struct nfp_net_adapter {
-- 
1.9.1

[dpdk-dev] [PATCH 11/16] nfp: allocate eth_dev from pf probe function

[Alejandro Lucero]

NFP can support several physical ports per PF device. Depending on
firmware info, one or more eth_dev objects will need to be created.

This patch adds the call to create just one eth_dev by now with future
commits supporting the multiport option. Once the eth_dev has been
created, probe function invokes pmd initialization with the new eth_dev.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 50 +++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 42 insertions(+), 8 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 7c23b7a..6005e41 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2677,13 +2677,16 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			    struct rte_pci_device *dev)
 {
+	struct rte_eth_dev *eth_dev;
+	struct nfp_net_hw *hw;
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
 	uint64_t offset_symbol;
 	int major, minor;
+	int ret = -ENODEV;
 
 	if (!dev)
-		return -ENODEV;
+		return ret;
 
 	nfpu_desc = rte_malloc("nfp nfpu", sizeof(nfpu_desc_t), 0);
 	if (!nfpu_desc)
@@ -2697,29 +2700,60 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 	nspu_desc = nfpu_desc->nspu;
 
-
 	/* Check NSP ABI version */
 	if (nfp_nsp_get_abi_version(nspu_desc, &major, &minor) < 0) {
 		RTE_LOG(INFO, PMD, "NFP NSP not present\n");
-		goto no_abi;
+		goto error;
 	}
 	PMD_INIT_LOG(INFO, "nspu ABI version: %d.%d\n", major, minor);
 
 	if (minor < 20) {
 		RTE_LOG(INFO, PMD, "NFP NSP ABI version too old. Required 0.20 or higher\n");
-		goto no_abi;
+		goto error;
 	}
 
-	nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports", &offset_symbol);
+	ret = nfp_nsp_fw_setup(nspu_desc, "nfd_cfg_pf0_num_ports",
+			       &offset_symbol);
+	if (ret)
+		goto error;
 
-	/* No port is created yet */
+	eth_dev = rte_eth_dev_allocate(dev->device.name);
+	if (!eth_dev) {
+		ret = -ENODEV;
+		goto error;
+	}
 
-no_abi:
+	eth_dev->data->dev_private = rte_zmalloc("nfp_pf_port",
+						 sizeof(struct nfp_net_adapter),
+						 RTE_CACHE_LINE_SIZE);
+	if (!eth_dev->data->dev_private) {
+		rte_eth_dev_release_port(eth_dev);
+		ret = -ENODEV;
+		goto error;
+	}
+
+	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private);
+	hw->nspu_desc = nspu_desc;
+	hw->nfpu_desc = nfpu_desc;
+	hw->is_pf = 1;
+
+	eth_dev->device = &dev->device;
+	rte_eth_copy_pci_info(eth_dev, dev);
+
+	ret = nfp_net_init(eth_dev);
+
+	if (!ret)
+		return 0;
+
+	/* something went wrong */
+	rte_eth_dev_release_port(eth_dev);
+
+error:
 	nfpu_close(nfpu_desc);
 nfpu_error:
 	rte_free(nfpu_desc);
 
-	return -ENODEV;
+	return ret;
 }
 
 static const struct rte_pci_id pci_id_nfp_pf_net_map[] = {
-- 
1.9.1

[dpdk-dev] [PATCH 12/16] nfp: support pf multiport

[Alejandro Lucero]

A NFP PF PCI device can have several physical ports, up to 8. Because
DPDK core creates one eth_dev per PCI device, nfp pf probe function
is used. Number of PF ports is obtained from firmware symbol using
NSPU API. Inside PF probe function an eth_dev per port is created and
nfp_net_init invoked for each port.

There are some limitations reagarding multiport: rx interrupts and
device hotplug are not supported.

Interrupts are handled with the VFIO or UIO drivers help. Those
drivers just know about PCI devices, so it is not possible, without
changing how DPDK handles interrupts, manage interrupts assigned to
different PF ports.

About hotplug, the problem is this functionality is based on a PCI
device, and although device pluging is possible, which would add as
many ports as supported by firmware, unpluging is based on device name
linked to a eth_dev, and device name has a suffix now (_portX, with X
being the port index) which DPDK core is not aware of. While rx
interrupts with multiport could be likely solved with some layer of
indirection, hotplug would require changes to DPDK core.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c      | 197 ++++++++++++++++++++++++++++++++---------
 drivers/net/nfp/nfp_net_ctrl.h |   3 +
 drivers/net/nfp/nfp_net_pmd.h  |   2 +
 3 files changed, 162 insertions(+), 40 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 6005e41..f9ce204 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -687,6 +687,11 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 
 	/* check and configure queue intr-vector mapping */
 	if (dev->data->dev_conf.intr_conf.rxq != 0) {
+		if (hw->pf_multiport_enabled) {
+			PMD_INIT_LOG(ERR, "PMD rx interrupt is not supported "
+					  "with NFP multiport PF");
+				return -EINVAL;
+		}
 		if (intr_handle->type == RTE_INTR_HANDLE_UIO) {
 			/*
 			 * Better not to share LSC with RX interrupts.
@@ -2489,11 +2494,40 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	.rx_queue_intr_disable  = nfp_rx_queue_intr_disable,
 };
 
+/*
+ * All eth_dev created got its private data, but before nfp_net_init, that
+ * private data is referencing private data for all the PF ports. This is due
+ * to how the vNIC bars are mapped based on first port, so all ports need info
+ * about port 0 private data. Inside nfp_net_init the private data pointer is
+ * changed to the right address for each port once the bars have been mapped.
+ */
+static int
+get_pf_port_number(char *name)
+{
+	char *pf_str = name;
+	int size = 0;
+
+	while ((*pf_str != '_') && (*pf_str != '\0') && (size++ < 30))
+		pf_str++;
+
+	if (size == 30)
+		/*
+		 * This should not happen at all and it would mean major
+		 * implementation fault.
+		 */
+		rte_panic("nfp_net: problem with pf device name\n");
+
+	/* Expecting _portX with X within [0,7] */
+	pf_str += 5;
+
+	return (int)strtol(pf_str, NULL, 10);
+}
+
 static int
 nfp_net_init(struct rte_eth_dev *eth_dev)
 {
 	struct rte_pci_device *pci_dev;
-	struct nfp_net_hw *hw;
+	struct nfp_net_hw *hw, *hwport0;
 
 	uint64_t tx_bar_off = 0, rx_bar_off = 0;
 	uint32_t start_q;
@@ -2501,10 +2535,32 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 	nspu_desc_t *nspu_desc = NULL;
 	uint64_t bar_offset;
+	int port = 0;
 
 	PMD_INIT_FUNC_TRACE();
 
-	hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
+
+	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
+	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+		port = get_pf_port_number(eth_dev->data->name);
+		if (port < 0 || port > 7) {
+			RTE_LOG(ERR, PMD, "Port value is wrong\n");
+			return -ENODEV;
+		}
+
+		PMD_INIT_LOG(DEBUG, "Working with PF port value %d\n", port);
+
+		/* This points to port 0 private data */
+		hwport0 = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+
+		/* This points to the specific port private data */
+		hw = &hwport0[port];
+		hw->pf_port_idx = port;
+	} else {
+		hw = NFP_NET_DEV_PRIVATE_TO_HW(eth_dev->data->dev_private);
+		hwport0 = 0;
+	}
 
 	eth_dev->dev_ops = &nfp_net_eth_dev_ops;
 	eth_dev->rx_pkt_burst = &nfp_net_recv_pkts;
@@ -2514,9 +2570,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
 		return 0;
 
-	pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
 	rte_eth_copy_pci_info(eth_dev, pci_dev);
-	eth_dev->data->dev_flags |= RTE_ETH_DEV_DETACHABLE;
+	/* hotplug is not possible with multiport PF */
+	if (!hw->pf_multiport_enabled)
+		eth_dev->data->dev_flags |= RTE_ETH_DEV_DETACHABLE;
 
 	hw->device_id = pci_dev->id.device_id;
 	hw->vendor_id = pci_dev->id.vendor_id;
@@ -2535,9 +2592,7 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENODEV;
 	}
 
-	/* Is this a PF device? */
-	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
-	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+	if (hw->is_pf && port == 0) {
 		nspu_desc = hw->nspu_desc;
 
 		if (nfp_nsp_map_ctrl_bar(nspu_desc, &bar_offset) != 0) {
@@ -2551,9 +2606,19 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 		/* vNIC PF control BAR is a subset of PF PCI device BAR */
 		hw->ctrl_bar += bar_offset;
-		PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
 	}
 
+	if (port > 0) {
+		if (!hwport0->ctrl_bar)
+			return -ENODEV;
+
+		/* address based on port0 offset */
+		hw->ctrl_bar = hwport0->ctrl_bar +
+			       (port * NFP_PF_CSR_SLICE_SIZE);
+	}
+
+	PMD_INIT_LOG(DEBUG, "ctrl bar: %p\n", hw->ctrl_bar);
+
 	hw->max_rx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_RXRINGS);
 	hw->max_tx_queues = nn_cfg_readl(hw, NFP_NET_CFG_MAX_TXRINGS);
 
@@ -2575,18 +2640,21 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	PMD_INIT_LOG(DEBUG, "tx_bar_off: 0x%" PRIx64 "\n", tx_bar_off);
 	PMD_INIT_LOG(DEBUG, "rx_bar_off: 0x%" PRIx64 "\n", rx_bar_off);
 
-	if ((pci_dev->id.device_id == PCI_DEVICE_ID_NFP4000_PF_NIC) ||
-	    (pci_dev->id.device_id == PCI_DEVICE_ID_NFP6000_PF_NIC)) {
+	if (hw->is_pf && port == 0) {
 		/* configure access to tx/rx vNIC BARs */
 		nfp_nsp_map_queues_bar(nspu_desc, &bar_offset);
 		PMD_INIT_LOG(DEBUG, "tx/rx bar_offset: %" PRIx64 "\n",
 				    bar_offset);
-		hw->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
+		hwport0->hw_queues = (uint8_t *)pci_dev->mem_resource[0].addr;
 
 		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
-		hw->hw_queues += bar_offset;
-		hw->tx_bar = hw->hw_queues + tx_bar_off;
-		hw->rx_bar = hw->hw_queues + rx_bar_off;
+		hwport0->hw_queues += bar_offset;
+	}
+
+	if (hw->is_pf) {
+		hw->tx_bar = hwport0->hw_queues + tx_bar_off;
+		hw->rx_bar = hwport0->hw_queues + rx_bar_off;
+		eth_dev->data->dev_private = hw;
 	} else {
 		hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr +
 			     tx_bar_off;
@@ -2674,16 +2742,77 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	return 0;
 }
 
-static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
-			    struct rte_pci_device *dev)
+static int
+nfp_pf_create_dev(struct rte_pci_device *dev, int port, int ports,
+		  nfpu_desc_t *nfpu_desc, void **priv)
 {
 	struct rte_eth_dev *eth_dev;
 	struct nfp_net_hw *hw;
+	char *port_name;
+	int ret;
+
+	port_name = rte_zmalloc("nfp_pf_port_name", 100, 0);
+	if (!port_name)
+		return -ENOMEM;
+
+	if (ports > 1)
+		sprintf(port_name, "%s_port%d", dev->device.name, port);
+	else
+		sprintf(port_name, "%s", dev->device.name);
+
+	eth_dev = rte_eth_dev_allocate(port_name);
+	if (!eth_dev)
+		return -ENOMEM;
+
+	if (port == 0) {
+		*priv = rte_zmalloc(port_name,
+				    sizeof(struct nfp_net_adapter) * ports,
+				    RTE_CACHE_LINE_SIZE);
+		if (!*priv) {
+			rte_eth_dev_release_port(eth_dev);
+			return -ENOMEM;
+		}
+	}
+
+	eth_dev->data->dev_private = *priv;
+
+	/*
+	 * dev_private pointing to port0 dev_private because we need
+	 * to configure vNIC bars based on port0 at nfp_net_init.
+	 * Then dev_private is adjusted per port.
+	 */
+	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private) + port;
+	hw->nspu_desc = nfpu_desc->nspu;
+	hw->nfpu_desc = nfpu_desc;
+	hw->is_pf = 1;
+	if (ports > 1)
+		hw->pf_multiport_enabled = 1;
+
+	eth_dev->device = &dev->device;
+	rte_eth_copy_pci_info(eth_dev, dev);
+
+	ret = nfp_net_init(eth_dev);
+
+	if (ret)
+		rte_eth_dev_release_port(eth_dev);
+
+	rte_free(port_name);
+
+	return ret;
+}
+
+static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+			    struct rte_pci_device *dev)
+{
 	nfpu_desc_t *nfpu_desc;
 	nspu_desc_t *nspu_desc;
 	uint64_t offset_symbol;
+	uint8_t *bar_offset;
 	int major, minor;
+	int total_ports;
+	void *priv = 0;
 	int ret = -ENODEV;
+	int i;
 
 	if (!dev)
 		return ret;
@@ -2717,36 +2846,24 @@ static int nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret)
 		goto error;
 
-	eth_dev = rte_eth_dev_allocate(dev->device.name);
-	if (!eth_dev) {
-		ret = -ENODEV;
-		goto error;
-	}
+	bar_offset = (uint8_t *)dev->mem_resource[0].addr;
+	bar_offset += offset_symbol;
+	total_ports = (uint32_t)*bar_offset;
+	PMD_INIT_LOG(INFO, "Total pf ports: %d\n", total_ports);
 
-	eth_dev->data->dev_private = rte_zmalloc("nfp_pf_port",
-						 sizeof(struct nfp_net_adapter),
-						 RTE_CACHE_LINE_SIZE);
-	if (!eth_dev->data->dev_private) {
-		rte_eth_dev_release_port(eth_dev);
+	if (total_ports <= 0 || total_ports > 8) {
+		RTE_LOG(ERR, PMD, "nfd_cfg_pf0_num_ports symbol with wrong value");
 		ret = -ENODEV;
 		goto error;
 	}
 
-	hw = (struct nfp_net_hw *)(eth_dev->data->dev_private);
-	hw->nspu_desc = nspu_desc;
-	hw->nfpu_desc = nfpu_desc;
-	hw->is_pf = 1;
-
-	eth_dev->device = &dev->device;
-	rte_eth_copy_pci_info(eth_dev, dev);
-
-	ret = nfp_net_init(eth_dev);
-
-	if (!ret)
-		return 0;
+	for (i = 0; i < total_ports; i++) {
+		ret = nfp_pf_create_dev(dev, i, total_ports, nfpu_desc, &priv);
+		if (ret)
+			goto error;
+	}
 
-	/* something went wrong */
-	rte_eth_dev_release_port(eth_dev);
+	return 0;
 
 error:
 	nfpu_close(nfpu_desc);
diff --git a/drivers/net/nfp/nfp_net_ctrl.h b/drivers/net/nfp/nfp_net_ctrl.h
index c1cba0e..1ebd99c 100644
--- a/drivers/net/nfp/nfp_net_ctrl.h
+++ b/drivers/net/nfp/nfp_net_ctrl.h
@@ -334,6 +334,9 @@
 #define NFP_NET_CFG_RXR_STATS(_x)       (NFP_NET_CFG_RXR_STATS_BASE + \
 					 ((_x) * 0x10))
 
+/* PF multiport offset */
+#define NFP_PF_CSR_SLICE_SIZE	(32 * 1024)
+
 #endif /* _NFP_NET_CTRL_H_ */
 /*
  * Local variables:
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index 0f902fc..d7e38d4 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -439,6 +439,8 @@ struct nfp_net_hw {
 #endif
 	uint8_t *hw_queues;
 	uint8_t is_pf;
+	uint8_t pf_port_idx;
+	uint8_t pf_multiport_enabled;
 	nspu_desc_t *nspu_desc;
 	nfpu_desc_t *nfpu_desc;
 };
-- 
1.9.1

[dpdk-dev] [PATCH 13/16] nfp: add nsp support for hw link configuration

[Alejandro Lucero]

Adding a new NSPU command for being able to read and write the ethernet
port table from/to the NFP. This will allow the PMD to put the Link up
or down when a port is started or stopped. Until now, this was performed
by the firmware independently of PMD functionality.

The ethernet port table has also some other useful information that will
be used in further commits.

Usually NSPU is used at device probe time and that is sequential code
execution. However, reading and writing the NFP eth table can be done at
different times and from different cores, and it implies it could happen
a concurrent access. A spinlock is added to the global nspu object for
protecting the NFP and avoiding the concurrent access.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net_eth.h | 82 +++++++++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/nfp_nspu.c    | 79 +++++++++++++++++++++++++++++++++++++++--
 drivers/net/nfp/nfp_nspu.h    |  4 +++
 3 files changed, 162 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/nfp/nfp_net_eth.h

diff --git a/drivers/net/nfp/nfp_net_eth.h b/drivers/net/nfp/nfp_net_eth.h
new file mode 100644
index 0000000..af57f03
--- /dev/null
+++ b/drivers/net/nfp/nfp_net_eth.h
@@ -0,0 +1,82 @@
+/*
+ * Copyright (c) 2017 Netronome Systems, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *  this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *  notice, this list of conditions and the following disclaimer in the
+ *  documentation and/or other materials provided with the distribution
+ *
+ * 3. Neither the name of the copyright holder nor the names of its
+ *  contributors may be used to endorse or promote products derived from this
+ *  software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * vim:shiftwidth=8:noexpandtab
+ *
+ * @file dpdk/pmd/nfp_net_eth.h
+ *
+ * Netronome NFP_NET PDM driver
+ */
+
+union eth_table_entry {
+	struct {
+		uint64_t port;
+		uint64_t state;
+		uint8_t mac_addr[6];
+		uint8_t resv[2];
+		uint64_t control;
+	};
+	uint64_t raw[4];
+};
+
+#ifndef BIT_ULL
+#define BIT_ULL(a) (1ULL << (a))
+#endif
+
+#define NSP_ETH_NBI_PORT_COUNT          24
+#define NSP_ETH_MAX_COUNT               (2 * NSP_ETH_NBI_PORT_COUNT)
+#define NSP_ETH_TABLE_SIZE   (NSP_ETH_MAX_COUNT * sizeof(union eth_table_entry))
+
+#define NSP_ETH_PORT_LANES              0xf
+#define NSP_ETH_PORT_INDEX              0xff00
+#define NSP_ETH_PORT_LABEL              0x3f000000000000
+#define NSP_ETH_PORT_PHYLABEL           0xfc0000000000000
+
+#define NSP_ETH_PORT_LANES_MASK         rte_cpu_to_le_64(NSP_ETH_PORT_LANES)
+
+#define NSP_ETH_STATE_CONFIGURED        BIT_ULL(0)
+#define NSP_ETH_STATE_ENABLED           BIT_ULL(1)
+#define NSP_ETH_STATE_TX_ENABLED        BIT_ULL(2)
+#define NSP_ETH_STATE_RX_ENABLED        BIT_ULL(3)
+#define NSP_ETH_STATE_RATE              0xf00
+#define NSP_ETH_STATE_INTERFACE         0xff000
+#define NSP_ETH_STATE_MEDIA             0x300000
+#define NSP_ETH_STATE_OVRD_CHNG         BIT_ULL(22)
+#define NSP_ETH_STATE_ANEG              0x3800000
+
+#define NSP_ETH_CTRL_CONFIGURED         BIT_ULL(0)
+#define NSP_ETH_CTRL_ENABLED            BIT_ULL(1)
+#define NSP_ETH_CTRL_TX_ENABLED         BIT_ULL(2)
+#define NSP_ETH_CTRL_RX_ENABLED         BIT_ULL(3)
+#define NSP_ETH_CTRL_SET_RATE           BIT_ULL(4)
+#define NSP_ETH_CTRL_SET_LANES          BIT_ULL(5)
+#define NSP_ETH_CTRL_SET_ANEG           BIT_ULL(6)
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index df5af33..29325e6 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -8,8 +8,10 @@
 #include <fcntl.h>
 
 #include <rte_log.h>
+#include <rte_byteorder.h>
 
 #include "nfp_nfpu.h"
+#include "nfp_net_eth.h"
 
 #define CFG_EXP_BAR_ADDR_SZ     1
 #define CFG_EXP_BAR_MAP_TYPE	1
@@ -45,9 +47,11 @@
 #define NSP_STATUS_MINOR(x)      (int)(((x) >> 32) & 0xfff)
 
 /* NSP commands */
-#define NSP_CMD_RESET          1
-#define NSP_CMD_FW_LOAD        6
-#define NSP_CMD_GET_SYMBOL     14
+#define NSP_CMD_RESET                   1
+#define NSP_CMD_FW_LOAD                 6
+#define NSP_CMD_READ_ETH_TABLE          7
+#define NSP_CMD_WRITE_ETH_TABLE         8
+#define NSP_CMD_GET_SYMBOL             14
 
 #define NSP_BUFFER_CFG_SIZE_MASK	(0xff)
 
@@ -528,3 +532,72 @@
 	/* This is the pcie offset to use by the host */
 	*pcie_offset |= ((NFP_NET_PF_HW_QUEUES_EXP_BAR & 0x7) << (27 - 3));
 }
+
+int
+nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up)
+{
+	union eth_table_entry *entries, *entry;
+	int modified;
+	int ret, idx;
+	int i;
+
+	idx = port;
+
+	RTE_LOG(INFO, PMD, "Hw ethernet port %d configure...\n", port);
+	rte_spinlock_lock(&desc->nsp_lock);
+	entries = malloc(NSP_ETH_TABLE_SIZE);
+	if (!entries) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return -ENOMEM;
+	}
+
+	ret = nspu_command(desc, NSP_CMD_READ_ETH_TABLE, 1, 0, entries,
+			   NSP_ETH_TABLE_SIZE, 0);
+	if (ret) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return ret;
+	}
+
+	entry = entries;
+
+	for (i = 0; i < NSP_ETH_MAX_COUNT; i++) {
+		/* ports in use do not appear sequentially in the table */
+		if (!(entry->port & NSP_ETH_PORT_LANES_MASK)) {
+			/* entry not in use */
+			entry++;
+			continue;
+		}
+		if (idx == 0)
+			break;
+		idx--;
+		entry++;
+	}
+
+	if (i == NSP_ETH_MAX_COUNT) {
+		rte_spinlock_unlock(&desc->nsp_lock);
+		return -EINVAL;
+	}
+
+	if (up && !(entry->state & NSP_ETH_STATE_CONFIGURED)) {
+		entry->control |= NSP_ETH_STATE_CONFIGURED;
+		modified = 1;
+	}
+
+	if (!up && (entry->state & NSP_ETH_STATE_CONFIGURED)) {
+		entry->control &= ~NSP_ETH_STATE_CONFIGURED;
+		modified = 1;
+	}
+
+	if (modified) {
+		ret = nspu_command(desc, NSP_CMD_WRITE_ETH_TABLE, 0, 1, entries,
+				   0, NSP_ETH_TABLE_SIZE);
+		if (!ret)
+			RTE_LOG(INFO, PMD,
+				"Hw ethernet port %d configure done\n", port);
+		else
+			RTE_LOG(INFO, PMD,
+				"Hw ethernet port %d configure failed\n", port);
+	}
+	rte_spinlock_unlock(&desc->nsp_lock);
+	return ret;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 4b09d4f..4e58986 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -57,6 +57,8 @@
  * the fast path.
  */
 
+#include <rte_spinlock.h>
+
 typedef struct {
 	int nfp;        /* NFP device */
 	int pcie_bar;   /* PF PCI BAR to work with */
@@ -67,6 +69,7 @@
 	uint64_t windowsz; /* NSPU BAR window size */
 	void *cfg_base; /* Expansion BARs address */
 	void *mem_base; /* NSP interface */
+	rte_spinlock_t nsp_lock;
 } nspu_desc_t;
 
 int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
@@ -75,3 +78,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_fw_setup(nspu_desc_t *desc, const char *sym, uint64_t *pcie_offset);
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
+int nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up);
-- 
1.9.1

[dpdk-dev] [PATCH 14/16] nfp: add support for hw port link configuration

[Alejandro Lucero]

It is PMD task to configure the hardware port: link up when port started
and link down when port stopped. This is not required for VFs but it is
for PF ports.

A minor refactoring in PMD stop and close functions is done because the
Link down needs to happen just when device is stopped.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index f9ce204..aa611e1 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -737,6 +737,10 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 		goto error;
 	}
 
+	if (hw->is_pf)
+		/* Configure the physical port up */
+		nfp_nsp_eth_config(hw->nspu_desc, hw->pf_port_idx, 1);
+
 	hw->ctrl = new_ctrl;
 
 	return 0;
@@ -765,9 +769,12 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 nfp_net_stop(struct rte_eth_dev *dev)
 {
 	int i;
+	struct nfp_net_hw *hw;
 
 	PMD_INIT_LOG(DEBUG, "Stop");
 
+	hw = NFP_NET_DEV_PRIVATE_TO_HW(dev->data->dev_private);
+
 	nfp_net_disable_queues(dev);
 
 	/* Clear queues */
@@ -780,6 +787,10 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 		nfp_net_reset_rx_queue(
 			(struct nfp_net_rxq *)dev->data->rx_queues[i]);
 	}
+
+	if (hw->is_pf)
+		/* Configure the physical port down */
+		nfp_nsp_eth_config(hw->nspu_desc, hw->pf_port_idx, 0);
 }
 
 /* Reset and stop device. The device can not be restarted. */
@@ -788,6 +799,7 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 {
 	struct nfp_net_hw *hw;
 	struct rte_pci_device *pci_dev;
+	int i;
 
 	PMD_INIT_LOG(DEBUG, "Close");
 
@@ -799,7 +811,18 @@ static void nfp_net_read_mac(struct nfp_net_hw *hw)
 	 * threads/queues before calling the device close function.
 	 */
 
-	nfp_net_stop(dev);
+	nfp_net_disable_queues(dev);
+
+	/* Clear queues */
+	for (i = 0; i < dev->data->nb_tx_queues; i++) {
+		nfp_net_reset_tx_queue(
+			(struct nfp_net_txq *)dev->data->tx_queues[i]);
+	}
+
+	for (i = 0; i < dev->data->nb_rx_queues; i++) {
+		nfp_net_reset_rx_queue(
+			(struct nfp_net_rxq *)dev->data->rx_queues[i]);
+	}
 
 	rte_intr_disable(&pci_dev->intr_handle);
 	nn_cfg_writeb(hw, NFP_NET_CFG_LSC, 0xff);
-- 
1.9.1

[dpdk-dev] [PATCH 15/16] nfp: read pf port mac addr using nsp

[Alejandro Lucero]

During initialization, mac address is read from configuration bar. This is
the default option when using VFs.

This patch adds support for reading the mac address using the NSPU
interface when PMD works with the PF.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c     | 59 +++++++++++++++++++++++++++++++++++++++++--
 drivers/net/nfp/nfp_net_pmd.h |  1 +
 drivers/net/nfp/nfp_nspu.c    | 22 +++++++++++++++-
 drivers/net/nfp/nfp_nspu.h    |  2 ++
 4 files changed, 81 insertions(+), 3 deletions(-)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index aa611e1..9496b63 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -593,7 +593,55 @@ enum nfp_qcp_ptr {
 	hw->qcp_cfg = hw->tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 }
 
-static void nfp_net_read_mac(struct nfp_net_hw *hw)
+#define ETH_ADDR_LEN	6
+
+static void
+nfp_eth_copy_mac_reverse(uint8_t *dst, const uint8_t *src)
+{
+	int i;
+
+	for (i = 0; i < ETH_ADDR_LEN; i++)
+		dst[ETH_ADDR_LEN - i - 1] = src[i];
+}
+
+static int
+nfp_net_pf_read_mac(struct nfp_net_hw *hw, int port)
+{
+	union eth_table_entry *entry;
+	int idx, i;
+
+	idx = port;
+	entry = hw->eth_table;
+
+	/* Reading NFP ethernet table obtained before */
+	for (i = 0; i < NSP_ETH_MAX_COUNT; i++) {
+		if (!(entry->port & NSP_ETH_PORT_LANES_MASK)) {
+			/* port not in use */
+			entry++;
+			continue;
+		}
+		if (idx == 0)
+			break;
+		idx--;
+		entry++;
+	}
+
+	if (i == NSP_ETH_MAX_COUNT)
+		return -EINVAL;
+
+	/*
+	 * hw points to port0 private data. We need hw now pointing to
+	 * right port.
+	 */
+	hw += port;
+	nfp_eth_copy_mac_reverse((uint8_t *)&hw->mac_addr,
+				 (uint8_t *)&entry->mac_addr);
+
+	return 0;
+}
+
+static void
+nfp_net_vf_read_mac(struct nfp_net_hw *hw)
 {
 	uint32_t tmp;
 
@@ -2672,6 +2720,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 
 		/* vNIC PF tx/rx BARs are a subset of PF PCI device */
 		hwport0->hw_queues += bar_offset;
+
+		/* Lets seize the chance to read eth table from hw */
+		if (nfp_nsp_eth_read_table(nspu_desc, &hw->eth_table))
+			return -ENODEV;
 	}
 
 	if (hw->is_pf) {
@@ -2732,7 +2784,10 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 		return -ENOMEM;
 	}
 
-	nfp_net_read_mac(hw);
+	if (hw->is_pf)
+		nfp_net_pf_read_mac(hwport0, port);
+	else
+		nfp_net_vf_read_mac(hw);
 
 	if (!is_valid_assigned_ether_addr((struct ether_addr *)&hw->mac_addr)) {
 		/* Using random mac addresses for VFs */
diff --git a/drivers/net/nfp/nfp_net_pmd.h b/drivers/net/nfp/nfp_net_pmd.h
index d7e38d4..20ade1a 100644
--- a/drivers/net/nfp/nfp_net_pmd.h
+++ b/drivers/net/nfp/nfp_net_pmd.h
@@ -441,6 +441,7 @@ struct nfp_net_hw {
 	uint8_t is_pf;
 	uint8_t pf_port_idx;
 	uint8_t pf_multiport_enabled;
+	union eth_table_entry *eth_table;
 	nspu_desc_t *nspu_desc;
 	nfpu_desc_t *nfpu_desc;
 };
diff --git a/drivers/net/nfp/nfp_nspu.c b/drivers/net/nfp/nfp_nspu.c
index 29325e6..36f2b30 100644
--- a/drivers/net/nfp/nfp_nspu.c
+++ b/drivers/net/nfp/nfp_nspu.c
@@ -11,7 +11,6 @@
 #include <rte_byteorder.h>
 
 #include "nfp_nfpu.h"
-#include "nfp_net_eth.h"
 
 #define CFG_EXP_BAR_ADDR_SZ     1
 #define CFG_EXP_BAR_MAP_TYPE	1
@@ -601,3 +600,24 @@
 	rte_spinlock_unlock(&desc->nsp_lock);
 	return ret;
 }
+
+int
+nfp_nsp_eth_read_table(nspu_desc_t *desc, union eth_table_entry **table)
+{
+	int ret;
+
+	RTE_LOG(INFO, PMD, "Reading hw ethernet table...\n");
+	/* port 0 allocates the eth table and read it using NSPU */
+	*table = malloc(NSP_ETH_TABLE_SIZE);
+	if (!table)
+		return -ENOMEM;
+
+	ret = nspu_command(desc, NSP_CMD_READ_ETH_TABLE, 1, 0, *table,
+			   NSP_ETH_TABLE_SIZE, 0);
+	if (ret)
+		return ret;
+
+	RTE_LOG(INFO, PMD, "Done\n");
+
+	return 0;
+}
diff --git a/drivers/net/nfp/nfp_nspu.h b/drivers/net/nfp/nfp_nspu.h
index 4e58986..8c33835 100644
--- a/drivers/net/nfp/nfp_nspu.h
+++ b/drivers/net/nfp/nfp_nspu.h
@@ -58,6 +58,7 @@
  */
 
 #include <rte_spinlock.h>
+#include "nfp_net_eth.h"
 
 typedef struct {
 	int nfp;        /* NFP device */
@@ -79,3 +80,4 @@ int nfp_nspu_init(nspu_desc_t *desc, int nfp, int pcie_bar, size_t pcie_barsz,
 int nfp_nsp_map_ctrl_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 void nfp_nsp_map_queues_bar(nspu_desc_t *desc, uint64_t *pcie_offset);
 int nfp_nsp_eth_config(nspu_desc_t *desc, int port, int up);
+int nfp_nsp_eth_read_table(nspu_desc_t *desc, union eth_table_entry **table);
-- 
1.9.1

[dpdk-dev] [PATCH 16/16] doc: update nfp with pf support information

[Alejandro Lucero]

NFP PMF has now support for both, PF and VFs. This patch updates
the guide and give some information about implications.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 doc/guides/nics/nfp.rst | 71 ++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 52 insertions(+), 19 deletions(-)

diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst
index c732fb1..69ae952 100644
--- a/doc/guides/nics/nfp.rst
+++ b/doc/guides/nics/nfp.rst
@@ -1,5 +1,5 @@
 ..  BSD LICENSE
-    Copyright(c) 2015 Netronome Systems, Inc. All rights reserved.
+    Copyright(c) 2015-2017 Netronome Systems, Inc. All rights reserved.
     All rights reserved.
 
     Redistribution and use in source and binary forms, with or without
@@ -38,31 +38,31 @@ up to 400 Gbps.
 
 This document explains how to use DPDK with the Netronome Poll Mode
 Driver (PMD) supporting Netronome's Network Flow Processor 6xxx
-(NFP-6xxx).
+(NFP-6xxx) and Netronome's Flow Processor 4xxx (NFP-4xxx).
 
-Currently the driver supports virtual functions (VFs) only.
+NFP is a SRIOV capable device and the PMD driver supports the physical
+function (PF) and virtual functions (VFs).
 
 Dependencies
 ------------
 
-Before using the Netronome's DPDK PMD some NFP-6xxx configuration,
+Before using the Netronome's DPDK PMD some NFP configuration,
 which is not related to DPDK, is required. The system requires
-installation of **Netronome's BSP (Board Support Package)** which includes
-Linux drivers, programs and libraries.
+installation of **Netronome's BSP (Board Support Package)** along
+with some specific NFP firmware application.
 
-If you have a NFP-6xxx device you should already have the code and
-documentation for doing this configuration. Contact
+If you have a NFP device you should already have the code and
+documentation for doing all this configuration. Contact
 **support@netronome.com** to obtain the latest available firmware.
 
-The NFP Linux kernel drivers (including the required PF driver for the
-NFP) are available on Github at
+The NFP Linux netdev kernel driver for VFs is part of vanilla kernel
+since kernel vesion 4.5, and support for the PF since kernel version
+4.11. Support for older kernels can be obtained on Github at
 **https://github.com/Netronome/nfp-drv-kmods** along with build
 instructions.
 
-DPDK runs in userspace and PMDs uses the Linux kernel UIO interface to
-allow access to physical devices from userspace. The NFP PMD requires
-the **igb_uio** UIO driver, available with DPDK, to perform correct
-initialization.
+NFP PMD needs to be used along with UIO **igb_uio** or VFIO (vfio-pci)
+Linux kernel driver.
 
 Building the software
 ---------------------
@@ -71,7 +71,7 @@ Netronome's PMD code is provided in the **drivers/net/nfp** directory.
 Although NFP PMD has Netronome´s BSP dependencies, it is possible to
 compile it along with other DPDK PMDs even if no BSP was installed before.
 Of course, a DPDK app will require such a BSP installed for using the
-NFP PMD.
+NFP PMD, along with a specific NFP firmware application.
 
 Default PMD configuration is at **common_linuxapp configuration** file:
 
@@ -88,13 +88,46 @@ Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and
 for details.
 
 
+Using the PF
+------------
+
+NFP PMD has support for using the NFP PF as another DPDK port, but it has not
+any functionality for controlling VFs. In fact, it is not possible to use the
+PMD with the VFs if the PF is being used by DPDK, this is, NFP PF bound to
+igb_uio or vfio-pci kernel drivers. Future DPDK version will have a PMD able
+to work with the PF and VFs at the same time and with the PF implementing VF
+management along wih other PF-only functionalities/offloads.
+
+The PMD PF has extra work to do which will delay the DPDK app initialization
+like checking if a firmware is already available in the device, uploading the
+firmware if necessary, and configure the Link state properly when starting or
+stopping a PF port. Note that firmware upload is not always necessary which is
+the main delay for NFP PF PMD initialization.
+
+PF multiport support
+--------------------
+
+Some NFP cards support several physical ports with just one single PCI device.
+DPDK core is designed with the 1:1 relationship between PCI devices and DPDK
+ports, so NFP PMD PF support requires to handle the multiport case specifically.
+During NFP PF initialization, the PMD will extract the information about the
+number of PF ports from the firmware and will create as many DPDK ports as
+needed.
+
+Because the unusual relationship between a single PCI device and several DPDK
+ports, there are some limitations when using more than one PF DPDK ports: there
+is no support for RX interrrupts and it is not possible either to use those PF
+ports with the device hotplug functionality.
+
 System configuration
 --------------------
 
-#. **Enable SR-IOV on the NFP-6xxx device:** The current NFP PMD works with
-   Virtual Functions (VFs) on a NFP device. Make sure that one of the Physical
-   Function (PF) drivers from the above Github repository is installed and
-   loaded.
+#. **Enable SR-IOV on the NFP device:** The current NFP PMD supports the PF and
+   the VFs on a NFP device. However, it is not possible to work with both at the
+   same time because the VFs require the PF being bound to the NFP PF Linux
+   netdev driver.  Make sure you are working with a kernel with NFP PF support or
+   get the drivers from the above Github repository and follow the instructions
+   for building and installing it.
 
    Virtual Functions need to be enabled before they can be used with the PMD.
    Before enabling the VFs it is useful to obtain information about the
-- 
1.9.1

Re: [dpdk-dev] [PATCH 00/16] nfp: add pf support

[Ferruh Yigit]

On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> NFP PMD has just had support for SRIOV VFs until now. This patchset adds
> support for the PF, but just for being used as another DPDK port. No VF
> management is added by now.
> 
> NFP is a programmable device and it supports virtual NICs (vNICs) through
> firmware implementation. Different firmware applications implement vNICs
> for PF devices and VF devices, being number of vNICs dependent on the
> firmware and the NFP card available. PF vNIC (virtual) BARs are a subset
> of PF PCI device BARs while VF vNIC BARs are same than VF PCI BARs. 
> 
> Working with VF vNICs requires a PF driver uploading the firmware and 
> doing some NFP configuration. This needs can be only done with the kernel
> NFP PF netdev driver by now.
> 
> Working with PF vNIC requires the PMD doing the NFP configuration and for
> accessing the NFP a specific user space interface is created. NFP Service 
> Processor Userspace (NSPU) interface allows to create specific PCI BAR
> windows for accessing different parts of the NFP device, including the
> Network Service Processor (NSP) itself. The NSPU interface is implemented
> as the base for working with the PF.
> 
> Alejandro Lucero (16):
>   nfp: add nsp user space interface
>   nfp: add specific pf probe function
>   nfp: add support for new pci id
>   nfp: add nsp support for commands
>   nfp: add nsp fw upload command
>   nfp: add nsp symbol resolution command
>   nfp: add fw upload logic
>   nfp: add support for vnic config bar mapping
>   nfp: add support for vNIC rx/tx bar mappings
>   nfp: support pf devices inside pmd initialization
>   nfp: allocate eth_dev from pf probe function
>   nfp: support pf multiport
>   nfp: add nsp support for hw link configuration
>   nfp: add support for hw port link configuration
>   nfp: read pf port mac addr using nsp
>   doc: update nfp with pf support information

For all patches in the set, can you please run the tool
./devtools/check-git-log.sh [1], it checks for some common commit log
errors.


[1]
./devtools/check-git-log.sh checks the applied commits in the tree and
gets commit number to check as parameter:
"./devtools/check-git-log.sh -16" for this case.

Re: [dpdk-dev] [PATCH 00/16] nfp: add pf support

[Alejandro Lucero]

On Mon, Aug 28, 2017 at 5:42 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2017 5:20 PM, Alejandro Lucero wrote:
> > NFP PMD has just had support for SRIOV VFs until now. This patchset adds
> > support for the PF, but just for being used as another DPDK port. No VF
> > management is added by now.
> >
> > NFP is a programmable device and it supports virtual NICs (vNICs) through
> > firmware implementation. Different firmware applications implement vNICs
> > for PF devices and VF devices, being number of vNICs dependent on the
> > firmware and the NFP card available. PF vNIC (virtual) BARs are a subset
> > of PF PCI device BARs while VF vNIC BARs are same than VF PCI BARs.
> >
> > Working with VF vNICs requires a PF driver uploading the firmware and
> > doing some NFP configuration. This needs can be only done with the kernel
> > NFP PF netdev driver by now.
> >
> > Working with PF vNIC requires the PMD doing the NFP configuration and for
> > accessing the NFP a specific user space interface is created. NFP Service
> > Processor Userspace (NSPU) interface allows to create specific PCI BAR
> > windows for accessing different parts of the NFP device, including the
> > Network Service Processor (NSP) itself. The NSPU interface is implemented
> > as the base for working with the PF.
> >
> > Alejandro Lucero (16):
> >   nfp: add nsp user space interface
> >   nfp: add specific pf probe function
> >   nfp: add support for new pci id
> >   nfp: add nsp support for commands
> >   nfp: add nsp fw upload command
> >   nfp: add nsp symbol resolution command
> >   nfp: add fw upload logic
> >   nfp: add support for vnic config bar mapping
> >   nfp: add support for vNIC rx/tx bar mappings
> >   nfp: support pf devices inside pmd initialization
> >   nfp: allocate eth_dev from pf probe function
> >   nfp: support pf multiport
> >   nfp: add nsp support for hw link configuration
> >   nfp: add support for hw port link configuration
> >   nfp: read pf port mac addr using nsp
> >   doc: update nfp with pf support information
>
> For all patches in the set, can you please run the tool
> ./devtools/check-git-log.sh [1], it checks for some common commit log
> errors.
>
>
> [1]
> ./devtools/check-git-log.sh checks the applied commits in the tree and
> gets commit number to check as parameter:
> "./devtools/check-git-log.sh -16" for this case.
>
>
I will fix this in next patch set version.

Thanks