[dpdk-dev] support MAC changes when no live changes allowed

[dpdk-dev] support MAC changes when no live changes allowed

[Alejandro Lucero]

This is a patched to fix a functionality coming with the first public
release: changing/setting MAC address.

The original patch assumes all NICs can safely change or set the MAC
in any case. However, this is not always true. NFP depends on the firmware
capabilities and this is not always supported. There are other NICs with
this same limitation, although, as far as I know, not in DPDK. Linux kernel
has a IFF_LIVE_ADDR_CHANGE flag and two NICs are checking this flag for
allowing or not live MAC changes.

The flag proposed in this patch is just the opposite: advertise if live
change not supported and assuming it is supported other way.

Although most NICs support rte_eth_dev_default_mac_addr_set and this
function returns and error when live change is not supported, note that
this function is invoked during port start but the value returned is not
checked. It is likely this is good enough for most of the cases, but
bonding is relying on this start then mac set/change, and a PMD ports is
not properly configured for being used as an slave port in some bonding
modes.

v2:
 - add RTE_ETH_DEV_NOLIVE_MAC_ADDR comment in rte_eth_dev_default_mac_addr_set doc
 - add rte_eth_dev_start change in release API changes

[dpdk-dev] [PATCH v2 1/3] ethdev: fix MAC changes when live change not supported

[Alejandro Lucero]

Current code assumes a MAC change can occur when the port has been
started. In fact, there are some NICs which require this port state
for being successful, but other NICs not always support MAC change
in that case.

This patch supports a new device flag for a device advertising this
limitation, and if the flag is set, the MAC is changed before the
port starts.

Fixes: af75078fece3 ("first public release")
Cc: stable@dpdk.org

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 lib/librte_ethdev/rte_ethdev.c | 28 +++++++++++++++++++---------
 lib/librte_ethdev/rte_ethdev.h |  6 ++++++
 2 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index f32722f..16825bf 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -1219,19 +1219,14 @@ struct rte_eth_dev *
 }
 
 static void
-rte_eth_dev_config_restore(uint16_t port_id)
+rte_eth_dev_mac_restore(struct rte_eth_dev *dev,
+			struct rte_eth_dev_info *dev_info)
 {
-	struct rte_eth_dev *dev;
-	struct rte_eth_dev_info dev_info;
 	struct ether_addr *addr;
 	uint16_t i;
 	uint32_t pool = 0;
 	uint64_t pool_mask;
 
-	dev = &rte_eth_devices[port_id];
-
-	rte_eth_dev_info_get(port_id, &dev_info);
-
 	/* replay MAC address configuration including default MAC */
 	addr = &dev->data->mac_addrs[0];
 	if (*dev->dev_ops->mac_addr_set != NULL)
@@ -1240,7 +1235,7 @@ struct rte_eth_dev *
 		(*dev->dev_ops->mac_addr_add)(dev, addr, 0, pool);
 
 	if (*dev->dev_ops->mac_addr_add != NULL) {
-		for (i = 1; i < dev_info.max_mac_addrs; i++) {
+		for (i = 1; i < dev_info->max_mac_addrs; i++) {
 			addr = &dev->data->mac_addrs[i];
 
 			/* skip zero address */
@@ -1259,6 +1254,14 @@ struct rte_eth_dev *
 			} while (pool_mask);
 		}
 	}
+}
+
+static void
+rte_eth_dev_config_restore(struct rte_eth_dev *dev,
+			   struct rte_eth_dev_info *dev_info, uint16_t port_id)
+{
+	if (!(*dev_info->dev_flags & RTE_ETH_DEV_NOLIVE_MAC_ADDR))
+		rte_eth_dev_mac_restore(dev, dev_info);
 
 	/* replay promiscuous configuration */
 	if (rte_eth_promiscuous_get(port_id) == 1)
@@ -1277,6 +1280,7 @@ struct rte_eth_dev *
 rte_eth_dev_start(uint16_t port_id)
 {
 	struct rte_eth_dev *dev;
+	struct rte_eth_dev_info dev_info;
 	int diag;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
@@ -1292,13 +1296,19 @@ struct rte_eth_dev *
 		return 0;
 	}
 
+	rte_eth_dev_info_get(port_id, &dev_info);
+
+	/* Lets restore MAC now if device does not support live change */
+	if (*dev_info.dev_flags & RTE_ETH_DEV_NOLIVE_MAC_ADDR)
+		rte_eth_dev_mac_restore(dev, &dev_info);
+
 	diag = (*dev->dev_ops->dev_start)(dev);
 	if (diag == 0)
 		dev->data->dev_started = 1;
 	else
 		return eth_err(port_id, diag);
 
-	rte_eth_dev_config_restore(port_id);
+	rte_eth_dev_config_restore(dev, &dev_info, port_id);
 
 	if (dev->data->dev_conf.intr_conf.lsc == 0) {
 		RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->link_update, -ENOTSUP);
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 7070e9a..f6c45bc 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1268,6 +1268,8 @@ struct rte_eth_dev_owner {
 #define RTE_ETH_DEV_INTR_RMV     0x0008
 /** Device is port representor */
 #define RTE_ETH_DEV_REPRESENTOR  0x0010
+/** Device does not support MAC change after started */
+#define RTE_ETH_DEV_NOLIVE_MAC_ADDR  0x0020
 
 /**
  * Iterates over valid ethdev ports owned by a specific owner.
@@ -2839,6 +2841,10 @@ int rte_eth_dev_mac_addr_add(uint16_t port_id, struct ether_addr *mac_addr,
 /**
  * Set the default MAC address.
  *
+ * A NIC not supporting MAC change after started should set
+ * RTE_ETH_DEV_NOLIVE_MAC_ADDR and this function should check such a flag
+ * and NIC state.
+ *
  * @param port_id
  *   The port identifier of the Ethernet device.
  * @param mac_addr
-- 
1.9.1

Re: [dpdk-dev] [PATCH v2 1/3] ethdev: fix MAC changes when live change not supported

[Ferruh Yigit]

On 8/24/2018 12:15 PM, Alejandro Lucero wrote:
> Current code assumes a MAC change can occur when the port has been
> started. In fact, there are some NICs which require this port state
> for being successful, but other NICs not always support MAC change
> in that case.
> 
> This patch supports a new device flag for a device advertising this
> limitation, and if the flag is set, the MAC is changed before the
> port starts.
> 
> Fixes: af75078fece3 ("first public release")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>

<...>

> @@ -2839,6 +2841,10 @@ int rte_eth_dev_mac_addr_add(uint16_t port_id, struct ether_addr *mac_addr,
>  /**
>   * Set the default MAC address.
>   *
> + * A NIC not supporting MAC change after started should set
> + * RTE_ETH_DEV_NOLIVE_MAC_ADDR and this function should check such a flag
> + * and NIC state.
> + *

Only rte_eth_dev_start() API effected from this change, API behavior changes
based on if PMD provides this flag or not, I was thinking to document this in
rte_eth_dev_start(), something like:
"Driver RTE_ETH_DEV_NOLIVE_MAC_ADDR flag cause MAC address to be set before
start dev_ops"

As you mentioned in cover letter, rte_eth_dev_mac_addr_add() will return an
error if not supported, this is not changed with RTE_ETH_DEV_NOLIVE_MAC_ADDR
flag, so I think no need to add this comment.

Re: [dpdk-dev] [PATCH v2 1/3] ethdev: fix MAC changes when live change not supported

[Alejandro Lucero]

On Fri, Aug 24, 2018 at 2:27 PM, Ferruh Yigit <ferruh.yigit@intel.com>
wrote:

> On 8/24/2018 12:15 PM, Alejandro Lucero wrote:
> > Current code assumes a MAC change can occur when the port has been
> > started. In fact, there are some NICs which require this port state
> > for being successful, but other NICs not always support MAC change
> > in that case.
> >
> > This patch supports a new device flag for a device advertising this
> > limitation, and if the flag is set, the MAC is changed before the
> > port starts.
> >
> > Fixes: af75078fece3 ("first public release")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
>
> <...>
>
> > @@ -2839,6 +2841,10 @@ int rte_eth_dev_mac_addr_add(uint16_t port_id,
> struct ether_addr *mac_addr,
> >  /**
> >   * Set the default MAC address.
> >   *
> > + * A NIC not supporting MAC change after started should set
> > + * RTE_ETH_DEV_NOLIVE_MAC_ADDR and this function should check such a
> flag
> > + * and NIC state.
> > + *
>
> Only rte_eth_dev_start() API effected from this change, API behavior
> changes
> based on if PMD provides this flag or not, I was thinking to document this
> in
> rte_eth_dev_start(), something like:
> "Driver RTE_ETH_DEV_NOLIVE_MAC_ADDR flag cause MAC address to be set before
> start dev_ops"
>
> As you mentioned in cover letter, rte_eth_dev_mac_addr_add() will return an
> error if not supported, this is not changed with
> RTE_ETH_DEV_NOLIVE_MAC_ADDR
> flag, so I think no need to add this comment.
>

Ok. I will send another version.

Thanks

Re: [dpdk-dev] [PATCH v2 1/3] ethdev: fix MAC changes when live change not supported

[Ferruh Yigit]

On 8/24/2018 12:15 PM, Alejandro Lucero wrote:
> Current code assumes a MAC change can occur when the port has been
> started. In fact, there are some NICs which require this port state
> for being successful, but other NICs not always support MAC change
> in that case.
> 
> This patch supports a new device flag for a device advertising this
> limitation, and if the flag is set, the MAC is changed before the
> port starts.
> 
> Fixes: af75078fece3 ("first public release")
> Cc: stable@dpdk.org

And this looks like behavior change, controlled with a new flag. Do you thinks
is this fix or should be really backported stable trees?

[dpdk-dev] [PATCH v2 2/3] net/nfp: fix live MAC changes not supported

[Alejandro Lucero]

Some NFP firmwares support live changes to the MAC address, but
this is not always true and the firmware advertises it accordingly.

This patch checks if firmware does not support live changes and
sets RTE_ETH_DEV_NOLIVE_MAC_ADDR in that case.

Cc: stable@dpdk.org

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 drivers/net/nfp/nfp_net.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/nfp/nfp_net.c b/drivers/net/nfp/nfp_net.c
index 6e5e305..ee743e9 100644
--- a/drivers/net/nfp/nfp_net.c
+++ b/drivers/net/nfp/nfp_net.c
@@ -2886,6 +2886,9 @@ uint32_t nfp_net_txq_full(struct nfp_net_txq *txq)
 	ether_addr_copy((struct ether_addr *)hw->mac_addr,
 			&eth_dev->data->mac_addrs[0]);
 
+	if (!(hw->cap & NFP_NET_CFG_CTRL_LIVE_ADDR))
+		eth_dev->data->dev_flags |= RTE_ETH_DEV_NOLIVE_MAC_ADDR;
+
 	PMD_INIT_LOG(INFO, "port %d VendorID=0x%x DeviceID=0x%x "
 		     "mac=%02x:%02x:%02x:%02x:%02x:%02x",
 		     eth_dev->data->port_id, pci_dev->id.vendor_id,
-- 
1.9.1

[dpdk-dev] [PATCH v2 3/3] doc: comment rte_eth_dev_start change

[Alejandro Lucero]

The new device flag RTE_ETH_DEV_NOLIVE_MAC_ADDR modifies how MAC is set
when calling rte_eth_dev_start.

Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
---
 doc/guides/rel_notes/release_18_11.rst | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index 3ae6b3f..b0c73bd 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,7 +67,11 @@ API Changes
    This section is a comment. Do not overwrite or remove it.
    Also, make sure to start the actual text at the margin.
    =========================================================
-
+   * A new device flag, RTE_ETH_DEV_NOLIVE_MAC_ADDR, changes the order of
+     actions inside rte_eth_dev_start regarding MAC set. Some NICs do not
+     support MAC changes once the port has started and with this new device
+     flag the MAC can be properly configured in any case. This is particularly
+     important for bonding.
 
 ABI Changes
 -----------
-- 
1.9.1

Re: [dpdk-dev] [PATCH v2 3/3] doc: comment rte_eth_dev_start change

[Ferruh Yigit]

On 8/24/2018 12:15 PM, Alejandro Lucero wrote:
> The new device flag RTE_ETH_DEV_NOLIVE_MAC_ADDR modifies how MAC is set
> when calling rte_eth_dev_start.
> 
> Signed-off-by: Alejandro Lucero <alejandro.lucero@netronome.com>
> ---
>  doc/guides/rel_notes/release_18_11.rst | 6 +++++-

I think it is better to merge this with implementation patch, 1/3.