[dpdk-dev] [PATCH 00/22] enable hotplug on multi-process

[dpdk-dev] [PATCH 00/22] enable hotplug on multi-process

[Qi Zhang]

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process will
broadcast to all other processes through mp channel then device information
will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.

detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on most
pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued but
not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is required

Limitation:
===========

The solution does not cover the case that primary process exit while
secondary processes still be active. Though this is not a typial use
case, but if this happens:
1. secondary process can't attach / detach any shared device since no
primary exist.
2. secondary process still can attach / detach private device.
3. secondary process still can detach a share device privately but may
not attach it back, that ethdev slot will become zombie slot.

Example:
========

The patchset also contains a example to demonstrate device management
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./devmgm_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0


Qi Zhang (22):
  eal: introduce one device scan
  bus/vdev: enable one device scan
  ethdev: add function to release port in local process
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/e1000: enable port detach on secondary process
  net/igb: enable port detach on secondary process
  net/fm10k: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/failsafe: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/devmgm_mp: add simple device management sample

 drivers/bus/vdev/vdev.c                   |  30 ++
 drivers/net/af_packet/rte_eth_af_packet.c |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c    |  11 +
 drivers/net/e1000/em_ethdev.c             |   9 +
 drivers/net/e1000/igb_ethdev.c            |   9 +
 drivers/net/failsafe/failsafe.c           |  16 +
 drivers/net/fm10k/fm10k_ethdev.c          |   9 +
 drivers/net/i40e/i40e_ethdev.c            |   2 +
 drivers/net/i40e/i40e_ethdev_vf.c         |   9 +
 drivers/net/ixgbe/ixgbe_ethdev.c          |  12 +
 drivers/net/kni/rte_eth_kni.c             |  11 +
 drivers/net/null/rte_eth_null.c           |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c    |  13 +
 drivers/net/pcap/rte_eth_pcap.c           |  15 +-
 drivers/net/softnic/rte_eth_softnic.c     |  19 +-
 drivers/net/tap/rte_eth_tap.c             |  17 +-
 drivers/net/vhost/rte_eth_vhost.c         |  11 +
 examples/devmgm_mp/Makefile               |  64 +++
 examples/devmgm_mp/commands.c             | 383 ++++++++++++++++++
 examples/devmgm_mp/commands.h             |  10 +
 examples/devmgm_mp/main.c                 |  41 ++
 examples/devmgm_mp/meson.build            |  11 +
 lib/librte_eal/common/eal_common_dev.c    |  17 +-
 lib/librte_eal/common/eal_private.h       |   8 +
 lib/librte_eal/common/include/rte_bus.h   |   4 +
 lib/librte_eal/linuxapp/eal/eal.c         |   6 +
 lib/librte_ethdev/Makefile                |   2 +
 lib/librte_ethdev/rte_ethdev.c            | 246 ++++++++++-
 lib/librte_ethdev/rte_ethdev.h            | 101 +++++
 lib/librte_ethdev/rte_ethdev_core.h       |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h     |  40 ++
 lib/librte_ethdev/rte_ethdev_lock.c       | 102 +++++
 lib/librte_ethdev/rte_ethdev_lock.h       |  23 ++
 lib/librte_ethdev/rte_ethdev_mp.c         | 653 ++++++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_mp.h         |  45 ++
 35 files changed, 1950 insertions(+), 31 deletions(-)
 create mode 100644 examples/devmgm_mp/Makefile
 create mode 100644 examples/devmgm_mp/commands.c
 create mode 100644 examples/devmgm_mp/commands.h
 create mode 100644 examples/devmgm_mp/main.c
 create mode 100644 examples/devmgm_mp/meson.build
 create mode 100644 lib/librte_ethdev/rte_ethdev_lock.c
 create mode 100644 lib/librte_ethdev/rte_ethdev_lock.h
 create mode 100644 lib/librte_ethdev/rte_ethdev_mp.c
 create mode 100644 lib/librte_ethdev/rte_ethdev_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH 01/22] eal: introduce one device scan

[Qi Zhang]

When hot plug a new device, it is not necessary to scan everything
on the bus since the devname and devargs are already there. So new
rte_bus ops "scan_one" is introduced, bus driver can implement this
function to simply the hotplug process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
 lib/librte_eal/common/include/rte_bus.h |  4 ++++
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 61cb3b162..1ad033536 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	if (ret)
 		goto err_devarg;
 
-	ret = bus->scan();
-	if (ret)
-		goto err_devarg;
+	/**
+	 * if bus support to scan specific device by devargs,
+	 * we don't need to scan all devices on the bus.
+	 */
+	if (bus->scan_one) {
+		dev = bus->scan_one(da);
+	} else {
+		ret = bus->scan();
+		if (ret)
+			goto err_devarg;
+
+		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
+	}
 
-	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
 	if (dev == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
 			devname);
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..b15cff892 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -83,6 +83,7 @@ enum rte_iova_mode {
  */
 typedef int (*rte_bus_scan_t)(void);
 
+typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *);
 /**
  * Implementation specific probe function which is responsible for linking
  * devices on that bus with applicable drivers.
@@ -95,6 +96,8 @@ typedef int (*rte_bus_scan_t)(void);
  */
 typedef int (*rte_bus_probe_t)(void);
 
+
+
 /**
  * Device iterator to find a device on a bus.
  *
@@ -204,6 +207,7 @@ struct rte_bus {
 	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
 	const char *name;            /**< Name of the bus */
 	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
+	rte_bus_scan_one_t scan_one; /**< Scan one device by devargs */
 	rte_bus_probe_t probe;       /**< Probe devices on bus */
 	rte_bus_find_device_t find_device; /**< Find a device on the bus */
 	rte_bus_plug_t plug;         /**< Probe single device for drivers */
-- 
2.13.6

[dpdk-dev] [PATCH 02/22] bus/vdev: enable one device scan

[Qi Zhang]

Implemented the bus ops scan_one, besides this improve the scan
efficiency in hotplug case, it aslo avoid sync IPC invoke (which
happens in vdev->scan on secondary process). The benifit is it
removes the potiential deadlock in the case when secondary process
receive a request from primary process to attach a new device, since
vdev->scan will be invoked on mp thread itself at this case.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 6139dd551..cdbd77df0 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -467,6 +467,35 @@ vdev_scan(void)
 	return 0;
 }
 
+static struct rte_device *vdev_scan_one(struct rte_devargs *devargs)
+{
+	struct rte_vdev_device *dev = NULL;
+
+	dev = calloc(1, sizeof(*dev));
+	if (!dev) {
+		VDEV_LOG(ERR, "failed to allocate memory for new device");
+		return NULL;
+	}
+
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
+
+	if (find_vdev(devargs->name)) {
+		VDEV_LOG(ERR, "device %s already exist", devargs->name);
+		free(dev);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+		return NULL;
+	}
+
+	dev->device.devargs = devargs;
+	dev->device.numa_node = SOCKET_ID_ANY;
+	dev->device.name = devargs->name;
+	TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
+
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+
+	return &dev->device;
+}
+
 static int
 vdev_probe(void)
 {
@@ -531,6 +560,7 @@ vdev_unplug(struct rte_device *dev)
 
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
+	.scan_one = vdev_scan_one,
 	.probe = vdev_probe,
 	.find_device = vdev_find_device,
 	.plug = vdev_plug,
-- 
2.13.6

[dpdk-dev] [PATCH 03/22] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_local to support the requirement
that an ethdev only be released on secondary process, so only local
state be set to unused , share data will not be reset so primary
process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index cd4bfd3c6..ec14adb91 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_local(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+
+	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
+
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
-	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
+		eth_dev->state = RTE_ETH_DEV_UNUSED;
+		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	}
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
 
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..261335426 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_local(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
-- 
2.13.6

[dpdk-dev] [PATCH 04/22] ethdev: enable hotplug on multi-process

[Qi Zhang]

The patch introduce the solution to handle different hotplug cases in
multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced, it will be implemented in
following separate patch.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 3, 4:
This will be implemented in following patch.

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowd to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs chenages:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_private.h   |   8 ++
 lib/librte_eal/linuxapp/eal/eal.c     |   6 ++
 lib/librte_ethdev/Makefile            |   1 +
 lib/librte_ethdev/rte_ethdev.c        | 183 ++++++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h        |  37 +++++++
 lib/librte_ethdev/rte_ethdev_core.h   |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h |  27 +++++
 lib/librte_ethdev/rte_ethdev_mp.c     | 195 ++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_mp.h     |  44 ++++++++
 9 files changed, 489 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/rte_ethdev_mp.c
 create mode 100644 lib/librte_ethdev/rte_ethdev_mp.h

diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..92fa59bed 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,12 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Register mp channel callback functions of ethdev layer.
+ *
+ * @return
+ *  0 on success.
+ *  (<0) on failure.
+ */
+int rte_eth_dev_mp_init(void);
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..b276e1caa 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1041,6 +1041,12 @@ rte_eal_init(int argc, char **argv)
 
 	rte_eal_mcfg_complete();
 
+	if (rte_eth_dev_mp_init()) {
+		rte_eal_init_alert("rte_eth_dev_mp_init() failed\n");
+		rte_errno = ENOEXEC;
+		return -1;
+	}
+
 	return fctret;
 }
 
diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..04e93f337 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += rte_ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index ec14adb91..24360f522 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,12 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "rte_ethdev_mp.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -656,9 +657,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -703,14 +703,104 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	return ret;
+
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primray
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = rte_eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR, "Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	rte_eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -721,22 +811,81 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primray
+		 * to start the process.
+		 */
+		ret = rte_eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR, "Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR, "Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR, "Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..bb03d613b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,27 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1511,22 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index 261335426..616add313 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -338,6 +338,33 @@ typedef int (*ethdev_uninit_t)(struct rte_eth_dev *ethdev);
 int __rte_experimental
 rte_eth_dev_destroy(struct rte_eth_dev *ethdev, ethdev_uninit_t ethdev_uninit);
 
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev_mp.c b/lib/librte_ethdev/rte_ethdev_mp.c
new file mode 100644
index 000000000..8ede8151d
--- /dev/null
+++ b/lib/librte_ethdev/rte_ethdev_mp.c
@@ -0,0 +1,195 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include "rte_ethdev_driver.h"
+#include "rte_ethdev_mp.h"
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_local(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	(void)msg;
+	(void)(peer);
+	return -ENOTSUP;
+}
+
+static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+{
+	(void)msg;
+	(void)(peer);
+	return -ENOTSUP;
+}
+
+static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp = {0};
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, peer) < 0) {
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	(void)req;
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req = {0};
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_eth_dev_mp_init(void)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return -1;
+		}
+	} else {
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_RESPONSE,
+					   handle_primary_response)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_RESPONSE);
+			return -1;
+		}
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+		}
+	}
+
+	return 0;
+}
+
diff --git a/lib/librte_ethdev/rte_ethdev_mp.h b/lib/librte_ethdev/rte_ethdev_mp.h
new file mode 100644
index 000000000..c3e55dfec
--- /dev/null
+++ b/lib/librte_ethdev/rte_ethdev_mp.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+/* Register mp channel callback functions of ethdev layer.*/
+int rte_eth_dev_mp_init(void);
+
+#endif
-- 
2.13.6

[dpdk-dev] [PATCH 05/22] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/rte_ethdev.c      |  41 ++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h      |  64 ++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_lock.c | 102 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_lock.h |  23 ++++++++
 lib/librte_ethdev/rte_ethdev_mp.c   |   3 +-
 6 files changed, 232 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/rte_ethdev_lock.c
 create mode 100644 lib/librte_ethdev/rte_ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index 04e93f337..5c4646469 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += rte_ethdev_mp.c
+SRCS-y += rte_ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 24360f522..6494e71a4 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -42,6 +42,7 @@
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
 #include "rte_ethdev_mp.h"
+#include "rte_ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -787,7 +788,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -828,6 +828,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -870,6 +874,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -883,6 +888,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4683,6 +4692,36 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
+		 void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return register_lock_callback(port_id, dev_is_busy, NULL);
+	else
+		return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
+		   void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return unregister_lock_callback(port_id, dev_is_busy, NULL);
+	else
+		return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index bb03d613b..506b6acdd 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4356,6 +4356,70 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached(ref. rte_eth_dev_lock and rte_eth_dev_unlock).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * Lock an Ethernet Device directly or register a callback function
+ * for condition check at runtime, this help application to prevent
+ * a device be detached unexpectly.
+ * NOTE: Lock a device mutliple times with same parmeter will increase
+ * a ref_count, and coresponding unlock decrease the ref_count, the
+ * device will be unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   !NULL the callback function will be added into a pre-detach list,
+ *         it will be invoked when a device is going to be detached. The
+ *         return value will decide if continue detach the device or not.
+ *   NULL  lock the device directly, basically this just regiter a empty
+ *         callback function(dev_is_busy) that return -EBUSY, so we can
+ *         handle the pre-detach check in unified way.
+ * @param user_args
+ *   parameter will be parsed to callback function, only valid when
+ *   callback != NULL.
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
+		     void *user_args);
+
+/**
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   NULL  decrease the ref_count of default callback function.
+ *   !NULL decrease the ref_count of specific callback with matched
+ *         user_args.
+ * @param user_args
+ *   parameter to match, only valid when callback != NULL.
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
+		       void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev_lock.c b/lib/librte_ethdev/rte_ethdev_lock.c
new file mode 100644
index 000000000..688d1d70a
--- /dev/null
+++ b/lib/librte_ethdev/rte_ethdev_lock.c
@@ -0,0 +1,102 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "rte_ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+		       rte_eth_dev_lock_callback_t callback,
+		       void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (!le) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (!le) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			 rte_eth_dev_lock_callback_t callback,
+			 void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le) {
+		le->ref_count--;
+		if (!le->ref_count) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+int
+process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/rte_ethdev_lock.h b/lib/librte_ethdev/rte_ethdev_lock.h
new file mode 100644
index 000000000..7b370c926
--- /dev/null
+++ b/lib/librte_ethdev/rte_ethdev_lock.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+int
+register_lock_callback(uint16_t port_id,
+		       rte_eth_dev_lock_callback_t callback,
+		       void *user_args);
+
+int
+unregister_lock_callback(uint16_t port_id,
+			 rte_eth_dev_lock_callback_t callback,
+			 void *user_args);
+
+int
+process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/rte_ethdev_mp.c b/lib/librte_ethdev/rte_ethdev_mp.c
index 8ede8151d..e23c8b010 100644
--- a/lib/librte_ethdev/rte_ethdev_mp.c
+++ b/lib/librte_ethdev/rte_ethdev_mp.c
@@ -4,6 +4,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "rte_ethdev_mp.h"
+#include "rte_ethdev_lock.h"
 
 static int detach_on_secondary(uint16_t port_id)
 {
@@ -101,7 +102,7 @@ static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
-- 
2.13.6

[dpdk-dev] [PATCH 06/22] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process, the implementation
references malloc_mp.c.

device attach on secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.

device detach on secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev_mp.c | 477 +++++++++++++++++++++++++++++++++++++-
 lib/librte_ethdev/rte_ethdev_mp.h |   1 +
 2 files changed, 468 insertions(+), 10 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev_mp.c b/lib/librte_ethdev/rte_ethdev_mp.c
index e23c8b010..6dbd23fa5 100644
--- a/lib/librte_ethdev/rte_ethdev_mp.c
+++ b/lib/librte_ethdev/rte_ethdev_mp.c
@@ -2,10 +2,69 @@
  * Copyright(c) 2010-2018 Intel Corporation
  */
 
+#include <sys/time.h>
+
 #include "rte_ethdev_driver.h"
 #include "rte_ethdev_mp.h"
 #include "rte_ethdev_lock.h"
 
+enum req_state {
+	REQ_STATE_INACTIVE = 0,
+	REQ_STATE_ACTIVE,
+	REQ_STATE_COMPLETE
+};
+
+struct mp_request {
+	TAILQ_ENTRY(mp_request) next;
+	struct eth_dev_mp_req user_req; /**< contents of request */
+	pthread_cond_t cond; /**< variable we use to time out on this request */
+	enum req_state state; /**< indicate status of this request */
+};
+
+/*
+ * We could've used just a single request, but it may be possible for
+ * secondaries to timeout earlier than the primary, and send a new request while
+ * primary is still expecting replies to the old one. Therefore, each new
+ * request will get assigned a new ID, which is how we will distinguish between
+ * expected and unexpected messages.
+ */
+TAILQ_HEAD(mp_request_list, mp_request);
+static struct {
+	struct mp_request_list list;
+	pthread_mutex_t lock;
+} mp_request_list = {
+	.list = TAILQ_HEAD_INITIALIZER(mp_request_list.list),
+	.lock = PTHREAD_MUTEX_INITIALIZER
+};
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+static struct mp_request *
+find_request_by_id(uint64_t id)
+{
+	struct mp_request *req;
+
+	TAILQ_FOREACH(req, &mp_request_list.list, next) {
+		if (req->user_req.id == id)
+			break;
+	}
+	return req;
+}
+
+static uint64_t
+get_unique_id(void)
+{
+	uint64_t id;
+
+	do {
+		id = rte_rand();
+	} while (find_request_by_id(id) != NULL);
+	return id;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req);
+
 static int detach_on_secondary(uint16_t port_id)
 {
 	struct rte_device *dev;
@@ -72,18 +131,325 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 	return 0;
 }
 
-static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+static int
+check_reply(const struct eth_dev_mp_req *req,
+	    const struct rte_mp_reply *reply)
+{
+	struct eth_dev_mp_req *resp;
+	int i;
+
+	if (reply->nb_received != reply->nb_sent)
+		return -EINVAL;
+
+	for (i = 0; i < reply->nb_received; i++) {
+		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
+
+		if (resp->t != req->t) {
+			ethdev_log(ERR, "Unexpected response to async request\n");
+			return -EINVAL;
+		}
+
+		if (resp->id != req->id) {
+			ethdev_log(ERR, "response to wrong async request\n");
+			return -EINVAL;
+		}
+
+		if (resp->result)
+			return resp->result;
+	}
+
+	return 0;
+}
+
+static int
+send_response_to_secondary(const struct eth_dev_mp_req *req, int result)
+{
+	struct rte_mp_msg resp_msg = {0};
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)resp_msg.param;
+	int ret = 0;
+
+	resp_msg.len_param = sizeof(*resp);
+	strcpy(resp_msg.name, ETH_DEV_MP_ACTION_RESPONSE);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_sendmsg(&resp_msg);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static int
+handle_async_attach_response(const struct rte_mp_msg *request,
+			     const struct rte_mp_reply *reply)
+{
+	struct mp_request *entry;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct eth_dev_mp_req tmp_req;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_ATTACH_ROLLBACK;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, 0);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+handle_async_detach_response(const struct rte_mp_msg *request,
+			     const struct rte_mp_reply *reply)
+{
+	struct mp_request *entry;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		send_response_to_secondary(req, ret);
+	} else {
+		do_eth_dev_detach(req->port_id);
+		send_response_to_secondary(req, 0);
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+handle_async_pre_detach_response(const struct rte_mp_msg *request,
+				 const struct rte_mp_reply *reply)
+{
+	struct mp_request *entry;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct eth_dev_mp_req tmp_req;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (!ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_DETACH;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, ret);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return 0;
+}
+
+static int
+handle_async_rollback_response(const struct rte_mp_msg *request,
+			       const struct rte_mp_reply *reply __rte_unused)
+{
+	struct mp_request *entry;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	/* we have nothing to do if rollback still fail, just detach */
+	do_eth_dev_detach(req->port_id);
+	/* send response to secondary with the reason of rollback */
+	send_response_to_secondary(req, req->result);
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req)
 {
-	(void)msg;
-	(void)(peer);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req = {0};
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	rte_mp_async_reply_t clb;
+	struct mp_request *entry;
+	int ret = 0;
+
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	if (req->t == REQ_TYPE_ATTACH)
+		clb = handle_async_attach_response;
+	else if (req->t == REQ_TYPE_PRE_DETACH)
+		clb = handle_async_pre_detach_response;
+	else if (req->t == REQ_TYPE_DETACH)
+		clb = handle_async_detach_response;
+	else if (req->t == REQ_TYPE_ATTACH_ROLLBACK)
+		clb = handle_async_rollback_response;
+	else
+		return -1;
+	do {
+		ret = rte_mp_request_async(&mp_req, &ts, clb);
+	} while (ret != 0 && rte_errno == EEXIST);
+
+	if (ret)
+		ethdev_log(ERR, "couldn't send async request\n");
+	entry = find_request_by_id(req->id);
+	(void)entry;
+	return ret;
+}
+
+static int handle_secondary_request(const struct rte_mp_msg *msg,
+				    const void *peer __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	struct mp_request *entry;
+	uint16_t port_id;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		ethdev_log(ERR, "duplicate request id\n");
+		ret = -EEXIST;
+		goto finish;
+	}
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		ret = -ENOMEM;
+		goto finish;
+	}
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.port_id = port_id;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+		goto finish;
+	}
+
+	if (ret) {
+		ret = send_response_to_secondary(req, ret);
+		if (ret) {
+			ethdev_log(ERR, "failed to send response to secondary\n");
+			goto finish;
+		}
+	} else {
+		memcpy(&entry->user_req, req, sizeof(*req));
+		entry->state = REQ_STATE_ACTIVE;
+		TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+		entry = NULL;
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	if (entry)
+		free(entry);
+	return ret;
 }
 
-static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+static int handle_primary_response(const struct rte_mp_msg *msg,
+				   const void *peer __rte_unused)
 {
-	(void)msg;
-	(void)(peer);
-	return -ENOTSUP;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct mp_request *entry;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		entry->user_req.result = req->result;
+		entry->user_req.port_id = req->port_id;
+		entry->state = REQ_STATE_COMPLETE;
+
+		pthread_cond_signal(&entry->cond);
+	}
+
+	pthread_mutex_unlock(&mp_request_list.lock);
+
+	return 0;
 }
 
 static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
@@ -124,10 +490,101 @@ static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer
 	return 0;
 }
 
+/**
+ * secondary to primary request.
+ *
+ * device attach:
+ * a) seconary send request to primary.
+ * b) primary attach the new device if failed goto i).
+ * c) primary forward attach request to all secondary.
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail response to secondary, goto k).
+ * j) send success response to secondary.
+ * k) end.
+
+ * device detach:
+ * a) secondary send request to primary.
+ * b) primary perform pre-detach check, if device is locked, got j).
+ * c) primary send pre-detach check request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success response to secondary, goto k).
+ * j) send fail response to secondary.
+ * k) end.
+ */
 int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	(void)req;
-	return -ENOTSUP;
+	struct rte_mp_msg msg = {0};
+	struct eth_dev_mp_req *msg_req = (struct eth_dev_mp_req *)msg.param;
+	struct mp_request *entry;
+	struct timespec ts = {0};
+	struct timeval now;
+	int ret = 0;
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		return -ENOMEM;
+	}
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	ret = gettimeofday(&now, NULL);
+	if (ret) {
+		ethdev_log(ERR, "cannot get current time\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ts.tv_nsec = (now.tv_usec * 1000) % 1000000000;
+	ts.tv_sec = now.tv_sec + MP_TIMEOUT_S +
+			(now.tv_usec * 1000) / 1000000000;
+
+	pthread_cond_init(&entry->cond, NULL);
+
+	msg.len_param = sizeof(*req);
+	strcpy(msg.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	req->id = get_unique_id();
+
+	memcpy(msg_req, req, sizeof(*req));
+
+	ret = rte_mp_sendmsg(&msg);
+	if (ret) {
+		ethdev_log(ERR, "cannot send message to primary");
+		goto finish;
+	}
+
+	memcpy(&entry->user_req, req, sizeof(*req));
+
+	entry->state = REQ_STATE_ACTIVE;
+
+	TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+
+	do {
+		ret = pthread_cond_timedwait(&entry->cond,
+				&mp_request_list.lock, &ts);
+	} while (ret != 0 && ret != ETIMEDOUT);
+
+	if (entry->state != REQ_STATE_COMPLETE) {
+		RTE_LOG(ERR, EAL, "request time out\n");
+		ret = -ETIMEDOUT;
+	} else {
+		req->result = entry->user_req.result;
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	free(entry);
+	return ret;
 }
 
 /**
diff --git a/lib/librte_ethdev/rte_ethdev_mp.h b/lib/librte_ethdev/rte_ethdev_mp.h
index c3e55dfec..6d10dfdad 100644
--- a/lib/librte_ethdev/rte_ethdev_mp.h
+++ b/lib/librte_ethdev/rte_ethdev_mp.h
@@ -18,6 +18,7 @@ enum eth_dev_req_type {
 };
 
 struct eth_dev_mp_req {
+	uint64_t id;
 	enum eth_dev_req_type t;
 	char devargs[MAX_DEV_ARGS_LEN];
 	uint16_t port_id;
-- 
2.13.6

[dpdk-dev] [PATCH 07/22] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c    | 2 ++
 drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..1de3c1499 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 804e44530..2b1ece851 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH 08/22] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..260640e50 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
@@ -1809,6 +1812,15 @@ static struct rte_pci_driver rte_ixgbe_pmd = {
 static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+
+	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	return rte_eth_dev_pci_generic_probe(pci_dev,
 		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
 }
-- 
2.13.6

[dpdk-dev] [PATCH 09/22] net/e1000: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/em_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index 7039dc100..e626cb10c 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -349,6 +349,15 @@ static int eth_em_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_em_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_em_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH 10/22] net/igb: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/igb_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index edc7be319..bd2b2d218 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -1089,6 +1089,15 @@ static int eth_igb_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_igb_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_igb_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH 11/22] net/fm10k: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 3ff1b0e0f..c7042be4e 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -3264,6 +3264,15 @@ static int eth_fm10k_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_fm10k_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_local(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_fm10k_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH 12/22] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..e1afbfc14 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH 13/22] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 02d94b1b1..1221f62b2 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3065,6 +3065,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3171,6 +3172,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH 14/22] net/failsafe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c
index eafbb75df..aa676069d 100644
--- a/drivers/net/failsafe/failsafe.c
+++ b/drivers/net/failsafe/failsafe.c
@@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &failsafe_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 static int
 rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)
 {
+	struct rte_eth_dev *eth_dev;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
 	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
+
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario.
+		 */
+	}
+
 	return fs_rte_eth_free(name);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH 15/22] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..3ee527ab2 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH 16/22] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..e5b8d2f03 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH 17/22] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..b42b69896 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_local(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH 18/22] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..4c366a92b 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH 19/22] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..fdb2f0825 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_local(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH 20/22] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index 5531fe9d9..56d3b6cc9 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH 21/22] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_local, we can support this with
minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..4ab34cefd 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_local(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH 22/22] examples/devmgm_mp: add simple device management sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./devmgm_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/devmgm_mp/Makefile    |  64 +++++++
 examples/devmgm_mp/commands.c  | 383 +++++++++++++++++++++++++++++++++++++++++
 examples/devmgm_mp/commands.h  |  10 ++
 examples/devmgm_mp/main.c      |  41 +++++
 examples/devmgm_mp/meson.build |  11 ++
 5 files changed, 509 insertions(+)
 create mode 100644 examples/devmgm_mp/Makefile
 create mode 100644 examples/devmgm_mp/commands.c
 create mode 100644 examples/devmgm_mp/commands.h
 create mode 100644 examples/devmgm_mp/main.c
 create mode 100644 examples/devmgm_mp/meson.build

diff --git a/examples/devmgm_mp/Makefile b/examples/devmgm_mp/Makefile
new file mode 100644
index 000000000..e6c0cb0c5
--- /dev/null
+++ b/examples/devmgm_mp/Makefile
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+# binary name
+APP = devmgm_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+# Build using pkg-config variables if possible
+$(shell pkg-config --exists libdpdk)
+ifeq ($(.SHELLSTATUS),0)
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+	ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+	ln -sf $(APP)-static build/$(APP)
+
+PC_FILE := $(shell pkg-config --path libdpdk)
+CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
+LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
+LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+	@mkdir -p $@
+
+.PHONY: clean
+clean:
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rmdir --ignore-fail-on-non-empty build
+
+else
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = devmgm_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS_parse_obj_list.o := -D_GNU_SOURCE
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/devmgm_mp/commands.c b/examples/devmgm_mp/commands.c
new file mode 100644
index 000000000..145cb766e
--- /dev/null
+++ b/examples/devmgm_mp/commands.c
@@ -0,0 +1,383 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation.
+ * Copyright (c) 2009, Olivier MATZ <zer0@droids-corp.org>
+ * All rights reserved.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <netinet/in.h>
+#include <termios.h>
+#ifndef __linux__
+	#ifdef __FreeBSD__
+		#include <sys/socket.h>
+	#else
+		#include <net/socket.h>
+	#endif
+#endif
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		/* Secondary process's ethdev->state may not be
+		 * updated after detach on primary process,  but
+		 * ethdev->data should already be reset, so
+		 * use strlen(dev_name) == 0 to know the port is
+		 * not used.
+		 *
+		 * TODO: Secondary process should be informed when a
+		 * port is released on primary through mp channel.
+		 */
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id, NULL, NULL);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id, NULL, NULL);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/devmgm_mp/commands.h b/examples/devmgm_mp/commands.h
new file mode 100644
index 000000000..791204547
--- /dev/null
+++ b/examples/devmgm_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/devmgm_mp/main.c b/examples/devmgm_mp/main.c
new file mode 100644
index 000000000..f2f2e5a2f
--- /dev/null
+++ b/examples/devmgm_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2014 Intel Corporation.
+ * Copyright (c) 2009, Olivier MATZ <zer0@droids-corp.org>
+ * All rights reserved.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	return 0;
+}
diff --git a/examples/devmgm_mp/meson.build b/examples/devmgm_mp/meson.build
new file mode 100644
index 000000000..f916eb9af
--- /dev/null
+++ b/examples/devmgm_mp/meson.build
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2017 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+sources = files(
+	'commands.c', 'main.c'
+)
-- 
2.13.6

Re: [dpdk-dev] [PATCH 13/22] net/bonding: enable port detach on secondary process

[Chas Williams]

On Thu, Jun 7, 2018 at 8:38 AM, Qi Zhang <qi.z.zhang@intel.com> wrote:

> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_local, we can support this with
> minor change.
>

This commit message needs some work.  Otherwise, I think this particular
patch is fine.


>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>  drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c
> b/drivers/net/bonding/rte_eth_bond_pmd.c
> index 02d94b1b1..1221f62b2 100644
> --- a/drivers/net/bonding/rte_eth_bond_pmd.c
> +++ b/drivers/net/bonding/rte_eth_bond_pmd.c
> @@ -3065,6 +3065,7 @@ bond_probe(struct rte_vdev_device *dev)
>                 }
>                 /* TODO: request info from primary to set up Rx and Tx */
>                 eth_dev->dev_ops = &default_dev_ops;
> +               eth_dev->device = &dev->device;
>                 rte_eth_dev_probing_finish(eth_dev);
>                 return 0;
>         }
> @@ -3171,6 +3172,16 @@ bond_remove(struct rte_vdev_device *dev)
>         if (eth_dev == NULL)
>                 return -ENODEV;
>
> +       if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +               /* detach device on local pprocess only */
>

process?


> +               if (strlen(rte_vdev_device_args(dev)) == 0)
> +                       return rte_eth_dev_release_port_local(eth_dev);
> +               /**
> +                * else this is a private device for current process
> +                * so continue with normal detach scenario
> +                */
> +       }
> +
>         RTE_ASSERT(eth_dev->device == &dev->device);
>
>         internals = eth_dev->data->dev_private;
> --
> 2.13.6
>
>

Re: [dpdk-dev] [PATCH 20/22] net/tap: enable port detach on secondary process

[Wiles, Keith]

> On Jun 7, 2018, at 5:38 AM, Qi Zhang <qi.z.zhang@intel.com> wrote:
> 
> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_local, we can support this with
> minor change.

Previously, detach ports on secondary process will mess with the primary
process and cause the device to be not able to attach again. Taking
advantage of the rte_eth_release_port_local call we can fix the problem
with minor changes.

> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
> 1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
> index 5531fe9d9..56d3b6cc9 100644
> --- a/drivers/net/tap/rte_eth_tap.c
> +++ b/drivers/net/tap/rte_eth_tap.c
> @@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
> 		}
> 		/* TODO: request info from primary to set up Rx and Tx */
> 		eth_dev->dev_ops = &ops;
> +		eth_dev->device = &dev->device;
> 		rte_eth_dev_probing_finish(eth_dev);
> 		return 0;
> 	}
> @@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
> {
> 	struct rte_eth_dev *eth_dev = NULL;
> 	struct pmd_internals *internals;
> +	const char *name;
> 	int i;
> 
> +	name = rte_vdev_device_name(dev);
> 	/* find the ethdev entry */
> -	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
> +	eth_dev = rte_eth_dev_allocated(name);
> 	if (!eth_dev)
> -		return 0;
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/* detach device on local pprocess only */

/pprocess/process/
> +		if (strlen(rte_vdev_device_args(dev)) == 0)

What does strlen() do with a null string returned by rte_vdev_device_args(), I believe it just returns with zero, but we need to make sure. If it does not then we must protect strlen().

> +			return rte_eth_dev_release_port_local(eth_dev);
> +		/**
> +		 * else this is a private device for current process
> +		 * so continue with normal detach scenario
> +		 */
> +	}
> 
> 	internals = eth_dev->data->dev_private;
> 
> -- 
> 2.13.6
> 

Regards,
Keith

Re: [dpdk-dev] [PATCH 01/22] eal: introduce one device scan

[Shreyansh Jain]

On 6/7/2018 6:08 PM, Qi Zhang wrote:
> When hot plug a new device, it is not necessary to scan everything
> on the bus since the devname and devargs are already there. So new
> rte_bus ops "scan_one" is introduced, bus driver can implement this
> function to simply the hotplug process.
              ^^^^^^^^^
              simplify

> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
>   lib/librte_eal/common/include/rte_bus.h |  4 ++++
>   2 files changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
> index 61cb3b162..1ad033536 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
>   	if (ret)
>   		goto err_devarg;
>   
> -	ret = bus->scan();
> -	if (ret)
> -		goto err_devarg;
> +	/**
> +	 * if bus support to scan specific device by devargs,
> +	 * we don't need to scan all devices on the bus.
> +	 */
> +	if (bus->scan_one) {
> +		dev = bus->scan_one(da);
> +	} else {
> +		ret = bus->scan();
> +		if (ret)
> +			goto err_devarg;
> +
> +		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
> +	}
>   
> -	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
>   	if (dev == NULL) {
>   		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
>   			devname);
> diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
> index eb9eded4e..b15cff892 100644
> --- a/lib/librte_eal/common/include/rte_bus.h
> +++ b/lib/librte_eal/common/include/rte_bus.h
> @@ -83,6 +83,7 @@ enum rte_iova_mode {
>    */
>   typedef int (*rte_bus_scan_t)(void);
>   
> +typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *);

You should add comments over the declaration, just like the other 
similar declarations.
And, a new line should be here.

>   /**
>    * Implementation specific probe function which is responsible for linking
>    * devices on that bus with applicable drivers.
> @@ -95,6 +96,8 @@ typedef int (*rte_bus_scan_t)(void);
>    */
>   typedef int (*rte_bus_probe_t)(void);
>   
> +
> +

And please remove the extra lines added above in next version of patch.

>   /**
>    * Device iterator to find a device on a bus.
>    *
> @@ -204,6 +207,7 @@ struct rte_bus {
>   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
>   	const char *name;            /**< Name of the bus */
>   	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
> +	rte_bus_scan_one_t scan_one; /**< Scan one device by devargs */

I think you mean "Scan one device using devargs" rather than "Scan one 
device by devargs".

>   	rte_bus_probe_t probe;       /**< Probe devices on bus */
>   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
>   	rte_bus_plug_t plug;         /**< Probe single device for drivers */
> 

Re: [dpdk-dev] [PATCH 02/22] bus/vdev: enable one device scan

[Shreyansh Jain]

On 6/7/2018 6:08 PM, Qi Zhang wrote:
> Implemented the bus ops scan_one, besides this improve the scan
> efficiency in hotplug case, it aslo avoid sync IPC invoke (which
                                  ^^^^
                                  also

> happens in vdev->scan on secondary process). The benifit is it
                                                    ^^^^^^^
                                                    benefit

> removes the potiential deadlock in the case when secondary process
               ^^^^^^^^^^
               potential

> receive a request from primary process to attach a new device, since
> vdev->scan will be invoked on mp thread itself at this case.
                                                  ^^^^^^^
                                                  in that


Besides the above spells, is it possible to re-write the commit?
You mention it "...improves the scan efficiency..." - how? Is that an 
implicit output of introducing the new scan_one for vdev?

> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
>   1 file changed, 30 insertions(+)
> 
> diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
> index 6139dd551..cdbd77df0 100644
> --- a/drivers/bus/vdev/vdev.c
> +++ b/drivers/bus/vdev/vdev.c
> @@ -467,6 +467,35 @@ vdev_scan(void)
>   	return 0;
>   }
>   

[...]

Re: [dpdk-dev] [PATCH 01/22] eal: introduce one device scan

[Zhang, Qi Z]

Hi Shreyansh:
	Thanks for your review.
	Will fix base on your comments in v2.
Regards
Qi

> -----Original Message-----
> From: Shreyansh Jain [mailto:shreyansh.jain@nxp.com]
> Sent: Friday, June 8, 2018 7:12 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 01/22] eal: introduce one device scan
> 
> On 6/7/2018 6:08 PM, Qi Zhang wrote:
> > When hot plug a new device, it is not necessary to scan everything on
> > the bus since the devname and devargs are already there. So new
> > rte_bus ops "scan_one" is introduced, bus driver can implement this
> > function to simply the hotplug process.
>               ^^^^^^^^^
>               simplify
> 
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
> >   lib/librte_eal/common/include/rte_bus.h |  4 ++++
> >   2 files changed, 17 insertions(+), 4 deletions(-)
> >
> > diff --git a/lib/librte_eal/common/eal_common_dev.c
> > b/lib/librte_eal/common/eal_common_dev.c
> > index 61cb3b162..1ad033536 100644
> > --- a/lib/librte_eal/common/eal_common_dev.c
> > +++ b/lib/librte_eal/common/eal_common_dev.c
> > @@ -147,11 +147,20 @@ int __rte_experimental
> rte_eal_hotplug_add(const char *busname, const char *devn
> >   	if (ret)
> >   		goto err_devarg;
> >
> > -	ret = bus->scan();
> > -	if (ret)
> > -		goto err_devarg;
> > +	/**
> > +	 * if bus support to scan specific device by devargs,
> > +	 * we don't need to scan all devices on the bus.
> > +	 */
> > +	if (bus->scan_one) {
> > +		dev = bus->scan_one(da);
> > +	} else {
> > +		ret = bus->scan();
> > +		if (ret)
> > +			goto err_devarg;
> > +
> > +		dev = bus->find_device(NULL, cmp_detached_dev_name,
> devname);
> > +	}
> >
> > -	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
> >   	if (dev == NULL) {
> >   		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
> >   			devname);
> > diff --git a/lib/librte_eal/common/include/rte_bus.h
> > b/lib/librte_eal/common/include/rte_bus.h
> > index eb9eded4e..b15cff892 100644
> > --- a/lib/librte_eal/common/include/rte_bus.h
> > +++ b/lib/librte_eal/common/include/rte_bus.h
> > @@ -83,6 +83,7 @@ enum rte_iova_mode {
> >    */
> >   typedef int (*rte_bus_scan_t)(void);
> >
> > +typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs
> > +*);
> 
> You should add comments over the declaration, just like the other similar
> declarations.
> And, a new line should be here.
> 
> >   /**
> >    * Implementation specific probe function which is responsible for
> linking
> >    * devices on that bus with applicable drivers.
> > @@ -95,6 +96,8 @@ typedef int (*rte_bus_scan_t)(void);
> >    */
> >   typedef int (*rte_bus_probe_t)(void);
> >
> > +
> > +
> 
> And please remove the extra lines added above in next version of patch.
> 
> >   /**
> >    * Device iterator to find a device on a bus.
> >    *
> > @@ -204,6 +207,7 @@ struct rte_bus {
> >   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list
> */
> >   	const char *name;            /**< Name of the bus */
> >   	rte_bus_scan_t scan;         /**< Scan for devices attached to
> bus */
> > +	rte_bus_scan_one_t scan_one; /**< Scan one device by devargs */
> 
> I think you mean "Scan one device using devargs" rather than "Scan one
> device by devargs".
> 
> >   	rte_bus_probe_t probe;       /**< Probe devices on bus */
> >   	rte_bus_find_device_t find_device; /**< Find a device on the bus
> */
> >   	rte_bus_plug_t plug;         /**< Probe single device for drivers
> */
> >

Re: [dpdk-dev] [PATCH 02/22] bus/vdev: enable one device scan

[Zhang, Qi Z]

> -----Original Message-----
> From: Shreyansh Jain [mailto:shreyansh.jain@nxp.com]
> Sent: Friday, June 8, 2018 8:08 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 02/22] bus/vdev: enable one device scan
> 
> On 6/7/2018 6:08 PM, Qi Zhang wrote:
> > Implemented the bus ops scan_one, besides this improve the scan
> > efficiency in hotplug case, it aslo avoid sync IPC invoke (which
>                                   ^^^^
>                                   also
> 
> > happens in vdev->scan on secondary process). The benifit is it
>                                                     ^^^^^^^
>                                                     benefit
> 
> > removes the potiential deadlock in the case when secondary process
>                ^^^^^^^^^^
>                potential
> 
> > receive a request from primary process to attach a new device, since
> > vdev->scan will be invoked on mp thread itself at this case.
>                                                   ^^^^^^^
>                                                   in that
> 
> 
> Besides the above spells, is it possible to re-write the commit?
> You mention it "...improves the scan efficiency..." - how? Is that an implicit
> output of introducing the new scan_one for vdev?

"Improve scan efficiency" should be general to all buses in hot plug case.
since compare to bus->scan, bus->scan_one no need to iterate all devargs.
But yes, it's not the original purpose for this patch set, but a bonus.

I will re-write comment with below format to make it more clear.
The patch implemented bus ops scan_one for vdev, it gives two benefits
1. improve scan efficiency ....
2. avoid sync IPC invoke .....

Regards
Qi

> 
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
> >   1 file changed, 30 insertions(+)
> >
> > diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c index
> > 6139dd551..cdbd77df0 100644
> > --- a/drivers/bus/vdev/vdev.c
> > +++ b/drivers/bus/vdev/vdev.c
> > @@ -467,6 +467,35 @@ vdev_scan(void)
> >   	return 0;
> >   }
> >
> 
> [...]

Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process

[Burakov, Anatoly]

Hi Qi,

I haven't read the code yet, and i'll be the first to admit that i'm not 
too well versed on how shared/private device data works, so my apologies 
in advance if all of below comments are addressed by implementation 
details or are way off base!



On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 

<...>

> Case 7, 8:
> Secondary process can also temporally to detach a share device
> "privately" then attach it back later, this action also not impact other
> processes.
>

Do we really need to implement these cases? It seems to me that this 
"reattach it later" introduces unnecessary complexity. If secondary has 
detached the device, i think it is safer if we cannot reattach it, 
period, because it was a shared device. What if we try to attach it when 
a handshake has already completed and all other processes expect to 
detach it?

(in fact, do we differentiate between non-existent device and shared 
device that has been "privately detached"? I would expect that we keep 
the device as detached as opposed to forgetting about it, so that, come 
handshake, we can safely reply "yeah, we can detach the device", but 
maybe it's OK to not treat request to detach a non-existent device as an 
error... thoughts? am i getting something wrong?)

> APIs chenages:
> ==============
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in
> secondary process.
> 
> New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached. This help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.
> Aslo the new API let application to register a callback function
> which will be invoked before a device is going to be detached,
> the return value of the function will decide if device will continue
> be detached or not, this support application to do condition check
> at runtime.

I assume that you've added device locking to avoid having to do 
handshake before detach, correct? Is this a shared lock of some kind, or 
is it a private lock? If it's shared lock, what happens if the process 
holding that lock crashes?

> 
> PMD Impact:
> ===========
> 
> Currently device removing is not handled well in secondary process on most
> pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
> primary process since it reset all shared data. So we introduced new API
> rte_eth_dev_release_port_local which only reset ethdev's state to unsued but
> not touch shared data so other process will not be impacted.
> Since not all device driver is target to support primary-secondary
> process model, so the patch set only fix this on all Intel devices and
> vdev, it can be refereneced by other driver when equevalent fix is required

Nitpick - why the naming mismatch between *_private() and *_local()?

> 
> Limitation:
> ===========
> 
> The solution does not cover the case that primary process exit while
> secondary processes still be active. Though this is not a typial use
> case, but if this happens:
> 1. secondary process can't attach / detach any shared device since no
> primary exist.
> 2. secondary process still can attach / detach private device.
> 3. secondary process still can detach a share device privately but may
> not attach it back, that ethdev slot will become zombie slot.

I think this should be explicit and by design. Shared devices can only 
be communicated to all secondaries through a primary process. No primary 
- no shared devices. I don't think we can do anything about it unless we 
implement some kind of peer-to-peer IPC (which isn't happening as far as 
i'm aware).



Thanks for your work on this patchset!

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 05/22] ethdev: introduce device lock

[Burakov, Anatoly]

On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.
> 
> Aslo the new API let application to register a callback function
> which will be invoked before a device is going to be detached,
> the return value of the function will decide if device will continue
> be detached or not, this support application to do condition check
> at runtime.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +
> +int
> +rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> +		 void *user_args)
> +{
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +
> +	if (callback == NULL)
> +		return register_lock_callback(port_id, dev_is_busy, NULL);
> +	else
> +		return register_lock_callback(port_id, callback, user_args);

As much as i don't like seeing negative errno values as return, the rest 
of ethdev library uses those, so this is OK :)

> +}
> +
> +int
> +rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> +		   void *user_args)
> +{
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> +

<snip>

> + * Also, any callback function return !0 value will prevent device be
> + * detached(ref. rte_eth_dev_lock and rte_eth_dev_unlock).
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param user_args
> + *   This is parameter "user_args" be saved when callback function is
> + *   registered(rte_dev_eth_lock).
> + *
> + * @return
> + *   0  device is allowed be detached.
> + *   !0 device is not allowed be detached.

!0 can be negative or positive. Are we expecting positive return values 
from this API?

> + */
> +typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
> +
> +/**
> + * Lock an Ethernet Device directly or register a callback function
> + * for condition check at runtime, this help application to prevent
> + * a device be detached unexpectly.
> + * NOTE: Lock a device mutliple times with same parmeter will increase
> + * a ref_count, and coresponding unlock decrease the ref_count, the
> + * device will be unlocked when ref_count reach 0.

Nitpick: "note" sections should be done with @note marker.

Also, i would mention that this is a per-process lock that does not 
affect other processes (assuming i understood the code correctly, of 
course...).

> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param callback
> + *   !NULL the callback function will be added into a pre-detach list,
> + *         it will be invoked when a device is going to be detached. The
> + *         return value will decide if continue detach the device or not.
> + *   NULL  lock the device directly, basically this just regiter a empty
> + *         callback function(dev_is_busy) that return -EBUSY, so we can
> + *         handle the pre-detach check in unified way.
> + * @param user_args
> + *   parameter will be parsed to callback function, only valid when
> + *   callback != NULL.
> + * @return
> + *   0 on success, negative on error.
> + */
> +int rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> +		     void *user_args);

Nitpicks: DPDK style guide discourages using spaces as indentation 
(other parts of this patch, and other patches have this issue as well).

> +
> +/**
> + * Reverse operation of rte_eth_dev_lock.
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param callback
> + *   NULL  decrease the ref_count of default callback function.
> + *   !NULL decrease the ref_count of specific callback with matched
> + *         user_args.
> + * @param user_args
> + *   parameter to match, only valid when callback != NULL.
> + * @return
> + *   0 on success, negative on error.
> + */
> +int rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> +		       void *user_args);
> +
>   #ifdef __cplusplus
>   }
>   #endif
> diff --git a/lib/librte_ethdev/rte_ethdev_lock.c b/lib/librte_ethdev/rte_ethdev_lock.c

rte_ethdev_lock.* seem to be internal-only files. Perhaps you should 
name them without the rte_ prefix to indicate that they're not exported?

> new file mode 100644
> index 000000000..688d1d70a
> --- /dev/null
> +++ b/lib/librte_ethdev/rte_ethdev_lock.c
> @@ -0,0 +1,102 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +#include "rte_ethdev_lock.h"
> +
> +struct lock_entry {
> +	TAILQ_ENTRY(lock_entry) next;
> +	rte_eth_dev_lock_callback_t callback;
> +	uint16_t port_id;

<snip>

> +register_lock_callback(uint16_t port_id,
> +		       rte_eth_dev_lock_callback_t callback,
> +		       void *user_args)
> +{
> +	struct lock_entry *le;
> +
> +	rte_spinlock_lock(&lock_entry_lock);
> +
> +	TAILQ_FOREACH(le, &lock_entry_list, next) {
> +		if (le->port_id == port_id &&
> +		    le->callback == callback &&
> +		    le->user_args == user_args)
> +			break;
> +	}
> +
> +	if (!le) {
> +		le = calloc(1, sizeof(struct lock_entry));
> +		if (!le) {

Nitpick: generally, DPDK style guide prefers "if (value)" or "if 
(!value)" to only be reserved for boolean values, and use explicit 
comparison (e.g. "if (value == NULL)" or "if (value == 0)") for all 
other cases.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 04/22] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> The patch introduce the solution to handle different hotplug cases in
> multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced, it will be implemented in
> following separate patch.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 3, 4:
> This will be implemented in following patch.
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowd to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs chenages:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_eal/common/eal_private.h   |   8 ++
>   lib/librte_eal/linuxapp/eal/eal.c     |   6 ++
>   lib/librte_ethdev/Makefile            |   1 +
>   lib/librte_ethdev/rte_ethdev.c        | 183 ++++++++++++++++++++++++++++---
>   lib/librte_ethdev/rte_ethdev.h        |  37 +++++++
>   lib/librte_ethdev/rte_ethdev_core.h   |   5 +
>   lib/librte_ethdev/rte_ethdev_driver.h |  27 +++++
>   lib/librte_ethdev/rte_ethdev_mp.c     | 195 ++++++++++++++++++++++++++++++++++
>   lib/librte_ethdev/rte_ethdev_mp.h     |  44 ++++++++
>   9 files changed, 489 insertions(+), 17 deletions(-)
>   create mode 100644 lib/librte_ethdev/rte_ethdev_mp.c
>   create mode 100644 lib/librte_ethdev/rte_ethdev_mp.h
> 

Haven't looked at the code yet, but general comment: please don't prefix 
internal-only files with rte_, it makes it look like they are part of 
external API.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 05/22] ethdev: introduce device lock

[Stephen Hemminger]

On Thu,  7 Jun 2018 20:38:32 +0800
Qi Zhang <qi.z.zhang@intel.com> wrote:

> +/**
> + * Lock an Ethernet Device directly or register a callback function
> + * for condition check at runtime, this help application to prevent
> + * a device be detached unexpectly.
> + * NOTE: Lock a device mutliple times with same parmeter will increase
> + * a ref_count, and coresponding unlock decrease the ref_count, the
> + * device will be unlocked when ref_count reach 0.
> + *
> + * @param port_id
> + *   The port identifier of the Ethernet device.
> + * @param callback
> + *   !NULL the callback function will be added into a pre-detach list,
> + *         it will be invoked when a device is going to be detached. The
> + *         return value will decide if continue detach the device or not.
> + *   NULL  lock the device directly, basically this just regiter a empty
> + *         callback function(dev_is_busy) that return -EBUSY, so we can
> + *         handle the pre-detach check in unified way.
> + * @param user_args
> + *   parameter will be parsed to callback function, only valid when
> + *   callback != NULL.
> + * @return
> + *   0 on success, negative on error.
> + */
> +int rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> +		     void *user_args);

I prefer API's that do one thing with one function.
Why not
	rte_eth_dev_lock(uint16_t port_id);
	rte_eth_dev_ondetach(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
           		     void *user_args);

Re: [dpdk-dev] [PATCH 04/22] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> The patch introduce the solution to handle different hotplug cases in
> multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced, it will be implemented in
> following separate patch.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 3, 4:
> This will be implemented in following patch.

If these will be implemented in following patch, why spend half the 
commit message talking about it? :) This commit doesn't implement 
secondary process functionality at all, so the commit message should 
probably be reworded to only include primary process logic, no?

> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowd to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs chenages:

Multiple typos - "chenages", "temporally", "allowd", etc.

> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
>  > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

>   	rte_eal_mcfg_complete();
>   
> +	if (rte_eth_dev_mp_init()) {
> +		rte_eal_init_alert("rte_eth_dev_mp_init() failed\n");
> +		rte_errno = ENOEXEC;
> +		return -1;
> +	}
> +

Why is this done after the end of init? rte_eal_mcfg_complete() makes it 
so that secondaries can initialize, at that point all initialization 
should have been finished. I would expect this to be called after 
(before?) bus probe, since this is device-related.

>   	return fctret;
>   }
>   
> diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
> index c2f2f7d82..04e93f337 100644
> --- a/lib/librte_ethdev/Makefile
> +++ b/lib/librte_ethdev/Makefile
> @@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
>   LIBABIVER := 9
>   

<snip>

> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +
> +		/**
> +		 * If secondary process, we just send request to primray
> +		 * to start the process.
> +		 */
> +		req.t = REQ_TYPE_ATTACH;
> +		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> +
> +		ret = rte_eth_dev_request_to_primary(&req);
> +		if (ret) {
> +			ethdev_log(ERR, "Failed to send device attach request to primary\n");

The log message is a little misleading. It can be that secondary has 
failed to send request. It can also be that it succeeded, but the attach 
itself has failed. I think a better message would be "attach request has 
failed" or something to that effect.

> +			return ret;
> +		}
> +
> +		*port_id = req.port_id;
> +		return req.result;
> +	}
> +
> +	ret = do_eth_dev_attach(devargs, port_id);
> +	if (ret)
> +		return ret;
> +
> +	/* send attach request to seoncary */
> +	req.t = REQ_TYPE_ATTACH;
> +	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> +	req.port_id = *port_id;
> +	ret = rte_eth_dev_request_to_secondary(&req);
> +	if (ret) {
> +		ethdev_log(ERR, "Failed to send device attach request to secondary\n");

Same as above - log message can/might be misleading. There are a few 
other places where similar log message is present, those should be 
corrected too.

> +		goto rollback;
> +	}
> +
> +	if (req.result)
> +		goto rollback;
> +
> +	return 0;

<snip>

> +{
> +	uint32_t dev_flags;
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +		return -ENOTSUP;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
> +
> +	dev_flags = rte_eth_devices[port_id].data->dev_flags;
> +	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
> +		ethdev_log(ERR,
> +			"Port %" PRIu16 " is bonded, cannot detach", port_id);
> +		return -ENOTSUP;
> +	}

Do we have to do a similar check for failsafe devices?

> +
> +	return do_eth_dev_detach(port_id);
> +}
> +
>   static int
>   rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
>   {
> diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
> index 36e3984ea..bb03d613b 100644
> --- a/lib/librte_ethdev/rte_ethdev.h
> +++ b/lib/librte_ethdev/rte_ethdev.h

<snip>

>   /**
> + * Attach a private Ethernet device specified by arguments.
> + * A private device is invisible to other process.
> + * Can only be invoked in secondary process.
> + *
> + * @param devargs
> + *  A pointer to a strings array describing the new device
> + *  to be attached. The strings should be a pci address like
> + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
> + * @param port_id
> + *  A pointer to a port identifier actually attached.
> + * @return
> + *  0 on success and port_id is filled, negative on error
> + */
> +int rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);

New API's should be marked as __rte_experimental.

> +
> +/**
>    * Detach a Ethernet device specified by port identifier.
>    * This function must be called when the device is in the
>    * closed state.
> + * In multi-process mode, it will sync with other process
> + * to detach the device.
>    *
>    * @param port_id
>    *   The port identifier of the device to detach.
> @@ -1490,6 +1511,22 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);

<snip>

> + * Detach a Ethernet device in current process.
> + *
> + * @param port_id
> + *   The port identifier of the device to detach.
> + * @param devname
> + *   A pointer to a buffer that will be filled with the device name.
> + *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
> + * @return
> + *  0 on success and devname is filled, negative on error
> + */
> +int do_eth_dev_detach(uint16_t port_id);
> +

Why is this made part of an external API? You should have a separate, 
private header file for these.

>   #ifdef __cplusplus
>   }
>   #endif
> diff --git a/lib/librte_ethdev/rte_ethdev_mp.c b/lib/librte_ethdev/rte_ethdev_mp.c
> new file mode 100644
> index 000000000..8ede8151d
> --- /dev/null
> +++ b/lib/librte_ethdev/rte_ethdev_mp.c
> @@ -0,0 +1,195 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation
> + */
> +
> +#include "rte_ethdev_driver.h"
> +#include "rte_ethdev_mp.h"
> +
> +static int detach_on_secondary(uint16_t port_id)

<snip>

> +	free(da.args);
> +	return 0;
> +}
> +
> +static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	(void)msg;
> +	(void)(peer);
> +	return -ENOTSUP;

Please either mark arguments as __rte_unused, or use RTE_SET_USED(blah) 
macro. Same in other similar places.

> +}
> +
> +static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	(void)msg;
> +	(void)(peer);
> +	return -ENOTSUP;
> +}
> +
> +static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;

<snip>

> +	case REQ_TYPE_DETACH:
> +	case REQ_TYPE_ATTACH_ROLLBACK:
> +		ret = detach_on_secondary(req->port_id);
> +		break;
> +	default:
> +		ret = -EINVAL;
> +	}
> +
> +	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);

Here and in other places: rte_strlcpy?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 06/22] ethdev: support attach or detach share device from secondary

[Burakov, Anatoly]

On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a share device
> attach/detach request be issued from secondary process, the implementation
> references malloc_mp.c.
> 
> device attach on secondary:
> a) seconary send asycn request to primary and wait on a condition
>     which will be released by matched response from primary.
> b) primary receive the request and attach the new device if failed
>     goto i).
> c) primary forward attach request to all secondary as async request
>     (because this in mp thread context, use sync request will deadlock)
> d) secondary receive request and attach device and send reply.
> e) primary check the reply if all success go to j).
> f) primary send attach rollback async request to all secondary.
> g) secondary receive the request and detach device and send reply.
> h) primary receive the reply and detach device as rollback action.
> i) send fail response to secondary, goto k).
> j) send success response to secondary.
> k) secondary process receive response and return.
> 
> device detach on secondary:
> a) secondary send async request to primary and wait on a condition
>     which will be released by matched response from primary.
> b) primary receive the request and  perform pre-detach check, if device
>     is locked, goto j).
> c) primary send pre-detach async request to all secondary.
> d) secondary perform pre-detach check and send reply.
> e) primary check the reply if any fail goto j).
> f) primary send detach async request to all secondary
> g) secondary detach the device and send reply
> h) primary detach the device.
> i) send success response to secondary, goto k).
> j) send fail response to secondary.
> k) secondary process receive response and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +TAILQ_HEAD(mp_request_list, mp_request);
> +static struct {
> +	struct mp_request_list list;
> +	pthread_mutex_t lock;
> +} mp_request_list = {
> +	.list = TAILQ_HEAD_INITIALIZER(mp_request_list.list),
> +	.lock = PTHREAD_MUTEX_INITIALIZER
> +};
> +
> +#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */

Patch number 4 should've used this #define to set up its timeout.

> +
> +static struct mp_request *
> +find_request_by_id(uint64_t id)
> +{
> +	struct mp_request *req;
> +
> +	TAILQ_FOREACH(req, &mp_request_list.list, next) {
> +		if (req->user_req.id == id)
> +			break;
> +	}
> +	return req;
> +}
> +

<snip>

> +		if (resp->result)
> +			return resp->result;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +send_response_to_secondary(const struct eth_dev_mp_req *req, int result)
> +{
> +	struct rte_mp_msg resp_msg = {0};

I've been bitten by this in the past - some compilers (*cough* clang 
*cough*) don't like this kind of zero-initialization depending on which 
type of parameter comes first in the structure, so i would refrain from 
using it and used memset(0) instead.

> +	struct eth_dev_mp_req *resp =
> +		(struct eth_dev_mp_req *)resp_msg.param;
> +	int ret = 0;
> +
> +	resp_msg.len_param = sizeof(*resp);
> +	strcpy(resp_msg.name, ETH_DEV_MP_ACTION_RESPONSE);

here and in other places - strlcpy()?

> +	memcpy(resp, req, sizeof(*req));
> +	resp->result = result;
> +
> +	ret = rte_mp_sendmsg(&resp_msg);
> +	if (ret)
> +		ethdev_log(ERR, "failed to send response to secondary\n");
> +
> +	return ret;
> +}
> +
> +static int
> +handle_async_attach_response(const struct rte_mp_msg *request,
> +			     const struct rte_mp_reply *reply)
> +{

<snip>

> +	else
> +		return -1;
> +	do {
> +		ret = rte_mp_request_async(&mp_req, &ts, clb);
> +	} while (ret != 0 && rte_errno == EEXIST);
> +
> +	if (ret)
> +		ethdev_log(ERR, "couldn't send async request\n");
> +	entry = find_request_by_id(req->id > +	(void)entry;

Why did you look up entry and then marked it as used without checking 
the return value? Leftover? Some code missing?

> +	return ret;
> +}
> +
> +static int handle_secondary_request(const struct rte_mp_msg *msg,
> +				    const void *peer __rte_unused)
> +{
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	struct eth_dev_mp_req tmp_req;

<snip>

> @@ -124,10 +490,101 @@ static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer
>   	return 0;
>   }
>   
> +/**
> + * secondary to primary request.
> + *
> + * device attach:
> + * a) seconary send request to primary.
> + * b) primary attach the new device if failed goto i).
> + * c) primary forward attach request to all secondary.
> + * d) secondary receive request and attach device and send reply.
> + * e) primary check the reply if all success go to j).
> + * f) primary send attach rollback request to all secondary.
> + * g) secondary receive the request and detach device and send reply.
> + * h) primary receive the reply and detach device as rollback action.
> + * i) send fail response to secondary, goto k).
> + * j) send success response to secondary.
> + * k) end.
> +
> + * device detach:
> + * a) secondary send request to primary.
> + * b) primary perform pre-detach check, if device is locked, got j).
> + * c) primary send pre-detach check request to all secondary.
> + * d) secondary perform pre-detach check and send reply.
> + * e) primary check the reply if any fail goto j).
> + * f) primary send detach request to all secondary
> + * g) secondary detach the device and send reply
> + * h) primary detach the device.
> + * i) send success response to secondary, goto k).
> + * j) send fail response to secondary.
> + * k) end.
> + */

I think this comment should be at the top of this file.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 22/22] examples/devmgm_mp: add simple device management sample

[Burakov, Anatoly]

On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> The sample code demonstrate device (ethdev only) management
> at multi-process envrionment. User can attach/detach a device
> on primary process and see it is synced on secondary process
> automatically, also user can lock a device to prevent it be
> detached or unlock it to go back to default behaviour.
> 
> How to start?
> ./devmgm_mp --proc-type=auto
> 
> Command Line Example:
> 
>> help
>> list
> 
> /* attach a af_packet vdev */
>> attach net_af_packet,iface=eth0
> 
> /* detach port 0 */
>> detach 0
> 
> /* attach a private af_packet vdev (secondary process only)*/
>> attachp net_af_packet,iface=eth0
> 
> /* detach a private device (secondary process only) */
>> detachp 0
> 
> /* lock port 0 */
>> lock 0
> 
> /* unlock port 0 */
>> unlock 0
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

I think the "devmgm_mp" is not a descriptive enough name. What this 
example demonstrates, is device hotplug. So how about naming the example 
app "hotplug"? (or "mp_hotplug" to indicate that it specifically sets 
out to demonstrate multiprocess hotplug)

>   examples/devmgm_mp/Makefile    |  64 +++++++
>   examples/devmgm_mp/commands.c  | 383 +++++++++++++++++++++++++++++++++++++++++
>   examples/devmgm_mp/commands.h  |  10 ++
>   examples/devmgm_mp/main.c      |  41 +++++
>   examples/devmgm_mp/meson.build |  11 ++
>   5 files changed, 509 insertions(+)
>   create mode 100644 examples/devmgm_mp/Makefile
>   create mode 100644 examples/devmgm_mp/commands.c
>   create mode 100644 examples/devmgm_mp/commands.h
>   create mode 100644 examples/devmgm_mp/main.c
>   create mode 100644 examples/devmgm_mp/meson.build
> 

<snip>

> +#include <stdio.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <stdlib.h>
> +#include <stdarg.h>
> +#include <errno.h>
> +#include <netinet/in.h>
> +#include <termios.h>
> +#ifndef __linux__
> +	#ifdef __FreeBSD__
> +		#include <sys/socket.h>
> +	#else
> +		#include <net/socket.h>
> +	#endif
> +#endif

This seems like a weird define. Care to elaborate why are we checking 
for __linux__ not being defined?

If you're trying to differentiate between Linux and FreeBSD, there's a 
readly RTE_EXEC_ENV_* config options, e.g.

#ifdef RTE_EXEC_ENV_LINUXAPP
// linux defines
#endif
#ifdef RTE_EXEC_ENV_BSDAPP
// bsd defines
#endif

or something to that effect.

> +
> +#include <cmdline_rdline.h>
> +#include <cmdline_parse.h>
> +#include <cmdline_parse_ipaddr.h>
> +#include <cmdline_parse_num.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +#include <rte_ethdev.h>

Generally (and as per DPDK coding guidelines), we prefer defines ordered 
as follows:

1) system defines enclosed in brackets
2) DPDK defines (rte_blah) enclosed in brackets
3) private/application-specific defines enclosed in quotes.

All three groups should be separated by newline.

So, these defines should've read as:

#include <stdblah.h>
#include <sys/blah.h>

#include <rte_blah.h>
#include <rte_foo.h>

#include "cmdline_blah.h"
#include "cmdline_foo.h"

> +
> +/**********************************************************/
> +
> +struct cmd_help_result {
> +	cmdline_fixed_string_t help;
> +};
> +
> +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,

<snip>

> +{
> +	uint16_t port_id;
> +	char dev_name[RTE_DEV_NAME_MAX_LEN];
> +
> +	cmdline_printf(cl, "list all etherdev\n");
> +
> +	RTE_ETH_FOREACH_DEV(port_id) {
> +		rte_eth_dev_get_name_by_port(port_id, dev_name);
> +		/* Secondary process's ethdev->state may not be
> +		 * updated after detach on primary process,  but
> +		 * ethdev->data should already be reset, so
> +		 * use strlen(dev_name) == 0 to know the port is
> +		 * not used.
> +		 *
> +		 * TODO: Secondary process should be informed when a
> +		 * port is released on primary through mp channel.
> +		 */

That seems like a weird thing to leave out for TODO - it looks like an 
API deficiency. Can this be automatically updated on multiprocess 
hotplug sync, or somehow managed inside RTE_ETH_FOREACH_DEV?

As i understand, per-process ethdev list is not protected by any locks, 
so doing this is racy. Since this is a multiprocess hotplug example app, 
it should demonstrate best practices. So, either RTE_ETH_FOREACH_DEV 
should be fixed to handle this case, or the application should 
demonstrate how to properly synchronize access to local device list. The 
latter is probably better as adding locking around ethdev device list is 
outside the scope of this patchset.

> +		if (strlen(dev_name) > 0)
> +			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
> +		else
> +			printf("empty dev_name is not expected!\n");
> +	}

<snip>

> +#include "commands.h"
> +
> +int main(int argc, char **argv)
> +{
> +	int ret;
> +	struct cmdline *cl;
> +
> +	ret = rte_eal_init(argc, argv);
> +	if (ret < 0)
> +		rte_panic("Cannot init EAL\n");
> +
> +	cl = cmdline_stdin_new(main_ctx, "example> ");
> +	if (cl == NULL)
> +		rte_panic("Cannot create cmdline instance\n");
> +	cmdline_interact(cl);
> +	cmdline_stdin_exit(cl);
> +
> +	return 0;

Application should call rte_eal_cleanup() before exit. Otherwise, each 
secondary started and stopped will leak memory.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 00/22] enable hotplug on multi-process

[Zhang, Qi Z]

Hi Anatoly:

	Thanks for the review, see my reply in inline.

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Friday, June 15, 2018 11:16 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH 00/22] enable hotplug on multi-process
> 
> Hi Qi,
> 
> I haven't read the code yet, and i'll be the first to admit that i'm not too well
> versed on how shared/private device data works, so my apologies in advance
> if all of below comments are addressed by implementation details or are way
> off base!
> 
> 
> 
> On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> >
> > 1. Attach a share device from primary
> > 2. Detach a share device from primary
> > 3. Attach a share device from secondary 4. Detach a share device from
> > secondary 5. Attach a private device from secondary 6. Detach a
> > private device from secondary 7. Detach a share device from secondary
> > privately 8. Attach a share device from secondary privately
> >
> 
> <...>
> 
> > Case 7, 8:
> > Secondary process can also temporally to detach a share device
> > "privately" then attach it back later, this action also not impact other
> > processes.
> >
> 
> Do we really need to implement these cases? It seems to me that this
> "reattach it later" introduces unnecessary complexity. 

I agree it's not necessary, but this looks like a free feature based on current implementation :)


>If secondary has
> detached the device, I think it is safer if we cannot reattach it,
> period, because it was a shared device. What if we try to attach it when
> a handshake has already completed and all other processes expect to
> detach it?

in the case: attach back a shared device already be detached will fail as expected.

For PCI devices, it will fail at driver probe.
For vdev, it will failed at rte_eth_dev_attach_secondary.

> 
> (in fact, do we differentiate between non-existent device and shared
> device that has been "privately detached"? I would expect that we keep
> the device as detached as opposed to forgetting about it, so that, come
> handshake, we can safely reply "yeah, we can detach the device", but
> maybe it's OK to not treat request to detach a non-existent device as an
> error... thoughts? am i getting something wrong?)
> 
> > APIs chenages:
> > ==============
> >
> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> > share device attach/detach in primary-secondary process model, it will
> > be called in case 1,2,3,4.
> >
> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> > introduced to cover case 5,6,7,8, this API can only be invoked in
> > secondary process.
> >
> > New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
> > application lock or unlock on specific ethdev, a locked device
> > can't be detached. This help applicaiton to prevent unexpected
> > device detaching, especially in multi-process envrionment.
> > Aslo the new API let application to register a callback function
> > which will be invoked before a device is going to be detached,
> > the return value of the function will decide if device will continue
> > be detached or not, this support application to do condition check
> > at runtime.
> 
> I assume that you've added device locking to avoid having to do
> handshake before detach, correct? 
Yes.
>Is this a shared lock of some kind, or
> is it a private lock? If it's shared lock, what happens if the process
> holding that lock crashes?

It’s a kind of process's private lock, but a shared device be locked any process will prevent it be detached.

> 
> >
> > PMD Impact:
> > ===========
> >
> > Currently device removing is not handled well in secondary process on
> most
> > pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
> > primary process since it reset all shared data. So we introduced new API
> > rte_eth_dev_release_port_local which only reset ethdev's state to unsued
> but
> > not touch shared data so other process will not be impacted.
> > Since not all device driver is target to support primary-secondary
> > process model, so the patch set only fix this on all Intel devices and
> > vdev, it can be refereneced by other driver when equevalent fix is required
> 
> Nitpick - why the naming mismatch between *_private() and *_local()?
Agree.
"private" make the API more identical.

> 
> >
> > Limitation:
> > ===========
> >
> > The solution does not cover the case that primary process exit while
> > secondary processes still be active. Though this is not a typial use
> > case, but if this happens:
> > 1. secondary process can't attach / detach any shared device since no
> > primary exist.
> > 2. secondary process still can attach / detach private device.
> > 3. secondary process still can detach a share device privately but may
> > not attach it back, that ethdev slot will become zombie slot.
> 
> I think this should be explicit and by design. Shared devices can only
> be communicated to all secondaries through a primary process. No primary
> - no shared devices. I don't think we can do anything about it unless we
> implement some kind of peer-to-peer IPC (which isn't happening as far as
> i'm aware).

Agree.

> 
> 
> 
> Thanks for your work on this patchset!

Thanks for the design review and all the helpful inputs.

Regards
Qi

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH 04/22] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, June 18, 2018 4:18 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH 04/22] ethdev: enable hotplug on multi-process
> 
> On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> > The patch introduce the solution to handle different hotplug cases in
> > multi-process situation, it include below scenario:
> >
> > 1. Attach a share device from primary
> > 2. Detach a share device from primary
> > 3. Attach a share device from secondary 4. Detach a share device from
> > secondary 5. Attach a private device from secondary 6. Detach a
> > private device from secondary 7. Detach a share device from secondary
> > privately 8. Attach a share device from secondary privately
> >
> > In primary-secondary process model, we assume device is shared by
> default.
> > that means attach or detach a device on any process will broadcast to
> > all other processes through mp channel then device information will be
> > synchronized on all processes.
> >
> > Any failure during attaching process will cause inconsistent status
> > between processes, so proper rollback action should be considered.
> > Also it is not safe to detach a share device when other process still
> > use it, so a handshake mechanism is introduced, it will be implemented
> > in following separate patch.
> >
> > Scenario for Case 1, 2:
> >
> > attach device
> > a) primary attach the new device if failed goto h).
> > b) primary send attach sync request to all secondary.
> > c) secondary receive request and attach device and send reply.
> > d) primary check the reply if all success go to i).
> > e) primary send attach rollback sync request to all secondary.
> > f) secondary receive the request and detach device and send reply.
> > g) primary receive the reply and detach device as rollback action.
> > h) attach fail
> > i) attach success
> >
> > detach device
> > a) primary perform pre-detach check, if device is locked, goto i).
> > b) primary send pre-detach sync request to all secondary.
> > c) secondary perform pre-detach check and send reply.
> > d) primary check the reply if any fail goto i).
> > e) primary send detach sync request to all secondary
> > f) secondary detach the device and send reply (assume no fail)
> > g) primary detach the device.
> > h) detach success
> > i) detach failed
> >
> > Case 3, 4:
> > This will be implemented in following patch.
> 
> If these will be implemented in following patch, why spend half the commit
> message talking about it? :) 

Sorry, I didn't get your point about "see half commit to talk about it" :)
This patch covered an overview, and also the implementation of case 1,2,5,6,7,8

For case 3, 4, just below 4 lines to describe it

3. Attach a share device from secondary.
4. Detach a share device from secondary.
Case 3, 4:
This will be implemented in following patch.

> is commit doesn't implement secondary
> process functionality at all, so the commit message should probably be
> reworded to only include primary process logic, no?

OK, I will reword it to highlight the patch's scope as description at above.

> 
> >
> > Case 5, 6:
> > Secondary process can attach private device which only visible to itself,
> > in this case no IPC is involved, primary process is not allowed to have
> > private device so far.
> >
> > Case 7, 8:
> > Secondary process can also temporally to detach a share device "privately"
> > then attach it back later, this action also not impact other processes.
> >
> > APIs chenages:
> 
> Multiple typos - "chenages", "temporally", "allowd", etc.

Thanks

> 
> >
> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> > share device attach/detach in primary-secondary process model, it will
> > be called in case 1,2,3,4.
> >
> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> > introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> > process.
> >  > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> >   	rte_eal_mcfg_complete();
> >
> > +	if (rte_eth_dev_mp_init()) {
> > +		rte_eal_init_alert("rte_eth_dev_mp_init() failed\n");
> > +		rte_errno = ENOEXEC;
> > +		return -1;
> > +	}
> > +
> 
> Why is this done after the end of init? rte_eal_mcfg_complete() makes it
> so that secondaries can initialize, at that point all initialization
> should have been finished. I would expect this to be called after
> (before?) bus probe, since this is device-related.

OK will move ahead.

> 
> >   	return fctret;
> >   }
> >
> > diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
> > index c2f2f7d82..04e93f337 100644
> > --- a/lib/librte_ethdev/Makefile
> > +++ b/lib/librte_ethdev/Makefile
> > @@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
> >   LIBABIVER := 9
> >
> 
> <snip>
> 
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > +
> > +		/**
> > +		 * If secondary process, we just send request to primray
> > +		 * to start the process.
> > +		 */
> > +		req.t = REQ_TYPE_ATTACH;
> > +		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> > +
> > +		ret = rte_eth_dev_request_to_primary(&req);
> > +		if (ret) {
> > +			ethdev_log(ERR, "Failed to send device attach request to
> primary\n");
> 
> The log message is a little misleading. It can be that secondary has
> failed to send request. It can also be that it succeeded, but the attach
> itself has failed. I think a better message would be "attach request has
> failed" or something to that effect.

The return value of rte_eth_dev_request_to_primary only means communication fail,
(message not able to send, or not get reply in time).
but not the fail on attach/detach itself. (which comes from req->result)

> 
> > +			return ret;
> > +		}
> > +
> > +		*port_id = req.port_id;
> > +		return req.result;
> > +	}
> > +
> > +	ret = do_eth_dev_attach(devargs, port_id);
> > +	if (ret)
> > +		return ret;
> > +
> > +	/* send attach request to seoncary */
> > +	req.t = REQ_TYPE_ATTACH;
> > +	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> > +	req.port_id = *port_id;
> > +	ret = rte_eth_dev_request_to_secondary(&req);
> > +	if (ret) {
> > +		ethdev_log(ERR, "Failed to send device attach request to
> secondary\n");
> 
> Same as above - log message can/might be misleading. There are a few
> other places where similar log message is present, those should be
> corrected too.

Same as above

> 
> > +		goto rollback;
> > +	}
> > +
> > +	if (req.result)
> > +		goto rollback;
> > +
> > +	return 0;
> 
> <snip>
> 
> > +{
> > +	uint32_t dev_flags;
> > +
> > +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > +		return -ENOTSUP;
> > +
> > +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
> > +
> > +	dev_flags = rte_eth_devices[port_id].data->dev_flags;
> > +	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
> > +		ethdev_log(ERR,
> > +			"Port %" PRIu16 " is bonded, cannot detach", port_id);
> > +		return -ENOTSUP;
> > +	}
> 
> Do we have to do a similar check for failsafe devices?

Just keep it same logic as before, it could be a separate patch to fix I guess.

> 
> > +
> > +	return do_eth_dev_detach(port_id);
> > +}
> > +
> >   static int
> >   rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t
> nb_queues)
> >   {
> > diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
> > index 36e3984ea..bb03d613b 100644
> > --- a/lib/librte_ethdev/rte_ethdev.h
> > +++ b/lib/librte_ethdev/rte_ethdev.h
> 
> <snip>
> 
> >   /**
> > + * Attach a private Ethernet device specified by arguments.
> > + * A private device is invisible to other process.
> > + * Can only be invoked in secondary process.
> > + *
> > + * @param devargs
> > + *  A pointer to a strings array describing the new device
> > + *  to be attached. The strings should be a pci address like
> > + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
> > + * @param port_id
> > + *  A pointer to a port identifier actually attached.
> > + * @return
> > + *  0 on success and port_id is filled, negative on error
> > + */
> > +int rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
> 
> New API's should be marked as __rte_experimental.

OK

> 
> > +
> > +/**
> >    * Detach a Ethernet device specified by port identifier.
> >    * This function must be called when the device is in the
> >    * closed state.
> > + * In multi-process mode, it will sync with other process
> > + * to detach the device.
> >    *
> >    * @param port_id
> >    *   The port identifier of the device to detach.
> > @@ -1490,6 +1511,22 @@ int rte_eth_dev_attach(const char *devargs,
> uint16_t *port_id);
> 
> <snip>
> 
> > + * Detach a Ethernet device in current process.
> > + *
> > + * @param port_id
> > + *   The port identifier of the device to detach.
> > + * @param devname
> > + *   A pointer to a buffer that will be filled with the device name.
> > + *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
> > + * @return
> > + *  0 on success and devname is filled, negative on error
> > + */
> > +int do_eth_dev_detach(uint16_t port_id);
> > +
> 
> Why is this made part of an external API? You should have a separate,
> private header file for these.

OK, will add to ethdev_private.h in v2.

> 
> >   #ifdef __cplusplus
> >   }
> >   #endif
> > diff --git a/lib/librte_ethdev/rte_ethdev_mp.c
> b/lib/librte_ethdev/rte_ethdev_mp.c
> > new file mode 100644
> > index 000000000..8ede8151d
> > --- /dev/null
> > +++ b/lib/librte_ethdev/rte_ethdev_mp.c
> > @@ -0,0 +1,195 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation
> > + */
> > +
> > +#include "rte_ethdev_driver.h"
> > +#include "rte_ethdev_mp.h"
> > +
> > +static int detach_on_secondary(uint16_t port_id)
> 
> <snip>
> 
> > +	free(da.args);
> > +	return 0;
> > +}
> > +
> > +static int handle_secondary_request(const struct rte_mp_msg *msg, const
> void *peer)
> > +{
> > +	(void)msg;
> > +	(void)(peer);
> > +	return -ENOTSUP;
> 
> Please either mark arguments as __rte_unused, or use RTE_SET_USED(blah)
> macro. Same in other similar places.

OK.

> 
> > +}
> > +
> > +static int handle_primary_response(const struct rte_mp_msg *msg, const
> void *peer)
> > +{
> > +	(void)msg;
> > +	(void)(peer);
> > +	return -ENOTSUP;
> > +}
> > +
> > +static int handle_primary_request(const struct rte_mp_msg *msg, const
> void *peer)
> > +{
> > +	const struct eth_dev_mp_req *req =
> > +		(const struct eth_dev_mp_req *)msg->param;
> 
> <snip>
> 
> > +	case REQ_TYPE_DETACH:
> > +	case REQ_TYPE_ATTACH_ROLLBACK:
> > +		ret = detach_on_secondary(req->port_id);
> > +		break;
> > +	default:
> > +		ret = -EINVAL;
> > +	}
> > +
> > +	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
> 
> Here and in other places: rte_strlcpy?

OK

Thanks!
Qi
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH 06/22] ethdev: support attach or detach share device from secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, June 18, 2018 4:51 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH 06/22] ethdev: support attach or detach share device
> from secondary
> 
> 
> > +	else
> > +		return -1;
> > +	do {
> > +		ret = rte_mp_request_async(&mp_req, &ts, clb);
> > +	} while (ret != 0 && rte_errno == EEXIST);
> > +
> > +	if (ret)
> > +		ethdev_log(ERR, "couldn't send async request\n");
> > +	entry = find_request_by_id(req->id > +	(void)entry;
> 
> Why did you look up entry and then marked it as used without checking the
> return value? Leftover? Some code missing?

Some debug code forgot be removed :)

BTW, also accept all other comments

Thanks
Qi

Re: [dpdk-dev] [PATCH 04/22] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 19-Jun-18 4:22 AM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Monday, June 18, 2018 4:18 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH 04/22] ethdev: enable hotplug on multi-process
>>
>> On 07-Jun-18 1:38 PM, Qi Zhang wrote:
>>> The patch introduce the solution to handle different hotplug cases in
>>> multi-process situation, it include below scenario:
>>>
>>> 1. Attach a share device from primary
>>> 2. Detach a share device from primary
>>> 3. Attach a share device from secondary 4. Detach a share device from
>>> secondary 5. Attach a private device from secondary 6. Detach a
>>> private device from secondary 7. Detach a share device from secondary
>>> privately 8. Attach a share device from secondary privately
>>>
>>> In primary-secondary process model, we assume device is shared by
>> default.
>>> that means attach or detach a device on any process will broadcast to
>>> all other processes through mp channel then device information will be
>>> synchronized on all processes.
>>>
>>> Any failure during attaching process will cause inconsistent status
>>> between processes, so proper rollback action should be considered.
>>> Also it is not safe to detach a share device when other process still
>>> use it, so a handshake mechanism is introduced, it will be implemented
>>> in following separate patch.
>>>
>>> Scenario for Case 1, 2:
>>>
>>> attach device
>>> a) primary attach the new device if failed goto h).
>>> b) primary send attach sync request to all secondary.
>>> c) secondary receive request and attach device and send reply.
>>> d) primary check the reply if all success go to i).
>>> e) primary send attach rollback sync request to all secondary.
>>> f) secondary receive the request and detach device and send reply.
>>> g) primary receive the reply and detach device as rollback action.
>>> h) attach fail
>>> i) attach success
>>>
>>> detach device
>>> a) primary perform pre-detach check, if device is locked, goto i).
>>> b) primary send pre-detach sync request to all secondary.
>>> c) secondary perform pre-detach check and send reply.
>>> d) primary check the reply if any fail goto i).
>>> e) primary send detach sync request to all secondary
>>> f) secondary detach the device and send reply (assume no fail)
>>> g) primary detach the device.
>>> h) detach success
>>> i) detach failed
>>>
>>> Case 3, 4:
>>> This will be implemented in following patch.
>>
>> If these will be implemented in following patch, why spend half the commit
>> message talking about it? :)
> 
> Sorry, I didn't get your point about "see half commit to talk about it" :)
> This patch covered an overview, and also the implementation of case 1,2,5,6,7,8
> 
> For case 3, 4, just below 4 lines to describe it
> 
> 3. Attach a share device from secondary.
> 4. Detach a share device from secondary.
> Case 3, 4:
> This will be implemented in following patch.
> 
>> is commit doesn't implement secondary
>> process functionality at all, so the commit message should probably be
>> reworded to only include primary process logic, no?
> 
> OK, I will reword it to highlight the patch's scope as description at above.

Thanks!

<snip>

> 
> The return value of rte_eth_dev_request_to_primary only means communication fail,
> (message not able to send, or not get reply in time).
> but not the fail on attach/detach itself. (which comes from req->result)
> 

Ah, yes, my apologies, you're right! The log message is fine then.

<snip>

>>
>> Do we have to do a similar check for failsafe devices?
> 
> Just keep it same logic as before, it could be a separate patch to fix I guess.

Sure.

<snip>

>> Here and in other places: rte_strlcpy?
> 
> OK

Apologies, this should read strlcpy, not rte_strlcpy.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 05/22] ethdev: introduce device lock

[Zhang, Qi Z]

Hi Stephen:


> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Saturday, June 16, 2018 12:09 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 05/22] ethdev: introduce device lock
> 
> On Thu,  7 Jun 2018 20:38:32 +0800
> Qi Zhang <qi.z.zhang@intel.com> wrote:
> 
> > +/**
> > + * Lock an Ethernet Device directly or register a callback function
> > + * for condition check at runtime, this help application to prevent
> > + * a device be detached unexpectly.
> > + * NOTE: Lock a device mutliple times with same parmeter will increase
> > + * a ref_count, and coresponding unlock decrease the ref_count, the
> > + * device will be unlocked when ref_count reach 0.
> > + *
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param callback
> > + *   !NULL the callback function will be added into a pre-detach list,
> > + *         it will be invoked when a device is going to be detached. The
> > + *         return value will decide if continue detach the device or not.
> > + *   NULL  lock the device directly, basically this just regiter a empty
> > + *         callback function(dev_is_busy) that return -EBUSY, so we can
> > + *         handle the pre-detach check in unified way.
> > + * @param user_args
> > + *   parameter will be parsed to callback function, only valid when
> > + *   callback != NULL.
> > + * @return
> > + *   0 on success, negative on error.
> > + */
> > +int rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t
> callback,
> > +		     void *user_args);
> 
> I prefer API's that do one thing with one function.

Agree

> Why not
> 	rte_eth_dev_lock(uint16_t port_id);
> 	rte_eth_dev_ondetach(uint16_t port_id, rte_eth_dev_lock_callback_t
> callback,
>            		     void *user_args);

Rte_eth_dev_ondetach looks like a callback function, 
but this is the function to register some condition check.
How about rte_eth_dev_lock and rte_eth_dev_lock_with_cond?

Thanks
Qi
  

Re: [dpdk-dev] [PATCH 05/22] ethdev: introduce device lock

[Zhang, Qi Z]

Hi Anatoly:
	Sorry to miss this email and reply late.

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Friday, June 15, 2018 11:43 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH 05/22] ethdev: introduce device lock
> 
> On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > application lock or unlock on specific ethdev, a locked device can't
> > be detached, this help applicaiton to prevent unexpected device
> > detaching, especially in multi-process envrionment.
> >
> > Aslo the new API let application to register a callback function which
> > will be invoked before a device is going to be detached, the return
> > value of the function will decide if device will continue be detached
> > or not, this support application to do condition check at runtime.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +
> > +int
> > +rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> > +		 void *user_args)
> > +{
> > +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> > +
> > +	if (callback == NULL)
> > +		return register_lock_callback(port_id, dev_is_busy, NULL);
> > +	else
> > +		return register_lock_callback(port_id, callback, user_args);
> 
> As much as i don't like seeing negative errno values as return, the rest of
> ethdev library uses those, so this is OK :)
> 
> > +}
> > +
> > +int
> > +rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t callback,
> > +		   void *user_args)
> > +{
> > +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
> > +
> 
> <snip>
> 
> > + * Also, any callback function return !0 value will prevent device be
> > + * detached(ref. rte_eth_dev_lock and rte_eth_dev_unlock).
> > + *
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param user_args
> > + *   This is parameter "user_args" be saved when callback function is
> > + *   registered(rte_dev_eth_lock).
> > + *
> > + * @return
> > + *   0  device is allowed be detached.
> > + *   !0 device is not allowed be detached.
> 
> !0 can be negative or positive. Are we expecting positive return values from
> this API?

I have no strong opinion, but if you think below or other option is better, I can change

>=0  device is allowed be detached.
<0 device is not allowed be detached.

> 
> > + */
> > +typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void
> > +*user_args);
> > +
> > +/**
> > + * Lock an Ethernet Device directly or register a callback function
> > + * for condition check at runtime, this help application to prevent
> > + * a device be detached unexpectedly 
> > + * NOTE: Lock a device multiple times with same parmeter will
> > +increase
> > + * a ref_count, and corresponding unlock decrease the ref_count, the
> > + * device will be unlocked when ref_count reach 0.
> 
> Nitpick: "note" sections should be done with @note marker.
> 
> Also, i would mention that this is a per-process lock that does not affect other
> processes (assuming i understood the code correctly, of course...).

OK, I will add more comment to explain this.

> 
> > + *
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param callback
> > + *   !NULL the callback function will be added into a pre-detach list,
> > + *         it will be invoked when a device is going to be detached. The
> > + *         return value will decide if continue detach the device or not.
> > + *   NULL  lock the device directly, basically this just regiter a empty
> > + *         callback function(dev_is_busy) that return -EBUSY, so we can
> > + *         handle the pre-detach check in unified way.
> > + * @param user_args
> > + *   parameter will be parsed to callback function, only valid when
> > + *   callback != NULL.
> > + * @return
> > + *   0 on success, negative on error.
> > + */
> > +int rte_eth_dev_lock(uint16_t port_id, rte_eth_dev_lock_callback_t
> callback,
> > +		     void *user_args);
> 
> Nitpicks: DPDK style guide discourages using spaces as indentation (other parts
> of this patch, and other patches have this issue as well).

OK, will fix all.
> 
> > +
> > +/**
> > + * Reverse operation of rte_eth_dev_lock.
> > + *
> > + * @param port_id
> > + *   The port identifier of the Ethernet device.
> > + * @param callback
> > + *   NULL  decrease the ref_count of default callback function.
> > + *   !NULL decrease the ref_count of specific callback with matched
> > + *         user_args.
> > + * @param user_args
> > + *   parameter to match, only valid when callback != NULL.
> > + * @return
> > + *   0 on success, negative on error.
> > + */
> > +int rte_eth_dev_unlock(uint16_t port_id, rte_eth_dev_lock_callback_t
> callback,
> > +		       void *user_args);
> > +
> >   #ifdef __cplusplus
> >   }
> >   #endif
> > diff --git a/lib/librte_ethdev/rte_ethdev_lock.c
> > b/lib/librte_ethdev/rte_ethdev_lock.c
> 
> rte_ethdev_lock.* seem to be internal-only files. Perhaps you should name
> them without the rte_ prefix to indicate that they're not exported?
> 
> > new file mode 100644
> > index 000000000..688d1d70a
> > --- /dev/null
> > +++ b/lib/librte_ethdev/rte_ethdev_lock.c
> > @@ -0,0 +1,102 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2018 Intel Corporation  */ #include
> > +"rte_ethdev_lock.h"
> > +
> > +struct lock_entry {
> > +	TAILQ_ENTRY(lock_entry) next;
> > +	rte_eth_dev_lock_callback_t callback;
> > +	uint16_t port_id;
> 
> <snip>
> 
> > +register_lock_callback(uint16_t port_id,
> > +		       rte_eth_dev_lock_callback_t callback,
> > +		       void *user_args)
> > +{
> > +	struct lock_entry *le;
> > +
> > +	rte_spinlock_lock(&lock_entry_lock);
> > +
> > +	TAILQ_FOREACH(le, &lock_entry_list, next) {
> > +		if (le->port_id == port_id &&
> > +		    le->callback == callback &&
> > +		    le->user_args == user_args)
> > +			break;
> > +	}
> > +
> > +	if (!le) {
> > +		le = calloc(1, sizeof(struct lock_entry));
> > +		if (!le) {
> 
> Nitpick: generally, DPDK style guide prefers "if (value)" or "if (!value)" to only
> be reserved for boolean values, and use explicit comparison (e.g. "if (value ==
> NULL)" or "if (value == 0)") for all other cases.

OK, will fix
> 
> --
> Thanks,
> Anatoly

[dpdk-dev] [PATCH v2 00/22] enable hotplug on multi-process

[Qi Zhang]

v2:

- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device management
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./devmgm_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (22):
  eal: introduce one device scan
  bus/vdev: enable one device scan
  ethdev: add function to release port in local process
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/e1000: enable port detach on secondary process
  net/igb: enable port detach on secondary process
  net/fm10k: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/failsafe: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/devmgm_mp: add simple device management sample

 drivers/bus/vdev/vdev.c                   |  30 ++
 drivers/net/af_packet/rte_eth_af_packet.c |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c    |  11 +
 drivers/net/e1000/em_ethdev.c             |   9 +
 drivers/net/e1000/igb_ethdev.c            |   9 +
 drivers/net/failsafe/failsafe.c           |  16 +
 drivers/net/fm10k/fm10k_ethdev.c          |   9 +
 drivers/net/i40e/i40e_ethdev.c            |   2 +
 drivers/net/i40e/i40e_ethdev_vf.c         |   9 +
 drivers/net/ixgbe/ixgbe_ethdev.c          |  12 +
 drivers/net/kni/rte_eth_kni.c             |  11 +
 drivers/net/null/rte_eth_null.c           |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c    |  16 +
 drivers/net/pcap/rte_eth_pcap.c           |  15 +-
 drivers/net/softnic/rte_eth_softnic.c     |  19 +-
 drivers/net/tap/rte_eth_tap.c             |  17 +-
 drivers/net/vhost/rte_eth_vhost.c         |  11 +
 examples/devmgm_mp/Makefile               |  64 +++
 examples/devmgm_mp/commands.c             | 381 +++++++++++++++++
 examples/devmgm_mp/commands.h             |  10 +
 examples/devmgm_mp/main.c                 |  39 ++
 examples/devmgm_mp/meson.build            |  11 +
 lib/librte_eal/common/eal_common_dev.c    |  17 +-
 lib/librte_eal/common/eal_private.h       |   8 +
 lib/librte_eal/common/include/rte_bus.h   |  16 +
 lib/librte_eal/linuxapp/eal/eal.c         |   7 +
 lib/librte_ethdev/Makefile                |   2 +
 lib/librte_ethdev/ethdev_lock.c           | 139 +++++++
 lib/librte_ethdev/ethdev_lock.h           |  31 ++
 lib/librte_ethdev/ethdev_mp.c             | 662 ++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h             |  45 ++
 lib/librte_ethdev/ethdev_private.h        |  39 ++
 lib/librte_ethdev/meson.build             |   2 +
 lib/librte_ethdev/rte_ethdev.c            | 266 +++++++++++-
 lib/librte_ethdev/rte_ethdev.h            | 169 ++++++++
 lib/librte_ethdev/rte_ethdev_core.h       |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h     |  13 +
 37 files changed, 2118 insertions(+), 31 deletions(-)
 create mode 100644 examples/devmgm_mp/Makefile
 create mode 100644 examples/devmgm_mp/commands.c
 create mode 100644 examples/devmgm_mp/commands.h
 create mode 100644 examples/devmgm_mp/main.c
 create mode 100644 examples/devmgm_mp/meson.build
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v2 01/22] eal: introduce one device scan

[Qi Zhang]

When hot plug a new device, it is not necessary to scan everything
on the bus since the devname and devargs are already there. So new
rte_bus ops "scan_one" is introduced, bus driver can implement this
function to simplify the hotplug process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- fix spelling
- add missing comments.

 lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
 lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 61cb3b162..1ad033536 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	if (ret)
 		goto err_devarg;
 
-	ret = bus->scan();
-	if (ret)
-		goto err_devarg;
+	/**
+	 * if bus support to scan specific device by devargs,
+	 * we don't need to scan all devices on the bus.
+	 */
+	if (bus->scan_one) {
+		dev = bus->scan_one(da);
+	} else {
+		ret = bus->scan();
+		if (ret)
+			goto err_devarg;
+
+		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
+	}
 
-	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
 	if (dev == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
 			devname);
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..3269ef78b 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -84,6 +84,21 @@ enum rte_iova_mode {
 typedef int (*rte_bus_scan_t)(void);
 
 /**
+ * Bus specific scan for one specific device attached on the bus.
+ * For each bus object, the scan would be responsible for finding the specific
+ * device and adding it to its private device list, and the device object will
+ * be return also.
+ *
+ * @param devargs
+ *	Device arguments be used to identify the device.
+ *
+ * @return
+ *	!NULL for successful scan
+ *	NULL for unsuccessful scan
+ */
+typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *devargs);
+
+/**
  * Implementation specific probe function which is responsible for linking
  * devices on that bus with applicable drivers.
  *
@@ -204,6 +219,7 @@ struct rte_bus {
 	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
 	const char *name;            /**< Name of the bus */
 	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
+	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
 	rte_bus_probe_t probe;       /**< Probe devices on bus */
 	rte_bus_find_device_t find_device; /**< Find a device on the bus */
 	rte_bus_plug_t plug;         /**< Probe single device for drivers */
-- 
2.13.6

[dpdk-dev] [PATCH v2 02/22] bus/vdev: enable one device scan

[Qi Zhang]

The patch implemented the ops scan_one for vdev bus, it gives two benefits
1. Improve scan efficiency when a device is attached as hotplug, since no
need to populate a new device by iterating all devargs in devargs_list.
2. It also avoid sync IPC invoke (which happens in vdev->scan on secondary
process). The benefit is this removes the potential deadlock in the case
when secondary process receive a request from primary process to attach a
new device, since vdev->scan will be invoked on mp thread itself in that
case.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- improve commit log

 drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 6139dd551..cdbd77df0 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -467,6 +467,35 @@ vdev_scan(void)
 	return 0;
 }
 
+static struct rte_device *vdev_scan_one(struct rte_devargs *devargs)
+{
+	struct rte_vdev_device *dev = NULL;
+
+	dev = calloc(1, sizeof(*dev));
+	if (!dev) {
+		VDEV_LOG(ERR, "failed to allocate memory for new device");
+		return NULL;
+	}
+
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
+
+	if (find_vdev(devargs->name)) {
+		VDEV_LOG(ERR, "device %s already exist", devargs->name);
+		free(dev);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+		return NULL;
+	}
+
+	dev->device.devargs = devargs;
+	dev->device.numa_node = SOCKET_ID_ANY;
+	dev->device.name = devargs->name;
+	TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
+
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+
+	return &dev->device;
+}
+
 static int
 vdev_probe(void)
 {
@@ -531,6 +560,7 @@ vdev_unplug(struct rte_device *dev)
 
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
+	.scan_one = vdev_scan_one,
 	.probe = vdev_probe,
 	.find_device = vdev_find_device,
 	.plug = vdev_plug,
-- 
2.13.6

[dpdk-dev] [PATCH v2 03/22] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- rename rte_eth_release_port_local to rte_eth_release_port_private.

 lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..205b2ee33 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+
+	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
+
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
-	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
+		eth_dev->state = RTE_ETH_DEV_UNUSED;
+		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	}
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
 
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
-- 
2.13.6

[dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- add experimental tag for rte_eth_dev_attach_private and
  rte_ethdev_detach_private.
- move do_eth_dev_attach and do_eth_dev_detach to ethdev_private.h
- move rte_eth_dev_mp_init before rte_eal_mcfg_complete.
- fix meson.build.
- improve commit log.

 lib/librte_eal/common/eal_private.h |   8 ++
 lib/librte_eal/linuxapp/eal/eal.c   |   7 ++
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 198 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  44 ++++++++
 lib/librte_ethdev/ethdev_private.h  |  39 +++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 184 +++++++++++++++++++++++++++++----
 lib/librte_ethdev/rte_ethdev.h      |  45 ++++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 10 files changed, 515 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..92fa59bed 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,12 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Register mp channel callback functions of ethdev layer.
+ *
+ * @return
+ *  0 on success.
+ *  (<0) on failure.
+ */
+int rte_eth_dev_mp_init(void);
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..b7788c42d 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1024,6 +1024,13 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	/* Initialize mp channel for ethdev layer */
+	if (rte_eth_dev_mp_init()) {
+		rte_eal_init_alert("rte_eth_dev_mp_init() failed\n");
+		rte_errno = ENOEXEC;
+		return -1;
+	}
+
 #ifdef VFIO_PRESENT
 	/* Register mp action after probe() so that we got enough info */
 	if (rte_vfio_is_enabled("vfio") && vfio_mp_sync_setup() < 0)
diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..73dd36485
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,198 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include <rte_string_fns.h>
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, peer) < 0) {
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_eth_dev_mp_init(void)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return -1;
+		}
+	} else {
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_RESPONSE,
+					   handle_primary_response)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_RESPONSE);
+			return -1;
+		}
+		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request)) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..c3e55dfec
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+/* Register mp channel callback functions of ethdev layer.*/
+int rte_eth_dev_mp_init(void);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 205b2ee33..77f53a634 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -656,9 +658,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -703,14 +704,104 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	return ret;
+
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = rte_eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR, "Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	rte_eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -721,22 +812,81 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = rte_eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR, "Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR, "Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = rte_eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR, "Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v2 05/22] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help application to prevent unexpected
device detaching, especially in multi-process environment.

Also introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- rename rte_ethdev_lock.* to ethdev_lock.*
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix meson.build

 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 139 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 357 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6c13e87a9
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,139 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 73dd36485..10c03d25f 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,6 +5,7 @@
 #include <rte_string_fns.h>
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -104,7 +105,7 @@ static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 77f53a634..b98ce6766 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -723,6 +724,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return ret;
 
@@ -788,7 +790,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -829,6 +830,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -871,6 +876,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -884,6 +890,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4686,6 +4696,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v2 06/22] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process, the implementation
references malloc_mp.c.

device attach on secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.

device detach on secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

v2:
- fix coding style.
- improve comments.
- remove debug code.

 lib/librte_ethdev/ethdev_mp.c | 485 +++++++++++++++++++++++++++++++++++++++++-
 lib/librte_ethdev/ethdev_mp.h |   1 +
 2 files changed, 475 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 10c03d25f..f2ea53fd6 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -3,12 +3,101 @@
  */
 
 #include <rte_string_fns.h>
+#include <sys/time.h>
+
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ * secondary to primary request.
+ * start from function rte_eth_dev_request_to_primary.
+ *
+ * device attach:
+ * a) secondary send request to primary.
+ * b) primary attach the new device if failed goto i).
+ * c) primary forward attach request to all secondary.
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail response to secondary, goto k).
+ * j) send success response to secondary.
+ * k) end.
+
+ * device detach:
+ * a) secondary send request to primary.
+ * b) primary perform pre-detach check, if device is locked, got j).
+ * c) primary send pre-detach check request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success response to secondary, goto k).
+ * j) send fail response to secondary.
+ * k) end.
+ */
+
+enum req_state {
+	REQ_STATE_INACTIVE = 0,
+	REQ_STATE_ACTIVE,
+	REQ_STATE_COMPLETE
+};
+
+struct mp_request {
+	TAILQ_ENTRY(mp_request) next;
+	struct eth_dev_mp_req user_req; /**< contents of request */
+	pthread_cond_t cond; /**< variable we use to time out on this request */
+	enum req_state state; /**< indicate status of this request */
+};
+
+/*
+ * We could've used just a single request, but it may be possible for
+ * secondaries to timeout earlier than the primary, and send a new request while
+ * primary is still expecting replies to the old one. Therefore, each new
+ * request will get assigned a new ID, which is how we will distinguish between
+ * expected and unexpected messages.
+ */
+TAILQ_HEAD(mp_request_list, mp_request);
+static struct {
+	struct mp_request_list list;
+	pthread_mutex_t lock;
+} mp_request_list = {
+	.list = TAILQ_HEAD_INITIALIZER(mp_request_list.list),
+	.lock = PTHREAD_MUTEX_INITIALIZER
+};
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
+static struct mp_request *
+find_request_by_id(uint64_t id)
+{
+	struct mp_request *req;
+
+	TAILQ_FOREACH(req, &mp_request_list.list, next) {
+		if (req->user_req.id == id)
+			break;
+	}
+	return req;
+}
+
+static uint64_t
+get_unique_id(void)
+{
+	uint64_t id;
+
+	do {
+		id = rte_rand();
+	} while (find_request_by_id(id) != NULL);
+	return id;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req);
+
 static int detach_on_secondary(uint16_t port_id)
 {
 	struct rte_device *dev;
@@ -75,21 +164,330 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 	return 0;
 }
 
-static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+static int
+check_reply(const struct eth_dev_mp_req *req, const struct rte_mp_reply *reply)
+{
+	struct eth_dev_mp_req *resp;
+	int i;
+
+	if (reply->nb_received != reply->nb_sent)
+		return -EINVAL;
+
+	for (i = 0; i < reply->nb_received; i++) {
+		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
+
+		if (resp->t != req->t) {
+			ethdev_log(ERR, "Unexpected response to async request\n");
+			return -EINVAL;
+		}
+
+		if (resp->id != req->id) {
+			ethdev_log(ERR, "response to wrong async request\n");
+			return -EINVAL;
+		}
+
+		if (resp->result)
+			return resp->result;
+	}
+
+	return 0;
+}
+
+static int
+send_response_to_secondary(const struct eth_dev_mp_req *req, int result)
+{
+	struct rte_mp_msg resp_msg;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)resp_msg.param;
+	int ret = 0;
+
+	memset(&resp_msg, 0, sizeof(resp_msg));
+	resp_msg.len_param = sizeof(*resp);
+	strcpy(resp_msg.name, ETH_DEV_MP_ACTION_RESPONSE);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_sendmsg(&resp_msg);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static int
+handle_async_attach_response(const struct rte_mp_msg *request,
+			     const struct rte_mp_reply *reply)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	struct eth_dev_mp_req tmp_req;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_ATTACH_ROLLBACK;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, 0);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+handle_async_detach_response(const struct rte_mp_msg *request,
+			const struct rte_mp_reply *reply)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		send_response_to_secondary(req, ret);
+	} else {
+		do_eth_dev_detach(req->port_id);
+		send_response_to_secondary(req, 0);
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
 }
 
-static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+static int
+handle_async_pre_detach_response(const struct rte_mp_msg *request,
+				const struct rte_mp_reply *reply)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct eth_dev_mp_req tmp_req;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (!ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_DETACH;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, ret);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return 0;
 }
 
-static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+static int
+handle_async_rollback_response(const struct rte_mp_msg *request,
+				const struct rte_mp_reply *reply __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	/* we have nothing to do if rollback still fail, just detach */
+	do_eth_dev_detach(req->port_id);
+	/* send response to secondary with the reason of rollback */
+	send_response_to_secondary(req, req->result);
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req)
+{
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct rte_mp_msg mp_req;
+	rte_mp_async_reply_t clb;
+	int ret = 0;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	if (req->t == REQ_TYPE_ATTACH)
+		clb = handle_async_attach_response;
+	else if (req->t == REQ_TYPE_PRE_DETACH)
+		clb = handle_async_pre_detach_response;
+	else if (req->t == REQ_TYPE_DETACH)
+		clb = handle_async_detach_response;
+	else if (req->t == REQ_TYPE_ATTACH_ROLLBACK)
+		clb = handle_async_rollback_response;
+	else
+		return -1;
+	do {
+		ret = rte_mp_request_async(&mp_req, &ts, clb);
+	} while (ret != 0 && rte_errno == EEXIST);
+
+	if (ret)
+		ethdev_log(ERR, "couldn't send async request\n");
+
+	return ret;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	struct mp_request *entry;
+	uint16_t port_id;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		ethdev_log(ERR, "duplicate request id\n");
+		ret = -EEXIST;
+		goto finish;
+	}
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		ret = -ENOMEM;
+		goto finish;
+	}
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.port_id = port_id;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+		goto finish;
+	}
+
+	if (ret) {
+		ret = send_response_to_secondary(req, ret);
+		if (ret) {
+			ethdev_log(ERR, "failed to send response to secondary\n");
+			goto finish;
+		}
+	} else {
+		memcpy(&entry->user_req, req, sizeof(*req));
+		entry->state = REQ_STATE_ACTIVE;
+		TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+		entry = NULL;
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	if (entry)
+		free(entry);
+	return ret;
+}
+
+static int
+handle_primary_response(const struct rte_mp_msg *msg,
+			const void *peer __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct mp_request *entry;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		entry->user_req.result = req->result;
+		entry->user_req.port_id = req->port_id;
+		entry->state = REQ_STATE_COMPLETE;
+
+		pthread_cond_signal(&entry->cond);
+	}
+
+	pthread_mutex_unlock(&mp_request_list.lock);
+
+	return 0;
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 {
 	const struct eth_dev_mp_req *req =
 		(const struct eth_dev_mp_req *)msg->param;
@@ -129,8 +527,73 @@ static int handle_primary_request(const struct rte_mp_msg *msg, const void *peer
 
 int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg msg;
+	struct eth_dev_mp_req *msg_req = (struct eth_dev_mp_req *)msg.param;
+	struct mp_request *entry;
+	struct timespec ts;
+	struct timeval now;
+	int ret = 0;
+
+	memset(&msg, 0, sizeof(msg));
+	memset(&ts, 0, sizeof(ts));
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		return -ENOMEM;
+	}
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	ret = gettimeofday(&now, NULL);
+	if (ret) {
+		ethdev_log(ERR, "cannot get current time\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ts.tv_nsec = (now.tv_usec * 1000) % 1000000000;
+	ts.tv_sec = now.tv_sec + MP_TIMEOUT_S +
+			(now.tv_usec * 1000) / 1000000000;
+
+	pthread_cond_init(&entry->cond, NULL);
+
+	msg.len_param = sizeof(*req);
+	strcpy(msg.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	req->id = get_unique_id();
+
+	memcpy(msg_req, req, sizeof(*req));
+
+	ret = rte_mp_sendmsg(&msg);
+	if (ret) {
+		ethdev_log(ERR, "cannot send message to primary");
+		goto finish;
+	}
+
+	memcpy(&entry->user_req, req, sizeof(*req));
+
+	entry->state = REQ_STATE_ACTIVE;
+
+	TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+
+	do {
+		ret = pthread_cond_timedwait(&entry->cond,
+				&mp_request_list.lock, &ts);
+	} while (ret != 0 && ret != ETIMEDOUT);
+
+	if (entry->state != REQ_STATE_COMPLETE) {
+		RTE_LOG(ERR, EAL, "request time out\n");
+		ret = -ETIMEDOUT;
+	} else {
+		req->result = entry->user_req.result;
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	free(entry);
+	return ret;
 }
 
 /**
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index c3e55dfec..6d10dfdad 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -18,6 +18,7 @@ enum eth_dev_req_type {
 };
 
 struct eth_dev_mp_req {
+	uint64_t id;
 	enum eth_dev_req_type t;
 	char devargs[MAX_DEV_ARGS_LEN];
 	uint16_t port_id;
-- 
2.13.6

[dpdk-dev] [PATCH v2 07/22] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c    | 2 ++
 drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 804e44530..fc6f079d5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 08/22] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..f9d560835 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
@@ -1809,6 +1812,15 @@ static struct rte_pci_driver rte_ixgbe_pmd = {
 static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+
+	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_probe(pci_dev,
 		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
 }
-- 
2.13.6

[dpdk-dev] [PATCH v2 09/22] net/e1000: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/em_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index 7039dc100..e6b7ce63a 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -349,6 +349,15 @@ static int eth_em_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_em_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_em_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 10/22] net/igb: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/igb_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index edc7be319..db07a83e3 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -1089,6 +1089,15 @@ static int eth_igb_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_igb_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_igb_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 11/22] net/fm10k: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 3ff1b0e0f..f73301182 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -3264,6 +3264,15 @@ static int eth_fm10k_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_fm10k_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_fm10k_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 12/22] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v2 13/22] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v2 14/22] net/failsafe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c
index eafbb75df..c5e8651f6 100644
--- a/drivers/net/failsafe/failsafe.c
+++ b/drivers/net/failsafe/failsafe.c
@@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &failsafe_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 static int
 rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)
 {
+	struct rte_eth_dev *eth_dev;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
 	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
+
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario.
+		 */
+	}
+
 	return fs_rte_eth_free(name);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 15/22] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v2 16/22] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v2 17/22] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 18/22] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v2 19/22] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v2 20/22] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v2 21/22] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v2 22/22] examples/devmgm_mp: add simple device management sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./devmgm_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/devmgm_mp/Makefile    |  64 +++++++
 examples/devmgm_mp/commands.c  | 381 +++++++++++++++++++++++++++++++++++++++++
 examples/devmgm_mp/commands.h  |  10 ++
 examples/devmgm_mp/main.c      |  39 +++++
 examples/devmgm_mp/meson.build |  11 ++
 5 files changed, 505 insertions(+)
 create mode 100644 examples/devmgm_mp/Makefile
 create mode 100644 examples/devmgm_mp/commands.c
 create mode 100644 examples/devmgm_mp/commands.h
 create mode 100644 examples/devmgm_mp/main.c
 create mode 100644 examples/devmgm_mp/meson.build

diff --git a/examples/devmgm_mp/Makefile b/examples/devmgm_mp/Makefile
new file mode 100644
index 000000000..0ad3ec80e
--- /dev/null
+++ b/examples/devmgm_mp/Makefile
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# binary name
+APP = devmgm_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+# Build using pkg-config variables if possible
+$(shell pkg-config --exists libdpdk)
+ifeq ($(.SHELLSTATUS),0)
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+	ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+	ln -sf $(APP)-static build/$(APP)
+
+PC_FILE := $(shell pkg-config --path libdpdk)
+CFLAGS += -O3 $(shell pkg-config --cflags libdpdk)
+LDFLAGS_SHARED = $(shell pkg-config --libs libdpdk)
+LDFLAGS_STATIC = -Wl,-Bstatic $(shell pkg-config --static --libs libdpdk)
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+	@mkdir -p $@
+
+.PHONY: clean
+clean:
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rmdir --ignore-fail-on-non-empty build
+
+else
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = devmgm_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS_parse_obj_list.o := -D_GNU_SOURCE
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/devmgm_mp/commands.c b/examples/devmgm_mp/commands.c
new file mode 100644
index 000000000..11eb66730
--- /dev/null
+++ b/examples/devmgm_mp/commands.c
@@ -0,0 +1,381 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <errno.h>
+#include <netinet/in.h>
+#include <termios.h>
+#ifndef __linux__
+	#ifdef __FreeBSD__
+		#include <sys/socket.h>
+	#else
+		#include <net/socket.h>
+	#endif
+#endif
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		/* Secondary process's ethdev->state may not be
+		 * updated after detach on primary process,  but
+		 * ethdev->data should already be reset, so
+		 * use strlen(dev_name) == 0 to know the port is
+		 * not used.
+		 *
+		 * TODO: Secondary process should be informed when a
+		 * port is released on primary through mp channel.
+		 */
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/devmgm_mp/commands.h b/examples/devmgm_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/devmgm_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/devmgm_mp/main.c b/examples/devmgm_mp/main.c
new file mode 100644
index 000000000..9bf0c9962
--- /dev/null
+++ b/examples/devmgm_mp/main.c
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	return 0;
+}
diff --git a/examples/devmgm_mp/meson.build b/examples/devmgm_mp/meson.build
new file mode 100644
index 000000000..cec7be717
--- /dev/null
+++ b/examples/devmgm_mp/meson.build
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+sources = files(
+	'commands.c', 'main.c'
+)
-- 
2.13.6

Re: [dpdk-dev] [PATCH v2 22/22] examples/devmgm_mp: add simple device management sample

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> The sample code demonstrate device (ethdev only) management
> at multi-process envrionment. User can attach/detach a device
> on primary process and see it is synced on secondary process
> automatically, also user can lock a device to prevent it be
> detached or unlock it to go back to default behaviour.
> 
> How to start?
> ./devmgm_mp --proc-type=auto
> 
> Command Line Example:
> 
>> help
>> list
> 
> /* attach a af_packet vdev */
>> attach net_af_packet,iface=eth0
> 
> /* detach port 0 */
>> detach 0
> 
> /* attach a private af_packet vdev (secondary process only)*/
>> attachp net_af_packet,iface=eth0
> 
> /* detach a private device (secondary process only) */
>> detachp 0
> 
> /* lock port 0 */
>> lock 0
> 
> /* unlock port 0 */
>> unlock 0
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Hi Qi,

I believe you've missed my comments for v1 of this patch.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 01/22] eal: introduce one device scan

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> When hot plug a new device, it is not necessary to scan everything
> on the bus since the devname and devargs are already there. So new
> rte_bus ops "scan_one" is introduced, bus driver can implement this
> function to simplify the hotplug process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> 

<snip>

> +/**
>    * Implementation specific probe function which is responsible for linking
>    * devices on that bus with applicable drivers.
>    *
> @@ -204,6 +219,7 @@ struct rte_bus {
>   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
>   	const char *name;            /**< Name of the bus */
>   	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
> +	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
>   	rte_bus_probe_t probe;       /**< Probe devices on bus */
>   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
>   	rte_bus_plug_t plug;         /**< Probe single device for drivers */
> 

Does this break ABI for bus?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 03/22] ethdev: add function to release port in local process

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> Add driver API rte_eth_release_port_private to support the
> requirement that an ethdev only be released on secondary process,
> so only local state be set to unused , share data will not be
> reset so primary process can still use it.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

>   
>   /**
>    * @internal
> + * Release the specified ethdev port in local process, only set to ethdev
> + * state to unused, but not reset share data since it assume other process
> + * is still using it, typically it is called by secondary process.
> + *
> + * @param eth_dev
> + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> +

As far as i can tell, even though the function is marked as internal, it 
should still be exported in the .map file (see rte_eth_dev_allocate() 
for example).

Thomas and others, does this count as new API? Should this be marked as 
__rte_experimental? Presumably, we guarantee ABI stability for internal 
functions too, so my expectation would be yes.

> +/**
> + * @internal
>    * Release device queues and clear its configuration to force the user
>    * application to reconfigure it. It is for internal use only.
>    *
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 03/22] ethdev: add function to release port in local process

[Thomas Monjalon]

21/06/2018 10:06, Burakov, Anatoly:
> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> > Add driver API rte_eth_release_port_private to support the
> > requirement that an ethdev only be released on secondary process,
> > so only local state be set to unused , share data will not be
> > reset so primary process can still use it.
> > 
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> >   
> >   /**
> >    * @internal
> > + * Release the specified ethdev port in local process, only set to ethdev
> > + * state to unused, but not reset share data since it assume other process
> > + * is still using it, typically it is called by secondary process.
> > + *
> > + * @param eth_dev
> > + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > + * @return
> > + *   - 0 on success, negative on error
> > + */
> > +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> > +
> 
> As far as i can tell, even though the function is marked as internal, it 
> should still be exported in the .map file (see rte_eth_dev_allocate() 
> for example).
> 
> Thomas and others, does this count as new API? Should this be marked as 
> __rte_experimental? Presumably, we guarantee ABI stability for internal 
> functions too, so my expectation would be yes.

You know the A in ABI stands for Application :)
If it is not called by application, it has no impact on ABI.

However, I am not sure about having this function at all.
Who is calling it?

Re: [dpdk-dev] [PATCH v2 03/22] ethdev: add function to release port in local process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 21, 2018 4:06 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v2 03/22] ethdev: add function to release port in local
> process
> 
> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> > Add driver API rte_eth_release_port_private to support the requirement
> > that an ethdev only be released on secondary process, so only local
> > state be set to unused , share data will not be reset so primary
> > process can still use it.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> >
> >   /**
> >    * @internal
> > + * Release the specified ethdev port in local process, only set to
> > +ethdev
> > + * state to unused, but not reset share data since it assume other
> > +process
> > + * is still using it, typically it is called by secondary process.
> > + *
> > + * @param eth_dev
> > + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > + * @return
> > + *   - 0 on success, negative on error
> > + */
> > +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> > +
> 
> As far as i can tell, even though the function is marked as internal, it should still
> be exported in the .map file (see rte_eth_dev_allocate() for example).
> 
> Thomas and others, does this count as new API? Should this be marked as
> __rte_experimental? Presumably, we guarantee ABI stability for internal
> functions too, so my expectation would be yes.

Sorry, I not intent to mark this as experimental, I must forgot to remove this
It should rte_eth_dev_attach/detach_private and rte_eth_dev_lock/unlock .

I guess internal API is not necessary to have this.
I will remove it in v3

Thanks
Qi

> 
> > +/**
> > + * @internal
> >    * Release device queues and clear its configuration to force the user
> >    * application to reconfigure it. It is for internal use only.
> >    *
> >
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
> 
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs changes:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +	memset(&da, 0, sizeof(da));
> +
> +	if (rte_devargs_parse(&da, "%s", devargs)) {
> +		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
> +	if (ret) {
> +		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
> +			   da.bus->name, da.name);
> +		free(da.args);
> +		return ret;
> +	}
> +
> +	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
> +		ethdev_log(ERR, "failed to attach to port %d, this is a pmd issue\n",
> +			   port_id);
> +		return -ENODEV;

^^^ Leaking da.args here?

> +	}
> +	free(da.args);
> +	return 0;
> +}
> +
> +static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	RTE_SET_USED(msg);
> +	RTE_SET_USED(peer);
> +	return -ENOTSUP;
> +}
> +
> +static int handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
> +{

<snip>

> +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> +	if (ret) {
> +		ethdev_log(ERR, "rte_mp_request_sync failed\n");
> +		return ret;
> +	}
> +
> +	req->result = 0;
> +	for (i = 0; i < mp_reply.nb_received; i++) {
> +		struct eth_dev_mp_req *resp =
> +			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
> +		if (resp->result) {
> +			req->result = resp->result;
> +			break;
> +		}
> +	}

Do we care if nb_sent != nb_received?

> +
> +	return 0;
> +}
> +
> +int rte_eth_dev_mp_init(void)
> +{
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,

<snip>

> +/**
> + * this is a synchronous wrapper for secondary process send
> + * request to primary process, this is invoked when an attach
> + * or detach request issued from primary.
> + */
> +int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> +
> +/**
> + * this is a synchronous wrapper for primary process send
> + * request to secondary process, this is invoked when an attach
> + * or detach request issued from secondary process.
> + */
> +int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req);

Nitpicking,  but the two above functions aren't used outside ethdev 
library. You can probably drop the rte_ prefix.

> +
> +/* Register mp channel callback functions of ethdev layer.*/
> +int rte_eth_dev_mp_init(void);

I don't quite understand what you're doing here. (Or rather, i 
understand the intention, but i don't understand the implementation :) )

This function is meant to be called from EAL at startup. First of all, 
why is it declared twice (once in eal_private, once in ethdev_private)?

Second of all, ethdev is a library, but this function is called from 
EAL. Which means it cannot be in a private header (nor should it be 
declared in EAL), and you cannot even call it from EAL because that 
would introduce a circular dependency between EAL and ethdev.

So, this needs to be redone the other way around - have ethdev register 
itself with EAL, and get called at some point, in a generic way (e.g. 
see how bus probe works for example). I don't know what this would look 
like - maybe some kind of generic multiprocess init?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 05/22] ethdev: introduce device lock

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help application to prevent unexpected
> device detaching, especially in multi-process environment.
> 
> Also introduce the new API rte_eth_dev_lock_with_callback and
> rte_eth_dev_unlock_with callback to let application to register
> a callback function which will be invoked before a device is going
> to be detached, the return value of the function will decide if
> device will continue be detached or not, this support application
> to do condition check at runtime.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +
> +static int clean_lock_callback_one(uint16_t port_id)
> +{
> +	struct lock_entry *le;
> +	int ret = 0;
> +
> +	rte_spinlock_lock(&lock_entry_lock);
> +
> +	TAILQ_FOREACH(le, &lock_entry_list, next) {
> +		if (le->port_id == port_id)
> +			break;
> +	}
> +
> +	if (le != NULL) {
> +		le->ref_count--;
> +		if (le->ref_count == 0) {
> +			TAILQ_REMOVE(&lock_entry_list, le, next);
> +			free(le);
> +		}
> +	} else {
> +		ret = -ENOENT;
> +	}
> +
> +	rte_spinlock_unlock(&lock_entry_lock);
> +	return ret;
> +
> +}
> +
> +void clean_lock_callback(uint16_t port_id)
> +{
> +	int ret;
> +
> +	for (;;) {
> +		ret = clean_lock_callback_one(port_id);
> +		if (ret == -ENOENT)
> +			break;
> +	}
> +}

Why not lock/unlock the list in clean_lock_callback() and proceed to 
cleaning callbacks one by one, instead of locking-and-unlocking the list 
over and over again?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 06/22] ethdev: support attach or detach share device from secondary

[Burakov, Anatoly]

On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a share device
> attach/detach request be issued from secondary process, the implementation
> references malloc_mp.c.
> 
> device attach on secondary:
> a) secondary send async request to primary and wait on a condition
>     which will be released by matched response from primary.
> b) primary receive the request and attach the new device if failed
>     goto i).
> c) primary forward attach request to all secondary as async request
>     (because this in mp thread context, use sync request will deadlock)
> d) secondary receive request and attach device and send reply.
> e) primary check the reply if all success go to j).
> f) primary send attach rollback async request to all secondary.
> g) secondary receive the request and detach device and send reply.
> h) primary receive the reply and detach device as rollback action.
> i) send fail response to secondary, goto k).
> j) send success response to secondary.
> k) secondary process receive response and return.
> 
> device detach on secondary:
> a) secondary send async request to primary and wait on a condition
>     which will be released by matched response from primary.
> b) primary receive the request and  perform pre-detach check, if device
>     is locked, goto j).
> c) primary send pre-detach async request to all secondary.
> d) secondary perform pre-detach check and send reply.
> e) primary check the reply if any fail goto j).
> f) primary send detach async request to all secondary
> g) secondary detach the device and send reply
> h) primary detach the device.
> i) send success response to secondary, goto k).
> j) send fail response to secondary.
> k) secondary process receive response and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> 

<snip>

> -static int handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
> +static int
> +check_reply(const struct eth_dev_mp_req *req, const struct rte_mp_reply *reply)
> +{
> +	struct eth_dev_mp_req *resp;
> +	int i;
> +
> +	if (reply->nb_received != reply->nb_sent)
> +		return -EINVAL;
> +
> +	for (i = 0; i < reply->nb_received; i++) {
> +		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
> +
> +		if (resp->t != req->t) {
> +			ethdev_log(ERR, "Unexpected response to async request\n");
> +			return -EINVAL;
> +		}
> +
> +		if (resp->id != req->id) {
> +			ethdev_log(ERR, "response to wrong async request\n");
> +			return -EINVAL;
> +		}
> +
> +		if (resp->result)
> +			return resp->result;
> +	}
> +
> +	return 0;
> +}

As far as i understand, return values from this will propagate all the 
way up to user return value. How would a user differentiate between 
-EINVAL returned from invalid parameters, and -EINVAL from failed reply? 
I think this error code should be different (don't know which one though 
:) ).

(as a side note, you keep returning -EINVAL all over the place, even 
when problem is not in user's arguments - you should probably fix those 
too. for example, if request ID not found, return code should probably 
be something like -ENOENT)


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 21, 2018 4:37 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v2 04/22] ethdev: enable hotplug on multi-process
> 
> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> > We are going to introduce the solution to handle different hotplug
> > cases in multi-process situation, it include below scenario:
> >
> > 1. Attach a share device from primary
> > 2. Detach a share device from primary
> > 3. Attach a share device from secondary 4. Detach a share device from
> > secondary 5. Attach a private device from secondary 6. Detach a
> > private device from secondary 7. Detach a share device from secondary
> > privately 8. Attach a share device from secondary privately
> >
> > In primary-secondary process model, we assume device is shared by default.
> > that means attach or detach a device on any process will broadcast to
> > all other processes through mp channel then device information will be
> > synchronized on all processes.
> >
> > Any failure during attaching process will cause inconsistent status
> > between processes, so proper rollback action should be considered.
> > Also it is not safe to detach a share device when other process still
> > use it, so a handshake mechanism is introduced.
> >
> > This patch covers the implementation of case 1,2,5,6,7,8.
> > Case 3,4 will be implemented on separate patch as well as handshake
> > mechanism.
> >
> > Scenario for Case 1, 2:
> >
> > attach device
> > a) primary attach the new device if failed goto h).
> > b) primary send attach sync request to all secondary.
> > c) secondary receive request and attach device and send reply.
> > d) primary check the reply if all success go to i).
> > e) primary send attach rollback sync request to all secondary.
> > f) secondary receive the request and detach device and send reply.
> > g) primary receive the reply and detach device as rollback action.
> > h) attach fail
> > i) attach success
> >
> > detach device
> > a) primary perform pre-detach check, if device is locked, goto i).
> > b) primary send pre-detach sync request to all secondary.
> > c) secondary perform pre-detach check and send reply.
> > d) primary check the reply if any fail goto i).
> > e) primary send detach sync request to all secondary
> > f) secondary detach the device and send reply (assume no fail)
> > g) primary detach the device.
> > h) detach success
> > i) detach failed
> >
> > Case 5, 6:
> > Secondary process can attach private device which only visible to
> > itself, in this case no IPC is involved, primary process is not
> > allowed to have private device so far.
> >
> > Case 7, 8:
> > Secondary process can also temporally to detach a share device "privately"
> > then attach it back later, this action also not impact other processes.
> >
> > APIs changes:
> >
> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> > share device attach/detach in primary-secondary process model, it will
> > be called in case 1,2,3,4.
> >
> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> > introduced to cover case 5,6,7,8, this API can only be invoked in
> > secondary process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +	memset(&da, 0, sizeof(da));
> > +
> > +	if (rte_devargs_parse(&da, "%s", devargs)) {
> > +		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
> > +		return -EINVAL;
> > +	}
> > +
> > +	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
> > +	if (ret) {
> > +		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
> > +			   da.bus->name, da.name);
> > +		free(da.args);
> > +		return ret;
> > +	}
> > +
> > +	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
> > +		ethdev_log(ERR, "failed to attach to port %d, this is a pmd issue\n",
> > +			   port_id);
> > +		return -ENODEV;
> 
> ^^^ Leaking da.args here?
> 
> > +	}
> > +	free(da.args);
> > +	return 0;
> > +}
> > +
> > +static int handle_secondary_request(const struct rte_mp_msg *msg,
> > +const void *peer) {
> > +	RTE_SET_USED(msg);
> > +	RTE_SET_USED(peer);
> > +	return -ENOTSUP;
> > +}
> > +
> > +static int handle_primary_response(const struct rte_mp_msg *msg,
> > +const void *peer) {
> 
> <snip>
> 
> > +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> > +	if (ret) {
> > +		ethdev_log(ERR, "rte_mp_request_sync failed\n");
> > +		return ret;
> > +	}
> > +
> > +	req->result = 0;
> > +	for (i = 0; i < mp_reply.nb_received; i++) {
> > +		struct eth_dev_mp_req *resp =
> > +			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
> > +		if (resp->result) {
> > +			req->result = resp->result;
> > +			break;
> > +		}
> > +	}
> 
> Do we care if nb_sent != nb_received?
> 
> > +
> > +	return 0;
> > +}
> > +
> > +int rte_eth_dev_mp_init(void)
> > +{
> > +
> > +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> > +		if (rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
> 
> <snip>
> 
> > +/**
> > + * this is a synchronous wrapper for secondary process send
> > + * request to primary process, this is invoked when an attach
> > + * or detach request issued from primary.
> > + */
> > +int rte_eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> > +
> > +/**
> > + * this is a synchronous wrapper for primary process send
> > + * request to secondary process, this is invoked when an attach
> > + * or detach request issued from secondary process.
> > + */
> > +int rte_eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
> 
> Nitpicking,  but the two above functions aren't used outside ethdev library.
> You can probably drop the rte_ prefix.
> 
> > +
> > +/* Register mp channel callback functions of ethdev layer.*/ int
> > +rte_eth_dev_mp_init(void);
> 
> I don't quite understand what you're doing here. (Or rather, i understand the
> intention, but i don't understand the implementation :) )
> 
> This function is meant to be called from EAL at startup. First of all, why is it
> declared twice (once in eal_private, once in ethdev_private)?

Ah, I forgot this, this is something in mess.

> 
> Second of all, ethdev is a library, but this function is called from EAL. Which
> means it cannot be in a private header (nor should it be declared in EAL), and
> you cannot even call it from EAL because that would introduce a circular
> dependency between EAL and ethdev.
> 
> So, this needs to be redone the other way around - have ethdev register itself
> with EAL, and get called at some point, in a generic way (e.g.
> see how bus probe works for example). I don't know what this would look like
> - maybe some kind of generic multiprocess init?

Yes, properly like this. I will re-work it.

Thanks
Qi

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v2 05/22] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 21, 2018 4:51 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v2 05/22] ethdev: introduce device lock
> 
> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > application lock or unlock on specific ethdev, a locked device can't
> > be detached, this help application to prevent unexpected device
> > detaching, especially in multi-process environment.
> >
> > Also introduce the new API rte_eth_dev_lock_with_callback and
> > rte_eth_dev_unlock_with callback to let application to register a
> > callback function which will be invoked before a device is going to be
> > detached, the return value of the function will decide if device will
> > continue be detached or not, this support application to do condition
> > check at runtime.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +
> > +static int clean_lock_callback_one(uint16_t port_id) {
> > +	struct lock_entry *le;
> > +	int ret = 0;
> > +
> > +	rte_spinlock_lock(&lock_entry_lock);
> > +
> > +	TAILQ_FOREACH(le, &lock_entry_list, next) {
> > +		if (le->port_id == port_id)
> > +			break;
> > +	}
> > +
> > +	if (le != NULL) {
> > +		le->ref_count--;
> > +		if (le->ref_count == 0) {
> > +			TAILQ_REMOVE(&lock_entry_list, le, next);
> > +			free(le);
> > +		}
> > +	} else {
> > +		ret = -ENOENT;
> > +	}
> > +
> > +	rte_spinlock_unlock(&lock_entry_lock);
> > +	return ret;
> > +
> > +}
> > +
> > +void clean_lock_callback(uint16_t port_id) {
> > +	int ret;
> > +
> > +	for (;;) {
> > +		ret = clean_lock_callback_one(port_id);
> > +		if (ret == -ENOENT)
> > +			break;
> > +	}
> > +}
> 
> Why not lock/unlock the list in clean_lock_callback() and proceed to cleaning
> callbacks one by one, instead of locking-and-unlocking the list over and over
> again?

Definitely!

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v2 20/22] net/tap: enable port detach on secondary process

[Wiles, Keith]

> On Jun 20, 2018, at 9:00 PM, Qi Zhang <qi.z.zhang@intel.com> wrote:
> 
> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_private, we can support this
> with minor change.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
> 1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
> index df396bfde..bb5f20b01 100644
> --- a/drivers/net/tap/rte_eth_tap.c
> +++ b/drivers/net/tap/rte_eth_tap.c
> @@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
> 		}
> 		/* TODO: request info from primary to set up Rx and Tx */
> 		eth_dev->dev_ops = &ops;
> +		eth_dev->device = &dev->device;
> 		rte_eth_dev_probing_finish(eth_dev);
> 		return 0;
> 	}
> @@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
> {
> 	struct rte_eth_dev *eth_dev = NULL;
> 	struct pmd_internals *internals;
> +	const char *name;
> 	int i;
> 
> +	name = rte_vdev_device_name(dev);
> 	/* find the ethdev entry */
> -	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
> +	eth_dev = rte_eth_dev_allocated(name);
> 	if (!eth_dev)
> -		return 0;
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/* detach device on local pprocess only */
> +		if (strlen(rte_vdev_device_args(dev)) == 0)
> +			return rte_eth_dev_release_port_private(eth_dev);
> +		/**
> +		 * else this is a private device for current process
> +		 * so continue with normal detach scenario
> +		 */
> +	}

Acked-by: Keith Wiles<keith.wiles@intel.com> for this patch of the tap PMD.

> 
> 	internals = eth_dev->data->dev_private;
> 
> -- 
> 2.13.6
> 

Regards,
Keith

Re: [dpdk-dev] [PATCH v2 06/22] ethdev: support attach or detach share device from secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 21, 2018 5:06 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v2 06/22] ethdev: support attach or detach share device
> from secondary
> 
> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
> > This patch cover the multi-process hotplug case when a share device
> > attach/detach request be issued from secondary process, the
> > implementation references malloc_mp.c.
> >
> > device attach on secondary:
> > a) secondary send async request to primary and wait on a condition
> >     which will be released by matched response from primary.
> > b) primary receive the request and attach the new device if failed
> >     goto i).
> > c) primary forward attach request to all secondary as async request
> >     (because this in mp thread context, use sync request will
> > deadlock)
> > d) secondary receive request and attach device and send reply.
> > e) primary check the reply if all success go to j).
> > f) primary send attach rollback async request to all secondary.
> > g) secondary receive the request and detach device and send reply.
> > h) primary receive the reply and detach device as rollback action.
> > i) send fail response to secondary, goto k).
> > j) send success response to secondary.
> > k) secondary process receive response and return.
> >
> > device detach on secondary:
> > a) secondary send async request to primary and wait on a condition
> >     which will be released by matched response from primary.
> > b) primary receive the request and  perform pre-detach check, if device
> >     is locked, goto j).
> > c) primary send pre-detach async request to all secondary.
> > d) secondary perform pre-detach check and send reply.
> > e) primary check the reply if any fail goto j).
> > f) primary send detach async request to all secondary
> > g) secondary detach the device and send reply
> > h) primary detach the device.
> > i) send success response to secondary, goto k).
> > j) send fail response to secondary.
> > k) secondary process receive response and return.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >
> 
> <snip>
> 
> > -static int handle_secondary_request(const struct rte_mp_msg *msg,
> > const void *peer)
> > +static int
> > +check_reply(const struct eth_dev_mp_req *req, const struct
> > +rte_mp_reply *reply) {
> > +	struct eth_dev_mp_req *resp;
> > +	int i;
> > +
> > +	if (reply->nb_received != reply->nb_sent)
> > +		return -EINVAL;
> > +
> > +	for (i = 0; i < reply->nb_received; i++) {
> > +		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
> > +
> > +		if (resp->t != req->t) {
> > +			ethdev_log(ERR, "Unexpected response to async request\n");
> > +			return -EINVAL;
> > +		}
> > +
> > +		if (resp->id != req->id) {
> > +			ethdev_log(ERR, "response to wrong async request\n");
> > +			return -EINVAL;
> > +		}
> > +
> > +		if (resp->result)
> > +			return resp->result;
> > +	}
> > +
> > +	return 0;
> > +}
> 
> As far as i understand, return values from this will propagate all the way up to
> user return value. 
Yes
>How would a user differentiate between -EINVAL returned
> from invalid parameters, and -EINVAL from failed reply?

My understanding is if 
 (resp->t != req->t) or (resp->id != req->id) is not expected to happen at any condition.
there should be a bug if it does happen.
So the return value is not necessary to be sensitive.
Am I right?

> I think this error code should be different (don't know which one though
> :) ).
> 
> (as a side note, you keep returning -EINVAL all over the place, even when
> problem is not in user's arguments - you should probably fix those too. for
> example, if request ID not found, return code should probably be something
> like -ENOENT)

Yes, -ENOENT is better than -EINVAL for id mismatch?

> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v2 06/22] ethdev: support attach or detach share device from secondary

[Burakov, Anatoly]

On 21-Jun-18 1:50 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Thursday, June 21, 2018 5:06 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH v2 06/22] ethdev: support attach or detach share device
>> from secondary
>>
>> On 21-Jun-18 3:00 AM, Qi Zhang wrote:
>>> This patch cover the multi-process hotplug case when a share device
>>> attach/detach request be issued from secondary process, the
>>> implementation references malloc_mp.c.
>>>
>>> device attach on secondary:
>>> a) secondary send async request to primary and wait on a condition
>>>      which will be released by matched response from primary.
>>> b) primary receive the request and attach the new device if failed
>>>      goto i).
>>> c) primary forward attach request to all secondary as async request
>>>      (because this in mp thread context, use sync request will
>>> deadlock)
>>> d) secondary receive request and attach device and send reply.
>>> e) primary check the reply if all success go to j).
>>> f) primary send attach rollback async request to all secondary.
>>> g) secondary receive the request and detach device and send reply.
>>> h) primary receive the reply and detach device as rollback action.
>>> i) send fail response to secondary, goto k).
>>> j) send success response to secondary.
>>> k) secondary process receive response and return.
>>>
>>> device detach on secondary:
>>> a) secondary send async request to primary and wait on a condition
>>>      which will be released by matched response from primary.
>>> b) primary receive the request and  perform pre-detach check, if device
>>>      is locked, goto j).
>>> c) primary send pre-detach async request to all secondary.
>>> d) secondary perform pre-detach check and send reply.
>>> e) primary check the reply if any fail goto j).
>>> f) primary send detach async request to all secondary
>>> g) secondary detach the device and send reply
>>> h) primary detach the device.
>>> i) send success response to secondary, goto k).
>>> j) send fail response to secondary.
>>> k) secondary process receive response and return.
>>>
>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>> ---
>>>
>>
>> <snip>
>>
>>> -static int handle_secondary_request(const struct rte_mp_msg *msg,
>>> const void *peer)
>>> +static int
>>> +check_reply(const struct eth_dev_mp_req *req, const struct
>>> +rte_mp_reply *reply) {
>>> +	struct eth_dev_mp_req *resp;
>>> +	int i;
>>> +
>>> +	if (reply->nb_received != reply->nb_sent)
>>> +		return -EINVAL;
>>> +
>>> +	for (i = 0; i < reply->nb_received; i++) {
>>> +		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
>>> +
>>> +		if (resp->t != req->t) {
>>> +			ethdev_log(ERR, "Unexpected response to async request\n");
>>> +			return -EINVAL;
>>> +		}
>>> +
>>> +		if (resp->id != req->id) {
>>> +			ethdev_log(ERR, "response to wrong async request\n");
>>> +			return -EINVAL;
>>> +		}
>>> +
>>> +		if (resp->result)
>>> +			return resp->result;
>>> +	}
>>> +
>>> +	return 0;
>>> +}
>>
>> As far as i understand, return values from this will propagate all the way up to
>> user return value.
> Yes
>> How would a user differentiate between -EINVAL returned
>> from invalid parameters, and -EINVAL from failed reply?
> 
> My understanding is if
>   (resp->t != req->t) or (resp->id != req->id) is not expected to happen at any condition.
> there should be a bug if it does happen.
> So the return value is not necessary to be sensitive.
> Am I right?

You're right, it won't happen under normal conditions. However, on the 
off-chance that it does, the error return should still be meaningful. 
Under normal conditions, malloc() doesn't fail either :)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH 22/22] examples/devmgm_mp: add simple device management sample

[Zhang, Qi Z]

Hi Anatoly:

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, June 18, 2018 6:36 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH 22/22] examples/devmgm_mp: add simple device
> management sample
> 
> On 07-Jun-18 1:38 PM, Qi Zhang wrote:
> > The sample code demonstrate device (ethdev only) management at
> > multi-process envrionment. User can attach/detach a device on primary
> > process and see it is synced on secondary process automatically, also
> > user can lock a device to prevent it be detached or unlock it to go
> > back to default behaviour.
> >
> > How to start?
> > ./devmgm_mp --proc-type=auto
> >
> > Command Line Example:
> >
> >> help
> >> list
> >
> > /* attach a af_packet vdev */
> >> attach net_af_packet,iface=eth0
> >
> > /* detach port 0 */
> >> detach 0
> >
> > /* attach a private af_packet vdev (secondary process only)*/
> >> attachp net_af_packet,iface=eth0
> >
> > /* detach a private device (secondary process only) */
> >> detachp 0
> >
> > /* lock port 0 */
> >> lock 0
> >
> > /* unlock port 0 */
> >> unlock 0
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> I think the "devmgm_mp" is not a descriptive enough name. What this
> example demonstrates, is device hotplug. So how about naming the example
> app "hotplug"? (or "mp_hotplug" to indicate that it specifically sets out to
> demonstrate multiprocess hotplug)


Ok, I saw all the multi-process samples are in examples/multi_process, so I think this the right place to add
it could be "hotplug_mp" to follow other samples naming rule.
> 
> >   examples/devmgm_mp/Makefile    |  64 +++++++
> >   examples/devmgm_mp/commands.c  | 383
> +++++++++++++++++++++++++++++++++++++++++
> >   examples/devmgm_mp/commands.h  |  10 ++
> >   examples/devmgm_mp/main.c      |  41 +++++
> >   examples/devmgm_mp/meson.build |  11 ++
> >   5 files changed, 509 insertions(+)
> >   create mode 100644 examples/devmgm_mp/Makefile
> >   create mode 100644 examples/devmgm_mp/commands.c
> >   create mode 100644 examples/devmgm_mp/commands.h
> >   create mode 100644 examples/devmgm_mp/main.c
> >   create mode 100644 examples/devmgm_mp/meson.build
> >
> 
> <snip>
> 
> > +#include <stdio.h>
> > +#include <stdint.h>
> > +#include <string.h>
> > +#include <stdlib.h>
> > +#include <stdarg.h>
> > +#include <errno.h>
> > +#include <netinet/in.h>
> > +#include <termios.h>
> > +#ifndef __linux__
> > +	#ifdef __FreeBSD__
> > +		#include <sys/socket.h>
> > +	#else
> > +		#include <net/socket.h>
> > +	#endif
> > +#endif
> 
> This seems like a weird define. Care to elaborate why are we checking for
> __linux__ not being defined?

OK, this is copy from exist sample code :), I will clean up the header file in v3.

> 
> If you're trying to differentiate between Linux and FreeBSD, there's a readly
> RTE_EXEC_ENV_* config options, e.g.
> 
> #ifdef RTE_EXEC_ENV_LINUXAPP
> // linux defines
> #endif
> #ifdef RTE_EXEC_ENV_BSDAPP
> // bsd defines
> #endif
> 
> or something to that effect.
> 
> > +
> > +#include <cmdline_rdline.h>
> > +#include <cmdline_parse.h>
> > +#include <cmdline_parse_ipaddr.h>
> > +#include <cmdline_parse_num.h>
> > +#include <cmdline_parse_string.h>
> > +#include <cmdline.h>
> > +#include <rte_ethdev.h>
> 
> Generally (and as per DPDK coding guidelines), we prefer defines ordered as
> follows:
> 
> 1) system defines enclosed in brackets
> 2) DPDK defines (rte_blah) enclosed in brackets
> 3) private/application-specific defines enclosed in quotes.
> 
> All three groups should be separated by newline.
> 
> So, these defines should've read as:
> 
> #include <stdblah.h>
> #include <sys/blah.h>
> 
> #include <rte_blah.h>
> #include <rte_foo.h>
> 
> #include "cmdline_blah.h"
> #include "cmdline_foo.h"

Got it, thanks

> 
> > +
> > +/**********************************************************/
> > +
> > +struct cmd_help_result {
> > +	cmdline_fixed_string_t help;
> > +};
> > +
> > +static void cmd_help_parsed(__attribute__((unused)) void
> > +*parsed_result,
> 
> <snip>
> 
> > +{
> > +	uint16_t port_id;
> > +	char dev_name[RTE_DEV_NAME_MAX_LEN];
> > +
> > +	cmdline_printf(cl, "list all etherdev\n");
> > +
> > +	RTE_ETH_FOREACH_DEV(port_id) {
> > +		rte_eth_dev_get_name_by_port(port_id, dev_name);
> > +		/* Secondary process's ethdev->state may not be
> > +		 * updated after detach on primary process,  but
> > +		 * ethdev->data should already be reset, so
> > +		 * use strlen(dev_name) == 0 to know the port is
> > +		 * not used.
> > +		 *
> > +		 * TODO: Secondary process should be informed when a
> > +		 * port is released on primary through mp channel.
> > +		 */
> 
> That seems like a weird thing to leave out for TODO - it looks like an API
> deficiency. Can this be automatically updated on multiprocess hotplug sync, or
> somehow managed inside RTE_ETH_FOREACH_DEV?
> 
> As i understand, per-process ethdev list is not protected by any locks, so doing
> this is racy. Since this is a multiprocess hotplug example app, it should
> demonstrate best practices. So, either RTE_ETH_FOREACH_DEV should be
> fixed to handle this case, or the application should demonstrate how to
> properly synchronize access to local device list. The latter is probably better as
> adding locking around ethdev device list is outside the scope of this patchset.

All this comment should be removed since TODO already done :)
Actually, we guarantee device be detached from secondary before primary.

> 
> > +		if (strlen(dev_name) > 0)
> > +			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
> > +		else
> > +			printf("empty dev_name is not expected!\n");
> > +	}
> 
> <snip>
> 
> > +#include "commands.h"
> > +
> > +int main(int argc, char **argv)
> > +{
> > +	int ret;
> > +	struct cmdline *cl;
> > +
> > +	ret = rte_eal_init(argc, argv);
> > +	if (ret < 0)
> > +		rte_panic("Cannot init EAL\n");
> > +
> > +	cl = cmdline_stdin_new(main_ctx, "example> ");
> > +	if (cl == NULL)
> > +		rte_panic("Cannot create cmdline instance\n");
> > +	cmdline_interact(cl);
> > +	cmdline_stdin_exit(cl);
> > +
> > +	return 0;
> 
> Application should call rte_eal_cleanup() before exit. Otherwise, each
> secondary started and stopped will leak memory.

OK, will add it.

Thanks
Qi

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process

[Andrew Rybchenko]

On 06/21/2018 05:00 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
>
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
>
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
>
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
>
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
>
> Scenario for Case 1, 2:
>
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
>
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
>
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
>
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
>
> APIs changes:
>
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
>
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>
> v2:
> - rename rte_ethdev_mp.* to ethdev_mp.*
> - add experimental tag for rte_eth_dev_attach_private and
>    rte_ethdev_detach_private.
> - move do_eth_dev_attach and do_eth_dev_detach to ethdev_private.h
> - move rte_eth_dev_mp_init before rte_eal_mcfg_complete.
> - fix meson.build.
> - improve commit log.
>
>   lib/librte_eal/common/eal_private.h |   8 ++
>   lib/librte_eal/linuxapp/eal/eal.c   |   7 ++
>   lib/librte_ethdev/Makefile          |   1 +
>   lib/librte_ethdev/ethdev_mp.c       | 198 ++++++++++++++++++++++++++++++++++++
>   lib/librte_ethdev/ethdev_mp.h       |  44 ++++++++
>   lib/librte_ethdev/ethdev_private.h  |  39 +++++++
>   lib/librte_ethdev/meson.build       |   1 +
>   lib/librte_ethdev/rte_ethdev.c      | 184 +++++++++++++++++++++++++++++----
>   lib/librte_ethdev/rte_ethdev.h      |  45 ++++++++
>   lib/librte_ethdev/rte_ethdev_core.h |   5 +
>   10 files changed, 515 insertions(+), 17 deletions(-)
>   create mode 100644 lib/librte_ethdev/ethdev_mp.c
>   create mode 100644 lib/librte_ethdev/ethdev_mp.h
>   create mode 100644 lib/librte_ethdev/ethdev_private.h
>
> diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
> index bdadc4d50..92fa59bed 100644
> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> @@ -258,4 +258,12 @@ int rte_mp_channel_init(void);
>    */
>   void dev_callback_process(char *device_name, enum rte_dev_event_type event);
>   
> +/**
> + * Register mp channel callback functions of ethdev layer.
> + *
> + * @return
> + *  0 on success.
> + *  (<0) on failure.
> + */
> +int rte_eth_dev_mp_init(void);

It looks like it makes cross-dependency between EAL and ethdev.
As far as I can see EAL does not have references to rte_eth_dev
functions yet. It looks really suspicious. The function is declared in
EAL, but implemented in ethdev.
Moreover, it is declared once again in ethdev_mp.h.

<...>

> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 33d12b3a2..2cb6de745 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -622,4 +622,9 @@ struct rte_eth_dev_data {
>    */
>   extern struct rte_eth_dev rte_eth_devices[];
>   
> +extern int ethdev_logtype;
> +#define ethdev_log(level, fmt, ...) \
> +	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)

It looks like it clashes with ethdev logging changes submitted by Ferruh.

Andrew.

Re: [dpdk-dev] [PATCH v2 07/22] net/i40e: enable port detach on secondary process

[Andrew Rybchenko]

On 06/21/2018 05:00 AM, Qi Zhang wrote:
> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_private, we can support this
> with minor change.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   drivers/net/i40e/i40e_ethdev.c    | 2 ++
>   drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
>   2 files changed, 11 insertions(+)
>
> diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
> index 13c5d3296..7d1f98422 100644
> --- a/drivers/net/i40e/i40e_ethdev.c
> +++ b/drivers/net/i40e/i40e_ethdev.c
> @@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
>   	if (!ethdev)
>   		return -ENODEV;
>   
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
>   
>   	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
>   		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
> diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
> index 804e44530..fc6f079d5 100644
> --- a/drivers/net/i40e/i40e_ethdev_vf.c
> +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> @@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>   
>   static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
>   {
> +	struct rte_eth_dev *ethdev;
> +		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
> +
> +	if (!ethdev)
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
> +
>   	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
>   }

As far as I can see similar changes are done in really many files.
Is it possible to avoid the duplication?
Can it be part of rte_eth_dev_pci_generic_remove() or correctly
named wrapper?

Andrew.

[dpdk-dev] [PATCH v3 00/22] enable hotplug on multi-process

[Qi Zhang]

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (23):
  eal: introduce one device scan
  bus/vdev: enable one device scan
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/e1000: enable port detach on secondary process
  net/igb: enable port detach on secondary process
  net/fm10k: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/failsafe: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample

 drivers/bus/vdev/vdev.c                      |  30 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/e1000/em_ethdev.c                |   9 +
 drivers/net/e1000/igb_ethdev.c               |   9 +
 drivers/net/failsafe/failsafe.c              |  16 +
 drivers/net/fm10k/fm10k_ethdev.c             |   9 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/i40e/i40e_ethdev_vf.c            |   9 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |  12 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 +
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 ++
 lib/librte_eal/common/eal_common_dev.c       |  17 +-
 lib/librte_eal/common/eal_common_proc.c      |  51 +-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_bus.h      |  16 +
 lib/librte_eal/common/include/rte_eal.h      |  34 ++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 ++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 714 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 ++
 lib/librte_ethdev/ethdev_private.h           |  39 ++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 272 +++++++++-
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 39 files changed, 2176 insertions(+), 32 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v3 01/23] eal: introduce one device scan

[Qi Zhang]

When hot plug a new device, it is not necessary to scan everything
on the bus since the devname and devargs are already there. So new
rte_bus ops "scan_one" is introduced, bus driver can implement this
function to simplify the hotplug process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
 lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 61cb3b162..1ad033536 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	if (ret)
 		goto err_devarg;
 
-	ret = bus->scan();
-	if (ret)
-		goto err_devarg;
+	/**
+	 * if bus support to scan specific device by devargs,
+	 * we don't need to scan all devices on the bus.
+	 */
+	if (bus->scan_one) {
+		dev = bus->scan_one(da);
+	} else {
+		ret = bus->scan();
+		if (ret)
+			goto err_devarg;
+
+		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
+	}
 
-	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
 	if (dev == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
 			devname);
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..3269ef78b 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -84,6 +84,21 @@ enum rte_iova_mode {
 typedef int (*rte_bus_scan_t)(void);
 
 /**
+ * Bus specific scan for one specific device attached on the bus.
+ * For each bus object, the scan would be responsible for finding the specific
+ * device and adding it to its private device list, and the device object will
+ * be return also.
+ *
+ * @param devargs
+ *	Device arguments be used to identify the device.
+ *
+ * @return
+ *	!NULL for successful scan
+ *	NULL for unsuccessful scan
+ */
+typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *devargs);
+
+/**
  * Implementation specific probe function which is responsible for linking
  * devices on that bus with applicable drivers.
  *
@@ -204,6 +219,7 @@ struct rte_bus {
 	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
 	const char *name;            /**< Name of the bus */
 	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
+	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
 	rte_bus_probe_t probe;       /**< Probe devices on bus */
 	rte_bus_find_device_t find_device; /**< Find a device on the bus */
 	rte_bus_plug_t plug;         /**< Probe single device for drivers */
-- 
2.13.6

[dpdk-dev] [PATCH v3 02/23] bus/vdev: enable one device scan

[Qi Zhang]

The patch implemented the ops scan_one for vdev bus, it gives two benifits
1. Improve scan efficiency when a device is attached as hotplug, since no
need to pupulate a new device by iterating all devargs in devargs_list.
2. It also avoid sync IPC invoke (which happens in vdev->scan on secondary
process). The benifit is this removes the potential deadlock in the case
when secondary process receive a request from primary process to attach a
new device, since vdev->scan will be invoked on mp thread itself in that
case.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 6139dd551..cdbd77df0 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -467,6 +467,35 @@ vdev_scan(void)
 	return 0;
 }
 
+static struct rte_device *vdev_scan_one(struct rte_devargs *devargs)
+{
+	struct rte_vdev_device *dev = NULL;
+
+	dev = calloc(1, sizeof(*dev));
+	if (!dev) {
+		VDEV_LOG(ERR, "failed to allocate memory for new device");
+		return NULL;
+	}
+
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
+
+	if (find_vdev(devargs->name)) {
+		VDEV_LOG(ERR, "device %s already exist", devargs->name);
+		free(dev);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+		return NULL;
+	}
+
+	dev->device.devargs = devargs;
+	dev->device.numa_node = SOCKET_ID_ANY;
+	dev->device.name = devargs->name;
+	TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
+
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+
+	return &dev->device;
+}
+
 static int
 vdev_probe(void)
 {
@@ -531,6 +560,7 @@ vdev_unplug(struct rte_device *dev)
 
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
+	.scan_one = vdev_scan_one,
 	.probe = vdev_probe,
 	.find_device = vdev_find_device,
 	.plug = vdev_plug,
-- 
2.13.6

[dpdk-dev] [PATCH v3 03/23] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..205b2ee33 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+
+	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
+
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
-	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
+		eth_dev->state = RTE_ETH_DEV_UNUSED;
+		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	}
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
 
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
-- 
2.13.6

[dpdk-dev] [PATCH v3 04/23] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  5 ++++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index 707d8ab30..fc0eb4d17 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
@@ -686,7 +722,20 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	return process_mp_init_callbacks();
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 05/23] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 220 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  41 +++++++
 lib/librte_ethdev/ethdev_private.h  |  39 +++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 190 ++++++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h      |  45 ++++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 8 files changed, 525 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..7b1e89a91
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,220 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include <rte_string_fns.h>
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static int
+handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, peer) < 0) {
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+		return -1;
+	}
+
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_RESPONSE,
+					   handle_primary_response);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_RESPONSE);
+			return ret;
+		}
+
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 205b2ee33..1a5861f30 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -656,9 +658,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -703,14 +704,105 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	return ret;
+
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -721,22 +813,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v3 06/23] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 140 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 7b1e89a91..b00c05c23 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,6 +5,7 @@
 #include <rte_string_fns.h>
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -109,7 +110,7 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 1a5861f30..575dd40ae 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -723,6 +724,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return ret;
 
@@ -789,7 +791,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -831,6 +832,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -877,6 +882,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -890,6 +896,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4692,6 +4702,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v3 07/23] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process, the implementation
references malloc_mp.c.

device attach on secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.

device detach on secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 513 +++++++++++++++++++++++++++++++++++++++++-
 lib/librte_ethdev/ethdev_mp.h |   1 +
 2 files changed, 504 insertions(+), 10 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index b00c05c23..af8cec8c0 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -3,12 +3,103 @@
  */
 
 #include <rte_string_fns.h>
+#include <sys/time.h>
+
+#include <rte_alarm.h>
+
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach:
+ * a) seconary send request to primary.
+ * b) primary attach the new device if failed goto i).
+ * c) primary forward attach request to all secondary.
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail response to secondary, goto k).
+ * j) send success response to secondary.
+ * k) end.
+
+ * device detach:
+ * a) secondary send request to primary.
+ * b) primary perform pre-detach check, if device is locked, got j).
+ * c) primary send pre-detach check request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success response to secondary, goto k).
+ * j) send fail response to secondary.
+ * k) end.
+ */
+
+enum req_state {
+	REQ_STATE_INACTIVE = 0,
+	REQ_STATE_ACTIVE,
+	REQ_STATE_COMPLETE
+};
+
+struct mp_request {
+	TAILQ_ENTRY(mp_request) next;
+	struct eth_dev_mp_req user_req; /**< contents of request */
+	pthread_cond_t cond; /**< variable we use to time out on this request */
+	enum req_state state; /**< indicate status of this request */
+};
+
+/*
+ * We could've used just a single request, but it may be possible for
+ * secondaries to timeout earlier than the primary, and send a new request while
+ * primary is still expecting replies to the old one. Therefore, each new
+ * request will get assigned a new ID, which is how we will distinguish between
+ * expected and unexpected messages.
+ */
+TAILQ_HEAD(mp_request_list, mp_request);
+static struct {
+	struct mp_request_list list;
+	pthread_mutex_t lock;
+} mp_request_list = {
+	.list = TAILQ_HEAD_INITIALIZER(mp_request_list.list),
+	.lock = PTHREAD_MUTEX_INITIALIZER
+};
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
+static struct mp_request *
+find_request_by_id(uint64_t id)
+{
+	struct mp_request *req;
+
+	TAILQ_FOREACH(req, &mp_request_list.list, next) {
+		if (req->user_req.id == id)
+			break;
+	}
+	return req;
+}
+
+static uint64_t
+get_unique_id(void)
+{
+	uint64_t id;
+
+	do {
+		id = rte_rand();
+	} while (find_request_by_id(id) != NULL);
+	return id;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req);
+
 static int detach_on_secondary(uint16_t port_id)
 {
 	struct rte_device *dev;
@@ -78,19 +169,355 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+check_reply(const struct eth_dev_mp_req *req, const struct rte_mp_reply *reply)
+{
+	struct eth_dev_mp_req *resp;
+	int i;
+
+	if (reply->nb_received != reply->nb_sent)
+		return -EINVAL;
+
+	for (i = 0; i < reply->nb_received; i++) {
+		resp = (struct eth_dev_mp_req *)reply->msgs[i].param;
+
+		if (resp->t != req->t) {
+			ethdev_log(ERR, "Unexpected response to async request\n");
+			return -EINVAL;
+		}
+
+		if (resp->id != req->id) {
+			ethdev_log(ERR, "response to wrong async request\n");
+			return -ENOENT;
+		}
+
+		if (resp->result)
+			return resp->result;
+	}
+
+	return 0;
+}
+
+static int
+send_response_to_secondary(const struct eth_dev_mp_req *req, int result)
+{
+	struct rte_mp_msg resp_msg;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)resp_msg.param;
+	int ret = 0;
+
+	memset(&resp_msg, 0, sizeof(resp_msg));
+	resp_msg.len_param = sizeof(*resp);
+	strcpy(resp_msg.name, ETH_DEV_MP_ACTION_RESPONSE);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_sendmsg(&resp_msg);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static int
+handle_async_attach_response(const struct rte_mp_msg *request,
+			     const struct rte_mp_reply *reply)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	struct eth_dev_mp_req tmp_req;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_ATTACH_ROLLBACK;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, 0);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+handle_async_detach_response(const struct rte_mp_msg *request,
+			const struct rte_mp_reply *reply)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (ret) {
+		send_response_to_secondary(req, ret);
+	} else {
+		do_eth_dev_detach(req->port_id);
+		send_response_to_secondary(req, 0);
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
 }
 
 static int
-handle_primary_response(const struct rte_mp_msg *msg, const void *peer)
+handle_async_pre_detach_response(const struct rte_mp_msg *request,
+				const struct rte_mp_reply *reply)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct eth_dev_mp_req tmp_req;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ret = check_reply(req, reply);
+	if (!ret) {
+		tmp_req = *req;
+		tmp_req.t = REQ_TYPE_DETACH;
+
+		ret = send_request_to_secondary_async(&tmp_req);
+		if (ret) {
+			ethdev_log(ERR, "couldn't send async request\n");
+			TAILQ_REMOVE(&mp_request_list.list, entry, next);
+			free(entry);
+		}
+	} else {
+		send_response_to_secondary(req, ret);
+		TAILQ_REMOVE(&mp_request_list.list, entry, next);
+		free(entry);
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return 0;
+}
+
+static int
+handle_async_rollback_response(const struct rte_mp_msg *request,
+				const struct rte_mp_reply *reply __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)request->param;
+	struct mp_request *entry;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (!entry) {
+		ethdev_log(ERR, "wrong request ID\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	/* we have nothing to do if rollback still fail, just detach */
+	do_eth_dev_detach(req->port_id);
+	/* send response to secondary with the reason of rollback */
+	send_response_to_secondary(req, req->result);
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+	free(entry);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	return ret;
+}
+
+static int
+send_request_to_secondary_async(const struct eth_dev_mp_req *req)
+{
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct rte_mp_msg mp_req;
+	rte_mp_async_reply_t clb;
+	int ret = 0;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	if (req->t == REQ_TYPE_ATTACH)
+		clb = handle_async_attach_response;
+	else if (req->t == REQ_TYPE_PRE_DETACH)
+		clb = handle_async_pre_detach_response;
+	else if (req->t == REQ_TYPE_DETACH)
+		clb = handle_async_detach_response;
+	else if (req->t == REQ_TYPE_ATTACH_ROLLBACK)
+		clb = handle_async_rollback_response;
+	else
+		return -1;
+	do {
+		ret = rte_mp_request_async(&mp_req, &ts, clb);
+	} while (ret != 0 && rte_errno == EEXIST);
+
+	if (ret)
+		ethdev_log(ERR, "couldn't send async request\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct rte_mp_msg *msg = param;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	struct mp_request *entry;
+	uint16_t port_id;
+	int ret = 0;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		ethdev_log(ERR, "duplicate request id\n");
+		ret = -EEXIST;
+		goto finish;
+	}
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		ret = -ENOMEM;
+		goto finish;
+	}
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.port_id = port_id;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req = *req;
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = send_request_to_secondary_async(&tmp_req);
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+		goto finish;
+	}
+
+	if (ret) {
+		ret = send_response_to_secondary(req, ret);
+		if (ret) {
+			ethdev_log(ERR, "failed to send response to secondary\n");
+			goto finish;
+		}
+	} else {
+		memcpy(&entry->user_req, req, sizeof(*req));
+		entry->state = REQ_STATE_ACTIVE;
+		TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+		entry = NULL;
+	}
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	if (entry)
+		free(entry);
+	free(msg);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer __rte_unused)
+{
+	struct rte_mp_msg *msg_cpy;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	msg_cpy = malloc(sizeof(*msg_cpy));
+	if (msg_cpy == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM);
+	}
+
+	memcpy(msg_cpy, msg, sizeof(*msg_cpy));
+
+	/**
+	 * We can't handle the secondary request in mp callback because
+	 * we are running in primary process, we are going to invoke SYNC IPC
+	 * in rte_malloc.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, msg_cpy);
+	if (ret) {
+		ethdev_log(ERR, "failed to set alarm callback\n");
+		return send_response_to_secondary(req, ret);
+	}
+	return 0;
+}
+
+static int
+handle_primary_response(const struct rte_mp_msg *msg,
+			const void *peer __rte_unused)
+{
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct mp_request *entry;
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	entry = find_request_by_id(req->id);
+	if (entry) {
+		entry->user_req.result = req->result;
+		entry->user_req.port_id = req->port_id;
+		entry->state = REQ_STATE_COMPLETE;
+
+		pthread_cond_signal(&entry->cond);
+	}
+
+	pthread_mutex_unlock(&mp_request_list.lock);
+
+	return 0;
 }
 
 static int
@@ -134,8 +561,74 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg msg;
+	struct eth_dev_mp_req *msg_req = (struct eth_dev_mp_req *)msg.param;
+	struct mp_request *entry;
+	struct timespec ts;
+	struct timeval now;
+	int ret = 0;
+
+	memset(&msg, 0, sizeof(msg));
+	memset(&ts, 0, sizeof(ts));
+
+	entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		ethdev_log(ERR, "not enough memory to allocate request entry\n");
+		return -ENOMEM;
+	}
+
+	pthread_mutex_lock(&mp_request_list.lock);
+
+	ret = gettimeofday(&now, NULL);
+	if (ret) {
+		ethdev_log(ERR, "cannot get current time\n");
+		ret = -EINVAL;
+		goto finish;
+	}
+
+	ts.tv_nsec = (now.tv_usec * 1000) % 1000000000;
+	ts.tv_sec = now.tv_sec + MP_TIMEOUT_S +
+			(now.tv_usec * 1000) / 1000000000;
+
+	pthread_cond_init(&entry->cond, NULL);
+
+	msg.len_param = sizeof(*req);
+	strcpy(msg.name, ETH_DEV_MP_ACTION_REQUEST);
+
+	req->id = get_unique_id();
+
+	memcpy(msg_req, req, sizeof(*req));
+
+	ret = rte_mp_sendmsg(&msg);
+	if (ret) {
+		ethdev_log(ERR, "cannot send message to primary");
+		goto finish;
+	}
+
+	memcpy(&entry->user_req, req, sizeof(*req));
+
+	entry->state = REQ_STATE_ACTIVE;
+
+	TAILQ_INSERT_TAIL(&mp_request_list.list, entry, next);
+
+	do {
+		ret = pthread_cond_timedwait(&entry->cond,
+				&mp_request_list.lock, &ts);
+	} while (ret != 0 && ret != ETIMEDOUT);
+
+	if (entry->state != REQ_STATE_COMPLETE) {
+		RTE_LOG(ERR, EAL, "request time out\n");
+		ret = -ETIMEDOUT;
+	} else {
+		req->port_id = entry->user_req.port_id;
+		req->result = entry->user_req.result;
+	}
+	TAILQ_REMOVE(&mp_request_list.list, entry, next);
+
+finish:
+	pthread_mutex_unlock(&mp_request_list.lock);
+	free(entry);
+	return ret;
 }
 
 /**
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..94ff21cdd 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -18,6 +18,7 @@ enum eth_dev_req_type {
 };
 
 struct eth_dev_mp_req {
+	uint64_t id;
 	enum eth_dev_req_type t;
 	char devargs[MAX_DEV_ARGS_LEN];
 	uint16_t port_id;
-- 
2.13.6

[dpdk-dev] [PATCH v3 08/23] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c    | 2 ++
 drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 804e44530..fc6f079d5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 09/23] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..f9d560835 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
@@ -1809,6 +1812,15 @@ static struct rte_pci_driver rte_ixgbe_pmd = {
 static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+
+	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_probe(pci_dev,
 		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
 }
-- 
2.13.6

[dpdk-dev] [PATCH v3 10/23] net/e1000: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/em_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index 7039dc100..e6b7ce63a 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -349,6 +349,15 @@ static int eth_em_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_em_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_em_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 11/23] net/igb: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/igb_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index edc7be319..db07a83e3 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -1089,6 +1089,15 @@ static int eth_igb_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_igb_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_igb_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 12/23] net/fm10k: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 3ff1b0e0f..f73301182 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -3264,6 +3264,15 @@ static int eth_fm10k_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_fm10k_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_fm10k_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 13/23] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v3 14/23] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v3 15/23] net/failsafe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c
index eafbb75df..c5e8651f6 100644
--- a/drivers/net/failsafe/failsafe.c
+++ b/drivers/net/failsafe/failsafe.c
@@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &failsafe_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 static int
 rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)
 {
+	struct rte_eth_dev *eth_dev;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
 	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
+
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario.
+		 */
+	}
+
 	return fs_rte_eth_free(name);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 16/23] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v3 17/23] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v3 18/23] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 19/23] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v3 20/23] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v3 21/23] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v3 22/23] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v3 23/23] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v3 08/23] net/i40e: enable port detach on secondary process

[Andrew Rybchenko]

On 06/25/2018 10:17 AM, Qi Zhang wrote:
> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_private, we can support this
> with minor change.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   drivers/net/i40e/i40e_ethdev.c    | 2 ++
>   drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
>   2 files changed, 11 insertions(+)
>
> diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
> index 13c5d3296..7d1f98422 100644
> --- a/drivers/net/i40e/i40e_ethdev.c
> +++ b/drivers/net/i40e/i40e_ethdev.c
> @@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
>   	if (!ethdev)
>   		return -ENODEV;
>   
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
>   
>   	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
>   		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
> diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
> index 804e44530..fc6f079d5 100644
> --- a/drivers/net/i40e/i40e_ethdev_vf.c
> +++ b/drivers/net/i40e/i40e_ethdev_vf.c
> @@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>   
>   static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
>   {
> +	struct rte_eth_dev *ethdev;
> +		ethdev = rte_eth_dev_allocated(pci_dev->device.name);

Above two lines looks strange. Is alignment incorrect?

> +
> +	if (!ethdev)
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
> +

These 9 lines are duplicated in 5 changeset (this one and 4 following).
Shouldn't it be done in rte_eth_dev_pci_generic_remove()?

>   	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
>   }

[dpdk-dev] [PATCH v4 00/24] enable hotplug on multi-process

[Qi Zhang]

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (24):
  eal: introduce one device scan
  bus/vdev: enable one device scan
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  eal: support mp task be invoked in a separate task
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/e1000: enable port detach on secondary process
  net/igb: enable port detach on secondary process
  net/fm10k: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/failsafe: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample

 drivers/bus/vdev/vdev.c                      |  30 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/e1000/em_ethdev.c                |   9 +
 drivers/net/e1000/igb_ethdev.c               |   9 +
 drivers/net/failsafe/failsafe.c              |  16 ++
 drivers/net/fm10k/fm10k_ethdev.c             |   9 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/i40e/i40e_ethdev_vf.c            |   9 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |  12 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 ++
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 ++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_dev.c       |  17 +-
 lib/librte_eal/common/eal_common_proc.c      | 116 ++++++++
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_bus.h      |  16 ++
 lib/librte_eal/common/include/rte_eal.h      |  65 +++++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 ++++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 +++
 lib/librte_ethdev/ethdev_mp.c                | 392 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  41 +++
 lib/librte_ethdev/ethdev_private.h           |  39 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 272 +++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 ++++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 39 files changed, 1950 insertions(+), 31 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Qi Zhang]

When hot plug a new device, it is not necessary to scan everything
on the bus since the devname and devargs are already there. So new
rte_bus ops "scan_one" is introduced, bus driver can implement this
function to simplify the hotplug process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

 lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
 lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 61cb3b162..1ad033536 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	if (ret)
 		goto err_devarg;
 
-	ret = bus->scan();
-	if (ret)
-		goto err_devarg;
+	/**
+	 * if bus support to scan specific device by devargs,
+	 * we don't need to scan all devices on the bus.
+	 */
+	if (bus->scan_one) {
+		dev = bus->scan_one(da);
+	} else {
+		ret = bus->scan();
+		if (ret)
+			goto err_devarg;
+
+		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
+	}
 
-	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
 	if (dev == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
 			devname);
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..3269ef78b 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -84,6 +84,21 @@ enum rte_iova_mode {
 typedef int (*rte_bus_scan_t)(void);
 
 /**
+ * Bus specific scan for one specific device attached on the bus.
+ * For each bus object, the scan would be responsible for finding the specific
+ * device and adding it to its private device list, and the device object will
+ * be return also.
+ *
+ * @param devargs
+ *	Device arguments be used to identify the device.
+ *
+ * @return
+ *	!NULL for successful scan
+ *	NULL for unsuccessful scan
+ */
+typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *devargs);
+
+/**
  * Implementation specific probe function which is responsible for linking
  * devices on that bus with applicable drivers.
  *
@@ -204,6 +219,7 @@ struct rte_bus {
 	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
 	const char *name;            /**< Name of the bus */
 	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
+	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
 	rte_bus_probe_t probe;       /**< Probe devices on bus */
 	rte_bus_find_device_t find_device; /**< Find a device on the bus */
 	rte_bus_plug_t plug;         /**< Probe single device for drivers */
-- 
2.13.6

[dpdk-dev] [PATCH v4 02/24] bus/vdev: enable one device scan

[Qi Zhang]

The patch implemented the ops scan_one for vdev bus, it gives two benifits
1. Improve scan efficiency when a device is attached as hotplug, since no
need to pupulate a new device by iterating all devargs in devargs_list.
2. It also avoid sync IPC invoke (which happens in vdev->scan on secondary
process). The benifit is this removes the potential deadlock in the case
when secondary process receive a request from primary process to attach a
new device, since vdev->scan will be invoked on mp thread itself in that
case.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

 drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 6139dd551..cdbd77df0 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -467,6 +467,35 @@ vdev_scan(void)
 	return 0;
 }
 
+static struct rte_device *vdev_scan_one(struct rte_devargs *devargs)
+{
+	struct rte_vdev_device *dev = NULL;
+
+	dev = calloc(1, sizeof(*dev));
+	if (!dev) {
+		VDEV_LOG(ERR, "failed to allocate memory for new device");
+		return NULL;
+	}
+
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
+
+	if (find_vdev(devargs->name)) {
+		VDEV_LOG(ERR, "device %s already exist", devargs->name);
+		free(dev);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+		return NULL;
+	}
+
+	dev->device.devargs = devargs;
+	dev->device.numa_node = SOCKET_ID_ANY;
+	dev->device.name = devargs->name;
+	TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
+
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+
+	return &dev->device;
+}
+
 static int
 vdev_probe(void)
 {
@@ -531,6 +560,7 @@ vdev_unplug(struct rte_device *dev)
 
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
+	.scan_one = vdev_scan_one,
 	.probe = vdev_probe,
 	.find_device = vdev_find_device,
 	.plug = vdev_plug,
-- 
2.13.6

[dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---

 lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..205b2ee33 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+
+	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
+
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
-	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
+		eth_dev->state = RTE_ETH_DEV_UNUSED;
+		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
+	}
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
 
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
-- 
2.13.6

[dpdk-dev] [PATCH v4 04/24] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  5 ++++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index 707d8ab30..fc0eb4d17 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
@@ -686,7 +722,20 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	return process_mp_init_callbacks();
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 05/24] eal: support mp task be invoked in a separate task

[Qi Zhang]

We know the limitation that sync IPC can't be invoked in mp handler
itself which will cause deadlock, the patch introduce new API
rte_eal_mp_task_add to support mp handler be delegated in a separate
task.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 67 +++++++++++++++++++++++++++++++++
 lib/librte_eal/common/include/rte_eal.h | 31 +++++++++++++++
 2 files changed, 98 insertions(+)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index fc0eb4d17..166bb0951 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -27,6 +27,7 @@
 #include <rte_lcore.h>
 #include <rte_log.h>
 #include <rte_tailq.h>
+#include <rte_spinlock.h>
 
 #include "eal_private.h"
 #include "eal_filesystem.h"
@@ -738,6 +739,72 @@ void rte_mp_init_callback_cleanup(void)
 	}
 }
 
+struct mp_task {
+	TAILQ_ENTRY(mp_task) next;
+	rte_eal_mp_task task;
+	void *args;
+};
+
+TAILQ_HEAD(mp_task_list, mp_task);
+static struct mp_task_list mp_task_list =
+	TAILQ_HEAD_INITIALIZER(mp_task_list);
+static rte_spinlock_t mp_task_lock = RTE_SPINLOCK_INITIALIZER;
+
+static void *schedule_mp_task(void *args __rte_unused)
+{
+	struct mp_task *task;
+
+	rte_spinlock_lock(&mp_task_lock);
+	while (!TAILQ_EMPTY(&mp_task_list)) {
+
+		task = TAILQ_FIRST(&mp_task_list);
+		rte_spinlock_unlock(&mp_task_lock);
+
+		task->task(task->args);
+
+		rte_spinlock_lock(&mp_task_lock);
+		TAILQ_REMOVE(&mp_task_list, task, next);
+		if (task->args)
+			free(task->args);
+		free(task);
+	}
+
+	rte_spinlock_unlock(&mp_task_lock);
+	return NULL;
+}
+
+int __rte_experimental
+rte_eal_mp_task_add(rte_eal_mp_task task, void *args)
+{
+	struct mp_task *t = calloc(1, sizeof(struct mp_task));
+	pthread_t tid;
+
+	if (t == NULL)
+		return -ENOMEM;
+
+	t->task = task;
+	t->args = args;
+
+	rte_spinlock_lock(&mp_task_lock);
+
+	if (TAILQ_EMPTY(&mp_task_list)) {
+		TAILQ_INSERT_TAIL(&mp_task_list, t, next);
+		rte_spinlock_unlock(&mp_task_lock);
+
+		if (rte_ctrl_thread_create(&tid,
+			"rte_mp_handle", NULL, schedule_mp_task, NULL) < 0) {
+			RTE_LOG(ERR, EAL, "failed to create mp thead: %s\n",
+				strerror(errno));
+		}
+		return 0;
+	}
+
+	TAILQ_INSERT_TAIL(&mp_task_list, t, next);
+	rte_spinlock_unlock(&mp_task_lock);
+
+	return 0;
+}
+
 /**
  * Return -1, as fail to send message and it's caused by the local side.
  * Return 0, as fail to send message and it's caused by the remote side.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 506f17f34..0ce49668c 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -546,6 +546,37 @@ typedef int (*rte_eal_mp_init_callback_t)(void);
 int __rte_experimental
 rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
 
+/**
+ * Function to perform the task that handle mp request,
+ * it will be scheduled on a separate task.
+ *
+ * @param args
+ *   argument parse to the function point.
+ */
+typedef void (*rte_eal_mp_task)(void *args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add a rte_eal_mp_task into a task list, it will invoked in a
+ * separate task, the purpose is to prevent deadlock if sync IPC
+ * is required in the task.
+ *
+ * @param task
+ *   function point to perform the task.
+ *
+ * @param args
+ *   argument parse to the function point.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_mp_task_add(rte_eal_mp_task task, void *args);
+
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 248 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  41 ++++++
 lib/librte_ethdev/ethdev_private.h  |  39 ++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 190 ++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h      |  45 +++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 8 files changed, 553 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..87fc430bf
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,248 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#include <rte_string_fns.h>
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	const void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	bundle->peer = peer;
+
+	ret = rte_eal_mp_task_add(__handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 205b2ee33..1a5861f30 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -656,9 +658,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -703,14 +704,105 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	return ret;
+
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -721,22 +813,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v4 07/24] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 140 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 87fc430bf..b94bd9501 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,6 +5,7 @@
 #include <rte_string_fns.h>
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -107,7 +108,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 1a5861f30..575dd40ae 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -723,6 +724,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return ret;
 
@@ -789,7 +791,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -831,6 +832,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -877,6 +882,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -890,6 +896,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4692,6 +4702,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v4 08/24] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 155 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 149 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index b94bd9501..7afbd4cf2 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -4,8 +4,44 @@
 
 #include <rte_string_fns.h>
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -83,11 +119,98 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req.port_id = port_id;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+		if (!ret && !tmp_req.result) {
+			tmp_req.t = REQ_TYPE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+		if (!ret)
+			ret = do_eth_dev_detach(req->port_id);
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
+{
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	bundle->peer = peer;
+
+	ret = rte_eal_mp_task_add(__handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -170,8 +293,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v4 09/24] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c    | 2 ++
 drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 804e44530..fc6f079d5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 10/24] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..f9d560835 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
@@ -1809,6 +1812,15 @@ static struct rte_pci_driver rte_ixgbe_pmd = {
 static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+
+	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_probe(pci_dev,
 		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
 }
-- 
2.13.6

[dpdk-dev] [PATCH v4 11/24] net/e1000: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/em_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index 7039dc100..e6b7ce63a 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -349,6 +349,15 @@ static int eth_em_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_em_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_em_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 12/24] net/igb: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/igb_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index edc7be319..db07a83e3 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -1089,6 +1089,15 @@ static int eth_igb_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_igb_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_igb_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 13/24] net/fm10k: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 3ff1b0e0f..f73301182 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -3264,6 +3264,15 @@ static int eth_fm10k_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_fm10k_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_fm10k_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 14/24] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v4 15/24] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v4 16/24] net/failsafe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c
index eafbb75df..c5e8651f6 100644
--- a/drivers/net/failsafe/failsafe.c
+++ b/drivers/net/failsafe/failsafe.c
@@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &failsafe_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 static int
 rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)
 {
+	struct rte_eth_dev *eth_dev;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
 	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
+
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario.
+		 */
+	}
+
 	return fs_rte_eth_free(name);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 17/24] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v4 18/24] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v4 19/24] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 20/24] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v4 21/24] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v4 22/24] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v4 23/24] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v4 24/24] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v4 05/24] eal: support mp task be invoked in a separate task

[Burakov, Anatoly]

On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> We know the limitation that sync IPC can't be invoked in mp handler
> itself which will cause deadlock, the patch introduce new API
> rte_eal_mp_task_add to support mp handler be delegated in a separate
> task.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

I would really like to find another solution to this problem. Creating a 
new thread per hotplug request seems like an overkill - even more so 
than having two threads. Creating a new thread potentially while the 
application is working may have other implications (e.g. there's a 
non-zero amount of time between thread created and thread affinitized, 
which may disrupt hotpaths).

It seems to me that the better solution would've been to leave the IPC 
thread in place. There are two IPC threads in the first place because 
there was a circular dependency between rte_malloc and alarm API. My 
patch fixes that - so how about we remove *one* IPC thread, but leave 
the other one in place?

Thomas, any thoughts? (quick description - hotplug needs IPC, and 
hotplug may need to allocate memory, which also needs IPC, which will 
cause a deadlock if IPC is one thread)

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 05/24] eal: support mp task be invoked in a separate task

[Thomas Monjalon]

26/06/2018 11:02, Burakov, Anatoly:
> On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > We know the limitation that sync IPC can't be invoked in mp handler
> > itself which will cause deadlock, the patch introduce new API
> > rte_eal_mp_task_add to support mp handler be delegated in a separate
> > task.
> > 
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> I would really like to find another solution to this problem. Creating a 
> new thread per hotplug request seems like an overkill - even more so 
> than having two threads. Creating a new thread potentially while the 
> application is working may have other implications (e.g. there's a 
> non-zero amount of time between thread created and thread affinitized, 
> which may disrupt hotpaths).
> 
> It seems to me that the better solution would've been to leave the IPC 
> thread in place. There are two IPC threads in the first place because 
> there was a circular dependency between rte_malloc and alarm API. My 
> patch fixes that - so how about we remove *one* IPC thread, but leave 
> the other one in place?
> 
> Thomas, any thoughts? (quick description - hotplug needs IPC, and 
> hotplug may need to allocate memory, which also needs IPC, which will 
> cause a deadlock if IPC is one thread)

We can keep one IPC thread until we find a better solution.

Re: [dpdk-dev] [PATCH v4 05/24] eal: support mp task be invoked in a separate task

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, June 26, 2018 5:24 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Zhang, Qi Z <qi.z.zhang@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>; dev@dpdk.org; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 05/24] eal: support mp task be invoked in a separate
> task
> 
> 26/06/2018 11:02, Burakov, Anatoly:
> > On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > > We know the limitation that sync IPC can't be invoked in mp handler
> > > itself which will cause deadlock, the patch introduce new API
> > > rte_eal_mp_task_add to support mp handler be delegated in a separate
> > > task.
> > >
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> >
> > I would really like to find another solution to this problem. Creating
> > a new thread per hotplug request seems like an overkill - even more so
> > than having two threads. Creating a new thread potentially while the
> > application is working may have other implications (e.g. there's a
> > non-zero amount of time between thread created and thread affinitized,
> > which may disrupt hotpaths).
> >
> > It seems to me that the better solution would've been to leave the IPC
> > thread in place. There are two IPC threads in the first place because
> > there was a circular dependency between rte_malloc and alarm API. My
> > patch fixes that - so how about we remove *one* IPC thread, but leave
> > the other one in place?
> >
> > Thomas, any thoughts? (quick description - hotplug needs IPC, and
> > hotplug may need to allocate memory, which also needs IPC, which will
> > cause a deadlock if IPC is one thread)
> 
> We can keep one IPC thread until we find a better solution.
> 
> 
OK, then I will delegate the task to interrupt thread and remove the temporal thread solution.

Thanks
Qi

> 

Re: [dpdk-dev] [PATCH v4 10/24] net/ixgbe: enable port detach on secondary process

[Remy Horton]

On 26/06/2018 08:08, Qi Zhang wrote:
[..]
>  static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>  	struct rte_pci_device *pci_dev)
>  {
> +	struct rte_eth_dev *ethdev;
> +
> +	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
> +	if (!ethdev)
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
> +
>  	return rte_eth_dev_pci_generic_probe(pci_dev,
>  		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
>  }
>

Is calling of rte_eth_dev_release_port_private() from the probe function 
intentional? To me it looks like the code has been pasted into the wrong 
place..

Re: [dpdk-dev] [PATCH v4 09/24] net/i40e: enable port detach on secondary process

[Remy Horton]

On 26/06/2018 08:08, Qi Zhang wrote:
[..]
>  static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
>  {
> +	struct rte_eth_dev *ethdev;
> +		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
> +
> +	if (!ethdev)
> +		return -ENODEV;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(ethdev);
> +
>  	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
>  }

This identical code appears in multiple drivers. Is there anything 
stopping it being folded into rte_eth_dev_pci_generic_remove()?

Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Remy Horton]

On 26/06/2018 08:08, Qi Zhang wrote:
[..]
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>
>  lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
>  lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++

Acked-by: Remy Horton <remy.horton@intel.com>

Re: [dpdk-dev] [PATCH v4 02/24] bus/vdev: enable one device scan

[Remy Horton]

On 26/06/2018 08:08, Qi Zhang wrote:
> The patch implemented the ops scan_one for vdev bus, it gives two benifits
> 1. Improve scan efficiency when a device is attached as hotplug, since no
> need to pupulate a new device by iterating all devargs in devargs_list.
> 2. It also avoid sync IPC invoke (which happens in vdev->scan on secondary
> process). The benifit is this removes the potential deadlock in the case
> when secondary process receive a request from primary process to attach a
> new device, since vdev->scan will be invoked on mp thread itself in that
> case.

Slight rewording and spelling corrections within description:

The patch implements the ops scan_one for the vdev bus, which gives two 
benefits:
1. Improves scan efficiency when a device is attached as hotplug, since 
there is no need to populate a new device by iterating all devargs in 
devargs_list
2. It also avoids sync IPC invoke (which happens in vdev->scan on 
secondary process). The benefit is this removes the potential deadlock 
in the case when a secondary process receives a request from primary 
process to attach a new device, since vdev->scan will be invoked on mp 
thread itself in that case.


> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>
>  drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
>  1 file changed, 30 insertions(+)

Acked-by: Remy Horton <remy.horton@intel.com>

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Remy Horton]

On 26/06/2018 08:08, Qi Zhang wrote:
[..]
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>
>  lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
>  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
>  2 files changed, 34 insertions(+), 3 deletions(-)

Acked-by: Remy Horton <remy.horton@intel.com>

Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Burakov, Anatoly]

On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> When hot plug a new device, it is not necessary to scan everything
> on the bus since the devname and devargs are already there. So new
> rte_bus ops "scan_one" is introduced, bus driver can implement this
> function to simplify the hotplug process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> 

<snip>

> + *	NULL for unsuccessful scan
> + */
> +typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *devargs);
> +
> +/**
>    * Implementation specific probe function which is responsible for linking
>    * devices on that bus with applicable drivers.
>    *
> @@ -204,6 +219,7 @@ struct rte_bus {
>   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
>   	const char *name;            /**< Name of the bus */
>   	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
> +	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
>   	rte_bus_probe_t probe;       /**< Probe devices on bus */
>   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
>   	rte_bus_plug_t plug;         /**< Probe single device for drivers */
> 

Does changing this structure break ABI for bus drivers?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Matan Azrad]

Hi Qi

Please see comments\questions..

From: Qi Zhang
> Add driver API rte_eth_release_port_private to support the requirement that
> an ethdev only be released on secondary process, so only local state be set to
> unused , share data will not be reset so primary process can still use it.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> 
>  lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
>  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
>  2 files changed, 34 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index a9977df97..205b2ee33 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char *name)  }
> 
>  int
> +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev) {
> +	if (eth_dev == NULL)
> +		return -EINVAL;
> +
> +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> NULL);
> +
> +	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);

I don't think you need the lock here because there is not shared data to reset.

> +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> +
> +	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> +
> +	return 0;
> +}
> +
> +int
>  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)  {
>  	if (eth_dev == NULL)
> @@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev
> *eth_dev)
> 
>  	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> 
> -	eth_dev->state = RTE_ETH_DEV_UNUSED;
> -
> -	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> +	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {

Can you explain why not to release the shared data in case of locally unused port?

> +		eth_dev->state = RTE_ETH_DEV_UNUSED;
> +		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> +	}
> 
>  	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> 
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> b/lib/librte_ethdev/rte_ethdev_driver.h
> index c9c825e3f..49c27223d 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> *eth_dev);
> 
>  /**
>   * @internal
> + * Release the specified ethdev port in local process, only set to
> +ethdev
> + * state to unused, but not reset share data since it assume other
> +process
> + * is still using it, typically it is called by secondary process.
> + *
> + * @param eth_dev
> + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> +
> +/**
> + * @internal
>   * Release device queues and clear its configuration to force the user
>   * application to reconfigure it. It is for internal use only.
>   *
> --
> 2.13.6

Re: [dpdk-dev] [PATCH v4 04/24] eal: enable multi process init callback

[Burakov, Anatoly]

On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> Introduce new API rte_eal_register_mp_init that help to register
> a callback function which will be invoked right after multi-process
> channel be established (rte_mp_channel_init). Typically the API
> will be used by other module that want it's mp channel action callbacks
> can be registered during rte_eal_init automatically.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
>   lib/librte_eal/common/eal_private.h     |  5 ++++
>   lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
>   lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
>   4 files changed, 91 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
> index 707d8ab30..fc0eb4d17 100644
> --- a/lib/librte_eal/common/eal_common_proc.c
> +++ b/lib/librte_eal/common/eal_common_proc.c
> @@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
>   	return 0;
>   }
>   
> +struct mp_init_entry {
> +	TAILQ_ENTRY(mp_init_entry) next;
> +	rte_eal_mp_init_callback_t callback;
> +};
> +
> +TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
> +static struct mp_init_entry_list mp_init_entry_list =
> +	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
> +
> +static int process_mp_init_callbacks(void)
> +{
> +	struct mp_init_entry *entry;
> +	int ret;
> +
> +	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
> +		ret = entry->callback();
> +		if (ret)
> +			return ret;
> +	}
> +	return 0;
> +}
> +
> +int __rte_experimental
> +rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
> +{
> +	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
> +
> +	if (entry == NULL)
> +		return -ENOMEM;
> +
> +	entry->callback = callback;
> +	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
> +
> +	return 0;
> +}
> +
>   int
>   rte_mp_channel_init(void)
>   {
> @@ -686,7 +722,20 @@ rte_mp_channel_init(void)
>   	flock(dir_fd, LOCK_UN);
>   	close(dir_fd);
>   
> -	return 0;
> +	return process_mp_init_callbacks();
> +}
> +

Perhaps some kind of log message in case of failure would be useful? 
Otherwise this failure would be pretty silent - neither 
process_mp_init_callbacks() nor rte_mp_channel_init() logs any errors.

> +void rte_mp_init_callback_cleanup(void)
> +{
> +	struct mp_init_entry *entry;
> +
> +	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
> +		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
> +			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
> +			free(entry);
> +			break;
> +		}
> +	}
>   }
>   
>   /**
> diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
> index bdadc4d50..bc230ee23 100644
> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> @@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
>   int rte_mp_channel_init(void);
>   
>   /**
> + * Cleanup all mp channel init callbacks.
> + */
> +void rte_mp_init_callback_cleanup(void);

This is a good idea, however i should note that none of the other 
services (EAL or otherwise) clean up after themselves. They leave open 
file descriptors, malloc'd memory etc. all over the place. Maybe this is 
something to fix for the future...

> +
> +/**
>    * Internal Executes all the user application registered callbacks for
>    * the specific device. It is for DPDK internal user only. User
>    * application should not call it directly.
> diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
> index 8de5d69e8..506f17f34 100644
> --- a/lib/librte_eal/common/include/rte_eal.h
> +++ b/lib/librte_eal/common/include/rte_eal.h
> @@ -512,6 +512,40 @@ __rte_deprecated
>   const char *
>   rte_eal_mbuf_default_mempool_ops(void);

Otherwise,

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 24/24] examples/multi_process: add hotplug sample

[Burakov, Anatoly]

On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> The sample code demonstrate device (ethdev only) management
> at multi-process envrionment. User can attach/detach a device
> on primary process and see it is synced on secondary process
> automatically, also user can lock a device to prevent it be
> detached or unlock it to go back to default behaviour.
> 
> How to start?
> ./hotplug_mp --proc-type=auto
> 
> Command Line Example:
> 
>> help
>> list
> 
> /* attach a af_packet vdev */
>> attach net_af_packet,iface=eth0
> 
> /* detach port 0 */
>> detach 0
> 
> /* attach a private af_packet vdev (secondary process only)*/
>> attachp net_af_packet,iface=eth0
> 
> /* detach a private device (secondary process only) */
>> detachp 0
> 
> /* lock port 0 */
>> lock 0
> 
> /* unlock port 0 */
>> unlock 0
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

LGTM

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
> 
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs changes:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +static int
> +handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +
> +	struct rte_mp_msg mp_resp;
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	struct eth_dev_mp_req *resp =
> +		(struct eth_dev_mp_req *)mp_resp.param;
> +	struct mp_reply_bundle *bundle;
> +	int ret = 0;
> +
> +	memset(&mp_resp, 0, sizeof(mp_resp));
> +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
> +	mp_resp.len_param = sizeof(*req);
> +	memcpy(resp, req, sizeof(*resp));
> +
> +	bundle = calloc(1, sizeof(*bundle));
> +	if (bundle == NULL) {
> +		resp->result = -ENOMEM;
> +		ret = rte_mp_reply(&mp_resp, peer);
> +		if (ret) {
> +			ethdev_log(ERR, "failed to send reply to primary request\n");
> +			return ret;
> +		}
> +	}
> +
> +	bundle->msg = *msg;
> +	bundle->peer = peer;
> +
> +	ret = rte_eal_mp_task_add(__handle_primary_request, bundle);
> +	if (ret) {
> +		resp->result = ret;
> +		ret = rte_mp_reply(&mp_resp, peer);
> +		if (ret) {
> +			ethdev_log(ERR, "failed to send reply to primary request\n");
> +			return ret;
> +		}
> +	}

What you're doing here is quite dangerous. The parameter "const void 
*peer" is only guaranteed to be valid at the time of the callback - not 
necessarily afterwards. So, if you're handing off sending replies to a 
separate thread, things might blow up because the pointer may no longer 
be valid.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 8:09 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > We are going to introduce the solution to handle different hotplug
> > cases in multi-process situation, it include below scenario:
> >
> > 1. Attach a share device from primary
> > 2. Detach a share device from primary
> > 3. Attach a share device from secondary 4. Detach a share device from
> > secondary 5. Attach a private device from secondary 6. Detach a
> > private device from secondary 7. Detach a share device from secondary
> > privately 8. Attach a share device from secondary privately
> >
> > In primary-secondary process model, we assume device is shared by default.
> > that means attach or detach a device on any process will broadcast to
> > all other processes through mp channel then device information will be
> > synchronized on all processes.
> >
> > Any failure during attaching process will cause inconsistent status
> > between processes, so proper rollback action should be considered.
> > Also it is not safe to detach a share device when other process still
> > use it, so a handshake mechanism is introduced.
> >
> > This patch covers the implementation of case 1,2,5,6,7,8.
> > Case 3,4 will be implemented on separate patch as well as handshake
> > mechanism.
> >
> > Scenario for Case 1, 2:
> >
> > attach device
> > a) primary attach the new device if failed goto h).
> > b) primary send attach sync request to all secondary.
> > c) secondary receive request and attach device and send reply.
> > d) primary check the reply if all success go to i).
> > e) primary send attach rollback sync request to all secondary.
> > f) secondary receive the request and detach device and send reply.
> > g) primary receive the reply and detach device as rollback action.
> > h) attach fail
> > i) attach success
> >
> > detach device
> > a) primary perform pre-detach check, if device is locked, goto i).
> > b) primary send pre-detach sync request to all secondary.
> > c) secondary perform pre-detach check and send reply.
> > d) primary check the reply if any fail goto i).
> > e) primary send detach sync request to all secondary
> > f) secondary detach the device and send reply (assume no fail)
> > g) primary detach the device.
> > h) detach success
> > i) detach failed
> >
> > Case 5, 6:
> > Secondary process can attach private device which only visible to
> > itself, in this case no IPC is involved, primary process is not
> > allowed to have private device so far.
> >
> > Case 7, 8:
> > Secondary process can also temporally to detach a share device "privately"
> > then attach it back later, this action also not impact other processes.
> >
> > APIs changes:
> >
> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> > share device attach/detach in primary-secondary process model, it will
> > be called in case 1,2,3,4.
> >
> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> > introduced to cover case 5,6,7,8, this API can only be invoked in
> > secondary process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +static int
> > +handle_primary_request(const struct rte_mp_msg *msg, const void
> > +*peer) {
> > +
> > +	struct rte_mp_msg mp_resp;
> > +	const struct eth_dev_mp_req *req =
> > +		(const struct eth_dev_mp_req *)msg->param;
> > +	struct eth_dev_mp_req *resp =
> > +		(struct eth_dev_mp_req *)mp_resp.param;
> > +	struct mp_reply_bundle *bundle;
> > +	int ret = 0;
> > +
> > +	memset(&mp_resp, 0, sizeof(mp_resp));
> > +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST,
> sizeof(mp_resp.name));
> > +	mp_resp.len_param = sizeof(*req);
> > +	memcpy(resp, req, sizeof(*resp));
> > +
> > +	bundle = calloc(1, sizeof(*bundle));
> > +	if (bundle == NULL) {
> > +		resp->result = -ENOMEM;
> > +		ret = rte_mp_reply(&mp_resp, peer);
> > +		if (ret) {
> > +			ethdev_log(ERR, "failed to send reply to primary request\n");
> > +			return ret;
> > +		}
> > +	}
> > +
> > +	bundle->msg = *msg;
> > +	bundle->peer = peer;
> > +
> > +	ret = rte_eal_mp_task_add(__handle_primary_request, bundle);
> > +	if (ret) {
> > +		resp->result = ret;
> > +		ret = rte_mp_reply(&mp_resp, peer);
> > +		if (ret) {
> > +			ethdev_log(ERR, "failed to send reply to primary request\n");
> > +			return ret;
> > +		}
> > +	}
> 
> What you're doing here is quite dangerous. The parameter "const void *peer"
> is only guaranteed to be valid at the time of the callback - not necessarily
> afterwards. So, if you're handing off sending replies to a separate thread,
> things might blow up because the pointer may no longer be valid.

OK, so what about clone the content a buffer, I think the content should be valid before reply is sent, right?

Thanks
Qi
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 7:48 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 01/24] eal: introduce one device scan
> 
> On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > When hot plug a new device, it is not necessary to scan everything on
> > the bus since the devname and devargs are already there. So new
> > rte_bus ops "scan_one" is introduced, bus driver can implement this
> > function to simplify the hotplug process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >
> 
> <snip>
> 
> > + *	NULL for unsuccessful scan
> > + */
> > +typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs
> > +*devargs);
> > +
> > +/**
> >    * Implementation specific probe function which is responsible for linking
> >    * devices on that bus with applicable drivers.
> >    *
> > @@ -204,6 +219,7 @@ struct rte_bus {
> >   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
> >   	const char *name;            /**< Name of the bus */
> >   	rte_bus_scan_t scan;         /**< Scan for devices attached to
> bus */
> > +	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
> >   	rte_bus_probe_t probe;       /**< Probe devices on bus */
> >   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
> >   	rte_bus_plug_t plug;         /**< Probe single device for drivers
> */
> >
> 
> Does changing this structure break ABI for bus drivers?

For bus driver, I think yes, but I'm not sure what I should do for this, since this is not for application


> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 26-Jun-18 1:19 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Tuesday, June 26, 2018 8:09 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
>>
>> On 26-Jun-18 8:08 AM, Qi Zhang wrote:
>>> We are going to introduce the solution to handle different hotplug
>>> cases in multi-process situation, it include below scenario:
>>>
>>> 1. Attach a share device from primary
>>> 2. Detach a share device from primary
>>> 3. Attach a share device from secondary 4. Detach a share device from
>>> secondary 5. Attach a private device from secondary 6. Detach a
>>> private device from secondary 7. Detach a share device from secondary
>>> privately 8. Attach a share device from secondary privately
>>>
>>> In primary-secondary process model, we assume device is shared by default.
>>> that means attach or detach a device on any process will broadcast to
>>> all other processes through mp channel then device information will be
>>> synchronized on all processes.
>>>
>>> Any failure during attaching process will cause inconsistent status
>>> between processes, so proper rollback action should be considered.
>>> Also it is not safe to detach a share device when other process still
>>> use it, so a handshake mechanism is introduced.
>>>
>>> This patch covers the implementation of case 1,2,5,6,7,8.
>>> Case 3,4 will be implemented on separate patch as well as handshake
>>> mechanism.
>>>
>>> Scenario for Case 1, 2:
>>>
>>> attach device
>>> a) primary attach the new device if failed goto h).
>>> b) primary send attach sync request to all secondary.
>>> c) secondary receive request and attach device and send reply.
>>> d) primary check the reply if all success go to i).
>>> e) primary send attach rollback sync request to all secondary.
>>> f) secondary receive the request and detach device and send reply.
>>> g) primary receive the reply and detach device as rollback action.
>>> h) attach fail
>>> i) attach success
>>>
>>> detach device
>>> a) primary perform pre-detach check, if device is locked, goto i).
>>> b) primary send pre-detach sync request to all secondary.
>>> c) secondary perform pre-detach check and send reply.
>>> d) primary check the reply if any fail goto i).
>>> e) primary send detach sync request to all secondary
>>> f) secondary detach the device and send reply (assume no fail)
>>> g) primary detach the device.
>>> h) detach success
>>> i) detach failed
>>>
>>> Case 5, 6:
>>> Secondary process can attach private device which only visible to
>>> itself, in this case no IPC is involved, primary process is not
>>> allowed to have private device so far.
>>>
>>> Case 7, 8:
>>> Secondary process can also temporally to detach a share device "privately"
>>> then attach it back later, this action also not impact other processes.
>>>
>>> APIs changes:
>>>
>>> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
>>> share device attach/detach in primary-secondary process model, it will
>>> be called in case 1,2,3,4.
>>>
>>> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
>>> introduced to cover case 5,6,7,8, this API can only be invoked in
>>> secondary process.
>>>
>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>> ---
>>
>> <snip>
>>
>>> +static int
>>> +handle_primary_request(const struct rte_mp_msg *msg, const void
>>> +*peer) {
>>> +
>>> +	struct rte_mp_msg mp_resp;
>>> +	const struct eth_dev_mp_req *req =
>>> +		(const struct eth_dev_mp_req *)msg->param;
>>> +	struct eth_dev_mp_req *resp =
>>> +		(struct eth_dev_mp_req *)mp_resp.param;
>>> +	struct mp_reply_bundle *bundle;
>>> +	int ret = 0;
>>> +
>>> +	memset(&mp_resp, 0, sizeof(mp_resp));
>>> +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST,
>> sizeof(mp_resp.name));
>>> +	mp_resp.len_param = sizeof(*req);
>>> +	memcpy(resp, req, sizeof(*resp));
>>> +
>>> +	bundle = calloc(1, sizeof(*bundle));
>>> +	if (bundle == NULL) {
>>> +		resp->result = -ENOMEM;
>>> +		ret = rte_mp_reply(&mp_resp, peer);
>>> +		if (ret) {
>>> +			ethdev_log(ERR, "failed to send reply to primary request\n");
>>> +			return ret;
>>> +		}
>>> +	}
>>> +
>>> +	bundle->msg = *msg;
>>> +	bundle->peer = peer;
>>> +
>>> +	ret = rte_eal_mp_task_add(__handle_primary_request, bundle);
>>> +	if (ret) {
>>> +		resp->result = ret;
>>> +		ret = rte_mp_reply(&mp_resp, peer);
>>> +		if (ret) {
>>> +			ethdev_log(ERR, "failed to send reply to primary request\n");
>>> +			return ret;
>>> +		}
>>> +	}
>>
>> What you're doing here is quite dangerous. The parameter "const void *peer"
>> is only guaranteed to be valid at the time of the callback - not necessarily
>> afterwards. So, if you're handing off sending replies to a separate thread,
>> things might blow up because the pointer may no longer be valid.
> 
> OK, so what about clone the content a buffer, I think the content should be valid before reply is sent, right?

Yes, but even if you clone the content of the buffer, where would you 
send it *to*? You'll need the peer parameter to know where to send your 
response.

> 
> Thanks
> Qi
>>
>> --
>> Thanks,
>> Anatoly


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 8:50 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 1:19 PM, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Tuesday, June 26, 2018 8:09 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> >>
> >> On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> >>> We are going to introduce the solution to handle different hotplug
> >>> cases in multi-process situation, it include below scenario:
> >>>
> >>> 1. Attach a share device from primary 2. Detach a share device from
> >>> primary 3. Attach a share device from secondary 4. Detach a share
> >>> device from secondary 5. Attach a private device from secondary 6.
> >>> Detach a private device from secondary 7. Detach a share device from
> >>> secondary privately 8. Attach a share device from secondary
> >>> privately
> >>>
> >>> In primary-secondary process model, we assume device is shared by
> default.
> >>> that means attach or detach a device on any process will broadcast
> >>> to all other processes through mp channel then device information
> >>> will be synchronized on all processes.
> >>>
> >>> Any failure during attaching process will cause inconsistent status
> >>> between processes, so proper rollback action should be considered.
> >>> Also it is not safe to detach a share device when other process
> >>> still use it, so a handshake mechanism is introduced.
> >>>
> >>> This patch covers the implementation of case 1,2,5,6,7,8.
> >>> Case 3,4 will be implemented on separate patch as well as handshake
> >>> mechanism.
> >>>
> >>> Scenario for Case 1, 2:
> >>>
> >>> attach device
> >>> a) primary attach the new device if failed goto h).
> >>> b) primary send attach sync request to all secondary.
> >>> c) secondary receive request and attach device and send reply.
> >>> d) primary check the reply if all success go to i).
> >>> e) primary send attach rollback sync request to all secondary.
> >>> f) secondary receive the request and detach device and send reply.
> >>> g) primary receive the reply and detach device as rollback action.
> >>> h) attach fail
> >>> i) attach success
> >>>
> >>> detach device
> >>> a) primary perform pre-detach check, if device is locked, goto i).
> >>> b) primary send pre-detach sync request to all secondary.
> >>> c) secondary perform pre-detach check and send reply.
> >>> d) primary check the reply if any fail goto i).
> >>> e) primary send detach sync request to all secondary
> >>> f) secondary detach the device and send reply (assume no fail)
> >>> g) primary detach the device.
> >>> h) detach success
> >>> i) detach failed
> >>>
> >>> Case 5, 6:
> >>> Secondary process can attach private device which only visible to
> >>> itself, in this case no IPC is involved, primary process is not
> >>> allowed to have private device so far.
> >>>
> >>> Case 7, 8:
> >>> Secondary process can also temporally to detach a share device "privately"
> >>> then attach it back later, this action also not impact other processes.
> >>>
> >>> APIs changes:
> >>>
> >>> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> >>> share device attach/detach in primary-secondary process model, it
> >>> will be called in case 1,2,3,4.
> >>>
> >>> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private
> >>> are introduced to cover case 5,6,7,8, this API can only be invoked
> >>> in secondary process.
> >>>
> >>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> >>> ---
> >>
> >> <snip>
> >>
> >>> +static int
> >>> +handle_primary_request(const struct rte_mp_msg *msg, const void
> >>> +*peer) {
> >>> +
> >>> +	struct rte_mp_msg mp_resp;
> >>> +	const struct eth_dev_mp_req *req =
> >>> +		(const struct eth_dev_mp_req *)msg->param;
> >>> +	struct eth_dev_mp_req *resp =
> >>> +		(struct eth_dev_mp_req *)mp_resp.param;
> >>> +	struct mp_reply_bundle *bundle;
> >>> +	int ret = 0;
> >>> +
> >>> +	memset(&mp_resp, 0, sizeof(mp_resp));
> >>> +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST,
> >> sizeof(mp_resp.name));
> >>> +	mp_resp.len_param = sizeof(*req);
> >>> +	memcpy(resp, req, sizeof(*resp));
> >>> +
> >>> +	bundle = calloc(1, sizeof(*bundle));
> >>> +	if (bundle == NULL) {
> >>> +		resp->result = -ENOMEM;
> >>> +		ret = rte_mp_reply(&mp_resp, peer);
> >>> +		if (ret) {
> >>> +			ethdev_log(ERR, "failed to send reply to primary
> request\n");
> >>> +			return ret;
> >>> +		}
> >>> +	}
> >>> +
> >>> +	bundle->msg = *msg;
> >>> +	bundle->peer = peer;
> >>> +
> >>> +	ret = rte_eal_mp_task_add(__handle_primary_request, bundle);
> >>> +	if (ret) {
> >>> +		resp->result = ret;
> >>> +		ret = rte_mp_reply(&mp_resp, peer);
> >>> +		if (ret) {
> >>> +			ethdev_log(ERR, "failed to send reply to primary
> request\n");
> >>> +			return ret;
> >>> +		}
> >>> +	}
> >>
> >> What you're doing here is quite dangerous. The parameter "const void
> *peer"
> >> is only guaranteed to be valid at the time of the callback - not
> >> necessarily afterwards. So, if you're handing off sending replies to
> >> a separate thread, things might blow up because the pointer may no longer
> be valid.
> >
> > OK, so what about clone the content a buffer, I think the content should be
> valid before reply is sent, right?
> 
> Yes, but even if you clone the content of the buffer, where would you send it
> *to*? You'll need the peer parameter to know where to send your response.

my understand is peer is identified by a string (or filename)
what I mean is clone the content of the buffer that peer point to , 
So I don't need to worry if the original peer be used to point to some other data



> 
> >
> > Thanks
> > Qi
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
> 
> my understand is peer is identified by a string (or filename)
> what I mean is clone the content of the buffer that peer point to ,
> So I don't need to worry if the original peer be used to point to some other data
> 

As far as the application is concerned, peer is an opaque pointer, and 
should be treated as such. Peeking behind a void pointer that is not 
designed for this purpose is not a good idea, even if technically you 
know what's in there.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 9:21 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
> >
> > my understand is peer is identified by a string (or filename) what I
> > mean is clone the content of the buffer that peer point to , So I
> > don't need to worry if the original peer be used to point to some
> > other data
> >
> 
> As far as the application is concerned, peer is an opaque pointer, and should
> be treated as such. Peeking behind a void pointer that is not designed for this
> purpose is not a good idea, even if technically you know what's in there.

We can expose a clone interface, like MP_PEER_CLONE, so we don't need to know what's inside, just need to know that it can be used on another thread?


> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Zhang, Qi Z]

Hi Matan:

> -----Original Message-----
> From: Matan Azrad [mailto:matan@mellanox.com]
> Sent: Tuesday, June 26, 2018 7:50 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port
> in local process
> 
> Hi Qi
> 
> Please see comments\questions..
> 
> From: Qi Zhang
> > Add driver API rte_eth_release_port_private to support the requirement
> > that an ethdev only be released on secondary process, so only local
> > state be set to unused , share data will not be reset so primary process can
> still use it.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >
> >  lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
> >  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
> >  2 files changed, 34 insertions(+), 3 deletions(-)
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > b/lib/librte_ethdev/rte_ethdev.c index a9977df97..205b2ee33 100644
> > --- a/lib/librte_ethdev/rte_ethdev.c
> > +++ b/lib/librte_ethdev/rte_ethdev.c
> > @@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char
> *name)  }
> >
> >  int
> > +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev) {
> > +	if (eth_dev == NULL)
> > +		return -EINVAL;
> > +
> > +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> > NULL);
> > +
> > +	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> 
> I don't think you need the lock here because there is not shared data to reset.

OK, will remove this.

> 
> > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > +
> > +	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > +
> > +	return 0;
> > +}
> > +
> > +int
> >  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)  {
> >  	if (eth_dev == NULL)
> > @@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev
> > *eth_dev)
> >
> >  	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> >
> > -	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > -
> > -	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > +	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
> 
> Can you explain why not to release the shared data in case of locally unused
> port?

Now, we have rte_eth_dev_release_port be called in rte_eth_dev_detach, its redundant for some driver, because
they already will release port in dev->remove, but I'm not sure if it is true for all drivers.

On secondary process, when detach a device, it is possible that 
rte_eth_dev_release_port be called after rte_eth_dev_release_port_local , so it will
reset share data which is not expected, since the primary process will fail on detach it because
 rte_eth_dev_allocated will return NULL.

There could be other way, but current implementation help to reuse exist code.,
I can't add more comment on this change in v5

Regards
Qi

> 
> > +		eth_dev->state = RTE_ETH_DEV_UNUSED;
> > +		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > +	}
> >
> >  	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > b/lib/librte_ethdev/rte_ethdev_driver.h
> > index c9c825e3f..49c27223d 100644
> > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> > *eth_dev);
> >
> >  /**
> >   * @internal
> > + * Release the specified ethdev port in local process, only set to
> > +ethdev
> > + * state to unused, but not reset share data since it assume other
> > +process
> > + * is still using it, typically it is called by secondary process.
> > + *
> > + * @param eth_dev
> > + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > + * @return
> > + *   - 0 on success, negative on error
> > + */
> > +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> > +
> > +/**
> > + * @internal
> >   * Release device queues and clear its configuration to force the user
> >   * application to reconfigure it. It is for internal use only.
> >   *
> > --
> > 2.13.6

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Zhang, Qi Z]

> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Tuesday, June 26, 2018 9:29 PM
> To: 'Matan Azrad' <matan@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port
> in local process
> 
> Hi Matan:
> 
> > -----Original Message-----
> > From: Matan Azrad [mailto:matan@mellanox.com]
> > Sent: Tuesday, June 26, 2018 7:50 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> > <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > release port in local process
> >
> > Hi Qi
> >
> > Please see comments\questions..
> >
> > From: Qi Zhang
> > > Add driver API rte_eth_release_port_private to support the
> > > requirement that an ethdev only be released on secondary process, so
> > > only local state be set to unused , share data will not be reset so
> > > primary process can
> > still use it.
> > >
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> > >
> > >  lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
> > >  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
> > >  2 files changed, 34 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > > b/lib/librte_ethdev/rte_ethdev.c index a9977df97..205b2ee33 100644
> > > --- a/lib/librte_ethdev/rte_ethdev.c
> > > +++ b/lib/librte_ethdev/rte_ethdev.c
> > > @@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char
> > *name)  }
> > >
> > >  int
> > > +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev) {
> > > +	if (eth_dev == NULL)
> > > +		return -EINVAL;
> > > +
> > > +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> > > NULL);
> > > +
> > > +	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> >
> > I don't think you need the lock here because there is not shared data to
> reset.
> 
> OK, will remove this.
> 
> >
> > > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > +
> > > +	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > > +
> > > +	return 0;
> > > +}
> > > +
> > > +int
> > >  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)  {
> > >  	if (eth_dev == NULL)
> > > @@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev
> > > *eth_dev)
> > >
> > >  	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> > >
> > > -	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > -
> > > -	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > +	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
> >
> > Can you explain why not to release the shared data in case of locally
> > unused port?
> 
> Now, we have rte_eth_dev_release_port be called in rte_eth_dev_detach, its
> redundant for some driver, because they already will release port in
> dev->remove, but I'm not sure if it is true for all drivers.
> 
> On secondary process, when detach a device, it is possible that
> rte_eth_dev_release_port be called after rte_eth_dev_release_port_local , so
> it will reset share data which is not expected, since the primary process will fail
> on detach it because  rte_eth_dev_allocated will return NULL.
> 
> There could be other way, but current implementation help to reuse exist
> code., I can't add more comment on this change in v5

I can add more comment

> 
> Regards
> Qi
> 
> >
> > > +		eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > +		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > +	}
> > >
> > >  	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > >
> > > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > > b/lib/librte_ethdev/rte_ethdev_driver.h
> > > index c9c825e3f..49c27223d 100644
> > > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > > @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> > > *eth_dev);
> > >
> > >  /**
> > >   * @internal
> > > + * Release the specified ethdev port in local process, only set to
> > > +ethdev
> > > + * state to unused, but not reset share data since it assume other
> > > +process
> > > + * is still using it, typically it is called by secondary process.
> > > + *
> > > + * @param eth_dev
> > > + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > > + * @return
> > > + *   - 0 on success, negative on error
> > > + */
> > > +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> > > +
> > > +/**
> > > + * @internal
> > >   * Release device queues and clear its configuration to force the user
> > >   * application to reconfigure it. It is for internal use only.
> > >   *
> > > --
> > > 2.13.6

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 26-Jun-18 2:25 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Tuesday, June 26, 2018 9:21 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
>>
>> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
>>>
>>> my understand is peer is identified by a string (or filename) what I
>>> mean is clone the content of the buffer that peer point to , So I
>>> don't need to worry if the original peer be used to point to some
>>> other data
>>>
>>
>> As far as the application is concerned, peer is an opaque pointer, and should
>> be treated as such. Peeking behind a void pointer that is not designed for this
>> purpose is not a good idea, even if technically you know what's in there.
> 
> We can expose a clone interface, like MP_PEER_CLONE, so we don't need to know what's inside, just need to know that it can be used on another thread?
> 

Well, that can probably work. Feels like a hacky workaround though.

Another way to do the same thing would be to store peer information 
right in the message, as opposed to providing it separately. Still a 
hack though, and will require far more changes, but it could be a better 
solution as (if done right) it would allow identifying which reply came 
from which peer.

Of course, an even better approach would be to devise some kind of 
addressing scheme (uuid?), so that peer addresses are no longer opaque 
pointers but rather are valid data types.

Thoughts?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 9:46 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 2:25 PM, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Tuesday, June 26, 2018 9:21 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> >>
> >> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
> >>>
> >>> my understand is peer is identified by a string (or filename) what I
> >>> mean is clone the content of the buffer that peer point to , So I
> >>> don't need to worry if the original peer be used to point to some
> >>> other data
> >>>
> >>
> >> As far as the application is concerned, peer is an opaque pointer,
> >> and should be treated as such. Peeking behind a void pointer that is
> >> not designed for this purpose is not a good idea, even if technically you
> know what's in there.
> >
> > We can expose a clone interface, like MP_PEER_CLONE, so we don't need to
> know what's inside, just need to know that it can be used on another thread?
> >
> 
> Well, that can probably work. Feels like a hacky workaround though.
> 
> Another way to do the same thing would be to store peer information right in
> the message, as opposed to providing it separately. Still a hack though, and will
> require far more changes, but it could be a better solution as (if done right) it
> would allow identifying which reply came from which peer.
> 
> Of course, an even better approach would be to devise some kind of
> addressing scheme (uuid?), so that peer addresses are no longer opaque
> pointers but rather are valid data types.
> 
> Thoughts?

I may not give insight comment from the IPC implementation, but from user's view, what required is a unique token,
 it can be used for reply at anywhere and at any time, to me, currently implementation looks like missing some mapping management between an abstract token to its real data.  

Thanks
Qi.

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 26-Jun-18 3:24 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Tuesday, June 26, 2018 9:46 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
>>
>> On 26-Jun-18 2:25 PM, Zhang, Qi Z wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Burakov, Anatoly
>>>> Sent: Tuesday, June 26, 2018 9:21 PM
>>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>>>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>>>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>>>> <narender.vangati@intel.com>
>>>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
>>>>
>>>> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
>>>>>
>>>>> my understand is peer is identified by a string (or filename) what I
>>>>> mean is clone the content of the buffer that peer point to , So I
>>>>> don't need to worry if the original peer be used to point to some
>>>>> other data
>>>>>
>>>>
>>>> As far as the application is concerned, peer is an opaque pointer,
>>>> and should be treated as such. Peeking behind a void pointer that is
>>>> not designed for this purpose is not a good idea, even if technically you
>> know what's in there.
>>>
>>> We can expose a clone interface, like MP_PEER_CLONE, so we don't need to
>> know what's inside, just need to know that it can be used on another thread?
>>>
>>
>> Well, that can probably work. Feels like a hacky workaround though.
>>
>> Another way to do the same thing would be to store peer information right in
>> the message, as opposed to providing it separately. Still a hack though, and will
>> require far more changes, but it could be a better solution as (if done right) it
>> would allow identifying which reply came from which peer.
>>
>> Of course, an even better approach would be to devise some kind of
>> addressing scheme (uuid?), so that peer addresses are no longer opaque
>> pointers but rather are valid data types.
>>
>> Thoughts?
> 
> I may not give insight comment from the IPC implementation, but from user's view, what required is a unique token,
>   it can be used for reply at anywhere and at any time, to me, currently implementation looks like missing some mapping management between an abstract token to its real data.
> 

Well, the idea was to not provide that :) We wanted to keep it simple 
and actively discourage any attempts to know which peer you're 
communicating with, as allowing that would imply some kind of addressing 
scheme, peer discovery and things like that, which we don't want to 
bother with. If you want replies, you use callbacks, which provide you 
with a peer address.

On the one hand, i understand the frustration of being forced to deal 
with deliberate simplicity of IPC API and to create workarounds like 
doing sendmsg() and storing requests that you're waiting on inside the 
application and launching interrupt threads, all for things that should 
Just Work in an IPC API. Believe me, i know about its deficiencies more 
than most people :)

On the other hand, we don't want to introduce hacks to solve a specific 
problem. If we are to solve this in IPC, we should do it properly. I 
don't think adding a workaround with "cloning" peer address is the 
solution. A proper solution would've been to implement addressing 
scheme, so that any response could be submitted at any time.

The hard route would be to add peer discovery, etc. The easy route would 
be to just change the peer address to something typed, something that we 
can use to reconstruct the original peer id. A great candidate for this 
is the uuid, but it's an external dependency. A workaround for this 
would be to just use a uint64 value obtained through some magic that is 
part of socket path. For example, replace 
"/var/run/dpdk/rte/mp_socket_pid_rdtsc" with something just as (likely) 
unique, but something that can be put into a uint64.

Without significantly changing the API and the internals, i think this 
is the best we will be able to do.

> Thanks
> Qi.
> 
>>
>> --
>> Thanks,
>> Anatoly


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Gaëtan Rivet]

On Tue, Jun 26, 2018 at 12:26:05PM +0000, Zhang, Qi Z wrote:
> 
> 
> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Tuesday, June 26, 2018 7:48 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: Re: [PATCH v4 01/24] eal: introduce one device scan
> > 
> > On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > > When hot plug a new device, it is not necessary to scan everything on
> > > the bus since the devname and devargs are already there. So new
> > > rte_bus ops "scan_one" is introduced, bus driver can implement this
> > > function to simplify the hotplug process.
> > >
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> > >
> > 
> > <snip>
> > 
> > > + *	NULL for unsuccessful scan
> > > + */
> > > +typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs
> > > +*devargs);
> > > +
> > > +/**
> > >    * Implementation specific probe function which is responsible for linking
> > >    * devices on that bus with applicable drivers.
> > >    *
> > > @@ -204,6 +219,7 @@ struct rte_bus {
> > >   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
> > >   	const char *name;            /**< Name of the bus */
> > >   	rte_bus_scan_t scan;         /**< Scan for devices attached to
> > bus */
> > > +	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
> > >   	rte_bus_probe_t probe;       /**< Probe devices on bus */
> > >   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
> > >   	rte_bus_plug_t plug;         /**< Probe single device for drivers
> > */
> > >
> > 
> > Does changing this structure break ABI for bus drivers?
> 
> For bus driver, I think yes, but I'm not sure what I should do for this, since this is not for application
> 
> 

This should be appropriately announced in advance, in general.
However, it seems there is some leeway if the new field will not move
the others and not make the structure grow (i.e. replace a padding).

There is an ABI check script that can be used.

This however breaks the bus ABI, which breaks the EAL ABI.
This is usually an issue.

More generally, I was in favor of changing the whole bus scan process to a
per-device iteration. I was shut down on this when adding hotplug.
As a result, bus->scan() process was made to require the operation to be
idempotent.

Adding a new ops adds noise to the bus API. It should be kept as clean
as possible. This new one seems unnecessary, now that all bus scans are
idempotent (when supporting hotplug).

Regards,
-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Matan Azrad]

Hi Zhang

 From: Zhang, Qi Z 
> Sent: Tuesday, June 26, 2018 4:30 PM
> To: Matan Azrad <matan@mellanox.com>; Thomas Monjalon
> <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port
> in local process
> 
> 
> 
> > -----Original Message-----
> > From: Zhang, Qi Z
> > Sent: Tuesday, June 26, 2018 9:29 PM
> > To: 'Matan Azrad' <matan@mellanox.com>; Thomas Monjalon
> > <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > release port in local process
> >
> > Hi Matan:
> >
> > > -----Original Message-----
> > > From: Matan Azrad [mailto:matan@mellanox.com]
> > > Sent: Tuesday, June 26, 2018 7:50 PM
> > > To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> > > <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> > > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > > dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit,
> > > Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > > <narender.vangati@intel.com>
> > > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > > release port in local process
> > >
> > > Hi Qi
> > >
> > > Please see comments\questions..
> > >
> > > From: Qi Zhang
> > > > Add driver API rte_eth_release_port_private to support the
> > > > requirement that an ethdev only be released on secondary process,
> > > > so only local state be set to unused , share data will not be
> > > > reset so primary process can
> > > still use it.
> > > >
> > > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > > ---
> > > >
> > > >  lib/librte_ethdev/rte_ethdev.c        | 24 +++++++++++++++++++++---
> > > >  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
> > > >  2 files changed, 34 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > > > b/lib/librte_ethdev/rte_ethdev.c index a9977df97..205b2ee33 100644
> > > > --- a/lib/librte_ethdev/rte_ethdev.c
> > > > +++ b/lib/librte_ethdev/rte_ethdev.c
> > > > @@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char
> > > *name)  }
> > > >
> > > >  int
> > > > +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev) {
> > > > +	if (eth_dev == NULL)
> > > > +		return -EINVAL;
> > > > +
> > > > +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> > > > NULL);
> > > > +
> > > > +	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> > >
> > > I don't think you need the lock here because there is not shared
> > > data to
> > reset.
> >
> > OK, will remove this.
> >
> > >
> > > > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > +
> > > > +	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +int
> > > >  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)  {
> > > >  	if (eth_dev == NULL)
> > > > @@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev
> > > > *eth_dev)
> > > >
> > > >  	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> > > >
> > > > -	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > -
> > > > -	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > > +	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
> > >
> > > Can you explain why not to release the shared data in case of
> > > locally unused port?
> >
> > Now, we have rte_eth_dev_release_port be called in rte_eth_dev_detach,
> > its redundant for some driver, because they already will release port
> > in
> > dev->remove, but I'm not sure if it is true for all drivers.

rte_eth_dev_release_port() like detach port should be called only by the port owner, by default is the application, so it should not be called by the PMD, maybe need to fix PMDs which does it.

> >
> > On secondary process, when detach a device, it is possible that
> > rte_eth_dev_release_port be called after
> > rte_eth_dev_release_port_local , so it will reset share data which is
> > not expected, since the primary process will fail on detach it because
> rte_eth_dev_allocated will return NULL.
> >
> > There could be other way, but current implementation help to reuse
> > exist code., I can't add more comment on this change in v5
> 
> I can add more comment

This solution is suffering from much more complexity,
The port id should be the same in all the processes, so you can pass it by the mp channel and not use rte_eth_dev_allocated.

I see it like this:

The initiator process asks from all the other processes to detach a port by port id, all the other processes using rte_eth_dev_release_port_local and sending ack back to the initiator,
Then, the initiator using the non-local release to release the port.
 
Am I missing something?


> >
> > Regards
> > Qi
> >
> > >
> > > > +		eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > +		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > > +	}
> > > >
> > > >  	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > > >
> > > > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > > > b/lib/librte_ethdev/rte_ethdev_driver.h
> > > > index c9c825e3f..49c27223d 100644
> > > > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > > > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > > > @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> > > > *eth_dev);
> > > >
> > > >  /**
> > > >   * @internal
> > > > + * Release the specified ethdev port in local process, only set
> > > > +to ethdev
> > > > + * state to unused, but not reset share data since it assume
> > > > +other process
> > > > + * is still using it, typically it is called by secondary process.
> > > > + *
> > > > + * @param eth_dev
> > > > + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > > > + * @return
> > > > + *   - 0 on success, negative on error
> > > > + */
> > > > +int rte_eth_dev_release_port_private(struct rte_eth_dev
> > > > +*eth_dev);
> > > > +
> > > > +/**
> > > > + * @internal
> > > >   * Release device queues and clear its configuration to force the user
> > > >   * application to reconfigure it. It is for internal use only.
> > > >   *
> > > > --
> > > > 2.13.6

Re: [dpdk-dev] [PATCH v4 06/24] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, June 26, 2018 11:12 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> 
> On 26-Jun-18 3:24 PM, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Tuesday, June 26, 2018 9:46 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on multi-process
> >>
> >> On 26-Jun-18 2:25 PM, Zhang, Qi Z wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Burakov, Anatoly
> >>>> Sent: Tuesday, June 26, 2018 9:21 PM
> >>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> >>>> dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> >>>> Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >>>> <narender.vangati@intel.com>
> >>>> Subject: Re: [PATCH v4 06/24] ethdev: enable hotplug on
> >>>> multi-process
> >>>>
> >>>> On 26-Jun-18 1:58 PM, Zhang, Qi Z wrote:
> >>>>>
> >>>>> my understand is peer is identified by a string (or filename) what
> >>>>> I mean is clone the content of the buffer that peer point to , So
> >>>>> I don't need to worry if the original peer be used to point to
> >>>>> some other data
> >>>>>
> >>>>
> >>>> As far as the application is concerned, peer is an opaque pointer,
> >>>> and should be treated as such. Peeking behind a void pointer that
> >>>> is not designed for this purpose is not a good idea, even if
> >>>> technically you
> >> know what's in there.
> >>>
> >>> We can expose a clone interface, like MP_PEER_CLONE, so we don't
> >>> need to
> >> know what's inside, just need to know that it can be used on another
> thread?
> >>>
> >>
> >> Well, that can probably work. Feels like a hacky workaround though.
> >>
> >> Another way to do the same thing would be to store peer information
> >> right in the message, as opposed to providing it separately. Still a
> >> hack though, and will require far more changes, but it could be a
> >> better solution as (if done right) it would allow identifying which reply came
> from which peer.
> >>
> >> Of course, an even better approach would be to devise some kind of
> >> addressing scheme (uuid?), so that peer addresses are no longer
> >> opaque pointers but rather are valid data types.
> >>
> >> Thoughts?
> >
> > I may not give insight comment from the IPC implementation, but from
> user's view, what required is a unique token,
> >   it can be used for reply at anywhere and at any time, to me, currently
> implementation looks like missing some mapping management between an
> abstract token to its real data.
> >
> 
> Well, the idea was to not provide that :) We wanted to keep it simple and
> actively discourage any attempts to know which peer you're communicating
> with, as allowing that would imply some kind of addressing scheme, peer
> discovery and things like that, which we don't want to bother with. If you want
> replies, you use callbacks, which provide you with a peer address.
> 
> On the one hand, i understand the frustration of being forced to deal with
> deliberate simplicity of IPC API and to create workarounds like doing sendmsg()
> and storing requests that you're waiting on inside the application and
> launching interrupt threads, all for things that should Just Work in an IPC API.
> Believe me, i know about its deficiencies more than most people :)
> 
> On the other hand, we don't want to introduce hacks to solve a specific
> problem. If we are to solve this in IPC, we should do it properly. I don't think
> adding a workaround with "cloning" peer address is the solution. A proper
> solution would've been to implement addressing scheme, so that any
> response could be submitted at any time.
> 
> The hard route would be to add peer discovery, etc. The easy route would be
> to just change the peer address to something typed, something that we can
> use to reconstruct the original peer id. A great candidate for this is the uuid,
> but it's an external dependency. A workaround for this would be to just use a
> uint64 value obtained through some magic that is part of socket path. For
> example, replace "/var/run/dpdk/rte/mp_socket_pid_rdtsc" with something
> just as (likely) unique, but something that can be put into a uint64.
> 
> Without significantly changing the API and the internals, i think this is the best
> we will be able to do.

Ok, let me summarize this.

Current sync IPC does not support reply to a sync request outside the callback function, 
Of cause it will not support reply on separate thread which required by mp hotplug implementation.

So, the question is do we agree that IPC should support a more relax sync reply? 
For my option, I think the requirement is reasonable, because it greatly simplified the implementation of the case when the receiver need to do some task on a separate thread and reply on that thread.(note, this is typical case since the IPC deadlock limitation)

If the answer is yes:

	The question is will we able to enable this in this release?
	For me I think I may not able to solve this quickly, I need more time to review the implementation detail of IPC , not sure if you can cover this on your new patchset for IPC. 
	if yes, I assume I don't need to change any code (or a little bit) of current mp implementation, 	
	because "peer"(or whatever) as an identity of reply target can be parsed to other thread

	If not, we can take workaround for mp hotplug, 
		since it is workaround I prefer the simplest way with less impact and enough comments, since we will fix this anyway.
		Yes, clone is the simplest way :), I can just implement it inside ethdev_mp.c and add comment say "Fix me, this is hack"., so it will not impact other IPC users.

If the answer is NO
	I have to give up all sync IPC, and use sendmsg to replace sync request and to match request / reply in ethdev_mp.c , though it looks like should be a part of IPC and this looks like we are going to enable another set of sync IPC which support reply on another thread.

Thanks
Qi
> 
> > Thanks
> > Qi.
> >
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port in local process

[Zhang, Qi Z]

> -----Original Message-----
> From: Matan Azrad [mailto:matan@mellanox.com]
> Sent: Wednesday, June 27, 2018 12:55 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to release port
> in local process
> 
> Hi Zhang
> 
>  From: Zhang, Qi Z
> > Sent: Tuesday, June 26, 2018 4:30 PM
> > To: Matan Azrad <matan@mellanox.com>; Thomas Monjalon
> > <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > release port in local process
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z
> > > Sent: Tuesday, June 26, 2018 9:29 PM
> > > To: 'Matan Azrad' <matan@mellanox.com>; Thomas Monjalon
> > > <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> > > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > > dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit,
> > > Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > > <narender.vangati@intel.com>
> > > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > > release port in local process
> > >
> > > Hi Matan:
> > >
> > > > -----Original Message-----
> > > > From: Matan Azrad [mailto:matan@mellanox.com]
> > > > Sent: Tuesday, June 26, 2018 7:50 PM
> > > > To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> > > > <thomas@monjalon.net>; Burakov, Anatoly
> > > > <anatoly.burakov@intel.com>
> > > > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > > > dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > > > Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > > > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > > > <narender.vangati@intel.com>
> > > > Subject: RE: [dpdk-dev] [PATCH v4 03/24] ethdev: add function to
> > > > release port in local process
> > > >
> > > > Hi Qi
> > > >
> > > > Please see comments\questions..
> > > >
> > > > From: Qi Zhang
> > > > > Add driver API rte_eth_release_port_private to support the
> > > > > requirement that an ethdev only be released on secondary
> > > > > process, so only local state be set to unused , share data will
> > > > > not be reset so primary process can
> > > > still use it.
> > > > >
> > > > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > > > ---
> > > > >
> > > > >  lib/librte_ethdev/rte_ethdev.c        | 24
> +++++++++++++++++++++---
> > > > >  lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
> > > > >  2 files changed, 34 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > > > > b/lib/librte_ethdev/rte_ethdev.c index a9977df97..205b2ee33
> > > > > 100644
> > > > > --- a/lib/librte_ethdev/rte_ethdev.c
> > > > > +++ b/lib/librte_ethdev/rte_ethdev.c
> > > > > @@ -359,6 +359,23 @@ rte_eth_dev_attach_secondary(const char
> > > > *name)  }
> > > > >
> > > > >  int
> > > > > +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev) {
> > > > > +	if (eth_dev == NULL)
> > > > > +		return -EINVAL;
> > > > > +
> > > > > +	_rte_eth_dev_callback_process(eth_dev,
> RTE_ETH_EVENT_DESTROY,
> > > > > NULL);
> > > > > +
> > > > > +	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> > > >
> > > > I don't think you need the lock here because there is not shared
> > > > data to
> > > reset.
> > >
> > > OK, will remove this.
> > >
> > > >
> > > > > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > > +
> > > > > +	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > > > > +
> > > > > +	return 0;
> > > > > +}
> > > > > +
> > > > > +int
> > > > >  rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)  {
> > > > >  	if (eth_dev == NULL)
> > > > > @@ -370,9 +387,10 @@ rte_eth_dev_release_port(struct rte_eth_dev
> > > > > *eth_dev)
> > > > >
> > > > >  	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
> > > > >
> > > > > -	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > > -
> > > > > -	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > > > +	if (eth_dev->state != RTE_ETH_DEV_UNUSED) {
> > > >
> > > > Can you explain why not to release the shared data in case of
> > > > locally unused port?
> > >
> > > Now, we have rte_eth_dev_release_port be called in
> > > rte_eth_dev_detach, its redundant for some driver, because they
> > > already will release port in
> > > dev->remove, but I'm not sure if it is true for all drivers.
> 
> rte_eth_dev_release_port() like detach port should be called only by the port
> owner, by default is the application, so it should not be called by the PMD,
> maybe need to fix PMDs which does it.

Agree, release_port is better only be handled by ethdev layer
But Just did a "git grep", seems rte_eth_dev_release_port is used by most PMDs :)
I don't know much about the background, not sure if there is some other reason
we need this in PMD.

There could be separate task to cleanup this.

> 
> > >
> > > On secondary process, when detach a device, it is possible that
> > > rte_eth_dev_release_port be called after
> > > rte_eth_dev_release_port_local , so it will reset share data which
> > > is not expected, since the primary process will fail on detach it
> > > because
> > rte_eth_dev_allocated will return NULL.
> > >
> > > There could be other way, but current implementation help to reuse
> > > exist code., I can't add more comment on this change in v5
> >
> > I can add more comment
> 
> This solution is suffering from much more complexity, The port id should be
> the same in all the processes, so you can pass it by the mp channel and not use
> rte_eth_dev_allocated.
> 
> I see it like this:
> 
> The initiator process asks from all the other processes to detach a port by port
> id, all the other processes using rte_eth_dev_release_port_local and sending
> ack back to the initiator, Then, the initiator using the non-local release to
> release the port.

yes, that will be a solution, but if we are able to figure out which release_port API should be
called in rte_eth_dev_detach scenario, we do don't need more IPC here.
I will add some condition check in rte_eth_dev_detach to make sure correct release port API be invoked so there will not be the case that rte_eth_dev_release_port will not be invoked after rte_eth_dev_release_port_private, And previous concerned change can be removed.

Thanks
Qi

> 
> Am I missing something?
> 
> 
> > >
> > > Regards
> > > Qi
> > >
> > > >
> > > > > +		eth_dev->state = RTE_ETH_DEV_UNUSED;
> > > > > +		memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> > > > > +	}
> > > > >
> > > > >  	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
> > > > >
> > > > > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > > > > b/lib/librte_ethdev/rte_ethdev_driver.h
> > > > > index c9c825e3f..49c27223d 100644
> > > > > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > > > > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > > > > @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct
> > > > > rte_eth_dev *eth_dev);
> > > > >
> > > > >  /**
> > > > >   * @internal
> > > > > + * Release the specified ethdev port in local process, only set
> > > > > +to ethdev
> > > > > + * state to unused, but not reset share data since it assume
> > > > > +other process
> > > > > + * is still using it, typically it is called by secondary process.
> > > > > + *
> > > > > + * @param eth_dev
> > > > > + * The *eth_dev* pointer is the address of the *rte_eth_dev*
> structure.
> > > > > + * @return
> > > > > + *   - 0 on success, negative on error
> > > > > + */
> > > > > +int rte_eth_dev_release_port_private(struct rte_eth_dev
> > > > > +*eth_dev);
> > > > > +
> > > > > +/**
> > > > > + * @internal
> > > > >   * Release device queues and clear its configuration to force the user
> > > > >   * application to reconfigure it. It is for internal use only.
> > > > >   *
> > > > > --
> > > > > 2.13.6

[dpdk-dev] [PATCH v5 00/24] enable hotplug on multi-process

[Qi Zhang]

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (24):
  eal: introduce one device scan
  bus/vdev: enable one device scan
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  ethdev: support attach private device as first
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/e1000: enable port detach on secondary process
  net/igb: enable port detach on secondary process
  net/fm10k: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/failsafe: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample

 drivers/bus/vdev/vdev.c                      |  30 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/e1000/em_ethdev.c                |   9 +
 drivers/net/e1000/igb_ethdev.c               |   9 +
 drivers/net/failsafe/failsafe.c              |  16 ++
 drivers/net/fm10k/fm10k_ethdev.c             |   9 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/i40e/i40e_ethdev_vf.c            |   9 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |  12 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 ++
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_dev.c       |  17 +-
 lib/librte_eal/common/eal_common_proc.c      |  51 +++-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_bus.h      |  16 ++
 lib/librte_eal/common/include/rte_eal.h      |  34 +++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 +++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 415 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 +++
 lib/librte_ethdev/ethdev_private.h           |  42 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 312 ++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 39 files changed, 1911 insertions(+), 41 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v5 01/24] eal: introduce one device scan

[Qi Zhang]

When hot plug a new device, it is not necessary to scan everything
on the bus since the devname and devargs are already there. So new
rte_bus ops "scan_one" is introduced, bus driver can implement this
function to simplify the hotplug process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c  | 17 +++++++++++++----
 lib/librte_eal/common/include/rte_bus.h | 16 ++++++++++++++++
 2 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 61cb3b162..1ad033536 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -147,11 +147,20 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	if (ret)
 		goto err_devarg;
 
-	ret = bus->scan();
-	if (ret)
-		goto err_devarg;
+	/**
+	 * if bus support to scan specific device by devargs,
+	 * we don't need to scan all devices on the bus.
+	 */
+	if (bus->scan_one) {
+		dev = bus->scan_one(da);
+	} else {
+		ret = bus->scan();
+		if (ret)
+			goto err_devarg;
+
+		dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
+	}
 
-	dev = bus->find_device(NULL, cmp_detached_dev_name, devname);
 	if (dev == NULL) {
 		RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
 			devname);
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..3269ef78b 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -84,6 +84,21 @@ enum rte_iova_mode {
 typedef int (*rte_bus_scan_t)(void);
 
 /**
+ * Bus specific scan for one specific device attached on the bus.
+ * For each bus object, the scan would be responsible for finding the specific
+ * device and adding it to its private device list, and the device object will
+ * be return also.
+ *
+ * @param devargs
+ *	Device arguments be used to identify the device.
+ *
+ * @return
+ *	!NULL for successful scan
+ *	NULL for unsuccessful scan
+ */
+typedef struct rte_device *(*rte_bus_scan_one_t)(struct rte_devargs *devargs);
+
+/**
  * Implementation specific probe function which is responsible for linking
  * devices on that bus with applicable drivers.
  *
@@ -204,6 +219,7 @@ struct rte_bus {
 	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
 	const char *name;            /**< Name of the bus */
 	rte_bus_scan_t scan;         /**< Scan for devices attached to bus */
+	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs */
 	rte_bus_probe_t probe;       /**< Probe devices on bus */
 	rte_bus_find_device_t find_device; /**< Find a device on the bus */
 	rte_bus_plug_t plug;         /**< Probe single device for drivers */
-- 
2.13.6

[dpdk-dev] [PATCH v5 02/24] bus/vdev: enable one device scan

[Qi Zhang]

The patch implements the ops scan_one for the vdev bus, which gives two
benefits:
1. Improves scan efficiency when a device is attached as hotplug, since
there is no need to populate a new device by iterating all devargs in
devargs_list
2. It also avoids sync IPC invoke (which happens in vdev->scan on
secondary process). The benefit is this removes the potential deadlock
in the case when a secondary process receives a request from primary
process to attach a new device, since vdev->scan will be invoked on mp
thread itself in that case

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 drivers/bus/vdev/vdev.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/drivers/bus/vdev/vdev.c b/drivers/bus/vdev/vdev.c
index 6139dd551..cdbd77df0 100644
--- a/drivers/bus/vdev/vdev.c
+++ b/drivers/bus/vdev/vdev.c
@@ -467,6 +467,35 @@ vdev_scan(void)
 	return 0;
 }
 
+static struct rte_device *vdev_scan_one(struct rte_devargs *devargs)
+{
+	struct rte_vdev_device *dev = NULL;
+
+	dev = calloc(1, sizeof(*dev));
+	if (!dev) {
+		VDEV_LOG(ERR, "failed to allocate memory for new device");
+		return NULL;
+	}
+
+	rte_spinlock_recursive_lock(&vdev_device_list_lock);
+
+	if (find_vdev(devargs->name)) {
+		VDEV_LOG(ERR, "device %s already exist", devargs->name);
+		free(dev);
+		rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+		return NULL;
+	}
+
+	dev->device.devargs = devargs;
+	dev->device.numa_node = SOCKET_ID_ANY;
+	dev->device.name = devargs->name;
+	TAILQ_INSERT_TAIL(&vdev_device_list, dev, next);
+
+	rte_spinlock_recursive_unlock(&vdev_device_list_lock);
+
+	return &dev->device;
+}
+
 static int
 vdev_probe(void)
 {
@@ -531,6 +560,7 @@ vdev_unplug(struct rte_device *dev)
 
 static struct rte_bus rte_vdev_bus = {
 	.scan = vdev_scan,
+	.scan_one = vdev_scan_one,
 	.probe = vdev_probe,
 	.find_device = vdev_find_device,
 	.plug = vdev_plug,
-- 
2.13.6

[dpdk-dev] [PATCH v5 03/24] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..2353fc921 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
 	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
 	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
-- 
2.13.6

[dpdk-dev] [PATCH v5 04/24] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  5 ++++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index 707d8ab30..fc0eb4d17 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
@@ -686,7 +722,20 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	return process_mp_init_callbacks();
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 05/24] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 257 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  41 ++++++
 lib/librte_ethdev/ethdev_private.h  |  39 ++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 210 ++++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h      |  45 +++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 8 files changed, 582 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..86daff934
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,257 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+#include <rte_string_fns.h>
+#include <rte_alarm.h>
+
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 2353fc921..6c5f465a2 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int solid_release;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+	/**
+	 * A private device on secondary need
+	 * rte_eth_dev_release_port.
+	 * 1) only vdev support private device.
+	 * 2) private device has no empty devargs.
+	 */
+		if (!strcmp(bus->name, "vdev") &&
+			dev->devargs != NULL &&
+			strlen(dev->devargs->args) > 0)
+			solid_release = 1;
+		else
+			solid_release = 0;
+	} else {
+		solid_release = 0;
+	}
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	if (solid_release)
+		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	else
+		return rte_eth_dev_release_port_private(
+			&rte_eth_devices[port_id]);
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v5 06/24] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 140 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 86daff934..0d9fcc77a 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -6,6 +6,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -108,7 +109,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 6c5f465a2..54d3d2369 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -734,6 +735,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	if (solid_release)
 		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	else
@@ -802,7 +804,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -844,6 +845,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -890,6 +895,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -903,6 +909,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4705,6 +4715,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v5 07/24] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 167 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 161 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0d9fcc77a..0a93fa669 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,8 +5,44 @@
 #include <rte_alarm.h>
 
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -84,11 +120,110 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req.port_id = port_id;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+		if (!ret) {
+			if (!tmp_req.result) {
+				tmp_req.t = REQ_TYPE_DETACH;
+				ret = eth_dev_request_to_secondary(&tmp_req);
+				if (!ret)
+					ret = do_eth_dev_detach(req->port_id);
+			} else {
+				ret = tmp_req.result;
+			}
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -179,8 +314,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v5 08/24] ethdev: support attach private device as first

[Qi Zhang]

When attach a private device from secondary as the first one, we need
to make sure rte_eth_dev_shared_data is initialized, the patch add
necessary IPC for secondary to inform primary to do initialization.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c      |  2 ++
 lib/librte_ethdev/ethdev_mp.h      |  1 +
 lib/librte_ethdev/ethdev_private.h |  3 +++
 lib/librte_ethdev/rte_ethdev.c     | 31 ++++++++++++++++++++-----------
 4 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0a93fa669..f2982c790 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -182,6 +182,8 @@ __handle_secondary_request(void *param)
 				ret = tmp_req.result;
 			}
 		}
+	} else if (req->t == REQ_TYPE_SHARE_DATA_PREPARE) {
+		eth_dev_shared_data_prepare();
 	} else {
 		ethdev_log(ERR, "unsupported secondary to primary request\n");
 		ret = -ENOTSUP;
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..61fc381da 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -15,6 +15,7 @@ enum eth_dev_req_type {
 	REQ_TYPE_PRE_DETACH,
 	REQ_TYPE_DETACH,
 	REQ_TYPE_ATTACH_ROLLBACK,
+	REQ_TYPE_SHARE_DATA_PREPARE,
 };
 
 struct eth_dev_mp_req {
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
index 981e7de8a..005d63afc 100644
--- a/lib/librte_ethdev/ethdev_private.h
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -36,4 +36,7 @@ int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
  */
 int do_eth_dev_detach(uint16_t port_id);
 
+/* Prepare shared data for multi-process */
+void eth_dev_shared_data_prepare(void);
+
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 54d3d2369..c0f68fe09 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -199,11 +199,14 @@ rte_eth_find_next(uint16_t port_id)
 	return port_id;
 }
 
-static void
-rte_eth_dev_shared_data_prepare(void)
+void
+eth_dev_shared_data_prepare(void)
 {
 	const unsigned flags = 0;
 	const struct rte_memzone *mz;
+	struct eth_dev_mp_req req;
+
+	memset(&req, 0, sizeof(req));
 
 	rte_spinlock_lock(&rte_eth_shared_data_lock);
 
@@ -215,6 +218,12 @@ rte_eth_dev_shared_data_prepare(void)
 					rte_socket_id(), flags);
 		} else
 			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		/* if secondary attach a private device first */
+		if (mz == NULL && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			req.t = REQ_TYPE_SHARE_DATA_PREPARE;
+			eth_dev_request_to_primary(&req);
+			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		}
 		if (mz == NULL)
 			rte_panic("Cannot allocate ethdev shared data\n");
 
@@ -255,7 +264,7 @@ rte_eth_dev_allocated(const char *name)
 {
 	struct rte_eth_dev *ethdev;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -300,7 +309,7 @@ rte_eth_dev_allocate(const char *name)
 	uint16_t port_id;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port creation between primary and secondary threads. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -339,7 +348,7 @@ rte_eth_dev_attach_secondary(const char *name)
 	uint16_t i;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port attachment to primary port creation and release. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -379,7 +388,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (eth_dev == NULL)
 		return -EINVAL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
 
@@ -432,7 +441,7 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id)
 int __rte_experimental
 rte_eth_dev_owner_new(uint64_t *owner_id)
 {
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -487,7 +496,7 @@ rte_eth_dev_owner_set(const uint16_t port_id,
 {
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -504,7 +513,7 @@ rte_eth_dev_owner_unset(const uint16_t port_id, const uint64_t owner_id)
 			{.id = RTE_ETH_DEV_NO_OWNER, .name = ""};
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -519,7 +528,7 @@ rte_eth_dev_owner_delete(const uint64_t owner_id)
 {
 	uint16_t port_id;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -541,7 +550,7 @@ rte_eth_dev_owner_get(const uint16_t port_id, struct rte_eth_dev_owner *owner)
 	int ret = 0;
 	struct rte_eth_dev *ethdev = &rte_eth_devices[port_id];
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 09/24] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c    | 2 ++
 drivers/net/i40e/i40e_ethdev_vf.c | 9 +++++++++
 2 files changed, 11 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
diff --git a/drivers/net/i40e/i40e_ethdev_vf.c b/drivers/net/i40e/i40e_ethdev_vf.c
index 804e44530..fc6f079d5 100644
--- a/drivers/net/i40e/i40e_ethdev_vf.c
+++ b/drivers/net/i40e/i40e_ethdev_vf.c
@@ -1500,6 +1500,15 @@ static int eth_i40evf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_i40evf_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+		ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, i40evf_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 10/24] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..f9d560835 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
@@ -1809,6 +1812,15 @@ static struct rte_pci_driver rte_ixgbe_pmd = {
 static int eth_ixgbevf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev;
+
+	ethdev = rte_eth_dev_allocated(pci_dev->device.name);
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_probe(pci_dev,
 		sizeof(struct ixgbe_adapter), eth_ixgbevf_dev_init);
 }
-- 
2.13.6

[dpdk-dev] [PATCH v5 11/24] net/e1000: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/em_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/em_ethdev.c b/drivers/net/e1000/em_ethdev.c
index 7039dc100..e6b7ce63a 100644
--- a/drivers/net/e1000/em_ethdev.c
+++ b/drivers/net/e1000/em_ethdev.c
@@ -349,6 +349,15 @@ static int eth_em_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_em_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_em_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 12/24] net/igb: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/e1000/igb_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/e1000/igb_ethdev.c b/drivers/net/e1000/igb_ethdev.c
index edc7be319..db07a83e3 100644
--- a/drivers/net/e1000/igb_ethdev.c
+++ b/drivers/net/e1000/igb_ethdev.c
@@ -1089,6 +1089,15 @@ static int eth_igb_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_igb_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_igb_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 13/24] net/fm10k: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 3ff1b0e0f..f73301182 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -3264,6 +3264,15 @@ static int eth_fm10k_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int eth_fm10k_pci_remove(struct rte_pci_device *pci_dev)
 {
+	struct rte_eth_dev *ethdev =
+		rte_eth_dev_allocated(pci_dev->device.name);
+
+	if (!ethdev)
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	return rte_eth_dev_pci_generic_remove(pci_dev, eth_fm10k_dev_uninit);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 14/24] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v5 15/24] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v5 16/24] net/failsafe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c
index eafbb75df..c5e8651f6 100644
--- a/drivers/net/failsafe/failsafe.c
+++ b/drivers/net/failsafe/failsafe.c
@@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &failsafe_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
 static int
 rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)
 {
+	struct rte_eth_dev *eth_dev;
 	const char *name;
 
 	name = rte_vdev_device_name(vdev);
 	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
+
+	eth_dev = rte_eth_dev_allocated(name);
+	if (!eth_dev)
+		return -ENODEV;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario.
+		 */
+	}
+
 	return fs_rte_eth_free(name);
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 17/24] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v5 18/24] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v5 19/24] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 20/24] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v5 21/24] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v5 22/24] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v5 23/24] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v5 24/24] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v5 23/24] net/vhost: enable port detach on secondary process

[Maxime Coquelin]

On 06/27/2018 09:17 AM, Qi Zhang wrote:
> Previously, detach port on secondary process will mess primary
> process and cause same device can't be attached again, by take
> advantage of rte_eth_release_port_private, we can support this
> with minor change.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
> index ba9d768a0..f773711b4 100644
> --- a/drivers/net/vhost/rte_eth_vhost.c
> +++ b/drivers/net/vhost/rte_eth_vhost.c
> @@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
>   		}
>   		/* TODO: request info from primary to set up Rx and Tx */
>   		eth_dev->dev_ops = &ops;
> +		eth_dev->device = &dev->device;
>   		rte_eth_dev_probing_finish(eth_dev);
>   		return 0;
>   	}
> @@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
>   	if (eth_dev == NULL)
>   		return -ENODEV;
>   
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/* detach device on local pprocess only */
> +		if (strlen(rte_vdev_device_args(dev)) == 0)
> +			return rte_eth_dev_release_port_private(eth_dev);
> +		/**
> +		 * else this is a private device for current process
> +		 * so continue with normal detach scenario
> +		 */
> +	}
> +

IMHO, the driver shouldn't have to manage primary/secondary process
stuff. It does not look like the right place to me.

>   	eth_dev_close(eth_dev);
>   
>   	rte_free(vring_states[eth_dev->data->port_id]);
> 

Re: [dpdk-dev] [PATCH v3 15/23] net/failsafe: enable port detach on secondary process

[Matan Azrad]

Hi Qi

From: Qi Zhang
> Previously, detach port on secondary process will mess primary process and
> cause same device can't be attached again, by take advantage of
> rte_eth_release_port_private, we can support this with minor change.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>

Failsafe doesn't support secondary process,
I don't think we need this adjustment for drivers which don't support secondary process.
 
> ---
>  drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/net/failsafe/failsafe.c b/drivers/net/failsafe/failsafe.c index
> eafbb75df..c5e8651f6 100644
> --- a/drivers/net/failsafe/failsafe.c
> +++ b/drivers/net/failsafe/failsafe.c
> @@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device *vdev)
>  		}
>  		/* TODO: request info from primary to set up Rx and Tx */
>  		eth_dev->dev_ops = &failsafe_ops;
> +		eth_dev->device = &vdev->device;
>  		rte_eth_dev_probing_finish(eth_dev);
>  		return 0;
>  	}
> @@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device
> *vdev)  static int  rte_pmd_failsafe_remove(struct rte_vdev_device *vdev)  {
> +	struct rte_eth_dev *eth_dev;
>  	const char *name;
> 
>  	name = rte_vdev_device_name(vdev);
>  	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
> +
> +	eth_dev = rte_eth_dev_allocated(name);
> +	if (!eth_dev)
> +		return -ENODEV;
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/* detach device on local pprocess only */
> +		if (strlen(rte_vdev_device_args(vdev)) == 0)
> +			return rte_eth_dev_release_port_private(eth_dev);
> +		/**
> +		 * else this is a private device for current process
> +		 * so continue with normal detach scenario.
> +		 */
> +	}
> +
>  	return fs_rte_eth_free(name);
>  }
> 
> --
> 2.13.6

Re: [dpdk-dev] [PATCH v3 15/23] net/failsafe: enable port detach on secondary process

[Zhang, Qi Z]

> -----Original Message-----
> From: Matan Azrad [mailto:matan@mellanox.com]
> Sent: Wednesday, June 27, 2018 4:29 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [dpdk-dev] [PATCH v3 15/23] net/failsafe: enable port detach on
> secondary process
> 
> Hi Qi
> 
> From: Qi Zhang
> > Previously, detach port on secondary process will mess primary process
> > and cause same device can't be attached again, by take advantage of
> > rte_eth_release_port_private, we can support this with minor change.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> 
> Failsafe doesn't support secondary process, I don't think we need this
> adjustment for drivers which don't support secondary process.

Sure, I can remove this, thanks for the information.

Regards
Qi

> 
> > ---
> >  drivers/net/failsafe/failsafe.c | 16 ++++++++++++++++
> >  1 file changed, 16 insertions(+)
> >
> > diff --git a/drivers/net/failsafe/failsafe.c
> > b/drivers/net/failsafe/failsafe.c index
> > eafbb75df..c5e8651f6 100644
> > --- a/drivers/net/failsafe/failsafe.c
> > +++ b/drivers/net/failsafe/failsafe.c
> > @@ -328,6 +328,7 @@ rte_pmd_failsafe_probe(struct rte_vdev_device
> *vdev)
> >  		}
> >  		/* TODO: request info from primary to set up Rx and Tx */
> >  		eth_dev->dev_ops = &failsafe_ops;
> > +		eth_dev->device = &vdev->device;
> >  		rte_eth_dev_probing_finish(eth_dev);
> >  		return 0;
> >  	}
> > @@ -338,10 +339,25 @@ rte_pmd_failsafe_probe(struct rte_vdev_device
> > *vdev)  static int  rte_pmd_failsafe_remove(struct rte_vdev_device
> > *vdev)  {
> > +	struct rte_eth_dev *eth_dev;
> >  	const char *name;
> >
> >  	name = rte_vdev_device_name(vdev);
> >  	INFO("Uninitializing " FAILSAFE_DRIVER_NAME " for %s", name);
> > +
> > +	eth_dev = rte_eth_dev_allocated(name);
> > +	if (!eth_dev)
> > +		return -ENODEV;
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > +		/* detach device on local pprocess only */
> > +		if (strlen(rte_vdev_device_args(vdev)) == 0)
> > +			return rte_eth_dev_release_port_private(eth_dev);
> > +		/**
> > +		 * else this is a private device for current process
> > +		 * so continue with normal detach scenario.
> > +		 */
> > +	}
> > +
> >  	return fs_rte_eth_free(name);
> >  }
> >
> > --
> > 2.13.6

Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan

[Zhang, Qi Z]

> -----Original Message-----
> From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com]
> Sent: Wednesday, June 27, 2018 12:34 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; thomas@monjalon.net;
> Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v4 01/24] eal: introduce one device scan
> 
> On Tue, Jun 26, 2018 at 12:26:05PM +0000, Zhang, Qi Z wrote:
> >
> >
> > > -----Original Message-----
> > > From: Burakov, Anatoly
> > > Sent: Tuesday, June 26, 2018 7:48 PM
> > > To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> > > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > > dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit,
> > > Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > > <narender.vangati@intel.com>
> > > Subject: Re: [PATCH v4 01/24] eal: introduce one device scan
> > >
> > > On 26-Jun-18 8:08 AM, Qi Zhang wrote:
> > > > When hot plug a new device, it is not necessary to scan everything
> > > > on the bus since the devname and devargs are already there. So new
> > > > rte_bus ops "scan_one" is introduced, bus driver can implement
> > > > this function to simplify the hotplug process.
> > > >
> > > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > > ---
> > > >
> > >
> > > <snip>
> > >
> > > > + *	NULL for unsuccessful scan
> > > > + */
> > > > +typedef struct rte_device *(*rte_bus_scan_one_t)(struct
> > > > +rte_devargs *devargs);
> > > > +
> > > > +/**
> > > >    * Implementation specific probe function which is responsible for
> linking
> > > >    * devices on that bus with applicable drivers.
> > > >    *
> > > > @@ -204,6 +219,7 @@ struct rte_bus {
> > > >   	TAILQ_ENTRY(rte_bus) next;   /**< Next bus object in linked list */
> > > >   	const char *name;            /**< Name of the bus */
> > > >   	rte_bus_scan_t scan;         /**< Scan for devices attached to
> > > bus */
> > > > +	rte_bus_scan_one_t scan_one; /**< Scan one device using devargs
> > > > +*/
> > > >   	rte_bus_probe_t probe;       /**< Probe devices on bus */
> > > >   	rte_bus_find_device_t find_device; /**< Find a device on the bus */
> > > >   	rte_bus_plug_t plug;         /**< Probe single device for drivers
> > > */
> > > >
> > >
> > > Does changing this structure break ABI for bus drivers?
> >
> > For bus driver, I think yes, but I'm not sure what I should do for
> > this, since this is not for application
> >
> >
> 
> This should be appropriately announced in advance, in general.
> However, it seems there is some leeway if the new field will not move the
> others and not make the structure grow (i.e. replace a padding).
> 
> There is an ABI check script that can be used.
> 
> This however breaks the bus ABI, which breaks the EAL ABI.
> This is usually an issue.


OK, since we are able to invoke IPC request in a separate thread and also with below
fix, there is no issue on the vdev->scan during hotplug on secondary.
https://patches.dpdk.org/patch/41647/

ABI break is not necessary, I will withdraw patch 1 and 2. 

Thanks
Qi


> 
> More generally, I was in favor of changing the whole bus scan process to a
> per-device iteration. I was shut down on this when adding hotplug.
> As a result, bus->scan() process was made to require the operation to be
> idempotent.
> 
> Adding a new ops adds noise to the bus API. It should be kept as clean as
> possible. This new one seems unnecessary, now that all bus scans are
> idempotent (when supporting hotplug).
> 
> Regards,
> --
> Gaëtan Rivet
> 6WIND

[dpdk-dev] [PATCH v5 00/24] enable hotplug on multi-process

[Qi Zhang]

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  ethdev: support attach private device as first
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst       |  21 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |   3 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 ++
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_proc.c      |  51 +++-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_eal.h      |  34 +++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 +++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 415 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 +++
 lib/librte_ethdev/ethdev_private.h           |  42 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 312 ++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 lib/librte_ethdev/rte_ethdev_pci.h           |   3 +
 33 files changed, 1815 insertions(+), 37 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v6 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..2353fc921 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
 	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
 	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..eeb944146 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,9 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v6 02/19] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  5 ++++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index f010ef59e..de87925c8 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
@@ -686,7 +722,20 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	return process_mp_init_callbacks();
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 03/19] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 257 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  41 ++++++
 lib/librte_ethdev/ethdev_private.h  |  39 ++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 210 ++++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h      |  45 +++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 8 files changed, 582 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..86daff934
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,257 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+#include <rte_string_fns.h>
+#include <rte_alarm.h>
+
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 2353fc921..6c5f465a2 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int solid_release;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+	/**
+	 * A private device on secondary need
+	 * rte_eth_dev_release_port.
+	 * 1) only vdev support private device.
+	 * 2) private device has no empty devargs.
+	 */
+		if (!strcmp(bus->name, "vdev") &&
+			dev->devargs != NULL &&
+			strlen(dev->devargs->args) > 0)
+			solid_release = 1;
+		else
+			solid_release = 0;
+	} else {
+		solid_release = 0;
+	}
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	if (solid_release)
+		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	else
+		return rte_eth_dev_release_port_private(
+			&rte_eth_devices[port_id]);
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v6 04/19] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 140 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 86daff934..0d9fcc77a 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -6,6 +6,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -108,7 +109,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 6c5f465a2..54d3d2369 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -734,6 +735,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	if (solid_release)
 		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	else
@@ -802,7 +804,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -844,6 +845,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -890,6 +895,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -903,6 +909,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4705,6 +4715,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v6 05/19] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 167 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 161 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0d9fcc77a..0a93fa669 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,8 +5,44 @@
 #include <rte_alarm.h>
 
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -84,11 +120,110 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req.port_id = port_id;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+		if (!ret) {
+			if (!tmp_req.result) {
+				tmp_req.t = REQ_TYPE_DETACH;
+				ret = eth_dev_request_to_secondary(&tmp_req);
+				if (!ret)
+					ret = do_eth_dev_detach(req->port_id);
+			} else {
+				ret = tmp_req.result;
+			}
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -179,8 +314,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Qi Zhang]

When attach a private device from secondary as the first one, we need
to make sure rte_eth_dev_shared_data is initialized, the patch add
necessary IPC for secondary to inform primary to do initialization.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c      |  2 ++
 lib/librte_ethdev/ethdev_mp.h      |  1 +
 lib/librte_ethdev/ethdev_private.h |  3 +++
 lib/librte_ethdev/rte_ethdev.c     | 31 ++++++++++++++++++++-----------
 4 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0a93fa669..f2982c790 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -182,6 +182,8 @@ __handle_secondary_request(void *param)
 				ret = tmp_req.result;
 			}
 		}
+	} else if (req->t == REQ_TYPE_SHARE_DATA_PREPARE) {
+		eth_dev_shared_data_prepare();
 	} else {
 		ethdev_log(ERR, "unsupported secondary to primary request\n");
 		ret = -ENOTSUP;
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..61fc381da 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -15,6 +15,7 @@ enum eth_dev_req_type {
 	REQ_TYPE_PRE_DETACH,
 	REQ_TYPE_DETACH,
 	REQ_TYPE_ATTACH_ROLLBACK,
+	REQ_TYPE_SHARE_DATA_PREPARE,
 };
 
 struct eth_dev_mp_req {
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
index 981e7de8a..005d63afc 100644
--- a/lib/librte_ethdev/ethdev_private.h
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -36,4 +36,7 @@ int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
  */
 int do_eth_dev_detach(uint16_t port_id);
 
+/* Prepare shared data for multi-process */
+void eth_dev_shared_data_prepare(void);
+
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 54d3d2369..c0f68fe09 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -199,11 +199,14 @@ rte_eth_find_next(uint16_t port_id)
 	return port_id;
 }
 
-static void
-rte_eth_dev_shared_data_prepare(void)
+void
+eth_dev_shared_data_prepare(void)
 {
 	const unsigned flags = 0;
 	const struct rte_memzone *mz;
+	struct eth_dev_mp_req req;
+
+	memset(&req, 0, sizeof(req));
 
 	rte_spinlock_lock(&rte_eth_shared_data_lock);
 
@@ -215,6 +218,12 @@ rte_eth_dev_shared_data_prepare(void)
 					rte_socket_id(), flags);
 		} else
 			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		/* if secondary attach a private device first */
+		if (mz == NULL && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			req.t = REQ_TYPE_SHARE_DATA_PREPARE;
+			eth_dev_request_to_primary(&req);
+			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		}
 		if (mz == NULL)
 			rte_panic("Cannot allocate ethdev shared data\n");
 
@@ -255,7 +264,7 @@ rte_eth_dev_allocated(const char *name)
 {
 	struct rte_eth_dev *ethdev;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -300,7 +309,7 @@ rte_eth_dev_allocate(const char *name)
 	uint16_t port_id;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port creation between primary and secondary threads. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -339,7 +348,7 @@ rte_eth_dev_attach_secondary(const char *name)
 	uint16_t i;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port attachment to primary port creation and release. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -379,7 +388,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (eth_dev == NULL)
 		return -EINVAL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
 
@@ -432,7 +441,7 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id)
 int __rte_experimental
 rte_eth_dev_owner_new(uint64_t *owner_id)
 {
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -487,7 +496,7 @@ rte_eth_dev_owner_set(const uint16_t port_id,
 {
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -504,7 +513,7 @@ rte_eth_dev_owner_unset(const uint16_t port_id, const uint64_t owner_id)
 			{.id = RTE_ETH_DEV_NO_OWNER, .name = ""};
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -519,7 +528,7 @@ rte_eth_dev_owner_delete(const uint64_t owner_id)
 {
 	uint16_t port_id;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -541,7 +550,7 @@ rte_eth_dev_owner_get(const uint16_t port_id, struct rte_eth_dev_owner *owner)
 	int ret = 0;
 	struct rte_eth_dev *ethdev = &rte_eth_devices[port_id];
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 07/19] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v6 08/19] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v6 09/19] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v6 10/19] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v6 11/19] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v6 12/19] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v6 13/19] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 14/19] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v6 15/19] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v6 16/19] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 17/19] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v6 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v6 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..fc4736814 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,20 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support ethernet device hotplug for primary-secondary model.**
+
+  In primary-secondary process model, ethernet devices are regarded as shared
+  by default, attach or detach a device on any process will broadcast to
+  other processes through mp channel then device information will be
+  synchronzied on all processes. While secondary process can still attach
+  or detach a private device (vdev only) with specific API.
+
+* **Support ethernet device lock.**
+
+  An ethernet device can be directly or conditionally locked. A directly
+  locked device can't be detached, while when try to detach a conditionally
+  locked device a pre-registered callback will be invoked to perform condition
+  check and decide if it can be detached or not.
 
 API Changes
 -----------
@@ -60,6 +74,13 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is extended.
+
+  In primary-secondary process model, ``rte_eth_dev_attach`` will guarantee that
+  device be attached on all processes, if any process failed to attach, it will
+  rollback to orignal status. ``rte_eth_dev_detach`` also guarantee device be
+  detached on all processes, if device is locked by any process, it will roll
+  back to original status.
 
 ABI Changes
 -----------
-- 
2.13.6

[dpdk-dev] [PATCH v6 00/19] enable hotplug on multi-process

[Qi Zhang]

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  ethdev: support attach private device as first
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst       |  21 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |   3 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 ++
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_proc.c      |  51 +++-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_eal.h      |  34 +++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 +++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 415 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 +++
 lib/librte_ethdev/ethdev_private.h           |  42 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 312 ++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 lib/librte_ethdev/rte_ethdev_pci.h           |   3 +
 33 files changed, 1815 insertions(+), 37 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v6 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..2353fc921 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
 	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
 	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..eeb944146 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,9 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v6 02/19] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 51 ++++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  5 ++++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index f010ef59e..de87925c8 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,6 +619,42 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
@@ -686,7 +722,20 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	return process_mp_init_callbacks();
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 03/19] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile          |   1 +
 lib/librte_ethdev/ethdev_mp.c       | 257 ++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h       |  41 ++++++
 lib/librte_ethdev/ethdev_private.h  |  39 ++++++
 lib/librte_ethdev/meson.build       |   1 +
 lib/librte_ethdev/rte_ethdev.c      | 210 ++++++++++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev.h      |  45 +++++++
 lib/librte_ethdev/rte_ethdev_core.h |   5 +
 8 files changed, 582 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..86daff934
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,257 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+#include <rte_string_fns.h>
+#include <rte_alarm.h>
+
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 2353fc921..6c5f465a2 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int solid_release;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+	/**
+	 * A private device on secondary need
+	 * rte_eth_dev_release_port.
+	 * 1) only vdev support private device.
+	 * 2) private device has no empty devargs.
+	 */
+		if (!strcmp(bus->name, "vdev") &&
+			dev->devargs != NULL &&
+			strlen(dev->devargs->args) > 0)
+			solid_release = 1;
+		else
+			solid_release = 0;
+	} else {
+		solid_release = 0;
+	}
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	if (solid_release)
+		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	else
+		return rte_eth_dev_release_port_private(
+			&rte_eth_devices[port_id]);
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
-- 
2.13.6

[dpdk-dev] [PATCH v6 04/19] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile      |   1 +
 lib/librte_ethdev/ethdev_lock.c | 140 ++++++++++++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h |  31 +++++++++
 lib/librte_ethdev/ethdev_mp.c   |   3 +-
 lib/librte_ethdev/meson.build   |   1 +
 lib/librte_ethdev/rte_ethdev.c  |  60 ++++++++++++++++-
 lib/librte_ethdev/rte_ethdev.h  | 124 +++++++++++++++++++++++++++++++++++
 7 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 86daff934..0d9fcc77a 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -6,6 +6,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -108,7 +109,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 6c5f465a2..54d3d2369 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -734,6 +735,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	if (solid_release)
 		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	else
@@ -802,7 +804,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -844,6 +845,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -890,6 +895,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -903,6 +909,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4705,6 +4715,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
-- 
2.13.6

[dpdk-dev] [PATCH v6 05/19] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 167 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 161 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0d9fcc77a..0a93fa669 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,8 +5,44 @@
 #include <rte_alarm.h>
 
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -84,11 +120,110 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (!ret) {
+			tmp_req.port_id = port_id;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id))
+			ret = -EINVAL;
+		if (!ret)
+			ret = process_lock_callbacks(req->port_id);
+		if (!ret) {
+			tmp_req.t = REQ_TYPE_PRE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+		}
+		if (!ret) {
+			if (!tmp_req.result) {
+				tmp_req.t = REQ_TYPE_DETACH;
+				ret = eth_dev_request_to_secondary(&tmp_req);
+				if (!ret)
+					ret = do_eth_dev_detach(req->port_id);
+			} else {
+				ret = tmp_req.result;
+			}
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -179,8 +314,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Qi Zhang]

When attach a private device from secondary as the first one, we need
to make sure rte_eth_dev_shared_data is initialized, the patch add
necessary IPC for secondary to inform primary to do initialization.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c      |  2 ++
 lib/librte_ethdev/ethdev_mp.h      |  1 +
 lib/librte_ethdev/ethdev_private.h |  3 +++
 lib/librte_ethdev/rte_ethdev.c     | 31 ++++++++++++++++++++-----------
 4 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0a93fa669..f2982c790 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -182,6 +182,8 @@ __handle_secondary_request(void *param)
 				ret = tmp_req.result;
 			}
 		}
+	} else if (req->t == REQ_TYPE_SHARE_DATA_PREPARE) {
+		eth_dev_shared_data_prepare();
 	} else {
 		ethdev_log(ERR, "unsupported secondary to primary request\n");
 		ret = -ENOTSUP;
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..61fc381da 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -15,6 +15,7 @@ enum eth_dev_req_type {
 	REQ_TYPE_PRE_DETACH,
 	REQ_TYPE_DETACH,
 	REQ_TYPE_ATTACH_ROLLBACK,
+	REQ_TYPE_SHARE_DATA_PREPARE,
 };
 
 struct eth_dev_mp_req {
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
index 981e7de8a..005d63afc 100644
--- a/lib/librte_ethdev/ethdev_private.h
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -36,4 +36,7 @@ int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
  */
 int do_eth_dev_detach(uint16_t port_id);
 
+/* Prepare shared data for multi-process */
+void eth_dev_shared_data_prepare(void);
+
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 54d3d2369..c0f68fe09 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -199,11 +199,14 @@ rte_eth_find_next(uint16_t port_id)
 	return port_id;
 }
 
-static void
-rte_eth_dev_shared_data_prepare(void)
+void
+eth_dev_shared_data_prepare(void)
 {
 	const unsigned flags = 0;
 	const struct rte_memzone *mz;
+	struct eth_dev_mp_req req;
+
+	memset(&req, 0, sizeof(req));
 
 	rte_spinlock_lock(&rte_eth_shared_data_lock);
 
@@ -215,6 +218,12 @@ rte_eth_dev_shared_data_prepare(void)
 					rte_socket_id(), flags);
 		} else
 			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		/* if secondary attach a private device first */
+		if (mz == NULL && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			req.t = REQ_TYPE_SHARE_DATA_PREPARE;
+			eth_dev_request_to_primary(&req);
+			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		}
 		if (mz == NULL)
 			rte_panic("Cannot allocate ethdev shared data\n");
 
@@ -255,7 +264,7 @@ rte_eth_dev_allocated(const char *name)
 {
 	struct rte_eth_dev *ethdev;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -300,7 +309,7 @@ rte_eth_dev_allocate(const char *name)
 	uint16_t port_id;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port creation between primary and secondary threads. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -339,7 +348,7 @@ rte_eth_dev_attach_secondary(const char *name)
 	uint16_t i;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port attachment to primary port creation and release. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -379,7 +388,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (eth_dev == NULL)
 		return -EINVAL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
 
@@ -432,7 +441,7 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id)
 int __rte_experimental
 rte_eth_dev_owner_new(uint64_t *owner_id)
 {
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -487,7 +496,7 @@ rte_eth_dev_owner_set(const uint16_t port_id,
 {
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -504,7 +513,7 @@ rte_eth_dev_owner_unset(const uint16_t port_id, const uint64_t owner_id)
 			{.id = RTE_ETH_DEV_NO_OWNER, .name = ""};
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -519,7 +528,7 @@ rte_eth_dev_owner_delete(const uint64_t owner_id)
 {
 	uint16_t port_id;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -541,7 +550,7 @@ rte_eth_dev_owner_get(const uint16_t port_id, struct rte_eth_dev_owner *owner)
 	int ret = 0;
 	struct rte_eth_dev *ethdev = &rte_eth_devices[port_id];
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 07/19] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v6 08/19] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v6 09/19] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v6 10/19] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v6 11/19] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v6 12/19] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v6 13/19] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 14/19] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v6 15/19] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v6 16/19] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v6 17/19] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v6 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v6 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..fc4736814 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,20 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support ethernet device hotplug for primary-secondary model.**
+
+  In primary-secondary process model, ethernet devices are regarded as shared
+  by default, attach or detach a device on any process will broadcast to
+  other processes through mp channel then device information will be
+  synchronzied on all processes. While secondary process can still attach
+  or detach a private device (vdev only) with specific API.
+
+* **Support ethernet device lock.**
+
+  An ethernet device can be directly or conditionally locked. A directly
+  locked device can't be detached, while when try to detach a conditionally
+  locked device a pre-registered callback will be invoked to perform condition
+  check and decide if it can be detached or not.
 
 API Changes
 -----------
@@ -60,6 +74,13 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is extended.
+
+  In primary-secondary process model, ``rte_eth_dev_attach`` will guarantee that
+  device be attached on all processes, if any process failed to attach, it will
+  rollback to orignal status. ``rte_eth_dev_detach`` also guarantee device be
+  detached on all processes, if device is locked by any process, it will roll
+  back to original status.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v6 03/19] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
> 
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs changes:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +	bundle = calloc(1, sizeof(*bundle));
> +	if (bundle == NULL) {
> +		resp->result = -ENOMEM;
> +		ret = rte_mp_reply(&mp_resp, peer);
> +		if (ret) {
> +			ethdev_log(ERR, "failed to send reply to primary request\n");
> +			return ret;
> +		}
> +	}
> +
> +	bundle->msg = *msg;
> +	/**
> +	 * We need to send reply on interrupt thread, but peer can't be
> +	 * parsed directly, so this is a temporal hack, need to be fixed
> +	 * when it is ready.
> +	 */
> +	bundle->peer = (void *)strdup(peer);
> +
> +	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);

Nitpick - it would've been good if you explained the reasoning behind 
processing the request on a separate thread in a comment.

Other than that,

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 03/19] ethdev: enable hotplug on multi-process

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
> 
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
> 
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
> 
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
> 
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
> 
> Scenario for Case 1, 2:
> 
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
> 
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
> 
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
> 
> APIs changes:
> 
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
> 
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Missed one issue - new API's should be added to the .map file.


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 04/19] ethdev: introduce device lock

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.
> 
> Aslo introduce the new API rte_eth_dev_lock_with_callback and
> rte_eth_dev_unlock_with callback to let application to register
> a callback function which will be invoked before a device is going
> to be detached, the return value of the function will decide if
> device will continue be detached or not, this support application
> to do condition check at runtime.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

New API's should be added to .map file.

Otherwise,

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 28, 2018 5:20 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v6 03/19] ethdev: enable hotplug on multi-process
> 
> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> > We are going to introduce the solution to handle different hotplug
> > cases in multi-process situation, it include below scenario:
> >
> > 1. Attach a share device from primary
> > 2. Detach a share device from primary
> > 3. Attach a share device from secondary 4. Detach a share device from
> > secondary 5. Attach a private device from secondary 6. Detach a
> > private device from secondary 7. Detach a share device from secondary
> > privately 8. Attach a share device from secondary privately
> >
> > In primary-secondary process model, we assume device is shared by default.
> > that means attach or detach a device on any process will broadcast to
> > all other processes through mp channel then device information will be
> > synchronized on all processes.
> >
> > Any failure during attaching process will cause inconsistent status
> > between processes, so proper rollback action should be considered.
> > Also it is not safe to detach a share device when other process still
> > use it, so a handshake mechanism is introduced.
> >
> > This patch covers the implementation of case 1,2,5,6,7,8.
> > Case 3,4 will be implemented on separate patch as well as handshake
> > mechanism.
> >
> > Scenario for Case 1, 2:
> >
> > attach device
> > a) primary attach the new device if failed goto h).
> > b) primary send attach sync request to all secondary.
> > c) secondary receive request and attach device and send reply.
> > d) primary check the reply if all success go to i).
> > e) primary send attach rollback sync request to all secondary.
> > f) secondary receive the request and detach device and send reply.
> > g) primary receive the reply and detach device as rollback action.
> > h) attach fail
> > i) attach success
> >
> > detach device
> > a) primary perform pre-detach check, if device is locked, goto i).
> > b) primary send pre-detach sync request to all secondary.
> > c) secondary perform pre-detach check and send reply.
> > d) primary check the reply if any fail goto i).
> > e) primary send detach sync request to all secondary
> > f) secondary detach the device and send reply (assume no fail)
> > g) primary detach the device.
> > h) detach success
> > i) detach failed
> >
> > Case 5, 6:
> > Secondary process can attach private device which only visible to
> > itself, in this case no IPC is involved, primary process is not
> > allowed to have private device so far.
> >
> > Case 7, 8:
> > Secondary process can also temporally to detach a share device "privately"
> > then attach it back later, this action also not impact other processes.
> >
> > APIs changes:
> >
> > rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> > share device attach/detach in primary-secondary process model, it will
> > be called in case 1,2,3,4.
> >
> > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> > introduced to cover case 5,6,7,8, this API can only be invoked in
> > secondary process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> Missed one issue - new API's should be added to the .map file.

Will fix thanks!
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v6 05/19] ethdev: support attach or detach share device from secondary

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a share device
> attach/detach request be issued from secondary process
> 
> device attach on secondary:
> a) seconary send sync request to primary.
> b) primary receive the request and attach the new device if failed
>     goto i).
> c) primary forward attach sync request to all secondary.
> d) secondary receive request and attach device and send reply.
> e) primary check the reply if all success go to j).
> f) primary send attach rollback sync request to all secondary.
> g) secondary receive the request and detach device and send reply.
> h) primary receive the reply and detach device as rollback action.
> i) send fail reply to secondary, goto k).
> j) send success reply to secondary.
> k) secondary process receive reply of step a) and return.
> 
> device detach on secondary:
> a) secondary send sync request to primary
> b) primary receive the request and perform pre-detach check, if device
>     is locked, goto j).
> c) primary send pre-detach sync request to all secondary.
> d) secondary perform pre-detach check and send reply.
> e) primary check the reply if any fail goto j).
> f) primary send detach sync request to all secondary
> g) secondary detach the device and send reply
> h) primary detach the device.
> i) send success reply to secondary, goto k).
> j) send fail reply to secondary.
> k) secondary process receive reply of step a) and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +	const struct rte_mp_msg *msg = &bundle->msg;
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	struct eth_dev_mp_req tmp_req;
> +	uint16_t port_id;
> +	int ret = 0;
> +
> +	tmp_req = *req;
> +
> +	if (req->t == REQ_TYPE_ATTACH) {
> +		ret = do_eth_dev_attach(req->devargs, &port_id);
> +		if (!ret) {
> +			tmp_req.port_id = port_id;
> +			ret = eth_dev_request_to_secondary(&tmp_req);
> +		}
> +	} else if (req->t == REQ_TYPE_DETACH) {
> +		if (!rte_eth_dev_is_valid_port(req->port_id))
> +			ret = -EINVAL;
> +		if (!ret)
> +			ret = process_lock_callbacks(req->port_id);
> +		if (!ret) {
> +			tmp_req.t = REQ_TYPE_PRE_DETACH;
> +			ret = eth_dev_request_to_secondary(&tmp_req);
> +		}
> +		if (!ret) {
> +			if (!tmp_req.result) {
> +				tmp_req.t = REQ_TYPE_DETACH;
> +				ret = eth_dev_request_to_secondary(&tmp_req);
> +				if (!ret)
> +					ret = do_eth_dev_detach(req->port_id);
> +			} else {
> +				ret = tmp_req.result;
> +			}
> +		}

This seems like a good opportunity to use goto, rather than checking 
value of ret every step of the way.

> +	} else {
> +		ethdev_log(ERR, "unsupported secondary to primary request\n");
> +		ret = -ENOTSUP;
> +	}
> +
> +	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
> +	if (ret)
> +		ethdev_log(ERR, "failed to send response to secondary\n");
> +
> +	free(bundle->peer);
> +	free(bundle);
> +}
> +
> +static int
> +handle_secondary_request(const struct rte_mp_msg *msg,
> +			const void *peer)
>   {
> -	RTE_SET_USED(msg);
> -	RTE_SET_USED(peer);
> -	return -ENOTSUP;
> +	struct mp_reply_bundle *bundle;
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	int ret = 0;
> +
> +	bundle = malloc(sizeof(*bundle));
> +	if (bundle == NULL) {
> +		ethdev_log(ERR, "not enough memory\n");
> +		return send_response_to_secondary(req, -ENOMEM, peer);
> +	}
> +
> +	bundle->msg = *msg;
> +	/**
> +	 * We need to send reply on interrupt thread, but peer can't be
> +	 * parsed directly, so this is a temporal hack, need to be fixed
> +	 * when it is ready.
> +	 */
> +	bundle->peer = strdup(peer);
> +
> +	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);

Again, some comments explaining why this is being done on a separate 
thread would be great.

Otherwise,

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> When attach a private device from secondary as the first one, we need
> to make sure rte_eth_dev_shared_data is initialized, the patch add
> necessary IPC for secondary to inform primary to do initialization.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Does this mean hotplug is broken before this patch? Can it be moved earlier?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 28, 2018 5:25 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device as first
> 
> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> > When attach a private device from secondary as the first one, we need
> > to make sure rte_eth_dev_shared_data is initialized, the patch add
> > necessary IPC for secondary to inform primary to do initialization.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> Does this mean hotplug is broken before this patch? Can it be moved earlier?

Basically patch 4,5,6 depends on patch 3, they have no dependency each other.

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v6 19/19] doc: update release notes for multi process hotplug

[Burakov, Anatoly]

On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> Update release notes for the new multi process hotplug feature.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

+CC John Mc as our resident native English speaker :)

>   doc/guides/rel_notes/release_18_08.rst | 21 +++++++++++++++++++++
>   1 file changed, 21 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
> index bc0124295..fc4736814 100644
> --- a/doc/guides/rel_notes/release_18_08.rst
> +++ b/doc/guides/rel_notes/release_18_08.rst
> @@ -46,6 +46,20 @@ New Features
>     Flow API support has been added to CXGBE Poll Mode Driver to offload
>     flows to Chelsio T5/T6 NICs.
>   
> +* **Support ethernet device hotplug for primary-secondary model.**
> +
> +  In primary-secondary process model, ethernet devices are regarded as shared
> +  by default, attach or detach a device on any process will broadcast to
> +  other processes through mp channel then device information will be
> +  synchronzied on all processes. While secondary process can still attach
> +  or detach a private device (vdev only) with specific API.

Suggested rewording:

**Support ethdev multiprocess hotplug.**

Hotplug and hot-unplug for ethdev devices will now be supported in 
multiprocessing scenario. Any ethdev devices created in the primary 
process will be regarded as shared and will be available for all DPDK 
processes, while secondary processes will have a choice between adding a 
private (non-shared) or a shared device. Synchronization between 
processes will be done using DPDK IPC.

------

I hope this is accurate enough, but please correct me if my 
interpretation of the changes is wrong!

> +
> +* **Support ethernet device lock.**
> +
> +  An ethernet device can be directly or conditionally locked. A directly
> +  locked device can't be detached, while when try to detach a conditionally
> +  locked device a pre-registered callback will be invoked to perform condition
> +  check and decide if it can be detached or not.

Suggested rewording:

**Support ethernet device locking.**

Application can now lock an ethernet device to prevent unexpected device 
removal. Devices can either be locked unconditionally, or an application 
can register for a callback before unplug for the purposes of performing 
cleanup before releasing the device (or have a chance to deny unplug).

>   
>   API Changes
>   -----------
> @@ -60,6 +74,13 @@ API Changes
>      Also, make sure to start the actual text at the margin.
>      =========================================================
>   
> +* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is extended.
> +
> +  In primary-secondary process model, ``rte_eth_dev_attach`` will guarantee that
> +  device be attached on all processes, if any process failed to attach, it will
> +  rollback to orignal status.

IMO no need to mention rollback - these are internal implementation 
details. The main thing is, the device is either attached everywhere, or 
attached nowhere.

  ``rte_eth_dev_detach`` also guarantee device be
> +  detached on all processes, if device is locked by any process, it will roll
> +  back to original status.

Again, no need to mention internal implementation details - it would be 
sufficient to mention that rte_eth_dev_detach will only succeed if the 
device is not locked by any of the participating processes.

>   
>   ABI Changes
>   -----------
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Burakov, Anatoly]

On 28-Jun-18 10:29 AM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Burakov, Anatoly
>> Sent: Thursday, June 28, 2018 5:25 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device as first
>>
>> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
>>> When attach a private device from secondary as the first one, we need
>>> to make sure rte_eth_dev_shared_data is initialized, the patch add
>>> necessary IPC for secondary to inform primary to do initialization.
>>>
>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>> ---
>>
>> Does this mean hotplug is broken before this patch? Can it be moved earlier?
> 
> Basically patch 4,5,6 depends on patch 3, they have no dependency each other.

So... is that a yes? :)

> 
>>
>> --
>> Thanks,
>> Anatoly


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 28, 2018 5:41 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device as first
> 
> On 28-Jun-18 10:29 AM, Zhang, Qi Z wrote:
> >
> >
> >> -----Original Message-----
> >> From: Burakov, Anatoly
> >> Sent: Thursday, June 28, 2018 5:25 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device
> >> as first
> >>
> >> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> >>> When attach a private device from secondary as the first one, we
> >>> need to make sure rte_eth_dev_shared_data is initialized, the patch
> >>> add necessary IPC for secondary to inform primary to do initialization.
> >>>
> >>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> >>> ---
> >>
> >> Does this mean hotplug is broken before this patch? Can it be moved
> earlier?
> >
> > Basically patch 4,5,6 depends on patch 3, they have no dependency each
> other.
> 
> So... is that a yes? :)

Yes, 3 ,6,4, 5 is better, will re-order
Thanks
Qi

> 
> >
> >>
> >> --
> >> Thanks,
> >> Anatoly
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Zhang, Qi Z]

> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Thursday, June 28, 2018 7:27 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: RE: [PATCH v6 06/19] ethdev: support attach private device as first
> 
> 
> 
> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Thursday, June 28, 2018 5:41 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: Re: [PATCH v6 06/19] ethdev: support attach private device as
> > first
> >
> > On 28-Jun-18 10:29 AM, Zhang, Qi Z wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Burakov, Anatoly
> > >> Sent: Thursday, June 28, 2018 5:25 PM
> > >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> > >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
> > >> dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
> > >> Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> > >> <narender.vangati@intel.com>
> > >> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device
> > >> as first
> > >>
> > >> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> > >>> When attach a private device from secondary as the first one, we
> > >>> need to make sure rte_eth_dev_shared_data is initialized, the
> > >>> patch add necessary IPC for secondary to inform primary to do
> initialization.
> > >>>
> > >>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > >>> ---
> > >>
> > >> Does this mean hotplug is broken before this patch? Can it be moved
> > earlier?
> > >
> > > Basically patch 4,5,6 depends on patch 3, they have no dependency
> > > each
> > other.
> >
> > So... is that a yes? :)
> 
> Yes, 3 ,6,4, 5 is better, will re-order

Apologies, actually patch 6 depends on patch 5, which build the secondary to primary request channel, so current sequence is necessary.

> Thanks
> Qi
> 
> >
> > >
> > >>
> > >> --
> > >> Thanks,
> > >> Anatoly
> >
> >
> > --
> > Thanks,
> > Anatoly

Re: [dpdk-dev] [PATCH v6 19/19] doc: update release notes for multi process hotplug

[Zhang, Qi Z]

Hi Anatoly:
	Your comments looks good to me.
Thanks!
Qi 

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 28, 2018 5:39 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; Mcnamara, John <john.mcnamara@intel.com>
> Subject: Re: [PATCH v6 19/19] doc: update release notes for multi process
> hotplug
> 
> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
> > Update release notes for the new multi process hotplug feature.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> +CC John Mc as our resident native English speaker :)
> 
> >   doc/guides/rel_notes/release_18_08.rst | 21 +++++++++++++++++++++
> >   1 file changed, 21 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/release_18_08.rst
> > b/doc/guides/rel_notes/release_18_08.rst
> > index bc0124295..fc4736814 100644
> > --- a/doc/guides/rel_notes/release_18_08.rst
> > +++ b/doc/guides/rel_notes/release_18_08.rst
> > @@ -46,6 +46,20 @@ New Features
> >     Flow API support has been added to CXGBE Poll Mode Driver to offload
> >     flows to Chelsio T5/T6 NICs.
> >
> > +* **Support ethernet device hotplug for primary-secondary model.**
> > +
> > +  In primary-secondary process model, ethernet devices are regarded
> > + as shared  by default, attach or detach a device on any process will
> > + broadcast to  other processes through mp channel then device
> > + information will be  synchronzied on all processes. While secondary
> > + process can still attach  or detach a private device (vdev only) with specific
> API.
> 
> Suggested rewording:
> 
> **Support ethdev multiprocess hotplug.**
> 
> Hotplug and hot-unplug for ethdev devices will now be supported in
> multiprocessing scenario. Any ethdev devices created in the primary process
> will be regarded as shared and will be available for all DPDK processes, while
> secondary processes will have a choice between adding a private (non-shared)
> or a shared device. Synchronization between processes will be done using
> DPDK IPC.
> 
> ------
> 
> I hope this is accurate enough, but please correct me if my interpretation of
> the changes is wrong!
> 
> > +
> > +* **Support ethernet device lock.**
> > +
> > +  An ethernet device can be directly or conditionally locked. A
> > + directly  locked device can't be detached, while when try to detach
> > + a conditionally  locked device a pre-registered callback will be
> > + invoked to perform condition  check and decide if it can be detached or
> not.
> 
> Suggested rewording:
> 
> **Support ethernet device locking.**
> 
> Application can now lock an ethernet device to prevent unexpected device
> removal. Devices can either be locked unconditionally, or an application can
> register for a callback before unplug for the purposes of performing cleanup
> before releasing the device (or have a chance to deny unplug).
> 
> >
> >   API Changes
> >   -----------
> > @@ -60,6 +74,13 @@ API Changes
> >      Also, make sure to start the actual text at the margin.
> >      =========================================================
> >
> > +* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is
> extended.
> > +
> > +  In primary-secondary process model, ``rte_eth_dev_attach`` will
> > + guarantee that  device be attached on all processes, if any process
> > + failed to attach, it will  rollback to orignal status.
> 
> IMO no need to mention rollback - these are internal implementation details.
> The main thing is, the device is either attached everywhere, or attached
> nowhere.
> 
>   ``rte_eth_dev_detach`` also guarantee device be
> > +  detached on all processes, if device is locked by any process, it
> > + will roll  back to original status.
> 
> Again, no need to mention internal implementation details - it would be
> sufficient to mention that rte_eth_dev_detach will only succeed if the device is
> not locked by any of the participating processes.
> 
> >
> >   ABI Changes
> >   -----------
> >
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v6 06/19] ethdev: support attach private device as first

[Burakov, Anatoly]

On 28-Jun-18 12:45 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Zhang, Qi Z
>> Sent: Thursday, June 28, 2018 7:27 PM
>> To: Burakov, Anatoly <anatoly.burakov@intel.com>; thomas@monjalon.net
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: RE: [PATCH v6 06/19] ethdev: support attach private device as first
>>
>>
>>
>>> -----Original Message-----
>>> From: Burakov, Anatoly
>>> Sent: Thursday, June 28, 2018 5:41 PM
>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>>> <narender.vangati@intel.com>
>>> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device as
>>> first
>>>
>>> On 28-Jun-18 10:29 AM, Zhang, Qi Z wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Burakov, Anatoly
>>>>> Sent: Thursday, June 28, 2018 5:25 PM
>>>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
>>>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>;
>>>>> dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>;
>>>>> Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H
>>>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>>>>> <narender.vangati@intel.com>
>>>>> Subject: Re: [PATCH v6 06/19] ethdev: support attach private device
>>>>> as first
>>>>>
>>>>> On 28-Jun-18 2:52 AM, Qi Zhang wrote:
>>>>>> When attach a private device from secondary as the first one, we
>>>>>> need to make sure rte_eth_dev_shared_data is initialized, the
>>>>>> patch add necessary IPC for secondary to inform primary to do
>> initialization.
>>>>>>
>>>>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>>>>> ---
>>>>>
>>>>> Does this mean hotplug is broken before this patch? Can it be moved
>>> earlier?
>>>>
>>>> Basically patch 4,5,6 depends on patch 3, they have no dependency
>>>> each
>>> other.
>>>
>>> So... is that a yes? :)
>>
>> Yes, 3 ,6,4, 5 is better, will re-order
> 
> Apologies, actually patch 6 depends on patch 5, which build the secondary to primary request channel, so current sequence is necessary.

OK, we can leave it as is then.

-- 
Thanks,
Anatoly

[dpdk-dev] [PATCH v7 00/19] enable hotplug on multi-process

[Qi Zhang]

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  ethdev: support attach private device as first
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst       |  20 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |   3 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 +
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 ++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_proc.c      |  51 +++-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_eal.h      |  34 +++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 +++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 430 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 +++
 lib/librte_ethdev/ethdev_private.h           |  42 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 312 +++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 lib/librte_ethdev/rte_ethdev_pci.h           |   3 +
 lib/librte_ethdev/rte_ethdev_version.map     |   4 +
 34 files changed, 1833 insertions(+), 37 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v7 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused , share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++
 3 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..2353fc921 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
 	eth_dev->state = RTE_ETH_DEV_UNUSED;
-
 	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
 
 	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..eeb944146 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,9 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v7 02/19] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 57 +++++++++++++++++++++++++++++++--
 lib/librte_eal/common/eal_private.h     |  5 +++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 4 files changed, 96 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index f010ef59e..f6d7c83e4 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,11 +619,47 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
 	char path[PATH_MAX];
-	int dir_fd;
+	int dir_fd, ret;
 	pthread_t mp_handle_tid, async_reply_handle_tid;
 
 	/* create filter path */
@@ -686,7 +722,24 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	ret = process_mp_init_callbacks();
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to process mp init callbacks\n");
+
+	return ret;
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
-- 
2.13.6

[dpdk-dev] [PATCH v7 03/19] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile               |   1 +
 lib/librte_ethdev/ethdev_mp.c            | 261 +++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h            |  41 +++++
 lib/librte_ethdev/ethdev_private.h       |  39 +++++
 lib/librte_ethdev/meson.build            |   1 +
 lib/librte_ethdev/rte_ethdev.c           | 210 +++++++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h           |  45 ++++++
 lib/librte_ethdev/rte_ethdev_core.h      |   5 +
 lib/librte_ethdev/rte_ethdev_version.map |   2 +
 9 files changed, 588 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..0f9d8990d
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,261 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+#include <rte_string_fns.h>
+#include <rte_alarm.h>
+
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 2353fc921..6c5f465a2 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int solid_release;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+	/**
+	 * A private device on secondary need
+	 * rte_eth_dev_release_port.
+	 * 1) only vdev support private device.
+	 * 2) private device has no empty devargs.
+	 */
+		if (!strcmp(bus->name, "vdev") &&
+			dev->devargs != NULL &&
+			strlen(dev->devargs->args) > 0)
+			solid_release = 1;
+		else
+			solid_release = 0;
+	} else {
+		solid_release = 0;
+	}
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	if (solid_release)
+		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	else
+		return rte_eth_dev_release_port_private(
+			&rte_eth_devices[port_id]);
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..813806e3c 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 40cf42b8a..fbcc03925 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -217,9 +217,11 @@ EXPERIMENTAL {
 	global:
 
 	rte_eth_devargs_parse;
+	rte_eth_dev_attach_private;
 	rte_eth_dev_count_total;
 	rte_eth_dev_create;
 	rte_eth_dev_destroy;
+	rte_eth_dev_detach_private;
 	rte_eth_dev_get_module_eeprom;
 	rte_eth_dev_get_module_info;
 	rte_eth_dev_is_removed;
-- 
2.13.6

[dpdk-dev] [PATCH v7 04/19] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_ethdev/Makefile               |   1 +
 lib/librte_ethdev/ethdev_lock.c          | 140 +++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h          |  31 +++++++
 lib/librte_ethdev/ethdev_mp.c            |   3 +-
 lib/librte_ethdev/meson.build            |   1 +
 lib/librte_ethdev/rte_ethdev.c           |  60 ++++++++++++-
 lib/librte_ethdev/rte_ethdev.h           | 124 +++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_version.map |   2 +
 8 files changed, 360 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0f9d8990d..1d148cd5e 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -6,6 +6,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -108,7 +109,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 6c5f465a2..54d3d2369 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -734,6 +735,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	if (solid_release)
 		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	else
@@ -802,7 +804,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -844,6 +845,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -890,6 +895,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -903,6 +909,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4705,6 +4715,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 813806e3c..1596b6e2b 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index fbcc03925..1e270a387 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -225,6 +225,7 @@ EXPERIMENTAL {
 	rte_eth_dev_get_module_eeprom;
 	rte_eth_dev_get_module_info;
 	rte_eth_dev_is_removed;
+	rte_eth_dev_lock;
 	rte_eth_dev_owner_delete;
 	rte_eth_dev_owner_get;
 	rte_eth_dev_owner_new;
@@ -232,6 +233,7 @@ EXPERIMENTAL {
 	rte_eth_dev_owner_unset;
 	rte_eth_dev_rx_offload_name;
 	rte_eth_dev_tx_offload_name;
+	rte_eth_dev_unlock;
 	rte_eth_switch_domain_alloc;
 	rte_eth_switch_domain_free;
 	rte_mtr_capabilities_get;
-- 
2.13.6

[dpdk-dev] [PATCH v7 05/19] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 179 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 173 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 1d148cd5e..8d13da591 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,8 +5,44 @@
 #include <rte_alarm.h>
 
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -84,11 +120,122 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (ret)
+			goto finish;
+
+		tmp_req.port_id = port_id;
+		ret = eth_dev_request_to_secondary(&tmp_req);
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id)) {
+			ret = -EINVAL;
+			goto finish;
+		}
+
+		ret = process_lock_callbacks(req->port_id);
+		if (ret)
+			goto finish;
+
+		tmp_req.t = REQ_TYPE_PRE_DETACH;
+		ret = eth_dev_request_to_secondary(&tmp_req);
+		if (ret)
+			goto finish;
+
+		if (!tmp_req.result) {
+			tmp_req.t = REQ_TYPE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+			if (ret)
+				goto finish;
+
+			ret = do_eth_dev_detach(req->port_id);
+		} else {
+			ret = tmp_req.result;
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -183,8 +330,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v7 06/19] ethdev: support attach private device as first

[Qi Zhang]

When attach a private device from secondary as the first one, we need
to make sure rte_eth_dev_shared_data is initialized, the patch add
necessary IPC for secondary to inform primary to do initialization.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c      |  2 ++
 lib/librte_ethdev/ethdev_mp.h      |  1 +
 lib/librte_ethdev/ethdev_private.h |  3 +++
 lib/librte_ethdev/rte_ethdev.c     | 31 ++++++++++++++++++++-----------
 4 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 8d13da591..28f89dba9 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -189,6 +189,8 @@ __handle_secondary_request(void *param)
 		} else {
 			ret = tmp_req.result;
 		}
+	} else if (req->t == REQ_TYPE_SHARE_DATA_PREPARE) {
+		eth_dev_shared_data_prepare();
 	} else {
 		ethdev_log(ERR, "unsupported secondary to primary request\n");
 		ret = -ENOTSUP;
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..61fc381da 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -15,6 +15,7 @@ enum eth_dev_req_type {
 	REQ_TYPE_PRE_DETACH,
 	REQ_TYPE_DETACH,
 	REQ_TYPE_ATTACH_ROLLBACK,
+	REQ_TYPE_SHARE_DATA_PREPARE,
 };
 
 struct eth_dev_mp_req {
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
index 981e7de8a..005d63afc 100644
--- a/lib/librte_ethdev/ethdev_private.h
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -36,4 +36,7 @@ int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
  */
 int do_eth_dev_detach(uint16_t port_id);
 
+/* Prepare shared data for multi-process */
+void eth_dev_shared_data_prepare(void);
+
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 54d3d2369..c0f68fe09 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -199,11 +199,14 @@ rte_eth_find_next(uint16_t port_id)
 	return port_id;
 }
 
-static void
-rte_eth_dev_shared_data_prepare(void)
+void
+eth_dev_shared_data_prepare(void)
 {
 	const unsigned flags = 0;
 	const struct rte_memzone *mz;
+	struct eth_dev_mp_req req;
+
+	memset(&req, 0, sizeof(req));
 
 	rte_spinlock_lock(&rte_eth_shared_data_lock);
 
@@ -215,6 +218,12 @@ rte_eth_dev_shared_data_prepare(void)
 					rte_socket_id(), flags);
 		} else
 			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		/* if secondary attach a private device first */
+		if (mz == NULL && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			req.t = REQ_TYPE_SHARE_DATA_PREPARE;
+			eth_dev_request_to_primary(&req);
+			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		}
 		if (mz == NULL)
 			rte_panic("Cannot allocate ethdev shared data\n");
 
@@ -255,7 +264,7 @@ rte_eth_dev_allocated(const char *name)
 {
 	struct rte_eth_dev *ethdev;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -300,7 +309,7 @@ rte_eth_dev_allocate(const char *name)
 	uint16_t port_id;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port creation between primary and secondary threads. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -339,7 +348,7 @@ rte_eth_dev_attach_secondary(const char *name)
 	uint16_t i;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port attachment to primary port creation and release. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -379,7 +388,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (eth_dev == NULL)
 		return -EINVAL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
 
@@ -432,7 +441,7 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id)
 int __rte_experimental
 rte_eth_dev_owner_new(uint64_t *owner_id)
 {
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -487,7 +496,7 @@ rte_eth_dev_owner_set(const uint16_t port_id,
 {
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -504,7 +513,7 @@ rte_eth_dev_owner_unset(const uint16_t port_id, const uint64_t owner_id)
 			{.id = RTE_ETH_DEV_NO_OWNER, .name = ""};
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -519,7 +528,7 @@ rte_eth_dev_owner_delete(const uint64_t owner_id)
 {
 	uint16_t port_id;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -541,7 +550,7 @@ rte_eth_dev_owner_get(const uint16_t port_id, struct rte_eth_dev_owner *owner)
 	int ret = 0;
 	struct rte_eth_dev *ethdev = &rte_eth_devices[port_id];
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-- 
2.13.6

[dpdk-dev] [PATCH v7 07/19] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v7 08/19] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v7 09/19] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v7 10/19] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v7 11/19] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v7 12/19] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v7 13/19] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v7 14/19] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v7 15/19] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v7 16/19] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v7 17/19] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v7 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v7 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..650f7aadb 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,21 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support ethernet device hotplug for primary-secondary model.**
+
+  Hotplug and hot-unplug for ethdev devices will now be supported in
+  multiprocessing scenario. Any ethdev devices created in the primary
+  process will be regarded as shared and will be available for all DPDK
+  processes, while secondary processes will have a choice between adding
+  a private (non-shared) or a shared device. Synchronization between
+  processes will be done using DPDK IPC.
+
+* **Support ethernet device lock.**
+
+  Application can now lock an ethernet device to prevent unexpected device
+  removal. Devices can either be locked unconditionally, or an application
+  can register for a callback before unplug for the purposes of performing
+  cleanup before releasing the device (or have a chance to deny unplug)
 
 API Changes
 -----------
@@ -60,6 +75,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is extended.
+
+  In primary-secondary process model, ``rte_eth_dev_attach`` will guarantee
+  that device be attached on all processes, while ``rte_eth_dev_detach``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v7 02/19] eal: enable multi process init callback

[Burakov, Anatoly]

On 28-Jun-18 1:56 PM, Qi Zhang wrote:
> Introduce new API rte_eal_register_mp_init that help to register
> a callback function which will be invoked right after multi-process
> channel be established (rte_mp_channel_init). Typically the API
> will be used by other module that want it's mp channel action callbacks
> can be registered during rte_eal_init automatically.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> ---

<snip>

> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Register a callback function that will be invoked right after
> + * multi-process channel be established (rte_mp_channel_init). Typically
> + * the function is used by other module that want it's mp channel
> + * action callbacks can be registered during rte_eal_init automatically.
> + *
> + * @note
> + *   This function only take effect when be called before rte_eal_init,
> + *   and all registered callback will be clear during rte_eal_cleanup.
> + *
> + * @param callback
> + *   function be called at that moment.
> + *
> + * @return
> + *  - 0 on success.
> + *  - (<0) on failure.
> + */
> +int __rte_experimental
> +rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);

This function is missing from the .map file.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v7 02/19] eal: enable multi process init callback

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, June 28, 2018 9:09 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v7 02/19] eal: enable multi process init callback
> 
> On 28-Jun-18 1:56 PM, Qi Zhang wrote:
> > Introduce new API rte_eal_register_mp_init that help to register a
> > callback function which will be invoked right after multi-process
> > channel be established (rte_mp_channel_init). Typically the API will
> > be used by other module that want it's mp channel action callbacks can
> > be registered during rte_eal_init automatically.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
> > ---
> 
> <snip>
> 
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice
> > + *
> > + * Register a callback function that will be invoked right after
> > + * multi-process channel be established (rte_mp_channel_init).
> > +Typically
> > + * the function is used by other module that want it's mp channel
> > + * action callbacks can be registered during rte_eal_init automatically.
> > + *
> > + * @note
> > + *   This function only take effect when be called before rte_eal_init,
> > + *   and all registered callback will be clear during rte_eal_cleanup.
> > + *
> > + * @param callback
> > + *   function be called at that moment.
> > + *
> > + * @return
> > + *  - 0 on success.
> > + *  - (<0) on failure.
> > + */
> > +int __rte_experimental
> > +rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
> 
> This function is missing from the .map file.

OK, will capture this and some minor reword in release notes in v8
Now, wait for other comments.

Thanks
Qi

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v7 01/19] ethdev: add function to release port in local process

[Andrew Rybchenko]

On 06/28/2018 03:56 PM, Qi Zhang wrote:
> Add driver API rte_eth_release_port_private to support the
> requirement that an ethdev only be released on secondary process,
> so only local state be set to unused , share data will not be

Extra space before comma above.

> reset so primary process can still use it.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> Acked-by: Remy Horton <remy.horton@intel.com>
> ---
>   lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-
>   lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
>   lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++
>   3 files changed, 28 insertions(+), 1 deletion(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index a9977df97..2353fc921 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c

<...>

> @@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>   	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
>   
>   	eth_dev->state = RTE_ETH_DEV_UNUSED;
> -
>   	memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
>   
>   	rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);

Just a nit: above looks like unrelated change.

Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>

Re: [dpdk-dev] [PATCH v7 03/19] ethdev: enable hotplug on multi-process

[Andrew Rybchenko]

On 06/28/2018 03:56 PM, Qi Zhang wrote:
> We are going to introduce the solution to handle different hotplug
> cases in multi-process situation, it include below scenario:
>
> 1. Attach a share device from primary
> 2. Detach a share device from primary
> 3. Attach a share device from secondary
> 4. Detach a share device from secondary
> 5. Attach a private device from secondary
> 6. Detach a private device from secondary
> 7. Detach a share device from secondary privately
> 8. Attach a share device from secondary privately
>
> In primary-secondary process model, we assume device is shared by default.
> that means attach or detach a device on any process will broadcast to
> all other processes through mp channel then device information will be
> synchronized on all processes.
>
> Any failure during attaching process will cause inconsistent status
> between processes, so proper rollback action should be considered.
> Also it is not safe to detach a share device when other process still use
> it, so a handshake mechanism is introduced.
>
> This patch covers the implementation of case 1,2,5,6,7,8.
> Case 3,4 will be implemented on separate patch as well as handshake
> mechanism.
>
> Scenario for Case 1, 2:
>
> attach device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach device and send reply.
> d) primary check the reply if all success go to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach device and send reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
>
> detach device
> a) primary perform pre-detach check, if device is locked, goto i).
> b) primary send pre-detach sync request to all secondary.
> c) secondary perform pre-detach check and send reply.
> d) primary check the reply if any fail goto i).
> e) primary send detach sync request to all secondary
> f) secondary detach the device and send reply (assume no fail)
> g) primary detach the device.
> h) detach success
> i) detach failed
>
> Case 5, 6:
> Secondary process can attach private device which only visible to itself,
> in this case no IPC is involved, primary process is not allowed to have
> private device so far.
>
> Case 7, 8:
> Secondary process can also temporally to detach a share device "privately"
> then attach it back later, this action also not impact other processes.
>
> APIs changes:
>
> rte_eth_dev_attach and rte_eth_dev_attach are extended to support
> share device attach/detach in primary-secondary process model, it will
> be called in case 1,2,3,4.
>
> New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
> introduced to cover case 5,6,7,8, this API can only be invoked in secondary
> process.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_ethdev/Makefile               |   1 +
>   lib/librte_ethdev/ethdev_mp.c            | 261 +++++++++++++++++++++++++++++++
>   lib/librte_ethdev/ethdev_mp.h            |  41 +++++
>   lib/librte_ethdev/ethdev_private.h       |  39 +++++
>   lib/librte_ethdev/meson.build            |   1 +
>   lib/librte_ethdev/rte_ethdev.c           | 210 +++++++++++++++++++++++--
>   lib/librte_ethdev/rte_ethdev.h           |  45 ++++++
>   lib/librte_ethdev/rte_ethdev_core.h      |   5 +
>   lib/librte_ethdev/rte_ethdev_version.map |   2 +
>   9 files changed, 588 insertions(+), 17 deletions(-)
>   create mode 100644 lib/librte_ethdev/ethdev_mp.c
>   create mode 100644 lib/librte_ethdev/ethdev_mp.h
>   create mode 100644 lib/librte_ethdev/ethdev_private.h
>
> diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
> index c2f2f7d82..d0a059b83 100644
> --- a/lib/librte_ethdev/Makefile
> +++ b/lib/librte_ethdev/Makefile
> @@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
>   LIBABIVER := 9
>   
>   SRCS-y += rte_ethdev.c
> +SRCS-y += ethdev_mp.c
>   SRCS-y += rte_flow.c
>   SRCS-y += rte_tm.c
>   SRCS-y += rte_mtr.c
> diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
> new file mode 100644
> index 000000000..0f9d8990d
> --- /dev/null
> +++ b/lib/librte_ethdev/ethdev_mp.c
> @@ -0,0 +1,261 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation
> + */
> +#include <rte_string_fns.h>
> +#include <rte_alarm.h>
> +
> +#include "rte_ethdev_driver.h"
> +#include "ethdev_mp.h"
> +
> +#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
> +
> +struct mp_reply_bundle {
> +	struct rte_mp_msg msg;
> +	void *peer;
> +};
> +
> +static int detach_on_secondary(uint16_t port_id)

Looking at lib/librte_ethdev sources and 1.7.2 DPDK contribution guide:
  - "The function type should be on a line by itself preceding the 
function."
  - rte_ethdev always uses rte_eth_dev_ prefix, but strictly speaking it 
is not required
    by coding style

> +{
> +	struct rte_device *dev;
> +	struct rte_bus *bus;
> +	int ret = 0;
> +
> +	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
> +		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
> +			   port_id);
> +		return -ENODEV;
> +	}
> +
> +	dev = rte_eth_devices[port_id].device;
> +	if (dev == NULL)
> +		return -EINVAL;
> +
> +	bus = rte_bus_find_by_device(dev);
> +	if (bus == NULL)
> +		return -ENOENT;
> +
> +	ret = rte_eal_hotplug_remove(bus->name, dev->name);
> +	if (ret) {
> +		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
> +			   bus->name, dev->name);
> +		return ret;
> +	}
> +
> +	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
> +	return ret;
> +}
> +
> +static int attach_on_secondary(const char *devargs, uint16_t port_id)
> +{
> +	struct rte_devargs da;
> +	int ret;
> +
> +	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
> +		ethdev_log(ERR, "port %d already in used, failed to attach\n",
> +			   port_id);
> +		return -EINVAL;
> +	}
> +
> +	memset(&da, 0, sizeof(da));
> +
> +	if (rte_devargs_parse(&da, "%s", devargs)) {
> +		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
> +		return -EINVAL;
> +	}
> +
> +	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
> +	if (ret) {
> +		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
> +			   da.bus->name, da.name);
> +		free(da.args);
> +		return ret;
> +	}
> +
> +	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
> +		ethdev_log(ERR,
> +			"failed to attach to port %d, this is a pmd issue\n",
> +			   port_id);
> +		free(da.args);
> +		return -ENODEV;
> +	}
> +	free(da.args);
> +	return 0;
> +}
> +
> +static int
> +handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +	RTE_SET_USED(msg);
> +	RTE_SET_USED(peer);
> +	return -ENOTSUP;
> +}
> +
> +static void __handle_primary_request(void *param)
> +{
> +	struct mp_reply_bundle *bundle = param;
> +	struct rte_mp_msg *msg = &bundle->msg;
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	struct rte_mp_msg mp_resp;
> +	struct eth_dev_mp_req *resp =
> +		(struct eth_dev_mp_req *)mp_resp.param;
> +	int ret = 0;
> +
> +	memset(&mp_resp, 0, sizeof(mp_resp));
> +
> +	switch (req->t) {
> +	case REQ_TYPE_ATTACH:
> +		ret = attach_on_secondary(req->devargs, req->port_id);
> +		break;
> +	case REQ_TYPE_PRE_DETACH:
> +		ret = 0;
> +		break;
> +	case REQ_TYPE_DETACH:
> +	case REQ_TYPE_ATTACH_ROLLBACK:
> +		ret = detach_on_secondary(req->port_id);
> +		break;
> +	default:
> +		ret = -EINVAL;
> +	}
> +
> +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
> +	mp_resp.len_param = sizeof(*req);
> +	memcpy(resp, req, sizeof(*resp));
> +	resp->result = ret;
> +	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
> +		ethdev_log(ERR, "failed to send reply to primary request\n");
> +
> +	free(bundle->peer);
> +	free(bundle);
> +}
> +
> +static int
> +handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
> +{
> +
> +	struct rte_mp_msg mp_resp;
> +	const struct eth_dev_mp_req *req =
> +		(const struct eth_dev_mp_req *)msg->param;
> +	struct eth_dev_mp_req *resp =
> +		(struct eth_dev_mp_req *)mp_resp.param;
> +	struct mp_reply_bundle *bundle;
> +	int ret = 0;
> +
> +	memset(&mp_resp, 0, sizeof(mp_resp));
> +	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
> +	mp_resp.len_param = sizeof(*req);
> +	memcpy(resp, req, sizeof(*resp));
> +
> +	bundle = calloc(1, sizeof(*bundle));
> +	if (bundle == NULL) {
> +		resp->result = -ENOMEM;
> +		ret = rte_mp_reply(&mp_resp, peer);
> +		if (ret) {
> +			ethdev_log(ERR, "failed to send reply to primary request\n");
> +			return ret;
> +		}
> +	}
> +
> +	bundle->msg = *msg;
> +	/**
> +	 * We need to send reply on interrupt thread, but peer can't be
> +	 * parsed directly, so this is a temporal hack, need to be fixed
> +	 * when it is ready.
> +	 */
> +	bundle->peer = (void *)strdup(peer);
> +
> +	/**
> +	 * We are at IPC callback thread, sync IPC is not allowed due to
> +	 * dead lock, so we delegate the task to interrupt thread.
> +	 */
> +	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
> +	if (ret) {

I think it is required to free  bundle and bundle->peer here.

> +		resp->result = ret;
> +		ret = rte_mp_reply(&mp_resp, peer);
> +		if (ret) {
> +			ethdev_log(ERR, "failed to send reply to primary request\n");
> +			return ret;
> +		}
> +	}
> +	return 0;
> +}
> +
> +int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
> +{
> +	RTE_SET_USED(req);
> +	return -ENOTSUP;
> +}
> +
> +/**
> + * Request from primary to secondary.
> + *
> + * Be invoked when try to attach or detach a share device
> + * from primary process.
> + */
> +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
> +{
> +	struct rte_mp_msg mp_req;
> +	struct rte_mp_reply mp_reply;
> +	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
> +	int ret;
> +	int i;
> +
> +	memset(&mp_req, 0, sizeof(mp_req));
> +	memcpy(mp_req.param, req, sizeof(*req));
> +	mp_req.len_param = sizeof(*req);
> +	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
> +
> +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> +	if (ret) {
> +		ethdev_log(ERR, "rte_mp_request_sync failed\n");
> +		return ret;
> +	}
> +
> +	if (mp_reply.nb_sent != mp_reply.nb_received) {
> +		ethdev_log(ERR, "not all secondary reply\n");
> +		return -1;
> +	}
> +
> +	req->result = 0;
> +	for (i = 0; i < mp_reply.nb_received; i++) {
> +		struct eth_dev_mp_req *resp =
> +			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
> +		if (resp->result) {
> +			req->result = resp->result;
> +			break;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +static int on_mp_init(void)
> +{
> +	int ret;
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
> +					   handle_secondary_request);
> +		if (ret) {
> +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> +				ETH_DEV_MP_ACTION_REQUEST);
> +			return ret;
> +		}
> +	} else {
> +		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
> +					   handle_primary_request);
> +		if (ret) {
> +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> +				ETH_DEV_MP_ACTION_REQUEST);
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +RTE_INIT(ethdev_mp_init)
> +{
> +	if (rte_eal_register_mp_init(on_mp_init))
> +		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
> +}
> diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
> new file mode 100644
> index 000000000..40be46c89
> --- /dev/null
> +++ b/lib/librte_ethdev/ethdev_mp.h
> @@ -0,0 +1,41 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#ifndef _RTE_ETHDEV_MP_H_
> +#define _RTE_ETHDEV_MP_H_
> +
> +#define MAX_DEV_ARGS_LEN 0x80
> +
> +#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
> +#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
> +
> +enum eth_dev_req_type {
> +	REQ_TYPE_ATTACH,
> +	REQ_TYPE_PRE_DETACH,
> +	REQ_TYPE_DETACH,
> +	REQ_TYPE_ATTACH_ROLLBACK,
> +};
> +
> +struct eth_dev_mp_req {
> +	enum eth_dev_req_type t;
> +	char devargs[MAX_DEV_ARGS_LEN];
> +	uint16_t port_id;
> +	int result;
> +};
> +
> +/**
> + * this is a synchronous wrapper for secondary process send
> + * request to primary process, this is invoked when an attach
> + * or detach request issued from primary.
> + */
> +int eth_dev_request_to_primary(struct eth_dev_mp_req *req);

I guess it is internal ethdev API, but nothing in the header says so.
The function description does not say so as well.
Also right now even internal functions in ethdev have rte_ prefix and
I think it is good especially in the static libraries case.

> +
> +/**
> + * this is a synchronous wrapper for primary process send
> + * request to secondary process, this is invoked when an attach
> + * or detach request issued from secondary process.
> + */
> +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
> +
> +#endif
> diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
> new file mode 100644
> index 000000000..981e7de8a
> --- /dev/null
> +++ b/lib/librte_ethdev/ethdev_private.h
> @@ -0,0 +1,39 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation
> + */
> +
> +#ifndef _ETHDEV_PRIVATE_H_
> +#define _ETHDEV_PRIVATE_H_
> +
> +/**
> + * Attach a new Ethernet device in current process.
> + *
> + * @param devargs
> + *  A pointer to a strings array describing the new device
> + *  to be attached. The strings should be a pci address like
> + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
> + *
> + * @param port_id
> + *  A pointer to a port identifier actually attached.
> + *
> + * @return
> + *  0 on success and port_id is filled, negative on error
> + */
> +int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> +
> +/**
> + * Detach a Ethernet device in current process.
> + *
> + * @param port_id
> + *   The port identifier of the device to detach.
> + *
> + * @param devname
> + *   A pointer to a buffer that will be filled with the device name.
> + *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
> + *
> + * @return
> + *  0 on success and devname is filled, negative on error
> + */
> +int do_eth_dev_detach(uint16_t port_id);
> +
> +#endif
> diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
> index aed5d2265..b60256855 100644
> --- a/lib/librte_ethdev/meson.build
> +++ b/lib/librte_ethdev/meson.build
> @@ -5,6 +5,7 @@ name = 'ethdev'
>   version = 9
>   allow_experimental_apis = true
>   sources = files('ethdev_profile.c',
> +	'ethdev_mp.c'
>   	'rte_ethdev.c',
>   	'rte_flow.c',
>   	'rte_mtr.c',
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index 2353fc921..6c5f465a2 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -41,11 +41,13 @@
>   #include "rte_ethdev.h"
>   #include "rte_ethdev_driver.h"
>   #include "ethdev_profile.h"
> +#include "ethdev_mp.h"
> +#include "ethdev_private.h"
>   
> -static int ethdev_logtype;
> +int ethdev_logtype;
>   
> -#define ethdev_log(level, fmt, ...) \
> -	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
> +#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
> +#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
>   
>   static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
>   struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
> @@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)
>   	return ret;
>   }
>   
> -/* attach the new device, then store port_id of the device */
>   int
> -rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
> +do_eth_dev_attach(const char *devargs, uint16_t *port_id)
>   {
>   	int current = rte_eth_dev_count_total();
>   	struct rte_devargs da;
> @@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
>   	return ret;
>   }
>   
> -/* detach the device, then store the name of the device */
>   int
> -rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
> +do_eth_dev_detach(uint16_t port_id)
>   {
>   	struct rte_device *dev;
>   	struct rte_bus *bus;
> +	int solid_release;
> +	int ret = 0;
> +
> +	dev = rte_eth_devices[port_id].device;
> +	if (dev == NULL)
> +		return -EINVAL;
> +
> +	bus = rte_bus_find_by_device(dev);
> +	if (bus == NULL)
> +		return -ENOENT;
> +
> +	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
> +	/**
> +	 * A private device on secondary need
> +	 * rte_eth_dev_release_port.
> +	 * 1) only vdev support private device.
> +	 * 2) private device has no empty devargs.
> +	 */

Is alignment correct/intended above?

> +		if (!strcmp(bus->name, "vdev") &&
> +			dev->devargs != NULL &&
> +			strlen(dev->devargs->args) > 0)
> +			solid_release = 1;
> +		else
> +			solid_release = 0;
> +	} else {
> +		solid_release = 0;
> +	}
> +
> +	ret = rte_eal_hotplug_remove(bus->name, dev->name);
> +	if (ret < 0)
> +		return ret;
> +
> +	if (solid_release)
> +		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
> +	else
> +		return rte_eth_dev_release_port_private(
> +			&rte_eth_devices[port_id]);
> +}
> +
> +/* attach the new device, then store port_id of the device */
> +int
> +rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
> +{
> +	struct eth_dev_mp_req req;
> +	int ret;
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +
> +		/**
> +		 * If secondary process, we just send request to primary
> +		 * to start the process.
> +		 */
> +		req.t = REQ_TYPE_ATTACH;
> +		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> +
> +		ret = eth_dev_request_to_primary(&req);
> +		if (ret) {
> +			ethdev_log(ERR,
> +				"Failed to send device attach request to primary\n");
> +			return ret;
> +		}
> +
> +		*port_id = req.port_id;
> +		return req.result;
> +	}
> +
> +	ret = do_eth_dev_attach(devargs, port_id);
> +	if (ret)
> +		return ret;
> +
> +	/* send attach request to seoncary */
> +	req.t = REQ_TYPE_ATTACH;
> +	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
> +	req.port_id = *port_id;
> +	ret = eth_dev_request_to_secondary(&req);
> +	if (ret) {
> +		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
> +		goto rollback;
> +	}
> +
> +	if (req.result)
> +		goto rollback;
> +
> +	return 0;
> +
> +rollback:
> +	/* send rollback request to secondary since some one fail to attach */
> +	req.t = REQ_TYPE_ATTACH_ROLLBACK;
> +	req.port_id = *port_id;
> +	eth_dev_request_to_secondary(&req);
> +
> +	do_eth_dev_detach(*port_id);
> +
> +	return -ENODEV;
> +}
> +
> +/* attach the new device, then store port_id of the device */
> +int
> +rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
> +{
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +		return -ENOTSUP;
> +
> +	return do_eth_dev_attach(devargs, port_id);
> +}
> +
> +/* detach the device, then store the name of the device */
> +int
> +rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
> +{
> +	struct eth_dev_mp_req req = {0};
> +	int ret;
>   	uint32_t dev_flags;
> -	int ret = -1;
>   
>   	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
>   
> @@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
>   		return -ENOTSUP;
>   	}
>   
> -	dev = rte_eth_devices[port_id].device;
> -	if (dev == NULL)
> -		return -EINVAL;
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		req.t = REQ_TYPE_DETACH;
> +		req.port_id = port_id;
>   
> -	bus = rte_bus_find_by_device(dev);
> -	if (bus == NULL)
> -		return -ENOENT;
> +		/**
> +		 * If secondary process, we just send request to primary
> +		 * to start the process.
> +		 */
> +		ret = eth_dev_request_to_primary(&req);
> +		if (ret) {
> +			ethdev_log(ERR,
> +				"Failed to send device detach request to primary\n");
> +			return ret;
> +		}
>   
> -	ret = rte_eal_hotplug_remove(bus->name, dev->name);
> -	if (ret < 0)
> +		return req.result;
> +	}
> +
> +	/* check pre_detach */
> +	req.t = REQ_TYPE_PRE_DETACH;
> +	req.port_id = port_id;
> +	ret = eth_dev_request_to_secondary(&req);
> +	if (ret) {
> +		ethdev_log(ERR,
> +			"Failed to send device pre-detach request to secondary\n");
> +		return ret;
> +	}
> +
> +	if (req.result) {
> +		ethdev_log(ERR,
> +			"Device is busy on secondary, can't be detached\n");
> +		return req.result;
> +	}
> +
> +	/* detach on seconary first */
> +	req.t = REQ_TYPE_DETACH;
> +	ret = eth_dev_request_to_secondary(&req);
> +	if (ret) {
> +		ethdev_log(ERR,
> +			"Failed to send device detach request to secondary\n");
> +		return ret;
> +	}
> +
> +	if (req.result)
> +		/**
> +		 * this should rarely happen, something wrong in secondary
> +		 * process, will not block primary detach.
> +		 */
> +		ethdev_log(ERR,
> +			"Failed to detach device on secondary process\n");
> +
> +	/* detach on primary */
> +	ret =  do_eth_dev_detach(port_id);
> +	if (ret)
>   		return ret;
>   
> -	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
>   	return 0;
>   }
>   
> +/* detach the device, then store the name of the device */
> +int
> +rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
> +{
> +	uint32_t dev_flags;
> +
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +		return -ENOTSUP;
> +
> +	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
> +
> +	dev_flags = rte_eth_devices[port_id].data->dev_flags;
> +	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
> +		ethdev_log(ERR,
> +			"Port %" PRIu16 " is bonded, cannot detach", port_id);
> +		return -ENOTSUP;
> +	}
> +
> +	return do_eth_dev_detach(port_id);
> +}
> +
>   static int
>   rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
>   {
> diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
> index 36e3984ea..813806e3c 100644
> --- a/lib/librte_ethdev/rte_ethdev.h
> +++ b/lib/librte_ethdev/rte_ethdev.h
> @@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
>   
>   /**
>    * Attach a new Ethernet device specified by arguments.
> + * In multi-process mode, it will sync with other process
> + * to make sure all processes attach the device, any
> + * failure on other process will rollback the action.
>    *
>    * @param devargs
>    *  A pointer to a strings array describing the new device
> @@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
>   int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
>   
>   /**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Attach a private Ethernet device specified by arguments.
> + * A private device is invisible to other process.
> + * Can only be invoked in secondary process.
> + *
> + * @param devargs
> + *  A pointer to a strings array describing the new device
> + *  to be attached. The strings should be a pci address like
> + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
> + * @param port_id
> + *  A pointer to a port identifier actually attached.
> + * @return
> + *  0 on success and port_id is filled, negative on error
> + */
> +int __rte_experimental
> +rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
> +
> +/**
>    * Detach a Ethernet device specified by port identifier.
>    * This function must be called when the device is in the
>    * closed state.
> + * In multi-process mode, it will sync with other process
> + * to detach the device.
>    *
>    * @param port_id
>    *   The port identifier of the device to detach.
> @@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
>   int rte_eth_dev_detach(uint16_t port_id, char *devname);
>   
>   /**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Detach a private Ethernet device specified by port identifier

Dot at the end is lost above.

> + * This function must be called when the device is in the
> + * closed state.
> + * Can only be invoked in secondary process.
> + *
> + * @param port_id
> + *   The port identifier of the device to detach.
> + * @param devname
> + *   A pointer to a buffer that will be filled with the device name.
> + *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
> + * @return
> + *  0 on success and devname is filled, negative on error
> + */
> +int __rte_experimental
> +rte_eth_dev_detach_private(uint16_t port_id, char *devname);
> +
> +/**
>    * Convert a numerical speed in Mbps to a bitmap flag that can be used in
>    * the bitmap link_speeds of the struct rte_eth_conf
>    *
> diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
> index 33d12b3a2..2cb6de745 100644
> --- a/lib/librte_ethdev/rte_ethdev_core.h
> +++ b/lib/librte_ethdev/rte_ethdev_core.h
> @@ -622,4 +622,9 @@ struct rte_eth_dev_data {
>    */
>   extern struct rte_eth_dev rte_eth_devices[];
>   
> +extern int ethdev_logtype;
> +#define ethdev_log(level, fmt, ...) \
> +	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
> +
> +
>   #endif /* _RTE_ETHDEV_CORE_H_ */
> diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
> index 40cf42b8a..fbcc03925 100644
> --- a/lib/librte_ethdev/rte_ethdev_version.map
> +++ b/lib/librte_ethdev/rte_ethdev_version.map
> @@ -217,9 +217,11 @@ EXPERIMENTAL {
>   	global:
>   
>   	rte_eth_devargs_parse;
> +	rte_eth_dev_attach_private;
>   	rte_eth_dev_count_total;
>   	rte_eth_dev_create;
>   	rte_eth_dev_destroy;
> +	rte_eth_dev_detach_private;
>   	rte_eth_dev_get_module_eeprom;
>   	rte_eth_dev_get_module_info;
>   	rte_eth_dev_is_removed;

Re: [dpdk-dev] [PATCH v7 04/19] ethdev: introduce device lock

[Andrew Rybchenko]

On 06/28/2018 03:56 PM, Qi Zhang wrote:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.

I think that locking deserves a bit more details on why it is needed.
When/why should it be used by applications or other libraries.
Right now the description is too generic and real usecases are unclear.
Should applications always lock device if some data cores are polling
its Rx/Tx queues? Does it imply that all apps which would like to be
hotplug-aware should be updated accordingly?
Do you have guidelines or is it too early stage for now?

> Aslo introduce the new API rte_eth_dev_lock_with_callback and
> rte_eth_dev_unlock_with callback to let application to register
> a callback function which will be invoked before a device is going
> to be detached, the return value of the function will decide if
> device will continue be detached or not, this support application
> to do condition check at runtime.

What should/will happen if two callbacks are registered and the
first says OK, but the second denies detach. The device will be
detached from the first callback point of view, but finally remains.

> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

Re: [dpdk-dev] [PATCH v7 01/19] ethdev: add function to release port in local process

[Zhang, Qi Z]

From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
Sent: Thursday, June 28, 2018 10:34 PM
To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender <narender.vangati@intel.com>
Subject: Re: [dpdk-dev] [PATCH v7 01/19] ethdev: add function to release port in local process

On 06/28/2018 03:56 PM, Qi Zhang wrote:

Add driver API rte_eth_release_port_private to support the

requirement that an ethdev only be released on secondary process,

so only local state be set to unused , share data will not be

Extra space before comma above.



reset so primary process can still use it.



Signed-off-by: Qi Zhang <qi.z.zhang@intel.com><mailto:qi.z.zhang@intel.com>

Acked-by: Remy Horton <remy.horton@intel.com><mailto:remy.horton@intel.com>

---

 lib/librte_ethdev/rte_ethdev.c        | 13 ++++++++++++-

 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++

 lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++

 3 files changed, 28 insertions(+), 1 deletion(-)



diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c

index a9977df97..2353fc921 100644

--- a/lib/librte_ethdev/rte_ethdev.c

+++ b/lib/librte_ethdev/rte_ethdev.c

<...>



@@ -371,7 +383,6 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)

        rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);



        eth_dev->state = RTE_ETH_DEV_UNUSED;

-

        memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));



        rte_spinlock_unlock(&rte_eth_dev_shared_data->ownership_lock);

Just a nit: above looks like unrelated change.

Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com><mailto:arybchenko@solarflare.com>

will fix both issue in v8.
Thanks,
Qi

Re: [dpdk-dev] [PATCH v7 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
Sent: Friday, June 29, 2018 12:33 AM
To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender <narender.vangati@intel.com>
Subject: Re: [dpdk-dev] [PATCH v7 03/19] ethdev: enable hotplug on multi-process

On 06/28/2018 03:56 PM, Qi Zhang wrote:

We are going to introduce the solution to handle different hotplug

cases in multi-process situation, it include below scenario:



1. Attach a share device from primary

2. Detach a share device from primary

3. Attach a share device from secondary

4. Detach a share device from secondary

5. Attach a private device from secondary

6. Detach a private device from secondary

7. Detach a share device from secondary privately

8. Attach a share device from secondary privately



In primary-secondary process model, we assume device is shared by default.

that means attach or detach a device on any process will broadcast to

all other processes through mp channel then device information will be

synchronized on all processes.



Any failure during attaching process will cause inconsistent status

between processes, so proper rollback action should be considered.

Also it is not safe to detach a share device when other process still use

it, so a handshake mechanism is introduced.



This patch covers the implementation of case 1,2,5,6,7,8.

Case 3,4 will be implemented on separate patch as well as handshake

mechanism.



Scenario for Case 1, 2:



attach device

a) primary attach the new device if failed goto h).

b) primary send attach sync request to all secondary.

c) secondary receive request and attach device and send reply.

d) primary check the reply if all success go to i).

e) primary send attach rollback sync request to all secondary.

f) secondary receive the request and detach device and send reply.

g) primary receive the reply and detach device as rollback action.

h) attach fail

i) attach success



detach device

a) primary perform pre-detach check, if device is locked, goto i).

b) primary send pre-detach sync request to all secondary.

c) secondary perform pre-detach check and send reply.

d) primary check the reply if any fail goto i).

e) primary send detach sync request to all secondary

f) secondary detach the device and send reply (assume no fail)

g) primary detach the device.

h) detach success

i) detach failed



Case 5, 6:

Secondary process can attach private device which only visible to itself,

in this case no IPC is involved, primary process is not allowed to have

private device so far.



Case 7, 8:

Secondary process can also temporally to detach a share device "privately"

then attach it back later, this action also not impact other processes.



APIs changes:



rte_eth_dev_attach and rte_eth_dev_attach are extended to support

share device attach/detach in primary-secondary process model, it will

be called in case 1,2,3,4.



New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are

introduced to cover case 5,6,7,8, this API can only be invoked in secondary

process.



Signed-off-by: Qi Zhang <qi.z.zhang@intel.com><mailto:qi.z.zhang@intel.com>

---

 lib/librte_ethdev/Makefile               |   1 +

 lib/librte_ethdev/ethdev_mp.c            | 261 +++++++++++++++++++++++++++++++

 lib/librte_ethdev/ethdev_mp.h            |  41 +++++

 lib/librte_ethdev/ethdev_private.h       |  39 +++++

 lib/librte_ethdev/meson.build            |   1 +

 lib/librte_ethdev/rte_ethdev.c           | 210 +++++++++++++++++++++++--

 lib/librte_ethdev/rte_ethdev.h           |  45 ++++++

 lib/librte_ethdev/rte_ethdev_core.h      |   5 +

 lib/librte_ethdev/rte_ethdev_version.map |   2 +

 9 files changed, 588 insertions(+), 17 deletions(-)

 create mode 100644 lib/librte_ethdev/ethdev_mp.c

 create mode 100644 lib/librte_ethdev/ethdev_mp.h

 create mode 100644 lib/librte_ethdev/ethdev_private.h



diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile

index c2f2f7d82..d0a059b83 100644

--- a/lib/librte_ethdev/Makefile

+++ b/lib/librte_ethdev/Makefile

@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map

 LIBABIVER := 9



 SRCS-y += rte_ethdev.c

+SRCS-y += ethdev_mp.c

 SRCS-y += rte_flow.c

 SRCS-y += rte_tm.c

 SRCS-y += rte_mtr.c

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c

new file mode 100644

index 000000000..0f9d8990d

--- /dev/null

+++ b/lib/librte_ethdev/ethdev_mp.c

@@ -0,0 +1,261 @@

+/* SPDX-License-Identifier: BSD-3-Clause

+ * Copyright(c) 2010-2018 Intel Corporation

+ */

+#include <rte_string_fns.h>

+#include <rte_alarm.h>

+

+#include "rte_ethdev_driver.h"

+#include "ethdev_mp.h"

+

+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */

+

+struct mp_reply_bundle {

+       struct rte_mp_msg msg;

+       void *peer;

+};

+

+static int detach_on_secondary(uint16_t port_id)

Looking at lib/librte_ethdev sources and 1.7.2 DPDK contribution guide:
 - "The function type should be on a line by itself preceding the function."
 - rte_ethdev always uses rte_eth_dev_ prefix, but strictly speaking it is not required
   by coding style





+{

+       struct rte_device *dev;

+       struct rte_bus *bus;

+       int ret = 0;

+

+       if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {

+               ethdev_log(ERR, "detach on secondary: invalid port %d\n",

+                          port_id);

+               return -ENODEV;

+       }

+

+       dev = rte_eth_devices[port_id].device;

+       if (dev == NULL)

+               return -EINVAL;

+

+       bus = rte_bus_find_by_device(dev);

+       if (bus == NULL)

+               return -ENOENT;

+

+       ret = rte_eal_hotplug_remove(bus->name, dev->name);

+       if (ret) {

+               ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",

+                          bus->name, dev->name);

+               return ret;

+       }

+

+       rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);

+       return ret;

+}

+

+static int attach_on_secondary(const char *devargs, uint16_t port_id)

+{

+       struct rte_devargs da;

+       int ret;

+

+       if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {

+               ethdev_log(ERR, "port %d already in used, failed to attach\n",

+                          port_id);

+               return -EINVAL;

+       }

+

+       memset(&da, 0, sizeof(da));

+

+       if (rte_devargs_parse(&da, "%s", devargs)) {

+               ethdev_log(ERR, "failed to parse devargs %s\n", devargs);

+               return -EINVAL;

+       }

+

+       ret = rte_eal_hotplug_add(da.bus->name, da.name, "");

+       if (ret) {

+               ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",

+                          da.bus->name, da.name);

+               free(da.args);

+               return ret;

+       }

+

+       if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {

+               ethdev_log(ERR,

+                       "failed to attach to port %d, this is a pmd issue\n",

+                          port_id);

+               free(da.args);

+               return -ENODEV;

+       }

+       free(da.args);

+       return 0;

+}

+

+static int

+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)

+{

+       RTE_SET_USED(msg);

+       RTE_SET_USED(peer);

+       return -ENOTSUP;

+}

+

+static void __handle_primary_request(void *param)

+{

+       struct mp_reply_bundle *bundle = param;

+       struct rte_mp_msg *msg = &bundle->msg;

+       const struct eth_dev_mp_req *req =

+               (const struct eth_dev_mp_req *)msg->param;

+       struct rte_mp_msg mp_resp;

+       struct eth_dev_mp_req *resp =

+               (struct eth_dev_mp_req *)mp_resp.param;

+       int ret = 0;

+

+       memset(&mp_resp, 0, sizeof(mp_resp));

+

+       switch (req->t) {

+       case REQ_TYPE_ATTACH:

+               ret = attach_on_secondary(req->devargs, req->port_id);

+               break;

+       case REQ_TYPE_PRE_DETACH:

+               ret = 0;

+               break;

+       case REQ_TYPE_DETACH:

+       case REQ_TYPE_ATTACH_ROLLBACK:

+               ret = detach_on_secondary(req->port_id);

+               break;

+       default:

+               ret = -EINVAL;

+       }

+

+       strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));

+       mp_resp.len_param = sizeof(*req);

+       memcpy(resp, req, sizeof(*resp));

+       resp->result = ret;

+       if (rte_mp_reply(&mp_resp, bundle->peer) < 0)

+               ethdev_log(ERR, "failed to send reply to primary request\n");

+

+       free(bundle->peer);

+       free(bundle);

+}

+

+static int

+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)

+{

+

+       struct rte_mp_msg mp_resp;

+       const struct eth_dev_mp_req *req =

+               (const struct eth_dev_mp_req *)msg->param;

+       struct eth_dev_mp_req *resp =

+               (struct eth_dev_mp_req *)mp_resp.param;

+       struct mp_reply_bundle *bundle;

+       int ret = 0;

+

+       memset(&mp_resp, 0, sizeof(mp_resp));

+       strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));

+       mp_resp.len_param = sizeof(*req);

+       memcpy(resp, req, sizeof(*resp));

+

+       bundle = calloc(1, sizeof(*bundle));

+       if (bundle == NULL) {

+               resp->result = -ENOMEM;

+               ret = rte_mp_reply(&mp_resp, peer);

+               if (ret) {

+                       ethdev_log(ERR, "failed to send reply to primary request\n");

+                       return ret;

+               }

+       }

+

+       bundle->msg = *msg;

+       /**

+        * We need to send reply on interrupt thread, but peer can't be

+        * parsed directly, so this is a temporal hack, need to be fixed

+        * when it is ready.

+        */

+       bundle->peer = (void *)strdup(peer);

+

+       /**

+        * We are at IPC callback thread, sync IPC is not allowed due to

+        * dead lock, so we delegate the task to interrupt thread.

+        */

+       ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);

+       if (ret) {

I think it is required to free  bundle and bundle->peer here.

[Qi]. Bundle and bundle->peer will be freed in __handle_primary_request.




+               resp->result = ret;

+               ret = rte_mp_reply(&mp_resp, peer);

+               if (ret) {

+                       ethdev_log(ERR, "failed to send reply to primary request\n");

+                       return ret;

+               }

+       }

+       return 0;

+}

+

+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)

+{

+       RTE_SET_USED(req);

+       return -ENOTSUP;

+}

+

+/**

+ * Request from primary to secondary.

+ *

+ * Be invoked when try to attach or detach a share device

+ * from primary process.

+ */

+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)

+{

+       struct rte_mp_msg mp_req;

+       struct rte_mp_reply mp_reply;

+       struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};

+       int ret;

+       int i;

+

+       memset(&mp_req, 0, sizeof(mp_req));

+       memcpy(mp_req.param, req, sizeof(*req));

+       mp_req.len_param = sizeof(*req);

+       strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));

+

+       ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);

+       if (ret) {

+               ethdev_log(ERR, "rte_mp_request_sync failed\n");

+               return ret;

+       }

+

+       if (mp_reply.nb_sent != mp_reply.nb_received) {

+               ethdev_log(ERR, "not all secondary reply\n");

+               return -1;

+       }

+

+       req->result = 0;

+       for (i = 0; i < mp_reply.nb_received; i++) {

+               struct eth_dev_mp_req *resp =

+                       (struct eth_dev_mp_req *)mp_reply.msgs[i].param;

+               if (resp->result) {

+                       req->result = resp->result;

+                       break;

+               }

+       }

+

+       return 0;

+}

+

+static int on_mp_init(void)

+{

+       int ret;

+

+       if (rte_eal_process_type() == RTE_PROC_PRIMARY) {

+               ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,

+                                         handle_secondary_request);

+               if (ret) {

+                       RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",

+                              ETH_DEV_MP_ACTION_REQUEST);

+                       return ret;

+               }

+       } else {

+               ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,

+                                         handle_primary_request);

+               if (ret) {

+                       RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",

+                              ETH_DEV_MP_ACTION_REQUEST);

+                       return ret;

+               }

+       }

+

+       return 0;

+}

+

+RTE_INIT(ethdev_mp_init)

+{

+       if (rte_eal_register_mp_init(on_mp_init))

+               RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");

+}

diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h

new file mode 100644

index 000000000..40be46c89

--- /dev/null

+++ b/lib/librte_ethdev/ethdev_mp.h

@@ -0,0 +1,41 @@

+/* SPDX-License-Identifier: BSD-3-Clause

+ * Copyright(c) 2018 Intel Corporation

+ */

+

+#ifndef _RTE_ETHDEV_MP_H_

+#define _RTE_ETHDEV_MP_H_

+

+#define MAX_DEV_ARGS_LEN 0x80

+

+#define ETH_DEV_MP_ACTION_REQUEST     "eth_dev_mp_request"

+#define ETH_DEV_MP_ACTION_RESPONSE    "eth_dev_mp_response"

+

+enum eth_dev_req_type {

+       REQ_TYPE_ATTACH,

+       REQ_TYPE_PRE_DETACH,

+       REQ_TYPE_DETACH,

+       REQ_TYPE_ATTACH_ROLLBACK,

+};

+

+struct eth_dev_mp_req {

+       enum eth_dev_req_type t;

+       char devargs[MAX_DEV_ARGS_LEN];

+       uint16_t port_id;

+       int result;

+};

+

+/**

+ * this is a synchronous wrapper for secondary process send

+ * request to primary process, this is invoked when an attach

+ * or detach request issued from primary.

+ */

+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);

I guess it is internal ethdev API, but nothing in the header says so.
The function description does not say so as well.
[Qi], I think it is not necessary to specify this is an internal API like other ethdev_xxx.h, the header file is
not exposed, so user will not see it.

Also right now even internal functions in ethdev have rte_ prefix and
[Qi], I don’t think this is a good idea.

I think it is good especially in the static libraries case.




+

+/**

+ * this is a synchronous wrapper for primary process send

+ * request to secondary process, this is invoked when an attach

+ * or detach request issued from secondary process.

+ */

+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);

+

+#endif

diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h

new file mode 100644

index 000000000..981e7de8a

--- /dev/null

+++ b/lib/librte_ethdev/ethdev_private.h

@@ -0,0 +1,39 @@

+/* SPDX-License-Identifier: BSD-3-Clause

+ * Copyright(c) 2010-2018 Intel Corporation

+ */

+

+#ifndef _ETHDEV_PRIVATE_H_

+#define _ETHDEV_PRIVATE_H_

+

+/**

+ * Attach a new Ethernet device in current process.

+ *

+ * @param devargs

+ *  A pointer to a strings array describing the new device

+ *  to be attached. The strings should be a pci address like

+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.

+ *

+ * @param port_id

+ *  A pointer to a port identifier actually attached.

+ *

+ * @return

+ *  0 on success and port_id is filled, negative on error

+ */

+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);

+

+/**

+ * Detach a Ethernet device in current process.

+ *

+ * @param port_id

+ *   The port identifier of the device to detach.

+ *

+ * @param devname

+ *   A pointer to a buffer that will be filled with the device name.

+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.

+ *

+ * @return

+ *  0 on success and devname is filled, negative on error

+ */

+int do_eth_dev_detach(uint16_t port_id);

+

+#endif

diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build

index aed5d2265..b60256855 100644

--- a/lib/librte_ethdev/meson.build

+++ b/lib/librte_ethdev/meson.build

@@ -5,6 +5,7 @@ name = 'ethdev'

 version = 9

 allow_experimental_apis = true

 sources = files('ethdev_profile.c',

+       'ethdev_mp.c'

        'rte_ethdev.c',

        'rte_flow.c',

        'rte_mtr.c',

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c

index 2353fc921..6c5f465a2 100644

--- a/lib/librte_ethdev/rte_ethdev.c

+++ b/lib/librte_ethdev/rte_ethdev.c

@@ -41,11 +41,13 @@

 #include "rte_ethdev.h"

 #include "rte_ethdev_driver.h"

 #include "ethdev_profile.h"

+#include "ethdev_mp.h"

+#include "ethdev_private.h"



-static int ethdev_logtype;

+int ethdev_logtype;



-#define ethdev_log(level, fmt, ...) \

-       rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)

+#define RTE_ETH_MP_ACTION_REQUEST     "rte_eth_mp_request"

+#define RTE_ETH_MP_ACTION_RESPONSE    "rte_eth_mp_response"



 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";

 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];

@@ -649,9 +651,8 @@ eth_err(uint16_t port_id, int ret)

        return ret;

 }



-/* attach the new device, then store port_id of the device */

 int

-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)

+do_eth_dev_attach(const char *devargs, uint16_t *port_id)

 {

        int current = rte_eth_dev_count_total();

        struct rte_devargs da;

@@ -696,14 +697,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)

        return ret;

 }



-/* detach the device, then store the name of the device */

 int

-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)

+do_eth_dev_detach(uint16_t port_id)

 {

        struct rte_device *dev;

        struct rte_bus *bus;

+       int solid_release;

+       int ret = 0;

+

+       dev = rte_eth_devices[port_id].device;

+       if (dev == NULL)

+               return -EINVAL;

+

+       bus = rte_bus_find_by_device(dev);

+       if (bus == NULL)

+               return -ENOENT;

+

+       if (rte_eal_process_type() == RTE_PROC_SECONDARY) {

+       /**

+        * A private device on secondary need

+        * rte_eth_dev_release_port.

+        * 1) only vdev support private device.

+        * 2) private device has no empty devargs.

+        */

Is alignment correct/intended above?

[Qi] Will fix






+               if (!strcmp(bus->name, "vdev") &&

+                       dev->devargs != NULL &&

+                       strlen(dev->devargs->args) > 0)

+                       solid_release = 1;

+               else

+                       solid_release = 0;

+       } else {

+               solid_release = 0;

+       }

+

+       ret = rte_eal_hotplug_remove(bus->name, dev->name);

+       if (ret < 0)

+               return ret;

+

+       if (solid_release)

+               return rte_eth_dev_release_port(&rte_eth_devices[port_id]);

+       else

+               return rte_eth_dev_release_port_private(

+                       &rte_eth_devices[port_id]);

+}

+

+/* attach the new device, then store port_id of the device */

+int

+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)

+{

+       struct eth_dev_mp_req req;

+       int ret;

+

+       if (rte_eal_process_type() != RTE_PROC_PRIMARY) {

+

+               /**

+                * If secondary process, we just send request to primary

+                * to start the process.

+                */

+               req.t = REQ_TYPE_ATTACH;

+               strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);

+

+               ret = eth_dev_request_to_primary(&req);

+               if (ret) {

+                       ethdev_log(ERR,

+                              "Failed to send device attach request to primary\n");

+                       return ret;

+               }

+

+               *port_id = req.port_id;

+               return req.result;

+       }

+

+       ret = do_eth_dev_attach(devargs, port_id);

+       if (ret)

+               return ret;

+

+       /* send attach request to seoncary */

+       req.t = REQ_TYPE_ATTACH;

+       strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);

+       req.port_id = *port_id;

+       ret = eth_dev_request_to_secondary(&req);

+       if (ret) {

+               ethdev_log(ERR, "Failed to send device attach request to secondary\n");

+               goto rollback;

+       }

+

+       if (req.result)

+               goto rollback;

+

+       return 0;

+

+rollback:

+       /* send rollback request to secondary since some one fail to attach */

+       req.t = REQ_TYPE_ATTACH_ROLLBACK;

+       req.port_id = *port_id;

+       eth_dev_request_to_secondary(&req);

+

+       do_eth_dev_detach(*port_id);

+

+       return -ENODEV;

+}

+

+/* attach the new device, then store port_id of the device */

+int

+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)

+{

+

+       if (rte_eal_process_type() == RTE_PROC_PRIMARY)

+               return -ENOTSUP;

+

+       return do_eth_dev_attach(devargs, port_id);

+}

+

+/* detach the device, then store the name of the device */

+int

+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)

+{

+       struct eth_dev_mp_req req = {0};

+       int ret;

        uint32_t dev_flags;

-       int ret = -1;



        RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);



@@ -714,22 +826,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)

                return -ENOTSUP;

        }



-       dev = rte_eth_devices[port_id].device;

-       if (dev == NULL)

-               return -EINVAL;

+       if (rte_eal_process_type() != RTE_PROC_PRIMARY) {

+               req.t = REQ_TYPE_DETACH;

+               req.port_id = port_id;



-       bus = rte_bus_find_by_device(dev);

-       if (bus == NULL)

-               return -ENOENT;

+               /**

+                * If secondary process, we just send request to primary

+                * to start the process.

+                */

+               ret = eth_dev_request_to_primary(&req);

+               if (ret) {

+                       ethdev_log(ERR,

+                              "Failed to send device detach request to primary\n");

+                       return ret;

+               }



-       ret = rte_eal_hotplug_remove(bus->name, dev->name);

-       if (ret < 0)

+               return req.result;

+       }

+

+       /* check pre_detach */

+       req.t = REQ_TYPE_PRE_DETACH;

+       req.port_id = port_id;

+       ret = eth_dev_request_to_secondary(&req);

+       if (ret) {

+               ethdev_log(ERR,

+                       "Failed to send device pre-detach request to secondary\n");

+               return ret;

+       }

+

+       if (req.result) {

+               ethdev_log(ERR,

+                      "Device is busy on secondary, can't be detached\n");

+               return req.result;

+       }

+

+       /* detach on seconary first */

+       req.t = REQ_TYPE_DETACH;

+       ret = eth_dev_request_to_secondary(&req);

+       if (ret) {

+               ethdev_log(ERR,

+                       "Failed to send device detach request to secondary\n");

+               return ret;

+       }

+

+       if (req.result)

+               /**

+                * this should rarely happen, something wrong in secondary

+                * process, will not block primary detach.

+                */

+               ethdev_log(ERR,

+                       "Failed to detach device on secondary process\n");

+

+       /* detach on primary */

+       ret =  do_eth_dev_detach(port_id);

+       if (ret)

                return ret;



-       rte_eth_dev_release_port(&rte_eth_devices[port_id]);

        return 0;

 }



+/* detach the device, then store the name of the device */

+int

+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)

+{

+       uint32_t dev_flags;

+

+       if (rte_eal_process_type() == RTE_PROC_PRIMARY)

+               return -ENOTSUP;

+

+       RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);

+

+       dev_flags = rte_eth_devices[port_id].data->dev_flags;

+       if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {

+               ethdev_log(ERR,

+                       "Port %" PRIu16 " is bonded, cannot detach", port_id);

+               return -ENOTSUP;

+       }

+

+       return do_eth_dev_detach(port_id);

+}

+

 static int

 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)

 {

diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h

index 36e3984ea..813806e3c 100644

--- a/lib/librte_ethdev/rte_ethdev.h

+++ b/lib/librte_ethdev/rte_ethdev.h

@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);



 /**

  * Attach a new Ethernet device specified by arguments.

+ * In multi-process mode, it will sync with other process

+ * to make sure all processes attach the device, any

+ * failure on other process will rollback the action.

  *

  * @param devargs

  *  A pointer to a strings array describing the new device

@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);

 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);



 /**

+ * @warning

+ * @b EXPERIMENTAL: this API may change without prior notice

+ *

+ * Attach a private Ethernet device specified by arguments.

+ * A private device is invisible to other process.

+ * Can only be invoked in secondary process.

+ *

+ * @param devargs

+ *  A pointer to a strings array describing the new device

+ *  to be attached. The strings should be a pci address like

+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.

+ * @param port_id

+ *  A pointer to a port identifier actually attached.

+ * @return

+ *  0 on success and port_id is filled, negative on error

+ */

+int __rte_experimental

+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);

+

+/**

  * Detach a Ethernet device specified by port identifier.

  * This function must be called when the device is in the

  * closed state.

+ * In multi-process mode, it will sync with other process

+ * to detach the device.

  *

  * @param port_id

  *   The port identifier of the device to detach.

@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);

 int rte_eth_dev_detach(uint16_t port_id, char *devname);



 /**

+ * @warning

+ * @b EXPERIMENTAL: this API may change without prior notice

+ *

+ * Detach a private Ethernet device specified by port identifier

Dot at the end is lost above.

[Qi] will fix

Thanks
Qi


+ * This function must be called when the device is in the

+ * closed state.

+ * Can only be invoked in secondary process.

+ *

+ * @param port_id

+ *   The port identifier of the device to detach.

+ * @param devname

+ *   A pointer to a buffer that will be filled with the device name.

+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.

+ * @return

+ *  0 on success and devname is filled, negative on error

+ */

+int __rte_experimental

+rte_eth_dev_detach_private(uint16_t port_id, char *devname);

+

+/**

  * Convert a numerical speed in Mbps to a bitmap flag that can be used in

  * the bitmap link_speeds of the struct rte_eth_conf

  *

diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h

index 33d12b3a2..2cb6de745 100644

--- a/lib/librte_ethdev/rte_ethdev_core.h

+++ b/lib/librte_ethdev/rte_ethdev_core.h

@@ -622,4 +622,9 @@ struct rte_eth_dev_data {

  */

 extern struct rte_eth_dev rte_eth_devices[];



+extern int ethdev_logtype;

+#define ethdev_log(level, fmt, ...) \

+       rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)

+

+

 #endif /* _RTE_ETHDEV_CORE_H_ */

diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map

index 40cf42b8a..fbcc03925 100644

--- a/lib/librte_ethdev/rte_ethdev_version.map

+++ b/lib/librte_ethdev/rte_ethdev_version.map

@@ -217,9 +217,11 @@ EXPERIMENTAL {

        global:



        rte_eth_devargs_parse;

+       rte_eth_dev_attach_private;

        rte_eth_dev_count_total;

        rte_eth_dev_create;

        rte_eth_dev_destroy;

+       rte_eth_dev_detach_private;

        rte_eth_dev_get_module_eeprom;

        rte_eth_dev_get_module_info;

        rte_eth_dev_is_removed;

Re: [dpdk-dev] [PATCH v7 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
Sent: Friday, June 29, 2018 12:47 AM
To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>
Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton, Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender <narender.vangati@intel.com>
Subject: Re: [dpdk-dev] [PATCH v7 04/19] ethdev: introduce device lock

On 06/28/2018 03:56 PM, Qi Zhang wrote:

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let

application lock or unlock on specific ethdev, a locked device

can't be detached, this help applicaiton to prevent unexpected

device detaching, especially in multi-process envrionment.

I think that locking deserves a bit more details on why it is needed.
When/why should it be used by applications or other libraries.
Right now the description is too generic and real usecases are unclear.

[Qi], the typical use case is described in cover letter.
Primary works as resource management process and secondary process do the network stuff.
And we need handshake to prevent a running device be detached unexpected.
So we introduce the lock/unlock API for this requirement,

I can add these information in commit log, if you think it’s necessary.


Should applications always lock device if some data cores are polling
its Rx/Tx queues?
[Qi] Application should know if it is possible to detach a running device on a separate process or separate thread.
The lock API helps application to handle such kind of case.

Does it imply that all apps which would like to be
hotplug-aware should be updated accordingly?
Do you have guidelines or is it too early stage for now?

Basically we didn’t break any thing what we already have, we just provide a new option which looks helpful to simplify
the development of application that need to handle hotplug.


Aslo introduce the new API rte_eth_dev_lock_with_callback and

rte_eth_dev_unlock_with callback to let application to register

a callback function which will be invoked before a device is going

to be detached, the return value of the function will decide if

device will continue be detached or not, this support application

to do condition check at runtime.

What should/will happen if two callbacks are registered and the
first says OK, but the second denies detach. The device will be
detached from the first callback point of view, but finally remains.

[Qi] Though I’m not very sure if this will be the real case, but if that happens, one method for
the lock owner to know that a device is exactly detached or not is also register a callback for
RTE_ETH_EVENT_DESTROY, so it will be notified when device is detached and do some following cleanup

But yes there will be an issue if some resource is freed during the first callback, it will not rollback.
So, application should consider this.

Thanks
Qi



Signed-off-by: Qi Zhang <qi.z.zhang@intel.com><mailto:qi.z.zhang@intel.com>

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com><mailto:anatoly.burakov@intel.com>

[dpdk-dev] [PATCH v8 00/19] enable hotplug on multi-process

[Qi Zhang]

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also, it is not safe to detach a share device when other process still
use it, so a handshake mechanism is introduced.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

Case 5, 6:
Secondary process can attach private device which only visible to
itself, in this case no IPC is involved, primary process is not allowed
to have private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device
"privately" then attach it back later, this action also not impact other
processes.

APIs chenages:
==============

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in
secondary process.

New API rte_eth_dev_lock and rte_eth_dev_unlock are introduced to let
application lock or unlock on specific ethdev, a locked device
can't be detached. This help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.
Aslo the new API let application to register a callback function
which will be invoked before a device is going to be detached,
the return value of the function will decide if device will continue
be detached or not, this support application to do condition check
at runtime.

PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Limitation:
===========

1. The solution does not cover the case that primary process exit while
   secondary processes still be active. Though this is not a typial use
   case, but if this happens:
   a. secondary process can't attach / detach any shared device since no
      primary exist.
   b. secondary process still can attach / detach private device.
   c. secondary process still can detach a share device privately but may
      not attach it back, that ethdev slot will become zombie slot.

2. So for, for PCI bus, case 5,6 is not supported. PCI bus scan/probe
   mechanism can be improved to support attach private device on secondary
   process, but this is not the scope of this patchset.

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  eal: enable multi process init callback
  ethdev: enable hotplug on multi-process
  ethdev: introduce device lock
  ethdev: support attach or detach share device from secondary
  ethdev: support attach private device as first
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst       |  20 ++
 drivers/net/af_packet/rte_eth_af_packet.c    |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c       |  11 +
 drivers/net/i40e/i40e_ethdev.c               |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c             |   3 +
 drivers/net/kni/rte_eth_kni.c                |  11 +
 drivers/net/null/rte_eth_null.c              |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c       |  16 +
 drivers/net/pcap/rte_eth_pcap.c              |  15 +-
 drivers/net/softnic/rte_eth_softnic.c        |  19 +-
 drivers/net/tap/rte_eth_tap.c                |  17 +-
 drivers/net/vhost/rte_eth_vhost.c            |  11 +
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 ++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/common/eal_common_proc.c      |  57 +++-
 lib/librte_eal/common/eal_private.h          |   5 +
 lib/librte_eal/common/include/rte_eal.h      |  34 +++
 lib/librte_eal/linuxapp/eal/eal.c            |   2 +
 lib/librte_eal/rte_eal_version.map           |   1 +
 lib/librte_ethdev/Makefile                   |   2 +
 lib/librte_ethdev/ethdev_lock.c              | 140 +++++++++
 lib/librte_ethdev/ethdev_lock.h              |  31 ++
 lib/librte_ethdev/ethdev_mp.c                | 431 +++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h                |  42 +++
 lib/librte_ethdev/ethdev_private.h           |  42 +++
 lib/librte_ethdev/meson.build                |   2 +
 lib/librte_ethdev/rte_ethdev.c               | 311 +++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h               | 169 +++++++++++
 lib/librte_ethdev/rte_ethdev_core.h          |   5 +
 lib/librte_ethdev/rte_ethdev_driver.h        |  13 +
 lib/librte_ethdev/rte_ethdev_pci.h           |   3 +
 lib/librte_ethdev/rte_ethdev_version.map     |   4 +
 35 files changed, 1840 insertions(+), 37 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

-- 
2.13.6

[dpdk-dev] [PATCH v8 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
requirement that an ethdev only be released on secondary process,
so only local state be set to unused, share data will not be
reset so primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 13 +++++++++++++
 lib/librte_ethdev/rte_ethdev_pci.h    |  3 +++
 3 files changed, 28 insertions(+)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..49c27223d 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in local process, only set to ethdev
+ * state to unused, but not reset share data since it assume other process
+ * is still using it, typically it is called by secondary process.
+ *
+ * @param eth_dev
+ * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..eeb944146 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,9 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback

[Qi Zhang]

Introduce new API rte_eal_register_mp_init that help to register
a callback function which will be invoked right after multi-process
channel be established (rte_mp_channel_init). Typically the API
will be used by other module that want it's mp channel action callbacks
can be registered during rte_eal_init automatically.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/eal_common_proc.c | 57 +++++++++++++++++++++++++++++++--
 lib/librte_eal/common/eal_private.h     |  5 +++
 lib/librte_eal/common/include/rte_eal.h | 34 ++++++++++++++++++++
 lib/librte_eal/linuxapp/eal/eal.c       |  2 ++
 lib/librte_eal/rte_eal_version.map      |  1 +
 5 files changed, 97 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c
index f010ef59e..f6d7c83e4 100644
--- a/lib/librte_eal/common/eal_common_proc.c
+++ b/lib/librte_eal/common/eal_common_proc.c
@@ -619,11 +619,47 @@ unlink_sockets(const char *filter)
 	return 0;
 }
 
+struct mp_init_entry {
+	TAILQ_ENTRY(mp_init_entry) next;
+	rte_eal_mp_init_callback_t callback;
+};
+
+TAILQ_HEAD(mp_init_entry_list, mp_init_entry);
+static struct mp_init_entry_list mp_init_entry_list =
+	TAILQ_HEAD_INITIALIZER(mp_init_entry_list);
+
+static int process_mp_init_callbacks(void)
+{
+	struct mp_init_entry *entry;
+	int ret;
+
+	TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+		ret = entry->callback();
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback)
+{
+	struct mp_init_entry *entry = calloc(1, sizeof(struct mp_init_entry));
+
+	if (entry == NULL)
+		return -ENOMEM;
+
+	entry->callback = callback;
+	TAILQ_INSERT_TAIL(&mp_init_entry_list, entry, next);
+
+	return 0;
+}
+
 int
 rte_mp_channel_init(void)
 {
 	char path[PATH_MAX];
-	int dir_fd;
+	int dir_fd, ret;
 	pthread_t mp_handle_tid, async_reply_handle_tid;
 
 	/* create filter path */
@@ -686,7 +722,24 @@ rte_mp_channel_init(void)
 	flock(dir_fd, LOCK_UN);
 	close(dir_fd);
 
-	return 0;
+	ret = process_mp_init_callbacks();
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to process mp init callbacks\n");
+
+	return ret;
+}
+
+void rte_mp_init_callback_cleanup(void)
+{
+	struct mp_init_entry *entry;
+
+	while (!TAILQ_EMPTY(&mp_init_entry_list)) {
+		TAILQ_FOREACH(entry, &mp_init_entry_list, next) {
+			TAILQ_REMOVE(&mp_init_entry_list, entry, next);
+			free(entry);
+			break;
+		}
+	}
 }
 
 /**
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..bc230ee23 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -247,6 +247,11 @@ struct rte_bus *rte_bus_find_by_device_name(const char *str);
 int rte_mp_channel_init(void);
 
 /**
+ * Cleanup all mp channel init callbacks.
+ */
+void rte_mp_init_callback_cleanup(void);
+
+/**
  * Internal Executes all the user application registered callbacks for
  * the specific device. It is for DPDK internal user only. User
  * application should not call it directly.
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 8de5d69e8..506f17f34 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -512,6 +512,40 @@ __rte_deprecated
 const char *
 rte_eal_mbuf_default_mempool_ops(void);
 
+/**
+ * Callback function right after multi-process channel be established.
+ * Typical implementation of these functions is to register mp channel
+ * action callbacks
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+typedef int (*rte_eal_mp_init_callback_t)(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register a callback function that will be invoked right after
+ * multi-process channel be established (rte_mp_channel_init). Typically
+ * the function is used by other module that want it's mp channel
+ * action callbacks can be registered during rte_eal_init automatically.
+ *
+ * @note
+ *   This function only take effect when be called before rte_eal_init,
+ *   and all registered callback will be clear during rte_eal_cleanup.
+ *
+ * @param callback
+ *   function be called at that moment.
+ *
+ * @return
+ *  - 0 on success.
+ *  - (<0) on failure.
+ */
+int __rte_experimental
+rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..45cccff7e 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -1048,6 +1048,8 @@ int __rte_experimental
 rte_eal_cleanup(void)
 {
 	rte_service_finalize();
+	rte_mp_init_callback_cleanup();
+
 	return 0;
 }
 
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index f7dd0e7bc..67e3548e8 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -260,6 +260,7 @@ EXPERIMENTAL {
 	rte_eal_hotplug_add;
 	rte_eal_hotplug_remove;
 	rte_eal_mbuf_user_pool_ops;
+	rte_eal_register_mp_init;
 	rte_fbarray_attach;
 	rte_fbarray_destroy;
 	rte_fbarray_detach;
-- 
2.13.6

[dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle different hotplug
cases in multi-process situation, it include below scenario:

1. Attach a share device from primary
2. Detach a share device from primary
3. Attach a share device from secondary
4. Detach a share device from secondary
5. Attach a private device from secondary
6. Detach a private device from secondary
7. Detach a share device from secondary privately
8. Attach a share device from secondary privately

In primary-secondary process model, we assume device is shared by default.
that means attach or detach a device on any process will broadcast to
all other processes through mp channel then device information will be
synchronized on all processes.

Any failure during attaching process will cause inconsistent status
between processes, so proper rollback action should be considered.
Also it is not safe to detach a share device when other process still use
it, so a handshake mechanism is introduced.

This patch covers the implementation of case 1,2,5,6,7,8.
Case 3,4 will be implemented on separate patch as well as handshake
mechanism.

Scenario for Case 1, 2:

attach device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Case 5, 6:
Secondary process can attach private device which only visible to itself,
in this case no IPC is involved, primary process is not allowed to have
private device so far.

Case 7, 8:
Secondary process can also temporally to detach a share device "privately"
then attach it back later, this action also not impact other processes.

APIs changes:

rte_eth_dev_attach and rte_eth_dev_attach are extended to support
share device attach/detach in primary-secondary process model, it will
be called in case 1,2,3,4.

New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are
introduced to cover case 5,6,7,8, this API can only be invoked in secondary
process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/Makefile               |   1 +
 lib/librte_ethdev/ethdev_mp.c            | 261 +++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_mp.h            |  41 +++++
 lib/librte_ethdev/ethdev_private.h       |  39 +++++
 lib/librte_ethdev/meson.build            |   1 +
 lib/librte_ethdev/rte_ethdev.c           | 210 +++++++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev.h           |  45 ++++++
 lib/librte_ethdev/rte_ethdev_core.h      |   5 +
 lib/librte_ethdev/rte_ethdev_version.map |   2 +
 9 files changed, 588 insertions(+), 17 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_mp.c
 create mode 100644 lib/librte_ethdev/ethdev_mp.h
 create mode 100644 lib/librte_ethdev/ethdev_private.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index c2f2f7d82..d0a059b83 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -19,6 +19,7 @@ EXPORT_MAP := rte_ethdev_version.map
 LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
+SRCS-y += ethdev_mp.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
new file mode 100644
index 000000000..0f9d8990d
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -0,0 +1,261 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+#include <rte_string_fns.h>
+#include <rte_alarm.h>
+
+#include "rte_ethdev_driver.h"
+#include "ethdev_mp.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int detach_on_secondary(uint16_t port_id)
+{
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "detach on secondary: invalid port %d\n",
+			   port_id);
+		return -ENODEV;
+	}
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret) {
+		ethdev_log(ERR, "failed to hot unplug bus: %s, device:%s\n",
+			   bus->name, dev->name);
+		return ret;
+	}
+
+	rte_eth_dev_release_port_private(&rte_eth_devices[port_id]);
+	return ret;
+}
+
+static int attach_on_secondary(const char *devargs, uint16_t port_id)
+{
+	struct rte_devargs da;
+	int ret;
+
+	if (rte_eth_devices[port_id].state != RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR, "port %d already in used, failed to attach\n",
+			   port_id);
+		return -EINVAL;
+	}
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", devargs)) {
+		ethdev_log(ERR, "failed to parse devargs %s\n", devargs);
+		return -EINVAL;
+	}
+
+	ret = rte_eal_hotplug_add(da.bus->name, da.name, "");
+	if (ret) {
+		ethdev_log(ERR, "failed to hotplug bus:%s, device:%s\n",
+			   da.bus->name, da.name);
+		free(da.args);
+		return ret;
+	}
+
+	if (rte_eth_devices[port_id].state == RTE_ETH_DEV_UNUSED) {
+		ethdev_log(ERR,
+			"failed to attach to port %d, this is a pmd issue\n",
+			   port_id);
+		free(da.args);
+		return -ENODEV;
+	}
+	free(da.args);
+	return 0;
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case REQ_TYPE_ATTACH:
+		ret = attach_on_secondary(req->devargs, req->port_id);
+		break;
+	case REQ_TYPE_PRE_DETACH:
+		ret = 0;
+		break;
+	case REQ_TYPE_DETACH:
+	case REQ_TYPE_ATTACH_ROLLBACK:
+		ret = detach_on_secondary(req->port_id);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		ethdev_log(ERR, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+
+	struct rte_mp_msg mp_resp;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			ethdev_log(ERR, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+/**
+ * Request from primary to secondary.
+ *
+ * Be invoked when try to attach or detach a share device
+ * from primary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		ethdev_log(ERR, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eth_dev_mp_req *resp =
+			(struct eth_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+static int on_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST,
+					   handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				ETH_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+RTE_INIT(ethdev_mp_init)
+{
+	if (rte_eal_register_mp_init(on_mp_init))
+		RTE_LOG(ERR, EAL, "ethdev mp channel init failed\n");
+}
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
new file mode 100644
index 000000000..40be46c89
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_MP_H_
+#define _RTE_ETHDEV_MP_H_
+
+#define MAX_DEV_ARGS_LEN 0x80
+
+#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
+#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
+
+enum eth_dev_req_type {
+	REQ_TYPE_ATTACH,
+	REQ_TYPE_PRE_DETACH,
+	REQ_TYPE_DETACH,
+	REQ_TYPE_ATTACH_ROLLBACK,
+};
+
+struct eth_dev_mp_req {
+	enum eth_dev_req_type t;
+	char devargs[MAX_DEV_ARGS_LEN];
+	uint16_t port_id;
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary.
+ */
+int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
new file mode 100644
index 000000000..981e7de8a
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2010-2018 Intel Corporation
+ */
+
+#ifndef _ETHDEV_PRIVATE_H_
+#define _ETHDEV_PRIVATE_H_
+
+/**
+ * Attach a new Ethernet device in current process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ *
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ *
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
+
+/**
+ * Detach a Ethernet device in current process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ *
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ *
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int do_eth_dev_detach(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index aed5d2265..b60256855 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -5,6 +5,7 @@ name = 'ethdev'
 version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
+	'ethdev_mp.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 52a97694c..91d9f9a37 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -41,11 +41,13 @@
 #include "rte_ethdev.h"
 #include "rte_ethdev_driver.h"
 #include "ethdev_profile.h"
+#include "ethdev_mp.h"
+#include "ethdev_private.h"
 
-static int ethdev_logtype;
+int ethdev_logtype;
 
-#define ethdev_log(level, fmt, ...) \
-	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+#define RTE_ETH_MP_ACTION_REQUEST	"rte_eth_mp_request"
+#define RTE_ETH_MP_ACTION_RESPONSE	"rte_eth_mp_response"
 
 static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data";
 struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS];
@@ -650,9 +652,8 @@ eth_err(uint16_t port_id, int ret)
 	return ret;
 }
 
-/* attach the new device, then store port_id of the device */
 int
-rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+do_eth_dev_attach(const char *devargs, uint16_t *port_id)
 {
 	int current = rte_eth_dev_count_total();
 	struct rte_devargs da;
@@ -697,14 +698,125 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 	return ret;
 }
 
-/* detach the device, then store the name of the device */
 int
-rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+do_eth_dev_detach(uint16_t port_id)
 {
 	struct rte_device *dev;
 	struct rte_bus *bus;
+	int solid_release;
+	int ret = 0;
+
+	dev = rte_eth_devices[port_id].device;
+	if (dev == NULL)
+		return -EINVAL;
+
+	bus = rte_bus_find_by_device(dev);
+	if (bus == NULL)
+		return -ENOENT;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+		/**
+		 * A private device on secondary need
+		 * rte_eth_dev_release_port.
+		 * 1) only vdev support private device.
+		 * 2) private device has no empty devargs.
+		 */
+		if (!strcmp(bus->name, "vdev") &&
+			dev->devargs != NULL &&
+			strlen(dev->devargs->args) > 0)
+			solid_release = 1;
+		else
+			solid_release = 0;
+	} else {
+		solid_release = 0;
+	}
+
+	ret = rte_eal_hotplug_remove(bus->name, dev->name);
+	if (ret < 0)
+		return ret;
+
+	if (solid_release)
+		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
+	else
+		return rte_eth_dev_release_port_private(
+			&rte_eth_devices[port_id]);
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
+{
+	struct eth_dev_mp_req req;
+	int ret;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		req.t = REQ_TYPE_ATTACH;
+		strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device attach request to primary\n");
+			return ret;
+		}
+
+		*port_id = req.port_id;
+		return req.result;
+	}
+
+	ret = do_eth_dev_attach(devargs, port_id);
+	if (ret)
+		return ret;
+
+	/* send attach request to seoncary */
+	req.t = REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, MAX_DEV_ARGS_LEN);
+	req.port_id = *port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR, "Failed to send device attach request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	return 0;
+
+rollback:
+	/* send rollback request to secondary since some one fail to attach */
+	req.t = REQ_TYPE_ATTACH_ROLLBACK;
+	req.port_id = *port_id;
+	eth_dev_request_to_secondary(&req);
+
+	do_eth_dev_detach(*port_id);
+
+	return -ENODEV;
+}
+
+/* attach the new device, then store port_id of the device */
+int
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
+{
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	return do_eth_dev_attach(devargs, port_id);
+}
+
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
+{
+	struct eth_dev_mp_req req = {0};
+	int ret;
 	uint32_t dev_flags;
-	int ret = -1;
 
 	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
 
@@ -715,22 +827,86 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
-	dev = rte_eth_devices[port_id].device;
-	if (dev == NULL)
-		return -EINVAL;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		req.t = REQ_TYPE_DETACH;
+		req.port_id = port_id;
 
-	bus = rte_bus_find_by_device(dev);
-	if (bus == NULL)
-		return -ENOENT;
+		/**
+		 * If secondary process, we just send request to primary
+		 * to start the process.
+		 */
+		ret = eth_dev_request_to_primary(&req);
+		if (ret) {
+			ethdev_log(ERR,
+				"Failed to send device detach request to primary\n");
+			return ret;
+		}
 
-	ret = rte_eal_hotplug_remove(bus->name, dev->name);
-	if (ret < 0)
+		return req.result;
+	}
+
+	/* check pre_detach */
+	req.t = REQ_TYPE_PRE_DETACH;
+	req.port_id = port_id;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device pre-detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		ethdev_log(ERR,
+			"Device is busy on secondary, can't be detached\n");
+		return req.result;
+	}
+
+	/* detach on seconary first */
+	req.t = REQ_TYPE_DETACH;
+	ret = eth_dev_request_to_secondary(&req);
+	if (ret) {
+		ethdev_log(ERR,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result)
+		/**
+		 * this should rarely happen, something wrong in secondary
+		 * process, will not block primary detach.
+		 */
+		ethdev_log(ERR,
+			"Failed to detach device on secondary process\n");
+
+	/* detach on primary */
+	ret =  do_eth_dev_detach(port_id);
+	if (ret)
 		return ret;
 
-	rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	return 0;
 }
 
+/* detach the device, then store the name of the device */
+int
+rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
+{
+	uint32_t dev_flags;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
+
+	dev_flags = rte_eth_devices[port_id].data->dev_flags;
+	if (dev_flags & RTE_ETH_DEV_BONDED_SLAVE) {
+		ethdev_log(ERR,
+			"Port %" PRIu16 " is bonded, cannot detach", port_id);
+		return -ENOTSUP;
+	}
+
+	return do_eth_dev_detach(port_id);
+}
+
 static int
 rte_eth_dev_rx_queue_config(struct rte_eth_dev *dev, uint16_t nb_queues)
 {
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 36e3984ea..4d207e2fe 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -1462,6 +1462,9 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 
 /**
  * Attach a new Ethernet device specified by arguments.
+ * In multi-process mode, it will sync with other process
+ * to make sure all processes attach the device, any
+ * failure on other process will rollback the action.
  *
  * @param devargs
  *  A pointer to a strings array describing the new device
@@ -1475,9 +1478,31 @@ uint16_t __rte_experimental rte_eth_dev_count_total(void);
 int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Attach a private Ethernet device specified by arguments.
+ * A private device is invisible to other process.
+ * Can only be invoked in secondary process.
+ *
+ * @param devargs
+ *  A pointer to a strings array describing the new device
+ *  to be attached. The strings should be a pci address like
+ *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
+ * @param port_id
+ *  A pointer to a port identifier actually attached.
+ * @return
+ *  0 on success and port_id is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id);
+
+/**
  * Detach a Ethernet device specified by port identifier.
  * This function must be called when the device is in the
  * closed state.
+ * In multi-process mode, it will sync with other process
+ * to detach the device.
  *
  * @param port_id
  *   The port identifier of the device to detach.
@@ -1490,6 +1515,26 @@ int rte_eth_dev_attach(const char *devargs, uint16_t *port_id);
 int rte_eth_dev_detach(uint16_t port_id, char *devname);
 
 /**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Detach a private Ethernet device specified by port identifier.
+ * This function must be called when the device is in the
+ * closed state.
+ * Can only be invoked in secondary process.
+ *
+ * @param port_id
+ *   The port identifier of the device to detach.
+ * @param devname
+ *   A pointer to a buffer that will be filled with the device name.
+ *   This buffer must be at least RTE_DEV_NAME_MAX_LEN long.
+ * @return
+ *  0 on success and devname is filled, negative on error
+ */
+int __rte_experimental
+rte_eth_dev_detach_private(uint16_t port_id, char *devname);
+
+/**
  * Convert a numerical speed in Mbps to a bitmap flag that can be used in
  * the bitmap link_speeds of the struct rte_eth_conf
  *
diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h
index 33d12b3a2..2cb6de745 100644
--- a/lib/librte_ethdev/rte_ethdev_core.h
+++ b/lib/librte_ethdev/rte_ethdev_core.h
@@ -622,4 +622,9 @@ struct rte_eth_dev_data {
  */
 extern struct rte_eth_dev rte_eth_devices[];
 
+extern int ethdev_logtype;
+#define ethdev_log(level, fmt, ...) \
+	rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__)
+
+
 #endif /* _RTE_ETHDEV_CORE_H_ */
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 40cf42b8a..fbcc03925 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -217,9 +217,11 @@ EXPERIMENTAL {
 	global:
 
 	rte_eth_devargs_parse;
+	rte_eth_dev_attach_private;
 	rte_eth_dev_count_total;
 	rte_eth_dev_create;
 	rte_eth_dev_destroy;
+	rte_eth_dev_detach_private;
 	rte_eth_dev_get_module_eeprom;
 	rte_eth_dev_get_module_info;
 	rte_eth_dev_is_removed;
-- 
2.13.6

[dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Qi Zhang]

Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
application lock or unlock on specific ethdev, a locked device
can't be detached, this help applicaiton to prevent unexpected
device detaching, especially in multi-process envrionment.

Aslo introduce the new API rte_eth_dev_lock_with_callback and
rte_eth_dev_unlock_with callback to let application to register
a callback function which will be invoked before a device is going
to be detached, the return value of the function will decide if
device will continue be detached or not, this support application
to do condition check at runtime.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_ethdev/Makefile               |   1 +
 lib/librte_ethdev/ethdev_lock.c          | 140 +++++++++++++++++++++++++++++++
 lib/librte_ethdev/ethdev_lock.h          |  31 +++++++
 lib/librte_ethdev/ethdev_mp.c            |   3 +-
 lib/librte_ethdev/meson.build            |   1 +
 lib/librte_ethdev/rte_ethdev.c           |  60 ++++++++++++-
 lib/librte_ethdev/rte_ethdev.h           | 124 +++++++++++++++++++++++++++
 lib/librte_ethdev/rte_ethdev_version.map |   2 +
 8 files changed, 360 insertions(+), 2 deletions(-)
 create mode 100644 lib/librte_ethdev/ethdev_lock.c
 create mode 100644 lib/librte_ethdev/ethdev_lock.h

diff --git a/lib/librte_ethdev/Makefile b/lib/librte_ethdev/Makefile
index d0a059b83..62bef03fc 100644
--- a/lib/librte_ethdev/Makefile
+++ b/lib/librte_ethdev/Makefile
@@ -20,6 +20,7 @@ LIBABIVER := 9
 
 SRCS-y += rte_ethdev.c
 SRCS-y += ethdev_mp.c
+SRCS-y += ethdev_lock.c
 SRCS-y += rte_flow.c
 SRCS-y += rte_tm.c
 SRCS-y += rte_mtr.c
diff --git a/lib/librte_ethdev/ethdev_lock.c b/lib/librte_ethdev/ethdev_lock.c
new file mode 100644
index 000000000..6379519e3
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.c
@@ -0,0 +1,140 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include "ethdev_lock.h"
+
+struct lock_entry {
+	TAILQ_ENTRY(lock_entry) next;
+	rte_eth_dev_lock_callback_t callback;
+	uint16_t port_id;
+	void *user_args;
+	int ref_count;
+};
+
+TAILQ_HEAD(lock_entry_list, lock_entry);
+static struct lock_entry_list lock_entry_list =
+	TAILQ_HEAD_INITIALIZER(lock_entry_list);
+static rte_spinlock_t lock_entry_lock = RTE_SPINLOCK_INITIALIZER;
+
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le == NULL) {
+		le = calloc(1, sizeof(struct lock_entry));
+		if (le == NULL) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -ENOMEM;
+		}
+		le->callback = callback;
+		le->port_id = port_id;
+		le->user_args = user_args;
+		TAILQ_INSERT_TAIL(&lock_entry_list, le, next);
+	}
+	le->ref_count++;
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
+
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id &&
+		    le->callback == callback &&
+		    le->user_args == user_args)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return ret;
+}
+
+static int clean_lock_callback_one(uint16_t port_id)
+{
+	struct lock_entry *le;
+	int ret = 0;
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id == port_id)
+			break;
+	}
+
+	if (le != NULL) {
+		le->ref_count--;
+		if (le->ref_count == 0) {
+			TAILQ_REMOVE(&lock_entry_list, le, next);
+			free(le);
+		}
+	} else {
+		ret = -ENOENT;
+	}
+
+	return ret;
+
+}
+
+void clean_lock_callback(uint16_t port_id)
+{
+	int ret;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	for (;;) {
+		ret = clean_lock_callback_one(port_id);
+		if (ret == -ENOENT)
+			break;
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+}
+
+int process_lock_callbacks(uint16_t port_id)
+{
+	struct lock_entry *le;
+
+	rte_spinlock_lock(&lock_entry_lock);
+
+	TAILQ_FOREACH(le, &lock_entry_list, next) {
+		if (le->port_id != port_id)
+			continue;
+
+		if (le->callback(port_id, le->user_args)) {
+			rte_spinlock_unlock(&lock_entry_lock);
+			return -EBUSY;
+		}
+	}
+
+	rte_spinlock_unlock(&lock_entry_lock);
+	return 0;
+}
diff --git a/lib/librte_ethdev/ethdev_lock.h b/lib/librte_ethdev/ethdev_lock.h
new file mode 100644
index 000000000..82132eb0c
--- /dev/null
+++ b/lib/librte_ethdev/ethdev_lock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _RTE_ETHDEV_LOCK_H_
+#define _RTE_ETHDEV_LOCK_H_
+
+#include "rte_ethdev.h"
+
+/* Register lock callback function on specific port */
+int
+register_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/* Unregister lock callback function on specific port */
+int
+unregister_lock_callback(uint16_t port_id,
+			rte_eth_dev_lock_callback_t callback,
+			void *user_args);
+
+/**
+ * Unregister all callback function on specific port.
+ * This will be called when a device is detached.
+ */
+void clean_lock_callback(uint16_t port_id);
+
+/* Run each callback one by one. */
+int process_lock_callbacks(uint16_t port_id);
+
+#endif
diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 0f9d8990d..1d148cd5e 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -6,6 +6,7 @@
 
 #include "rte_ethdev_driver.h"
 #include "ethdev_mp.h"
+#include "ethdev_lock.h"
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -108,7 +109,7 @@ static void __handle_primary_request(void *param)
 		ret = attach_on_secondary(req->devargs, req->port_id);
 		break;
 	case REQ_TYPE_PRE_DETACH:
-		ret = 0;
+		ret = process_lock_callbacks(req->port_id);
 		break;
 	case REQ_TYPE_DETACH:
 	case REQ_TYPE_ATTACH_ROLLBACK:
diff --git a/lib/librte_ethdev/meson.build b/lib/librte_ethdev/meson.build
index b60256855..9bb0aec7f 100644
--- a/lib/librte_ethdev/meson.build
+++ b/lib/librte_ethdev/meson.build
@@ -6,6 +6,7 @@ version = 9
 allow_experimental_apis = true
 sources = files('ethdev_profile.c',
 	'ethdev_mp.c'
+	'ethdev_lock.c'
 	'rte_ethdev.c',
 	'rte_flow.c',
 	'rte_mtr.c',
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 91d9f9a37..7d89d9f95 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -43,6 +43,7 @@
 #include "ethdev_profile.h"
 #include "ethdev_mp.h"
 #include "ethdev_private.h"
+#include "ethdev_lock.h"
 
 int ethdev_logtype;
 
@@ -735,6 +736,7 @@ do_eth_dev_detach(uint16_t port_id)
 	if (ret < 0)
 		return ret;
 
+	clean_lock_callback(port_id);
 	if (solid_release)
 		return rte_eth_dev_release_port(&rte_eth_devices[port_id]);
 	else
@@ -803,7 +805,6 @@ rte_eth_dev_attach(const char *devargs, uint16_t *port_id)
 int
 rte_eth_dev_attach_private(const char *devargs, uint16_t *port_id)
 {
-
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
 
@@ -845,6 +846,10 @@ rte_eth_dev_detach(uint16_t port_id, char *name __rte_unused)
 		return req.result;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	/* check pre_detach */
 	req.t = REQ_TYPE_PRE_DETACH;
 	req.port_id = port_id;
@@ -891,6 +896,7 @@ int
 rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 {
 	uint32_t dev_flags;
+	int ret;
 
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		return -ENOTSUP;
@@ -904,6 +910,10 @@ rte_eth_dev_detach_private(uint16_t port_id, char *name __rte_unused)
 		return -ENOTSUP;
 	}
 
+	ret = process_lock_callbacks(port_id);
+	if (ret)
+		return ret;
+
 	return do_eth_dev_detach(port_id);
 }
 
@@ -4706,6 +4716,54 @@ rte_eth_devargs_parse(const char *dargs, struct rte_eth_devargs *eth_da)
 	return result;
 }
 
+static int
+dev_is_busy(uint16_t port_id __rte_unused, void *user_args __rte_unused)
+{
+	return -EBUSY;
+}
+
+int
+rte_eth_dev_lock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return register_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return register_lock_callback(port_id, callback, user_args);
+}
+
+int
+rte_eth_dev_unlock(uint16_t port_id)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	return unregister_lock_callback(port_id, dev_is_busy, NULL);
+}
+
+int
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args)
+{
+	RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV);
+
+	if (callback == NULL)
+		return -EINVAL;
+
+	return unregister_lock_callback(port_id, callback, user_args);
+}
+
 RTE_INIT(ethdev_init_log);
 static void
 ethdev_init_log(void)
diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h
index 4d207e2fe..b2c2de600 100644
--- a/lib/librte_ethdev/rte_ethdev.h
+++ b/lib/librte_ethdev/rte_ethdev.h
@@ -4364,6 +4364,130 @@ rte_eth_tx_buffer(uint16_t port_id, uint16_t queue_id,
 	return rte_eth_tx_buffer_flush(port_id, queue_id, buffer);
 }
 
+/**
+ * Callback function before device is detached.
+ *
+ * This type of function will be added into a function list, and will be
+ * invoked before device be detached. Application can register a callback
+ * function so it can be notified and do some cleanup before detach happen.
+ * Also, any callback function return !0 value will prevent device be
+ * detached (ref. rte_eth_dev_lock_with_callback and
+ * rte_eth_dev_unlock_with_callback).
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param user_args
+ *   This is parameter "user_args" be saved when callback function is
+ *   registered(rte_dev_eth_lock).
+ *
+ * @return
+ *   0  device is allowed be detached.
+ *   !0 device is not allowed be detached.
+ */
+typedef int (*rte_eth_dev_lock_callback_t)(uint16_t port_id, void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet Device, this help application to prevent a device
+ * be detached unexpectedly.
+ *
+ * @note
+ *   In multi-process situation, any process lock a share device will
+ *   prevent it be detached from all process. Also this is per-process
+ *   lock, which means unlock a device from process A take no effect
+ *   if the device is locked from process B.
+ *
+ * @note
+ *   Lock a device multiple times will increase a ref_count, and
+ *   corresponding unlock decrease the ref_count, the device will be
+ *   unlocked when ref_count reach 0.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_lock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Lock an Ethernet device base on a callback function which can performs
+ * condition check at the moment before device be detached. if the
+ * condition check not pass, the device will not be detached, else,
+ * continue to detach or not rely on return value of other callbacks
+ * on the same port.
+ *
+ * @note
+ *   Same as rte_eth_dev_lock, it is per-process lock.
+ *
+ * @note
+ *   Lock a device with different callback or user_args will add different
+ *   lock entries (<callback, user_args> pair) in a list. Lock a device
+ *   multiple times with same callback and args will only increase a
+ *   ref_count of specific lock entry, and corresponding unlock decrease
+ *   the ref_count, an entry will be removed if its ref_count reach 0.
+ *
+ * @note
+ *   All callbacks be attached to specific port will be removed
+ *   automatically if the device is detached.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   the callback function will be added into a pre-detach list,
+ *   it will be invoked when a device is going to be detached. The
+ *   return value will decide if continue detach the device or not.
+ * @param user_args
+ *   parameter will be parsed to callback function.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_lock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental rte_eth_dev_unlock(uint16_t port_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Reverse operation of rte_eth_dev_lock_with_callback.
+ *
+ * @param port_id
+ *   The port identifier of the Ethernet device.
+ * @param callback
+ *   parameter to match a lock entry.
+ * @param user_args
+ *   parameter to match a lock entry.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_eth_dev_unlock_with_callback(uint16_t port_id,
+				rte_eth_dev_lock_callback_t callback,
+				void *user_args);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index fbcc03925..1e270a387 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -225,6 +225,7 @@ EXPERIMENTAL {
 	rte_eth_dev_get_module_eeprom;
 	rte_eth_dev_get_module_info;
 	rte_eth_dev_is_removed;
+	rte_eth_dev_lock;
 	rte_eth_dev_owner_delete;
 	rte_eth_dev_owner_get;
 	rte_eth_dev_owner_new;
@@ -232,6 +233,7 @@ EXPERIMENTAL {
 	rte_eth_dev_owner_unset;
 	rte_eth_dev_rx_offload_name;
 	rte_eth_dev_tx_offload_name;
+	rte_eth_dev_unlock;
 	rte_eth_switch_domain_alloc;
 	rte_eth_switch_domain_free;
 	rte_mtr_capabilities_get;
-- 
2.13.6

[dpdk-dev] [PATCH v8 05/19] ethdev: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a share device
attach/detach request be issued from secondary process

device attach on secondary:
a) seconary send sync request to primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail reply to secondary, goto k).
j) send success reply to secondary.
k) secondary process receive reply of step a) and return.

device detach on secondary:
a) secondary send sync request to primary
b) primary receive the request and perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach sync request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach sync request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success reply to secondary, goto k).
j) send fail reply to secondary.
k) secondary process receive reply of step a) and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c | 179 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 173 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 1d148cd5e..8d13da591 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -5,8 +5,44 @@
 #include <rte_alarm.h>
 
 #include "rte_ethdev_driver.h"
+
 #include "ethdev_mp.h"
 #include "ethdev_lock.h"
+#include "ethdev_private.h"
+
+/**
+ *
+ * secondary to primary request.
+ * start from function eth_dev_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) seconary send sycn request to primary
+ * b) primary receive the request and attach the new device thread,
+ *    if failed goto i).
+ * c) primary forward attach request to all secondary as sync request
+ * d) secondary receive request and attach device and send reply.
+ * e) primary check the reply if all success go to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach device and send reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send fail sync reply to secondary, goto k).
+ * j) send success sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ *
+ * device detach on secondary:
+ * a) secondary send detach sync request to primary
+ * b) primary receive the request and perform pre-detach check, if device
+ *    is locked, goto j).
+ * c) primary send pre-detach sync request to all secondary.
+ * d) secondary perform pre-detach check and send reply.
+ * e) primary check the reply if any fail goto j).
+ * f) primary send detach sync request to all secondary
+ * g) secondary detach the device and send reply
+ * h) primary detach the device.
+ * i) send success sync reply to secondary, goto k).
+ * j) send fail sync reply to secondary.
+ * k) secondary process receive reply of step a) and return.
+ */
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
@@ -84,11 +120,122 @@ static int attach_on_secondary(const char *devargs, uint16_t port_id)
 }
 
 static int
-handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+send_response_to_secondary(const struct eth_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eth_dev_mp_req *resp =
+		(struct eth_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strcpy(mp_resp.name, ETH_DEV_MP_ACTION_REQUEST);
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	struct eth_dev_mp_req tmp_req;
+	uint16_t port_id;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == REQ_TYPE_ATTACH) {
+		ret = do_eth_dev_attach(req->devargs, &port_id);
+		if (ret)
+			goto finish;
+
+		tmp_req.port_id = port_id;
+		ret = eth_dev_request_to_secondary(&tmp_req);
+	} else if (req->t == REQ_TYPE_DETACH) {
+		if (!rte_eth_dev_is_valid_port(req->port_id)) {
+			ret = -EINVAL;
+			goto finish;
+		}
+
+		ret = process_lock_callbacks(req->port_id);
+		if (ret)
+			goto finish;
+
+		tmp_req.t = REQ_TYPE_PRE_DETACH;
+		ret = eth_dev_request_to_secondary(&tmp_req);
+		if (ret)
+			goto finish;
+
+		if (!tmp_req.result) {
+			tmp_req.t = REQ_TYPE_DETACH;
+			ret = eth_dev_request_to_secondary(&tmp_req);
+			if (ret)
+				goto finish;
+
+			ret = do_eth_dev_detach(req->port_id);
+		} else {
+			ret = tmp_req.result;
+		}
+	} else {
+		ethdev_log(ERR, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		ethdev_log(ERR, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg,
+			const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eth_dev_mp_req *req =
+		(const struct eth_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		ethdev_log(ERR, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		ethdev_log(ERR, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -183,8 +330,28 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eth_dev_request_to_primary(struct eth_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eth_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, ETH_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		ethdev_log(ERR, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eth_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+	req->port_id = resp->port_id;
+
+	return ret;
 }
 
 /**
-- 
2.13.6

[dpdk-dev] [PATCH v8 06/19] ethdev: support attach private device as first

[Qi Zhang]

When attach a private device from secondary as the first one, we need
to make sure rte_eth_dev_shared_data is initialized, the patch add
necessary IPC for secondary to inform primary to do initialization.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/ethdev_mp.c      |  2 ++
 lib/librte_ethdev/ethdev_mp.h      |  1 +
 lib/librte_ethdev/ethdev_private.h |  3 +++
 lib/librte_ethdev/rte_ethdev.c     | 31 ++++++++++++++++++++-----------
 4 files changed, 26 insertions(+), 11 deletions(-)

diff --git a/lib/librte_ethdev/ethdev_mp.c b/lib/librte_ethdev/ethdev_mp.c
index 8d13da591..28f89dba9 100644
--- a/lib/librte_ethdev/ethdev_mp.c
+++ b/lib/librte_ethdev/ethdev_mp.c
@@ -189,6 +189,8 @@ __handle_secondary_request(void *param)
 		} else {
 			ret = tmp_req.result;
 		}
+	} else if (req->t == REQ_TYPE_SHARE_DATA_PREPARE) {
+		eth_dev_shared_data_prepare();
 	} else {
 		ethdev_log(ERR, "unsupported secondary to primary request\n");
 		ret = -ENOTSUP;
diff --git a/lib/librte_ethdev/ethdev_mp.h b/lib/librte_ethdev/ethdev_mp.h
index 40be46c89..61fc381da 100644
--- a/lib/librte_ethdev/ethdev_mp.h
+++ b/lib/librte_ethdev/ethdev_mp.h
@@ -15,6 +15,7 @@ enum eth_dev_req_type {
 	REQ_TYPE_PRE_DETACH,
 	REQ_TYPE_DETACH,
 	REQ_TYPE_ATTACH_ROLLBACK,
+	REQ_TYPE_SHARE_DATA_PREPARE,
 };
 
 struct eth_dev_mp_req {
diff --git a/lib/librte_ethdev/ethdev_private.h b/lib/librte_ethdev/ethdev_private.h
index 981e7de8a..005d63afc 100644
--- a/lib/librte_ethdev/ethdev_private.h
+++ b/lib/librte_ethdev/ethdev_private.h
@@ -36,4 +36,7 @@ int do_eth_dev_attach(const char *devargs, uint16_t *port_id);
  */
 int do_eth_dev_detach(uint16_t port_id);
 
+/* Prepare shared data for multi-process */
+void eth_dev_shared_data_prepare(void);
+
 #endif
diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 7d89d9f95..408a49f44 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -199,11 +199,14 @@ rte_eth_find_next(uint16_t port_id)
 	return port_id;
 }
 
-static void
-rte_eth_dev_shared_data_prepare(void)
+void
+eth_dev_shared_data_prepare(void)
 {
 	const unsigned flags = 0;
 	const struct rte_memzone *mz;
+	struct eth_dev_mp_req req;
+
+	memset(&req, 0, sizeof(req));
 
 	rte_spinlock_lock(&rte_eth_shared_data_lock);
 
@@ -215,6 +218,12 @@ rte_eth_dev_shared_data_prepare(void)
 					rte_socket_id(), flags);
 		} else
 			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		/* if secondary attach a private device first */
+		if (mz == NULL && rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			req.t = REQ_TYPE_SHARE_DATA_PREPARE;
+			eth_dev_request_to_primary(&req);
+			mz = rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
+		}
 		if (mz == NULL)
 			rte_panic("Cannot allocate ethdev shared data\n");
 
@@ -255,7 +264,7 @@ rte_eth_dev_allocated(const char *name)
 {
 	struct rte_eth_dev *ethdev;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -300,7 +309,7 @@ rte_eth_dev_allocate(const char *name)
 	uint16_t port_id;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port creation between primary and secondary threads. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -339,7 +348,7 @@ rte_eth_dev_attach_secondary(const char *name)
 	uint16_t i;
 	struct rte_eth_dev *eth_dev = NULL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	/* Synchronize port attachment to primary port creation and release. */
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
@@ -379,7 +388,7 @@ rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 	if (eth_dev == NULL)
 		return -EINVAL;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
 
@@ -433,7 +442,7 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id)
 int __rte_experimental
 rte_eth_dev_owner_new(uint64_t *owner_id)
 {
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -488,7 +497,7 @@ rte_eth_dev_owner_set(const uint16_t port_id,
 {
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -505,7 +514,7 @@ rte_eth_dev_owner_unset(const uint16_t port_id, const uint64_t owner_id)
 			{.id = RTE_ETH_DEV_NO_OWNER, .name = ""};
 	int ret;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -520,7 +529,7 @@ rte_eth_dev_owner_delete(const uint64_t owner_id)
 {
 	uint16_t port_id;
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
@@ -542,7 +551,7 @@ rte_eth_dev_owner_get(const uint16_t port_id, struct rte_eth_dev_owner *owner)
 	int ret = 0;
 	struct rte_eth_dev *ethdev = &rte_eth_devices[port_id];
 
-	rte_eth_dev_shared_data_prepare();
+	eth_dev_shared_data_prepare();
 
 	rte_spinlock_lock(&rte_eth_dev_shared_data->ownership_lock);
 
-- 
2.13.6

[dpdk-dev] [PATCH v8 07/19] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v8 08/19] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v8 09/19] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v8 10/19] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v8 11/19] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v8 12/19] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v8 13/19] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v8 14/19] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v8 15/19] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v8 16/19] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v8 17/19] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on secondary process will mess primary
process and cause same device can't be attached again, by take
advantage of rte_eth_release_port_private, we can support this
with minor change.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v8 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrate device (ethdev only) management
at multi-process envrionment. User can attach/detach a device
on primary process and see it is synced on secondary process
automatically, also user can lock a device to prevent it be
detached or unlock it to go back to default behaviour.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

/* attach a private af_packet vdev (secondary process only)*/
>attachp net_af_packet,iface=eth0

/* detach a private device (secondary process only) */
>detachp 0

/* lock port 0 */
>lock 0

/* unlock port 0 */
>unlock 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 356 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 5 files changed, 431 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..31f9e2e15
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,356 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- attachp <devargs>\n"
+		       "- detachp <port_id>\n"
+		       "- lock <port_id>\n"
+		       "- unlock <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attachp_result {
+	cmdline_fixed_string_t attachp;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attachp_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach_private(res->device, &port_id))
+		cmdline_printf(cl, "attached prviate device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached private device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attachp_attachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, attachp,
+				 "attachp");
+cmdline_parse_token_string_t cmd_dev_attachp_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attachp_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attachp_device = {
+	.f = cmd_dev_attachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attachp_attachp,
+		(void *)&cmd_dev_attachp_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detachp_result {
+	cmdline_fixed_string_t detachp;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detachp_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detachp_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach_private(port_id, dev_name))
+		cmdline_printf(cl, "detached private device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to detach private device at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detachp_detachp =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detachp_result, detachp,
+				 "detachp");
+cmdline_parse_token_num_t cmd_dev_detachp_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detachp_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detachp_device = {
+	.f = cmd_dev_detachp_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a private device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detachp_detachp,
+		(void *)&cmd_dev_detachp_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_lock_result {
+	cmdline_fixed_string_t lock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_lock_parsed(void *parsed_result,
+				struct cmdline *cl,
+				__attribute__((unused)) void *data)
+{
+	struct cmd_dev_lock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_lock(res->port_id);
+	cmdline_printf(cl, "lock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_lock_lock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_lock_result, lock, "lock");
+cmdline_parse_token_num_t cmd_dev_lock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_lock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_lock_device = {
+	.f = cmd_dev_lock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "lock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_lock_lock,
+		(void *)&cmd_dev_lock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_unlock_result {
+	cmdline_fixed_string_t unlock;
+	uint16_t port_id;
+};
+
+static void cmd_dev_unlock_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_unlock_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	int ret = 0;
+
+	ret = rte_eth_dev_unlock(res->port_id);
+	cmdline_printf(cl, "unlock port %d, ret = %d\n", port_id, ret);
+}
+
+cmdline_parse_token_string_t cmd_dev_unlock_unlock =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_unlock_result, unlock,
+				 "unlock");
+cmdline_parse_token_num_t cmd_dev_unlock_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_unlock_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_unlock_device = {
+	.f = cmd_dev_unlock_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "unlock a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_unlock_unlock,
+		(void *)&cmd_dev_unlock_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	(cmdline_parse_inst_t *)&cmd_attachp_device,
+	(cmdline_parse_inst_t *)&cmd_detachp_device,
+	(cmdline_parse_inst_t *)&cmd_lock_device,
+	(cmdline_parse_inst_t *)&cmd_unlock_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v8 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..93a813340 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,21 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support etherdev multi-process hotplug.**
+
+  Hotplug and hot-unplug for ethdev devices will now be supported in
+  multiprocessing scenario. Any ethdev devices created in the primary
+  process will be regarded as shared and will be available for all DPDK
+  processes, while secondary processes will have a choice between adding
+  a private (non-shared) or a shared device. Synchronization between
+  processes will be done using DPDK IPC.
+
+* **Support etherdev locking.**
+
+  Application can now lock an ethernet device to prevent unexpected device
+  removal. Devices can either be locked unconditionally, or an application
+  can register for a callback before unplug for the purposes of performing
+  cleanup before releasing the device (or have a chance to deny unplug)
 
 API Changes
 -----------
@@ -60,6 +75,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* ethdev: scope of rte_eth_dev_attach and rte_eth_dev_detach is extended.
+
+  In primary-secondary process model, ``rte_eth_dev_attach`` will guarantee
+  that device be attached on all processes, while ``rte_eth_dev_detach``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v8 01/19] ethdev: add function to release port in local process

[Thomas Monjalon]

Hi,

02/07/2018 07:44, Qi Zhang:
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -70,6 +70,19 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
>  
>  /**
>   * @internal
> + * Release the specified ethdev port in local process, only set to ethdev
> + * state to unused, but not reset share data since it assume other process
> + * is still using it, typically it is called by secondary process.

Please check grammar in doxygen comments, and do not make long sentences.

"only set to ethdev state to unused" -> "only set ethdev state to unused"
"share" -> "shared"
"it assume" -> "it assumes"

Please split sentences with a dot and an uppercase.
I expect all patches of the series must be reviewed for english wording.

> + *
> + * @param eth_dev
> + * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.

This comment is useless.
You can just say "Device to be detached".

> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);


[...]
> --- a/lib/librte_ethdev/rte_ethdev_pci.h
> +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> @@ -197,6 +197,9 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(eth_dev);

This is a behaviour change.
It means a PCI device cannot be globally detached directly by a secondary.
It must be justified by a comment in the code.

Please explain.

Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback

[Thomas Monjalon]

Hi Qi,

Sorry I do not understand neither the commit log, nor the doxygen.
If it can help, consider your reader is in a bad mood
and needs a pleasant description.

02/07/2018 07:44, Qi Zhang:
> Introduce new API rte_eal_register_mp_init that help to register
> a callback function which will be invoked right after multi-process
> channel be established (rte_mp_channel_init). Typically the API
> will be used by other module that want it's mp channel action callbacks
> can be registered during rte_eal_init automatically.
[...]
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice
> + *
> + * Register a callback function that will be invoked right after
> + * multi-process channel be established (rte_mp_channel_init). Typically
> + * the function is used by other module that want it's mp channel
> + * action callbacks can be registered during rte_eal_init automatically.
> + *
> + * @note
> + *   This function only take effect when be called before rte_eal_init,
> + *   and all registered callback will be clear during rte_eal_cleanup.
> + *
> + * @param callback
> + *   function be called at that moment.
> + *
> + * @return
> + *  - 0 on success.
> + *  - (<0) on failure.
> + */
> +int __rte_experimental
> +rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);

It looks the prefix of this function should be rte_mp_

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Thomas Monjalon]

Hi,

02/07/2018 07:44, Qi Zhang:
> --- /dev/null
> +++ b/lib/librte_ethdev/ethdev_mp.h
> @@ -0,0 +1,41 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#ifndef _RTE_ETHDEV_MP_H_
> +#define _RTE_ETHDEV_MP_H_
> +
> +#define MAX_DEV_ARGS_LEN 0x80
> +
> +#define ETH_DEV_MP_ACTION_REQUEST	"eth_dev_mp_request"
> +#define ETH_DEV_MP_ACTION_RESPONSE	"eth_dev_mp_response"
> +
> +enum eth_dev_req_type {
> +	REQ_TYPE_ATTACH,
> +	REQ_TYPE_PRE_DETACH,
> +	REQ_TYPE_DETACH,
> +	REQ_TYPE_ATTACH_ROLLBACK,
> +};

These constants are missing an ethdev prefix.

> +
> +struct eth_dev_mp_req {
> +	enum eth_dev_req_type t;
> +	char devargs[MAX_DEV_ARGS_LEN];
> +	uint16_t port_id;
> +	int result;
> +};
> +
> +/**
> + * this is a synchronous wrapper for secondary process send
> + * request to primary process, this is invoked when an attach
> + * or detach request issued from primary.
> + */
> +int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> +
> +/**
> + * this is a synchronous wrapper for primary process send
> + * request to secondary process, this is invoked when an attach
> + * or detach request issued from secondary process.
> + */
> +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);


Why do we need ethdev functions for IPC (mp request/response)?
How this model can reasonnably scale to other device classes
(crypto, compression, bbdev, eventdev, etc)?


> --- /dev/null
> +++ b/lib/librte_ethdev/ethdev_private.h

What is the purpose of a file ethdev_private.h?

> @@ -0,0 +1,39 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2010-2018 Intel Corporation

Are you sure about the years?

> +/**
> + * Attach a new Ethernet device in current process.
> + *
> + * @param devargs
> + *  A pointer to a strings array describing the new device
> + *  to be attached. The strings should be a pci address like
> + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.

No, no. The devargs syntax is being changed, so you should not duplicate
its description here. Better to reference an unique source of doc.

> + *
> + * @param port_id
> + *  A pointer to a port identifier actually attached.
> + *
> + * @return
> + *  0 on success and port_id is filled, negative on error
> + */
> +int do_eth_dev_attach(const char *devargs, uint16_t *port_id);

So you are duplicating rte_eth_dev_attach which is flawed in its design
and should be deprecated...

As you may have noticed, rte_eth_dev_attach() is calling
rte_eal_hotplug_add() which manages the EAL device.
It is wrong because the relation between an ethdev port and
an EAL device is not a 1:1 mapping.
We must manage the ethdev port as one of the possible abstractions
of a device represented by rte_device.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

02/07/2018 07:44, Qi Zhang:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.

Trying to understand: a process of an application could try to detach
a port while another process is against this decision.
Why an application needs to be protected against itself?
I guess it is only an application inter-process management.
If we really want to provide such helper in DPDK, it should not be
limited to ethdev.
(for info, see class implementation: https://patches.dpdk.org/patch/41605/)

What about hardware unplug?
Can we detach the locked ports associated to the unplugged device?

> Aslo introduce the new API rte_eth_dev_lock_with_callback and
> rte_eth_dev_unlock_with callback to let application to register
> a callback function which will be invoked before a device is going
> to be detached, the return value of the function will decide if
> device will continue be detached or not, this support application
> to do condition check at runtime.

You don't need 2 flavors for the lock.
We can have only the "_with_callback" flavour and provide a default
callback which says always no.

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

Hi Thomas:
<...>

> > +enum eth_dev_req_type {
> > +	REQ_TYPE_ATTACH,
> > +	REQ_TYPE_PRE_DETACH,
> > +	REQ_TYPE_DETACH,
> > +	REQ_TYPE_ATTACH_ROLLBACK,
> > +};
> 
> These constants are missing an ethdev prefix.

OK, will fix.

> 
> > +
> > +struct eth_dev_mp_req {
> > +	enum eth_dev_req_type t;
> > +	char devargs[MAX_DEV_ARGS_LEN];
> > +	uint16_t port_id;
> > +	int result;
> > +};
> > +
> > +/**
> > + * this is a synchronous wrapper for secondary process send
> > + * request to primary process, this is invoked when an attach
> > + * or detach request issued from primary.
> > + */
> > +int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> > +
> > +/**
> > + * this is a synchronous wrapper for primary process send
> > + * request to secondary process, this is invoked when an attach
> > + * or detach request issued from secondary process.
> > + */
> > +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
> 
> 
> Why do we need ethdev functions for IPC (mp request/response)?
> How this model can reasonnably scale to other device classes (crypto,
> compression, bbdev, eventdev, etc)?

Yes it will be more generic to more the multi-process device sync mechanism into eal layer.(rte_eal_hotplug_add/rte_eal_hotplug_remove)
I didn't do this is I'm not very sure if all anothers kinds of device type need this, but if you think this is a good direction and we need to enable for all devices, 
I think this could be our next step. BTW, I guess ethdev still need some IPC to sync port_id which is specific for itself, and other device type may have similar requirement.

> > --- /dev/null
> > +++ b/lib/librte_ethdev/ethdev_private.h
> 
> What is the purpose of a file ethdev_private.h?
> 
> > @@ -0,0 +1,39 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright(c) 2010-2018 Intel Corporation
> 
> Are you sure about the years?

NO, will fix:)

> 
> > +/**
> > + * Attach a new Ethernet device in current process.
> > + *
> > + * @param devargs
> > + *  A pointer to a strings array describing the new device
> > + *  to be attached. The strings should be a pci address like
> > + *  '0000:01:00.0' or virtual device name like 'net_pcap0'.
> 
> No, no. The devargs syntax is being changed, so you should not duplicate its
> description here. Better to reference an unique source of doc.

OK, will check and replace with more correct description.

> 
> > + *
> > + * @param port_id
> > + *  A pointer to a port identifier actually attached.
> > + *
> > + * @return
> > + *  0 on success and port_id is filled, negative on error  */ int
> > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> 
> So you are duplicating rte_eth_dev_attach which is flawed in its design and
> should be deprecated...

OK, just to know this, but I guess it will not be the issue, if we move the dev sync mechanism into eal layer in future right?

Regards
Qi

> 
> As you may have noticed, rte_eth_dev_attach() is calling
> rte_eal_hotplug_add() which manages the EAL device.
> It is wrong because the relation between an ethdev port and an EAL device is
> not a 1:1 mapping.
> We must manage the ethdev port as one of the possible abstractions of a
> device represented by rte_device.
> 

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Thomas Monjalon]

03/07/2018 14:59, Zhang, Qi Z:
> > > +/**
> > > + * this is a synchronous wrapper for secondary process send
> > > + * request to primary process, this is invoked when an attach
> > > + * or detach request issued from primary.
> > > + */
> > > +int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> > > +
> > > +/**
> > > + * this is a synchronous wrapper for primary process send
> > > + * request to secondary process, this is invoked when an attach
> > > + * or detach request issued from secondary process.
> > > + */
> > > +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
> > 
> > 
> > Why do we need ethdev functions for IPC (mp request/response)?
> > How this model can reasonnably scale to other device classes (crypto,
> > compression, bbdev, eventdev, etc)?
> 
> Yes it will be more generic to more the multi-process device sync mechanism into eal layer.(rte_eal_hotplug_add/rte_eal_hotplug_remove)
> I didn't do this is I'm not very sure if all anothers kinds of device type need this, but if you think this is a good direction and we need to enable for all devices, 
> I think this could be our next step. BTW, I guess ethdev still need some IPC to sync port_id which is specific for itself, and other device type may have similar requirement.

I don't understand what is specific to ethdev in this IPC.
If it is just about a port id, I think it can be done elsewhere (EAL?)

> > > --- /dev/null
> > > +++ b/lib/librte_ethdev/ethdev_private.h
> > 
> > What is the purpose of a file ethdev_private.h?

You did not reply this question.

> > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > 
> > So you are duplicating rte_eth_dev_attach which is flawed in its design and
> > should be deprecated...
> 
> OK, just to know this, but I guess it will not be the issue, if we move the dev sync mechanism into eal layer in future right?

Future is now :)
We must stop mixing devargs and port id in the same layer.

> > As you may have noticed, rte_eth_dev_attach() is calling
> > rte_eal_hotplug_add() which manages the EAL device.
> > It is wrong because the relation between an ethdev port and an EAL device is
> > not a 1:1 mapping.
> > We must manage the ethdev port as one of the possible abstractions of a
> > device represented by rte_device.

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, July 3, 2018 10:11 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on
> multi-process
> 
> 03/07/2018 14:59, Zhang, Qi Z:
> > > > +/**
> > > > + * this is a synchronous wrapper for secondary process send
> > > > + * request to primary process, this is invoked when an attach
> > > > + * or detach request issued from primary.
> > > > + */
> > > > +int eth_dev_request_to_primary(struct eth_dev_mp_req *req);
> > > > +
> > > > +/**
> > > > + * this is a synchronous wrapper for primary process send
> > > > + * request to secondary process, this is invoked when an attach
> > > > + * or detach request issued from secondary process.
> > > > + */
> > > > +int eth_dev_request_to_secondary(struct eth_dev_mp_req *req);
> > >
> > >
> > > Why do we need ethdev functions for IPC (mp request/response)?
> > > How this model can reasonnably scale to other device classes
> > > (crypto, compression, bbdev, eventdev, etc)?
> >
> > Yes it will be more generic to more the multi-process device sync
> > mechanism into eal layer.(rte_eal_hotplug_add/rte_eal_hotplug_remove)
> > I didn't do this is I'm not very sure if all anothers kinds of device
> > type need this, but if you think this is a good direction and we need to enable
> for all devices, I think this could be our next step. BTW, I guess ethdev still
> need some IPC to sync port_id which is specific for itself, and other device type
> may have similar requirement.
> 
> I don't understand what is specific to ethdev in this IPC.
> If it is just about a port id, I think it can be done elsewhere (EAL?)

Yes what I mean is port _id, I'm not sure if it can be done elsewhere, I need to check.

> 
> > > > --- /dev/null
> > > > +++ b/lib/librte_ethdev/ethdev_private.h
> > >
> > > What is the purpose of a file ethdev_private.h?
> 
> You did not reply this question.

the header file is used to declare the functions only be used by libethdev internally 
like eal_private.h, or I don't know where the three functions I should declared.

> 
> > > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > >
> > > So you are duplicating rte_eth_dev_attach which is flawed in its
> > > design and should be deprecated...
> >
> > OK, just to know this, but I guess it will not be the issue, if we move the dev
> sync mechanism into eal layer in future right?
> 
> Future is now :)
> We must stop mixing devargs and port id in the same layer.

Ok, is there any RFC I can learn?

> 
> > > As you may have noticed, rte_eth_dev_attach() is calling
> > > rte_eal_hotplug_add() which manages the EAL device.
> > > It is wrong because the relation between an ethdev port and an EAL
> > > device is not a 1:1 mapping.
> > > We must manage the ethdev port as one of the possible abstractions
> > > of a device represented by rte_device.
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, July 3, 2018 5:56 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 02/07/2018 07:44, Qi Zhang:
> > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > application lock or unlock on specific ethdev, a locked device can't
> > be detached, this help applicaiton to prevent unexpected device
> > detaching, especially in multi-process envrionment.
> 
> Trying to understand: a process of an application could try to detach a port
> while another process is against this decision.
> Why an application needs to be protected against itself?

I think we can regard this as a help function, it help application to simplified the situation when one process want to detach a device while another one is still using it.
Application can register a callback which can do to necessary clean up (like stop traffic, release memory ...) before device be detached.

> I guess it is only an application inter-process management.
> If we really want to provide such helper in DPDK, it should not be limited to
> ethdev.

Once we move to eal layer, we will have rte_eal_dev_lock/unlock(devname, busname).
But its also better we keep rte_eth_dev_lock/unlock to make ethdev API more completed (any port be locked means underline rte_device also be locked?)
and this is same for other device family.


> (for info, see class implementation: https://patches.dpdk.org/patch/41605/)
> 
> What about hardware unplug?
> Can we detach the locked ports associated to the unplugged device?

NO, we can't.
But do you think, we need to introduce a "force detach" version, which will ignore all locks.

> 
> > Aslo introduce the new API rte_eth_dev_lock_with_callback and
> > rte_eth_dev_unlock_with callback to let application to register a
> > callback function which will be invoked before a device is going to be
> > detached, the return value of the function will decide if device will
> > continue be detached or not, this support application to do condition
> > check at runtime.
> 
> You don't need 2 flavors for the lock.
> We can have only the "_with_callback" flavour and provide a default callback
> which says always no.

OK, I can rollback this.

Regards
Qi

> 

Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, July 3, 2018 5:28 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback
> 
> Hi Qi,
> 
> Sorry I do not understand neither the commit log, nor the doxygen.
> If it can help, consider your reader is in a bad mood and needs a pleasant
> description.

OK, I think is not a grammar issue since it almost passed a grammar check :)

So more explanation: 
Basically we need to register mp action callback in ethdev layer, but this should happens during rte_eal_init 
ethdev don't have a general init function that can be invoked by eal during rte_eal_init, actually I guess, all no-eal module don't have 
in vdev bus, it register the mp action at the first bus scan happen. but in ethdev, we can't do that way, because, we don' t know when device will be attached or detached. 
so we need that function to help register a callback function which will be invoked during rte_eal_init.


> 
> 02/07/2018 07:44, Qi Zhang:
> > Introduce new API rte_eal_register_mp_init that help to register a
> > callback function which will be invoked right after multi-process
> > channel be established (rte_mp_channel_init). Typically the API will
> > be used by other module that want it's mp channel action callbacks can
> > be registered during rte_eal_init automatically.
> [...]
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice
> > + *
> > + * Register a callback function that will be invoked right after
> > + * multi-process channel be established (rte_mp_channel_init).
> > +Typically
> > + * the function is used by other module that want it's mp channel
> > + * action callbacks can be registered during rte_eal_init automatically.
> > + *
> > + * @note
> > + *   This function only take effect when be called before rte_eal_init,
> > + *   and all registered callback will be clear during rte_eal_cleanup.
> > + *
> > + * @param callback
> > + *   function be called at that moment.
> > + *
> > + * @return
> > + *  - 0 on success.
> > + *  - (<0) on failure.
> > + */
> > +int __rte_experimental
> > +rte_eal_register_mp_init(rte_eal_mp_init_callback_t callback);
> 
> It looks the prefix of this function should be rte_mp_

OK, will fix.

Regards
Qi
> 

Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback

[Thomas Monjalon]

03/07/2018 17:16, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 
> > Hi Qi,
> > 
> > Sorry I do not understand neither the commit log, nor the doxygen.
> > If it can help, consider your reader is in a bad mood and needs a pleasant
> > description.
> 
> OK, I think is not a grammar issue since it almost passed a grammar check :)
> 
> So more explanation: 
> Basically we need to register mp action callback in ethdev layer,

What is "mp action callback"?
Why do you need it in ethdev?

> but this should happens during rte_eal_init 

Why it should be done in rte_eal_init?

> ethdev don't have a general init function that can be invoked by eal
> during rte_eal_init, actually I guess, all no-eal module don't have 

What is "no-eal module"?

> in vdev bus, it register the mp action at the first bus scan happen.
> but in ethdev, we can't do that way, because, we don' t know
> when device will be attached or detached. 
> so we need that function to help register a callback function which will be invoked during rte_eal_init.
> 
> 
> > 
> > 02/07/2018 07:44, Qi Zhang:
> > > Introduce new API rte_eal_register_mp_init that help to register a
> > > callback function which will be invoked right after multi-process
> > > channel be established (rte_mp_channel_init). Typically the API will
> > > be used by other module that want it's mp channel action callbacks can
> > > be registered during rte_eal_init automatically.
> > [...]
> > > +/**
> > > + * @warning
> > > + * @b EXPERIMENTAL: this API may change without prior notice
> > > + *
> > > + * Register a callback function that will be invoked right after
> > > + * multi-process channel be established (rte_mp_channel_init).
> > > +Typically
> > > + * the function is used by other module that want it's mp channel
> > > + * action callbacks can be registered during rte_eal_init automatically.
> > > + *
> > > + * @note
> > > + *   This function only take effect when be called before rte_eal_init,
> > > + *   and all registered callback will be clear during rte_eal_cleanup.

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Thomas Monjalon]

03/07/2018 17:03, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 03/07/2018 14:59, Zhang, Qi Z:
> > > > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > > >
> > > > So you are duplicating rte_eth_dev_attach which is flawed in its
> > > > design and should be deprecated...
> > >
> > > OK, just to know this, but I guess it will not be the issue, if we move the dev
> > sync mechanism into eal layer in future right?
> > 
> > Future is now :)
> > We must stop mixing devargs and port id in the same layer.
> 
> Ok, is there any RFC I can learn?

RFC for what?
It is just a design issue that we must stop propagating.

> > > > As you may have noticed, rte_eth_dev_attach() is calling
> > > > rte_eal_hotplug_add() which manages the EAL device.
> > > > It is wrong because the relation between an ethdev port and an EAL
> > > > device is not a 1:1 mapping.
> > > > We must manage the ethdev port as one of the possible abstractions
> > > > of a device represented by rte_device.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

03/07/2018 17:08, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 02/07/2018 07:44, Qi Zhang:
> > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > > application lock or unlock on specific ethdev, a locked device can't
> > > be detached, this help applicaiton to prevent unexpected device
> > > detaching, especially in multi-process envrionment.
> > 
> > Trying to understand: a process of an application could try to detach a port
> > while another process is against this decision.
> > Why an application needs to be protected against itself?
> 
> I think we can regard this as a help function, it help application to simplified the situation when one process want to detach a device while another one is still using it.
> Application can register a callback which can do to necessary clean up (like stop traffic, release memory ...) before device be detached.

Yes I agree such hook can be a good idea.


> > I guess it is only an application inter-process management.
> > If we really want to provide such helper in DPDK, it should not be limited to
> > ethdev.
> 
> Once we move to eal layer, we will have rte_eal_dev_lock/unlock(devname, busname).
> But its also better we keep rte_eth_dev_lock/unlock to make ethdev API more completed (any port be locked means underline rte_device also be locked?)
> and this is same for other device family.

No. Again, a port is not exactly a device.
There can be several ports per device.

I think the right place for this hook is in port-level API
(ethdev, crypto, etc). And as we improve only ethdev currently,
without any common genericity for other device classes,
it is probably fine in ethdev for now.
> 
> > (for info, see class implementation: https://patches.dpdk.org/patch/41605/)
> > 
> > What about hardware unplug?
> > Can we detach the locked ports associated to the unplugged device?
> 
> NO, we can't.
> But do you think, we need to introduce a "force detach" version, which will ignore all locks.

No, I don't think so.
I am just trying to show that you cannot really "lock" a port.
Maybe you should just rename those functions.
After all, it is just a pre-detach hook.
Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
Perhaps we just need to improve the handling of the DESTROY event?

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Thomas Monjalon]

03/07/2018 23:57, Thomas Monjalon:
> 03/07/2018 17:03, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 03/07/2018 14:59, Zhang, Qi Z:
> > > > > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > > > >
> > > > > So you are duplicating rte_eth_dev_attach which is flawed in its
> > > > > design and should be deprecated...
> > > >
> > > > OK, just to know this, but I guess it will not be the issue, if we move the dev
> > > sync mechanism into eal layer in future right?
> > > 
> > > Future is now :)
> > > We must stop mixing devargs and port id in the same layer.
> > 
> > Ok, is there any RFC I can learn?
> 
> RFC for what?
> It is just a design issue that we must stop propagating.

Please read at this commit, which is 2 years old:
	http://git.dpdk.org/dpdk/commit/?id=b0fb26685570
It was starting to fix early design mistakes, but unfortunately it is not
yet totally fixed today.

> > > > > As you may have noticed, rte_eth_dev_attach() is calling
> > > > > rte_eal_hotplug_add() which manages the EAL device.
> > > > > It is wrong because the relation between an ethdev port and an EAL
> > > > > device is not a 1:1 mapping.
> > > > > We must manage the ethdev port as one of the possible abstractions
> > > > > of a device represented by rte_device.

Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Wednesday, July 4, 2018 5:51 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 02/19] eal: enable multi process init callback
> 
> 03/07/2018 17:16, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > >
> > > Hi Qi,
> > >
> > > Sorry I do not understand neither the commit log, nor the doxygen.
> > > If it can help, consider your reader is in a bad mood and needs a
> > > pleasant description.
> >
> > OK, I think is not a grammar issue since it almost passed a grammar
> > check :)
> >
> > So more explanation:
> > Basically we need to register mp action callback in ethdev layer,
> 
> What is "mp action callback"?

rte_mp_action_register(ETH_DEV_MP_ACTION_REQUEST, handle_secondary_request);

in this case ETH_DEV_MP_ACTION_REQUEST is the action,
and handle_secondary_request is the callback which will be invoked when the action received from another process.

> Why do you need it in ethdev?

Based on current implementation, IPC is required during dev_attach/dev_detach.
so we need to make sure callback is registered before that happen.

> 
> > but this should happens during rte_eal_init
> 
> Why it should be done in rte_eal_init?
Either we can expose an ethdev API like rte_eth_dev_mp_init, to let application do this explicitly, or we can let application assume all necessary callback has been registered when DPDK is initialized. (rte_eal_init)
> 
> > ethdev don't have a general init function that can be invoked by eal
> > during rte_eal_init, actually I guess, all no-eal module don't have
> 
> What is "no-eal module"?

A module like lib_ethdev which lib_eal should not depend on.

> 
> > in vdev bus, it register the mp action at the first bus scan happen.
> > but in ethdev, we can't do that way, because, we don' t know when
> > device will be attached or detached.
> > so we need that function to help register a callback function which will be
> invoked during rte_eal_init.
> >
> >
> > >
> > > 02/07/2018 07:44, Qi Zhang:
> > > > Introduce new API rte_eal_register_mp_init that help to register a
> > > > callback function which will be invoked right after multi-process
> > > > channel be established (rte_mp_channel_init). Typically the API
> > > > will be used by other module that want it's mp channel action
> > > > callbacks can be registered during rte_eal_init automatically.
> > > [...]
> > > > +/**
> > > > + * @warning
> > > > + * @b EXPERIMENTAL: this API may change without prior notice
> > > > + *
> > > > + * Register a callback function that will be invoked right after
> > > > + * multi-process channel be established (rte_mp_channel_init).
> > > > +Typically
> > > > + * the function is used by other module that want it's mp channel
> > > > + * action callbacks can be registered during rte_eal_init automatically.
> > > > + *
> > > > + * @note
> > > > + *   This function only take effect when be called before rte_eal_init,
> > > > + *   and all registered callback will be clear during rte_eal_cleanup.
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Wednesday, July 4, 2018 6:14 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 03/07/2018 17:08, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 02/07/2018 07:44, Qi Zhang:
> > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > > > application lock or unlock on specific ethdev, a locked device
> > > > can't be detached, this help applicaiton to prevent unexpected
> > > > device detaching, especially in multi-process envrionment.
> > >
> > > Trying to understand: a process of an application could try to
> > > detach a port while another process is against this decision.
> > > Why an application needs to be protected against itself?
> >
> > I think we can regard this as a help function, it help application to simplified
> the situation when one process want to detach a device while another one is
> still using it.
> > Application can register a callback which can do to necessary clean up (like
> stop traffic, release memory ...) before device be detached.
> 
> Yes I agree such hook can be a good idea.
> 
> 
> > > I guess it is only an application inter-process management.
> > > If we really want to provide such helper in DPDK, it should not be
> > > limited to ethdev.
> >
> > Once we move to eal layer, we will have rte_eal_dev_lock/unlock(devname,
> busname).
> > But its also better we keep rte_eth_dev_lock/unlock to make ethdev API
> > more completed (any port be locked means underline rte_device also be
> locked?) and this is same for other device family.
> 
> No. Again, a port is not exactly a device.
> There can be several ports per device.

Yes, I know that.
what I mean is, we should assume lock any port of that rte_device will prevent the device be detached.

> 
> I think the right place for this hook is in port-level API (ethdev, crypto, etc). And
> as we improve only ethdev currently, without any common genericity for other
> device classes, it is probably fine in ethdev for now.
> >
> > > (for info, see class implementation:
> > > https://patches.dpdk.org/patch/41605/)
> > >
> > > What about hardware unplug?
> > > Can we detach the locked ports associated to the unplugged device?
> >
> > NO, we can't.
> > But do you think, we need to introduce a "force detach" version, which will
> ignore all locks.
> 
> No, I don't think so.
> I am just trying to show that you cannot really "lock" a port.
> Maybe you should just rename those functions.
> After all, it is just a pre-detach hook.

> Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> Perhaps we just need to improve the handling of the DESTROY event?

I have thought about this before.
Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here need to give feedback, pass or fail will impact the following behavior, this make it special, so I separate it from all exist rte_eth_event_type handle mechanism. 

The alternative solution is 
we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH and reuse all exist API 
rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.

But in _rte_eth_dev_callback_process we need to add a code branch for PRE_DETACH handle.

If (event = RTE_ETH_EVENT_PRE_DETACH)
	<...>.
else {
	<....>
}

And we may also need to keep rte_eth_dev_lock/unlock which will register a default callback for PRE_DETACH.

What do you think about?








 



> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Yuanhan Liu]

On Mon, Jul 2, 2018, at 1:44 PM, Qi Zhang wrote:
> Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> application lock or unlock on specific ethdev, a locked device
> can't be detached, this help applicaiton to prevent unexpected
> device detaching, especially in multi-process envrionment.

I'm just wondering why the need of introducing such 2 (very specific)
APIs, judging that it  looks like a ref count could solve this kind of issues
perfectly.

Also, the API name (_dev_lock/unlock) looks like an over killing, judging
that you just want to pin a device (if I understand this commit log correctly).
What firstly comes to my mind when I see "lock" was "no one can *access*
a device after another has grabbed the lock", which doesn't quite match
what you described here.

That said, if I were you, I would go with introducing few generic refcount
APIs.

        --yliu


> 
> Aslo introduce the new API rte_eth_dev_lock_with_callback and
> rte_eth_dev_unlock_with callback to let application to register
> a callback function which will be invoked before a device is going
> to be detached, the return value of the function will decide if
> device will continue be detached or not, this support application
> to do condition check at runtime.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Wednesday, July 4, 2018 6:36 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on
> multi-process
> 
> 03/07/2018 23:57, Thomas Monjalon:
> > 03/07/2018 17:03, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 03/07/2018 14:59, Zhang, Qi Z:
> > > > > > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > > > > >
> > > > > > So you are duplicating rte_eth_dev_attach which is flawed in
> > > > > > its design and should be deprecated...
> > > > >
> > > > > OK, just to know this, but I guess it will not be the issue, if
> > > > > we move the dev
> > > > sync mechanism into eal layer in future right?
> > > >
> > > > Future is now :)
> > > > We must stop mixing devargs and port id in the same layer.
> > >
> > > Ok, is there any RFC I can learn?
> >
> > RFC for what?
> > It is just a design issue that we must stop propagating.
> 
> Please read at this commit, which is 2 years old:
> 	http://git.dpdk.org/dpdk/commit/?id=b0fb26685570
> It was starting to fix early design mistakes, but unfortunately it is not yet totally
> fixed today.

OK, rte_eth_dev_attach is going to be deprecated.
Do you mean we will use rte_eal_hotplug_add to attach a device directly, 
then the device driver will be responsible for propagating all the ethdev port, 
and application could register callback for RTE_ETH_EVENT_NEW to know new ports are created, 
is that correct?

> 
> > > > > > As you may have noticed, rte_eth_dev_attach() is calling
> > > > > > rte_eal_hotplug_add() which manages the EAL device.
> > > > > > It is wrong because the relation between an ethdev port and an
> > > > > > EAL device is not a 1:1 mapping.
> > > > > > We must manage the ethdev port as one of the possible
> > > > > > abstractions of a device represented by rte_device.
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

Hi Yuanhan:

> -----Original Message-----
> From: Yuanhan Liu [mailto:yliu@fridaylinux.org]
> Sent: Wednesday, July 4, 2018 10:20 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
> Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 
> 
> On Mon, Jul 2, 2018, at 1:44 PM, Qi Zhang wrote:
> > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > application lock or unlock on specific ethdev, a locked device can't
> > be detached, this help applicaiton to prevent unexpected device
> > detaching, especially in multi-process envrionment.
> 
> I'm just wondering why the need of introducing such 2 (very specific) APIs,
> judging that it  looks like a ref count could solve this kind of issues perfectly.

Yes, if we separate the callback stuff, ref_count can solve the issue.
Actually this is my original implementation, but since we also need a callback for runtime check, 
so I just reuse it by registering a default callback.

But if we decide to replace lock_with_callback by adding a new PRE_DETACH event in rte_eth_dev_register_callback, 
We can take the ref_count way for simple device pin.

> 
> Also, the API name (_dev_lock/unlock) looks like an over killing, judging that
> you just want to pin a device (if I understand this commit log correctly).
> What firstly comes to my mind when I see "lock" was "no one can *access* a
> device after another has grabbed the lock", which doesn't quite match what
> you described here.

Yes, device is shared, any processes can access it, we are not going to introduce exclusive access.
But probably we should rename the API's name as rte_eth_dev_get/put? or rte_eth_dev_pin/unpin?

Thanks!
Qi

> 
> That said, if I were you, I would go with introducing few generic refcount APIs.
> 
>         --yliu
> 
> 
> >
> > Aslo introduce the new API rte_eth_dev_lock_with_callback and
> > rte_eth_dev_unlock_with callback to let application to register a
> > callback function which will be invoked before a device is going to be
> > detached, the return value of the function will decide if device will
> > continue be detached or not, this support application to do condition
> > check at runtime.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

04/07/2018 03:47, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 03/07/2018 17:08, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 02/07/2018 07:44, Qi Zhang:
> > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > > > > application lock or unlock on specific ethdev, a locked device
> > > > > can't be detached, this help applicaiton to prevent unexpected
> > > > > device detaching, especially in multi-process envrionment.
> > > >
> > > > Trying to understand: a process of an application could try to
> > > > detach a port while another process is against this decision.
> > > > Why an application needs to be protected against itself?
> > >
> > > I think we can regard this as a help function, it help application to simplified
> > the situation when one process want to detach a device while another one is
> > still using it.
> > > Application can register a callback which can do to necessary clean up (like
> > stop traffic, release memory ...) before device be detached.
> > 
> > Yes I agree such hook can be a good idea.
> > 
> > 
> > > > I guess it is only an application inter-process management.
> > > > If we really want to provide such helper in DPDK, it should not be
> > > > limited to ethdev.
> > >
> > > Once we move to eal layer, we will have rte_eal_dev_lock/unlock(devname,
> > busname).
> > > But its also better we keep rte_eth_dev_lock/unlock to make ethdev API
> > > more completed (any port be locked means underline rte_device also be
> > locked?) and this is same for other device family.
> > 
> > No. Again, a port is not exactly a device.
> > There can be several ports per device.
> 
> Yes, I know that.
> what I mean is, we should assume lock any port of that rte_device will prevent the device be detached.
> 
> > 
> > I think the right place for this hook is in port-level API (ethdev, crypto, etc). And
> > as we improve only ethdev currently, without any common genericity for other
> > device classes, it is probably fine in ethdev for now.
> > >
> > > > (for info, see class implementation:
> > > > https://patches.dpdk.org/patch/41605/)
> > > >
> > > > What about hardware unplug?
> > > > Can we detach the locked ports associated to the unplugged device?
> > >
> > > NO, we can't.
> > > But do you think, we need to introduce a "force detach" version, which will
> > ignore all locks.
> > 
> > No, I don't think so.
> > I am just trying to show that you cannot really "lock" a port.
> > Maybe you should just rename those functions.
> > After all, it is just a pre-detach hook.
> 
> > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > Perhaps we just need to improve the handling of the DESTROY event?
> 
> I have thought about this before.
> Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here need to give feedback, pass or fail will impact the following behavior, this make it special, so I separate it from all exist rte_eth_event_type handle mechanism. 

Look at _rte_eth_dev_callback_process, there is a "ret_param".

> The alternative solution is 
> we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH and reuse all exist API 
> rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.

I don't think we need a new event.
Let's try to use RTE_ETH_EVENT_DESTROY.

> But in _rte_eth_dev_callback_process we need to add a code branch for PRE_DETACH handle.
> 
> If (event = RTE_ETH_EVENT_PRE_DETACH)
> 	<...>.
> else {
> 	<....>
> }
> 
> And we may also need to keep rte_eth_dev_lock/unlock which will register a default callback for PRE_DETACH.

The default callback can be registered by the application.

> What do you think about?

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Thomas Monjalon]

04/07/2018 04:26, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 03/07/2018 23:57, Thomas Monjalon:
> > > 03/07/2018 17:03, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 03/07/2018 14:59, Zhang, Qi Z:
> > > > > > > > +do_eth_dev_attach(const char *devargs, uint16_t *port_id);
> > > > > > >
> > > > > > > So you are duplicating rte_eth_dev_attach which is flawed in
> > > > > > > its design and should be deprecated...
> > > > > >
> > > > > > OK, just to know this, but I guess it will not be the issue, if
> > > > > > we move the dev
> > > > > sync mechanism into eal layer in future right?
> > > > >
> > > > > Future is now :)
> > > > > We must stop mixing devargs and port id in the same layer.
> > > >
> > > > Ok, is there any RFC I can learn?
> > >
> > > RFC for what?
> > > It is just a design issue that we must stop propagating.
> > 
> > Please read at this commit, which is 2 years old:
> > 	http://git.dpdk.org/dpdk/commit/?id=b0fb26685570
> > It was starting to fix early design mistakes, but unfortunately it is not yet totally
> > fixed today.
> 
> OK, rte_eth_dev_attach is going to be deprecated.
> Do you mean we will use rte_eal_hotplug_add to attach a device directly, 
> then the device driver will be responsible for propagating all the ethdev port, 
> and application could register callback for RTE_ETH_EVENT_NEW to know new ports are created, 
> is that correct?

Exact!
All what you describe is already implemented.

To make it clear, EAL and ethdev must stay 2 separate layers.
The bridge between these 2 layers is done only by PMDs.


> > > > > > > As you may have noticed, rte_eth_dev_attach() is calling
> > > > > > > rte_eal_hotplug_add() which manages the EAL device.
> > > > > > > It is wrong because the relation between an ethdev port and an
> > > > > > > EAL device is not a 1:1 mapping.
> > > > > > > We must manage the ethdev port as one of the possible
> > > > > > > abstractions of a device represented by rte_device.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Wednesday, July 4, 2018 3:27 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 04/07/2018 03:47, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > > > > > application lock or unlock on specific ethdev, a locked device
> > > > > > can't be detached, this help applicaiton to prevent unexpected
> > > > > > device detaching, especially in multi-process envrionment.
> > > > >
> > > > > Trying to understand: a process of an application could try to
> > > > > detach a port while another process is against this decision.
> > > > > Why an application needs to be protected against itself?
> > > >
> > > > I think we can regard this as a help function, it help application
> > > > to simplified
> > > the situation when one process want to detach a device while another
> > > one is still using it.
> > > > Application can register a callback which can do to necessary
> > > > clean up (like
> > > stop traffic, release memory ...) before device be detached.
> > >
> > > Yes I agree such hook can be a good idea.
> > >
> > >
> > > > > I guess it is only an application inter-process management.
> > > > > If we really want to provide such helper in DPDK, it should not
> > > > > be limited to ethdev.
> > > >
> > > > Once we move to eal layer, we will have
> > > > rte_eal_dev_lock/unlock(devname,
> > > busname).
> > > > But its also better we keep rte_eth_dev_lock/unlock to make ethdev
> > > > API more completed (any port be locked means underline rte_device
> > > > also be
> > > locked?) and this is same for other device family.
> > >
> > > No. Again, a port is not exactly a device.
> > > There can be several ports per device.
> >
> > Yes, I know that.
> > what I mean is, we should assume lock any port of that rte_device will
> prevent the device be detached.
> >
> > >
> > > I think the right place for this hook is in port-level API (ethdev,
> > > crypto, etc). And as we improve only ethdev currently, without any
> > > common genericity for other device classes, it is probably fine in ethdev for
> now.
> > > >
> > > > > (for info, see class implementation:
> > > > > https://patches.dpdk.org/patch/41605/)
> > > > >
> > > > > What about hardware unplug?
> > > > > Can we detach the locked ports associated to the unplugged device?
> > > >
> > > > NO, we can't.
> > > > But do you think, we need to introduce a "force detach" version,
> > > > which will
> > > ignore all locks.
> > >
> > > No, I don't think so.
> > > I am just trying to show that you cannot really "lock" a port.
> > > Maybe you should just rename those functions.
> > > After all, it is just a pre-detach hook.
> >
> > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > Perhaps we just need to improve the handling of the DESTROY event?
> >
> > I have thought about this before.
> > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here
> need to give feedback, pass or fail will impact the following behavior, this
> make it special, so I separate it from all exist rte_eth_event_type handle
> mechanism.
> 
> Look at _rte_eth_dev_callback_process, there is a "ret_param".

OK, that should work.
> 
> > The alternative solution is
> > we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH and
> > reuse all exist API
> rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> 
> I don't think we need a new event.
> Let's try to use RTE_ETH_EVENT_DESTROY.

The problem is RTE_ETH_EVENT_DESTROY is used in rte_eth_dev_release_port already.
And in PMD, rte_eth_dev_release_port is called after dev_uninit, that mean its too late to reject a detach
So , do you mean we can remove _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in rte_eth_dev_release_port 

And where is right place to call _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
If can't be called in rte_eth_dev_detach, because if device is removed by rte_eal_hotplug_remove, it will be skipped.
probably we need to call this at the beginning of each PMD driver->remove?, that means, we need to change all PMD drivers?




> 
> > But in _rte_eth_dev_callback_process we need to add a code branch for
> PRE_DETACH handle.
> >
> > If (event = RTE_ETH_EVENT_PRE_DETACH)
> > 	<...>.
> > else {
> > 	<....>
> > }
> >
> > And we may also need to keep rte_eth_dev_lock/unlock which will register a
> default callback for PRE_DETACH.
> 
> The default callback can be registered by the application.

OK.
> 
> > What do you think about?
> 
> 

Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Wednesday, July 4, 2018 3:34 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; gaetan.rivet@6wind.com
> Subject: Re: [dpdk-dev] [PATCH v8 03/19] ethdev: enable hotplug on
> multi-process
> 
> 04/07/2018 04:26, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 03/07/2018 23:57, Thomas Monjalon:
> > > > 03/07/2018 17:03, Zhang, Qi Z:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 03/07/2018 14:59, Zhang, Qi Z:
> > > > > > > > > +do_eth_dev_attach(const char *devargs, uint16_t
> > > > > > > > > +*port_id);
> > > > > > > >
> > > > > > > > So you are duplicating rte_eth_dev_attach which is flawed
> > > > > > > > in its design and should be deprecated...
> > > > > > >
> > > > > > > OK, just to know this, but I guess it will not be the issue,
> > > > > > > if we move the dev
> > > > > > sync mechanism into eal layer in future right?
> > > > > >
> > > > > > Future is now :)
> > > > > > We must stop mixing devargs and port id in the same layer.
> > > > >
> > > > > Ok, is there any RFC I can learn?
> > > >
> > > > RFC for what?
> > > > It is just a design issue that we must stop propagating.
> > >
> > > Please read at this commit, which is 2 years old:
> > > 	http://git.dpdk.org/dpdk/commit/?id=b0fb26685570
> > > It was starting to fix early design mistakes, but unfortunately it
> > > is not yet totally fixed today.
> >
> > OK, rte_eth_dev_attach is going to be deprecated.
> > Do you mean we will use rte_eal_hotplug_add to attach a device
> > directly, then the device driver will be responsible for propagating
> > all the ethdev port, and application could register callback for
> > RTE_ETH_EVENT_NEW to know new ports are created, is that correct?
> 
> Exact!
> All what you describe is already implemented.
> 
> To make it clear, EAL and ethdev must stay 2 separate layers.
> The bridge between these 2 layers is done only by PMDs.

OK, I will move the IPC stuff from ethdev layer into eal layer in v9
Thanks!

> 
> 
> > > > > > > > As you may have noticed, rte_eth_dev_attach() is calling
> > > > > > > > rte_eal_hotplug_add() which manages the EAL device.
> > > > > > > > It is wrong because the relation between an ethdev port
> > > > > > > > and an EAL device is not a 1:1 mapping.
> > > > > > > > We must manage the ethdev port as one of the possible
> > > > > > > > abstractions of a device represented by rte_device.
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

04/07/2018 12:49, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 04/07/2018 03:47, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to let
> > > > > > > application lock or unlock on specific ethdev, a locked device
> > > > > > > can't be detached, this help applicaiton to prevent unexpected
> > > > > > > device detaching, especially in multi-process envrionment.
> > > > > >
> > > > > > Trying to understand: a process of an application could try to
> > > > > > detach a port while another process is against this decision.
> > > > > > Why an application needs to be protected against itself?
> > > > >
> > > > > I think we can regard this as a help function, it help application
> > > > > to simplified
> > > > the situation when one process want to detach a device while another
> > > > one is still using it.
> > > > > Application can register a callback which can do to necessary
> > > > > clean up (like
> > > > stop traffic, release memory ...) before device be detached.
> > > >
> > > > Yes I agree such hook can be a good idea.
[...]
> > > > After all, it is just a pre-detach hook.
> > >
> > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > Perhaps we just need to improve the handling of the DESTROY event?
> > >
> > > I have thought about this before.
> > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here
> > need to give feedback, pass or fail will impact the following behavior, this
> > make it special, so I separate it from all exist rte_eth_event_type handle
> > mechanism.
> > 
> > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> 
> OK, that should work.
> > 
> > > The alternative solution is
> > > we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH and
> > > reuse all exist API
> > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > 
> > I don't think we need a new event.
> > Let's try to use RTE_ETH_EVENT_DESTROY.
> 
> The problem is RTE_ETH_EVENT_DESTROY is used in rte_eth_dev_release_port already.
> And in PMD, rte_eth_dev_release_port is called after dev_uninit, that mean its too late to reject a detach

You're right.

It's a real mess currently.
The right order should be to remove ethdev ports before
removing the underlying EAL device. But it's strangely not the case.

We need to separate things.
The function rte_eth_dev_close can be used to remove an ethdev port
if we add a call to rte_eth_dev_release_port.
So we could call rte_eth_dev_close in PMD remove functions.
Is "close" a good time to ask confirmation to the application?
Or should we ask confirmation a step before, on "stop"?

> So , do you mean we can remove _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in rte_eth_dev_release_port 

I would say we need RTE_ETH_EVENT_DESTROY to notify that the port
is really destroyed.
Maybe the right thing to do is to add a new event
RTE_ETH_EVENT_CLOSE_REQUEST or something else.
Note that we already have 2 removal events in ethdev:
	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used anymore
	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted

> And where is right place to call _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> If can't be called in rte_eth_dev_detach, because if device is removed by rte_eal_hotplug_remove, it will be skipped.

No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2 different things.
One is a mix of ethdev and EAL (and should be deprecated),
the other one is for the underlying device at EAL level.

> probably we need to call this at the beginning of each PMD driver->remove?, that means, we need to change all PMD drivers?

Yes, we can call rte_eth_dev_stop and rte_eth_dev_close
at the beginning of PMD remove.
Note that there is already a helper rte_eth_dev_destroy called in
some PMD to achieve the removal, but curiously, it doesn't call
stop and close functions.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Thursday, July 5, 2018 5:42 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; arybchenko@solarflare.com
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 04/07/2018 12:49, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to
> > > > > > > > let application lock or unlock on specific ethdev, a
> > > > > > > > locked device can't be detached, this help applicaiton to
> > > > > > > > prevent unexpected device detaching, especially in multi-process
> envrionment.
> > > > > > >
> > > > > > > Trying to understand: a process of an application could try
> > > > > > > to detach a port while another process is against this decision.
> > > > > > > Why an application needs to be protected against itself?
> > > > > >
> > > > > > I think we can regard this as a help function, it help
> > > > > > application to simplified
> > > > > the situation when one process want to detach a device while
> > > > > another one is still using it.
> > > > > > Application can register a callback which can do to necessary
> > > > > > clean up (like
> > > > > stop traffic, release memory ...) before device be detached.
> > > > >
> > > > > Yes I agree such hook can be a good idea.
> [...]
> > > > > After all, it is just a pre-detach hook.
> > > >
> > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > Perhaps we just need to improve the handling of the DESTROY event?
> > > >
> > > > I have thought about this before.
> > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here
> > > need to give feedback, pass or fail will impact the following
> > > behavior, this make it special, so I separate it from all exist
> > > rte_eth_event_type handle mechanism.
> > >
> > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> >
> > OK, that should work.
> > >
> > > > The alternative solution is
> > > > we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH
> > > > and reuse all exist API
> > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > >
> > > I don't think we need a new event.
> > > Let's try to use RTE_ETH_EVENT_DESTROY.
> >
> > The problem is RTE_ETH_EVENT_DESTROY is used in
> rte_eth_dev_release_port already.
> > And in PMD, rte_eth_dev_release_port is called after dev_uninit, that
> > mean its too late to reject a detach
> 
> You're right.
> 
> It's a real mess currently.
> The right order should be to remove ethdev ports before removing the
> underlying EAL device. But it's strangely not the case.
> 
> We need to separate things.
> The function rte_eth_dev_close can be used to remove an ethdev port if we
> add a call to rte_eth_dev_release_port.
> So we could call rte_eth_dev_close in PMD remove functions.
> Is "close" a good time to ask confirmation to the application?
> Or should we ask confirmation a step before, on "stop"?

I think the confirmation should before any cleanup stage, it should at the beginning of driver->remove.
Also we should not put it into rte_eth_dev_stop, because, rte_eth_dev_stop can invoked by application directly, in that case, we don't what any callback be invoked.

> 
> > So , do you mean we can remove
> > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in
> > rte_eth_dev_release_port
> 
> I would say we need RTE_ETH_EVENT_DESTROY to notify that the port is really
> destroyed.
> Maybe the right thing to do is to add a new event
> RTE_ETH_EVENT_CLOSE_REQUEST or something else.
> Note that we already have 2 removal events in ethdev:
> 	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used anymore
> 	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted
> 
> > And where is right place to call
> _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> > If can't be called in rte_eth_dev_detach, because if device is removed by
> rte_eal_hotplug_remove, it will be skipped.
> 
> No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2 different things.
> One is a mix of ethdev and EAL (and should be deprecated), the other one is
> for the underlying device at EAL level.
> 
> > probably we need to call this at the beginning of each PMD driver->remove?,
> that means, we need to change all PMD drivers?
> 
> Yes, we can call rte_eth_dev_stop and rte_eth_dev_close at the beginning of
> PMD remove.
> Note that there is already a helper rte_eth_dev_destroy called in some PMD to
> achieve the removal, but curiously, it doesn't call stop and close functions.

Currently PMD implement driver->remove with different way, rte_eth_dev_stop / rte_eth_dev_close / rte_eth_dev_destroy is not always be invoked.
So Before we standardize what ethdev API and what sequence should be called in driver->remove (I think this is a separate task)
I will suggest 
1. Create another help function like _rte_eth_dev_allow_to_remove, 
2. the help function will call _rte_eth_dev_callback_process(RTE_ETH_EVENT_PRE_REMOVE) and update ret_param which contain a reject count.
3. the help function should to invoked at beginning at driver->remove and driver->remove will abort if the help function failed.

But once we standardized that , we can do cleanup to merge it into another rte_eth_xxx API in next step.

What do you think?

> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

05/07/2018 03:38, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 04/07/2018 12:49, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock to
> > > > > > > > > let application lock or unlock on specific ethdev, a
> > > > > > > > > locked device can't be detached, this help applicaiton to
> > > > > > > > > prevent unexpected device detaching, especially in multi-process
> > envrionment.
> > > > > > > >
> > > > > > > > Trying to understand: a process of an application could try
> > > > > > > > to detach a port while another process is against this decision.
> > > > > > > > Why an application needs to be protected against itself?
> > > > > > >
> > > > > > > I think we can regard this as a help function, it help
> > > > > > > application to simplified
> > > > > > the situation when one process want to detach a device while
> > > > > > another one is still using it.
> > > > > > > Application can register a callback which can do to necessary
> > > > > > > clean up (like
> > > > > > stop traffic, release memory ...) before device be detached.
> > > > > >
> > > > > > Yes I agree such hook can be a good idea.
> > [...]
> > > > > > After all, it is just a pre-detach hook.
> > > > >
> > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > > Perhaps we just need to improve the handling of the DESTROY event?
> > > > >
> > > > > I have thought about this before.
> > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook here
> > > > need to give feedback, pass or fail will impact the following
> > > > behavior, this make it special, so I separate it from all exist
> > > > rte_eth_event_type handle mechanism.
> > > >
> > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > >
> > > OK, that should work.
> > > >
> > > > > The alternative solution is
> > > > > we just introduce a new event type like RTE_ETH_EVENT_PRE_DETACH
> > > > > and reuse all exist API
> > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > >
> > > > I don't think we need a new event.
> > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > >
> > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > rte_eth_dev_release_port already.
> > > And in PMD, rte_eth_dev_release_port is called after dev_uninit, that
> > > mean its too late to reject a detach
> > 
> > You're right.
> > 
> > It's a real mess currently.
> > The right order should be to remove ethdev ports before removing the
> > underlying EAL device. But it's strangely not the case.
> > 
> > We need to separate things.
> > The function rte_eth_dev_close can be used to remove an ethdev port if we
> > add a call to rte_eth_dev_release_port.
> > So we could call rte_eth_dev_close in PMD remove functions.
> > Is "close" a good time to ask confirmation to the application?
> > Or should we ask confirmation a step before, on "stop"?
> 
> I think the confirmation should before any cleanup stage, it should at the beginning of driver->remove.

So you stop a port, even if the app policy is against detaching it?

> Also we should not put it into rte_eth_dev_stop, because, rte_eth_dev_stop can invoked by application directly, in that case, we don't what any callback be invoked.

It it the same to detach a port: it is invoked directly by application.
I thought you wanted a callback as helper for inter-process management?

> > > So , do you mean we can remove
> > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in
> > > rte_eth_dev_release_port
> > 
> > I would say we need RTE_ETH_EVENT_DESTROY to notify that the port is really
> > destroyed.
> > Maybe the right thing to do is to add a new event
> > RTE_ETH_EVENT_CLOSE_REQUEST or something else.
> > Note that we already have 2 removal events in ethdev:
> > 	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used anymore
> > 	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted
> > 
> > > And where is right place to call
> > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> > > If can't be called in rte_eth_dev_detach, because if device is removed by
> > rte_eal_hotplug_remove, it will be skipped.
> > 
> > No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2 different things.
> > One is a mix of ethdev and EAL (and should be deprecated), the other one is
> > for the underlying device at EAL level.
> > 
> > > probably we need to call this at the beginning of each PMD driver->remove?,
> > that means, we need to change all PMD drivers?
> > 
> > Yes, we can call rte_eth_dev_stop and rte_eth_dev_close at the beginning of
> > PMD remove.
> > Note that there is already a helper rte_eth_dev_destroy called in some PMD to
> > achieve the removal, but curiously, it doesn't call stop and close functions.
> 
> Currently PMD implement driver->remove with different way, rte_eth_dev_stop / rte_eth_dev_close / rte_eth_dev_destroy is not always be invoked.
> So Before we standardize what ethdev API and what sequence should be called in driver->remove (I think this is a separate task)
> I will suggest 
> 1. Create another help function like _rte_eth_dev_allow_to_remove, 
> 2. the help function will call _rte_eth_dev_callback_process(RTE_ETH_EVENT_PRE_REMOVE) and update ret_param which contain a reject count.
> 3. the help function should to invoked at beginning at driver->remove and driver->remove will abort if the help function failed.
> 
> But once we standardized that , we can do cleanup to merge it into another rte_eth_xxx API in next step.
> 
> What do you think?

No
All the problems we have today are because we preferred add more and more
functions instead of fixing the basic stuff. And it is especially the case
for all the detach crap.
So no.
Let's fix stuff first.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Thursday, July 5, 2018 9:55 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; arybchenko@solarflare.com
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 05/07/2018 03:38, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 04/07/2018 12:49, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock
> > > > > > > > > > to let application lock or unlock on specific ethdev,
> > > > > > > > > > a locked device can't be detached, this help
> > > > > > > > > > applicaiton to prevent unexpected device detaching,
> > > > > > > > > > especially in multi-process
> > > envrionment.
> > > > > > > > >
> > > > > > > > > Trying to understand: a process of an application could
> > > > > > > > > try to detach a port while another process is against this
> decision.
> > > > > > > > > Why an application needs to be protected against itself?
> > > > > > > >
> > > > > > > > I think we can regard this as a help function, it help
> > > > > > > > application to simplified
> > > > > > > the situation when one process want to detach a device while
> > > > > > > another one is still using it.
> > > > > > > > Application can register a callback which can do to
> > > > > > > > necessary clean up (like
> > > > > > > stop traffic, release memory ...) before device be detached.
> > > > > > >
> > > > > > > Yes I agree such hook can be a good idea.
> > > [...]
> > > > > > > After all, it is just a pre-detach hook.
> > > > > >
> > > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > > > Perhaps we just need to improve the handling of the DESTROY
> event?
> > > > > >
> > > > > > I have thought about this before.
> > > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook
> > > > > > here
> > > > > need to give feedback, pass or fail will impact the following
> > > > > behavior, this make it special, so I separate it from all exist
> > > > > rte_eth_event_type handle mechanism.
> > > > >
> > > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > > >
> > > > OK, that should work.
> > > > >
> > > > > > The alternative solution is
> > > > > > we just introduce a new event type like
> > > > > > RTE_ETH_EVENT_PRE_DETACH and reuse all exist API
> > > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > > >
> > > > > I don't think we need a new event.
> > > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > > >
> > > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > > rte_eth_dev_release_port already.
> > > > And in PMD, rte_eth_dev_release_port is called after dev_uninit,
> > > > that mean its too late to reject a detach
> > >
> > > You're right.
> > >
> > > It's a real mess currently.
> > > The right order should be to remove ethdev ports before removing the
> > > underlying EAL device. But it's strangely not the case.
> > >
> > > We need to separate things.
> > > The function rte_eth_dev_close can be used to remove an ethdev port
> > > if we add a call to rte_eth_dev_release_port.
> > > So we could call rte_eth_dev_close in PMD remove functions.
> > > Is "close" a good time to ask confirmation to the application?
> > > Or should we ask confirmation a step before, on "stop"?
> >
> > I think the confirmation should before any cleanup stage, it should at the
> beginning of driver->remove.
> 
> So you stop a port, even if the app policy is against detaching it?

My understanding is, stop and detach is different, we may stop a device and reconfigure it then restart it.
but for detach, properly we will not use it, unless it be probed again.
For dev_close , it should be called after dev_stop.
so we have to like below.

If (dev->started) {
	dev_stop /* but still problem here, if traffic is ongoing */
	if (dev_close()) {
		dev_start()
		return -EBUSY.
	}
} else {
	If (dev_close())
		Return _EBUSY
}

So for me, neither rte_eth_dev_stop and rte_eth_dev_close is the right place to check this.
But rte_eth_dev_destroy looks like a good one. We can put all the ethdev general logic into it, 
and PMD specific dev_unit will be called at last

> 
> > Also we should not put it into rte_eth_dev_stop, because, rte_eth_dev_stop
> can invoked by application directly, in that case, we don't what any callback be
> invoked.
> 
> It it the same to detach a port: it is invoked directly by application.
> I thought you wanted a callback as helper for inter-process management?
> 
> > > > So , do you mean we can remove
> > > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in
> > > > rte_eth_dev_release_port
> > >
> > > I would say we need RTE_ETH_EVENT_DESTROY to notify that the port is
> > > really destroyed.
> > > Maybe the right thing to do is to add a new event
> > > RTE_ETH_EVENT_CLOSE_REQUEST or something else.
> > > Note that we already have 2 removal events in ethdev:
> > > 	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used anymore
> > > 	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted
> > >
> > > > And where is right place to call
> > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> > > > If can't be called in rte_eth_dev_detach, because if device is
> > > > removed by
> > > rte_eal_hotplug_remove, it will be skipped.
> > >
> > > No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2 different
> things.
> > > One is a mix of ethdev and EAL (and should be deprecated), the other
> > > one is for the underlying device at EAL level.
> > >
> > > > probably we need to call this at the beginning of each PMD
> > > > driver->remove?,
> > > that means, we need to change all PMD drivers?
> > >
> > > Yes, we can call rte_eth_dev_stop and rte_eth_dev_close at the
> > > beginning of PMD remove.
> > > Note that there is already a helper rte_eth_dev_destroy called in
> > > some PMD to achieve the removal, but curiously, it doesn't call stop and
> close functions.
> >
> > Currently PMD implement driver->remove with different way,
> rte_eth_dev_stop / rte_eth_dev_close / rte_eth_dev_destroy is not always be
> invoked.
> > So Before we standardize what ethdev API and what sequence should be
> > called in driver->remove (I think this is a separate task) I will
> > suggest 1. Create another help function like
> > _rte_eth_dev_allow_to_remove, 2. the help function will call
> _rte_eth_dev_callback_process(RTE_ETH_EVENT_PRE_REMOVE) and update
> ret_param which contain a reject count.
> > 3. the help function should to invoked at beginning at driver->remove and
> driver->remove will abort if the help function failed.
> >
> > But once we standardized that , we can do cleanup to merge it into another
> rte_eth_xxx API in next step.
> >
> > What do you think?
> 
> No
> All the problems we have today are because we preferred add more and more
> functions instead of fixing the basic stuff. And it is especially the case for all the
> detach crap.
> So no.
> Let's fix stuff first.
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

05/07/2018 05:37, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 05/07/2018 03:38, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 04/07/2018 12:49, Zhang, Qi Z:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > > > Introduce API rte_eth_dev_lock and rte_eth_dev_unlock
> > > > > > > > > > > to let application lock or unlock on specific ethdev,
> > > > > > > > > > > a locked device can't be detached, this help
> > > > > > > > > > > applicaiton to prevent unexpected device detaching,
> > > > > > > > > > > especially in multi-process
> > > > envrionment.
> > > > > > > > > >
> > > > > > > > > > Trying to understand: a process of an application could
> > > > > > > > > > try to detach a port while another process is against this
> > decision.
> > > > > > > > > > Why an application needs to be protected against itself?
> > > > > > > > >
> > > > > > > > > I think we can regard this as a help function, it help
> > > > > > > > > application to simplified
> > > > > > > > the situation when one process want to detach a device while
> > > > > > > > another one is still using it.
> > > > > > > > > Application can register a callback which can do to
> > > > > > > > > necessary clean up (like
> > > > > > > > stop traffic, release memory ...) before device be detached.
> > > > > > > >
> > > > > > > > Yes I agree such hook can be a good idea.
> > > > [...]
> > > > > > > > After all, it is just a pre-detach hook.
> > > > > > >
> > > > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > > > > Perhaps we just need to improve the handling of the DESTROY
> > event?
> > > > > > >
> > > > > > > I have thought about this before.
> > > > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the hook
> > > > > > > here
> > > > > > need to give feedback, pass or fail will impact the following
> > > > > > behavior, this make it special, so I separate it from all exist
> > > > > > rte_eth_event_type handle mechanism.
> > > > > >
> > > > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > > > >
> > > > > OK, that should work.
> > > > > >
> > > > > > > The alternative solution is
> > > > > > > we just introduce a new event type like
> > > > > > > RTE_ETH_EVENT_PRE_DETACH and reuse all exist API
> > > > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > > > >
> > > > > > I don't think we need a new event.
> > > > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > > > >
> > > > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > > > rte_eth_dev_release_port already.
> > > > > And in PMD, rte_eth_dev_release_port is called after dev_uninit,
> > > > > that mean its too late to reject a detach
> > > >
> > > > You're right.
> > > >
> > > > It's a real mess currently.
> > > > The right order should be to remove ethdev ports before removing the
> > > > underlying EAL device. But it's strangely not the case.
> > > >
> > > > We need to separate things.
> > > > The function rte_eth_dev_close can be used to remove an ethdev port
> > > > if we add a call to rte_eth_dev_release_port.
> > > > So we could call rte_eth_dev_close in PMD remove functions.
> > > > Is "close" a good time to ask confirmation to the application?
> > > > Or should we ask confirmation a step before, on "stop"?
> > >
> > > I think the confirmation should before any cleanup stage, it should at the
> > beginning of driver->remove.
> > 
> > So you stop a port, even if the app policy is against detaching it?
> 
> My understanding is, stop and detach is different, we may stop a device and reconfigure it then restart it.
> but for detach, properly we will not use it, unless it be probed again.
> For dev_close , it should be called after dev_stop.
> so we have to like below.
> 
> If (dev->started) {
> 	dev_stop /* but still problem here, if traffic is ongoing */
> 	if (dev_close()) {
> 		dev_start()
> 		return -EBUSY.
> 	}
> } else {
> 	If (dev_close())
> 		Return _EBUSY
> }
> 
> So for me, neither rte_eth_dev_stop and rte_eth_dev_close is the right place to check this.
> But rte_eth_dev_destroy looks like a good one. We can put all the ethdev general logic into it, 
> and PMD specific dev_unit will be called at last

If you want to detach a port, you need to stop it.
If one process try to detach a port, but another process decides
(via callback) that the port should not be detached,
you will have stopped a port for no good reason.
To me it is a real design issue.


> > > Also we should not put it into rte_eth_dev_stop, because, rte_eth_dev_stop
> > can invoked by application directly, in that case, we don't what any callback be
> > invoked.
> > 
> > It it the same to detach a port: it is invoked directly by application.
> > I thought you wanted a callback as helper for inter-process management?
> > 
> > > > > So , do you mean we can remove
> > > > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in
> > > > > rte_eth_dev_release_port
> > > >
> > > > I would say we need RTE_ETH_EVENT_DESTROY to notify that the port is
> > > > really destroyed.
> > > > Maybe the right thing to do is to add a new event
> > > > RTE_ETH_EVENT_CLOSE_REQUEST or something else.
> > > > Note that we already have 2 removal events in ethdev:
> > > > 	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used anymore
> > > > 	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted
> > > >
> > > > > And where is right place to call
> > > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> > > > > If can't be called in rte_eth_dev_detach, because if device is
> > > > > removed by
> > > > rte_eal_hotplug_remove, it will be skipped.
> > > >
> > > > No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2 different
> > things.
> > > > One is a mix of ethdev and EAL (and should be deprecated), the other
> > > > one is for the underlying device at EAL level.
> > > >
> > > > > probably we need to call this at the beginning of each PMD
> > > > > driver->remove?,
> > > > that means, we need to change all PMD drivers?
> > > >
> > > > Yes, we can call rte_eth_dev_stop and rte_eth_dev_close at the
> > > > beginning of PMD remove.
> > > > Note that there is already a helper rte_eth_dev_destroy called in
> > > > some PMD to achieve the removal, but curiously, it doesn't call stop and
> > close functions.
> > >
> > > Currently PMD implement driver->remove with different way,
> > rte_eth_dev_stop / rte_eth_dev_close / rte_eth_dev_destroy is not always be
> > invoked.
> > > So Before we standardize what ethdev API and what sequence should be
> > > called in driver->remove (I think this is a separate task) I will
> > > suggest 1. Create another help function like
> > > _rte_eth_dev_allow_to_remove, 2. the help function will call
> > _rte_eth_dev_callback_process(RTE_ETH_EVENT_PRE_REMOVE) and update
> > ret_param which contain a reject count.
> > > 3. the help function should to invoked at beginning at driver->remove and
> > driver->remove will abort if the help function failed.
> > >
> > > But once we standardized that , we can do cleanup to merge it into another
> > rte_eth_xxx API in next step.
> > >
> > > What do you think?
> > 
> > No
> > All the problems we have today are because we preferred add more and more
> > functions instead of fixing the basic stuff. And it is especially the case for all the
> > detach crap.
> > So no.
> > Let's fix stuff first.
> > 
> > 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Thursday, July 5, 2018 3:23 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; arybchenko@solarflare.com
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 05/07/2018 05:37, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 05/07/2018 03:38, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 04/07/2018 12:49, Zhang, Qi Z:
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > > > > Introduce API rte_eth_dev_lock and
> > > > > > > > > > > > rte_eth_dev_unlock to let application lock or
> > > > > > > > > > > > unlock on specific ethdev, a locked device can't
> > > > > > > > > > > > be detached, this help applicaiton to prevent
> > > > > > > > > > > > unexpected device detaching, especially in
> > > > > > > > > > > > multi-process
> > > > > envrionment.
> > > > > > > > > > >
> > > > > > > > > > > Trying to understand: a process of an application
> > > > > > > > > > > could try to detach a port while another process is
> > > > > > > > > > > against this
> > > decision.
> > > > > > > > > > > Why an application needs to be protected against itself?
> > > > > > > > > >
> > > > > > > > > > I think we can regard this as a help function, it help
> > > > > > > > > > application to simplified
> > > > > > > > > the situation when one process want to detach a device
> > > > > > > > > while another one is still using it.
> > > > > > > > > > Application can register a callback which can do to
> > > > > > > > > > necessary clean up (like
> > > > > > > > > stop traffic, release memory ...) before device be detached.
> > > > > > > > >
> > > > > > > > > Yes I agree such hook can be a good idea.
> > > > > [...]
> > > > > > > > > After all, it is just a pre-detach hook.
> > > > > > > >
> > > > > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > > > > > Perhaps we just need to improve the handling of the
> > > > > > > > > DESTROY
> > > event?
> > > > > > > >
> > > > > > > > I have thought about this before.
> > > > > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the
> > > > > > > > hook here
> > > > > > > need to give feedback, pass or fail will impact the
> > > > > > > following behavior, this make it special, so I separate it
> > > > > > > from all exist rte_eth_event_type handle mechanism.
> > > > > > >
> > > > > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > > > > >
> > > > > > OK, that should work.
> > > > > > >
> > > > > > > > The alternative solution is we just introduce a new event
> > > > > > > > type like RTE_ETH_EVENT_PRE_DETACH and reuse all exist API
> > > > > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > > > > >
> > > > > > > I don't think we need a new event.
> > > > > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > > > > >
> > > > > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > > > > rte_eth_dev_release_port already.
> > > > > > And in PMD, rte_eth_dev_release_port is called after
> > > > > > dev_uninit, that mean its too late to reject a detach
> > > > >
> > > > > You're right.
> > > > >
> > > > > It's a real mess currently.
> > > > > The right order should be to remove ethdev ports before removing
> > > > > the underlying EAL device. But it's strangely not the case.
> > > > >
> > > > > We need to separate things.
> > > > > The function rte_eth_dev_close can be used to remove an ethdev
> > > > > port if we add a call to rte_eth_dev_release_port.
> > > > > So we could call rte_eth_dev_close in PMD remove functions.
> > > > > Is "close" a good time to ask confirmation to the application?
> > > > > Or should we ask confirmation a step before, on "stop"?
> > > >
> > > > I think the confirmation should before any cleanup stage, it
> > > > should at the
> > > beginning of driver->remove.
> > >
> > > So you stop a port, even if the app policy is against detaching it?
> >
> > My understanding is, stop and detach is different, we may stop a device and
> reconfigure it then restart it.
> > but for detach, properly we will not use it, unless it be probed again.
> > For dev_close , it should be called after dev_stop.
> > so we have to like below.
> >
> > If (dev->started) {
> > 	dev_stop /* but still problem here, if traffic is ongoing */
> > 	if (dev_close()) {
> > 		dev_start()
> > 		return -EBUSY.
> > 	}
> > } else {
> > 	If (dev_close())
> > 		Return _EBUSY
> > }
> >
> > So for me, neither rte_eth_dev_stop and rte_eth_dev_close is the right place
> to check this.
> > But rte_eth_dev_destroy looks like a good one. We can put all the
> > ethdev general logic into it, and PMD specific dev_unit will be called
> > at last
> 
> If you want to detach a port, you need to stop it.
> If one process try to detach a port, but another process decides (via callback)
> that the port should not be detached, you will have stopped a port for no good
> reason.
> To me it is a real design issue.

Yes, so I think we still need two iterates for detach.
First iterate to get agreement on all processes.
Secondary iterate to do the detach.

But how?
> 
> 
> > > > Also we should not put it into rte_eth_dev_stop, because,
> > > > rte_eth_dev_stop
> > > can invoked by application directly, in that case, we don't what any
> > > callback be invoked.
> > >
> > > It it the same to detach a port: it is invoked directly by application.
> > > I thought you wanted a callback as helper for inter-process management?
> > >
> > > > > > So , do you mean we can remove
> > > > > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROPY) in
> > > > > > rte_eth_dev_release_port
> > > > >
> > > > > I would say we need RTE_ETH_EVENT_DESTROY to notify that the
> > > > > port is really destroyed.
> > > > > Maybe the right thing to do is to add a new event
> > > > > RTE_ETH_EVENT_CLOSE_REQUEST or something else.
> > > > > Note that we already have 2 removal events in ethdev:
> > > > > 	- RTE_ETH_EVENT_INTR_RMV when the port cannot be used
> anymore
> > > > > 	- RTE_ETH_EVENT_DESTROY when the port is going to be deleted
> > > > >
> > > > > > And where is right place to call
> > > > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_DESTROY)?
> > > > > > If can't be called in rte_eth_dev_detach, because if device is
> > > > > > removed by
> > > > > rte_eal_hotplug_remove, it will be skipped.
> > > > >
> > > > > No, rte_eth_dev_detach and rte_eal_hotplug_remove are 2
> > > > > different
> > > things.
> > > > > One is a mix of ethdev and EAL (and should be deprecated), the
> > > > > other one is for the underlying device at EAL level.
> > > > >
> > > > > > probably we need to call this at the beginning of each PMD
> > > > > > driver->remove?,
> > > > > that means, we need to change all PMD drivers?
> > > > >
> > > > > Yes, we can call rte_eth_dev_stop and rte_eth_dev_close at the
> > > > > beginning of PMD remove.
> > > > > Note that there is already a helper rte_eth_dev_destroy called
> > > > > in some PMD to achieve the removal, but curiously, it doesn't
> > > > > call stop and
> > > close functions.
> > > >
> > > > Currently PMD implement driver->remove with different way,
> > > rte_eth_dev_stop / rte_eth_dev_close / rte_eth_dev_destroy is not
> > > always be invoked.
> > > > So Before we standardize what ethdev API and what sequence should
> > > > be called in driver->remove (I think this is a separate task) I
> > > > will suggest 1. Create another help function like
> > > > _rte_eth_dev_allow_to_remove, 2. the help function will call
> > > _rte_eth_dev_callback_process(RTE_ETH_EVENT_PRE_REMOVE) and
> update
> > > ret_param which contain a reject count.
> > > > 3. the help function should to invoked at beginning at
> > > > driver->remove and
> > > driver->remove will abort if the help function failed.
> > > >
> > > > But once we standardized that , we can do cleanup to merge it into
> > > > another
> > > rte_eth_xxx API in next step.
> > > >
> > > > What do you think?
> > >
> > > No
> > > All the problems we have today are because we preferred add more and
> > > more functions instead of fixing the basic stuff. And it is
> > > especially the case for all the detach crap.
> > > So no.
> > > Let's fix stuff first.
> > >
> > >
> >
> 
> 
> 
> 

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Thomas Monjalon]

05/07/2018 11:54, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 05/07/2018 05:37, Zhang, Qi Z:
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > 05/07/2018 03:38, Zhang, Qi Z:
> > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > 04/07/2018 12:49, Zhang, Qi Z:
> > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > > > > > Introduce API rte_eth_dev_lock and
> > > > > > > > > > > > > rte_eth_dev_unlock to let application lock or
> > > > > > > > > > > > > unlock on specific ethdev, a locked device can't
> > > > > > > > > > > > > be detached, this help applicaiton to prevent
> > > > > > > > > > > > > unexpected device detaching, especially in
> > > > > > > > > > > > > multi-process
> > > > > > envrionment.
> > > > > > > > > > > >
> > > > > > > > > > > > Trying to understand: a process of an application
> > > > > > > > > > > > could try to detach a port while another process is
> > > > > > > > > > > > against this
> > > > decision.
> > > > > > > > > > > > Why an application needs to be protected against itself?
> > > > > > > > > > >
> > > > > > > > > > > I think we can regard this as a help function, it help
> > > > > > > > > > > application to simplified
> > > > > > > > > > the situation when one process want to detach a device
> > > > > > > > > > while another one is still using it.
> > > > > > > > > > > Application can register a callback which can do to
> > > > > > > > > > > necessary clean up (like
> > > > > > > > > > stop traffic, release memory ...) before device be detached.
> > > > > > > > > >
> > > > > > > > > > Yes I agree such hook can be a good idea.
> > > > > > [...]
> > > > > > > > > > After all, it is just a pre-detach hook.
> > > > > > > > >
> > > > > > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY callback?
> > > > > > > > > > Perhaps we just need to improve the handling of the
> > > > > > > > > > DESTROY
> > > > event?
> > > > > > > > >
> > > > > > > > > I have thought about this before.
> > > > > > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook, the
> > > > > > > > > hook here
> > > > > > > > need to give feedback, pass or fail will impact the
> > > > > > > > following behavior, this make it special, so I separate it
> > > > > > > > from all exist rte_eth_event_type handle mechanism.
> > > > > > > >
> > > > > > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > > > > > >
> > > > > > > OK, that should work.
> > > > > > > >
> > > > > > > > > The alternative solution is we just introduce a new event
> > > > > > > > > type like RTE_ETH_EVENT_PRE_DETACH and reuse all exist API
> > > > > > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > > > > > >
> > > > > > > > I don't think we need a new event.
> > > > > > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > > > > > >
> > > > > > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > > > > > rte_eth_dev_release_port already.
> > > > > > > And in PMD, rte_eth_dev_release_port is called after
> > > > > > > dev_uninit, that mean its too late to reject a detach
> > > > > >
> > > > > > You're right.
> > > > > >
> > > > > > It's a real mess currently.
> > > > > > The right order should be to remove ethdev ports before removing
> > > > > > the underlying EAL device. But it's strangely not the case.
> > > > > >
> > > > > > We need to separate things.
> > > > > > The function rte_eth_dev_close can be used to remove an ethdev
> > > > > > port if we add a call to rte_eth_dev_release_port.
> > > > > > So we could call rte_eth_dev_close in PMD remove functions.
> > > > > > Is "close" a good time to ask confirmation to the application?
> > > > > > Or should we ask confirmation a step before, on "stop"?
> > > > >
> > > > > I think the confirmation should before any cleanup stage, it
> > > > > should at the
> > > > beginning of driver->remove.
> > > >
> > > > So you stop a port, even if the app policy is against detaching it?
> > >
> > > My understanding is, stop and detach is different, we may stop a device and
> > reconfigure it then restart it.
> > > but for detach, properly we will not use it, unless it be probed again.
> > > For dev_close , it should be called after dev_stop.
> > > so we have to like below.
> > >
> > > If (dev->started) {
> > > 	dev_stop /* but still problem here, if traffic is ongoing */
> > > 	if (dev_close()) {
> > > 		dev_start()
> > > 		return -EBUSY.
> > > 	}
> > > } else {
> > > 	If (dev_close())
> > > 		Return _EBUSY
> > > }
> > >
> > > So for me, neither rte_eth_dev_stop and rte_eth_dev_close is the right place
> > to check this.
> > > But rte_eth_dev_destroy looks like a good one. We can put all the
> > > ethdev general logic into it, and PMD specific dev_unit will be called
> > > at last
> > 
> > If you want to detach a port, you need to stop it.
> > If one process try to detach a port, but another process decides (via callback)
> > that the port should not be detached, you will have stopped a port for no good
> > reason.
> > To me it is a real design issue.
> 
> Yes, so I think we still need two iterates for detach.
> First iterate to get agreement on all processes.
> Secondary iterate to do the detach.
> 
> But how?

An option is to let the application manages itself
its process synchronization and authorization for detaching devices.

Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Thursday, July 5, 2018 6:55 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; arybchenko@solarflare.com
> Subject: Re: [dpdk-dev] [PATCH v8 04/19] ethdev: introduce device lock
> 
> 05/07/2018 11:54, Zhang, Qi Z:
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > 05/07/2018 05:37, Zhang, Qi Z:
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > 05/07/2018 03:38, Zhang, Qi Z:
> > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > 04/07/2018 12:49, Zhang, Qi Z:
> > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > 04/07/2018 03:47, Zhang, Qi Z:
> > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > > 03/07/2018 17:08, Zhang, Qi Z:
> > > > > > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > > > > > > > > > > 02/07/2018 07:44, Qi Zhang:
> > > > > > > > > > > > > > Introduce API rte_eth_dev_lock and
> > > > > > > > > > > > > > rte_eth_dev_unlock to let application lock or
> > > > > > > > > > > > > > unlock on specific ethdev, a locked device
> > > > > > > > > > > > > > can't be detached, this help applicaiton to
> > > > > > > > > > > > > > prevent unexpected device detaching,
> > > > > > > > > > > > > > especially in multi-process
> > > > > > > envrionment.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Trying to understand: a process of an
> > > > > > > > > > > > > application could try to detach a port while
> > > > > > > > > > > > > another process is against this
> > > > > decision.
> > > > > > > > > > > > > Why an application needs to be protected against itself?
> > > > > > > > > > > >
> > > > > > > > > > > > I think we can regard this as a help function, it
> > > > > > > > > > > > help application to simplified
> > > > > > > > > > > the situation when one process want to detach a
> > > > > > > > > > > device while another one is still using it.
> > > > > > > > > > > > Application can register a callback which can do
> > > > > > > > > > > > to necessary clean up (like
> > > > > > > > > > > stop traffic, release memory ...) before device be detached.
> > > > > > > > > > >
> > > > > > > > > > > Yes I agree such hook can be a good idea.
> > > > > > > [...]
> > > > > > > > > > > After all, it is just a pre-detach hook.
> > > > > > > > > >
> > > > > > > > > > > Wait, how is it different of RTE_ETH_EVENT_DESTROY
> callback?
> > > > > > > > > > > Perhaps we just need to improve the handling of the
> > > > > > > > > > > DESTROY
> > > > > event?
> > > > > > > > > >
> > > > > > > > > > I have thought about this before.
> > > > > > > > > > Not like RTE_ETH_EVENT_DESTROY and other event hook,
> > > > > > > > > > the hook here
> > > > > > > > > need to give feedback, pass or fail will impact the
> > > > > > > > > following behavior, this make it special, so I separate
> > > > > > > > > it from all exist rte_eth_event_type handle mechanism.
> > > > > > > > >
> > > > > > > > > Look at _rte_eth_dev_callback_process, there is a "ret_param".
> > > > > > > >
> > > > > > > > OK, that should work.
> > > > > > > > >
> > > > > > > > > > The alternative solution is we just introduce a new
> > > > > > > > > > event type like RTE_ETH_EVENT_PRE_DETACH and reuse all
> > > > > > > > > > exist API
> > > > > > > > > rte_eth_dev_callback_register/rte_eth_dev_callback_unregister.
> > > > > > > > >
> > > > > > > > > I don't think we need a new event.
> > > > > > > > > Let's try to use RTE_ETH_EVENT_DESTROY.
> > > > > > > >
> > > > > > > > The problem is RTE_ETH_EVENT_DESTROY is used in
> > > > > > > rte_eth_dev_release_port already.
> > > > > > > > And in PMD, rte_eth_dev_release_port is called after
> > > > > > > > dev_uninit, that mean its too late to reject a detach
> > > > > > >
> > > > > > > You're right.
> > > > > > >
> > > > > > > It's a real mess currently.
> > > > > > > The right order should be to remove ethdev ports before
> > > > > > > removing the underlying EAL device. But it's strangely not the case.
> > > > > > >
> > > > > > > We need to separate things.
> > > > > > > The function rte_eth_dev_close can be used to remove an
> > > > > > > ethdev port if we add a call to rte_eth_dev_release_port.
> > > > > > > So we could call rte_eth_dev_close in PMD remove functions.
> > > > > > > Is "close" a good time to ask confirmation to the application?
> > > > > > > Or should we ask confirmation a step before, on "stop"?
> > > > > >
> > > > > > I think the confirmation should before any cleanup stage, it
> > > > > > should at the
> > > > > beginning of driver->remove.
> > > > >
> > > > > So you stop a port, even if the app policy is against detaching it?
> > > >
> > > > My understanding is, stop and detach is different, we may stop a
> > > > device and
> > > reconfigure it then restart it.
> > > > but for detach, properly we will not use it, unless it be probed again.
> > > > For dev_close , it should be called after dev_stop.
> > > > so we have to like below.
> > > >
> > > > If (dev->started) {
> > > > 	dev_stop /* but still problem here, if traffic is ongoing */
> > > > 	if (dev_close()) {
> > > > 		dev_start()
> > > > 		return -EBUSY.
> > > > 	}
> > > > } else {
> > > > 	If (dev_close())
> > > > 		Return _EBUSY
> > > > }
> > > >
> > > > So for me, neither rte_eth_dev_stop and rte_eth_dev_close is the
> > > > right place
> > > to check this.
> > > > But rte_eth_dev_destroy looks like a good one. We can put all the
> > > > ethdev general logic into it, and PMD specific dev_unit will be
> > > > called at last
> > >
> > > If you want to detach a port, you need to stop it.
> > > If one process try to detach a port, but another process decides
> > > (via callback) that the port should not be detached, you will have
> > > stopped a port for no good reason.
> > > To me it is a real design issue.
> >
> > Yes, so I think we still need two iterates for detach.
> > First iterate to get agreement on all processes.
> > Secondary iterate to do the detach.
> >
> > But how?
> 
> An option is to let the application manages itself its process synchronization
> and authorization for detaching devices.

Yes, I agree
Before we figure out a comprehensive solution, leave this to application's handle so far.
I will remove this one from patchset.
> 

[dpdk-dev] [PATCH v9 00/19] enable hotplug on multi-process

[Qi Zhang]

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable port detach on secondary process
  net/ixgbe: enable port detach on secondary process
  net/af_packet: enable port detach on secondary process
  net/bonding: enable port detach on secondary process
  net/kni: enable port detach on secondary process
  net/null: enable port detach on secondary process
  net/octeontx: enable port detach on secondary process
  net/pcap: enable port detach on secondary process
  net/softnic: enable port detach on secondary process
  net/tap: enable port detach on secondary process
  net/vhost: enable port detach on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               |  88 ++++++-
 drivers/net/af_packet/rte_eth_af_packet.c      |  11 +
 drivers/net/bonding/rte_eth_bond_pmd.c         |  11 +
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |  11 +
 drivers/net/null/rte_eth_null.c                |  16 +-
 drivers/net/octeontx/octeontx_ethdev.c         |  16 ++
 drivers/net/pcap/rte_eth_pcap.c                |  15 +-
 drivers/net/softnic/rte_eth_softnic.c          |  19 +-
 drivers/net/tap/rte_eth_tap.c                  |  17 +-
 drivers/net/vhost/rte_eth_vhost.c              |  11 +
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 197 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 168 +++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 346 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  44 ++++
 lib/librte_eal/common/include/rte_bus.h        |   3 +
 lib/librte_eal/common/include/rte_dev.h        |   9 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 34 files changed, 1147 insertions(+), 62 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v9 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..269586d88 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..a46d9e182 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	/**
+	 * PCI device can only be globally detached directly by a
+	 * primary process. In secondary process, we only need to
+	 * release port.
+	 */
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v9 02/19] bus/pci: fix PCI address compare

[Qi Zhang]

When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
is valid. So compare the 8th byte will cause the unexpected result, which
happens when repeatedly attach/detach a device.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..dd25c3542 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_vfio_tailq)
 
+/* Compair two pci address */
+static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
+{
+	if (addr1->domain == addr2->domain &&
+		addr1->bus == addr2->bus &&
+		addr1->devid == addr2->devid &&
+		addr1->function == addr2->function)
+		return 0;
+	return 1;
+}
+
 int
 pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
 		    void *buf, size_t len, off_t offs)
@@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 	/* Get vfio_res */
 	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
 			continue;
 		break;
 	}
-- 
2.13.6

[dpdk-dev] [PATCH v9 03/19] bus/pci: enable vfio unmap resource for secondary

[Qi Zhang]

Subroutine to unmap VFIO resource is shared by secondary and
primary, and it does not work on the secondary process.
The patch adds a dedicate function to handle the situation
when a device is unmapped on a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 75 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index dd25c3542..72481ac45 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -595,6 +595,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 		dev->mem_resource[i].addr = maps[i].addr;
 	}
 
+	/* we need save vfio_dev_fd, so it can be used during release */
+	dev->intr_handle.vfio_dev_fd = vfio_dev_fd;
+
 	return 0;
 err_vfio_dev_fd:
 	close(vfio_dev_fd);
@@ -614,8 +617,8 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		return pci_vfio_map_resource_secondary(dev);
 }
 
-int
-pci_vfio_unmap_resource(struct rte_pci_device *dev)
+static int
+pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
 {
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
@@ -687,6 +690,74 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	return 0;
 }
 
+static int
+pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
+{
+	char pci_addr[PATH_MAX] = {0};
+	struct rte_pci_addr *loc = &dev->addr;
+	int i, ret;
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct mapped_pci_res_list *vfio_res_list;
+
+	struct pci_map *maps;
+
+	/* store PCI address string */
+	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
+			loc->domain, loc->bus, loc->devid, loc->function);
+
+	ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+				  dev->intr_handle.vfio_dev_fd);
+	if (ret < 0) {
+		RTE_LOG(ERR, EAL,
+			"%s(): cannot release device\n", __func__);
+		return ret;
+	}
+
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	/* Get vfio_res */
+	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
+			continue;
+		break;
+	}
+	/* if we haven't found our tailq entry, something's wrong */
+	if (vfio_res == NULL) {
+		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
+				pci_addr);
+		return -1;
+	}
+
+	/* unmap BARs */
+	maps = vfio_res->maps;
+
+	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
+		pci_addr);
+	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+
+		/*
+		 * We do not need to be aware of MSI-X table BAR mappings as
+		 * when mapping. Just using current maps array is enough
+		 */
+		if (maps[i].addr) {
+			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
+				pci_addr, maps[i].addr);
+			pci_unmap_resource(maps[i].addr, maps[i].size);
+		}
+	}
+
+	return 0;
+}
+
+int
+pci_vfio_unmap_resource(struct rte_pci_device *dev)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return pci_vfio_unmap_resource_primary(dev);
+	else
+		return pci_vfio_unmap_resource_secondary(dev);
+}
+
 int
 pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
 		    struct rte_pci_ioport *p)
-- 
2.13.6

[dpdk-dev] [PATCH v9 04/19] vfio: remove uneccessary IPC for group fd clear

[Qi Zhang]

Clear vfio_group_fd is not necessary to involve any IPC.
Also, current IPC implementation for SOCKET_CLR_GROUP is not
correct. rte_vfio_clear_group on secondary will always fail,
that prevent device be detached correctly on a secondary process.
The patch simply removes all IPC related stuff in
rte_vfio_clear_group.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_vfio.c         | 45 +++++---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |  1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |  8 -----
 3 files changed, 8 insertions(+), 46 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index a2bbdfbf4..c0eccddc3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -575,10 +575,6 @@ int
 rte_vfio_clear_group(int vfio_group_fd)
 {
 	int i;
-	struct rte_mp_msg mp_req, *mp_rep;
-	struct rte_mp_reply mp_reply;
-	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
-	struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
 	struct vfio_config *vfio_cfg;
 
 	vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
@@ -587,40 +583,15 @@ rte_vfio_clear_group(int vfio_group_fd)
 		return -1;
 	}
 
-	if (internal_config.process_type == RTE_PROC_PRIMARY) {
-
-		i = get_vfio_group_idx(vfio_group_fd);
-		if (i < 0)
-			return -1;
-		vfio_cfg->vfio_groups[i].group_num = -1;
-		vfio_cfg->vfio_groups[i].fd = -1;
-		vfio_cfg->vfio_groups[i].devices = 0;
-		vfio_cfg->vfio_active_groups--;
-		return 0;
-	}
-
-	p->req = SOCKET_CLR_GROUP;
-	p->group_num = vfio_group_fd;
-	strcpy(mp_req.name, EAL_VFIO_MP);
-	mp_req.len_param = sizeof(*p);
-	mp_req.num_fds = 0;
-
-	if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
-	    mp_reply.nb_received == 1) {
-		mp_rep = &mp_reply.msgs[0];
-		p = (struct vfio_mp_param *)mp_rep->param;
-		if (p->result == SOCKET_OK) {
-			free(mp_reply.msgs);
-			return 0;
-		} else if (p->result == SOCKET_NO_FD)
-			RTE_LOG(ERR, EAL, "  BAD VFIO group fd!\n");
-		else
-			RTE_LOG(ERR, EAL, "  no such VFIO group fd!\n");
-
-		free(mp_reply.msgs);
-	}
+	i = get_vfio_group_idx(vfio_group_fd);
+	if (i < 0)
+		return -1;
+	vfio_cfg->vfio_groups[i].group_num = -1;
+	vfio_cfg->vfio_groups[i].fd = -1;
+	vfio_cfg->vfio_groups[i].devices = 0;
+	vfio_cfg->vfio_active_groups--;
 
-	return -1;
+	return 0;
 }
 
 int
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index e65b10374..68d4750a5 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -129,7 +129,6 @@ int vfio_mp_sync_setup(void);
 
 #define SOCKET_REQ_CONTAINER 0x100
 #define SOCKET_REQ_GROUP 0x200
-#define SOCKET_CLR_GROUP 0x300
 #define SOCKET_OK 0x0
 #define SOCKET_NO_FD 0x1
 #define SOCKET_ERR 0xFF
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
index 9c202bb08..680a24aae 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
@@ -55,14 +55,6 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
 			reply.fds[0] = fd;
 		}
 		break;
-	case SOCKET_CLR_GROUP:
-		r->req = SOCKET_CLR_GROUP;
-		r->group_num = m->group_num;
-		if (rte_vfio_clear_group(m->group_num) < 0)
-			r->result = SOCKET_NO_FD;
-		else
-			r->result = SOCKET_OK;
-		break;
 	case SOCKET_REQ_CONTAINER:
 		r->req = SOCKET_REQ_CONTAINER;
 		fd = rte_vfio_get_container_fd();
-- 
2.13.6

[dpdk-dev] [PATCH v9 05/19] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 140 +++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 +++++++
 lib/librte_eal/common/hotplug_mp.c      | 181 ++++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  44 ++++++++
 lib/librte_eal/common/include/rte_bus.h |   3 +
 lib/librte_eal/common/include/rte_dev.h |   9 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 ++
 10 files changed, 418 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 3fd33f1e4..4ecc73b42 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 14c5f05fa..fb1a122ae 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -16,8 +16,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -102,8 +104,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -168,8 +171,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -197,11 +199,139 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
 	return ret;
 }
 
 int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+	if (ret)
+		/* step h) */
+		return ret;
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return -ENODEV;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
+	return -ENODEV;
+}
+
+int __rte_experimental
 rte_dev_event_callback_register(const char *device_name,
 				rte_dev_event_cb_fn cb_fn,
 				void *cb_arg)
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..1883b05d1 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,41 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..261d17fe6
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+		handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..f24ba107b
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[RTE_BUS_NAME_MAX_LEN];
+	char devname[RTE_DEV_NAME_MAX_LEN];
+	char devargs[RTE_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..720f7c3c8 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -197,6 +197,9 @@ struct rte_bus_conf {
 typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
 
 
+/* Max length for a bus name */
+#define RTE_BUS_NAME_MAX_LEN 32
+
 /**
  * A structure describing a generic bus.
  */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 3879ff3ca..667df20f0 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -152,6 +152,9 @@ struct rte_driver {
  */
 #define RTE_DEV_NAME_MAX_LEN 64
 
+/* Max devargs length be allowed */
+#define RTE_DEV_ARGS_MAX_LEN 128
+
 /**
  * A structure describing a generic device.
  */
@@ -193,6 +196,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +218,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 8a3dcfee0..0f2cf9a1a 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -25,6 +25,7 @@ common_sources = files(
 	'eal_common_tailqs.c',
 	'eal_common_thread.c',
 	'eal_common_timer.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 3719ec9d7..6c225b54a 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..0c6984990 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -850,6 +850,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v9 06/19] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c |  36 ++++++-
 lib/librte_eal/common/hotplug_mp.c     | 175 ++++++++++++++++++++++++++++++++-
 2 files changed, 202 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index fb1a122ae..195e1fe00 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -218,8 +218,22 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
 	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
 	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
 
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -ENOTSUP;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
 
 	/**
 	 * attach a device from primary start from here:
@@ -279,8 +293,22 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
 	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
 
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -ENOTSUP;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
 
 	/**
 	 * detach a device from primary start from here:
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 261d17fe6..68ca18bbe 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (!tmp_req.result) {
+			ret = do_dev_hotplug_remove(req->busname,
+						req->devname);
+			if (ret) {
+				RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+				goto rollback;
+			}
+		} else {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+	if (ret)
+		RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -114,8 +260,27 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v9 07/19] net/i40e: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v9 08/19] net/ixgbe: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v9 09/19] net/af_packet: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..33ac19de8 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -935,6 +935,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +987,16 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v9 10/19] net/bonding: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..da45ba9ba 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3062,6 +3062,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3169,16 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v9 11/19] net/kni: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..e5679c76a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -419,6 +419,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +464,16 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v9 12/19] net/null: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..2f040729b 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -623,6 +623,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +668,31 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v9 13/19] net/octeontx: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..497bacdc6 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,18 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			/* detach device on local pprocess only */
+			if (strlen(rte_vdev_device_args(dev)) == 0) {
+				rte_eth_dev_release_port_private(eth_dev);
+				continue;
+			}
+			/**
+			 * else this is a private device for current process
+			 * so continue with normal detach scenario
+			 */
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1161,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v9 14/19] net/pcap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..6cc20c2b2 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -925,6 +925,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1017,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1025,22 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v9 15/19] net/softnic: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..a45a7b0dd 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -750,6 +750,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +804,29 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(vdev)) == 0)
+			return rte_eth_dev_release_port_private(dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v9 16/19] net/tap: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..bb5f20b01 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1759,6 +1759,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1828,24 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v9 17/19] net/vhost: enable port detach on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..f773711b4 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1353,6 +1353,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1436,16 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/* detach device on local pprocess only */
+		if (strlen(rte_vdev_device_args(dev)) == 0)
+			return rte_eth_dev_release_port_private(eth_dev);
+		/**
+		 * else this is a private device for current process
+		 * so continue with normal detach scenario
+		 */
+	}
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v9 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++++
 examples/multi_process/hotplug_mp/commands.c | 197 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 ++++++
 5 files changed, 272 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..c09a57bfa
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2010-2014 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..fb7198d51
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v9 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..1251e4b5b 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,12 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -60,6 +66,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

[dpdk-dev] [PATCH v10 00/19] enable hotplug on multi-process

[Qi Zhang]

v10:
  Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               |  88 ++++++-
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 197 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 168 +++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 346 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  44 ++++
 lib/librte_eal/common/include/rte_bus.h        |   3 +
 lib/librte_eal/common/include/rte_dev.h        |   9 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 34 files changed, 1092 insertions(+), 78 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v10 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..269586d88 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..a46d9e182 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	/**
+	 * PCI device can only be globally detached directly by a
+	 * primary process. In secondary process, we only need to
+	 * release port.
+	 */
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v10 02/19] bus/pci: fix PCI address compare

[Qi Zhang]

When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
is valid. So compare the 8th byte will cause the unexpected result, which
happens when repeatedly attach/detach a device.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..dd25c3542 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_vfio_tailq)
 
+/* Compair two pci address */
+static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
+{
+	if (addr1->domain == addr2->domain &&
+		addr1->bus == addr2->bus &&
+		addr1->devid == addr2->devid &&
+		addr1->function == addr2->function)
+		return 0;
+	return 1;
+}
+
 int
 pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
 		    void *buf, size_t len, off_t offs)
@@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 	/* Get vfio_res */
 	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
 			continue;
 		break;
 	}
-- 
2.13.6

[dpdk-dev] [PATCH v10 03/19] bus/pci: enable vfio unmap resource for secondary

[Qi Zhang]

Subroutine to unmap VFIO resource is shared by secondary and
primary, and it does not work on the secondary process.
The patch adds a dedicate function to handle the situation
when a device is unmapped on a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 75 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index dd25c3542..72481ac45 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -595,6 +595,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 		dev->mem_resource[i].addr = maps[i].addr;
 	}
 
+	/* we need save vfio_dev_fd, so it can be used during release */
+	dev->intr_handle.vfio_dev_fd = vfio_dev_fd;
+
 	return 0;
 err_vfio_dev_fd:
 	close(vfio_dev_fd);
@@ -614,8 +617,8 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		return pci_vfio_map_resource_secondary(dev);
 }
 
-int
-pci_vfio_unmap_resource(struct rte_pci_device *dev)
+static int
+pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
 {
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
@@ -687,6 +690,74 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	return 0;
 }
 
+static int
+pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
+{
+	char pci_addr[PATH_MAX] = {0};
+	struct rte_pci_addr *loc = &dev->addr;
+	int i, ret;
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct mapped_pci_res_list *vfio_res_list;
+
+	struct pci_map *maps;
+
+	/* store PCI address string */
+	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
+			loc->domain, loc->bus, loc->devid, loc->function);
+
+	ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+				  dev->intr_handle.vfio_dev_fd);
+	if (ret < 0) {
+		RTE_LOG(ERR, EAL,
+			"%s(): cannot release device\n", __func__);
+		return ret;
+	}
+
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	/* Get vfio_res */
+	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
+			continue;
+		break;
+	}
+	/* if we haven't found our tailq entry, something's wrong */
+	if (vfio_res == NULL) {
+		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
+				pci_addr);
+		return -1;
+	}
+
+	/* unmap BARs */
+	maps = vfio_res->maps;
+
+	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
+		pci_addr);
+	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+
+		/*
+		 * We do not need to be aware of MSI-X table BAR mappings as
+		 * when mapping. Just using current maps array is enough
+		 */
+		if (maps[i].addr) {
+			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
+				pci_addr, maps[i].addr);
+			pci_unmap_resource(maps[i].addr, maps[i].size);
+		}
+	}
+
+	return 0;
+}
+
+int
+pci_vfio_unmap_resource(struct rte_pci_device *dev)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return pci_vfio_unmap_resource_primary(dev);
+	else
+		return pci_vfio_unmap_resource_secondary(dev);
+}
+
 int
 pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
 		    struct rte_pci_ioport *p)
-- 
2.13.6

[dpdk-dev] [PATCH v10 04/19] vfio: remove uneccessary IPC for group fd clear

[Qi Zhang]

Clear vfio_group_fd is not necessary to involve any IPC.
Also, current IPC implementation for SOCKET_CLR_GROUP is not
correct. rte_vfio_clear_group on secondary will always fail,
that prevent device be detached correctly on a secondary process.
The patch simply removes all IPC related stuff in
rte_vfio_clear_group.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_vfio.c         | 45 +++++---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |  1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |  8 -----
 3 files changed, 8 insertions(+), 46 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index a2bbdfbf4..c0eccddc3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -575,10 +575,6 @@ int
 rte_vfio_clear_group(int vfio_group_fd)
 {
 	int i;
-	struct rte_mp_msg mp_req, *mp_rep;
-	struct rte_mp_reply mp_reply;
-	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
-	struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
 	struct vfio_config *vfio_cfg;
 
 	vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
@@ -587,40 +583,15 @@ rte_vfio_clear_group(int vfio_group_fd)
 		return -1;
 	}
 
-	if (internal_config.process_type == RTE_PROC_PRIMARY) {
-
-		i = get_vfio_group_idx(vfio_group_fd);
-		if (i < 0)
-			return -1;
-		vfio_cfg->vfio_groups[i].group_num = -1;
-		vfio_cfg->vfio_groups[i].fd = -1;
-		vfio_cfg->vfio_groups[i].devices = 0;
-		vfio_cfg->vfio_active_groups--;
-		return 0;
-	}
-
-	p->req = SOCKET_CLR_GROUP;
-	p->group_num = vfio_group_fd;
-	strcpy(mp_req.name, EAL_VFIO_MP);
-	mp_req.len_param = sizeof(*p);
-	mp_req.num_fds = 0;
-
-	if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
-	    mp_reply.nb_received == 1) {
-		mp_rep = &mp_reply.msgs[0];
-		p = (struct vfio_mp_param *)mp_rep->param;
-		if (p->result == SOCKET_OK) {
-			free(mp_reply.msgs);
-			return 0;
-		} else if (p->result == SOCKET_NO_FD)
-			RTE_LOG(ERR, EAL, "  BAD VFIO group fd!\n");
-		else
-			RTE_LOG(ERR, EAL, "  no such VFIO group fd!\n");
-
-		free(mp_reply.msgs);
-	}
+	i = get_vfio_group_idx(vfio_group_fd);
+	if (i < 0)
+		return -1;
+	vfio_cfg->vfio_groups[i].group_num = -1;
+	vfio_cfg->vfio_groups[i].fd = -1;
+	vfio_cfg->vfio_groups[i].devices = 0;
+	vfio_cfg->vfio_active_groups--;
 
-	return -1;
+	return 0;
 }
 
 int
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index e65b10374..68d4750a5 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -129,7 +129,6 @@ int vfio_mp_sync_setup(void);
 
 #define SOCKET_REQ_CONTAINER 0x100
 #define SOCKET_REQ_GROUP 0x200
-#define SOCKET_CLR_GROUP 0x300
 #define SOCKET_OK 0x0
 #define SOCKET_NO_FD 0x1
 #define SOCKET_ERR 0xFF
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
index 9c202bb08..680a24aae 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
@@ -55,14 +55,6 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
 			reply.fds[0] = fd;
 		}
 		break;
-	case SOCKET_CLR_GROUP:
-		r->req = SOCKET_CLR_GROUP;
-		r->group_num = m->group_num;
-		if (rte_vfio_clear_group(m->group_num) < 0)
-			r->result = SOCKET_NO_FD;
-		else
-			r->result = SOCKET_OK;
-		break;
 	case SOCKET_REQ_CONTAINER:
 		r->req = SOCKET_REQ_CONTAINER;
 		fd = rte_vfio_get_container_fd();
-- 
2.13.6

[dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 140 +++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 +++++++
 lib/librte_eal/common/hotplug_mp.c      | 181 ++++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  44 ++++++++
 lib/librte_eal/common/include/rte_bus.h |   3 +
 lib/librte_eal/common/include/rte_dev.h |   9 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 ++
 10 files changed, 418 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 3fd33f1e4..4ecc73b42 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 14c5f05fa..fb1a122ae 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -16,8 +16,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -102,8 +104,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -168,8 +171,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -197,11 +199,139 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
 	return ret;
 }
 
 int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+	if (ret)
+		/* step h) */
+		return ret;
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return -ENODEV;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return -ENOTSUP;
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
+	return -ENODEV;
+}
+
+int __rte_experimental
 rte_dev_event_callback_register(const char *device_name,
 				rte_dev_event_cb_fn cb_fn,
 				void *cb_arg)
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..1883b05d1 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,41 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..261d17fe6
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+		handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..f24ba107b
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,44 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[RTE_BUS_NAME_MAX_LEN];
+	char devname[RTE_DEV_NAME_MAX_LEN];
+	char devargs[RTE_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
index eb9eded4e..720f7c3c8 100644
--- a/lib/librte_eal/common/include/rte_bus.h
+++ b/lib/librte_eal/common/include/rte_bus.h
@@ -197,6 +197,9 @@ struct rte_bus_conf {
 typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
 
 
+/* Max length for a bus name */
+#define RTE_BUS_NAME_MAX_LEN 32
+
 /**
  * A structure describing a generic bus.
  */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 3879ff3ca..667df20f0 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -152,6 +152,9 @@ struct rte_driver {
  */
 #define RTE_DEV_NAME_MAX_LEN 64
 
+/* Max devargs length be allowed */
+#define RTE_DEV_ARGS_MAX_LEN 128
+
 /**
  * A structure describing a generic device.
  */
@@ -193,6 +196,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +218,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 8a3dcfee0..0f2cf9a1a 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -25,6 +25,7 @@ common_sources = files(
 	'eal_common_tailqs.c',
 	'eal_common_thread.c',
 	'eal_common_timer.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 3719ec9d7..6c225b54a 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..0c6984990 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -850,6 +850,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v10 06/19] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c |  36 ++++++-
 lib/librte_eal/common/hotplug_mp.c     | 175 ++++++++++++++++++++++++++++++++-
 2 files changed, 202 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index fb1a122ae..195e1fe00 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -218,8 +218,22 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
 	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
 	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
 
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -ENOTSUP;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
 
 	/**
 	 * attach a device from primary start from here:
@@ -279,8 +293,22 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
 	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
 
-	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -ENOTSUP;
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
 
 	/**
 	 * detach a device from primary start from here:
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 261d17fe6..68ca18bbe 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (!tmp_req.result) {
+			ret = do_dev_hotplug_remove(req->busname,
+						req->devname);
+			if (ret) {
+				RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+				goto rollback;
+			}
+		} else {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+	if (ret)
+		RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -114,8 +260,27 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v10 07/19] net/i40e: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v10 08/19] net/ixgbe: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v10 09/19] net/af_packet: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..9b0dabeeb 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -935,6 +934,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v10 10/19] net/bonding: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..91c352c4c 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3053,8 +3053,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3062,6 +3061,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3168,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v10 11/19] net/kni: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..984d0d29a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -419,6 +418,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v10 12/19] net/null: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..4d5989bd1 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -623,6 +622,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +667,24 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v10 13/19] net/octeontx: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..1b98f4086 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_private(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1154,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v10 14/19] net/pcap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..3c27a98df 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -916,8 +916,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -925,6 +924,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1016,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1024,15 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v10 15/19] net/softnic: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..f3b6863ae 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -741,8 +741,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 	/* Parse input arguments */
 	params = rte_vdev_device_args(vdev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -750,6 +749,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +803,22 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(dev);
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v10 16/19] net/tap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..a58d5ebe6 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1750,8 +1750,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -1759,6 +1758,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1827,17 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v10 17/19] net/vhost: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..3a3f8f6cc 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1353,6 +1352,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v10 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++++
 examples/multi_process/hotplug_mp/commands.c | 197 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 ++++++
 5 files changed, 272 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..fb7198d51
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v10 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..1251e4b5b 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,12 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -60,6 +66,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v10 03/19] bus/pci: enable vfio unmap resource for secondary

[Burakov, Anatoly]

On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> Subroutine to unmap VFIO resource is shared by secondary and
> primary, and it does not work on the secondary process.
> The patch adds a dedicate function to handle the situation
> when a device is unmapped on a secondary process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Hi Qi,

Please correct me if i'm wrong here, but it seems like the unmapping 
code is shared between primary and secondary, and the difference comes 
from interrupts, bus mastering, and removing the device from tailq. Can 
we separate out the common code somehow?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v10 04/19] vfio: remove uneccessary IPC for group fd clear

[Burakov, Anatoly]

On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> Clear vfio_group_fd is not necessary to involve any IPC.
> Also, current IPC implementation for SOCKET_CLR_GROUP is not
> correct. rte_vfio_clear_group on secondary will always fail,
> that prevent device be detached correctly on a secondary process.
> The patch simply removes all IPC related stuff in
> rte_vfio_clear_group.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v10 03/19] bus/pci: enable vfio unmap resource for secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Monday, July 9, 2018 10:38 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v10 03/19] bus/pci: enable vfio unmap resource for
> secondary
> 
> On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> > Subroutine to unmap VFIO resource is shared by secondary and primary,
> > and it does not work on the secondary process.
> > The patch adds a dedicate function to handle the situation when a
> > device is unmapped on a secondary process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> Hi Qi,
> 
> Please correct me if i'm wrong here, but it seems like the unmapping code is
> shared between primary and secondary, and the difference comes from
> interrupts, bus mastering, and removing the device from tailq. Can we
> separate out the common code somehow?

Yes, you are right and I will separate out the common code.

Thanks
Qi
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on multi-process

[Burakov, Anatoly]

On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle hotplug in
> multi-process, it includes the below scenario:
> 
> 1. Attach a device from the primary
> 2. Detach a device from the primary
> 3. Attach a device from a secondary
> 4. Detach a device from a secondary
> 
> In the primary-secondary process model, we assume devices are shared
> by default. that means attaches or detaches a device on any process
> will broadcast to all other processes through mp channel then device
> information will be synchronized on all processes.
> 
> Any failure during attaching/detaching process will cause inconsistent
> status between processes, so proper rollback action should be considered.
> 
> This patch covers the implementation of case 1,2.
> Case 3,4 will be implemented on a separate patch.
> 
> IPC scenario for Case 1, 2:
> 
> attach a device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach the device and send a reply.
> d) primary check the reply if all success goes to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach the device and send a reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach a device
> a) primary send detach sync request to all secondary
> b) secondary detach the device and send reply
> c) primary check the reply if all success goes to f).
> d) primary send detach rollback sync request to all secondary.
> e) secondary receive the request and attach back device. goto g)
> f) primary detach the device if success goto g), else goto d)
> g) detach fail.
> h) detach success.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +	req.t = EAL_DEV_REQ_TYPE_ATTACH;
> +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> +	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return -ENOTSUP;

Nitpick, but maybe do this before strlcpy?

> +
> +	/**
> +	 * attach a device from primary start from here:
> +	 *
> +	 * a) primary attach the new device if failed goto h).
> +	 * b) primary send attach sync request to all secondary.
> +	 * c) secondary receive request and attach the device and send a reply.
> +	 * d) primary check the reply if all success goes to i).

<snip>

> +
> +	memset(&req, 0, sizeof(req));
> +	req.t = EAL_DEV_REQ_TYPE_DETACH;
> +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return -ENOTSUP;

Same nitpick, probably move this above.

> +
> +	/**
> +	 * detach a device from primary start from here:
> +	 *
> +	 * a) primary send detach sync request to all secondary
> +	 * b) secondary detach the device and send reply

<snip>

> +	struct mp_reply_bundle *bundle = param;
> +	struct rte_mp_msg *msg = &bundle->msg;
> +	const struct eal_dev_mp_req *req =
> +		(const struct eal_dev_mp_req *)msg->param;
> +	struct rte_mp_msg mp_resp;
> +	struct eal_dev_mp_req *resp =
> +		(struct eal_dev_mp_req *)mp_resp.param;
> +	int ret = 0;
> +
> +	memset(&mp_resp, 0, sizeof(mp_resp));
> +
> +	switch (req->t) {
> +	case EAL_DEV_REQ_TYPE_ATTACH:
> +	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
> +		ret = do_dev_hotplug_add(req->busname, req->devname, "");

I'm not too familiar with devargs and hotplug, but why are we passing 
empty devargs string here? Is it possible for it to be not empty?

> +		break;
> +	case EAL_DEV_REQ_TYPE_DETACH:
> +	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
> +		ret = do_dev_hotplug_remove(req->busname, req->devname);
> +		break;
> +	default:
> +		ret = -EINVAL;
> +	}

<snip>

> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> +					handle_secondary_request);
> +		if (ret) {
> +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> +				EAL_DEV_MP_ACTION_REQUEST);
> +			return ret;
> +		}
> +	} else {
> +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> +		handle_primary_request);

^^ wrong indentation.

> +		if (ret) {
> +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> +				EAL_DEV_MP_ACTION_REQUEST);
> +			return ret;
> +		}
> +	}

<snip>

> +
> +#endif /* _HOTPLUG_MP_H_ */
> diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h
> index eb9eded4e..720f7c3c8 100644
> --- a/lib/librte_eal/common/include/rte_bus.h
> +++ b/lib/librte_eal/common/include/rte_bus.h
> @@ -197,6 +197,9 @@ struct rte_bus_conf {
>   typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
>   
>   
> +/* Max length for a bus name */
> +#define RTE_BUS_NAME_MAX_LEN 32

Is this enforced anywhere in the bus codebase? Can we guarantee that bus 
name will never be bigger than this?

> +
>   /**
>    * A structure describing a generic bus.
>    */
> diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
> index 3879ff3ca..667df20f0 100644
> --- a/lib/librte_eal/common/include/rte_dev.h
> +++ b/lib/librte_eal/common/include/rte_dev.h
> @@ -152,6 +152,9 @@ struct rte_driver {
>    */
>   #define RTE_DEV_NAME_MAX_LEN 64
>   
> +/* Max devargs length be allowed */
> +#define RTE_DEV_ARGS_MAX_LEN 128

Same - is this enforced anywhere in the codebase related to devargs?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v10 06/19] eal: support attach or detach share device from secondary

[Burakov, Anatoly]

On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a device
> attach/detach request be issued from a secondary process
> 
> device attach on secondary:
> a) secondary send sync request to the primary.
> b) primary receive the request and attach the new device if
>     failed goto i).
> c) primary forward attach sync request to all secondary.
> d) secondary receive the request and attach the device and send a reply.
> e) primary check the reply if all success goes to j).
> f) primary send attach rollback sync request to all secondary.
> g) secondary receive the request and detach the device and send a reply.
> h) primary receive the reply and detach device as rollback action.
> i) send attach fail to secondary as a reply of step a), goto k).
> j) send attach success to secondary as a reply of step a).
> k) secondary receive reply and return.
> 
> device detach on secondary:
> a) secondary send sync request to the primary.
> b) primary send detach sync request to all secondary.
> c) secondary detach the device and send a reply.
> d) primary check the reply if all success goes to g).
> e) primary send detach rollback sync request to all secondary.
> f) secondary receive the request and attach back device. goto h).
> g) primary detach the device if success goto i), else goto e).
> h) primary send detach fail to secondary as a reply of step a), goto j).
> i) primary send detach success to secondary as a reply of step a).
> j) secondary receive reply and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

<snip>

> +
> +	memset(&mp_req, 0, sizeof(mp_req));
> +	memcpy(mp_req.param, req, sizeof(*req));
> +	mp_req.len_param = sizeof(*req);
> +	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
> +
> +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> +	if (ret) {
> +		RTE_LOG(ERR, EAL, "cannot send request to primary");
> +		return ret;
> +	}
> +
> +	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;

This looks like a potential buffer overflow - you don't check if there's 
a first message to read a response from.

> +	req->result = resp->result;
> +
> +	return ret;
>   }
>   
>   int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, July 10, 2018 10:01 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v10 05/19] eal: enable hotplug on multi-process
> 
> On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> > We are going to introduce the solution to handle hotplug in
> > multi-process, it includes the below scenario:
> >
> > 1. Attach a device from the primary
> > 2. Detach a device from the primary
> > 3. Attach a device from a secondary
> > 4. Detach a device from a secondary
> >
> > In the primary-secondary process model, we assume devices are shared
> > by default. that means attaches or detaches a device on any process
> > will broadcast to all other processes through mp channel then device
> > information will be synchronized on all processes.
> >
> > Any failure during attaching/detaching process will cause inconsistent
> > status between processes, so proper rollback action should be considered.
> >
> > This patch covers the implementation of case 1,2.
> > Case 3,4 will be implemented on a separate patch.
> >
> > IPC scenario for Case 1, 2:
> >
> > attach a device
> > a) primary attach the new device if failed goto h).
> > b) primary send attach sync request to all secondary.
> > c) secondary receive request and attach the device and send a reply.
> > d) primary check the reply if all success goes to i).
> > e) primary send attach rollback sync request to all secondary.
> > f) secondary receive the request and detach the device and send a reply.
> > g) primary receive the reply and detach device as rollback action.
> > h) attach fail
> > i) attach success
> >
> > detach a device
> > a) primary send detach sync request to all secondary
> > b) secondary detach the device and send reply
> > c) primary check the reply if all success goes to f).
> > d) primary send detach rollback sync request to all secondary.
> > e) secondary receive the request and attach back device. goto g)
> > f) primary detach the device if success goto g), else goto d)
> > g) detach fail.
> > h) detach success.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +	req.t = EAL_DEV_REQ_TYPE_ATTACH;
> > +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> > +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> > +	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
> > +
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return -ENOTSUP;
> 
> Nitpick, but maybe do this before strlcpy?

On the next patch, these strlcpy can be reused when implemented secondary process case

> 
> > +
> > +	/**
> > +	 * attach a device from primary start from here:
> > +	 *
> > +	 * a) primary attach the new device if failed goto h).
> > +	 * b) primary send attach sync request to all secondary.
> > +	 * c) secondary receive request and attach the device and send a reply.
> > +	 * d) primary check the reply if all success goes to i).
> 
> <snip>
> 
> > +
> > +	memset(&req, 0, sizeof(req));
> > +	req.t = EAL_DEV_REQ_TYPE_DETACH;
> > +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> > +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> > +
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return -ENOTSUP;
> 
> Same nitpick, probably move this above.
> 
> > +
> > +	/**
> > +	 * detach a device from primary start from here:
> > +	 *
> > +	 * a) primary send detach sync request to all secondary
> > +	 * b) secondary detach the device and send reply
> 
> <snip>
> 
> > +	struct mp_reply_bundle *bundle = param;
> > +	struct rte_mp_msg *msg = &bundle->msg;
> > +	const struct eal_dev_mp_req *req =
> > +		(const struct eal_dev_mp_req *)msg->param;
> > +	struct rte_mp_msg mp_resp;
> > +	struct eal_dev_mp_req *resp =
> > +		(struct eal_dev_mp_req *)mp_resp.param;
> > +	int ret = 0;
> > +
> > +	memset(&mp_resp, 0, sizeof(mp_resp));
> > +
> > +	switch (req->t) {
> > +	case EAL_DEV_REQ_TYPE_ATTACH:
> > +	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
> > +		ret = do_dev_hotplug_add(req->busname, req->devname, "");
> 
> I'm not too familiar with devargs and hotplug, but why are we passing empty
> devargs string here? Is it possible for it to be not empty?

For secondary process, devargs is ignored, so we just need the device unique identity <busname, devname>

> 
> > +		break;
> > +	case EAL_DEV_REQ_TYPE_DETACH:
> > +	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
> > +		ret = do_dev_hotplug_remove(req->busname, req->devname);
> > +		break;
> > +	default:
> > +		ret = -EINVAL;
> > +	}
> 
> <snip>
> 
> > +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> > +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> > +					handle_secondary_request);
> > +		if (ret) {
> > +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> > +				EAL_DEV_MP_ACTION_REQUEST);
> > +			return ret;
> > +		}
> > +	} else {
> > +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> > +		handle_primary_request);
> 
> ^^ wrong indentation.

Will fix.
> 
> > +		if (ret) {
> > +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> > +				EAL_DEV_MP_ACTION_REQUEST);
> > +			return ret;
> > +		}
> > +	}
> 
> <snip>
> 
> > +
> > +#endif /* _HOTPLUG_MP_H_ */
> > diff --git a/lib/librte_eal/common/include/rte_bus.h
> > b/lib/librte_eal/common/include/rte_bus.h
> > index eb9eded4e..720f7c3c8 100644
> > --- a/lib/librte_eal/common/include/rte_bus.h
> > +++ b/lib/librte_eal/common/include/rte_bus.h
> > @@ -197,6 +197,9 @@ struct rte_bus_conf {
> >   typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
> >
> >
> > +/* Max length for a bus name */
> > +#define RTE_BUS_NAME_MAX_LEN 32
> 
> Is this enforced anywhere in the bus codebase? Can we guarantee that bus
> name will never be bigger than this?

I think 32 should be enough for a bus name even in future.

> 
> > +
> >   /**
> >    * A structure describing a generic bus.
> >    */
> > diff --git a/lib/librte_eal/common/include/rte_dev.h
> > b/lib/librte_eal/common/include/rte_dev.h
> > index 3879ff3ca..667df20f0 100644
> > --- a/lib/librte_eal/common/include/rte_dev.h
> > +++ b/lib/librte_eal/common/include/rte_dev.h
> > @@ -152,6 +152,9 @@ struct rte_driver {
> >    */
> >   #define RTE_DEV_NAME_MAX_LEN 64
> >
> > +/* Max devargs length be allowed */
> > +#define RTE_DEV_ARGS_MAX_LEN 128
> 
> Same - is this enforced anywhere in the codebase related to devargs?

I'm not sure, but I guess it is big enough for all exist driver :)

Thanks
Qi

> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zhang, Qi Z
> Sent: Wednesday, July 11, 2018 9:26 AM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on
> multi-process
> 
> 
> 
> > -----Original Message-----
> > From: Burakov, Anatoly
> > Sent: Tuesday, July 10, 2018 10:01 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> > Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> > Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> > <ferruh.yigit@intel.com>; Shelton, Benjamin H
> > <benjamin.h.shelton@intel.com>; Vangati, Narender
> > <narender.vangati@intel.com>
> > Subject: Re: [PATCH v10 05/19] eal: enable hotplug on multi-process
> >
> > On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> > > We are going to introduce the solution to handle hotplug in
> > > multi-process, it includes the below scenario:
> > >
> > > 1. Attach a device from the primary
> > > 2. Detach a device from the primary
> > > 3. Attach a device from a secondary
> > > 4. Detach a device from a secondary
> > >
> > > In the primary-secondary process model, we assume devices are shared
> > > by default. that means attaches or detaches a device on any process
> > > will broadcast to all other processes through mp channel then device
> > > information will be synchronized on all processes.
> > >
> > > Any failure during attaching/detaching process will cause
> > > inconsistent status between processes, so proper rollback action should be
> considered.
> > >
> > > This patch covers the implementation of case 1,2.
> > > Case 3,4 will be implemented on a separate patch.
> > >
> > > IPC scenario for Case 1, 2:
> > >
> > > attach a device
> > > a) primary attach the new device if failed goto h).
> > > b) primary send attach sync request to all secondary.
> > > c) secondary receive request and attach the device and send a reply.
> > > d) primary check the reply if all success goes to i).
> > > e) primary send attach rollback sync request to all secondary.
> > > f) secondary receive the request and detach the device and send a reply.
> > > g) primary receive the reply and detach device as rollback action.
> > > h) attach fail
> > > i) attach success
> > >
> > > detach a device
> > > a) primary send detach sync request to all secondary
> > > b) secondary detach the device and send reply
> > > c) primary check the reply if all success goes to f).
> > > d) primary send detach rollback sync request to all secondary.
> > > e) secondary receive the request and attach back device. goto g)
> > > f) primary detach the device if success goto g), else goto d)
> > > g) detach fail.
> > > h) detach success.
> > >
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> >
> > <snip>
> >
> > > +	req.t = EAL_DEV_REQ_TYPE_ATTACH;
> > > +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> > > +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> > > +	strlcpy(req.devargs, devargs, RTE_DEV_ARGS_MAX_LEN);
> > > +
> > > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > > +		return -ENOTSUP;
> >
> > Nitpick, but maybe do this before strlcpy?
> 
> On the next patch, these strlcpy can be reused when implemented secondary
> process case
> 
> >
> > > +
> > > +	/**
> > > +	 * attach a device from primary start from here:
> > > +	 *
> > > +	 * a) primary attach the new device if failed goto h).
> > > +	 * b) primary send attach sync request to all secondary.
> > > +	 * c) secondary receive request and attach the device and send a reply.
> > > +	 * d) primary check the reply if all success goes to i).
> >
> > <snip>
> >
> > > +
> > > +	memset(&req, 0, sizeof(req));
> > > +	req.t = EAL_DEV_REQ_TYPE_DETACH;
> > > +	strlcpy(req.busname, busname, RTE_BUS_NAME_MAX_LEN);
> > > +	strlcpy(req.devname, devname, RTE_DEV_NAME_MAX_LEN);
> > > +
> > > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > > +		return -ENOTSUP;
> >
> > Same nitpick, probably move this above.
> >
> > > +
> > > +	/**
> > > +	 * detach a device from primary start from here:
> > > +	 *
> > > +	 * a) primary send detach sync request to all secondary
> > > +	 * b) secondary detach the device and send reply
> >
> > <snip>
> >
> > > +	struct mp_reply_bundle *bundle = param;
> > > +	struct rte_mp_msg *msg = &bundle->msg;
> > > +	const struct eal_dev_mp_req *req =
> > > +		(const struct eal_dev_mp_req *)msg->param;
> > > +	struct rte_mp_msg mp_resp;
> > > +	struct eal_dev_mp_req *resp =
> > > +		(struct eal_dev_mp_req *)mp_resp.param;
> > > +	int ret = 0;
> > > +
> > > +	memset(&mp_resp, 0, sizeof(mp_resp));
> > > +
> > > +	switch (req->t) {
> > > +	case EAL_DEV_REQ_TYPE_ATTACH:
> > > +	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
> > > +		ret = do_dev_hotplug_add(req->busname, req->devname, "");
> >
> > I'm not too familiar with devargs and hotplug, but why are we passing
> > empty devargs string here? Is it possible for it to be not empty?
> 
> For secondary process, devargs is ignored, so we just need the device unique
> identity <busname, devname>
> 
> >
> > > +		break;
> > > +	case EAL_DEV_REQ_TYPE_DETACH:
> > > +	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
> > > +		ret = do_dev_hotplug_remove(req->busname, req->devname);
> > > +		break;
> > > +	default:
> > > +		ret = -EINVAL;
> > > +	}
> >
> > <snip>
> >
> > > +	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> > > +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> > > +					handle_secondary_request);
> > > +		if (ret) {
> > > +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> > > +				EAL_DEV_MP_ACTION_REQUEST);
> > > +			return ret;
> > > +		}
> > > +	} else {
> > > +		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
> > > +		handle_primary_request);
> >
> > ^^ wrong indentation.
> 
> Will fix.
> >
> > > +		if (ret) {
> > > +			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
> > > +				EAL_DEV_MP_ACTION_REQUEST);
> > > +			return ret;
> > > +		}
> > > +	}
> >
> > <snip>
> >
> > > +
> > > +#endif /* _HOTPLUG_MP_H_ */
> > > diff --git a/lib/librte_eal/common/include/rte_bus.h
> > > b/lib/librte_eal/common/include/rte_bus.h
> > > index eb9eded4e..720f7c3c8 100644
> > > --- a/lib/librte_eal/common/include/rte_bus.h
> > > +++ b/lib/librte_eal/common/include/rte_bus.h
> > > @@ -197,6 +197,9 @@ struct rte_bus_conf {
> > >   typedef enum rte_iova_mode (*rte_bus_get_iommu_class_t)(void);
> > >
> > >
> > > +/* Max length for a bus name */
> > > +#define RTE_BUS_NAME_MAX_LEN 32
> >
> > Is this enforced anywhere in the bus codebase? Can we guarantee that
> > bus name will never be bigger than this?
> 
> I think 32 should be enough for a bus name even in future.

Sorry, I missed your point, I think it is not enforced, we still can add a new bus exceed 32, 
but for RTE_DEV_NAME_MAX_LEN which is used in rte_devargs to enforce all device name not exceed 64.
So, it's better to move RTE_BUS_NAME_MAX_LEN into hotplug_mp as internal , and this can be regarded as a limitation for hotplug so far, though it should be enough for all exist cases.
And same for RTE_DEV_ARGS_MAX_LEN.

> 
> >
> > > +
> > >   /**
> > >    * A structure describing a generic bus.
> > >    */
> > > diff --git a/lib/librte_eal/common/include/rte_dev.h
> > > b/lib/librte_eal/common/include/rte_dev.h
> > > index 3879ff3ca..667df20f0 100644
> > > --- a/lib/librte_eal/common/include/rte_dev.h
> > > +++ b/lib/librte_eal/common/include/rte_dev.h
> > > @@ -152,6 +152,9 @@ struct rte_driver {
> > >    */
> > >   #define RTE_DEV_NAME_MAX_LEN 64
> > >
> > > +/* Max devargs length be allowed */ #define RTE_DEV_ARGS_MAX_LEN
> > > +128
> >
> > Same - is this enforced anywhere in the codebase related to devargs?
> 
> I'm not sure, but I guess it is big enough for all exist driver :)
> 
> Thanks
> Qi
> 
> >
> > --
> > Thanks,
> > Anatoly

Re: [dpdk-dev] [PATCH v10 06/19] eal: support attach or detach share device from secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Tuesday, July 10, 2018 10:12 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v10 06/19] eal: support attach or detach share device from
> secondary
> 
> On 09-Jul-18 4:36 AM, Qi Zhang wrote:
> > This patch cover the multi-process hotplug case when a device
> > attach/detach request be issued from a secondary process
> >
> > device attach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary receive the request and attach the new device if
> >     failed goto i).
> > c) primary forward attach sync request to all secondary.
> > d) secondary receive the request and attach the device and send a reply.
> > e) primary check the reply if all success goes to j).
> > f) primary send attach rollback sync request to all secondary.
> > g) secondary receive the request and detach the device and send a reply.
> > h) primary receive the reply and detach device as rollback action.
> > i) send attach fail to secondary as a reply of step a), goto k).
> > j) send attach success to secondary as a reply of step a).
> > k) secondary receive reply and return.
> >
> > device detach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary send detach sync request to all secondary.
> > c) secondary detach the device and send a reply.
> > d) primary check the reply if all success goes to g).
> > e) primary send detach rollback sync request to all secondary.
> > f) secondary receive the request and attach back device. goto h).
> > g) primary detach the device if success goto i), else goto e).
> > h) primary send detach fail to secondary as a reply of step a), goto j).
> > i) primary send detach success to secondary as a reply of step a).
> > j) secondary receive reply and return.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> 
> <snip>
> 
> > +
> > +	memset(&mp_req, 0, sizeof(mp_req));
> > +	memcpy(mp_req.param, req, sizeof(*req));
> > +	mp_req.len_param = sizeof(*req);
> > +	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST,
> > +sizeof(mp_req.name));
> > +
> > +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> > +	if (ret) {
> > +		RTE_LOG(ERR, EAL, "cannot send request to primary");
> > +		return ret;
> > +	}
> > +
> > +	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;

Ok, I will add nb_receivedd check.

> 
> This looks like a potential buffer overflow - you don't check if there's a first
> message to read a response from.
> 
> > +	req->result = resp->result;
> > +
> > +	return ret;
> >   }
> >
> >   int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
> >
> 
> 
> --
> Thanks,
> Anatoly

[dpdk-dev] [PATCH v11 00/19] enable hotplug on multi-process

[Qi Zhang]

v11:
- move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               | 129 +++++++--
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 197 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 184 ++++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 346 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  48 ++++
 lib/librte_eal/common/include/rte_dev.h        |   6 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 33 files changed, 1122 insertions(+), 103 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..269586d88 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..a46d9e182 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	/**
+	 * PCI device can only be globally detached directly by a
+	 * primary process. In secondary process, we only need to
+	 * release port.
+	 */
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v11 02/19] bus/pci: fix PCI address compare

[Qi Zhang]

When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
is valid. So compare the 8th byte will cause the unexpected result, which
happens when repeatedly attach/detach a device.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..dd25c3542 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_vfio_tailq)
 
+/* Compair two pci address */
+static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
+{
+	if (addr1->domain == addr2->domain &&
+		addr1->bus == addr2->bus &&
+		addr1->devid == addr2->devid &&
+		addr1->function == addr2->function)
+		return 0;
+	return 1;
+}
+
 int
 pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
 		    void *buf, size_t len, off_t offs)
@@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 	/* Get vfio_res */
 	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
 			continue;
 		break;
 	}
-- 
2.13.6

[dpdk-dev] [PATCH v11 03/19] bus/pci: enable vfio unmap resource for secondary

[Qi Zhang]

Subroutine to unmap VFIO resource is shared by secondary and
primary, and it does not work on the secondary process. Since
for secondary process, it is not necessary to close interrupt
handler, set pci bus mastering and remove vfio_res from
vfio_res_list. So, the patch adds a dedicate function to handle
the situation when a device is unmapped on a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 118 +++++++++++++++++++++++++++++----------
 1 file changed, 90 insertions(+), 28 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index dd25c3542..93f0ac07e 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -595,6 +595,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 		dev->mem_resource[i].addr = maps[i].addr;
 	}
 
+	/* we need save vfio_dev_fd, so it can be used during release */
+	dev->intr_handle.vfio_dev_fd = vfio_dev_fd;
+
 	return 0;
 err_vfio_dev_fd:
 	close(vfio_dev_fd);
@@ -614,22 +617,58 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		return pci_vfio_map_resource_secondary(dev);
 }
 
-int
-pci_vfio_unmap_resource(struct rte_pci_device *dev)
+static struct mapped_pci_resource *
+find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,
+			struct rte_pci_device *dev,
+			const char *pci_addr)
+{
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct pci_map *maps;
+	int i;
+
+	/* Get vfio_res */
+	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
+			continue;
+		break;
+	}
+
+	if  (vfio_res == NULL)
+		return vfio_res;
+
+	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
+		pci_addr);
+
+	maps = vfio_res->maps;
+	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+
+		/*
+		 * We do not need to be aware of MSI-X table BAR mappings as
+		 * when mapping. Just using current maps array is enough
+		 */
+		if (maps[i].addr) {
+			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
+				pci_addr, maps[i].addr);
+			pci_unmap_resource(maps[i].addr, maps[i].size);
+		}
+	}
+
+	return vfio_res;
+}
+
+static int
+pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
 {
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
-	int i, ret;
 	struct mapped_pci_resource *vfio_res = NULL;
 	struct mapped_pci_res_list *vfio_res_list;
-
-	struct pci_map *maps;
+	int ret;
 
 	/* store PCI address string */
 	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
 			loc->domain, loc->bus, loc->devid, loc->function);
 
-
 	if (close(dev->intr_handle.fd) < 0) {
 		RTE_LOG(INFO, EAL, "Error when closing eventfd file descriptor for %s\n",
 			pci_addr);
@@ -650,13 +689,10 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return ret;
 	}
 
-	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
-	/* Get vfio_res */
-	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
-			continue;
-		break;
-	}
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
 	/* if we haven't found our tailq entry, something's wrong */
 	if (vfio_res == NULL) {
 		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
@@ -664,30 +700,56 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return -1;
 	}
 
-	/* unmap BARs */
-	maps = vfio_res->maps;
+	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
 
-	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
-		pci_addr);
-	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+	return 0;
+}
 
-		/*
-		 * We do not need to be aware of MSI-X table BAR mappings as
-		 * when mapping. Just using current maps array is enough
-		 */
-		if (maps[i].addr) {
-			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
-				pci_addr, maps[i].addr);
-			pci_unmap_resource(maps[i].addr, maps[i].size);
-		}
+static int
+pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
+{
+	char pci_addr[PATH_MAX] = {0};
+	struct rte_pci_addr *loc = &dev->addr;
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct mapped_pci_res_list *vfio_res_list;
+	int ret;
+
+	/* store PCI address string */
+	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
+			loc->domain, loc->bus, loc->devid, loc->function);
+
+	ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+				  dev->intr_handle.vfio_dev_fd);
+	if (ret < 0) {
+		RTE_LOG(ERR, EAL,
+			"%s(): cannot release device\n", __func__);
+		return ret;
 	}
 
-	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
+	/* if we haven't found our tailq entry, something's wrong */
+	if (vfio_res == NULL) {
+		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
+				pci_addr);
+		return -1;
+	}
 
 	return 0;
 }
 
 int
+pci_vfio_unmap_resource(struct rte_pci_device *dev)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return pci_vfio_unmap_resource_primary(dev);
+	else
+		return pci_vfio_unmap_resource_secondary(dev);
+}
+
+int
 pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
 		    struct rte_pci_ioport *p)
 {
-- 
2.13.6

[dpdk-dev] [PATCH v11 04/19] vfio: remove uneccessary IPC for group fd clear

[Qi Zhang]

Clear vfio_group_fd is not necessary to involve any IPC.
Also, current IPC implementation for SOCKET_CLR_GROUP is not
correct. rte_vfio_clear_group on secondary will always fail,
that prevent device be detached correctly on a secondary process.
The patch simply removes all IPC related stuff in
rte_vfio_clear_group.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_vfio.c         | 45 +++++---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |  1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |  8 -----
 3 files changed, 8 insertions(+), 46 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index a2bbdfbf4..c0eccddc3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -575,10 +575,6 @@ int
 rte_vfio_clear_group(int vfio_group_fd)
 {
 	int i;
-	struct rte_mp_msg mp_req, *mp_rep;
-	struct rte_mp_reply mp_reply;
-	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
-	struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
 	struct vfio_config *vfio_cfg;
 
 	vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
@@ -587,40 +583,15 @@ rte_vfio_clear_group(int vfio_group_fd)
 		return -1;
 	}
 
-	if (internal_config.process_type == RTE_PROC_PRIMARY) {
-
-		i = get_vfio_group_idx(vfio_group_fd);
-		if (i < 0)
-			return -1;
-		vfio_cfg->vfio_groups[i].group_num = -1;
-		vfio_cfg->vfio_groups[i].fd = -1;
-		vfio_cfg->vfio_groups[i].devices = 0;
-		vfio_cfg->vfio_active_groups--;
-		return 0;
-	}
-
-	p->req = SOCKET_CLR_GROUP;
-	p->group_num = vfio_group_fd;
-	strcpy(mp_req.name, EAL_VFIO_MP);
-	mp_req.len_param = sizeof(*p);
-	mp_req.num_fds = 0;
-
-	if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
-	    mp_reply.nb_received == 1) {
-		mp_rep = &mp_reply.msgs[0];
-		p = (struct vfio_mp_param *)mp_rep->param;
-		if (p->result == SOCKET_OK) {
-			free(mp_reply.msgs);
-			return 0;
-		} else if (p->result == SOCKET_NO_FD)
-			RTE_LOG(ERR, EAL, "  BAD VFIO group fd!\n");
-		else
-			RTE_LOG(ERR, EAL, "  no such VFIO group fd!\n");
-
-		free(mp_reply.msgs);
-	}
+	i = get_vfio_group_idx(vfio_group_fd);
+	if (i < 0)
+		return -1;
+	vfio_cfg->vfio_groups[i].group_num = -1;
+	vfio_cfg->vfio_groups[i].fd = -1;
+	vfio_cfg->vfio_groups[i].devices = 0;
+	vfio_cfg->vfio_active_groups--;
 
-	return -1;
+	return 0;
 }
 
 int
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index e65b10374..68d4750a5 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -129,7 +129,6 @@ int vfio_mp_sync_setup(void);
 
 #define SOCKET_REQ_CONTAINER 0x100
 #define SOCKET_REQ_GROUP 0x200
-#define SOCKET_CLR_GROUP 0x300
 #define SOCKET_OK 0x0
 #define SOCKET_NO_FD 0x1
 #define SOCKET_ERR 0xFF
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
index 9c202bb08..680a24aae 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
@@ -55,14 +55,6 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
 			reply.fds[0] = fd;
 		}
 		break;
-	case SOCKET_CLR_GROUP:
-		r->req = SOCKET_CLR_GROUP;
-		r->group_num = m->group_num;
-		if (rte_vfio_clear_group(m->group_num) < 0)
-			r->result = SOCKET_NO_FD;
-		else
-			r->result = SOCKET_OK;
-		break;
 	case SOCKET_REQ_CONTAINER:
 		r->req = SOCKET_REQ_CONTAINER;
 		fd = rte_vfio_get_container_fd();
-- 
2.13.6

[dpdk-dev] [PATCH v11 05/19] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 166 ++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 +++++++
 lib/librte_eal/common/hotplug_mp.c      | 181 ++++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  48 +++++++++
 lib/librte_eal/common/include/rte_dev.h |   6 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 ++
 9 files changed, 442 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 3fd33f1e4..4ecc73b42 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 14c5f05fa..ab2b25558 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -16,8 +16,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -102,8 +104,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -168,8 +171,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -197,11 +199,165 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
 	return ret;
 }
 
 int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+	if (ret)
+		/* step h) */
+		return ret;
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		goto rollback;
+	}
+
+	if (req.result)
+		goto rollback;
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return -ENODEV;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary  process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return ret;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
+	return -ENODEV;
+}
+
+int __rte_experimental
 rte_dev_event_callback_register(const char *device_name,
 				rte_dev_event_cb_fn cb_fn,
 				void *cb_arg)
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..1883b05d1 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,41 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..d3a8a5576
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..22efa073e
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[EAL_DEV_MP_BUS_NAME_MAX_LEN];
+	char devname[EAL_DEV_MP_DEV_NAME_MAX_LEN];
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 3879ff3ca..ab179c778 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -193,6 +193,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +215,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 8a3dcfee0..0f2cf9a1a 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -25,6 +25,7 @@ common_sources = files(
 	'eal_common_tailqs.c',
 	'eal_common_thread.c',
 	'eal_common_timer.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 3719ec9d7..6c225b54a 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..0c6984990 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -850,6 +850,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v11 06/19] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/eal_common_dev.c |  20 +++-
 lib/librte_eal/common/hotplug_mp.c     | 175 ++++++++++++++++++++++++++++++++-
 2 files changed, 189 insertions(+), 6 deletions(-)

diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index ab2b25558..ee09720c4 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		/**
 		 * If in secondary process, just send IPC request to
-		 * primary process
+		 * primary process.
 		 */
 		ret = eal_dev_hotplug_request_to_primary(&req);
 		if (ret) {
@@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
 				"Failed to hotplug add device\n");
 		return req.result;
 	}
+
 	/**
 	 * attach a device from primary start from here:
 	 *
@@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
 		/**
 		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return ret;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
 		 * primary  process.
 		 */
 		ret = eal_dev_hotplug_request_to_primary(&req);
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index d3a8a5576..321050884 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (!tmp_req.result) {
+			ret = do_dev_hotplug_remove(req->busname,
+						req->devname);
+			if (ret) {
+				RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+				goto rollback;
+			}
+		} else {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+	if (ret)
+		RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -114,8 +260,27 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v11 07/19] net/i40e: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v11 08/19] net/ixgbe: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v11 09/19] net/af_packet: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..9b0dabeeb 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -935,6 +934,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v11 10/19] net/bonding: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..91c352c4c 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3053,8 +3053,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3062,6 +3061,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3168,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v11 11/19] net/kni: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..984d0d29a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -419,6 +418,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v11 12/19] net/null: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..4d5989bd1 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -623,6 +622,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +667,24 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v11 13/19] net/octeontx: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..1b98f4086 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_private(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1154,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v11 14/19] net/pcap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..3c27a98df 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -916,8 +916,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -925,6 +924,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1016,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1024,15 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v11 15/19] net/softnic: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..f3b6863ae 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -741,8 +741,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 	/* Parse input arguments */
 	params = rte_vdev_device_args(vdev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -750,6 +749,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +803,22 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(dev);
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v11 16/19] net/tap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..a58d5ebe6 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1750,8 +1750,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -1759,6 +1758,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1827,17 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v11 17/19] net/vhost: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..3a3f8f6cc 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1353,6 +1352,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v11 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++++
 examples/multi_process/hotplug_mp/commands.c | 197 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 ++++++
 5 files changed, 272 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..fb7198d51
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v11 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..1251e4b5b 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,12 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -60,6 +66,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v10 05/19] eal: enable hotplug on multi-process

[Burakov, Anatoly]

On 11-Jul-18 3:11 AM, Zhang, Qi Z wrote:
>>>> +/* Max length for a bus name */
>>>> +#define RTE_BUS_NAME_MAX_LEN 32
>>>
>>> Is this enforced anywhere in the bus codebase? Can we guarantee that
>>> bus name will never be bigger than this?
>>
>> I think 32 should be enough for a bus name even in future.
> 
> Sorry, I missed your point, I think it is not enforced, we still can add a new bus exceed 32,
> but for RTE_DEV_NAME_MAX_LEN which is used in rte_devargs to enforce all device name not exceed 64.
> So, it's better to move RTE_BUS_NAME_MAX_LEN into hotplug_mp as internal , and this can be regarded as a limitation for hotplug so far, though it should be enough for all exist cases.
> And same for RTE_DEV_ARGS_MAX_LEN.

Can we fix it in this patchset, or would that involve an ABI break of 
some sort?

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v11 03/19] bus/pci: enable vfio unmap resource for secondary

[Burakov, Anatoly]

On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> Subroutine to unmap VFIO resource is shared by secondary and
> primary, and it does not work on the secondary process. Since
> for secondary process, it is not necessary to close interrupt
> handler, set pci bus mastering and remove vfio_res from
> vfio_res_list. So, the patch adds a dedicate function to handle
> the situation when a device is unmapped on a secondary process.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Andrew Rybchenko]

On 11.07.2018 06:08, Qi Zhang wrote:
> Add driver API rte_eth_release_port_private to support the
> case when an ethdev need to be detached on a secondary process.
> Local state is set to unused and shared data will not be reset
> so the primary process can still use it.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
> Acked-by: Remy Horton <remy.horton@intel.com>
> ---
>   lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
>   lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
>   lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
>   3 files changed, 35 insertions(+), 1 deletion(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index a9977df97..52a97694c 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
>   }
>   
>   int
> +rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
> +{
> +	if (eth_dev == NULL)
> +		return -EINVAL;
> +
> +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
> +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> +
> +	return 0;
> +}
> +
> +int
>   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>   {
>   	if (eth_dev == NULL)
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
> index c9c825e3f..269586d88 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
>    * Release the specified ethdev port.
>    *
>    * @param eth_dev
> - * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> + * Device to be detached.
>    * @return
>    *   - 0 on success, negative on error
>    */
> @@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
>   
>   /**
>    * @internal
> + * Release the specified ethdev port in the local process.
> + * Only set ethdev state to unused, but not reset shared data since
> + * it assume other processes is still using it. typically it is
> + * called by a secondary process.
> + *
> + * @param eth_dev
> + * Device to be detached.
> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
> +
> +/**
> + * @internal
>    * Release device queues and clear its configuration to force the user
>    * application to reconfigure it. It is for internal use only.
>    *
> diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
> index 2cfd37274..a46d9e182 100644
> --- a/lib/librte_ethdev/rte_ethdev_pci.h
> +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> @@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
>   	if (!eth_dev)
>   		return -ENODEV;
>   
> +	/**
> +	 * PCI device can only be globally detached directly by a
> +	 * primary process. In secondary process, we only need to
> +	 * release port.
> +	 */
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_private(eth_dev);

I've realized that some uninit functions which will not be called anymore
in secondary processes have check for process type and handling of
secondary process case. It makes code inconsistent and should be fixed.

> +
>   	if (dev_uninit) {
>   		ret = dev_uninit(eth_dev);
>   		if (ret)

Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Zhang, Qi Z]

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Wednesday, July 11, 2018 5:27 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
> Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port
> in local process
> 
> On 11.07.2018 06:08, Qi Zhang wrote:
> > Add driver API rte_eth_release_port_private to support the case when
> > an ethdev need to be detached on a secondary process.
> > Local state is set to unused and shared data will not be reset so the
> > primary process can still use it.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
> > Acked-by: Remy Horton <remy.horton@intel.com>
> > ---

<...>
> > +	/**
> > +	 * PCI device can only be globally detached directly by a
> > +	 * primary process. In secondary process, we only need to
> > +	 * release port.
> > +	 */
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return rte_eth_dev_release_port_private(eth_dev);
> 
> I've realized that some uninit functions which will not be called anymore in
> secondary processes have check for process type and handling of secondary
> process case. It makes code inconsistent and should be fixed.

Good point, I did a scan and check all the places that rte_eth_dev_pci_generic_remove be involved.
I found only sfc driver (sfc_eth_dev_unit) will call some cleanup on secondary process as below.

		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
                sfc_eth_dev_secondary_clear_ops(dev);
                return 0;
        }

But in sfc_eth_dev_secondary_clear_ops

static void
sfc_eth_dev_secondary_clear_ops(struct rte_eth_dev *dev)
{
        dev->dev_ops = NULL;
        dev->tx_pkt_burst = NULL;
        dev->rx_pkt_burst = NULL;
}

So my understand is current change is not a problem for all exist drivers.

Please let me know if I missed something

Thanks
Qi

> 
> > +
> >   	if (dev_uninit) {
> >   		ret = dev_uninit(eth_dev);
> >   		if (ret)

Re: [dpdk-dev] [PATCH v11 06/19] eal: support attach or detach share device from secondary

[Burakov, Anatoly]

On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a device
> attach/detach request be issued from a secondary process
> 
> device attach on secondary:
> a) secondary send sync request to the primary.
> b) primary receive the request and attach the new device if
>     failed goto i).
> c) primary forward attach sync request to all secondary.
> d) secondary receive the request and attach the device and send a reply.
> e) primary check the reply if all success goes to j).
> f) primary send attach rollback sync request to all secondary.
> g) secondary receive the request and detach the device and send a reply.
> h) primary receive the reply and detach device as rollback action.
> i) send attach fail to secondary as a reply of step a), goto k).
> j) send attach success to secondary as a reply of step a).
> k) secondary receive reply and return.
> 
> device detach on secondary:
> a) secondary send sync request to the primary.
> b) primary send detach sync request to all secondary.
> c) secondary detach the device and send a reply.
> d) primary check the reply if all success goes to g).
> e) primary send detach rollback sync request to all secondary.
> f) secondary receive the request and attach back device. goto h).
> g) primary detach the device if success goto i), else goto e).
> h) primary send detach fail to secondary as a reply of step a), goto j).
> i) primary send detach success to secondary as a reply of step a).
> j) secondary receive reply and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_eal/common/eal_common_dev.c |  20 +++-
>   lib/librte_eal/common/hotplug_mp.c     | 175 ++++++++++++++++++++++++++++++++-
>   2 files changed, 189 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
> index ab2b25558..ee09720c4 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
>   	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
>   		/**
>   		 * If in secondary process, just send IPC request to
> -		 * primary process
> +		 * primary process.
>   		 */
>   		ret = eal_dev_hotplug_request_to_primary(&req);
>   		if (ret) {
> @@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
>   				"Failed to hotplug add device\n");
>   		return req.result;
>   	}
> +
>   	/**

The above two changes look like unintended noise (or perhaps you meant 
to merge them into one of the previous commits.

>   	 * attach a device from primary start from here:
>   	 *
> @@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
>   	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
>   		/**
>   		 * If in secondary process, just send IPC request to
> +		 * primary process.

<snip>

> +	memset(&mp_req, 0, sizeof(mp_req));
> +	memcpy(mp_req.param, req, sizeof(*req));
> +	mp_req.len_param = sizeof(*req);
> +	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
> +
> +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> +	if (ret || mp_reply.nb_received != 1) {
> +		RTE_LOG(ERR, EAL, "cannot send request to primary");
> +		return ret;
> +	}

In case of ret = 0 and nb_received = 0, you would be returning 0 as ret 
= 0. I don't think that's what you intended here :)

> +
> +	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
> +	req->result = resp->result;
> +
> +	return ret;
>   }
>   
>   int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v11 05/19] eal: enable hotplug on multi-process

[Burakov, Anatoly]

On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> We are going to introduce the solution to handle hotplug in
> multi-process, it includes the below scenario:
> 
> 1. Attach a device from the primary
> 2. Detach a device from the primary
> 3. Attach a device from a secondary
> 4. Detach a device from a secondary
> 
> In the primary-secondary process model, we assume devices are shared
> by default. that means attaches or detaches a device on any process
> will broadcast to all other processes through mp channel then device
> information will be synchronized on all processes.
> 
> Any failure during attaching/detaching process will cause inconsistent
> status between processes, so proper rollback action should be considered.
> 
> This patch covers the implementation of case 1,2.
> Case 3,4 will be implemented on a separate patch.
> 
> IPC scenario for Case 1, 2:
> 
> attach a device
> a) primary attach the new device if failed goto h).
> b) primary send attach sync request to all secondary.
> c) secondary receive request and attach the device and send a reply.
> d) primary check the reply if all success goes to i).
> e) primary send attach rollback sync request to all secondary.
> f) secondary receive the request and detach the device and send a reply.
> g) primary receive the reply and detach device as rollback action.
> h) attach fail
> i) attach success
> 
> detach a device
> a) primary send detach sync request to all secondary
> b) secondary detach the device and send reply
> c) primary check the reply if all success goes to f).
> d) primary send detach rollback sync request to all secondary.
> e) secondary receive the request and attach back device. goto g)
> f) primary detach the device if success goto g), else goto d)
> g) detach fail.
> h) detach success.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Multiprocess part looks OK to me, so

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v11 06/19] eal: support attach or detach share device from secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Wednesday, July 11, 2018 8:34 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v11 06/19] eal: support attach or detach share device from
> secondary
> 
> On 11-Jul-18 4:09 AM, Qi Zhang wrote:
> > This patch cover the multi-process hotplug case when a device
> > attach/detach request be issued from a secondary process
> >
> > device attach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary receive the request and attach the new device if
> >     failed goto i).
> > c) primary forward attach sync request to all secondary.
> > d) secondary receive the request and attach the device and send a reply.
> > e) primary check the reply if all success goes to j).
> > f) primary send attach rollback sync request to all secondary.
> > g) secondary receive the request and detach the device and send a reply.
> > h) primary receive the reply and detach device as rollback action.
> > i) send attach fail to secondary as a reply of step a), goto k).
> > j) send attach success to secondary as a reply of step a).
> > k) secondary receive reply and return.
> >
> > device detach on secondary:
> > a) secondary send sync request to the primary.
> > b) primary send detach sync request to all secondary.
> > c) secondary detach the device and send a reply.
> > d) primary check the reply if all success goes to g).
> > e) primary send detach rollback sync request to all secondary.
> > f) secondary receive the request and attach back device. goto h).
> > g) primary detach the device if success goto i), else goto e).
> > h) primary send detach fail to secondary as a reply of step a), goto j).
> > i) primary send detach success to secondary as a reply of step a).
> > j) secondary receive reply and return.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   lib/librte_eal/common/eal_common_dev.c |  20 +++-
> >   lib/librte_eal/common/hotplug_mp.c     | 175
> ++++++++++++++++++++++++++++++++-
> >   2 files changed, 189 insertions(+), 6 deletions(-)
> >
> > diff --git a/lib/librte_eal/common/eal_common_dev.c
> > b/lib/librte_eal/common/eal_common_dev.c
> > index ab2b25558..ee09720c4 100644
> > --- a/lib/librte_eal/common/eal_common_dev.c
> > +++ b/lib/librte_eal/common/eal_common_dev.c
> > @@ -221,7 +221,7 @@ rte_eal_hotplug_add(const char *busname, const
> char *devname,
> >   	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> >   		/**
> >   		 * If in secondary process, just send IPC request to
> > -		 * primary process
> > +		 * primary process.
> >   		 */
> >   		ret = eal_dev_hotplug_request_to_primary(&req);
> >   		if (ret) {
> > @@ -234,6 +234,7 @@ rte_eal_hotplug_add(const char *busname, const
> char *devname,
> >   				"Failed to hotplug add device\n");
> >   		return req.result;
> >   	}
> > +
> >   	/**
> 
> The above two changes look like unintended noise (or perhaps you meant to
> merge them into one of the previous commits.



> 
> >   	 * attach a device from primary start from here:
> >   	 *
> > @@ -295,6 +296,23 @@ rte_eal_hotplug_remove(const char *busname,
> const char *devname)
> >   	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> >   		/**
> >   		 * If in secondary process, just send IPC request to
> > +		 * primary process.
> 
> <snip>
> 
> > +	memset(&mp_req, 0, sizeof(mp_req));
> > +	memcpy(mp_req.param, req, sizeof(*req));
> > +	mp_req.len_param = sizeof(*req);
> > +	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST,
> > +sizeof(mp_req.name));
> > +
> > +	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
> > +	if (ret || mp_reply.nb_received != 1) {
> > +		RTE_LOG(ERR, EAL, "cannot send request to primary");
> > +		return ret;
> > +	}
> 
> In case of ret = 0 and nb_received = 0, you would be returning 0 as ret = 0. I
> don't think that's what you intended here :)

Ah.. yes, I missed that, thanks
 
> 
> > +
> > +	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
> > +	req->result = resp->result;
> > +
> > +	return ret;
> >   }
> >
> >   int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
> >
> 
> 
> --
> Thanks,
> Anatoly

[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Qi Zhang]

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11:
- move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               | 129 +++++++--
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 197 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 177 ++++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 348 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  48 ++++
 lib/librte_eal/common/include/rte_dev.h        |   6 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 33 files changed, 1117 insertions(+), 103 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v12 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..269586d88 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..a46d9e182 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	/**
+	 * PCI device can only be globally detached directly by a
+	 * primary process. In secondary process, we only need to
+	 * release port.
+	 */
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v12 02/19] bus/pci: fix PCI address compare

[Qi Zhang]

When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
is valid. So compare the 8th byte will cause the unexpected result, which
happens when repeatedly attach/detach a device.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..dd25c3542 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_vfio_tailq)
 
+/* Compair two pci address */
+static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
+{
+	if (addr1->domain == addr2->domain &&
+		addr1->bus == addr2->bus &&
+		addr1->devid == addr2->devid &&
+		addr1->function == addr2->function)
+		return 0;
+	return 1;
+}
+
 int
 pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
 		    void *buf, size_t len, off_t offs)
@@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 	/* Get vfio_res */
 	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
 			continue;
 		break;
 	}
-- 
2.13.6

[dpdk-dev] [PATCH v12 03/19] bus/pci: enable vfio unmap resource for secondary

[Qi Zhang]

Subroutine to unmap VFIO resource is shared by secondary and
primary, and it does not work on the secondary process. Since
for secondary process, it is not necessary to close interrupt
handler, set pci bus mastering and remove vfio_res from
vfio_res_list. So, the patch adds a dedicate function to handle
the situation when a device is unmapped on a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 118 +++++++++++++++++++++++++++++----------
 1 file changed, 90 insertions(+), 28 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index dd25c3542..93f0ac07e 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -595,6 +595,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 		dev->mem_resource[i].addr = maps[i].addr;
 	}
 
+	/* we need save vfio_dev_fd, so it can be used during release */
+	dev->intr_handle.vfio_dev_fd = vfio_dev_fd;
+
 	return 0;
 err_vfio_dev_fd:
 	close(vfio_dev_fd);
@@ -614,22 +617,58 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		return pci_vfio_map_resource_secondary(dev);
 }
 
-int
-pci_vfio_unmap_resource(struct rte_pci_device *dev)
+static struct mapped_pci_resource *
+find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,
+			struct rte_pci_device *dev,
+			const char *pci_addr)
+{
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct pci_map *maps;
+	int i;
+
+	/* Get vfio_res */
+	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
+			continue;
+		break;
+	}
+
+	if  (vfio_res == NULL)
+		return vfio_res;
+
+	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
+		pci_addr);
+
+	maps = vfio_res->maps;
+	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+
+		/*
+		 * We do not need to be aware of MSI-X table BAR mappings as
+		 * when mapping. Just using current maps array is enough
+		 */
+		if (maps[i].addr) {
+			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
+				pci_addr, maps[i].addr);
+			pci_unmap_resource(maps[i].addr, maps[i].size);
+		}
+	}
+
+	return vfio_res;
+}
+
+static int
+pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
 {
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
-	int i, ret;
 	struct mapped_pci_resource *vfio_res = NULL;
 	struct mapped_pci_res_list *vfio_res_list;
-
-	struct pci_map *maps;
+	int ret;
 
 	/* store PCI address string */
 	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
 			loc->domain, loc->bus, loc->devid, loc->function);
 
-
 	if (close(dev->intr_handle.fd) < 0) {
 		RTE_LOG(INFO, EAL, "Error when closing eventfd file descriptor for %s\n",
 			pci_addr);
@@ -650,13 +689,10 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return ret;
 	}
 
-	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
-	/* Get vfio_res */
-	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
-			continue;
-		break;
-	}
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
 	/* if we haven't found our tailq entry, something's wrong */
 	if (vfio_res == NULL) {
 		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
@@ -664,30 +700,56 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return -1;
 	}
 
-	/* unmap BARs */
-	maps = vfio_res->maps;
+	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
 
-	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
-		pci_addr);
-	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+	return 0;
+}
 
-		/*
-		 * We do not need to be aware of MSI-X table BAR mappings as
-		 * when mapping. Just using current maps array is enough
-		 */
-		if (maps[i].addr) {
-			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
-				pci_addr, maps[i].addr);
-			pci_unmap_resource(maps[i].addr, maps[i].size);
-		}
+static int
+pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
+{
+	char pci_addr[PATH_MAX] = {0};
+	struct rte_pci_addr *loc = &dev->addr;
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct mapped_pci_res_list *vfio_res_list;
+	int ret;
+
+	/* store PCI address string */
+	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
+			loc->domain, loc->bus, loc->devid, loc->function);
+
+	ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+				  dev->intr_handle.vfio_dev_fd);
+	if (ret < 0) {
+		RTE_LOG(ERR, EAL,
+			"%s(): cannot release device\n", __func__);
+		return ret;
 	}
 
-	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
+	/* if we haven't found our tailq entry, something's wrong */
+	if (vfio_res == NULL) {
+		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
+				pci_addr);
+		return -1;
+	}
 
 	return 0;
 }
 
 int
+pci_vfio_unmap_resource(struct rte_pci_device *dev)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return pci_vfio_unmap_resource_primary(dev);
+	else
+		return pci_vfio_unmap_resource_secondary(dev);
+}
+
+int
 pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
 		    struct rte_pci_ioport *p)
 {
-- 
2.13.6

[dpdk-dev] [PATCH v12 04/19] vfio: remove uneccessary IPC for group fd clear

[Qi Zhang]

Clear vfio_group_fd is not necessary to involve any IPC.
Also, current IPC implementation for SOCKET_CLR_GROUP is not
correct. rte_vfio_clear_group on secondary will always fail,
that prevent device be detached correctly on a secondary process.
The patch simply removes all IPC related stuff in
rte_vfio_clear_group.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_vfio.c         | 45 +++++---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |  1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |  8 -----
 3 files changed, 8 insertions(+), 46 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index a2bbdfbf4..c0eccddc3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -575,10 +575,6 @@ int
 rte_vfio_clear_group(int vfio_group_fd)
 {
 	int i;
-	struct rte_mp_msg mp_req, *mp_rep;
-	struct rte_mp_reply mp_reply;
-	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
-	struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
 	struct vfio_config *vfio_cfg;
 
 	vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
@@ -587,40 +583,15 @@ rte_vfio_clear_group(int vfio_group_fd)
 		return -1;
 	}
 
-	if (internal_config.process_type == RTE_PROC_PRIMARY) {
-
-		i = get_vfio_group_idx(vfio_group_fd);
-		if (i < 0)
-			return -1;
-		vfio_cfg->vfio_groups[i].group_num = -1;
-		vfio_cfg->vfio_groups[i].fd = -1;
-		vfio_cfg->vfio_groups[i].devices = 0;
-		vfio_cfg->vfio_active_groups--;
-		return 0;
-	}
-
-	p->req = SOCKET_CLR_GROUP;
-	p->group_num = vfio_group_fd;
-	strcpy(mp_req.name, EAL_VFIO_MP);
-	mp_req.len_param = sizeof(*p);
-	mp_req.num_fds = 0;
-
-	if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
-	    mp_reply.nb_received == 1) {
-		mp_rep = &mp_reply.msgs[0];
-		p = (struct vfio_mp_param *)mp_rep->param;
-		if (p->result == SOCKET_OK) {
-			free(mp_reply.msgs);
-			return 0;
-		} else if (p->result == SOCKET_NO_FD)
-			RTE_LOG(ERR, EAL, "  BAD VFIO group fd!\n");
-		else
-			RTE_LOG(ERR, EAL, "  no such VFIO group fd!\n");
-
-		free(mp_reply.msgs);
-	}
+	i = get_vfio_group_idx(vfio_group_fd);
+	if (i < 0)
+		return -1;
+	vfio_cfg->vfio_groups[i].group_num = -1;
+	vfio_cfg->vfio_groups[i].fd = -1;
+	vfio_cfg->vfio_groups[i].devices = 0;
+	vfio_cfg->vfio_active_groups--;
 
-	return -1;
+	return 0;
 }
 
 int
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index e65b10374..68d4750a5 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -129,7 +129,6 @@ int vfio_mp_sync_setup(void);
 
 #define SOCKET_REQ_CONTAINER 0x100
 #define SOCKET_REQ_GROUP 0x200
-#define SOCKET_CLR_GROUP 0x300
 #define SOCKET_OK 0x0
 #define SOCKET_NO_FD 0x1
 #define SOCKET_ERR 0xFF
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
index 9c202bb08..680a24aae 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
@@ -55,14 +55,6 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
 			reply.fds[0] = fd;
 		}
 		break;
-	case SOCKET_CLR_GROUP:
-		r->req = SOCKET_CLR_GROUP;
-		r->group_num = m->group_num;
-		if (rte_vfio_clear_group(m->group_num) < 0)
-			r->result = SOCKET_NO_FD;
-		else
-			r->result = SOCKET_OK;
-		break;
 	case SOCKET_REQ_CONTAINER:
 		r->req = SOCKET_REQ_CONTAINER;
 		fd = rte_vfio_get_container_fd();
-- 
2.13.6

[dpdk-dev] [PATCH v12 05/19] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 177 ++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 +++++++
 lib/librte_eal/common/hotplug_mp.c      | 181 ++++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  48 +++++++++
 lib/librte_eal/common/include/rte_dev.h |   6 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 ++
 9 files changed, 453 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 3fd33f1e4..4ecc73b42 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 14c5f05fa..ad5705b45 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -16,8 +16,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -102,8 +104,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -168,8 +171,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -197,7 +199,172 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+	if (ret) {
+		/* step h) */
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+		return ret;
+	}
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+		goto rollback;
+	}
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return -ENOMSG;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..1883b05d1 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,41 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..d3a8a5576
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..22efa073e
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[EAL_DEV_MP_BUS_NAME_MAX_LEN];
+	char devname[EAL_DEV_MP_DEV_NAME_MAX_LEN];
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 3879ff3ca..ab179c778 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -193,6 +193,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +215,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 8a3dcfee0..0f2cf9a1a 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -25,6 +25,7 @@ common_sources = files(
 	'eal_common_tailqs.c',
 	'eal_common_thread.c',
 	'eal_common_timer.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 3719ec9d7..6c225b54a 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..0c6984990 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -850,6 +850,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v12 06/19] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 177 +++++++++++++++++++++++++++++++++++--
 1 file changed, 172 insertions(+), 5 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index d3a8a5576..e24a72c9d 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (!tmp_req.result) {
+			ret = do_dev_hotplug_remove(req->busname,
+						req->devname);
+			if (ret) {
+				RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+				goto rollback;
+			}
+		} else {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+	if (ret)
+		RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -114,8 +260,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v12 07/19] net/i40e: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v12 08/19] net/ixgbe: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v12 09/19] net/af_packet: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..9b0dabeeb 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -935,6 +934,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v12 10/19] net/bonding: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..91c352c4c 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3053,8 +3053,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3062,6 +3061,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3168,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v12 11/19] net/kni: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..984d0d29a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -419,6 +418,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v12 12/19] net/null: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..4d5989bd1 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -623,6 +622,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +667,24 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v12 13/19] net/octeontx: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..1b98f4086 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_private(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1154,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v12 14/19] net/pcap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..3c27a98df 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -916,8 +916,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -925,6 +924,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1016,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1024,15 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v12 15/19] net/softnic: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..f3b6863ae 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -741,8 +741,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 	/* Parse input arguments */
 	params = rte_vdev_device_args(vdev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -750,6 +749,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +803,22 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(dev);
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v12 16/19] net/tap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..a58d5ebe6 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1750,8 +1750,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -1759,6 +1758,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1827,17 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v12 17/19] net/vhost: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..3a3f8f6cc 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1353,6 +1352,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v12 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a af_packet vdev */
>attach net_af_packet,iface=eth0

/* detach port 0 */
>detach 0

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++++
 examples/multi_process/hotplug_mp/commands.c | 197 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 ++++++
 5 files changed, 272 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..fb7198d51
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <port_id>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t device;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	uint16_t port_id;
+
+	if (!rte_eth_dev_attach(res->device, &port_id))
+		cmdline_printf(cl, "attached device %s at port %d\n",
+			res->device, port_id);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+			res->device);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_device =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, device, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_device,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	uint16_t port_id;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	uint16_t port_id = res->port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	printf("detaching...\n");
+	if (!rte_eth_dev_detach(port_id, dev_name))
+		cmdline_printf(cl, "detached device at port %d\n",
+			port_id);
+	else
+		cmdline_printf(cl, "failed to dettached at port %d\n",
+			port_id);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+cmdline_parse_token_num_t cmd_dev_detach_port_id =
+	TOKEN_NUM_INITIALIZER(struct cmd_dev_detach_result, port_id, UINT16);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_port_id,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v12 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..1251e4b5b 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,12 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -60,6 +66,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v12 06/19] eal: support attach or detach share device from secondary

[Burakov, Anatoly]

On 11-Jul-18 2:47 PM, Qi Zhang wrote:
> This patch cover the multi-process hotplug case when a device
> attach/detach request be issued from a secondary process
> 
> device attach on secondary:
> a) secondary send sync request to the primary.
> b) primary receive the request and attach the new device if
>     failed goto i).
> c) primary forward attach sync request to all secondary.
> d) secondary receive the request and attach the device and send a reply.
> e) primary check the reply if all success goes to j).
> f) primary send attach rollback sync request to all secondary.
> g) secondary receive the request and detach the device and send a reply.
> h) primary receive the reply and detach device as rollback action.
> i) send attach fail to secondary as a reply of step a), goto k).
> j) send attach success to secondary as a reply of step a).
> k) secondary receive reply and return.
> 
> device detach on secondary:
> a) secondary send sync request to the primary.
> b) primary send detach sync request to all secondary.
> c) secondary detach the device and send a reply.
> d) primary check the reply if all success goes to g).
> e) primary send detach rollback sync request to all secondary.
> f) secondary receive the request and attach back device. goto h).
> g) primary detach the device if success goto i), else goto e).
> h) primary send detach fail to secondary as a reply of step a), goto j).
> i) primary send detach success to secondary as a reply of step a).
> j) secondary receive reply and return.
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Andrew Rybchenko]

On 11.07.2018 15:30, Zhang, Qi Z wrote:
>
>> -----Original Message-----
>> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
>> Sent: Wednesday, July 11, 2018 5:27 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
>> Anatoly <anatoly.burakov@intel.com>
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port
>> in local process
>>
>> On 11.07.2018 06:08, Qi Zhang wrote:
>>> Add driver API rte_eth_release_port_private to support the case when
>>> an ethdev need to be detached on a secondary process.
>>> Local state is set to unused and shared data will not be reset so the
>>> primary process can still use it.
>>>
>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
>>> Acked-by: Remy Horton <remy.horton@intel.com>
>>> ---
> <...>
>>> +	/**
>>> +	 * PCI device can only be globally detached directly by a
>>> +	 * primary process. In secondary process, we only need to
>>> +	 * release port.
>>> +	 */
>>> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
>>> +		return rte_eth_dev_release_port_private(eth_dev);
>> I've realized that some uninit functions which will not be called anymore in
>> secondary processes have check for process type and handling of secondary
>> process case. It makes code inconsistent and should be fixed.
> Good point, I did a scan and check all the places that rte_eth_dev_pci_generic_remove be involved.
> I found only sfc driver (sfc_eth_dev_unit) will call some cleanup on secondary process as below.

The patch makes impossible dev_uninit to be executed for secondary process
for all cases if rte_eth_dev_pci_generic_remove() is used. However, many 
drivers
still check for process type. Yes, sfc does cleanup, but some drivers 
return -EPERM,
some return 0. In fact it does not matter. It leaves dead code which is 
really confusing.

>
> 		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
>                  sfc_eth_dev_secondary_clear_ops(dev);
>                  return 0;
>          }
>
> But in sfc_eth_dev_secondary_clear_ops
>
> static void
> sfc_eth_dev_secondary_clear_ops(struct rte_eth_dev *dev)
> {
>          dev->dev_ops = NULL;
>          dev->tx_pkt_burst = NULL;
>          dev->rx_pkt_burst = NULL;
> }
>
> So my understand is current change is not a problem for all exist drivers.
>
> Please let me know if I missed something
>
> Thanks
> Qi
>
>>> +
>>>    	if (dev_uninit) {
>>>    		ret = dev_uninit(eth_dev);
>>>    		if (ret)

Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Zhang, Qi Z]

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Thursday, July 12, 2018 12:05 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
> Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port
> in local process
> 
> On 11.07.2018 15:30, Zhang, Qi Z wrote:
> >
> >> -----Original Message-----
> >> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> >> Sent: Wednesday, July 11, 2018 5:27 PM
> >> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
> >> Anatoly <anatoly.burakov@intel.com>
> >> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> >> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> >> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> >> <benjamin.h.shelton@intel.com>; Vangati, Narender
> >> <narender.vangati@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to
> >> release port in local process
> >>
> >> On 11.07.2018 06:08, Qi Zhang wrote:
> >>> Add driver API rte_eth_release_port_private to support the case when
> >>> an ethdev need to be detached on a secondary process.
> >>> Local state is set to unused and shared data will not be reset so
> >>> the primary process can still use it.
> >>>
> >>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> >>> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
> >>> Acked-by: Remy Horton <remy.horton@intel.com>
> >>> ---
> > <...>
> >>> +	/**
> >>> +	 * PCI device can only be globally detached directly by a
> >>> +	 * primary process. In secondary process, we only need to
> >>> +	 * release port.
> >>> +	 */
> >>> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> >>> +		return rte_eth_dev_release_port_private(eth_dev);
> >> I've realized that some uninit functions which will not be called
> >> anymore in secondary processes have check for process type and
> >> handling of secondary process case. It makes code inconsistent and should
> be fixed.
> > Good point, I did a scan and check all the places that
> rte_eth_dev_pci_generic_remove be involved.
> > I found only sfc driver (sfc_eth_dev_unit) will call some cleanup on
> secondary process as below.
> 
> The patch makes impossible dev_uninit to be executed for secondary process
> for all cases if rte_eth_dev_pci_generic_remove() is used. However, many
> drivers still check for process type. Yes, sfc does cleanup, but some drivers
> return -EPERM, some return 0. In fact it does not matter. It leaves dead code
> which is really confusing.

OK, l can do a cleanup in a separate patchset if this one will be merged.

> 
> >
> > 		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> >                  sfc_eth_dev_secondary_clear_ops(dev);
> >                  return 0;
> >          }
> >
> > But in sfc_eth_dev_secondary_clear_ops
> >
> > static void
> > sfc_eth_dev_secondary_clear_ops(struct rte_eth_dev *dev) {
> >          dev->dev_ops = NULL;
> >          dev->tx_pkt_burst = NULL;
> >          dev->rx_pkt_burst = NULL;
> > }
> >
> > So my understand is current change is not a problem for all exist drivers.
> >
> > Please let me know if I missed something
> >
> > Thanks
> > Qi
> >
> >>> +
> >>>    	if (dev_uninit) {
> >>>    		ret = dev_uninit(eth_dev);
> >>>    		if (ret)

[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Qi Zhang]

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11:
- move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               | 129 +++++++--
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 214 +++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 177 ++++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 348 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  48 ++++
 lib/librte_eal/common/include/rte_dev.h        |   6 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 33 files changed, 1134 insertions(+), 103 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v13 01/19] ethdev: add function to release port in local process

[Qi Zhang]

Add driver API rte_eth_release_port_private to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: Remy Horton <remy.horton@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c        | 12 ++++++++++++
 lib/librte_ethdev/rte_ethdev_driver.h | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h    |  8 ++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index a9977df97..52a97694c 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c9c825e3f..269586d88 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -62,7 +62,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -70,6 +70,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_private(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index 2cfd37274..a46d9e182 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -197,6 +197,14 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device *pci_dev,
 	if (!eth_dev)
 		return -ENODEV;
 
+	/**
+	 * PCI device can only be globally detached directly by a
+	 * primary process. In secondary process, we only need to
+	 * release port.
+	 */
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	if (dev_uninit) {
 		ret = dev_uninit(eth_dev);
 		if (ret)
-- 
2.13.6

[dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare

[Qi Zhang]

When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
is valid. So compare the 8th byte will cause the unexpected result, which
happens when repeatedly attach/detach a device.

Fixes: c752998b5e2e ("pci: introduce library and driver")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..dd25c3542 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
 };
 EAL_REGISTER_TAILQ(rte_vfio_tailq)
 
+/* Compair two pci address */
+static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
+{
+	if (addr1->domain == addr2->domain &&
+		addr1->bus == addr2->bus &&
+		addr1->devid == addr2->devid &&
+		addr1->function == addr2->function)
+		return 0;
+	return 1;
+}
+
 int
 pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
 		    void *buf, size_t len, off_t offs)
@@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 	/* Get vfio_res */
 	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
 			continue;
 		break;
 	}
-- 
2.13.6

[dpdk-dev] [PATCH v13 03/19] bus/pci: enable vfio unmap resource for secondary

[Qi Zhang]

Subroutine to unmap VFIO resource is shared by secondary and
primary, and it does not work on the secondary process. Since
for secondary process, it is not necessary to close interrupt
handler, set pci bus mastering and remove vfio_res from
vfio_res_list. So, the patch adds a dedicate function to handle
the situation when a device is unmapped on a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/bus/pci/linux/pci_vfio.c | 118 +++++++++++++++++++++++++++++----------
 1 file changed, 90 insertions(+), 28 deletions(-)

diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index dd25c3542..93f0ac07e 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -595,6 +595,9 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 		dev->mem_resource[i].addr = maps[i].addr;
 	}
 
+	/* we need save vfio_dev_fd, so it can be used during release */
+	dev->intr_handle.vfio_dev_fd = vfio_dev_fd;
+
 	return 0;
 err_vfio_dev_fd:
 	close(vfio_dev_fd);
@@ -614,22 +617,58 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 		return pci_vfio_map_resource_secondary(dev);
 }
 
-int
-pci_vfio_unmap_resource(struct rte_pci_device *dev)
+static struct mapped_pci_resource *
+find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,
+			struct rte_pci_device *dev,
+			const char *pci_addr)
+{
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct pci_map *maps;
+	int i;
+
+	/* Get vfio_res */
+	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
+		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
+			continue;
+		break;
+	}
+
+	if  (vfio_res == NULL)
+		return vfio_res;
+
+	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
+		pci_addr);
+
+	maps = vfio_res->maps;
+	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+
+		/*
+		 * We do not need to be aware of MSI-X table BAR mappings as
+		 * when mapping. Just using current maps array is enough
+		 */
+		if (maps[i].addr) {
+			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
+				pci_addr, maps[i].addr);
+			pci_unmap_resource(maps[i].addr, maps[i].size);
+		}
+	}
+
+	return vfio_res;
+}
+
+static int
+pci_vfio_unmap_resource_primary(struct rte_pci_device *dev)
 {
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
-	int i, ret;
 	struct mapped_pci_resource *vfio_res = NULL;
 	struct mapped_pci_res_list *vfio_res_list;
-
-	struct pci_map *maps;
+	int ret;
 
 	/* store PCI address string */
 	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
 			loc->domain, loc->bus, loc->devid, loc->function);
 
-
 	if (close(dev->intr_handle.fd) < 0) {
 		RTE_LOG(INFO, EAL, "Error when closing eventfd file descriptor for %s\n",
 			pci_addr);
@@ -650,13 +689,10 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return ret;
 	}
 
-	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
-	/* Get vfio_res */
-	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
-		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
-			continue;
-		break;
-	}
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
 	/* if we haven't found our tailq entry, something's wrong */
 	if (vfio_res == NULL) {
 		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
@@ -664,30 +700,56 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
 		return -1;
 	}
 
-	/* unmap BARs */
-	maps = vfio_res->maps;
+	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
 
-	RTE_LOG(INFO, EAL, "Releasing pci mapped resource for %s\n",
-		pci_addr);
-	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+	return 0;
+}
 
-		/*
-		 * We do not need to be aware of MSI-X table BAR mappings as
-		 * when mapping. Just using current maps array is enough
-		 */
-		if (maps[i].addr) {
-			RTE_LOG(INFO, EAL, "Calling pci_unmap_resource for %s at %p\n",
-				pci_addr, maps[i].addr);
-			pci_unmap_resource(maps[i].addr, maps[i].size);
-		}
+static int
+pci_vfio_unmap_resource_secondary(struct rte_pci_device *dev)
+{
+	char pci_addr[PATH_MAX] = {0};
+	struct rte_pci_addr *loc = &dev->addr;
+	struct mapped_pci_resource *vfio_res = NULL;
+	struct mapped_pci_res_list *vfio_res_list;
+	int ret;
+
+	/* store PCI address string */
+	snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT,
+			loc->domain, loc->bus, loc->devid, loc->function);
+
+	ret = rte_vfio_release_device(rte_pci_get_sysfs_path(), pci_addr,
+				  dev->intr_handle.vfio_dev_fd);
+	if (ret < 0) {
+		RTE_LOG(ERR, EAL,
+			"%s(): cannot release device\n", __func__);
+		return ret;
 	}
 
-	TAILQ_REMOVE(vfio_res_list, vfio_res, next);
+	vfio_res_list =
+		RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
+	vfio_res = find_and_unmap_vfio_resource(vfio_res_list, dev, pci_addr);
+
+	/* if we haven't found our tailq entry, something's wrong */
+	if (vfio_res == NULL) {
+		RTE_LOG(ERR, EAL, "  %s cannot find TAILQ entry for PCI device!\n",
+				pci_addr);
+		return -1;
+	}
 
 	return 0;
 }
 
 int
+pci_vfio_unmap_resource(struct rte_pci_device *dev)
+{
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return pci_vfio_unmap_resource_primary(dev);
+	else
+		return pci_vfio_unmap_resource_secondary(dev);
+}
+
+int
 pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
 		    struct rte_pci_ioport *p)
 {
-- 
2.13.6

[dpdk-dev] [PATCH v13 04/19] vfio: remove uneccessary IPC for group fd clear

[Qi Zhang]

Clear vfio_group_fd is not necessary to involve any IPC.
Also, current IPC implementation for SOCKET_CLR_GROUP is not
correct. rte_vfio_clear_group on secondary will always fail,
that prevent device be detached correctly on a secondary process.
The patch simply removes all IPC related stuff in
rte_vfio_clear_group.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/linuxapp/eal/eal_vfio.c         | 45 +++++---------------------
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |  1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |  8 -----
 3 files changed, 8 insertions(+), 46 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.c b/lib/librte_eal/linuxapp/eal/eal_vfio.c
index a2bbdfbf4..c0eccddc3 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.c
@@ -575,10 +575,6 @@ int
 rte_vfio_clear_group(int vfio_group_fd)
 {
 	int i;
-	struct rte_mp_msg mp_req, *mp_rep;
-	struct rte_mp_reply mp_reply;
-	struct timespec ts = {.tv_sec = 5, .tv_nsec = 0};
-	struct vfio_mp_param *p = (struct vfio_mp_param *)mp_req.param;
 	struct vfio_config *vfio_cfg;
 
 	vfio_cfg = get_vfio_cfg_by_group_fd(vfio_group_fd);
@@ -587,40 +583,15 @@ rte_vfio_clear_group(int vfio_group_fd)
 		return -1;
 	}
 
-	if (internal_config.process_type == RTE_PROC_PRIMARY) {
-
-		i = get_vfio_group_idx(vfio_group_fd);
-		if (i < 0)
-			return -1;
-		vfio_cfg->vfio_groups[i].group_num = -1;
-		vfio_cfg->vfio_groups[i].fd = -1;
-		vfio_cfg->vfio_groups[i].devices = 0;
-		vfio_cfg->vfio_active_groups--;
-		return 0;
-	}
-
-	p->req = SOCKET_CLR_GROUP;
-	p->group_num = vfio_group_fd;
-	strcpy(mp_req.name, EAL_VFIO_MP);
-	mp_req.len_param = sizeof(*p);
-	mp_req.num_fds = 0;
-
-	if (rte_mp_request_sync(&mp_req, &mp_reply, &ts) == 0 &&
-	    mp_reply.nb_received == 1) {
-		mp_rep = &mp_reply.msgs[0];
-		p = (struct vfio_mp_param *)mp_rep->param;
-		if (p->result == SOCKET_OK) {
-			free(mp_reply.msgs);
-			return 0;
-		} else if (p->result == SOCKET_NO_FD)
-			RTE_LOG(ERR, EAL, "  BAD VFIO group fd!\n");
-		else
-			RTE_LOG(ERR, EAL, "  no such VFIO group fd!\n");
-
-		free(mp_reply.msgs);
-	}
+	i = get_vfio_group_idx(vfio_group_fd);
+	if (i < 0)
+		return -1;
+	vfio_cfg->vfio_groups[i].group_num = -1;
+	vfio_cfg->vfio_groups[i].fd = -1;
+	vfio_cfg->vfio_groups[i].devices = 0;
+	vfio_cfg->vfio_active_groups--;
 
-	return -1;
+	return 0;
 }
 
 int
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio.h b/lib/librte_eal/linuxapp/eal/eal_vfio.h
index e65b10374..68d4750a5 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio.h
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio.h
@@ -129,7 +129,6 @@ int vfio_mp_sync_setup(void);
 
 #define SOCKET_REQ_CONTAINER 0x100
 #define SOCKET_REQ_GROUP 0x200
-#define SOCKET_CLR_GROUP 0x300
 #define SOCKET_OK 0x0
 #define SOCKET_NO_FD 0x1
 #define SOCKET_ERR 0xFF
diff --git a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
index 9c202bb08..680a24aae 100644
--- a/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
+++ b/lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c
@@ -55,14 +55,6 @@ vfio_mp_primary(const struct rte_mp_msg *msg, const void *peer)
 			reply.fds[0] = fd;
 		}
 		break;
-	case SOCKET_CLR_GROUP:
-		r->req = SOCKET_CLR_GROUP;
-		r->group_num = m->group_num;
-		if (rte_vfio_clear_group(m->group_num) < 0)
-			r->result = SOCKET_NO_FD;
-		else
-			r->result = SOCKET_OK;
-		break;
 	case SOCKET_REQ_CONTAINER:
 		r->req = SOCKET_REQ_CONTAINER;
 		fd = rte_vfio_get_container_fd();
-- 
2.13.6

[dpdk-dev] [PATCH v13 05/19] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 177 ++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 +++++++
 lib/librte_eal/common/hotplug_mp.c      | 181 ++++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  48 +++++++++
 lib/librte_eal/common/include/rte_dev.h |   6 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 ++
 9 files changed, 453 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 3fd33f1e4..4ecc73b42 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -59,6 +59,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 14c5f05fa..ad5705b45 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -16,8 +16,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -102,8 +104,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -168,8 +171,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -197,7 +199,172 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+	if (ret) {
+		/* step h) */
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+		return ret;
+	}
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+		goto rollback;
+	}
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return -ENOMSG;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index bdadc4d50..1883b05d1 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -258,4 +258,41 @@ int rte_mp_channel_init(void);
  */
 void dev_callback_process(char *device_name, enum rte_dev_event_type event);
 
+/**
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..d3a8a5576
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,181 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..22efa073e
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[EAL_DEV_MP_BUS_NAME_MAX_LEN];
+	char devname[EAL_DEV_MP_DEV_NAME_MAX_LEN];
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 3879ff3ca..ab179c778 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -193,6 +193,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +215,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 8a3dcfee0..0f2cf9a1a 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -25,6 +25,7 @@ common_sources = files(
 	'eal_common_tailqs.c',
 	'eal_common_thread.c',
 	'eal_common_timer.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 3719ec9d7..6c225b54a 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -67,6 +67,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_thread.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 8655b8691..0c6984990 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -850,6 +850,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v13 06/19] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 177 +++++++++++++++++++++++++++++++++++--
 1 file changed, 172 insertions(+), 5 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index d3a8a5576..e24a72c9d 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,158 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			goto finish;
+		}
+		if (!tmp_req.result) {
+			ret = do_dev_hotplug_remove(req->busname,
+						req->devname);
+			if (ret) {
+				RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+				goto rollback;
+			}
+		} else {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+	if (ret)
+		RTE_LOG(ERR, EAL, "Failed to send hotplug rollback request to secondary\n");
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -114,8 +260,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v13 07/19] net/i40e: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_ethdev.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_ethdev.c b/drivers/net/i40e/i40e_ethdev.c
index 13c5d3296..7d1f98422 100644
--- a/drivers/net/i40e/i40e_ethdev.c
+++ b/drivers/net/i40e/i40e_ethdev.c
@@ -678,6 +678,8 @@ static int eth_i40e_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
 
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, i40e_vf_representor_uninit);
-- 
2.13.6

[dpdk-dev] [PATCH v13 08/19] net/ixgbe: enable hotplug on secondary process

[Qi Zhang]

Previously, detach port on a secondary process will mess primary
process and cause the same device can't be attached back again.
A secondary process should use rte_eth_release_port_private to
release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 87d2ad090..161a15f05 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -1792,6 +1792,9 @@ static int eth_ixgbe_pci_remove(struct rte_pci_device *pci_dev)
 	if (!ethdev)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(ethdev);
+
 	if (ethdev->data->dev_flags & RTE_ETH_DEV_REPRESENTOR)
 		return rte_eth_dev_destroy(ethdev, ixgbe_vf_representor_uninit);
 	else
-- 
2.13.6

[dpdk-dev] [PATCH v13 09/19] net/af_packet: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index ea47abbf8..9b0dabeeb 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -935,6 +934,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -986,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
-- 
2.13.6

[dpdk-dev] [PATCH v13 10/19] net/bonding: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bonding/rte_eth_bond_pmd.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index f155ff779..91c352c4c 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3053,8 +3053,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3062,6 +3061,7 @@ bond_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &default_dev_ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -3168,6 +3168,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v13 11/19] net/kni: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/kni/rte_eth_kni.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index ab63ea427..984d0d29a 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -419,6 +418,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &eth_kni_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -463,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
-- 
2.13.6

[dpdk-dev] [PATCH v13 12/19] net/null: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/null/rte_eth_null.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 1d2e6b9e9..4d5989bd1 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -623,6 +622,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -667,18 +667,24 @@ static int
 rte_pmd_null_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	if (!dev)
 		return -EINVAL;
 
+	name = rte_vdev_device_name(dev);
+
 	PMD_LOG(INFO, "Closing null ethdev on numa socket %u",
 			rte_socket_id());
 
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v13 13/19] net/octeontx: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/octeontx/octeontx_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 1eb453b21..1b98f4086 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1016,6 +1016,7 @@ octeontx_create(struct rte_vdev_device *dev, int port, uint8_t evdev,
 
 		eth_dev->tx_pkt_burst = octeontx_xmit_pkts;
 		eth_dev->rx_pkt_burst = octeontx_recv_pkts;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1138,6 +1139,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_private(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1148,6 +1154,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
-- 
2.13.6

[dpdk-dev] [PATCH v13 14/19] net/pcap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/pcap/rte_eth_pcap.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 6bd4a7d79..3c27a98df 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -916,8 +916,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -925,6 +924,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1016,6 +1016,7 @@ static int
 pmd_pcap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
+	const char *name;
 
 	PMD_LOG(INFO, "Closing pcap ethdev on numa socket %d",
 			rte_socket_id());
@@ -1023,11 +1024,15 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (!dev)
 		return -1;
 
+	name = rte_vdev_device_name(dev);
 	/* reserve an ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v13 15/19] net/softnic: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/softnic/rte_eth_softnic.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/net/softnic/rte_eth_softnic.c b/drivers/net/softnic/rte_eth_softnic.c
index 6b3c13e5c..f3b6863ae 100644
--- a/drivers/net/softnic/rte_eth_softnic.c
+++ b/drivers/net/softnic/rte_eth_softnic.c
@@ -741,8 +741,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 	/* Parse input arguments */
 	params = rte_vdev_device_args(vdev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -750,6 +749,7 @@ pmd_probe(struct rte_vdev_device *vdev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &pmd_ops;
+		eth_dev->device = &vdev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -803,17 +803,22 @@ pmd_remove(struct rte_vdev_device *vdev)
 {
 	struct rte_eth_dev *dev = NULL;
 	struct pmd_internals *p;
+	const char *name;
 
 	if (!vdev)
 		return -EINVAL;
 
-	PMD_LOG(INFO, "Removing device \"%s\"",
-		rte_vdev_device_name(vdev));
+	name = rte_vdev_device_name(vdev);
+	PMD_LOG(INFO, "Removing device \"%s\"", name);
 
 	/* Find the ethdev entry */
-	dev = rte_eth_dev_allocated(rte_vdev_device_name(vdev));
+	dev = rte_eth_dev_allocated(name);
 	if (dev == NULL)
 		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(dev);
+
 	p = dev->data->dev_private;
 
 	/* Free device data structures*/
-- 
2.13.6

[dpdk-dev] [PATCH v13 16/19] net/tap: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Keith Wiles <keith.wiles@intel.com>
---
 drivers/net/tap/rte_eth_tap.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index df396bfde..a58d5ebe6 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1750,8 +1750,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -1759,6 +1758,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1827,12 +1827,17 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 {
 	struct rte_eth_dev *eth_dev = NULL;
 	struct pmd_internals *internals;
+	const char *name;
 	int i;
 
+	name = rte_vdev_device_name(dev);
 	/* find the ethdev entry */
-	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
+	eth_dev = rte_eth_dev_allocated(name);
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
-- 
2.13.6

[dpdk-dev] [PATCH v13 17/19] net/vhost: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_release_port_private to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index ba9d768a0..3a3f8f6cc 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1353,6 +1352,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 		}
 		/* TODO: request info from primary to set up Rx and Tx */
 		eth_dev->dev_ops = &ops;
+		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
 	}
@@ -1435,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_private(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v13 18/19] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach a pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
>attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
>detach net_af_packet

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +++
 examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 +++++
 5 files changed, 289 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..62f328261
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <devargs>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	if (!rte_eal_hotplug_add(da.bus->name, da.name, da.args))
+		cmdline_printf(cl, "attached device %s\n", da.name);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+				da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parse(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	printf("detaching...\n");
+	if (!rte_eal_hotplug_remove(da.bus->name, da.name))
+		cmdline_printf(cl, "detached device %s\n",
+			da.name);
+	else
+		cmdline_printf(cl, "failed to dettach device %s\n",
+			da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+
+cmdline_parse_token_string_t cmd_dev_detach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v13 19/19] doc: update release notes for multi process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_08.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst
index bc0124295..1251e4b5b 100644
--- a/doc/guides/rel_notes/release_18_08.rst
+++ b/doc/guides/rel_notes/release_18_08.rst
@@ -46,6 +46,12 @@ New Features
   Flow API support has been added to CXGBE Poll Mode Driver to offload
   flows to Chelsio T5/T6 NICs.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -60,6 +66,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

[dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Qi Zhang]

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11:
- move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               | 129 +++++++--
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 214 +++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 177 ++++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 348 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  48 ++++
 lib/librte_eal/common/include/rte_dev.h        |   6 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 33 files changed, 1134 insertions(+), 103 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v13 00/19] enable hotplug on multi-process

[Qi Zhang]

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11:
- move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_local which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this on all Intel devices and
vdev, it can be refereneced by other driver when equevalent fix is
required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (19):
  ethdev: add function to release port in local process
  bus/pci: fix PCI address compare
  bus/pci: enable vfio unmap resource for secondary
  vfio: remove uneccessary IPC for group fd clear
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  net/i40e: enable hotplug on secondary process
  net/ixgbe: enable hotplug on secondary process
  net/af_packet: enable hotplug on secondary process
  net/bonding: enable hotplug on secondary process
  net/kni: enable hotplug on secondary process
  net/null: enable hotplug on secondary process
  net/octeontx: enable hotplug on secondary process
  net/pcap: enable hotplug on secondary process
  net/softnic: enable hotplug on secondary process
  net/tap: enable hotplug on secondary process
  net/vhost: enable hotplug on secondary process
  examples/multi_process: add hotplug sample
  doc: update release notes for multi process hotplug

 doc/guides/rel_notes/release_18_08.rst         |  11 +
 drivers/bus/pci/linux/pci_vfio.c               | 129 +++++++--
 drivers/net/af_packet/rte_eth_af_packet.c      |   7 +-
 drivers/net/bonding/rte_eth_bond_pmd.c         |   7 +-
 drivers/net/i40e/i40e_ethdev.c                 |   2 +
 drivers/net/ixgbe/ixgbe_ethdev.c               |   3 +
 drivers/net/kni/rte_eth_kni.c                  |   7 +-
 drivers/net/null/rte_eth_null.c                |  12 +-
 drivers/net/octeontx/octeontx_ethdev.c         |   9 +
 drivers/net/pcap/rte_eth_pcap.c                |  11 +-
 drivers/net/softnic/rte_eth_softnic.c          |  15 +-
 drivers/net/tap/rte_eth_tap.c                  |  13 +-
 drivers/net/vhost/rte_eth_vhost.c              |   7 +-
 examples/multi_process/Makefile                |   1 +
 examples/multi_process/hotplug_mp/Makefile     |  23 ++
 examples/multi_process/hotplug_mp/commands.c   | 214 +++++++++++++++
 examples/multi_process/hotplug_mp/commands.h   |  10 +
 examples/multi_process/hotplug_mp/main.c       |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile             |   1 +
 lib/librte_eal/common/eal_common_dev.c         | 177 ++++++++++++-
 lib/librte_eal/common/eal_private.h            |  37 +++
 lib/librte_eal/common/hotplug_mp.c             | 348 +++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h             |  48 ++++
 lib/librte_eal/common/include/rte_dev.h        |   6 +
 lib/librte_eal/common/meson.build              |   1 +
 lib/librte_eal/linuxapp/eal/Makefile           |   1 +
 lib/librte_eal/linuxapp/eal/eal.c              |   6 +
 lib/librte_eal/linuxapp/eal/eal_vfio.c         |  45 +---
 lib/librte_eal/linuxapp/eal/eal_vfio.h         |   1 -
 lib/librte_eal/linuxapp/eal/eal_vfio_mp_sync.c |   8 -
 lib/librte_ethdev/rte_ethdev.c                 |  12 +
 lib/librte_ethdev/rte_ethdev_driver.h          |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h             |   8 +
 33 files changed, 1134 insertions(+), 103 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Thomas Monjalon]

12/07/2018 03:14, Qi Zhang:
> v13:
> - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
>   so, modify the sample code to use rte_eal_hotplug_add and
>   rte_eal_hotplug_remove to attach/detach device.

Yes, this is what I tried to explain you.

I think it is now too late for 18.08.
We see that this patchset deserves more reviews.

Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Thursday, July 12, 2018 4:30 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; Burakov, Anatoly <anatoly.burakov@intel.com>; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Shelton,
> Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process
> 
> 12/07/2018 03:14, Qi Zhang:
> > v13:
> > - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
> >   so, modify the sample code to use rte_eal_hotplug_add and
> >   rte_eal_hotplug_remove to attach/detach device.
> 
> Yes, this is what I tried to explain you.
> 
> I think it is now too late for 18.08.

Understand, but probably patch 2,3,4 could be considered in 18.08, since they fix general issue but not just for hotplug mp.
What do you think?


> We see that this patchset deserves more reviews.
> 
> 

Re: [dpdk-dev] [PATCH v12 00/19] enable hotplug on multi-process

[Thomas Monjalon]

12/07/2018 11:11, Zhang, Qi Z:
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > 12/07/2018 03:14, Qi Zhang:
> > > v13:
> > > - Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
> > >   so, modify the sample code to use rte_eal_hotplug_add and
> > >   rte_eal_hotplug_remove to attach/detach device.
> > 
> > Yes, this is what I tried to explain you.
> > 
> > I think it is now too late for 18.08.
> 
> Understand, but probably patch 2,3,4 could be considered in 18.08, since they fix general issue but not just for hotplug mp.
> What do you think?

Yes, you are right.

Please send a separate patchset and try to get reviews.
Gaetan, Anatoly, please review patches 2 and 3.

Re: [dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare

[Burakov, Anatoly]

On 12-Jul-18 2:14 AM, Qi Zhang wrote:
> When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
> is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
> is valid. So compare the 8th byte will cause the unexpected result, which
> happens when repeatedly attach/detach a device.
> 
> Fixes: c752998b5e2e ("pci: introduce library and driver")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> index aeeaa9ed8..dd25c3542 100644
> --- a/drivers/bus/pci/linux/pci_vfio.c
> +++ b/drivers/bus/pci/linux/pci_vfio.c
> @@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
>   };
>   EAL_REGISTER_TAILQ(rte_vfio_tailq)
>   
> +/* Compair two pci address */
> +static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
> +{
> +	if (addr1->domain == addr2->domain &&
> +		addr1->bus == addr2->bus &&
> +		addr1->devid == addr2->devid &&
> +		addr1->function == addr2->function)
> +		return 0;
> +	return 1;
> +}

Generally, change looks OK to me, but I think we already have this 
function in PCI library - rte_pci_addr_cmp(). Is there a specific reason 
to reimplement it here?

> +
>   int
>   pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
>   		    void *buf, size_t len, off_t offs)
> @@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
>   	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
>   	/* Get vfio_res */
>   	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
> -		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
> +		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
>   			continue;
>   		break;
>   	}
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare

[Gaëtan Rivet]

Hi,

On Thu, Jul 12, 2018 at 10:24:44AM +0100, Burakov, Anatoly wrote:
> On 12-Jul-18 2:14 AM, Qi Zhang wrote:
> > When use memcmp to compare two PCI address, sizeof(struct rte_pci_addr)
> > is 4 bytes aligned, and it is 8. While only 7 byte of struct rte_pci_addr
> > is valid. So compare the 8th byte will cause the unexpected result, which
> > happens when repeatedly attach/detach a device.
> > 
> > Fixes: c752998b5e2e ("pci: introduce library and driver")

Shouldn't be the original commit be

Fixes: 94c0776b1bad ("vfio: support hotplug")

instead?

> > Cc: stable@dpdk.org
> > 
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
> >   1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
> > index aeeaa9ed8..dd25c3542 100644
> > --- a/drivers/bus/pci/linux/pci_vfio.c
> > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > @@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
> >   };
> >   EAL_REGISTER_TAILQ(rte_vfio_tailq)
> > +/* Compair two pci address */
> > +static int pci_addr_cmp(struct rte_pci_addr *addr1, struct rte_pci_addr *addr2)
> > +{
> > +	if (addr1->domain == addr2->domain &&
> > +		addr1->bus == addr2->bus &&
> > +		addr1->devid == addr2->devid &&
> > +		addr1->function == addr2->function)
> > +		return 0;
> > +	return 1;
> > +}
> 
> Generally, change looks OK to me, but I think we already have this function
> in PCI library - rte_pci_addr_cmp(). Is there a specific reason to
> reimplement it here?
> 

+1

> > +
> >   int
> >   pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
> >   		    void *buf, size_t len, off_t offs)
> > @@ -642,7 +653,7 @@ pci_vfio_unmap_resource(struct rte_pci_device *dev)
> >   	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
> >   	/* Get vfio_res */
> >   	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
> > -		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
> > +		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
> >   			continue;
> >   		break;
> >   	}
> > 
> 
> 
> -- 
> Thanks,
> Anatoly

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port in local process

[Andrew Rybchenko]

On 12.07.2018 03:23, Zhang, Qi Z wrote:
>
>> -----Original Message-----
>> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
>> Sent: Thursday, July 12, 2018 12:05 AM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
>> Anatoly <anatoly.burakov@intel.com>
>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to release port
>> in local process
>>
>> On 11.07.2018 15:30, Zhang, Qi Z wrote:
>>>> -----Original Message-----
>>>> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
>>>> Sent: Wednesday, July 11, 2018 5:27 PM
>>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Burakov,
>>>> Anatoly <anatoly.burakov@intel.com>
>>>> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
>>>> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
>>>> <ferruh.yigit@intel.com>; Shelton, Benjamin H
>>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>>>> <narender.vangati@intel.com>
>>>> Subject: Re: [dpdk-dev] [PATCH v11 01/19] ethdev: add function to
>>>> release port in local process
>>>>
>>>> On 11.07.2018 06:08, Qi Zhang wrote:
>>>>> Add driver API rte_eth_release_port_private to support the case when
>>>>> an ethdev need to be detached on a secondary process.
>>>>> Local state is set to unused and shared data will not be reset so
>>>>> the primary process can still use it.
>>>>>
>>>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>>>> Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
>>>>> Acked-by: Remy Horton <remy.horton@intel.com>
>>>>> ---
>>> <...>
>>>>> +	/**
>>>>> +	 * PCI device can only be globally detached directly by a
>>>>> +	 * primary process. In secondary process, we only need to
>>>>> +	 * release port.
>>>>> +	 */
>>>>> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
>>>>> +		return rte_eth_dev_release_port_private(eth_dev);
>>>> I've realized that some uninit functions which will not be called
>>>> anymore in secondary processes have check for process type and
>>>> handling of secondary process case. It makes code inconsistent and should
>> be fixed.
>>> Good point, I did a scan and check all the places that
>> rte_eth_dev_pci_generic_remove be involved.
>>> I found only sfc driver (sfc_eth_dev_unit) will call some cleanup on
>> secondary process as below.
>>
>> The patch makes impossible dev_uninit to be executed for secondary process
>> for all cases if rte_eth_dev_pci_generic_remove() is used. However, many
>> drivers still check for process type. Yes, sfc does cleanup, but some drivers
>> return -EPERM, some return 0. In fact it does not matter. It leaves dead code
>> which is really confusing.
> OK, l can do a cleanup in a separate patchset if this one will be merged.

For now, I'd like to revoke my Reviewed-by. I'll review once again when
complete solution is suggested.

Re: [dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare

[Zhang, Qi Z]

> -----Original Message-----
> From: Burakov, Anatoly
> Sent: Thursday, July 12, 2018 5:25 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; stable@dpdk.org
> Subject: Re: [PATCH v13 02/19] bus/pci: fix PCI address compare
> 
> On 12-Jul-18 2:14 AM, Qi Zhang wrote:
> > When use memcmp to compare two PCI address, sizeof(struct
> > rte_pci_addr) is 4 bytes aligned, and it is 8. While only 7 byte of
> > struct rte_pci_addr is valid. So compare the 8th byte will cause the
> > unexpected result, which happens when repeatedly attach/detach a device.
> >
> > Fixes: c752998b5e2e ("pci: introduce library and driver")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
> >   1 file changed, 12 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/bus/pci/linux/pci_vfio.c
> > b/drivers/bus/pci/linux/pci_vfio.c
> > index aeeaa9ed8..dd25c3542 100644
> > --- a/drivers/bus/pci/linux/pci_vfio.c
> > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > @@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
> >   };
> >   EAL_REGISTER_TAILQ(rte_vfio_tailq)
> >
> > +/* Compair two pci address */
> > +static int pci_addr_cmp(struct rte_pci_addr *addr1, struct
> > +rte_pci_addr *addr2) {
> > +	if (addr1->domain == addr2->domain &&
> > +		addr1->bus == addr2->bus &&
> > +		addr1->devid == addr2->devid &&
> > +		addr1->function == addr2->function)
> > +		return 0;
> > +	return 1;
> > +}
> 
> Generally, change looks OK to me, but I think we already have this function in
> PCI library - rte_pci_addr_cmp(). Is there a specific reason to reimplement it
> here?

NO, and rte_pci_addr_cmp is what I want :), thanks!
> 
> > +
> >   int
> >   pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
> >   		    void *buf, size_t len, off_t offs) @@ -642,7 +653,7 @@
> > pci_vfio_unmap_resource(struct rte_pci_device *dev)
> >   	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head,
> mapped_pci_res_list);
> >   	/* Get vfio_res */
> >   	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
> > -		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
> > +		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
> >   			continue;
> >   		break;
> >   	}
> >
> 
> 
> --
> Thanks,
> Anatoly

Re: [dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare

[Zhang, Qi Z]

> -----Original Message-----
> From: Gaëtan Rivet [mailto:gaetan.rivet@6wind.com]
> Sent: Thursday, July 12, 2018 5:32 PM
> To: Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org; Richardson,
> Bruce <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> Shelton, Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>; stable@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v13 02/19] bus/pci: fix PCI address compare
> 
> Hi,
> 
> On Thu, Jul 12, 2018 at 10:24:44AM +0100, Burakov, Anatoly wrote:
> > On 12-Jul-18 2:14 AM, Qi Zhang wrote:
> > > When use memcmp to compare two PCI address, sizeof(struct
> > > rte_pci_addr) is 4 bytes aligned, and it is 8. While only 7 byte of
> > > struct rte_pci_addr is valid. So compare the 8th byte will cause the
> > > unexpected result, which happens when repeatedly attach/detach a
> device.
> > >
> > > Fixes: c752998b5e2e ("pci: introduce library and driver")
> 
> Shouldn't be the original commit be
> 
> Fixes: 94c0776b1bad ("vfio: support hotplug")
> 
> instead?

You are right, this should be one that introduce the issue, thanks!

> 
> > > Cc: stable@dpdk.org
> > >
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> > >   drivers/bus/pci/linux/pci_vfio.c | 13 ++++++++++++-
> > >   1 file changed, 12 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/bus/pci/linux/pci_vfio.c
> > > b/drivers/bus/pci/linux/pci_vfio.c
> > > index aeeaa9ed8..dd25c3542 100644
> > > --- a/drivers/bus/pci/linux/pci_vfio.c
> > > +++ b/drivers/bus/pci/linux/pci_vfio.c
> > > @@ -43,6 +43,17 @@ static struct rte_tailq_elem rte_vfio_tailq = {
> > >   };
> > >   EAL_REGISTER_TAILQ(rte_vfio_tailq)
> > > +/* Compair two pci address */
> > > +static int pci_addr_cmp(struct rte_pci_addr *addr1, struct
> > > +rte_pci_addr *addr2) {
> > > +	if (addr1->domain == addr2->domain &&
> > > +		addr1->bus == addr2->bus &&
> > > +		addr1->devid == addr2->devid &&
> > > +		addr1->function == addr2->function)
> > > +		return 0;
> > > +	return 1;
> > > +}
> >
> > Generally, change looks OK to me, but I think we already have this
> > function in PCI library - rte_pci_addr_cmp(). Is there a specific
> > reason to reimplement it here?
> >
> 
> +1
> 
> > > +
> > >   int
> > >   pci_vfio_read_config(const struct rte_intr_handle *intr_handle,
> > >   		    void *buf, size_t len, off_t offs) @@ -642,7 +653,7 @@
> > > pci_vfio_unmap_resource(struct rte_pci_device *dev)
> > >   	vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head,
> mapped_pci_res_list);
> > >   	/* Get vfio_res */
> > >   	TAILQ_FOREACH(vfio_res, vfio_res_list, next) {
> > > -		if (memcmp(&vfio_res->pci_addr, &dev->addr, sizeof(dev->addr)))
> > > +		if (pci_addr_cmp(&vfio_res->pci_addr, &dev->addr))
> > >   			continue;
> > >   		break;
> > >   	}
> > >
> >
> >
> > --
> > Thanks,
> > Anatoly
> 
> --
> Gaëtan Rivet
> 6WIND

[dpdk-dev] [PATCH v14 0/6] enable hotplug on multi-process

[Qi Zhang]

v14:
- rebase.
- All changes belongs to patch 1/6.
  1) rename rte_eth_dev_release_port_private to rte_eth_dev_release_port_seondary
     since it is only used by secondary process.
  2) in rte_eth_dev_pci_generic_remove, even on the secondary process,
     I think its better to call rte_eth_dev_release_port_secondary after
     dev_uninit since it is possible that secondary process need to release
     some local resources in dev_uninit before release the port and return.
     Also this does not break all exist users of rte_eth_dev_pci_generic_remove,
     because there is no special handle in all exist dev_uninit for secondary
     process.
  3) add rte_eth_dev_release_port_secondary into rte_eth_dev_destroy as a
     general step, so we don't need patches for i40e and ixgbe.
  4) fix missing update on rte_ethdev_version.map.
- improve error handle for -EEXIST when attaching a device and -ENOENT
  when detaching a device. It is possible that device is not synced during
  some situation, so attach an exist device in primary still need to sync
  with secondary. Also, it's not necessary to rollback if we fail to
  attach an exist device or detach a not exist device on secondary.
- fix potential NULL point ref in handle_primary_request.
- merge all vdev driver patches into one patch.
- merge all pci driver patches into on patch.

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11: - move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_secondary which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this for PCI device those driver use
rte_eth_dev_pci_generic_remove or rte_eth_dev_destroy and all
vdev that support secondary process, it can be refereneced by other driver
when equevalent fix is required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (6):
  ethdev: add function to release port in secondary process
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  drivers/net: enable hotplug on secondary process
  drivers/net: enable device detach on secondary
  examples/multi_process: add hotplug sample

 drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
 drivers/net/bnxt/bnxt_ethdev.c               |   2 +-
 drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
 drivers/net/ena/ena_ethdev.c                 |   2 +-
 drivers/net/kni/rte_eth_kni.c                |   6 +-
 drivers/net/liquidio/lio_ethdev.c            |   2 +-
 drivers/net/null/rte_eth_null.c              |   6 +-
 drivers/net/octeontx/octeontx_ethdev.c       |   8 +
 drivers/net/pcap/rte_eth_pcap.c              |   6 +-
 drivers/net/tap/rte_eth_tap.c                |   8 +-
 drivers/net/vhost/rte_eth_vhost.c            |   6 +-
 drivers/net/virtio/virtio_ethdev.c           |   2 +-
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 214 ++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 ++++
 lib/librte_eal/bsdapp/eal/Makefile           |   1 +
 lib/librte_eal/common/eal_common_dev.c       | 177 +++++++++++++-
 lib/librte_eal/common/eal_private.h          |  37 +++
 lib/librte_eal/common/hotplug_mp.c           | 348 +++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h           |  48 ++++
 lib/librte_eal/common/include/rte_dev.h      |   6 +
 lib/librte_eal/common/meson.build            |   1 +
 lib/librte_eal/linuxapp/eal/Makefile         |   1 +
 lib/librte_eal/linuxapp/eal/eal.c            |   6 +
 lib/librte_ethdev/rte_ethdev.c               |  21 +-
 lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h           |   9 +-
 lib/librte_ethdev/rte_ethdev_version.map     |   7 +
 30 files changed, 997 insertions(+), 30 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v14 1/6] ethdev: add function to release port in secondary process

[Qi Zhang]

Add driver API rte_eth_release_port_secondary to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c           | 21 ++++++++++++++++++---
 lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h       |  9 +++++++--
 lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
 4 files changed, 47 insertions(+), 6 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 4c3202505..c653bf5fa 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -3532,10 +3544,13 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
 			return ret;
 	}
 
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(ethdev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(ethdev);
 
-	ethdev->data->dev_private = NULL;
+	if (ethdev->data->dev_private != NULL) {
+		rte_free(ethdev->data->dev_private);
+		ethdev->data->dev_private = NULL;
+	}
 
 	return rte_eth_dev_release_port(ethdev);
 }
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c6d9bc1a3..8fe82d2ab 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index f652596f4..c22022e4f 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -135,9 +135,14 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
 static inline void
 rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
 {
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(eth_dev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->device = NULL;
+		eth_dev->intr_handle = NULL;
+		rte_eth_dev_release_port_secondary(eth_dev);
+	}
 
+	/* primary process */
+	rte_free(eth_dev->data->dev_private);
 	eth_dev->data->dev_private = NULL;
 
 	/*
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 38f117f01..acc407f86 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -220,6 +220,13 @@ DPDK_18.08 {
 
 } DPDK_18.05;
 
+DPDK_18.11 {
+	global:
+
+	rte_eth_dev_release_port_secondary;
+
+} DPDK_18.08;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.13.6

[dpdk-dev] [PATCH v14 2/6] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 198 +++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 ++++++
 lib/librte_eal/common/hotplug_mp.c      | 185 +++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  48 ++++++++
 lib/librte_eal/common/include/rte_dev.h |   6 +
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 +
 9 files changed, 478 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index d27da3d15..4351c6a20 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 678dbcac7..d0a457ddb 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -19,8 +19,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -127,8 +129,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -193,8 +196,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -227,7 +229,193 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+
+	if (ret) {
+		/* step h) */
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+
+		/**
+		 * it is possible that secondary process failed to attached a
+		 * device that primary process have during initialization,
+		 * so for -EEXIST case, we still need to sync with secondary
+		 * process.
+		 */
+		if (ret != -EEXIST)
+			return ret;
+	}
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+
+		/* for -EEXIST, we don't need to rollback */
+		if (ret == -EEXIST)
+			return ret;
+		goto rollback;
+	}
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return -ENOMSG;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		/**
+		 * if -ENOENT, we don't need to rollback, since devices is
+		 * already detached on secondary process.
+		 */
+		if (ret != -ENOENT)
+			goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		/* if -ENOENT, we don't need to rollback */
+		if (ret == -ENOENT)
+			return ret;
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 4f809a83c..946bb01db 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -304,4 +304,41 @@ int
 rte_devargs_layers_parse(struct rte_devargs *devargs,
 			 const char *devstr);
 
+/*
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..03f7e4240
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,185 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret)
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+		return ret;
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			if (req->t == EAL_DEV_REQ_TYPE_ATTACH &&
+				req->result != -EEXIST)
+				break;
+			if (req->t == EAL_DEV_REQ_TYPE_DETACH &&
+				req->result != -ENOENT)
+				break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..22efa073e
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[EAL_DEV_MP_BUS_NAME_MAX_LEN];
+	char devname[EAL_DEV_MP_DEV_NAME_MAX_LEN];
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index b80a80598..c023bfbca 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -193,6 +193,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +215,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 56005bea8..16d4433d2 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -27,6 +27,7 @@ common_sources = files(
 	'eal_common_thread.c',
 	'eal_common_timer.c',
 	'eal_common_uuid.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index fd92c75c2..58455c1a6 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -70,6 +70,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index e59ac6577..f2c90c528 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -865,6 +865,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v14 3/6] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 188 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 183 insertions(+), 5 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 03f7e4240..debb0f720 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,169 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			if (ret != -EEXIST)
+				goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+			if (ret != -EEXIST)
+				goto rollback;
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto finish;
+		}
+
+		if (tmp_req.result) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		tmp_req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+		do_dev_hotplug_remove(req->busname, req->devname);
+	} else {
+		tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -113,8 +270,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v14 4/6] drivers/net: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_dev_release_port_secondary to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 6 ++++--
 drivers/net/bonding/rte_eth_bond_pmd.c    | 6 ++++--
 drivers/net/kni/rte_eth_kni.c             | 6 ++++--
 drivers/net/null/rte_eth_null.c           | 6 ++++--
 drivers/net/octeontx/octeontx_ethdev.c    | 8 ++++++++
 drivers/net/pcap/rte_eth_pcap.c           | 6 ++++--
 drivers/net/tap/rte_eth_tap.c             | 8 +++++---
 drivers/net/vhost/rte_eth_vhost.c         | 6 ++++--
 8 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index eb3cce3a6..80a390b19 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -927,8 +927,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -988,6 +987,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 58f7377c6..47d0f23cc 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3186,8 +3186,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3302,6 +3301,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index 085bb8452..3ce14e2b1 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -411,8 +411,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -465,6 +464,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 244f86545..ebe3e35f2 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -615,8 +615,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -681,6 +680,9 @@ rte_pmd_null_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 0f3d5d673..6d044c862 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1141,6 +1141,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_secondary(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1151,6 +1156,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index e8810a171..c601b45f0 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -1008,8 +1008,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -1108,6 +1107,9 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index feb92b48e..7794a49bb 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1993,8 +1993,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -2076,7 +2075,10 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 	/* find the ethdev entry */
 	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index e58f32211..992f53b56 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1345,8 +1345,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1437,6 +1436,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v14 5/6] drivers/net: enable device detach on secondary

[Qi Zhang]

With the enabling for hotplug on multi-process, it is not necessary
to prevent detaching a device from a secondary process.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bnxt/bnxt_ethdev.c     | 2 +-
 drivers/net/ena/ena_ethdev.c       | 2 +-
 drivers/net/liquidio/lio_ethdev.c  | 2 +-
 drivers/net/virtio/virtio_ethdev.c | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index cc7e4391c..dff9c70e2 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -3468,7 +3468,7 @@ bnxt_dev_uninit(struct rte_eth_dev *eth_dev)
 	int rc;
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	PMD_DRV_LOG(DEBUG, "Calling Device uninit\n");
 	bnxt_disable_int(bp);
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index c255dc6db..c29a581e8 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -1703,7 +1703,7 @@ static int eth_ena_dev_uninit(struct rte_eth_dev *eth_dev)
 		(struct ena_adapter *)(eth_dev->data->dev_private);
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	if (adapter->state != ENA_ADAPTER_STATE_CLOSED)
 		ena_close(eth_dev);
diff --git a/drivers/net/liquidio/lio_ethdev.c b/drivers/net/liquidio/lio_ethdev.c
index 93e89007a..0f59e4475 100644
--- a/drivers/net/liquidio/lio_ethdev.c
+++ b/drivers/net/liquidio/lio_ethdev.c
@@ -2038,7 +2038,7 @@ lio_eth_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	/* lio_free_sc_buffer_pool */
 	lio_free_sc_buffer_pool(lio_dev);
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 614357da7..0f110fde7 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1697,7 +1697,7 @@ eth_virtio_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
-		return -EPERM;
+		return 0;
 
 	virtio_dev_stop(eth_dev);
 	virtio_dev_close(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v14 6/6] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +++
 examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 +++++
 5 files changed, 289 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..b06859393
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <devargs>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	if (!rte_eal_hotplug_add(da.bus->name, da.name, da.args))
+		cmdline_printf(cl, "attached device %s\n", da.name);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+				da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	printf("detaching...\n");
+	if (!rte_eal_hotplug_remove(da.bus->name, da.name))
+		cmdline_printf(cl, "detached device %s\n",
+			da.name);
+	else
+		cmdline_printf(cl, "failed to dettach device %s\n",
+			da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+
+cmdline_parse_token_string_t cmd_dev_detach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v14 5/6] drivers/net: enable device detach on secondary

[Andrew Rybchenko]

On 10.08.2018 03:42, Qi Zhang wrote:
> With the enabling for hotplug on multi-process, it is not necessary
> to prevent detaching a device from a secondary process.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>

 From the patch itself including description it is absolutely unclear
why detach works and where it actually happens.
Why is it OK to return 0 instead of error and that's it.
Why is it necessary to call uninit at all in the case of secondary
processes?

Re: [dpdk-dev] [PATCH v14 4/6] drivers/net: enable hotplug on secondary process

[Andrew Rybchenko]

On 10.08.2018 03:42, Qi Zhang wrote:
> Attach port from secondary should ignore devargs since the private
> device is not necessary to support. Also previously, detach port on
> a secondary process will mess primary process and cause the same
> device can't be attached back again. A secondary process should use
> rte_eth_dev_release_port_secondary to release a port.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>

For me, it looks like duplication of the same code logic in
all vdev drivers. I'd say that remove should not be called
at all in the case of secondary process. Also I'd consider
to introduce separate callback for probe in the case of
secondary process: it would make it clear if secondary is
supported and enforce authors to think about secondary
process specifics on probe. As far as I can see it is always
absolutely different branch with own code.

Re: [dpdk-dev] [PATCH v14 1/6] ethdev: add function to release port in secondary process

[Andrew Rybchenko]

On 10.08.2018 03:42, Qi Zhang wrote:
> Add driver API rte_eth_release_port_secondary to support the
> case when an ethdev need to be detached on a secondary process.
> Local state is set to unused and shared data will not be reset
> so the primary process can still use it.
>
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_ethdev/rte_ethdev.c           | 21 ++++++++++++++++++---
>   lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
>   lib/librte_ethdev/rte_ethdev_pci.h       |  9 +++++++--
>   lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
>   4 files changed, 47 insertions(+), 6 deletions(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index 4c3202505..c653bf5fa 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c

<...>

> @@ -3532,10 +3544,13 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
>   			return ret;
>   	}
>   
> -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> -		rte_free(ethdev->data->dev_private);
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_secondary(ethdev);
>   
> -	ethdev->data->dev_private = NULL;
> +	if (ethdev->data->dev_private != NULL) {

The check is not necessary, rte_free() perfectly works with NULL.

> +		rte_free(ethdev->data->dev_private);
> +		ethdev->data->dev_private = NULL;
> +	}
>   
>   	return rte_eth_dev_release_port(ethdev);
>   }
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
> index c6d9bc1a3..8fe82d2ab 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
>    * Release the specified ethdev port.
>    *
>    * @param eth_dev
> - * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> + * Device to be detached.
>    * @return
>    *   - 0 on success, negative on error
>    */
> @@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
>   
>   /**
>    * @internal
> + * Release the specified ethdev port in the local process.
> + * Only set ethdev state to unused, but not reset shared data since
> + * it assume other processes is still using it. typically it is
> + * called by a secondary process.
> + *
> + * @param eth_dev
> + * Device to be detached.
> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
> +
> +/**
> + * @internal
>    * Release device queues and clear its configuration to force the user
>    * application to reconfigure it. It is for internal use only.
>    *
> diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
> index f652596f4..c22022e4f 100644
> --- a/lib/librte_ethdev/rte_ethdev_pci.h
> +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> @@ -135,9 +135,14 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
>   static inline void
>   rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
>   {@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
>   }
>   
>   int
> +rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
> +{
> +	if (eth_dev == NULL)
> +		return -EINVAL;
> +
> +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
> +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> +
> +	return 0;
> +}
> +
> +int
>   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>   {
>   	if (eth_dev == NULL)
>
>
> -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> -		rte_free(eth_dev->data->dev_private);
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		eth_dev->device = NULL;
> +		eth_dev->intr_handle = NULL;
> +		rte_eth_dev_release_port_secondary(eth_dev);

Is return missing here. Below comment pretends that it is primary 
process code.

> +	}
>   
> +	/* primary process */
> +	rte_free(eth_dev->data->dev_private);
>   	eth_dev->data->dev_private = NULL;
>   
>   	/*
> diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
> index 38f117f01..acc407f86 100644
> --- a/lib/librte_ethdev/rte_ethdev_version.map
> +++ b/lib/librte_ethdev/rte_ethdev_version.map
> @@ -220,6 +220,13 @@ DPDK_18.08 {
>   
>   } DPDK_18.05;
>   
> +DPDK_18.11 {
> +	global:
> +
> +	rte_eth_dev_release_port_secondary;
> +
> +} DPDK_18.08;
> +
>   EXPERIMENTAL {
>   	global:
>   

Re: [dpdk-dev] [PATCH v14 1/6] ethdev: add function to release port in secondary process

[Zhang, Qi Z]

Thanks for the capture, will fix in v15.

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Sunday, August 12, 2018 7:05 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net;
> gaetan.rivet@6wind.com; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v14 1/6] ethdev: add function to release port
> in secondary process
> 
> On 10.08.2018 03:42, Qi Zhang wrote:
> > Add driver API rte_eth_release_port_secondary to support the case when
> > an ethdev need to be detached on a secondary process.
> > Local state is set to unused and shared data will not be reset so the
> > primary process can still use it.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   lib/librte_ethdev/rte_ethdev.c           | 21
> ++++++++++++++++++---
> >   lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
> >   lib/librte_ethdev/rte_ethdev_pci.h       |  9 +++++++--
> >   lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
> >   4 files changed, 47 insertions(+), 6 deletions(-)
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > b/lib/librte_ethdev/rte_ethdev.c index 4c3202505..c653bf5fa 100644
> > --- a/lib/librte_ethdev/rte_ethdev.c
> > +++ b/lib/librte_ethdev/rte_ethdev.c
> 
> <...>
> 
> > @@ -3532,10 +3544,13 @@ rte_eth_dev_destroy(struct rte_eth_dev
> *ethdev,
> >   			return ret;
> >   	}
> >
> > -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > -		rte_free(ethdev->data->dev_private);
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return rte_eth_dev_release_port_secondary(ethdev);
> >
> > -	ethdev->data->dev_private = NULL;
> > +	if (ethdev->data->dev_private != NULL) {
> 
> The check is not necessary, rte_free() perfectly works with NULL.
> 
> > +		rte_free(ethdev->data->dev_private);
> > +		ethdev->data->dev_private = NULL;
> > +	}
> >
> >   	return rte_eth_dev_release_port(ethdev);
> >   }
> > diff --git a/lib/librte_ethdev/rte_ethdev_driver.h
> > b/lib/librte_ethdev/rte_ethdev_driver.h
> > index c6d9bc1a3..8fe82d2ab 100644
> > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > @@ -61,7 +61,7 @@ struct rte_eth_dev
> *rte_eth_dev_attach_secondary(const char *name);
> >    * Release the specified ethdev port.
> >    *
> >    * @param eth_dev
> > - * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > + * Device to be detached.
> >    * @return
> >    *   - 0 on success, negative on error
> >    */
> > @@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> > *eth_dev);
> >
> >   /**
> >    * @internal
> > + * Release the specified ethdev port in the local process.
> > + * Only set ethdev state to unused, but not reset shared data since
> > + * it assume other processes is still using it. typically it is
> > + * called by a secondary process.
> > + *
> > + * @param eth_dev
> > + * Device to be detached.
> > + * @return
> > + *   - 0 on success, negative on error
> > + */
> > +int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
> > +
> > +/**
> > + * @internal
> >    * Release device queues and clear its configuration to force the user
> >    * application to reconfigure it. It is for internal use only.
> >    *
> > diff --git a/lib/librte_ethdev/rte_ethdev_pci.h
> > b/lib/librte_ethdev/rte_ethdev_pci.h
> > index f652596f4..c22022e4f 100644
> > --- a/lib/librte_ethdev/rte_ethdev_pci.h
> > +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> > @@ -135,9 +135,14 @@ rte_eth_dev_pci_allocate(struct rte_pci_device
> *dev, size_t private_data_size)
> >   static inline void
> >   rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
> >   {@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char
> *name)
> >   }
> >
> >   int
> > +rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev) {
> > +	if (eth_dev == NULL)
> > +		return -EINVAL;
> > +
> > +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> NULL);
> > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> > +
> > +	return 0;
> > +}
> > +
> > +int
> >   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
> >   {
> >   	if (eth_dev == NULL)
> >
> >
> > -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > -		rte_free(eth_dev->data->dev_private);
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > +		eth_dev->device = NULL;
> > +		eth_dev->intr_handle = NULL;
> > +		rte_eth_dev_release_port_secondary(eth_dev);
> 
> Is return missing here. Below comment pretends that it is primary process
> code.
> 
> > +	}
> >
> > +	/* primary process */
> > +	rte_free(eth_dev->data->dev_private);
> >   	eth_dev->data->dev_private = NULL;
> >
> >   	/*
> > diff --git a/lib/librte_ethdev/rte_ethdev_version.map
> > b/lib/librte_ethdev/rte_ethdev_version.map
> > index 38f117f01..acc407f86 100644
> > --- a/lib/librte_ethdev/rte_ethdev_version.map
> > +++ b/lib/librte_ethdev/rte_ethdev_version.map
> > @@ -220,6 +220,13 @@ DPDK_18.08 {
> >
> >   } DPDK_18.05;
> >
> > +DPDK_18.11 {
> > +	global:
> > +
> > +	rte_eth_dev_release_port_secondary;
> > +
> > +} DPDK_18.08;
> > +
> >   EXPERIMENTAL {
> >   	global:
> >

Re: [dpdk-dev] [PATCH v14 4/6] drivers/net: enable hotplug on secondary process

[Zhang, Qi Z]

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Sunday, August 12, 2018 7:00 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net;
> gaetan.rivet@6wind.com; Burakov, Anatoly <anatoly.burakov@intel.com>;
> arybchenko@solarflare.com
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v14 4/6] drivers/net: enable hotplug on
> secondary process
> 
> On 10.08.2018 03:42, Qi Zhang wrote:
> > Attach port from secondary should ignore devargs since the private
> > device is not necessary to support. Also previously, detach port on a
> > secondary process will mess primary process and cause the same device
> > can't be attached back again. A secondary process should use
> > rte_eth_dev_release_port_secondary to release a port.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> 
> For me, it looks like duplication of the same code logic in all vdev drivers. I'd
> say that remove should not be called at all in the case of secondary process.

But based on current framework, rte_eth_dev_release_port_secondary is required to be called in PMD, and driver->remove is the place to call it as what I see.

> Also I'd consider to introduce separate callback for probe in the case of
> secondary process: it would make it clear if secondary is supported and
> enforce authors to think about secondary process specifics on probe. As far
> as I can see it is always absolutely different branch with own code.

I like this idea. We should give the driver the flexibility to decide to expose device on a secondary process or not.
And this is not for vdev only, maybe another option is adding a flag in rte_driver to indicate if it supports secondary process or not, so we don't need to add callback for all sub bus drivers separately, but in that case we still have to handle secondary in the same probe/remove function if a driver support secondary.

Btw, this looks like involve a lot of change and break ABI. Also, it exceeds the scope of hotplug. I would like to see this in a separate patchset ( better a RFC first), what do you think?

Regards
Qi

Re: [dpdk-dev] [PATCH v14 5/6] drivers/net: enable device detach on secondary

[Zhang, Qi Z]

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Sunday, August 12, 2018 6:51 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net;
> gaetan.rivet@6wind.com; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v14 5/6] drivers/net: enable device detach on
> secondary
> 
> On 10.08.2018 03:42, Qi Zhang wrote:
> > With the enabling for hotplug on multi-process, it is not necessary to
> > prevent detaching a device from a secondary process.
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> 
>  From the patch itself including description it is absolutely unclear why
> detach works and where it actually happens.

OK, I will add more detail comment.

> Why is it OK to return 0 instead of error and that's it.

Obviously , something need to fix, since uninit is not only called by driver->remove

> Why is it necessary to call uninit at all in the case of secondary processes?

Will parse NULL to rte_eth_dev_pci_generic_remove for secondary process. 

[dpdk-dev] [PATCH v15 0/7] enable hotplug on multi-process

[Qi Zhang]

v15:
- fix missing return in rte_eth_dev_pci_release.
- minor fix and more detail comments for patch 5/7.
- update release notes for v18.11.

v14:
- rebase.
- All changes belongs to patch 1/6.
  1) rename rte_eth_dev_release_port_private to rte_eth_dev_release_port_seondary
     since it is only used by secondary process.
  2) in rte_eth_dev_pci_generic_remove, even on the secondary process,
     I think its better to call rte_eth_dev_release_port_secondary after
     dev_uninit since it is possible that secondary process need to release
     some local resources in dev_uninit before release the port and return.
     Also this does not break all exist users of rte_eth_dev_pci_generic_remove,
     because there is no special handle in all exist dev_uninit for secondary
     process.
  3) add rte_eth_dev_release_port_secondary into rte_eth_dev_destroy as a
     general step, so we don't need patches for i40e and ixgbe.
  4) fix missing update on rte_ethdev_version.map.
- improve error handle for -EEXIST when attaching a device and -ENOENT
  when detaching a device. It is possible that device is not synced during
  some situation, so attach an exist device in primary still need to sync
  with secondary. Also, it's not necessary to rollback if we fail to
  attach an exist device or detach a not exist device on secondary.
- fix potential NULL point ref in handle_primary_request.
- merge all vdev driver patches into one patch.
- merge all pci driver patches into on patch.

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11: - move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_secondary which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this for PCI device those driver use
rte_eth_dev_pci_generic_remove or rte_eth_dev_destroy and all
vdev that support secondary process, it can be refereneced by other driver
when equevalent fix is required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (7):
  ethdev: add function to release port in secondary process
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  drivers/net: enable hotplug on secondary process
  drivers/net: enable device detach on secondary
  examples/multi_process: add hotplug sample
  doc: update release notes for mulit-process hotplug

 doc/guides/rel_notes/release_18_11.rst       |  11 +
 drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
 drivers/net/bnxt/bnxt_ethdev.c               |   6 +-
 drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
 drivers/net/ena/ena_ethdev.c                 |   2 +-
 drivers/net/kni/rte_eth_kni.c                |   6 +-
 drivers/net/liquidio/lio_ethdev.c            |   2 +-
 drivers/net/null/rte_eth_null.c              |   6 +-
 drivers/net/octeontx/octeontx_ethdev.c       |   8 +
 drivers/net/pcap/rte_eth_pcap.c              |   6 +-
 drivers/net/tap/rte_eth_tap.c                |   8 +-
 drivers/net/vhost/rte_eth_vhost.c            |   6 +-
 drivers/net/virtio/virtio_ethdev.c           |   2 +-
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 214 ++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile           |   1 +
 lib/librte_eal/common/eal_common_dev.c       | 198 ++++++++++++++-
 lib/librte_eal/common/eal_private.h          |  37 +++
 lib/librte_eal/common/hotplug_mp.c           | 363 +++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h           |  48 ++++
 lib/librte_eal/common/include/rte_dev.h      |   6 +
 lib/librte_eal/common/meson.build            |   1 +
 lib/librte_eal/linuxapp/eal/Makefile         |   1 +
 lib/librte_eal/linuxapp/eal/eal.c            |   6 +
 lib/librte_ethdev/rte_ethdev.c               |  17 +-
 lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h           |  10 +-
 lib/librte_ethdev/rte_ethdev_version.map     |   7 +
 31 files changed, 1046 insertions(+), 29 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v15 1/7] ethdev: add function to release port in secondary process

[Qi Zhang]

Add driver API rte_eth_release_port_secondary to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c           | 17 +++++++++++++++--
 lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h       | 10 ++++++++--
 lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
 4 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 4c3202505..1a1cc1125 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -3532,9 +3544,10 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
 			return ret;
 	}
 
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(ethdev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(ethdev);
 
+	rte_free(ethdev->data->dev_private);
 	ethdev->data->dev_private = NULL;
 
 	return rte_eth_dev_release_port(ethdev);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index c6d9bc1a3..8fe82d2ab 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index f652596f4..70d2d2503 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -135,9 +135,15 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
 static inline void
 rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
 {
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(eth_dev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->device = NULL;
+		eth_dev->intr_handle = NULL;
+		rte_eth_dev_release_port_secondary(eth_dev);
+		return;
+	}
 
+	/* primary process */
+	rte_free(eth_dev->data->dev_private);
 	eth_dev->data->dev_private = NULL;
 
 	/*
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 38f117f01..acc407f86 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -220,6 +220,13 @@ DPDK_18.08 {
 
 } DPDK_18.05;
 
+DPDK_18.11 {
+	global:
+
+	rte_eth_dev_release_port_secondary;
+
+} DPDK_18.08;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.13.6

[dpdk-dev] [PATCH v15 2/7] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 198 +++++++++++++++++++++++++++++++-
 lib/librte_eal/common/eal_private.h     |  37 ++++++
 lib/librte_eal/common/hotplug_mp.c      | 185 +++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  48 ++++++++
 lib/librte_eal/common/include/rte_dev.h |   6 +
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 +
 9 files changed, 478 insertions(+), 5 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index d27da3d15..4351c6a20 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 678dbcac7..d0a457ddb 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -19,8 +19,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -127,8 +129,9 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devname,
-			const char *devargs)
+int
+do_dev_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -193,8 +196,7 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
 	return ret;
 }
 
-int __rte_experimental
-rte_eal_hotplug_remove(const char *busname, const char *devname)
+int do_dev_hotplug_remove(const char *busname, const char *devname)
 {
 	struct rte_bus *bus;
 	struct rte_device *dev;
@@ -227,7 +229,193 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(busname, devname);
+	else
+		rte_devargs_remove(busname, devname);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/**
+	 * attach a device from primary start from here:
+	 *
+	 * a) primary attach the new device if failed goto h).
+	 * b) primary send attach sync request to all secondary.
+	 * c) secondary receive request and attach the device and send a reply.
+	 * d) primary check the reply if all success goes to i).
+	 * e) primary send attach rollback sync request to all secondary.
+	 * f) secondary receive the request and detach the device and send a
+	 *    reply.
+	 * g) primary receive the reply and detach device as rollback action.
+	 * h) attach fail
+	 * i) attach success
+	 */
+
+	/* step a) */
+	ret = do_dev_hotplug_add(busname, devname, devargs);
+
+	if (ret) {
+		/* step h) */
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+
+		/**
+		 * it is possible that secondary process failed to attached a
+		 * device that primary process have during initialization,
+		 * so for -EEXIST case, we still need to sync with secondary
+		 * process.
+		 */
+		if (ret != -EEXIST)
+			return ret;
+	}
+
+	/* step b), c) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* step d) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+
+		/* for -EEXIST, we don't need to rollback */
+		if (ret == -EEXIST)
+			return ret;
+		goto rollback;
+	}
+
+	/* step i */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+	/* step e), f) */
+	eal_dev_hotplug_request_to_secondary(&req);
+	/* step g) */
+	do_dev_hotplug_remove(busname, devname);
+	/* step h */
+	return ret;
+}
+
+int __rte_experimental
+rte_eal_hotplug_remove(const char *busname, const char *devname)
+{
+	struct eal_dev_mp_req req;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.busname, busname, EAL_DEV_MP_BUS_NAME_MAX_LEN);
+	strlcpy(req.devname, devname, EAL_DEV_MP_DEV_NAME_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/**
+	 * detach a device from primary start from here:
+	 *
+	 * a) primary send detach sync request to all secondary
+	 * b) secondary detach the device and send reply
+	 * c) primary check the reply if all success goes to f).
+	 * d) primary send detach rollback sync request to all secondary.
+	 * e) secondary receive the request and attach back device. goto g)
+	 * f) primary detach the device if success goto g), else goto d)
+	 * g) detach fail.
+	 * h) detach success.
+	 */
+
+	/* step a), b) */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+	/* step c) */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		return -ENOMSG;
+	}
+
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		/**
+		 * if -ENOENT, we don't need to rollback, since devices is
+		 * already detached on secondary process.
+		 */
+		if (ret != -ENOENT)
+			goto rollback;
+	}
+
+	/* step f) */
+	ret = do_dev_hotplug_remove(busname, devname);
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		/* if -ENOENT, we don't need to rollback */
+		if (ret == -ENOENT)
+			return ret;
+		goto rollback;
+	}
+	/* step h */
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+	/* step d), e) */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(ERR, EAL,
+			"Failed to attach back device on secondary."
+			"Devices in secondary may not sync with primary\n");
+	/* step g) */
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 4f809a83c..946bb01db 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -304,4 +304,41 @@ int
 rte_devargs_layers_parse(struct rte_devargs *devargs,
 			 const char *devstr);
 
+/*
+ * Hotplug add a given device to a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is added to.
+ * @param devname
+ *   The device name. Based on this device name, eal will identify a driver
+ *   capable of handling it and pass it to the driver probing function.
+ * @param devargs
+ *   Device arguments to be passed to the driver.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_add(const char *busname, const char *devname,
+			const char *devargs);
+
+/**
+ * Hotplug remove a given device from a specific bus.
+ *
+ * @param busname
+ *   The bus name the device is removed from.
+ * @param devname
+ *   The device name being removed.
+ * @return
+ *   0 on success, negative on error.
+ */
+int do_dev_hotplug_remove(const char *busname,
+			const char *devname);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..03f7e4240
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,185 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = do_dev_hotplug_add(req->busname, req->devname, "");
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret)
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+		return ret;
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			if (req->t == EAL_DEV_REQ_TYPE_ATTACH &&
+				req->result != -EEXIST)
+				break;
+			if (req->t == EAL_DEV_REQ_TYPE_DETACH &&
+				req->result != -ENOENT)
+				break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..22efa073e
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char busname[EAL_DEV_MP_BUS_NAME_MAX_LEN];
+	char devname[EAL_DEV_MP_DEV_NAME_MAX_LEN];
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index b80a80598..c023bfbca 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -193,6 +193,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -212,6 +215,9 @@ int __rte_experimental rte_eal_hotplug_add(const char *busname, const char *devn
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index 56005bea8..16d4433d2 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -27,6 +27,7 @@ common_sources = files(
 	'eal_common_thread.c',
 	'eal_common_timer.c',
 	'eal_common_uuid.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index fd92c75c2..58455c1a6 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -70,6 +70,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index e59ac6577..f2c90c528 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -865,6 +865,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v15 3/7] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 188 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 183 insertions(+), 5 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 03f7e4240..debb0f720 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -17,12 +17,169 @@ struct mp_reply_bundle {
 	void *peer;
 };
 
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = do_dev_hotplug_add(req->busname, req->devname,
+					req->devargs);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			if (ret != -EEXIST)
+				goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+			if (ret != -EEXIST)
+				goto rollback;
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto finish;
+		}
+
+		if (tmp_req.result) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+
+		ret = do_dev_hotplug_remove(req->busname, req->devname);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		tmp_req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+		do_dev_hotplug_remove(req->busname, req->devname);
+	} else {
+		tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -113,8 +270,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v15 4/7] drivers/net: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_dev_release_port_secondary to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 6 ++++--
 drivers/net/bonding/rte_eth_bond_pmd.c    | 6 ++++--
 drivers/net/kni/rte_eth_kni.c             | 6 ++++--
 drivers/net/null/rte_eth_null.c           | 6 ++++--
 drivers/net/octeontx/octeontx_ethdev.c    | 8 ++++++++
 drivers/net/pcap/rte_eth_pcap.c           | 6 ++++--
 drivers/net/tap/rte_eth_tap.c             | 8 +++++---
 drivers/net/vhost/rte_eth_vhost.c         | 6 ++++--
 8 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index eb3cce3a6..80a390b19 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -927,8 +927,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -988,6 +987,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 58f7377c6..47d0f23cc 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3186,8 +3186,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3302,6 +3301,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index 085bb8452..3ce14e2b1 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -411,8 +411,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -465,6 +464,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index 244f86545..ebe3e35f2 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -615,8 +615,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -681,6 +680,9 @@ rte_pmd_null_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 0f3d5d673..6d044c862 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1141,6 +1141,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_secondary(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1151,6 +1156,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index e8810a171..c601b45f0 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -1008,8 +1008,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -1108,6 +1107,9 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index feb92b48e..7794a49bb 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1993,8 +1993,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -2076,7 +2075,10 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 	/* find the ethdev entry */
 	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index e58f32211..992f53b56 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1345,8 +1345,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1437,6 +1436,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v15 5/7] drivers/net: enable device detach on secondary

[Qi Zhang]

With the enabling for hotplug on multi-process,
rte_eth_dev_pci_generic_remove can be used to detach the device from
a secondary process also. But we need to take care of the uninit callback
parameter to make sure it handles the secondary case correctly.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bnxt/bnxt_ethdev.c     | 6 +++++-
 drivers/net/ena/ena_ethdev.c       | 2 +-
 drivers/net/liquidio/lio_ethdev.c  | 2 +-
 drivers/net/virtio/virtio_ethdev.c | 2 +-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index cc7e4391c..c092bdb06 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -3515,7 +3515,11 @@ static int bnxt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int bnxt_pci_remove(struct rte_pci_device *pci_dev)
 {
-	return rte_eth_dev_pci_generic_remove(pci_dev, bnxt_dev_uninit);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return rte_eth_dev_pci_generic_remove(pci_dev,
+				bnxt_dev_uninit);
+	else
+		return rte_eth_dev_pci_generic_remove(pci_dev, NULL);
 }
 
 static struct rte_pci_driver bnxt_rte_pmd = {
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index c255dc6db..c29a581e8 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -1703,7 +1703,7 @@ static int eth_ena_dev_uninit(struct rte_eth_dev *eth_dev)
 		(struct ena_adapter *)(eth_dev->data->dev_private);
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	if (adapter->state != ENA_ADAPTER_STATE_CLOSED)
 		ena_close(eth_dev);
diff --git a/drivers/net/liquidio/lio_ethdev.c b/drivers/net/liquidio/lio_ethdev.c
index 93e89007a..0f59e4475 100644
--- a/drivers/net/liquidio/lio_ethdev.c
+++ b/drivers/net/liquidio/lio_ethdev.c
@@ -2038,7 +2038,7 @@ lio_eth_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	/* lio_free_sc_buffer_pool */
 	lio_free_sc_buffer_pool(lio_dev);
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 614357da7..0f110fde7 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1697,7 +1697,7 @@ eth_virtio_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
-		return -EPERM;
+		return 0;
 
 	virtio_dev_stop(eth_dev);
 	virtio_dev_close(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v15 6/7] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +++
 examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 +++++
 5 files changed, 289 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..b06859393
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <devargs>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	if (!rte_eal_hotplug_add(da.bus->name, da.name, da.args))
+		cmdline_printf(cl, "attached device %s\n", da.name);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+				da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	printf("detaching...\n");
+	if (!rte_eal_hotplug_remove(da.bus->name, da.name))
+		cmdline_printf(cl, "detached device %s\n",
+			da.name);
+	else
+		cmdline_printf(cl, "failed to dettach device %s\n",
+			da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+
+cmdline_parse_token_string_t cmd_dev_detach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

[dpdk-dev] [PATCH v15 7/7] doc: update release notes for mulit-process hotplug

[Qi Zhang]

Update release notes for the new multi-process hotplug feature.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 doc/guides/rel_notes/release_18_11.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index 3ae6b3f58..f08793c17 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -54,6 +54,12 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization between
+  processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -68,6 +74,11 @@ API Changes
    Also, make sure to start the actual text at the margin.
    =========================================================
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
-- 
2.13.6

Re: [dpdk-dev] [PATCH v15 1/7] ethdev: add function to release port in secondary process

[Andrew Rybchenko]

On 16.08.2018 06:04, Qi Zhang wrote:
> Add driver API rte_eth_release_port_secondary to support the
> case when an ethdev need to be detached on a secondary process.
> Local state is set to unused and shared data will not be reset
> so the primary process can still use it.

There are few questions below, but in general I'm really puzzled
after looking at variety of release, destroy etc functions and how
call chains look like.

IMHO, introduction of the function is really wrong direction.
It duplicates part of rte_eth_dev_release_port() functionality,
it will complicate maintenance since it will be required to remember
to find and update both. Also it looks like it already has bugs
(missing init of shared data, missing lock).

I would prefer to update rte_eth_dev_release_port() to make it
secondary process aware.

> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>   lib/librte_ethdev/rte_ethdev.c           | 17 +++++++++++++++--
>   lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
>   lib/librte_ethdev/rte_ethdev_pci.h       | 10 ++++++++--
>   lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
>   4 files changed, 45 insertions(+), 5 deletions(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index 4c3202505..1a1cc1125 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
>   }
>   
>   int
> +rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
> +{
> +	if (eth_dev == NULL)
> +		return -EINVAL;
> +
> +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
> +	eth_dev->state = RTE_ETH_DEV_UNUSED;

rte_eth_dev_release_port() does it under ownership lock.
Why is lock not required here?

> +
> +	return 0;
> +}
> +
> +int
>   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
>   {
>   	if (eth_dev == NULL)
> @@ -3532,9 +3544,10 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
>   			return ret;
>   	}
>   
> -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> -		rte_free(ethdev->data->dev_private);
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> +		return rte_eth_dev_release_port_secondary(ethdev);
>   
> +	rte_free(ethdev->data->dev_private);
>   	ethdev->data->dev_private = NULL;
>   
>   	return rte_eth_dev_release_port(ethdev);
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
> index c6d9bc1a3..8fe82d2ab 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
>    * Release the specified ethdev port.
>    *
>    * @param eth_dev
> - * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> + * Device to be detached.
>    * @return
>    *   - 0 on success, negative on error
>    */
> @@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
>   
>   /**
>    * @internal
> + * Release the specified ethdev port in the local process.
> + * Only set ethdev state to unused, but not reset shared data since
> + * it assume other processes is still using it. typically it is
> + * called by a secondary process.
> + *
> + * @param eth_dev
> + * Device to be detached.
> + * @return
> + *   - 0 on success, negative on error
> + */
> +int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
> +
> +/**
> + * @internal
>    * Release device queues and clear its configuration to force the user
>    * application to reconfigure it. It is for internal use only.
>    *
> diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
> index f652596f4..70d2d2503 100644
> --- a/lib/librte_ethdev/rte_ethdev_pci.h
> +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> @@ -135,9 +135,15 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
>   static inline void
>   rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
>   {
> -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> -		rte_free(eth_dev->data->dev_private);
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		eth_dev->device = NULL;
> +		eth_dev->intr_handle = NULL;

Why are above two assignments done here in the PCI device release,
but not included in rte_eth_dev_release_port_secondary()?

> +		rte_eth_dev_release_port_secondary(eth_dev);
> +		return;
> +	}
>   
> +	/* primary process */
> +	rte_free(eth_dev->data->dev_private);
>   	eth_dev->data->dev_private = NULL;
>   
>   	/*
> diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
> index 38f117f01..acc407f86 100644
> --- a/lib/librte_ethdev/rte_ethdev_version.map
> +++ b/lib/librte_ethdev/rte_ethdev_version.map
> @@ -220,6 +220,13 @@ DPDK_18.08 {
>   
>   } DPDK_18.05;
>   
> +DPDK_18.11 {
> +	global:
> +
> +	rte_eth_dev_release_port_secondary;
> +
> +} DPDK_18.08;

Shouldn't it be experimental?

> +
>   EXPERIMENTAL {
>   	global:
>   

Re: [dpdk-dev] [PATCH v15 1/7] ethdev: add function to release port in secondary process

[Zhang, Qi Z]

> -----Original Message-----
> From: Andrew Rybchenko [mailto:arybchenko@solarflare.com]
> Sent: Monday, August 20, 2018 4:53 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net;
> gaetan.rivet@6wind.com; Burakov, Anatoly <anatoly.burakov@intel.com>
> Cc: Ananyev, Konstantin <konstantin.ananyev@intel.com>; dev@dpdk.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Shelton, Benjamin H
> <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [PATCH v15 1/7] ethdev: add function to release port in
> secondary process
> 
> On 16.08.2018 06:04, Qi Zhang wrote:
> > Add driver API rte_eth_release_port_secondary to support the case when
> > an ethdev need to be detached on a secondary process.
> > Local state is set to unused and shared data will not be reset so the
> > primary process can still use it.
> 
> There are few questions below, but in general I'm really puzzled after looking
> at variety of release, destroy etc functions and how call chains look like.
> 
> IMHO, introduction of the function is really wrong direction.
> It duplicates part of rte_eth_dev_release_port() functionality, it will
> complicate maintenance since it will be required to remember to find and
> update both. Also it looks like it already has bugs (missing init of shared data,
> missing lock).
> 
> I would prefer to update rte_eth_dev_release_port() to make it secondary
> process aware.

Well, this is on purpose to pair with rte_eth_dev_attach_secondary which is also a dedicate function for
secondary process. So we have 
in driver->probe: use rte_eth_dev_attach_secondary to attach an already registered port by primary to secondary
In driver->remove: use rte_eth_dev_release_port_secondary to detach the port from secondary and the input parameter is exactly the return value of previous.)

> 
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >   lib/librte_ethdev/rte_ethdev.c           | 17 +++++++++++++++--
> >   lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
> >   lib/librte_ethdev/rte_ethdev_pci.h       | 10 ++++++++--
> >   lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
> >   4 files changed, 45 insertions(+), 5 deletions(-)
> >
> > diff --git a/lib/librte_ethdev/rte_ethdev.c
> > b/lib/librte_ethdev/rte_ethdev.c index 4c3202505..1a1cc1125 100644
> > --- a/lib/librte_ethdev/rte_ethdev.c
> > +++ b/lib/librte_ethdev/rte_ethdev.c
> > @@ -359,6 +359,18 @@ rte_eth_dev_attach_secondary(const char *name)
> >   }
> >
> >   int
> > +rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev) {
> > +	if (eth_dev == NULL)
> > +		return -EINVAL;
> > +
> > +	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY,
> NULL);
> > +	eth_dev->state = RTE_ETH_DEV_UNUSED;
> 
> rte_eth_dev_release_port() does it under ownership lock.

Yes, because on primary process, it is possible that another thread is going to attach a new port which will also access the dev state and shared data.

> Why is lock not required here?

It's not necessary, since for secondary process, we only attach a port already be registered on primary, there will be no concurrent issue.


> 
> > +
> > +	return 0;
> > +}
> > +
> > +int
> >   rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
> >   {
> >   	if (eth_dev == NULL)
> > @@ -3532,9 +3544,10 @@ rte_eth_dev_destroy(struct rte_eth_dev
> *ethdev,
> >   			return ret;
> >   	}
> >
> > -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > -		rte_free(ethdev->data->dev_private);
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
> > +		return rte_eth_dev_release_port_secondary(ethdev);
> >
> > +	rte_free(ethdev->data->dev_private);
> >   	ethdev->data->dev_private = NULL;
> >
> >   	return rte_eth_dev_release_port(ethdev); diff --git
> > a/lib/librte_ethdev/rte_ethdev_driver.h
> > b/lib/librte_ethdev/rte_ethdev_driver.h
> > index c6d9bc1a3..8fe82d2ab 100644
> > --- a/lib/librte_ethdev/rte_ethdev_driver.h
> > +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> > @@ -61,7 +61,7 @@ struct rte_eth_dev
> *rte_eth_dev_attach_secondary(const char *name);
> >    * Release the specified ethdev port.
> >    *
> >    * @param eth_dev
> > - * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
> > + * Device to be detached.
> >    * @return
> >    *   - 0 on success, negative on error
> >    */
> > @@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev
> > *eth_dev);
> >
> >   /**
> >    * @internal
> > + * Release the specified ethdev port in the local process.
> > + * Only set ethdev state to unused, but not reset shared data since
> > + * it assume other processes is still using it. typically it is
> > + * called by a secondary process.
> > + *
> > + * @param eth_dev
> > + * Device to be detached.
> > + * @return
> > + *   - 0 on success, negative on error
> > + */
> > +int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
> > +
> > +/**
> > + * @internal
> >    * Release device queues and clear its configuration to force the user
> >    * application to reconfigure it. It is for internal use only.
> >    *
> > diff --git a/lib/librte_ethdev/rte_ethdev_pci.h
> > b/lib/librte_ethdev/rte_ethdev_pci.h
> > index f652596f4..70d2d2503 100644
> > --- a/lib/librte_ethdev/rte_ethdev_pci.h
> > +++ b/lib/librte_ethdev/rte_ethdev_pci.h
> > @@ -135,9 +135,15 @@ rte_eth_dev_pci_allocate(struct rte_pci_device
> *dev, size_t private_data_size)
> >   static inline void
> >   rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
> >   {
> > -	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > -		rte_free(eth_dev->data->dev_private);
> > +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> > +		eth_dev->device = NULL;
> > +		eth_dev->intr_handle = NULL;
> 
> Why are above two assignments done here in the PCI device release, but not
> included in rte_eth_dev_release_port_secondary()?

Since they are not in rte_eth_dev_release_port, so I keep them out of rte_eth_dev_release_port_secondary simply.
Probably there could be some consolidation, but should not for rte_eth_dev_release_port_secondary only from my view.


> 
> > +		rte_eth_dev_release_port_secondary(eth_dev);
> > +		return;
> > +	}
> >
> > +	/* primary process */
> > +	rte_free(eth_dev->data->dev_private);
> >   	eth_dev->data->dev_private = NULL;
> >
> >   	/*
> > diff --git a/lib/librte_ethdev/rte_ethdev_version.map
> > b/lib/librte_ethdev/rte_ethdev_version.map
> > index 38f117f01..acc407f86 100644
> > --- a/lib/librte_ethdev/rte_ethdev_version.map
> > +++ b/lib/librte_ethdev/rte_ethdev_version.map
> > @@ -220,6 +220,13 @@ DPDK_18.08 {
> >
> >   } DPDK_18.05;
> >
> > +DPDK_18.11 {
> > +	global:
> > +
> > +	rte_eth_dev_release_port_secondary;
> > +
> > +} DPDK_18.08;
> 
> Shouldn't it be experimental?

This API is a help function for PMD only which is defined in rte_ethdev_driver.h, it is not for external use.

Regards
Qi

[dpdk-dev] [PATCH v16 0/6] enable hotplug on multi-process

[Qi Zhang]

v16:
- rebase to patch "simplify parameters of hotplug functions"
  http://patchwork.dpdk.org/patch/45463/ include:
  * keep rte_eal_hotplug_add/rte_eal_hotplug_move unchanged.
  * the IPC sync logic is moved to rte_dev_probe/rte_dev_remove. 
  * simplify the IPC message by removing busname and devname from
    eal_dev_mp_req, since devargs string will encode those information
    already.
- combined release notes with related code changes.
- replace do_ prefix to local_ for local process only probe/remove function.
- improve comments 

v15:
- fix missing return in rte_eth_dev_pci_release.
- minor fix and more detail comments for patch 5/7.
- update release notes for v18.11.

v14:
- rebase.
- All changes belongs to patch 1/6.
  1) rename rte_eth_dev_release_port_private to rte_eth_dev_release_port_seondary
     since it is only used by secondary process.
  2) in rte_eth_dev_pci_generic_remove, even on the secondary process,
     I think its better to call rte_eth_dev_release_port_secondary after
     dev_uninit since it is possible that secondary process need to release
     some local resources in dev_uninit before release the port and return.
     Also this does not break all exist users of rte_eth_dev_pci_generic_remove,
     because there is no special handle in all exist dev_uninit for secondary
     process.
  3) add rte_eth_dev_release_port_secondary into rte_eth_dev_destroy as a
     general step, so we don't need patches for i40e and ixgbe.
  4) fix missing update on rte_ethdev_version.map.
- improve error handle for -EEXIST when attaching a device and -ENOENT
  when detaching a device. It is possible that device is not synced during
  some situation, so attach an exist device in primary still need to sync
  with secondary. Also, it's not necessary to rollback if we fail to
  attach an exist device or detach a not exist device on secondary.
- fix potential NULL point ref in handle_primary_request.
- merge all vdev driver patches into one patch.
- merge all pci driver patches into on patch.

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11: - move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_secondary which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this for PCI device those driver use
rte_eth_dev_pci_generic_remove or rte_eth_dev_destroy and all
vdev that support secondary process, it can be refereneced by other driver
when equevalent fix is required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (6):
  ethdev: add function to release port in secondary process
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  drivers/net: enable hotplug on secondary process
  drivers/net: enable device detach on secondary
  examples/multi_process: add hotplug sample

 doc/guides/rel_notes/release_18_11.rst       |  11 +
 drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
 drivers/net/bnxt/bnxt_ethdev.c               |   6 +-
 drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
 drivers/net/ena/ena_ethdev.c                 |   2 +-
 drivers/net/kni/rte_eth_kni.c                |   6 +-
 drivers/net/liquidio/lio_ethdev.c            |   2 +-
 drivers/net/null/rte_eth_null.c              |   6 +-
 drivers/net/octeontx/octeontx_ethdev.c       |   8 +
 drivers/net/pcap/rte_eth_pcap.c              |   6 +-
 drivers/net/tap/rte_eth_tap.c                |   8 +-
 drivers/net/vhost/rte_eth_vhost.c            |   6 +-
 drivers/net/virtio/virtio_ethdev.c           |   2 +-
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 214 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile           |   1 +
 lib/librte_eal/common/eal_common_dev.c       | 225 +++++++++++++-
 lib/librte_eal/common/eal_private.h          |  30 ++
 lib/librte_eal/common/hotplug_mp.c           | 426 +++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h           |  46 +++
 lib/librte_eal/common/include/rte_dev.h      |   9 +
 lib/librte_eal/common/meson.build            |   1 +
 lib/librte_eal/linuxapp/eal/Makefile         |   1 +
 lib/librte_eal/linuxapp/eal/eal.c            |   6 +
 lib/librte_ethdev/rte_ethdev.c               |  17 +-
 lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h           |  10 +-
 lib/librte_ethdev/rte_ethdev_version.map     |   7 +
 31 files changed, 1126 insertions(+), 33 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v16 1/6] ethdev: add function to release port in secondary process

[Qi Zhang]

Add driver API rte_eth_release_port_secondary to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c           | 17 +++++++++++++++--
 lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h       | 10 ++++++++--
 lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
 4 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index aa7730ce2..c58b1c6ed 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -360,6 +360,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -3540,9 +3552,10 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
 			return ret;
 	}
 
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(ethdev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(ethdev);
 
+	rte_free(ethdev->data->dev_private);
 	ethdev->data->dev_private = NULL;
 
 	return rte_eth_dev_release_port(ethdev);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index f158462a0..ca31b5777 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index f652596f4..70d2d2503 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -135,9 +135,15 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
 static inline void
 rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
 {
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(eth_dev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->device = NULL;
+		eth_dev->intr_handle = NULL;
+		rte_eth_dev_release_port_secondary(eth_dev);
+		return;
+	}
 
+	/* primary process */
+	rte_free(eth_dev->data->dev_private);
 	eth_dev->data->dev_private = NULL;
 
 	/*
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 38f117f01..acc407f86 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -220,6 +220,13 @@ DPDK_18.08 {
 
 } DPDK_18.05;
 
+DPDK_18.11 {
+	global:
+
+	rte_eth_dev_release_port_secondary;
+
+} DPDK_18.08;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.13.6

[dpdk-dev] [PATCH v16 2/6] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 doc/guides/rel_notes/release_18_11.rst  |  11 ++
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 225 ++++++++++++++++++++++++++++++--
 lib/librte_eal/common/eal_private.h     |  30 +++++
 lib/librte_eal/common/hotplug_mp.c      | 221 +++++++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  46 +++++++
 lib/librte_eal/common/include/rte_dev.h |   9 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 +
 10 files changed, 542 insertions(+), 9 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..f88910c7f 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,12 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization
+  between processes will be done using DPDK IPC.
 
 API Changes
 -----------
@@ -91,6 +97,11 @@ API Changes
   flag the MAC can be properly configured in any case. This is particularly
   important for bonding.
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
 
 ABI Changes
 -----------
diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index d27da3d15..4351c6a20 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 85eb1569f..314266041 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -19,8 +19,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -127,9 +129,10 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int
-rte_eal_hotplug_add(const char *busname, const char *devname,
-                    const char *drvargs)
+/* help funciton to build devargs, caller should free the memory */
+static char *
+build_devargs(const char *busname, const char *devname,
+	      const char *drvargs)
 {
 	char *devargs = NULL;
 	int size, length = -1;
@@ -140,19 +143,33 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
 		if (length >= size)
 			devargs = malloc(length + 1);
 		if (devargs == NULL)
-			return -ENOMEM;
+			break;
 	} while (size == 0);
 
+	return devargs;
+}
+
+int
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		    const char *drvargs)
+{
+	char *devargs = build_devargs(busname, devname, drvargs);
+
+	if (devargs == NULL)
+		return -ENOMEM;
+
 	return rte_dev_probe(devargs);
 }
 
-int __rte_experimental
-rte_dev_probe(const char *devargs)
+/* probe device at local process. */
+int
+local_dev_probe(const char *devargs, struct rte_device **new_dev)
 {
 	struct rte_device *dev;
 	struct rte_devargs *da;
 	int ret;
 
+	*new_dev = NULL;
 	da = calloc(1, sizeof(*da));
 	if (da == NULL)
 		return -ENOMEM;
@@ -195,6 +212,8 @@ rte_dev_probe(const char *devargs)
 			dev->name);
 		goto err_devarg;
 	}
+
+	*new_dev = dev;
 	return 0;
 
 err_devarg:
@@ -226,8 +245,9 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	return rte_dev_remove(dev);
 }
 
-int __rte_experimental
-rte_dev_remove(struct rte_device *dev)
+/* remove device at local process. */
+int
+local_dev_remove(struct rte_device *dev)
 {
 	struct rte_bus *bus;
 	int ret;
@@ -248,7 +268,194 @@ rte_dev_remove(struct rte_device *dev)
 	if (ret)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(dev->devargs);
+	else
+		rte_devargs_remove(dev->devargs);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_dev_probe(const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	struct rte_device *dev;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/* attach a shared device from primary start from here: */
+
+	/* primary attach the new device itself. */
+	ret = local_dev_probe(devargs, &dev);
+
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+
+		/**
+		 * it is possible that secondary process failed to attached a
+		 * device that primary process have during initialization,
+		 * so for -EEXIST case, we still need to sync with secondary
+		 * process.
+		 */
+		if (ret != -EEXIST)
+			return ret;
+	}
+
+	/* primary send attach sync request to secondary. */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* if any commnunication error, we need to rollback. */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	/**
+	 * if any secondary failed to attach, we need to consider if rollback
+	 * is necessary.
+	 */
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+
+		/* for -EEXIST, we don't need to rollback. */
+		if (ret == -EEXIST)
+			return ret;
+		goto rollback;
+	}
+
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+
+	/* primary send rollback request to secondary. */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device attach on secondary."
+			"Devices in secondary may not sync with primary\n");
+
+	/* primary rollback itself. */
+	if (local_dev_remove(dev))
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device attach on primary."
+			"Devices in secondary may not sync with primary\n");
+
+	return ret;
+}
+
+int __rte_experimental
+rte_dev_remove(struct rte_device *dev)
+{
+	struct eal_dev_mp_req req;
+	char *devargs;
+	int ret;
+
+	devargs = build_devargs(dev->devargs->bus->name, dev->name, "");
+	if (devargs == NULL)
+		return -ENOMEM;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+	free(devargs);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/* detach a device from primary start from here: */
+
+	/* primary send detach sync request to secondary */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/**
+	 * if communication error, we need to rollback, because it is possible
+	 * part of the secondary processes still detached it successfully.
+	 */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	/**
+	 * if any secondary failed to detach, we need to consider if rollback
+	 * is necessary.
+	 */
+	if (req.result) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		/**
+		 * if -ENOENT, we don't need to rollback, since devices is
+		 * already detached on secondary process.
+		 */
+		if (ret != -ENOENT)
+			goto rollback;
+	}
+
+	/* primary detach the device itself. */
+	ret = local_dev_remove(dev);
+
+	/* if primary failed, still need to consider if rollback is necessary */
+	if (ret) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		/* if -ENOENT, we don't need to rollback */
+		if (ret == -ENOENT)
+			return ret;
+		goto rollback;
+	}
+
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+
+	/* primary send rollback request to secondary. */
+	if (eal_dev_hotplug_request_to_secondary(&req))
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device detach on secondary."
+			"Devices in secondary may not sync with primary\n");
+
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 4f809a83c..83f10a9f8 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -304,4 +304,34 @@ int
 rte_devargs_layers_parse(struct rte_devargs *devargs,
 			 const char *devstr);
 
+/*
+ * probe a device at local process.
+ *
+ * @param devargs
+ *   Device arguments including bus, class and driver properties.
+ * @param new_dev
+ *   new device be probed as output.
+ * @return
+ *   0 on success, negative on error.
+ */
+int local_dev_probe(const char *devargs, struct rte_device **new_dev);
+
+/**
+ * Hotplug remove a given device from a specific bus at local process.
+ *
+ * @param dev
+ *   Data structure of the device to remove.
+ * @return
+ *   0 on success, negative on error.
+ */
+int local_dev_remove(struct rte_device *dev);
+
+/**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int rte_dev_hotplug_mp_init(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..1c92e44cb
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+#include <rte_devargs.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+static int cmp_dev_name(const struct rte_device *dev, const void *_name)
+{
+	const char *name = _name;
+
+	return strcmp(dev->name, name);
+}
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct rte_devargs *da;
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = local_dev_probe(req->devargs, &dev);
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		da = calloc(1, sizeof(*da));
+		if (da == NULL) {
+			ret = -ENOMEM;
+			goto quit;
+		}
+
+		ret = rte_devargs_parse(da, req->devargs);
+		if (ret)
+			goto quit;
+
+		bus = rte_bus_find_by_name(da->bus->name);
+		if (bus == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", da->bus->name);
+			ret = -ENOENT;
+			goto quit;
+		}
+
+		dev = bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find plugged device (%s)\n", da->name);
+			ret = -ENOENT;
+			goto quit;
+		}
+
+		ret = local_dev_remove(dev);
+quit:
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret)
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+		return ret;
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result) {
+			req->result = resp->result;
+			if (req->t == EAL_DEV_REQ_TYPE_ATTACH &&
+				req->result != -EEXIST)
+				break;
+			if (req->t == EAL_DEV_REQ_TYPE_DETACH &&
+				req->result != -ENOENT)
+				break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_dev_hotplug_mp_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..c95c8f1fb
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include <rte_dev.h>
+#include <rte_bus.h>
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * this is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 7a30362c0..266331acd 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -190,6 +190,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
 
 /**
  * Hotplug add a given device to a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug add the same device. Any failure on other process rollback
+ * the action.
  *
  * @param busname
  *   The bus name the device is added to.
@@ -219,6 +222,9 @@ int __rte_experimental rte_dev_probe(const char *devargs);
 
 /**
  * Hotplug remove a given device from a specific bus.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param busname
  *   The bus name the device is removed from.
@@ -234,6 +240,9 @@ int rte_eal_hotplug_remove(const char *busname, const char *devname);
  * @b EXPERIMENTAL: this API may change without prior notice
  *
  * Remove one device.
+ * In multi-process, this function will inform all other processes
+ * to hotplug remove the same device. Any failure on other process
+ * will rollback the action.
  *
  * @param dev
  *   Data structure of the device to remove.
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index b7fc98499..04c414356 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -28,6 +28,7 @@ common_sources = files(
 	'eal_common_thread.c',
 	'eal_common_timer.c',
 	'eal_common_uuid.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index fd92c75c2..58455c1a6 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -70,6 +70,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index e59ac6577..f2c90c528 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -865,6 +865,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register mp action callbacks for hotplug */
+	if (rte_dev_hotplug_mp_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v16 3/6] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 223 +++++++++++++++++++++++++++++++++++--
 1 file changed, 214 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 1c92e44cb..c31316bf7 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -13,6 +13,11 @@
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
 static int cmp_dev_name(const struct rte_device *dev, const void *_name)
 {
 	const char *name = _name;
@@ -20,17 +25,196 @@ static int cmp_dev_name(const struct rte_device *dev, const void *_name)
 	return strcmp(dev->name, name);
 }
 
-struct mp_reply_bundle {
-	struct rte_mp_msg msg;
-	void *peer;
-};
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	struct rte_devargs *da;
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = local_dev_probe(req->devargs, &dev);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			if (ret != -EEXIST)
+				goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+		if (tmp_req.result) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+			if (ret != -EEXIST)
+				goto rollback;
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		da = calloc(1, sizeof(*da));
+		if (da == NULL) {
+			ret = -ENOMEM;
+			goto finish;
+		}
+
+		ret = rte_devargs_parse(da, req->devargs);
+		if (ret)
+			goto finish;
+
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+
+		bus = rte_bus_find_by_name(da->bus->name);
+		if (bus == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", da->bus->name);
+			ret = -ENOENT;
+			goto finish;
+		}
+
+		dev = bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find plugged device (%s)\n", da->name);
+			ret = -ENOENT;
+			goto finish;
+		}
+
+		if (tmp_req.result) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+
+		ret = local_dev_remove(dev);
+		if (ret) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		tmp_req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+		local_dev_remove(dev);
+	} else {
+		tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
 
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -149,8 +333,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v16 4/6] drivers/net: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_dev_release_port_secondary to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 6 ++++--
 drivers/net/bonding/rte_eth_bond_pmd.c    | 6 ++++--
 drivers/net/kni/rte_eth_kni.c             | 6 ++++--
 drivers/net/null/rte_eth_null.c           | 6 ++++--
 drivers/net/octeontx/octeontx_ethdev.c    | 8 ++++++++
 drivers/net/pcap/rte_eth_pcap.c           | 6 ++++--
 drivers/net/tap/rte_eth_tap.c             | 8 +++++---
 drivers/net/vhost/rte_eth_vhost.c         | 6 ++++--
 8 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index bc7daed5e..376d76302 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -987,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 7814258f2..94eed800a 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3116,8 +3116,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3232,6 +3231,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index 8a7015a0b..72f3c16c1 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -464,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index de10b5bdf..1e8237a41 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -680,6 +679,9 @@ rte_pmd_null_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 71843c63a..5431b44dc 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1133,6 +1133,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_secondary(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1143,6 +1148,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index a015a9d48..8146602a7 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -1007,8 +1007,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -1107,6 +1106,9 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index ad5ae988b..84aaf2410 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1992,8 +1992,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -2075,7 +2074,10 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 	/* find the ethdev entry */
 	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index aa6052221..a7604ff23 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1436,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v16 5/6] drivers/net: enable device detach on secondary

[Qi Zhang]

With the enabling for hotplug on multi-process,
rte_eth_dev_pci_generic_remove can be used to detach the device from
a secondary process also. But we need to take care of the uninit callback
parameter to make sure it handles the secondary case correctly.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bnxt/bnxt_ethdev.c     | 6 +++++-
 drivers/net/ena/ena_ethdev.c       | 2 +-
 drivers/net/liquidio/lio_ethdev.c  | 2 +-
 drivers/net/virtio/virtio_ethdev.c | 2 +-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 70c761581..c06b4e828 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -3514,7 +3514,11 @@ static int bnxt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int bnxt_pci_remove(struct rte_pci_device *pci_dev)
 {
-	return rte_eth_dev_pci_generic_remove(pci_dev, bnxt_dev_uninit);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return rte_eth_dev_pci_generic_remove(pci_dev,
+				bnxt_dev_uninit);
+	else
+		return rte_eth_dev_pci_generic_remove(pci_dev, NULL);
 }
 
 static struct rte_pci_driver bnxt_rte_pmd = {
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index c255dc6db..c29a581e8 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -1703,7 +1703,7 @@ static int eth_ena_dev_uninit(struct rte_eth_dev *eth_dev)
 		(struct ena_adapter *)(eth_dev->data->dev_private);
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	if (adapter->state != ENA_ADAPTER_STATE_CLOSED)
 		ena_close(eth_dev);
diff --git a/drivers/net/liquidio/lio_ethdev.c b/drivers/net/liquidio/lio_ethdev.c
index 93e89007a..0f59e4475 100644
--- a/drivers/net/liquidio/lio_ethdev.c
+++ b/drivers/net/liquidio/lio_ethdev.c
@@ -2038,7 +2038,7 @@ lio_eth_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	/* lio_free_sc_buffer_pool */
 	lio_free_sc_buffer_pool(lio_dev);
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index b81df0a99..730c41707 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1697,7 +1697,7 @@ eth_virtio_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
-		return -EPERM;
+		return 0;
 
 	virtio_dev_stop(eth_dev);
 	virtio_dev_close(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v16 6/6] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +++
 examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 +++++
 5 files changed, 289 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..b06859393
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <devargs>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	if (!rte_eal_hotplug_add(da.bus->name, da.name, da.args))
+		cmdline_printf(cl, "attached device %s\n", da.name);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+				da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	printf("detaching...\n");
+	if (!rte_eal_hotplug_remove(da.bus->name, da.name))
+		cmdline_printf(cl, "detached device %s\n",
+			da.name);
+	else
+		cmdline_printf(cl, "failed to dettach device %s\n",
+			da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+
+cmdline_parse_token_string_t cmd_dev_detach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v16 0/6] enable hotplug on multi-process

[Raslan Darawsheh]

Hi Qi Zhang,

From: Qi Zhang
> v16:
> - rebase to patch "simplify parameters of hotplug functions"

Seems that we have some issues in applying this series such as following and it continues with the rest of the patches

Applying: eal: enable hotplug on multi-process
fatal: sha1 information is lacking or useless (lib/librte_eal/common/eal_common_dev.c).
error: could not build fake ancestor
Patch failed at 0002 eal: enable hotplug on multi-process
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".


Kindest regards
Raslan Darawsheh

Re: [dpdk-dev] [PATCH v16 2/6] eal: enable hotplug on multi-process

[Thomas Monjalon]

Hi Qi,

I wanted to push this old series, but I still have some questions
and comments. Please let's fix them quickly.

Note: I did not review the IPC mechanism and rollback. I trust you :)

28/09/2018 06:23, Qi Zhang:
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,12 @@ New Features
>    SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>    vdev_netvsc, tap, and failsafe drivers combination.
>  
> +* **Support device multi-process hotplug.**
> +
> +  Hotplug and hot-unplug for devices will now be supported in multiprocessing
> +  scenario. Any ethdev devices created in the primary process will be regarded
> +  as shared and will be available for all DPDK processes. Synchronization
> +  between processes will be done using DPDK IPC.
>  

A blank line is missing here.

>  API Changes
>  -----------
> @@ -91,6 +97,11 @@ API Changes
>    flag the MAC can be properly configured in any case. This is particularly
>    important for bonding.
>  
> +* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
> +
> +  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
> +  that device be attached on all processes, while ``rte_eal_hotplug_remove``
> +  will guarantee device be detached on all processes.
>  

Here too, double blank line before next heading.

> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> -int
> -rte_eal_hotplug_add(const char *busname, const char *devname,
> -                    const char *drvargs)
> +/* help funciton to build devargs, caller should free the memory */

"help funciton" -> "helper function"

> +static char *
> +build_devargs(const char *busname, const char *devname,
> +	      const char *drvargs)
>  {
>  	char *devargs = NULL;
>  	int size, length = -1;
> @@ -140,19 +143,33 @@ rte_eal_hotplug_add(const char *busname, const char *devname,
>  		if (length >= size)
>  			devargs = malloc(length + 1);
>  		if (devargs == NULL)
> -			return -ENOMEM;
> +			break;
>  	} while (size == 0);

It is an old code, please rebase on master.

> -int __rte_experimental
> -rte_dev_remove(struct rte_device *dev)
> +/* remove device at local process. */
> +int
> +local_dev_remove(struct rte_device *dev)
>  {
>  	struct rte_bus *bus;
>  	int ret;
> @@ -248,7 +268,194 @@ rte_dev_remove(struct rte_device *dev)
>  	if (ret)
>  		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
>  			dev->name);
> -	rte_devargs_remove(dev->devargs);
> +	else
> +		rte_devargs_remove(dev->devargs);

It looks you are fixing a bug here. Good catch!

> +int __rte_experimental
> +rte_dev_probe(const char *devargs)
> +{
> +	struct eal_dev_mp_req req;
> +	struct rte_device *dev;
> +	int ret;
> +
> +	memset(&req, 0, sizeof(req));
> +	req.t = EAL_DEV_REQ_TYPE_ATTACH;
> +	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/**
> +		 * If in secondary process, just send IPC request to
> +		 * primary process.
> +		 */
> +		ret = eal_dev_hotplug_request_to_primary(&req);
> +		if (ret) {
> +			RTE_LOG(ERR, EAL,
> +				"Failed to send hotplug request to primary\n");
> +			return -ENOMSG;
> +		}
> +		if (req.result)
> +			RTE_LOG(ERR, EAL,
> +				"Failed to hotplug add device\n");
> +		return req.result;
> +	}
> +
> +	/* attach a shared device from primary start from here: */
> +
> +	/* primary attach the new device itself. */
> +	ret = local_dev_probe(devargs, &dev);
> +
> +	if (ret) {
> +		RTE_LOG(ERR, EAL,
> +			"Failed to attach device on primary process\n");
> +
> +		/**
> +		 * it is possible that secondary process failed to attached a
> +		 * device that primary process have during initialization,
> +		 * so for -EEXIST case, we still need to sync with secondary
> +		 * process.
> +		 */
> +		if (ret != -EEXIST)
> +			return ret;
> +	}
> +
> +	/* primary send attach sync request to secondary. */
> +	ret = eal_dev_hotplug_request_to_secondary(&req);
> +
> +	/* if any commnunication error, we need to rollback. */

typo: communication

> +	if (ret) {
> +		RTE_LOG(ERR, EAL,
> +			"Failed to send hotplug add request to secondary\n");
> +		ret = -ENOMSG;
> +		goto rollback;
> +	}
> +
> +	/**
> +	 * if any secondary failed to attach, we need to consider if rollback
> +	 * is necessary.
> +	 */
> +	if (req.result) {
> +		RTE_LOG(ERR, EAL,
> +			"Failed to attach device on secondary process\n");
> +		ret = req.result;
> +
> +		/* for -EEXIST, we don't need to rollback. */
> +		if (ret == -EEXIST)
> +			return ret;
> +		goto rollback;
> +	}
> +
> +	return 0;
> +
> +rollback:
> +	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
> +
> +	/* primary send rollback request to secondary. */
> +	if (eal_dev_hotplug_request_to_secondary(&req))

For all occurences of "if (function())", the coding style is requesting
an explicit check of the return value: if (function() != 0)

> +		RTE_LOG(WARNING, EAL,
> +			"Failed to rollback device attach on secondary."
> +			"Devices in secondary may not sync with primary\n");
> +
> +	/* primary rollback itself. */
> +	if (local_dev_remove(dev))
> +		RTE_LOG(WARNING, EAL,
> +			"Failed to rollback device attach on primary."
> +			"Devices in secondary may not sync with primary\n");
> +
> +	return ret;
> +}
> +
> +int __rte_experimental
> +rte_dev_remove(struct rte_device *dev)
> +{
> +	struct eal_dev_mp_req req;
> +	char *devargs;
> +	int ret;
> +
> +	devargs = build_devargs(dev->devargs->bus->name, dev->name, "");
> +	if (devargs == NULL)
> +		return -ENOMEM;
> +
> +	memset(&req, 0, sizeof(req));
> +	req.t = EAL_DEV_REQ_TYPE_DETACH;
> +	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
> +	free(devargs);
> +
> +	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> +		/**
> +		 * If in secondary process, just send IPC request to
> +		 * primary process.
> +		 */
> +		ret = eal_dev_hotplug_request_to_primary(&req);
> +		if (ret) {
> +			RTE_LOG(ERR, EAL,
> +				"Failed to send hotplug request to primary\n");
> +			return -ENOMSG;
> +		}
> +		if (req.result)
> +			RTE_LOG(ERR, EAL,
> +				"Failed to hotplug remove device\n");
> +		return req.result;
> +	}
> +
> +	/* detach a device from primary start from here: */
> +
> +	/* primary send detach sync request to secondary */
> +	ret = eal_dev_hotplug_request_to_secondary(&req);
> +
> +	/**
> +	 * if communication error, we need to rollback, because it is possible
> +	 * part of the secondary processes still detached it successfully.
> +	 */
> +	if (ret) {

ret is not a boolean, please do explicit check != 0.

> +		RTE_LOG(ERR, EAL,
> +			"Failed to send device detach request to secondary\n");
> +		ret = -ENOMSG;
> +		goto rollback;
> +	}
> +
> +	/**
> +	 * if any secondary failed to detach, we need to consider if rollback
> +	 * is necessary.
> +	 */
> +	if (req.result) {

result is not a boolean, please do explicit check != 0.

> +		RTE_LOG(ERR, EAL,
> +			"Failed to detach device on secondary process\n");
> +		ret = req.result;
> +		/**
> +		 * if -ENOENT, we don't need to rollback, since devices is
> +		 * already detached on secondary process.
> +		 */
> +		if (ret != -ENOENT)
> +			goto rollback;
> +	}
> +
> +	/* primary detach the device itself. */
> +	ret = local_dev_remove(dev);
> +
> +	/* if primary failed, still need to consider if rollback is necessary */
> +	if (ret) {
> +		RTE_LOG(ERR, EAL,
> +			"Failed to detach device on primary process\n");
> +		/* if -ENOENT, we don't need to rollback */
> +		if (ret == -ENOENT)
> +			return ret;
> +		goto rollback;
> +	}
> +
> +	return 0;
> +
> +rollback:
> +	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
> +
> +	/* primary send rollback request to secondary. */
> +	if (eal_dev_hotplug_request_to_secondary(&req))
> +		RTE_LOG(WARNING, EAL,
> +			"Failed to rollback device detach on secondary."
> +			"Devices in secondary may not sync with primary\n");
> +
>  	return ret;
>  }

> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> +/**
> + * Register all mp action callbacks for hotplug.
> + *
> + * @return
> + *   0 on success, negative on error.
> + */
> +int rte_dev_hotplug_mp_init(void);

This function is called by the init, so it should not be private.
The app should be free to build its own init routine.

> --- /dev/null
> +++ b/lib/librte_eal/common/hotplug_mp.h
> +#include <rte_dev.h>
> +#include <rte_bus.h>

I think EAL headers should be included with quotes.

> +/**
> + * this is a synchronous wrapper for secondary process send

Missing uppercase at the beggining of sentence.

> + * request to primary process, this is invoked when an attach
> + * or detach request issued from primary process.

Missing "is" before "issued": request is issued.

> --- a/lib/librte_eal/common/include/rte_dev.h
> +++ b/lib/librte_eal/common/include/rte_dev.h
> @@ -190,6 +190,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
>  
>  /**
>   * Hotplug add a given device to a specific bus.
> + * In multi-process, this function will inform all other processes
> + * to hotplug add the same device. Any failure on other process rollback
> + * the action.

Better to leave a blank line before this comment.
Small reword:

 * Hotplug add a given device to a specific bus.
 *
 * In multi-process, it will request other processes to add the same device.
 * A failure, in any process, will rollback the action.

You should add the same comment for rte_dev_probe().

> @@ -219,6 +222,9 @@ int __rte_experimental rte_dev_probe(const char *devargs);
>  
>  /**
>   * Hotplug remove a given device from a specific bus.
> + * In multi-process, this function will inform all other processes
> + * to hotplug remove the same device. Any failure on other process
> + * will rollback the action.

Same reword:

 * Hotplug remove a given device from a specific bus.
 *
 * In multi-process, it will request other processes to remove the same device.
 * A failure, in any process, will rollback the action.

[...]
> --- a/lib/librte_eal/linuxapp/eal/eal.c
> +++ b/lib/librte_eal/linuxapp/eal/eal.c
> +	/* register mp action callbacks for hotplug */
> +	if (rte_dev_hotplug_mp_init() < 0) {

In the comment, better to say "multi-process" instead of the cryptic "mp".

[dpdk-dev] [PATCH v17 0/6] enable hotplug on multi-process

[Qi Zhang]

v17:
- fix format in release notes
- rework build_devargs
- fix devargs memory leak in rte_dev_hotplug_add
- always explicit if (<check> != 0)
- rename rte_dev_hotplug_mp_init to rte_mp_dev_hotplug_init and move
  funciton claim to rte_dev.h
- comment reword 

v16:
- rebase to patch "simplify parameters of hotplug functions"
  http://patchwork.dpdk.org/patch/45463/ include:
  * keep rte_eal_hotplug_add/rte_eal_hotplug_move unchanged.
  * the IPC sync logic is moved to rte_dev_probe/rte_dev_remove. 
  * simplify the IPC message by removing busname and devname from
    eal_dev_mp_req, since devargs string will encode those information
    already.
- combined release notes with related code changes.
- replace do_ prefix to local_ for local process only probe/remove function.
- improve comments 

v15:
- fix missing return in rte_eth_dev_pci_release.
- minor fix and more detail comments for patch 5/7.
- update release notes for v18.11.

v14:
- rebase.
- All changes belongs to patch 1/6.
  1) rename rte_eth_dev_release_port_private to rte_eth_dev_release_port_seondary
     since it is only used by secondary process.
  2) in rte_eth_dev_pci_generic_remove, even on the secondary process,
     I think its better to call rte_eth_dev_release_port_secondary after
     dev_uninit since it is possible that secondary process need to release
     some local resources in dev_uninit before release the port and return.
     Also this does not break all exist users of rte_eth_dev_pci_generic_remove,
     because there is no special handle in all exist dev_uninit for secondary
     process.
  3) add rte_eth_dev_release_port_secondary into rte_eth_dev_destroy as a
     general step, so we don't need patches for i40e and ixgbe.
  4) fix missing update on rte_ethdev_version.map.
- improve error handle for -EEXIST when attaching a device and -ENOENT
  when detaching a device. It is possible that device is not synced during
  some situation, so attach an exist device in primary still need to sync
  with secondary. Also, it's not necessary to rollback if we fail to
  attach an exist device or detach a not exist device on secondary.
- fix potential NULL point ref in handle_primary_request.
- merge all vdev driver patches into one patch.
- merge all pci driver patches into on patch.

v13:
- Since rte_eth_dev_attach/rte_eth_dev_detach will be deprecated,
  so, modify the sample code to use rte_eal_hotplug_add and
  rte_eal_hotplug_remove to attach/detach device.

v12:
- fix return value in eal_dev_hotplug_request_to_primary.
- add more error log in rte_eal_hotplug_add.
- fix return value in rte_eal_hotplug_add and rte_eal_hotplug_remove
  any failure due to IPC error will return -ENOMSG, but not -1.
- remove unnecessary changes from previous rework.

v11: - move out common code from pci_vfio_unmap_secondary and
  pci_vfio_unmap_primary.
- move RTE_BUS_NAME_MAX_LEN and RTE_DEV_ARGS_MAX_LEN into hotplug_mp.h
- fix reply check in eal_dev_hotplug_request_to_primary.
- move skeleton code for attaching device from secondary from patch 6/19
  to patch 5/19 to improve code readability.

v10:
- Since hotplug add/remove a vdev on a secondary process will sync on
  all processes now, it is not necessary to support private vdev for
  a secondary process which is identified by a not-NULL devargs in
  "--vdev". So re-work on all vdev driver changes to simpified device
  probe scenario on a secondary process, devargs will be ignored on
  secondary process now.
- fix lisence header in example/multi-process/hotplug_mp/Makefile.

v9:
- Move hotplug IPC from rte_eth_dev_attach/rte_eth_dev_detach to
  eal_dev_hotplug_add and eal_dev_hotplug_remove, now all kinds of
  devices will be synced in multi-process.
- Fix couple issue when a device is bound to vfio.
  1) The device can't be detached clearly in a secondary process, which
     also cause it can't be attached again, due to the error that
     /dev/vfio/<group_fd> is still busy.(see Patch 3/19 and 4/19)
  2) repeat detach/attach device will cause "cannot find TAILQ entry
     for PCI device" due to incorrect PCI address compare.
     (see patch 2/19).
- Removed device lock.
- Removed private device support.
- Fix commit log grammar issue

v8:
- update rte_eal_version.map due to new API added.
- minor reword on release note.
- minor fix on commit log and code style.

NOTE:
  Some issues which is not related with this patchset is expected when
  play with hotplug_mp sample as belows.

- Attach a PCI device twice may cause device can't be detached
  below fix is required:
  https://patches.dpdk.org/patch/42030/

- ixgbe device can't detached, below fix is required
  https://patches.dpdk.org/patch/42031/

v7:
- update rte_ethdev_version.map for new APIs.
- improve code readability in __handle_secondary_request by use goto.
- add comments to explain why need to call rte_eal_alarm_set.
- add error log when process_mp_init_callbacks failed.
- reword release notes base on Anatoly's suggestion.
- add back previous "Acked-by" and "Reviewed-by" in commit log.

  NOTE: current patchset depends on below IPC fix, or it may not be able
  to attach a shared vdev.
  https://patches.dpdk.org/patch/41647/

v6:
- remove bus->scan_one, since ABI break is not necessary.
- remove patch for failsafe PMD since it will not support secondary.
- fix wrong implemenation on ixgbe.
- add rte_eth_dev_release_port_private into rte_eth_dev_pci_generic_remove for
  secondary process, so we don't need to patch on PMD if PMD use the
  default remove function.
- add release notes update.
- agreed to use strdup(peer) as workaround for repling a sync request in seperate
  thread.

v5:
- since we will keep mp thread separate from interrupt thread,
  it is not necessary to use temporary thread, we use rte_eal_alarm_set.
- remove the change in rte_eth_dev_release_port, since there is a better
  way to prevent rte_eth_dev_release_port be called after
  rte_eth_dev_release_port_private.
- fix the issue that lock does not take effect on secondary due to
  previous re-work
- fix the issue when the first attached device is a private device from
  secondary. (patch 8/24)
- work around for reply a sync request in separate thread, this is still
  an open and in discussion as below.
  https://mails.dpdk.org/archives/dev/2018-June/105359.html

v4:
- since mp thread will be merged to interrupt thread, the fix on v3
  for sync IPC deadlock will not work. the new version enable the
  machanism to invoke a mp action callback in a temporary thread to
  avoid the IPC deadlock, with this, secondary to primary request
  impelemtation also be simplified, since we can use sync request
  directly in a separate thread.

v3:
- enable mp init callback register to help non-eal module to initialize
  mp channel during rte_eal_init
- fix when attach share device from secondary.
  1) dead lock due to sync IPC be invoked in rte_malloc in primary
     process when handle secondary request to attach device, the
     solution is primary process to issue share device attach/detach
     in interrupt thread.
  2) return port_id not correct.
- check nb_sent and nb_received in sync IPC.
- fix memory leak duirng error handling at attach_on_secondary.
- improve clean_lock_callback to only lock/unlock spinlock once
- improve error code return in check-reply during async IPC.
- remove rte_ prefix of internal function in ethdev_mp.c
- sample code improvement.
  1) rename sample to "hotplug_mp", and move to example/multi-process.
  2) cleanup header include.
  3) call rte_eal_cleanup before exit.

v2:
- rename rte_ethdev_mp.* to ethdev_mp.*
- rename rte_ethdev_lock.* to ethdev_lock.*
- move internal funciton to ethdev_private.h
- separate rte_eth_dev_[un]lock into rte_eth_dev_[un]lock and
  rte_eth_dev_[un]lock_with_callback
- lock callbacks will be removed automatically after device is detached.
- add experimental tag for all new APIs.
- fix coding style issue.
- fix wrong lisence header in sample code.
- fix spelling 
- fix meson.build.
- improve comments. 

Background:
===========

Currently secondary process will only sync ethdev from primary
process at init stage, but it will not be aware if device
is attached/detached on primary process at runtime.

While there is the requirement from application that take
primary-secondary process model. The primary process work as a
resource management process, it will create/destroy virtual device
at runtime, while the secondary process deal with the network stuff
with these devices.

Solution:
=========

So the orignial intention is to fix this gap, but beyond that
the patch set provide a more comprehesive solution to handle
different hotplug cases in multi-process situation, it cover below
scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In primary-secondary process model, we assume ethernet devices are
shared by default. that means attach or detach a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching or detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

Scenario for Case 1, 2:

attach device from primary
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach device and send reply.
d) primary check the reply if all success go to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach device and send reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach device from primary
a) primary perform pre-detach check, if device is locked, goto i).
b) primary send pre-detach sync request to all secondary.
c) secondary perform pre-detach check and send reply.
d) primary check the reply if any fail goto i).
e) primary send detach sync request to all secondary
f) secondary detach the device and send reply (assume no fail)
g) primary detach the device.
h) detach success
i) detach failed

Scenario for case 3, 4:

attach device from secondary:
a) seconary send asycn request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and attach the new device if failed
   goto i).
c) primary forward attach request to all secondary as async request
   (because this in mp thread context, use sync request will deadlock,
    same reason for all following async request.)
d) secondary receive request and attach device and send reply.
e) primary check the reply if all success go to j).
f) primary send attach rollback async request to all secondary.
g) secondary receive the request and detach device and send reply.
h) primary receive the reply and detach device as rollback action.
i) send fail response to secondary, goto k).
j) send success response to secondary.
k) secondary process receive response and return.
 
detach device from secondary:
a) secondary send async request to primary and wait on a condition
   which will be released by matched response from primary.
b) primary receive the request and  perform pre-detach check, if device
   is locked, goto j).
c) primary send pre-detach async request to all secondary.
d) secondary perform pre-detach check and send reply.
e) primary check the reply if any fail goto j).
f) primary send detach async request to all secondary
g) secondary detach the device and send reply
h) primary detach the device.
i) send success response to secondary, goto k).
j) send fail response to secondary.
k) secondary process receive response and return.

APIs chenages:
==============

scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
In primary-secondary process model, rte_eal_hotplug_add will guarantee
that device be attached on all processes, while rte_eal_hotplug_remove will
guarantee device be detached on all processes.


PMD Impact:
===========

Currently device removing is not handled well in secondary process on
most pmd drivers, rte_eth_dev_relase_port will be invoked and will mess up
primary process since it reset all shared data. So we introduced new API
rte_eth_dev_release_port_secondary which only reset ethdev's state to unsued
but not touch shared data so other process will not be impacted.
Since not all device driver is target to support primary-secondary
process model, so the patch set only fix this for PCI device those driver use
rte_eth_dev_pci_generic_remove or rte_eth_dev_destroy and all
vdev that support secondary process, it can be refereneced by other driver
when equevalent fix is required

Example:
========

The patchset also contains a example to demonstrate device hotplug
in multi-process model, below are detail instructions.

/* start sample code as primary then secondary */
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Qi Zhang (6):
  ethdev: add function to release port in secondary process
  eal: enable hotplug on multi-process
  eal: support attach or detach share device from  secondary
  drivers/net: enable hotplug on secondary process
  drivers/net: enable device detach on secondary
  examples/multi_process: add hotplug sample

 doc/guides/rel_notes/release_18_11.rst       |  13 +
 drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
 drivers/net/bnxt/bnxt_ethdev.c               |   6 +-
 drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
 drivers/net/ena/ena_ethdev.c                 |   2 +-
 drivers/net/kni/rte_eth_kni.c                |   6 +-
 drivers/net/liquidio/lio_ethdev.c            |   2 +-
 drivers/net/null/rte_eth_null.c              |   6 +-
 drivers/net/octeontx/octeontx_ethdev.c       |   8 +
 drivers/net/pcap/rte_eth_pcap.c              |   6 +-
 drivers/net/tap/rte_eth_tap.c                |   8 +-
 drivers/net/vhost/rte_eth_vhost.c            |   6 +-
 drivers/net/virtio/virtio_ethdev.c           |   2 +-
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 ++
 examples/multi_process/hotplug_mp/commands.c | 214 ++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 +
 examples/multi_process/hotplug_mp/main.c     |  41 +++
 lib/librte_eal/bsdapp/eal/Makefile           |   1 +
 lib/librte_eal/common/eal_common_dev.c       | 254 ++++++++++++++--
 lib/librte_eal/common/eal_private.h          |  22 ++
 lib/librte_eal/common/hotplug_mp.c           | 426 +++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h           |  46 +++
 lib/librte_eal/common/include/rte_dev.h      |  12 +
 lib/librte_eal/common/include/rte_eal.h      |   9 +
 lib/librte_eal/common/meson.build            |   1 +
 lib/librte_eal/linuxapp/eal/Makefile         |   1 +
 lib/librte_eal/linuxapp/eal/eal.c            |   6 +
 lib/librte_ethdev/rte_ethdev.c               |  17 +-
 lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
 lib/librte_ethdev/rte_ethdev_pci.h           |  10 +-
 lib/librte_ethdev/rte_ethdev_version.map     |   7 +
 32 files changed, 1151 insertions(+), 43 deletions(-)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

-- 
2.13.6

[dpdk-dev] [PATCH v17 1/6] ethdev: add function to release port in secondary process

[Qi Zhang]

Add driver API rte_eth_release_port_secondary to support the
case when an ethdev need to be detached on a secondary process.
Local state is set to unused and shared data will not be reset
so the primary process can still use it.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 lib/librte_ethdev/rte_ethdev.c           | 17 +++++++++++++++--
 lib/librte_ethdev/rte_ethdev_driver.h    | 16 +++++++++++++++-
 lib/librte_ethdev/rte_ethdev_pci.h       | 10 ++++++++--
 lib/librte_ethdev/rte_ethdev_version.map |  7 +++++++
 4 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 21f1dfbe4..741a864f7 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -363,6 +363,18 @@ rte_eth_dev_attach_secondary(const char *name)
 }
 
 int
+rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev)
+{
+	if (eth_dev == NULL)
+		return -EINVAL;
+
+	_rte_eth_dev_callback_process(eth_dev, RTE_ETH_EVENT_DESTROY, NULL);
+	eth_dev->state = RTE_ETH_DEV_UNUSED;
+
+	return 0;
+}
+
+int
 rte_eth_dev_release_port(struct rte_eth_dev *eth_dev)
 {
 	if (eth_dev == NULL)
@@ -3578,9 +3590,10 @@ rte_eth_dev_destroy(struct rte_eth_dev *ethdev,
 			return ret;
 	}
 
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(ethdev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(ethdev);
 
+	rte_free(ethdev->data->dev_private);
 	ethdev->data->dev_private = NULL;
 
 	return rte_eth_dev_release_port(ethdev);
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index f158462a0..ca31b5777 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -61,7 +61,7 @@ struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
  * Release the specified ethdev port.
  *
  * @param eth_dev
- * The *eth_dev* pointer is the address of the *rte_eth_dev* structure.
+ * Device to be detached.
  * @return
  *   - 0 on success, negative on error
  */
@@ -69,6 +69,20 @@ int rte_eth_dev_release_port(struct rte_eth_dev *eth_dev);
 
 /**
  * @internal
+ * Release the specified ethdev port in the local process.
+ * Only set ethdev state to unused, but not reset shared data since
+ * it assume other processes is still using it. typically it is
+ * called by a secondary process.
+ *
+ * @param eth_dev
+ * Device to be detached.
+ * @return
+ *   - 0 on success, negative on error
+ */
+int rte_eth_dev_release_port_secondary(struct rte_eth_dev *eth_dev);
+
+/**
+ * @internal
  * Release device queues and clear its configuration to force the user
  * application to reconfigure it. It is for internal use only.
  *
diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h
index f652596f4..70d2d2503 100644
--- a/lib/librte_ethdev/rte_ethdev_pci.h
+++ b/lib/librte_ethdev/rte_ethdev_pci.h
@@ -135,9 +135,15 @@ rte_eth_dev_pci_allocate(struct rte_pci_device *dev, size_t private_data_size)
 static inline void
 rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
 {
-	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		rte_free(eth_dev->data->dev_private);
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		eth_dev->device = NULL;
+		eth_dev->intr_handle = NULL;
+		rte_eth_dev_release_port_secondary(eth_dev);
+		return;
+	}
 
+	/* primary process */
+	rte_free(eth_dev->data->dev_private);
 	eth_dev->data->dev_private = NULL;
 
 	/*
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index dfa122c1a..3a4dd4790 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -220,6 +220,13 @@ DPDK_18.08 {
 
 } DPDK_18.05;
 
+DPDK_18.11 {
+	global:
+
+	rte_eth_dev_release_port_secondary;
+
+} DPDK_18.08;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.13.6

[dpdk-dev] [PATCH v17 2/6] eal: enable hotplug on multi-process

[Qi Zhang]

We are going to introduce the solution to handle hotplug in
multi-process, it includes the below scenario:

1. Attach a device from the primary
2. Detach a device from the primary
3. Attach a device from a secondary
4. Detach a device from a secondary

In the primary-secondary process model, we assume devices are shared
by default. that means attaches or detaches a device on any process
will broadcast to all other processes through mp channel then device
information will be synchronized on all processes.

Any failure during attaching/detaching process will cause inconsistent
status between processes, so proper rollback action should be considered.

This patch covers the implementation of case 1,2.
Case 3,4 will be implemented on a separate patch.

IPC scenario for Case 1, 2:

attach a device
a) primary attach the new device if failed goto h).
b) primary send attach sync request to all secondary.
c) secondary receive request and attach the device and send a reply.
d) primary check the reply if all success goes to i).
e) primary send attach rollback sync request to all secondary.
f) secondary receive the request and detach the device and send a reply.
g) primary receive the reply and detach device as rollback action.
h) attach fail
i) attach success

detach a device
a) primary send detach sync request to all secondary
b) secondary detach the device and send reply
c) primary check the reply if all success goes to f).
d) primary send detach rollback sync request to all secondary.
e) secondary receive the request and attach back device. goto g)
f) primary detach the device if success goto g), else goto d)
g) detach fail.
h) detach success.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 doc/guides/rel_notes/release_18_11.rst  |  13 ++
 lib/librte_eal/bsdapp/eal/Makefile      |   1 +
 lib/librte_eal/common/eal_common_dev.c  | 254 +++++++++++++++++++++++++++++---
 lib/librte_eal/common/eal_private.h     |  22 +++
 lib/librte_eal/common/hotplug_mp.c      | 221 +++++++++++++++++++++++++++
 lib/librte_eal/common/hotplug_mp.h      |  46 ++++++
 lib/librte_eal/common/include/rte_dev.h |  12 ++
 lib/librte_eal/common/include/rte_eal.h |   9 ++
 lib/librte_eal/common/meson.build       |   1 +
 lib/librte_eal/linuxapp/eal/Makefile    |   1 +
 lib/librte_eal/linuxapp/eal/eal.c       |   6 +
 11 files changed, 567 insertions(+), 19 deletions(-)
 create mode 100644 lib/librte_eal/common/hotplug_mp.c
 create mode 100644 lib/librte_eal/common/hotplug_mp.h

diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index 436b20e2b..da2236fea 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -159,6 +159,13 @@ New Features
   this application doesn't need to launch dedicated worker threads for vhost
   enqueue/dequeue operations.
 
+* **Support device multi-process hotplug.**
+
+  Hotplug and hot-unplug for devices will now be supported in multiprocessing
+  scenario. Any ethdev devices created in the primary process will be regarded
+  as shared and will be available for all DPDK processes. Synchronization
+  between processes will be done using DPDK IPC.
+
 
 API Changes
 -----------
@@ -213,6 +220,12 @@ API Changes
 * eventdev: Type of 2nd parameter to ``rte_event_eth_rx_adapter_caps_get()``
   has been changed from uint8_t to uint16_t.
 
+* eal: scope of rte_eal_hotplug_add and rte_eal_hotplug_remove is extended.
+
+  In primary-secondary process model, ``rte_eal_hotplug_add`` will guarantee
+  that device be attached on all processes, while ``rte_eal_hotplug_remove``
+  will guarantee device be detached on all processes.
+
 
 ABI Changes
 -----------
diff --git a/lib/librte_eal/bsdapp/eal/Makefile b/lib/librte_eal/bsdapp/eal/Makefile
index 97bff4852..6e9bc02c5 100644
--- a/lib/librte_eal/bsdapp/eal/Makefile
+++ b/lib/librte_eal/bsdapp/eal/Makefile
@@ -62,6 +62,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_BSDAPP) += malloc_mp.c
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 7663eaa3f..2209f8843 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -19,8 +19,10 @@
 #include <rte_log.h>
 #include <rte_spinlock.h>
 #include <rte_malloc.h>
+#include <rte_string_fns.h>
 
 #include "eal_private.h"
+#include "hotplug_mp.h"
 
 /**
  * The device event callback description.
@@ -127,37 +129,61 @@ int rte_eal_dev_detach(struct rte_device *dev)
 	return ret;
 }
 
-int
-rte_eal_hotplug_add(const char *busname, const char *devname,
-		    const char *drvargs)
+/* helper funciton to build devargs, caller should free the memory */
+static int
+build_devargs(const char *busname, const char *devname,
+	      const char *drvargs, char **devargs)
 {
-	int ret;
-	char *devargs = NULL;
 	int length;
+	char *da;
 
 	length = snprintf(NULL, 0, "%s:%s,%s", busname, devname, drvargs);
+
 	if (length < 0)
 		return -EINVAL;
-	devargs = malloc(length + 1);
-	if (devargs == NULL)
+
+	da = malloc(length + 1);
+	if (da == NULL)
 		return -ENOMEM;
-	ret = snprintf(devargs, length + 1, "%s:%s,%s", busname, devname, drvargs);
-	if (ret < 0)
+
+	if (snprintf(da, length + 1, "%s:%s,%s",
+		busname, devname, drvargs) < 0) {
+		free(da);
 		return -EINVAL;
+	}
 
-	ret = rte_dev_probe(devargs);
+	*devargs = da;
+	return 0;
+}
 
+int
+rte_eal_hotplug_add(const char *busname, const char *devname,
+		    const char *drvargs)
+{
+
+	char *devargs;
+	int ret;
+
+	ret = build_devargs(busname, devname, drvargs, &devargs);
+
+	if (ret != 0)
+		return ret;
+
+	ret = rte_dev_probe(devargs);
 	free(devargs);
+
 	return ret;
 }
 
-int __rte_experimental
-rte_dev_probe(const char *devargs)
+/* probe device at local process. */
+int
+local_dev_probe(const char *devargs, struct rte_device **new_dev)
 {
 	struct rte_device *dev;
 	struct rte_devargs *da;
 	int ret;
 
+	*new_dev = NULL;
 	da = calloc(1, sizeof(*da));
 	if (da == NULL)
 		return -ENOMEM;
@@ -174,11 +200,11 @@ rte_dev_probe(const char *devargs)
 	}
 
 	ret = rte_devargs_insert(da);
-	if (ret)
+	if (ret != 0)
 		goto err_devarg;
 
 	ret = da->bus->scan();
-	if (ret)
+	if (ret != 0)
 		goto err_devarg;
 
 	dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
@@ -195,11 +221,13 @@ rte_dev_probe(const char *devargs)
 	}
 
 	ret = dev->bus->plug(dev);
-	if (ret) {
+	if (ret != 0) {
 		RTE_LOG(ERR, EAL, "Driver cannot attach the device (%s)\n",
 			dev->name);
 		goto err_devarg;
 	}
+
+	*new_dev = dev;
 	return 0;
 
 err_devarg:
@@ -231,8 +259,9 @@ rte_eal_hotplug_remove(const char *busname, const char *devname)
 	return rte_dev_remove(dev);
 }
 
-int __rte_experimental
-rte_dev_remove(struct rte_device *dev)
+/* remove device at local process. */
+int
+local_dev_remove(struct rte_device *dev)
 {
 	int ret;
 
@@ -248,10 +277,197 @@ rte_dev_remove(struct rte_device *dev)
 	}
 
 	ret = dev->bus->unplug(dev);
-	if (ret)
+	if (ret != 0)
 		RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\n",
 			dev->name);
-	rte_devargs_remove(dev->devargs);
+	else
+		rte_devargs_remove(dev->devargs);
+
+	return ret;
+}
+
+int __rte_experimental
+rte_dev_probe(const char *devargs)
+{
+	struct eal_dev_mp_req req;
+	struct rte_device *dev;
+	int ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_ATTACH;
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result != 0)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug add device\n");
+		return req.result;
+	}
+
+	/* attach a shared device from primary start from here: */
+
+	/* primary attach the new device itself. */
+	ret = local_dev_probe(devargs, &dev);
+
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on primary process\n");
+
+		/**
+		 * it is possible that secondary process failed to attached a
+		 * device that primary process have during initialization,
+		 * so for -EEXIST case, we still need to sync with secondary
+		 * process.
+		 */
+		if (ret != -EEXIST)
+			return ret;
+	}
+
+	/* primary send attach sync request to secondary. */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/* if any communication error, we need to rollback. */
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send hotplug add request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	/**
+	 * if any secondary failed to attach, we need to consider if rollback
+	 * is necessary.
+	 */
+	if (req.result != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to attach device on secondary process\n");
+		ret = req.result;
+
+		/* for -EEXIST, we don't need to rollback. */
+		if (ret == -EEXIST)
+			return ret;
+		goto rollback;
+	}
+
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+
+	/* primary send rollback request to secondary. */
+	if (eal_dev_hotplug_request_to_secondary(&req) != 0)
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device attach on secondary."
+			"Devices in secondary may not sync with primary\n");
+
+	/* primary rollback itself. */
+	if (local_dev_remove(dev) != 0)
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device attach on primary."
+			"Devices in secondary may not sync with primary\n");
+
+	return ret;
+}
+
+int __rte_experimental
+rte_dev_remove(struct rte_device *dev)
+{
+	struct eal_dev_mp_req req;
+	char *devargs;
+	int ret;
+
+	ret = build_devargs(dev->devargs->bus->name, dev->name, "", &devargs);
+	if (ret != 0)
+		return ret;
+
+	memset(&req, 0, sizeof(req));
+	req.t = EAL_DEV_REQ_TYPE_DETACH;
+	strlcpy(req.devargs, devargs, EAL_DEV_MP_DEV_ARGS_MAX_LEN);
+	free(devargs);
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+		/**
+		 * If in secondary process, just send IPC request to
+		 * primary process.
+		 */
+		ret = eal_dev_hotplug_request_to_primary(&req);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL,
+				"Failed to send hotplug request to primary\n");
+			return -ENOMSG;
+		}
+		if (req.result != 0)
+			RTE_LOG(ERR, EAL,
+				"Failed to hotplug remove device\n");
+		return req.result;
+	}
+
+	/* detach a device from primary start from here: */
+
+	/* primary send detach sync request to secondary */
+	ret = eal_dev_hotplug_request_to_secondary(&req);
+
+	/**
+	 * if communication error, we need to rollback, because it is possible
+	 * part of the secondary processes still detached it successfully.
+	 */
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to send device detach request to secondary\n");
+		ret = -ENOMSG;
+		goto rollback;
+	}
+
+	/**
+	 * if any secondary failed to detach, we need to consider if rollback
+	 * is necessary.
+	 */
+	if (req.result != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on secondary process\n");
+		ret = req.result;
+		/**
+		 * if -ENOENT, we don't need to rollback, since devices is
+		 * already detached on secondary process.
+		 */
+		if (ret != -ENOENT)
+			goto rollback;
+	}
+
+	/* primary detach the device itself. */
+	ret = local_dev_remove(dev);
+
+	/* if primary failed, still need to consider if rollback is necessary */
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL,
+			"Failed to detach device on primary process\n");
+		/* if -ENOENT, we don't need to rollback */
+		if (ret == -ENOENT)
+			return ret;
+		goto rollback;
+	}
+
+	return 0;
+
+rollback:
+	req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+
+	/* primary send rollback request to secondary. */
+	if (eal_dev_hotplug_request_to_secondary(&req) != 0)
+		RTE_LOG(WARNING, EAL,
+			"Failed to rollback device detach on secondary."
+			"Devices in secondary may not sync with primary\n");
+
 	return ret;
 }
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 4f809a83c..2ad94e435 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -304,4 +304,26 @@ int
 rte_devargs_layers_parse(struct rte_devargs *devargs,
 			 const char *devstr);
 
+/*
+ * probe a device at local process.
+ *
+ * @param devargs
+ *   Device arguments including bus, class and driver properties.
+ * @param new_dev
+ *   new device be probed as output.
+ * @return
+ *   0 on success, negative on error.
+ */
+int local_dev_probe(const char *devargs, struct rte_device **new_dev);
+
+/**
+ * Hotplug remove a given device from a specific bus at local process.
+ *
+ * @param dev
+ *   Data structure of the device to remove.
+ * @return
+ *   0 on success, negative on error.
+ */
+int local_dev_remove(struct rte_device *dev);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
new file mode 100644
index 000000000..92d8f50d3
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+#include <string.h>
+
+#include <rte_eal.h>
+#include <rte_alarm.h>
+#include <rte_string_fns.h>
+#include <rte_devargs.h>
+
+#include "hotplug_mp.h"
+#include "eal_private.h"
+
+#define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
+
+static int cmp_dev_name(const struct rte_device *dev, const void *_name)
+{
+	const char *name = _name;
+
+	return strcmp(dev->name, name);
+}
+
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
+static int
+handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	RTE_SET_USED(msg);
+	RTE_SET_USED(peer);
+	return -ENOTSUP;
+}
+
+static void __handle_primary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+	struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct rte_devargs *da;
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+
+	switch (req->t) {
+	case EAL_DEV_REQ_TYPE_ATTACH:
+	case EAL_DEV_REQ_TYPE_DETACH_ROLLBACK:
+		ret = local_dev_probe(req->devargs, &dev);
+		break;
+	case EAL_DEV_REQ_TYPE_DETACH:
+	case EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK:
+		da = calloc(1, sizeof(*da));
+		if (da == NULL) {
+			ret = -ENOMEM;
+			goto quit;
+		}
+
+		ret = rte_devargs_parse(da, req->devargs);
+		if (ret != 0)
+			goto quit;
+
+		bus = rte_bus_find_by_name(da->bus->name);
+		if (bus == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", da->bus->name);
+			ret = -ENOENT;
+			goto quit;
+		}
+
+		dev = bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find plugged device (%s)\n", da->name);
+			ret = -ENOENT;
+			goto quit;
+		}
+
+		ret = local_dev_remove(dev);
+quit:
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+	resp->result = ret;
+	if (rte_mp_reply(&mp_resp, bundle->peer) < 0)
+		RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
+
+static int
+handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	struct mp_reply_bundle *bundle;
+	int ret = 0;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	mp_resp.len_param = sizeof(*req);
+	memcpy(resp, req, sizeof(*resp));
+
+	bundle = calloc(1, sizeof(*bundle));
+	if (bundle == NULL) {
+		resp->result = -ENOMEM;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if (ret)
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+		return ret;
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = (void *)strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_primary_request, bundle);
+	if (ret != 0) {
+		resp->result = ret;
+		ret = rte_mp_reply(&mp_resp, peer);
+		if  (ret != 0) {
+			RTE_LOG(ERR, EAL, "failed to send reply to primary request\n");
+			return ret;
+		}
+	}
+	return 0;
+}
+
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
+{
+	RTE_SET_USED(req);
+	return -ENOTSUP;
+}
+
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
+{
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	int ret;
+	int i;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL, "rte_mp_request_sync failed\n");
+		return ret;
+	}
+
+	if (mp_reply.nb_sent != mp_reply.nb_received) {
+		RTE_LOG(ERR, EAL, "not all secondary reply\n");
+		return -1;
+	}
+
+	req->result = 0;
+	for (i = 0; i < mp_reply.nb_received; i++) {
+		struct eal_dev_mp_req *resp =
+			(struct eal_dev_mp_req *)mp_reply.msgs[i].param;
+		if (resp->result != 0) {
+			req->result = resp->result;
+			if (req->t == EAL_DEV_REQ_TYPE_ATTACH &&
+				req->result != -EEXIST)
+				break;
+			if (req->t == EAL_DEV_REQ_TYPE_DETACH &&
+				req->result != -ENOENT)
+				break;
+		}
+	}
+
+	return 0;
+}
+
+int rte_mp_dev_hotplug_init(void)
+{
+	int ret;
+
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_secondary_request);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	} else {
+		ret = rte_mp_action_register(EAL_DEV_MP_ACTION_REQUEST,
+					handle_primary_request);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Couldn't register '%s' action\n",
+				EAL_DEV_MP_ACTION_REQUEST);
+			return ret;
+		}
+	}
+
+	return 0;
+}
diff --git a/lib/librte_eal/common/hotplug_mp.h b/lib/librte_eal/common/hotplug_mp.h
new file mode 100644
index 000000000..597fde3d7
--- /dev/null
+++ b/lib/librte_eal/common/hotplug_mp.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _HOTPLUG_MP_H_
+#define _HOTPLUG_MP_H_
+
+#include "rte_dev.h"
+#include "rte_bus.h"
+
+#define EAL_DEV_MP_ACTION_REQUEST      "eal_dev_mp_request"
+#define EAL_DEV_MP_ACTION_RESPONSE     "eal_dev_mp_response"
+
+#define EAL_DEV_MP_DEV_NAME_MAX_LEN RTE_DEV_NAME_MAX_LEN
+#define EAL_DEV_MP_BUS_NAME_MAX_LEN 32
+#define EAL_DEV_MP_DEV_ARGS_MAX_LEN 128
+
+enum eal_dev_req_type {
+	EAL_DEV_REQ_TYPE_ATTACH,
+	EAL_DEV_REQ_TYPE_DETACH,
+	EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK,
+	EAL_DEV_REQ_TYPE_DETACH_ROLLBACK,
+};
+
+struct eal_dev_mp_req {
+	enum eal_dev_req_type t;
+	char devargs[EAL_DEV_MP_DEV_ARGS_MAX_LEN];
+	int result;
+};
+
+/**
+ * This is a synchronous wrapper for secondary process send
+ * request to primary process, this is invoked when an attach
+ * or detach request is issued from primary process.
+ */
+int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req);
+
+/**
+ * this is a synchronous wrapper for primary process send
+ * request to secondary process, this is invoked when an attach
+ * or detach request issued from secondary process.
+ */
+int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req);
+
+
+#endif /* _HOTPLUG_MP_H_ */
diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
index 036180ff3..696cf7bbe 100644
--- a/lib/librte_eal/common/include/rte_dev.h
+++ b/lib/librte_eal/common/include/rte_dev.h
@@ -192,6 +192,9 @@ int rte_eal_dev_detach(struct rte_device *dev);
 /**
  * Hotplug add a given device to a specific bus.
  *
+ * In multi-process, it will request other processes to add the same device.
+ * A failure, in any process, will rollback the action
+ *
  * @param busname
  *   The bus name the device is added to.
  * @param devname
@@ -211,6 +214,9 @@ int rte_eal_hotplug_add(const char *busname, const char *devname,
  *
  * Add matching devices.
  *
+ * In multi-process, it will request other processes to add the same device.
+ * A failure, in any process, will rollback the action
+ *
  * @param devargs
  *   Device arguments including bus, class and driver properties.
  * @return
@@ -221,6 +227,9 @@ int __rte_experimental rte_dev_probe(const char *devargs);
 /**
  * Hotplug remove a given device from a specific bus.
  *
+ * In multi-process, it will request other processes to remove the same device.
+ * A failure, in any process, will rollback the action
+ *
  * @param busname
  *   The bus name the device is removed from.
  * @param devname
@@ -236,6 +245,9 @@ int rte_eal_hotplug_remove(const char *busname, const char *devname);
  *
  * Remove one device.
  *
+ * In multi-process, it will request other processes to remove the same device.
+ * A failure, in any process, will rollback the action
+ *
  * @param dev
  *   Data structure of the device to remove.
  * @return
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index e114dcbdc..3ee897c1d 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -378,6 +378,15 @@ int __rte_experimental
 rte_mp_reply(struct rte_mp_msg *msg, const char *peer);
 
 /**
+ * Register all mp action callbacks for hotplug.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int __rte_experimental
+rte_mp_dev_hotplug_init(void);
+
+/**
  * Usage function typedef used by the application usage function.
  *
  * Use this function typedef to define and call rte_set_application_usage_hook()
diff --git a/lib/librte_eal/common/meson.build b/lib/librte_eal/common/meson.build
index b7fc98499..04c414356 100644
--- a/lib/librte_eal/common/meson.build
+++ b/lib/librte_eal/common/meson.build
@@ -28,6 +28,7 @@ common_sources = files(
 	'eal_common_thread.c',
 	'eal_common_timer.c',
 	'eal_common_uuid.c',
+	'hotplug_mp.c',
 	'malloc_elem.c',
 	'malloc_heap.c',
 	'malloc_mp.c',
diff --git a/lib/librte_eal/linuxapp/eal/Makefile b/lib/librte_eal/linuxapp/eal/Makefile
index 5c16bc40f..736bc6569 100644
--- a/lib/librte_eal/linuxapp/eal/Makefile
+++ b/lib/librte_eal/linuxapp/eal/Makefile
@@ -70,6 +70,7 @@ SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_proc.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_fbarray.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += eal_common_uuid.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += rte_malloc.c
+SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_elem.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_heap.c
 SRCS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += malloc_mp.c
diff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c
index 950f33f2c..d342a04f0 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -888,6 +888,12 @@ rte_eal_init(int argc, char **argv)
 		}
 	}
 
+	/* register multi-process action callbacks for hotplug */
+	if (rte_mp_dev_hotplug_init() < 0) {
+		rte_eal_init_alert("failed to register mp callback for hotplug\n");
+		return -1;
+	}
+
 	if (rte_bus_scan()) {
 		rte_eal_init_alert("Cannot scan the buses for devices\n");
 		rte_errno = ENODEV;
-- 
2.13.6

[dpdk-dev] [PATCH v17 3/6] eal: support attach or detach share device from secondary

[Qi Zhang]

This patch cover the multi-process hotplug case when a device
attach/detach request be issued from a secondary process

device attach on secondary:
a) secondary send sync request to the primary.
b) primary receive the request and attach the new device if
   failed goto i).
c) primary forward attach sync request to all secondary.
d) secondary receive the request and attach the device and send a reply.
e) primary check the reply if all success goes to j).
f) primary send attach rollback sync request to all secondary.
g) secondary receive the request and detach the device and send a reply.
h) primary receive the reply and detach device as rollback action.
i) send attach fail to secondary as a reply of step a), goto k).
j) send attach success to secondary as a reply of step a).
k) secondary receive reply and return.

device detach on secondary:
a) secondary send sync request to the primary.
b) primary send detach sync request to all secondary.
c) secondary detach the device and send a reply.
d) primary check the reply if all success goes to g).
e) primary send detach rollback sync request to all secondary.
f) secondary receive the request and attach back device. goto h).
g) primary detach the device if success goto i), else goto e).
h) primary send detach fail to secondary as a reply of step a), goto j).
i) primary send detach success to secondary as a reply of step a).
j) secondary receive reply and return.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
 lib/librte_eal/common/hotplug_mp.c | 223 +++++++++++++++++++++++++++++++++++--
 1 file changed, 214 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/common/hotplug_mp.c b/lib/librte_eal/common/hotplug_mp.c
index 92d8f50d3..84f59d95b 100644
--- a/lib/librte_eal/common/hotplug_mp.c
+++ b/lib/librte_eal/common/hotplug_mp.c
@@ -13,6 +13,11 @@
 
 #define MP_TIMEOUT_S 5 /**< 5 seconds timeouts */
 
+struct mp_reply_bundle {
+	struct rte_mp_msg msg;
+	void *peer;
+};
+
 static int cmp_dev_name(const struct rte_device *dev, const void *_name)
 {
 	const char *name = _name;
@@ -20,17 +25,196 @@ static int cmp_dev_name(const struct rte_device *dev, const void *_name)
 	return strcmp(dev->name, name);
 }
 
-struct mp_reply_bundle {
-	struct rte_mp_msg msg;
-	void *peer;
-};
+/**
+ * Secondary to primary request.
+ * start from function eal_dev_hotplug_request_to_primary.
+ *
+ * device attach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary receive the request and attach the new device if
+ *    failed goto i).
+ * c) primary forward attach sync request to all secondary.
+ * d) secondary receive the request and attach the device and send a reply.
+ * e) primary check the reply if all success goes to j).
+ * f) primary send attach rollback sync request to all secondary.
+ * g) secondary receive the request and detach the device and send a reply.
+ * h) primary receive the reply and detach device as rollback action.
+ * i) send attach fail to secondary as a reply of step a), goto k).
+ * j) send attach success to secondary as a reply of step a).
+ * k) secondary receive reply and return.
+ *
+ * device detach on secondary:
+ * a) secondary send sync request to the primary.
+ * b) primary send detach sync request to all secondary.
+ * c) secondary detach the device and send a reply.
+ * d) primary check the reply if all success goes to g).
+ * e) primary send detach rollback sync request to all secondary.
+ * f) secondary receive the request and attach back device. goto h).
+ * g) primary detach the device if success goto i), else goto e).
+ * h) primary send detach fail to secondary as a reply of step a), goto j).
+ * i) primary send detach success to secondary as a reply of step a).
+ * j) secondary receive reply and return.
+ */
+
+static int
+send_response_to_secondary(const struct eal_dev_mp_req *req,
+			int result,
+			const void *peer)
+{
+	struct rte_mp_msg mp_resp;
+	struct eal_dev_mp_req *resp =
+		(struct eal_dev_mp_req *)mp_resp.param;
+	int ret;
+
+	memset(&mp_resp, 0, sizeof(mp_resp));
+	mp_resp.len_param = sizeof(*resp);
+	strlcpy(mp_resp.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_resp.name));
+	memcpy(resp, req, sizeof(*req));
+	resp->result = result;
+
+	ret = rte_mp_reply(&mp_resp, peer);
+	if (ret != 0)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	return ret;
+}
+
+static void
+__handle_secondary_request(void *param)
+{
+	struct mp_reply_bundle *bundle = param;
+		const struct rte_mp_msg *msg = &bundle->msg;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	struct eal_dev_mp_req tmp_req;
+	struct rte_devargs *da;
+	struct rte_device *dev;
+	struct rte_bus *bus;
+	int ret = 0;
+
+	tmp_req = *req;
+
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		ret = local_dev_probe(req->devargs, &dev);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on primary\n");
+			if (ret != -EEXIST)
+				goto finish;
+		}
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+		if (tmp_req.result != 0) {
+			ret = tmp_req.result;
+			RTE_LOG(ERR, EAL, "Failed to hotplug add device on secondary\n");
+			if (ret != -EEXIST)
+				goto rollback;
+		}
+	} else if (req->t == EAL_DEV_REQ_TYPE_DETACH) {
+		da = calloc(1, sizeof(*da));
+		if (da == NULL) {
+			ret = -ENOMEM;
+			goto finish;
+		}
+
+		ret = rte_devargs_parse(da, req->devargs);
+		if (ret != 0)
+			goto finish;
+
+		ret = eal_dev_hotplug_request_to_secondary(&tmp_req);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Failed to send hotplug request to secondary\n");
+			ret = -ENOMSG;
+			goto rollback;
+		}
+
+		bus = rte_bus_find_by_name(da->bus->name);
+		if (bus == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", da->bus->name);
+			ret = -ENOENT;
+			goto finish;
+		}
+
+		dev = bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Cannot find plugged device (%s)\n", da->name);
+			ret = -ENOENT;
+			goto finish;
+		}
+
+		if (tmp_req.result != 0) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on secondary\n");
+			ret = tmp_req.result;
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+
+		ret = local_dev_remove(dev);
+		if (ret != 0) {
+			RTE_LOG(ERR, EAL, "Failed to hotplug remove device on primary\n");
+			if (ret != -ENOENT)
+				goto rollback;
+		}
+	} else {
+		RTE_LOG(ERR, EAL, "unsupported secondary to primary request\n");
+		ret = -ENOTSUP;
+	}
+	goto finish;
+
+rollback:
+	if (req->t == EAL_DEV_REQ_TYPE_ATTACH) {
+		tmp_req.t = EAL_DEV_REQ_TYPE_ATTACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+		local_dev_remove(dev);
+	} else {
+		tmp_req.t = EAL_DEV_REQ_TYPE_DETACH_ROLLBACK;
+		eal_dev_hotplug_request_to_secondary(&tmp_req);
+	}
+
+finish:
+	ret = send_response_to_secondary(&tmp_req, ret, bundle->peer);
+	if (ret)
+		RTE_LOG(ERR, EAL, "failed to send response to secondary\n");
+
+	free(bundle->peer);
+	free(bundle);
+}
 
 static int
 handle_secondary_request(const struct rte_mp_msg *msg, const void *peer)
 {
-	RTE_SET_USED(msg);
-	RTE_SET_USED(peer);
-	return -ENOTSUP;
+	struct mp_reply_bundle *bundle;
+	const struct eal_dev_mp_req *req =
+		(const struct eal_dev_mp_req *)msg->param;
+	int ret = 0;
+
+	bundle = malloc(sizeof(*bundle));
+	if (bundle == NULL) {
+		RTE_LOG(ERR, EAL, "not enough memory\n");
+		return send_response_to_secondary(req, -ENOMEM, peer);
+	}
+
+	bundle->msg = *msg;
+	/**
+	 * We need to send reply on interrupt thread, but peer can't be
+	 * parsed directly, so this is a temporal hack, need to be fixed
+	 * when it is ready.
+	 */
+	bundle->peer = strdup(peer);
+
+	/**
+	 * We are at IPC callback thread, sync IPC is not allowed due to
+	 * dead lock, so we delegate the task to interrupt thread.
+	 */
+	ret = rte_eal_alarm_set(1, __handle_secondary_request, bundle);
+	if (ret != 0) {
+		RTE_LOG(ERR, EAL, "failed to add mp task\n");
+		return send_response_to_secondary(req, ret, peer);
+	}
+	return 0;
 }
 
 static void __handle_primary_request(void *param)
@@ -149,8 +333,29 @@ handle_primary_request(const struct rte_mp_msg *msg, const void *peer)
 
 int eal_dev_hotplug_request_to_primary(struct eal_dev_mp_req *req)
 {
-	RTE_SET_USED(req);
-	return -ENOTSUP;
+	struct rte_mp_msg mp_req;
+	struct rte_mp_reply mp_reply;
+	struct timespec ts = {.tv_sec = MP_TIMEOUT_S, .tv_nsec = 0};
+	struct eal_dev_mp_req *resp;
+	int ret;
+
+	memset(&mp_req, 0, sizeof(mp_req));
+	memcpy(mp_req.param, req, sizeof(*req));
+	mp_req.len_param = sizeof(*req);
+	strlcpy(mp_req.name, EAL_DEV_MP_ACTION_REQUEST, sizeof(mp_req.name));
+
+	ret = rte_mp_request_sync(&mp_req, &mp_reply, &ts);
+	if (ret || mp_reply.nb_received != 1) {
+		RTE_LOG(ERR, EAL, "cannot send request to primary");
+		if (!ret)
+			return -1;
+		return ret;
+	}
+
+	resp = (struct eal_dev_mp_req *)mp_reply.msgs[0].param;
+	req->result = resp->result;
+
+	return ret;
 }
 
 int eal_dev_hotplug_request_to_secondary(struct eal_dev_mp_req *req)
-- 
2.13.6

[dpdk-dev] [PATCH v17 4/6] drivers/net: enable hotplug on secondary process

[Qi Zhang]

Attach port from secondary should ignore devargs since the private
device is not necessary to support. Also previously, detach port on
a secondary process will mess primary process and cause the same
device can't be attached back again. A secondary process should use
rte_eth_dev_release_port_secondary to release a port.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/af_packet/rte_eth_af_packet.c | 6 ++++--
 drivers/net/bonding/rte_eth_bond_pmd.c    | 6 ++++--
 drivers/net/kni/rte_eth_kni.c             | 6 ++++--
 drivers/net/null/rte_eth_null.c           | 6 ++++--
 drivers/net/octeontx/octeontx_ethdev.c    | 8 ++++++++
 drivers/net/pcap/rte_eth_pcap.c           | 6 ++++--
 drivers/net/tap/rte_eth_tap.c             | 8 +++++---
 drivers/net/vhost/rte_eth_vhost.c         | 6 ++++--
 8 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/drivers/net/af_packet/rte_eth_af_packet.c b/drivers/net/af_packet/rte_eth_af_packet.c
index bc7daed5e..376d76302 100644
--- a/drivers/net/af_packet/rte_eth_af_packet.c
+++ b/drivers/net/af_packet/rte_eth_af_packet.c
@@ -926,8 +926,7 @@ rte_pmd_af_packet_probe(struct rte_vdev_device *dev)
 
 	PMD_LOG(INFO, "Initializing pmd_af_packet for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -987,6 +986,9 @@ rte_pmd_af_packet_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	for (q = 0; q < internals->nb_queues; q++) {
 		rte_free(internals->rx_queue[q].rd);
diff --git a/drivers/net/bonding/rte_eth_bond_pmd.c b/drivers/net/bonding/rte_eth_bond_pmd.c
index 31c83a286..b731132a5 100644
--- a/drivers/net/bonding/rte_eth_bond_pmd.c
+++ b/drivers/net/bonding/rte_eth_bond_pmd.c
@@ -3152,8 +3152,7 @@ bond_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	RTE_BOND_LOG(INFO, "Initializing pmd_bond for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			RTE_BOND_LOG(ERR, "Failed to probe %s", name);
@@ -3268,6 +3267,9 @@ bond_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	RTE_ASSERT(eth_dev->device == &dev->device);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/kni/rte_eth_kni.c b/drivers/net/kni/rte_eth_kni.c
index 8a7015a0b..72f3c16c1 100644
--- a/drivers/net/kni/rte_eth_kni.c
+++ b/drivers/net/kni/rte_eth_kni.c
@@ -410,8 +410,7 @@ eth_kni_probe(struct rte_vdev_device *vdev)
 	params = rte_vdev_device_args(vdev);
 	PMD_LOG(INFO, "Initializing eth_kni for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -464,6 +463,9 @@ eth_kni_remove(struct rte_vdev_device *vdev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_kni_dev_stop(eth_dev);
 
 	internals = eth_dev->data->dev_private;
diff --git a/drivers/net/null/rte_eth_null.c b/drivers/net/null/rte_eth_null.c
index de10b5bdf..1e8237a41 100644
--- a/drivers/net/null/rte_eth_null.c
+++ b/drivers/net/null/rte_eth_null.c
@@ -614,8 +614,7 @@ rte_pmd_null_probe(struct rte_vdev_device *dev)
 	params = rte_vdev_device_args(dev);
 	PMD_LOG(INFO, "Initializing pmd_null for %s", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -680,6 +679,9 @@ rte_pmd_null_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	rte_free(eth_dev->data->dev_private);
 
 	rte_eth_dev_release_port(eth_dev);
diff --git a/drivers/net/octeontx/octeontx_ethdev.c b/drivers/net/octeontx/octeontx_ethdev.c
index 71843c63a..5431b44dc 100644
--- a/drivers/net/octeontx/octeontx_ethdev.c
+++ b/drivers/net/octeontx/octeontx_ethdev.c
@@ -1133,6 +1133,11 @@ octeontx_remove(struct rte_vdev_device *dev)
 		if (eth_dev == NULL)
 			return -ENODEV;
 
+		if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+			rte_eth_dev_release_port_secondary(eth_dev);
+			continue;
+		}
+
 		nic = octeontx_pmd_priv(eth_dev);
 		rte_event_dev_stop(nic->evdev);
 		PMD_INIT_LOG(INFO, "Closing octeontx device %s", octtx_name);
@@ -1143,6 +1148,9 @@ octeontx_remove(struct rte_vdev_device *dev)
 		rte_event_dev_close(nic->evdev);
 	}
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return 0;
+
 	/* Free FC resource */
 	octeontx_pko_fc_free();
 
diff --git a/drivers/net/pcap/rte_eth_pcap.c b/drivers/net/pcap/rte_eth_pcap.c
index 51d405116..ede5ebb45 100644
--- a/drivers/net/pcap/rte_eth_pcap.c
+++ b/drivers/net/pcap/rte_eth_pcap.c
@@ -1122,8 +1122,7 @@ pmd_pcap_probe(struct rte_vdev_device *dev)
 	start_cycles = rte_get_timer_cycles();
 	hz = rte_get_timer_hz();
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			PMD_LOG(ERR, "Failed to probe %s", name);
@@ -1229,6 +1228,9 @@ pmd_pcap_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -1;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	internals = eth_dev->data->dev_private;
 	if (internals && internals->phy_mac)
 		rte_free(eth_dev->data->mac_addrs);
diff --git a/drivers/net/tap/rte_eth_tap.c b/drivers/net/tap/rte_eth_tap.c
index ad5ae988b..84aaf2410 100644
--- a/drivers/net/tap/rte_eth_tap.c
+++ b/drivers/net/tap/rte_eth_tap.c
@@ -1992,8 +1992,7 @@ rte_pmd_tap_probe(struct rte_vdev_device *dev)
 	name = rte_vdev_device_name(dev);
 	params = rte_vdev_device_args(dev);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(params) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			TAP_LOG(ERR, "Failed to probe %s", name);
@@ -2075,7 +2074,10 @@ rte_pmd_tap_remove(struct rte_vdev_device *dev)
 	/* find the ethdev entry */
 	eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
 	if (!eth_dev)
-		return 0;
+		return -ENODEV;
+
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
 
 	internals = eth_dev->data->dev_private;
 
diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index aa6052221..a7604ff23 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1344,8 +1344,7 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 
 	VHOST_LOG(INFO, "Initializing pmd_vhost for %s\n", name);
 
-	if (rte_eal_process_type() == RTE_PROC_SECONDARY &&
-	    strlen(rte_vdev_device_args(dev)) == 0) {
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
 		eth_dev = rte_eth_dev_attach_secondary(name);
 		if (!eth_dev) {
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
@@ -1436,6 +1435,9 @@ rte_pmd_vhost_remove(struct rte_vdev_device *dev)
 	if (eth_dev == NULL)
 		return -ENODEV;
 
+	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
+		return rte_eth_dev_release_port_secondary(eth_dev);
+
 	eth_dev_close(eth_dev);
 
 	rte_free(vring_states[eth_dev->data->port_id]);
-- 
2.13.6

[dpdk-dev] [PATCH v17 5/6] drivers/net: enable device detach on secondary

[Qi Zhang]

With the enabling for hotplug on multi-process,
rte_eth_dev_pci_generic_remove can be used to detach the device from
a secondary process also. But we need to take care of the uninit callback
parameter to make sure it handles the secondary case correctly.

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/bnxt/bnxt_ethdev.c     | 6 +++++-
 drivers/net/ena/ena_ethdev.c       | 2 +-
 drivers/net/liquidio/lio_ethdev.c  | 2 +-
 drivers/net/virtio/virtio_ethdev.c | 2 +-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c
index 858e65d00..c11fe9c8c 100644
--- a/drivers/net/bnxt/bnxt_ethdev.c
+++ b/drivers/net/bnxt/bnxt_ethdev.c
@@ -3542,7 +3542,11 @@ static int bnxt_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 
 static int bnxt_pci_remove(struct rte_pci_device *pci_dev)
 {
-	return rte_eth_dev_pci_generic_remove(pci_dev, bnxt_dev_uninit);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		return rte_eth_dev_pci_generic_remove(pci_dev,
+				bnxt_dev_uninit);
+	else
+		return rte_eth_dev_pci_generic_remove(pci_dev, NULL);
 }
 
 static struct rte_pci_driver bnxt_rte_pmd = {
diff --git a/drivers/net/ena/ena_ethdev.c b/drivers/net/ena/ena_ethdev.c
index c255dc6db..c29a581e8 100644
--- a/drivers/net/ena/ena_ethdev.c
+++ b/drivers/net/ena/ena_ethdev.c
@@ -1703,7 +1703,7 @@ static int eth_ena_dev_uninit(struct rte_eth_dev *eth_dev)
 		(struct ena_adapter *)(eth_dev->data->dev_private);
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	if (adapter->state != ENA_ADAPTER_STATE_CLOSED)
 		ena_close(eth_dev);
diff --git a/drivers/net/liquidio/lio_ethdev.c b/drivers/net/liquidio/lio_ethdev.c
index 93e89007a..0f59e4475 100644
--- a/drivers/net/liquidio/lio_ethdev.c
+++ b/drivers/net/liquidio/lio_ethdev.c
@@ -2038,7 +2038,7 @@ lio_eth_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() != RTE_PROC_PRIMARY)
-		return -EPERM;
+		return 0;
 
 	/* lio_free_sc_buffer_pool */
 	lio_free_sc_buffer_pool(lio_dev);
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index b81df0a99..730c41707 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1697,7 +1697,7 @@ eth_virtio_dev_uninit(struct rte_eth_dev *eth_dev)
 	PMD_INIT_FUNC_TRACE();
 
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY)
-		return -EPERM;
+		return 0;
 
 	virtio_dev_stop(eth_dev);
 	virtio_dev_close(eth_dev);
-- 
2.13.6

[dpdk-dev] [PATCH v17 6/6] examples/multi_process: add hotplug sample

[Qi Zhang]

The sample code demonstrates device (ethdev only) management
at a multi-process environment. The user can attach/detach a
device on primary process and see it is synced on secondary
process automatically.

How to start?
./hotplug_mp --proc-type=auto

Command Line Example:

>help
>list

/* attach a pci device */
> attach 0000:81:00.0

/* detach the pci device */
> detach 0000:81:00.0

/* attach a vdev af_packet device */
> attach net_af_packet,iface=eth0

/* detach the vdev af_packet device */
> detach net_af_packet

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 examples/multi_process/Makefile              |   1 +
 examples/multi_process/hotplug_mp/Makefile   |  23 +++
 examples/multi_process/hotplug_mp/commands.c | 214 +++++++++++++++++++++++++++
 examples/multi_process/hotplug_mp/commands.h |  10 ++
 examples/multi_process/hotplug_mp/main.c     |  41 +++++
 5 files changed, 289 insertions(+)
 create mode 100644 examples/multi_process/hotplug_mp/Makefile
 create mode 100644 examples/multi_process/hotplug_mp/commands.c
 create mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/main.c

diff --git a/examples/multi_process/Makefile b/examples/multi_process/Makefile
index a6708b7e4..b76b02fcb 100644
--- a/examples/multi_process/Makefile
+++ b/examples/multi_process/Makefile
@@ -13,5 +13,6 @@ include $(RTE_SDK)/mk/rte.vars.mk
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += client_server_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += simple_mp
 DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += symmetric_mp
+DIRS-$(CONFIG_RTE_EXEC_ENV_LINUXAPP) += hotplug_mp
 
 include $(RTE_SDK)/mk/rte.extsubdir.mk
diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
new file mode 100644
index 000000000..bc36aeaed
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# binary name
+APP = hotplug_mp
+
+# all source are stored in SRCS-y
+SRCS-y := main.c commands.c
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
new file mode 100644
index 000000000..b06859393
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -0,0 +1,214 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation.
+ */
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_parse_ipaddr.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+#include <rte_ethdev.h>
+
+/**********************************************************/
+
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_printf(cl,
+		       "commands:\n"
+		       "- attach <devargs>\n"
+		       "- detach <devargs>\n"
+		       "- list\n\n");
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "show help",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "quit",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_list_result {
+	cmdline_fixed_string_t list;
+};
+
+static void cmd_list_parsed(__attribute__((unused)) void *parsed_result,
+			    struct cmdline *cl,
+			    __attribute__((unused)) void *data)
+{
+	uint16_t port_id;
+	char dev_name[RTE_DEV_NAME_MAX_LEN];
+
+	cmdline_printf(cl, "list all etherdev\n");
+
+	RTE_ETH_FOREACH_DEV(port_id) {
+		rte_eth_dev_get_name_by_port(port_id, dev_name);
+		if (strlen(dev_name) > 0)
+			cmdline_printf(cl, "%d\t%s\n", port_id, dev_name);
+		else
+			printf("empty dev_name is not expected!\n");
+	}
+}
+
+cmdline_parse_token_string_t cmd_list_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
+
+cmdline_parse_inst_t cmd_list = {
+	.f = cmd_list_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_list_list,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_attach_result {
+	cmdline_fixed_string_t attach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_attach_parsed(void *parsed_result,
+				  struct cmdline *cl,
+				  __attribute__((unused)) void *data)
+{
+	struct cmd_dev_attach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	if (!rte_eal_hotplug_add(da.bus->name, da.name, da.args))
+		cmdline_printf(cl, "attached device %s\n", da.name);
+	else
+		cmdline_printf(cl, "failed to attached device %s\n",
+				da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_attach_attach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
+				 "attach");
+cmdline_parse_token_string_t cmd_dev_attach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_attach_device = {
+	.f = cmd_dev_attach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "attach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_attach_attach,
+		(void *)&cmd_dev_attach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+
+struct cmd_dev_detach_result {
+	cmdline_fixed_string_t detach;
+	cmdline_fixed_string_t devargs;
+};
+
+static void cmd_dev_detach_parsed(void *parsed_result,
+				   struct cmdline *cl,
+				   __attribute__((unused)) void *data)
+{
+	struct cmd_dev_detach_result *res = parsed_result;
+	struct rte_devargs da;
+
+	memset(&da, 0, sizeof(da));
+
+	if (rte_devargs_parsef(&da, "%s", res->devargs)) {
+		cmdline_printf(cl, "cannot parse devargs\n");
+		if (da.args)
+			free(da.args);
+		return;
+	}
+
+	printf("detaching...\n");
+	if (!rte_eal_hotplug_remove(da.bus->name, da.name))
+		cmdline_printf(cl, "detached device %s\n",
+			da.name);
+	else
+		cmdline_printf(cl, "failed to dettach device %s\n",
+			da.name);
+}
+
+cmdline_parse_token_string_t cmd_dev_detach_detach =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
+				 "detach");
+
+cmdline_parse_token_string_t cmd_dev_detach_devargs =
+	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
+
+cmdline_parse_inst_t cmd_detach_device = {
+	.f = cmd_dev_detach_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "detach a device",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_dev_detach_detach,
+		(void *)&cmd_dev_detach_devargs,
+		NULL,
+	},
+};
+
+/**********************************************************/
+/**********************************************************/
+/****** CONTEXT (list of instruction) */
+
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	(cmdline_parse_inst_t *)&cmd_list,
+	(cmdline_parse_inst_t *)&cmd_attach_device,
+	(cmdline_parse_inst_t *)&cmd_detach_device,
+	NULL,
+};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
new file mode 100644
index 000000000..afcf177db
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#ifndef _COMMANDS_H_
+#define _COMMANDS_H_
+
+extern cmdline_parse_ctx_t main_ctx[];
+
+#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/main.c b/examples/multi_process/hotplug_mp/main.c
new file mode 100644
index 000000000..d66858078
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/main.c
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdint.h>
+#include <errno.h>
+#include <termios.h>
+#include <sys/queue.h>
+
+#include <cmdline_rdline.h>
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline.h>
+
+#include <rte_memory.h>
+#include <rte_eal.h>
+#include <rte_debug.h>
+
+#include "commands.h"
+
+int main(int argc, char **argv)
+{
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_panic("Cannot init EAL\n");
+
+	cl = cmdline_stdin_new(main_ctx, "example> ");
+	if (cl == NULL)
+		rte_panic("Cannot create cmdline instance\n");
+	cmdline_interact(cl);
+	cmdline_stdin_exit(cl);
+
+	rte_eal_cleanup();
+
+	return 0;
+}
-- 
2.13.6

Re: [dpdk-dev] [PATCH v17 0/6] enable hotplug on multi-process

[Thomas Monjalon]

> Qi Zhang (6):
>   ethdev: add function to release port in secondary process
>   eal: enable hotplug on multi-process
>   eal: support attach or detach share device from  secondary
>   drivers/net: enable hotplug on secondary process
>   drivers/net: enable device detach on secondary
>   examples/multi_process: add hotplug sample
> 
>  doc/guides/rel_notes/release_18_11.rst       |  13 +
>  drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
>  drivers/net/bnxt/bnxt_ethdev.c               |   6 +-
>  drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
>  drivers/net/ena/ena_ethdev.c                 |   2 +-
>  drivers/net/kni/rte_eth_kni.c                |   6 +-
>  drivers/net/liquidio/lio_ethdev.c            |   2 +-
>  drivers/net/null/rte_eth_null.c              |   6 +-
>  drivers/net/octeontx/octeontx_ethdev.c       |   8 +
>  drivers/net/pcap/rte_eth_pcap.c              |   6 +-
>  drivers/net/tap/rte_eth_tap.c                |   8 +-
>  drivers/net/vhost/rte_eth_vhost.c            |   6 +-
>  drivers/net/virtio/virtio_ethdev.c           |   2 +-
>  examples/multi_process/Makefile              |   1 +
>  examples/multi_process/hotplug_mp/Makefile   |  23 ++
>  examples/multi_process/hotplug_mp/commands.c | 214 ++++++++++++++
>  examples/multi_process/hotplug_mp/commands.h |  10 +
>  examples/multi_process/hotplug_mp/main.c     |  41 +++
>  lib/librte_eal/bsdapp/eal/Makefile           |   1 +
>  lib/librte_eal/common/eal_common_dev.c       | 254 ++++++++++++++--
>  lib/librte_eal/common/eal_private.h          |  22 ++
>  lib/librte_eal/common/hotplug_mp.c           | 426 +++++++++++++++++++++++++++
>  lib/librte_eal/common/hotplug_mp.h           |  46 +++
>  lib/librte_eal/common/include/rte_dev.h      |  12 +
>  lib/librte_eal/common/include/rte_eal.h      |   9 +
>  lib/librte_eal/common/meson.build            |   1 +
>  lib/librte_eal/linuxapp/eal/Makefile         |   1 +
>  lib/librte_eal/linuxapp/eal/eal.c            |   6 +
>  lib/librte_ethdev/rte_ethdev.c               |  17 +-
>  lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
>  lib/librte_ethdev/rte_ethdev_pci.h           |  10 +-
>  lib/librte_ethdev/rte_ethdev_version.map     |   7 +
>  32 files changed, 1151 insertions(+), 43 deletions(-)

Applied, thanks.

Note: some PMDs are updated to support multi-process hotplug,
but not all of them. I think you focused on the generic ones
(af_packet, pcap, tap, bonding), but you forgot failsafe.
Please, could you follow-up in -rc2? Thank you

Re: [dpdk-dev] [PATCH v17 0/6] enable hotplug on multi-process

[Zhang, Qi Z]

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, October 16, 2018 3:52 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>
> Cc: dev@dpdk.org; gaetan.rivet@6wind.com; Burakov, Anatoly
> <anatoly.burakov@intel.com>; arybchenko@solarflare.com; Ananyev,
> Konstantin <konstantin.ananyev@intel.com>; Richardson, Bruce
> <bruce.richardson@intel.com>; Yigit, Ferruh <ferruh.yigit@intel.com>;
> Shelton, Benjamin H <benjamin.h.shelton@intel.com>; Vangati, Narender
> <narender.vangati@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v17 0/6] enable hotplug on multi-process
> 
> > Qi Zhang (6):
> >   ethdev: add function to release port in secondary process
> >   eal: enable hotplug on multi-process
> >   eal: support attach or detach share device from  secondary
> >   drivers/net: enable hotplug on secondary process
> >   drivers/net: enable device detach on secondary
> >   examples/multi_process: add hotplug sample
> >
> >  doc/guides/rel_notes/release_18_11.rst       |  13 +
> >  drivers/net/af_packet/rte_eth_af_packet.c    |   6 +-
> >  drivers/net/bnxt/bnxt_ethdev.c               |   6 +-
> >  drivers/net/bonding/rte_eth_bond_pmd.c       |   6 +-
> >  drivers/net/ena/ena_ethdev.c                 |   2 +-
> >  drivers/net/kni/rte_eth_kni.c                |   6 +-
> >  drivers/net/liquidio/lio_ethdev.c            |   2 +-
> >  drivers/net/null/rte_eth_null.c              |   6 +-
> >  drivers/net/octeontx/octeontx_ethdev.c       |   8 +
> >  drivers/net/pcap/rte_eth_pcap.c              |   6 +-
> >  drivers/net/tap/rte_eth_tap.c                |   8 +-
> >  drivers/net/vhost/rte_eth_vhost.c            |   6 +-
> >  drivers/net/virtio/virtio_ethdev.c           |   2 +-
> >  examples/multi_process/Makefile              |   1 +
> >  examples/multi_process/hotplug_mp/Makefile   |  23 ++
> >  examples/multi_process/hotplug_mp/commands.c | 214
> ++++++++++++++
> > examples/multi_process/hotplug_mp/commands.h |  10 +
> >  examples/multi_process/hotplug_mp/main.c     |  41 +++
> >  lib/librte_eal/bsdapp/eal/Makefile           |   1 +
> >  lib/librte_eal/common/eal_common_dev.c       | 254
> ++++++++++++++--
> >  lib/librte_eal/common/eal_private.h          |  22 ++
> >  lib/librte_eal/common/hotplug_mp.c           | 426
> +++++++++++++++++++++++++++
> >  lib/librte_eal/common/hotplug_mp.h           |  46 +++
> >  lib/librte_eal/common/include/rte_dev.h      |  12 +
> >  lib/librte_eal/common/include/rte_eal.h      |   9 +
> >  lib/librte_eal/common/meson.build            |   1 +
> >  lib/librte_eal/linuxapp/eal/Makefile         |   1 +
> >  lib/librte_eal/linuxapp/eal/eal.c            |   6 +
> >  lib/librte_ethdev/rte_ethdev.c               |  17 +-
> >  lib/librte_ethdev/rte_ethdev_driver.h        |  16 +-
> >  lib/librte_ethdev/rte_ethdev_pci.h           |  10 +-
> >  lib/librte_ethdev/rte_ethdev_version.map     |   7 +
> >  32 files changed, 1151 insertions(+), 43 deletions(-)
> 
> Applied, thanks.

Thank! 

> 
> Note: some PMDs are updated to support multi-process hotplug, but not all
> of them. I think you focused on the generic ones (af_packet, pcap, tap,
> bonding), but you forgot failsafe.
> Please, could you follow-up in -rc2? Thank you

As I remember, I've been informed that failsafe is not going to support secondary process, so I remove it as requested.

Hi Gaetan:
	Would you comment on this?
Thanks
Qi

>