[dpdk-stable] [PATCH v3 0/4] make vhost PMD available for secondary processes

[dpdk-stable] [PATCH v3 0/4] make vhost PMD available for secondary processes

[Itsuro Oda]

vhost PMD has not been available for secondary processes since
DPDK v18.11.  (https://bugs.dpdk.org/show_bug.cgi?id=194)
(for a long term !)
This series of patches intend to make vhost PMD available for
secondary processes.
Because now setting vhost driver to communicate with a vhost-user
master (ex. Qemu) is accomplished by the probe function of the
primary process, only the primary process can be a vhost-user
slave.
With this patch, setting vhost driver is delayed at eth_dev
configuration in order to be able to set it from a secondary
process. Because (in the first place,) setting vhost driver is not
necessary to be done at probe (it is enough to be done up to eth_dev
start), this fix is no problem for the primary process.
There is a precondition that the same process has to operate
a vhost interface from beginning to end (from eth_dev configuration
to eth_dev close). (This patch leaves it to user's responsibility.)
This precondition will not be a problem in most use cases
(including SPP).

v2:
- add signed-off-by
- fix spelling error

v3:
- rebase on dpdk-next-virtio master
- change patch order
- fix subject and commit message

Itsuro Oda (4):
  net/vhost: allocate interface name from heap
  net/vhost: delay vhost driver setup
  net/vhost: make secondary probe complete
  net/vhost: remove an unused member

 drivers/net/vhost/rte_eth_vhost.c | 152 +++++++++++++++++-------------
 1 file changed, 88 insertions(+), 64 deletions(-)

-- 
2.17.0

[dpdk-stable] [PATCH v3 1/4] net/vhost: allocate interface name from heap

[Itsuro Oda]

This patch allocates iface_name of pmd_internal from heap
in order to be able to refer from secondary processes.

Fixes: 4852aa8f6e21 (drivers/net: enable hotplug on secondary process)
Cc: stable@dpdk.org

Signed-off-by: Itsuro Oda <oda@valinux.co.jp>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index a63588986..cea2ead2d 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1009,7 +1009,7 @@ eth_dev_close(struct rte_eth_dev *dev)
 			rte_free(dev->data->tx_queues[i]);
 
 	free(internal->dev_name);
-	free(internal->iface_name);
+	rte_free(internal->iface_name);
 	rte_free(internal);
 
 	dev->data->dev_private = NULL;
@@ -1256,9 +1256,11 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	internal->dev_name = strdup(name);
 	if (internal->dev_name == NULL)
 		goto error;
-	internal->iface_name = strdup(iface_name);
+	internal->iface_name = rte_malloc_socket(name, strlen(iface_name) + 1,
+						 0, numa_node);
 	if (internal->iface_name == NULL)
 		goto error;
+	strcpy(internal->iface_name, iface_name);
 
 	list->eth_dev = eth_dev;
 	pthread_mutex_lock(&internal_list_lock);
@@ -1306,7 +1308,7 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 
 error:
 	if (internal) {
-		free(internal->iface_name);
+		rte_free(internal->iface_name);
 		free(internal->dev_name);
 	}
 	rte_free(vring_state);
-- 
2.17.0

[dpdk-stable] [PATCH v3 2/4] net/vhost: delay vhost driver setup

[Itsuro Oda]

Vhost driver setup is delayed at eth_dev configuration
in order to be able to set it from a secondary process.

Fixes: 4852aa8f6e21 (drivers/net: enable hotplug on secondary process)
Cc: stable@dpdk.org

Signed-off-by: Itsuro Oda <oda@valinux.co.jp>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 130 ++++++++++++++++++------------
 1 file changed, 78 insertions(+), 52 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index cea2ead2d..d7bba5c6e 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -97,6 +97,8 @@ struct pmd_internal {
 	rte_atomic32_t dev_attached;
 	char *dev_name;
 	char *iface_name;
+	uint64_t flags;
+	uint64_t disable_flags;
 	uint16_t max_queues;
 	int vid;
 	rte_atomic32_t started;
@@ -491,17 +493,6 @@ eth_vhost_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
 	return nb_tx;
 }
 
-static int
-eth_dev_configure(struct rte_eth_dev *dev __rte_unused)
-{
-	struct pmd_internal *internal = dev->data->dev_private;
-	const struct rte_eth_rxmode *rxmode = &dev->data->dev_conf.rxmode;
-
-	internal->vlan_strip = !!(rxmode->offloads & DEV_RX_OFFLOAD_VLAN_STRIP);
-
-	return 0;
-}
-
 static inline struct internal_list *
 find_internal_resource(char *ifname)
 {
@@ -877,6 +868,62 @@ static struct vhost_device_ops vhost_ops = {
 	.vring_state_changed = vring_state_changed,
 };
 
+static int
+vhost_driver_setup(struct rte_eth_dev *eth_dev)
+{
+	struct pmd_internal *internal = eth_dev->data->dev_private;
+	struct internal_list *list = NULL;
+	struct rte_vhost_vring_state *vring_state = NULL;
+	unsigned int numa_node = eth_dev->device->numa_node;
+	const char *name = eth_dev->device->name;
+
+	list = rte_zmalloc_socket(name, sizeof(*list), 0, numa_node);
+	if (list == NULL)
+		goto error;
+
+	vring_state = rte_zmalloc_socket(name, sizeof(*vring_state),
+					 0, numa_node);
+	if (vring_state == NULL)
+		goto error;
+
+	list->eth_dev = eth_dev;
+	pthread_mutex_lock(&internal_list_lock);
+	TAILQ_INSERT_TAIL(&internal_list, list, next);
+	pthread_mutex_unlock(&internal_list_lock);
+
+	rte_spinlock_init(&vring_state->lock);
+	vring_states[eth_dev->data->port_id] = vring_state;
+
+	if (rte_vhost_driver_register(internal->iface_name, internal->flags))
+		goto error;
+
+	if (internal->disable_flags) {
+		if (rte_vhost_driver_disable_features(internal->iface_name,
+						      internal->disable_flags))
+			goto error;
+	}
+
+	if (rte_vhost_driver_callback_register(internal->iface_name,
+					       &vhost_ops) < 0) {
+		VHOST_LOG(ERR, "Can't register callbacks\n");
+		goto error;
+	}
+
+	if (rte_vhost_driver_start(internal->iface_name) < 0) {
+		VHOST_LOG(ERR, "Failed to start driver for %s\n",
+			  internal->iface_name);
+		goto error;
+	}
+
+	return 0;
+
+error:
+	rte_free(vring_state);
+	rte_free(list);
+
+	return -1;
+}
+
 int
 rte_eth_vhost_get_queue_event(uint16_t port_id,
 		struct rte_eth_vhost_queue_event *event)
@@ -943,6 +990,24 @@ rte_eth_vhost_get_vid_from_port_id(uint16_t port_id)
 	return vid;
 }
 
+static int
+eth_dev_configure(struct rte_eth_dev *dev)
+{
+	struct pmd_internal *internal = dev->data->dev_private;
+	const struct rte_eth_rxmode *rxmode = &dev->data->dev_conf.rxmode;
+
+	/* NOTE: the same process has to operate a vhost interface
+	 * from beginning to end (from eth_dev configure to eth_dev close).
+	 * It is user's responsibility at the moment.
+	 */
+	if (vhost_driver_setup(dev) < 0)
+		return -1;
+
+	internal->vlan_strip = !!(rxmode->offloads & DEV_RX_OFFLOAD_VLAN_STRIP);
+
+	return 0;
+}
+
 static int
 eth_dev_start(struct rte_eth_dev *eth_dev)
 {
@@ -1219,16 +1284,10 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	struct pmd_internal *internal = NULL;
 	struct rte_eth_dev *eth_dev = NULL;
 	struct rte_ether_addr *eth_addr = NULL;
-	struct rte_vhost_vring_state *vring_state = NULL;
-	struct internal_list *list = NULL;
 
 	VHOST_LOG(INFO, "Creating VHOST-USER backend on numa socket %u\n",
 		numa_node);
 
-	list = rte_zmalloc_socket(name, sizeof(*list), 0, numa_node);
-	if (list == NULL)
-		goto error;
-
 	/* reserve an ethdev entry */
 	eth_dev = rte_eth_vdev_allocate(dev, sizeof(*internal));
 	if (eth_dev == NULL)
@@ -1242,11 +1301,6 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	*eth_addr = base_eth_addr;
 	eth_addr->addr_bytes[5] = eth_dev->data->port_id;
 
-	vring_state = rte_zmalloc_socket(name,
-			sizeof(*vring_state), 0, numa_node);
-	if (vring_state == NULL)
-		goto error;
-
 	/* now put it all together
 	 * - store queue data in internal,
 	 * - point eth_dev_data to internals
@@ -1262,18 +1316,12 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 		goto error;
 	strcpy(internal->iface_name, iface_name);
 
-	list->eth_dev = eth_dev;
-	pthread_mutex_lock(&internal_list_lock);
-	TAILQ_INSERT_TAIL(&internal_list, list, next);
-	pthread_mutex_unlock(&internal_list_lock);
-
-	rte_spinlock_init(&vring_state->lock);
-	vring_states[eth_dev->data->port_id] = vring_state;
-
 	data->nb_rx_queues = queues;
 	data->nb_tx_queues = queues;
 	internal->max_queues = queues;
 	internal->vid = -1;
+	internal->flags = flags;
+	internal->disable_flags = disable_flags;
 	data->dev_link = pmd_link;
 	data->dev_flags = RTE_ETH_DEV_INTR_LSC | RTE_ETH_DEV_CLOSE_REMOVE;
 
@@ -1283,26 +1331,6 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	eth_dev->rx_pkt_burst = eth_vhost_rx;
 	eth_dev->tx_pkt_burst = eth_vhost_tx;
 
-	if (rte_vhost_driver_register(iface_name, flags))
-		goto error;
-
-	if (disable_flags) {
-		if (rte_vhost_driver_disable_features(iface_name,
-					disable_flags))
-			goto error;
-	}
-
-	if (rte_vhost_driver_callback_register(iface_name, &vhost_ops) < 0) {
-		VHOST_LOG(ERR, "Can't register callbacks\n");
-		goto error;
-	}
-
-	if (rte_vhost_driver_start(iface_name) < 0) {
-		VHOST_LOG(ERR, "Failed to start driver for %s\n",
-			iface_name);
-		goto error;
-	}
-
 	rte_eth_dev_probing_finish(eth_dev);
 	return 0;
 
@@ -1311,9 +1339,7 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 		rte_free(internal->iface_name);
 		free(internal->dev_name);
 	}
-	rte_free(vring_state);
 	rte_eth_dev_release_port(eth_dev);
-	rte_free(list);
 
 	return -1;
 }
-- 
2.17.0

[dpdk-stable] [PATCH v3 3/4] net/vhost: make secondary probe complete

[Itsuro Oda]

This patch adds lacking member setting and makes secondary
probe complete.

Fixes: 4852aa8f6e21 (drivers/net: enable hotplug on secondary process)
Cc: stable@dpdk.org

Signed-off-by: Itsuro Oda <oda@valinux.co.jp>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index d7bba5c6e..307de2c68 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -1397,8 +1397,11 @@ rte_pmd_vhost_probe(struct rte_vdev_device *dev)
 			VHOST_LOG(ERR, "Failed to probe %s\n", name);
 			return -1;
 		}
-		/* TODO: request info from primary to set up Rx and Tx */
+		eth_dev->rx_pkt_burst = eth_vhost_rx;
+		eth_dev->tx_pkt_burst = eth_vhost_tx;
 		eth_dev->dev_ops = &ops;
+		if (dev->device.numa_node == SOCKET_ID_ANY)
+			dev->device.numa_node = rte_socket_id();
 		eth_dev->device = &dev->device;
 		rte_eth_dev_probing_finish(eth_dev);
 		return 0;
-- 
2.17.0

[dpdk-stable] [PATCH v3 4/4] net/vhost: remove an unused member

[Itsuro Oda]

This patch removes an unused member from pmd_internal.

Signed-off-by: Itsuro Oda <oda@valinux.co.jp>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/net/vhost/rte_eth_vhost.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
index 307de2c68..90263ae77 100644
--- a/drivers/net/vhost/rte_eth_vhost.c
+++ b/drivers/net/vhost/rte_eth_vhost.c
@@ -95,7 +95,6 @@ struct vhost_queue {
 
 struct pmd_internal {
 	rte_atomic32_t dev_attached;
-	char *dev_name;
 	char *iface_name;
 	uint64_t flags;
 	uint64_t disable_flags;
@@ -1073,7 +1072,6 @@ eth_dev_close(struct rte_eth_dev *dev)
 		for (i = 0; i < dev->data->nb_tx_queues; i++)
 			rte_free(dev->data->tx_queues[i]);
 
-	free(internal->dev_name);
 	rte_free(internal->iface_name);
 	rte_free(internal);
 
@@ -1307,9 +1305,6 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	 * - and point eth_dev structure to new eth_dev_data structure
 	 */
 	internal = eth_dev->data->dev_private;
-	internal->dev_name = strdup(name);
-	if (internal->dev_name == NULL)
-		goto error;
 	internal->iface_name = rte_malloc_socket(name, strlen(iface_name) + 1,
 						 0, numa_node);
 	if (internal->iface_name == NULL)
@@ -1335,10 +1330,8 @@ eth_dev_vhost_create(struct rte_vdev_device *dev, char *iface_name,
 	return 0;
 
 error:
-	if (internal) {
+	if (internal)
 		rte_free(internal->iface_name);
-		free(internal->dev_name);
-	}
 	rte_eth_dev_release_port(eth_dev);
 
 	return -1;
-- 
2.17.0

Re: [dpdk-stable] [PATCH v3 0/4] make vhost PMD available for secondary processes

[Maxime Coquelin]

On 2/6/20 2:39 AM, Itsuro Oda wrote:
> vhost PMD has not been available for secondary processes since
> DPDK v18.11.  (https://bugs.dpdk.org/show_bug.cgi?id=194)
> (for a long term !)
> This series of patches intend to make vhost PMD available for
> secondary processes.
> Because now setting vhost driver to communicate with a vhost-user
> master (ex. Qemu) is accomplished by the probe function of the
> primary process, only the primary process can be a vhost-user
> slave.
> With this patch, setting vhost driver is delayed at eth_dev
> configuration in order to be able to set it from a secondary
> process. Because (in the first place,) setting vhost driver is not
> necessary to be done at probe (it is enough to be done up to eth_dev
> start), this fix is no problem for the primary process.
> There is a precondition that the same process has to operate
> a vhost interface from beginning to end (from eth_dev configuration
> to eth_dev close). (This patch leaves it to user's responsibility.)
> This precondition will not be a problem in most use cases
> (including SPP).
> 
> v2:
> - add signed-off-by
> - fix spelling error
> 
> v3:
> - rebase on dpdk-next-virtio master
> - change patch order
> - fix subject and commit message
> 
> Itsuro Oda (4):
>   net/vhost: allocate interface name from heap
>   net/vhost: delay vhost driver setup
>   net/vhost: make secondary probe complete
>   net/vhost: remove an unused member
> 
>  drivers/net/vhost/rte_eth_vhost.c | 152 +++++++++++++++++-------------
>  1 file changed, 88 insertions(+), 64 deletions(-)
> 

Thanks Itsuro, it looks good to me.
I will pull the series for v20.02-rc3.

Maxime

Re: [dpdk-stable] [PATCH v3 0/4] make vhost PMD available for secondary processes

[Maxime Coquelin]

On 2/6/20 2:39 AM, Itsuro Oda wrote:
> vhost PMD has not been available for secondary processes since
> DPDK v18.11.  (https://bugs.dpdk.org/show_bug.cgi?id=194)
> (for a long term !)
> This series of patches intend to make vhost PMD available for
> secondary processes.
> Because now setting vhost driver to communicate with a vhost-user
> master (ex. Qemu) is accomplished by the probe function of the
> primary process, only the primary process can be a vhost-user
> slave.
> With this patch, setting vhost driver is delayed at eth_dev
> configuration in order to be able to set it from a secondary
> process. Because (in the first place,) setting vhost driver is not
> necessary to be done at probe (it is enough to be done up to eth_dev
> start), this fix is no problem for the primary process.
> There is a precondition that the same process has to operate
> a vhost interface from beginning to end (from eth_dev configuration
> to eth_dev close). (This patch leaves it to user's responsibility.)
> This precondition will not be a problem in most use cases
> (including SPP).
> 
> v2:
> - add signed-off-by
> - fix spelling error
> 
> v3:
> - rebase on dpdk-next-virtio master
> - change patch order
> - fix subject and commit message
> 
> Itsuro Oda (4):
>   net/vhost: allocate interface name from heap
>   net/vhost: delay vhost driver setup
>   net/vhost: make secondary probe complete
>   net/vhost: remove an unused member
> 
>  drivers/net/vhost/rte_eth_vhost.c | 152 +++++++++++++++++-------------
>  1 file changed, 88 insertions(+), 64 deletions(-)
> 

Applied to dpdk-next-virtio/master

Thanks,
Maxime