[PATCH 00/10] support software live migration

[PATCH 00/10] support software live migration

[Chaoyong He]

This patch series aims to add the support of software live migration
feature for NFP vDPA device.

Xinying Yu (10):
  mailmap: add new contributor
  vdpa/nfp: fix logic in hardware init
  vdpa/nfp: fix the logic of reconfiguration
  vdpa/nfp: refactor the logic of datapath update
  vdpa/nfp: add the live migration logic
  vdpa/nfp: add the interrupt logic of vring relay
  vdpa/nfp: setup the VF configure
  vdpa/nfp: recover the ring index on new host
  vdpa/nfp: setup vring relay thread
  doc: update nfp document

 .mailmap                             |   1 +
 doc/guides/vdpadevs/nfp.rst          |   9 +
 drivers/common/nfp/nfp_common_ctrl.h |  11 +-
 drivers/vdpa/nfp/nfp_vdpa.c          | 440 +++++++++++++++++++++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
 6 files changed, 564 insertions(+), 46 deletions(-)

-- 
2.39.1

[PATCH 01/10] mailmap: add new contributor

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add new contributor.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 .mailmap | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.mailmap b/.mailmap
index 848603bfc5..84a6ff84f9 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1597,6 +1597,7 @@ Xieming Katty <katty.xieming@huawei.com>
 Xinfeng Zhao <xinfengx.zhao@intel.com>
 Xingguang He <xingguang.he@intel.com>
 Xingyou Chen <niatlantice@gmail.com>
+Xinying Yu <xinying.yu@corigine.com>
 Xin Long <longxin.xl@alibaba-inc.com>
 Xi Zhang <xix.zhang@intel.com>
 Xuan Ding <xuan.ding@intel.com>
-- 
2.39.1

[PATCH 02/10] vdpa/nfp: fix logic in hardware init

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Reconfigure the NIC will fail because lack of the
initialization logic of queue configuration pointer.
Fix this by adding the correct initialization logic.

Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 7b877605e4..291798196c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -55,7 +55,10 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 		struct rte_pci_device *pci_dev)
 {
 	uint32_t queue;
+	uint8_t *tx_bar;
+	uint32_t start_q;
 	struct nfp_hw *hw;
+	uint32_t tx_bar_off;
 	uint8_t *notify_base;
 
 	hw = &vdpa_hw->super;
@@ -82,6 +85,12 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 				idx + 1, vdpa_hw->notify_addr[idx + 1]);
 	}
 
+	/* NFP vDPA cfg queue setup */
+	start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
+	tx_bar_off = start_q * NFP_QCP_QUEUE_ADDR_SZ;
+	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
+	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
-- 
2.39.1

Re: [PATCH 02/10] vdpa/nfp: fix logic in hardware init

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Reconfigure the NIC will fail because lack of the
> initialization logic of queue configuration pointer.
> Fix this by adding the correct initialization logic.
> 
> Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
> Cc: chaoyong.he@corigine.com
> Cc: stable@dpdk.org
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH 03/10] vdpa/nfp: fix the logic of reconfiguration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

The ctrl words of vDPA is locate on the extend word, so should
use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.

Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
with 'NFP_NET_CFG_CTRL_VIRTIO'.

Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  1 +
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 6badf769fc..a0e62b063d 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -184,6 +184,7 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_IPSEC_LM_LOOKUP  (0x1 << 4) /**< SA long match lookup */
 #define NFP_NET_CFG_CTRL_MULTI_PF         (0x1 << 5)
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
+#define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 291798196c..6d07356581 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -101,7 +101,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
-	return NFP_NET_CFG_CTRL_SCATTER |
+	return NFP_NET_CFG_CTRL_VIRTIO  |
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
@@ -112,6 +112,7 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	int ret;
 	uint32_t update;
 	uint32_t new_ctrl;
+	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
 	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
@@ -131,8 +132,6 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	new_ctrl = nfp_vdpa_check_offloads();
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -147,8 +146,17 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	/* Writing new MAC to the specific port BAR address */
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
+	new_ext_ctrl = nfp_vdpa_check_offloads();
+
+	update = NFP_NET_CFG_UPDATE_GEN;
+	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
+	if (ret != 0)
+		return -EIO;
+
+	hw->ctrl_ext = new_ext_ctrl;
+
 	/* Enable device */
-	new_ctrl |= NFP_NET_CFG_CTRL_ENABLE;
+	new_ctrl = NFP_NET_CFG_CTRL_ENABLE;
 
 	/* Signal the NIC about the change */
 	update = NFP_NET_CFG_UPDATE_MACADDR |
-- 
2.39.1

Re: [PATCH 03/10] vdpa/nfp: fix the logic of reconfiguration

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> The ctrl words of vDPA is locate on the extend word, so should
> use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.

located*
extended*

so it should*

> 
> Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
> with 'NFP_NET_CFG_CTRL_VIRTIO'.
> 
> Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
> Cc: chaoyong.he@corigine.com
> Cc: stable@dpdk.org
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |  1 +
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
>   2 files changed, 13 insertions(+), 4 deletions(-)
> 

With commit message fixed (I can do it if only change needed in the
series):

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

RE: [PATCH 03/10] vdpa/nfp: fix the logic of reconfiguration

[Chaoyong He]

> On 4/26/24 09:48, Chaoyong He wrote:
> > From: Xinying Yu <xinying.yu@corigine.com>
> >
> > The ctrl words of vDPA is locate on the extend word, so should use the
> > 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.
> 
> located*
> extended*
> 
> so it should*
> 

Will fix in the v2 patch series.
Thanks for your review.

> >
> > Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro with
> > 'NFP_NET_CFG_CTRL_VIRTIO'.
> >
> > Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
> > Cc: chaoyong.he@corigine.com
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > Reviewed-by: Long Wu <long.wu@corigine.com>
> > Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> > ---
> >   drivers/common/nfp/nfp_common_ctrl.h |  1 +
> >   drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
> >   2 files changed, 13 insertions(+), 4 deletions(-)
> >
> 
> With commit message fixed (I can do it if only change needed in the
> series):
> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Thanks,
> Maxime

[PATCH 04/10] vdpa/nfp: refactor the logic of datapath update

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

In order to add the new configuration logic of software live
migration, split the datapath update logic into two parts,
queue configuration and VF configuration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 18 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 6d07356581..79ecd2b4fc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -105,8 +105,8 @@ nfp_vdpa_check_offloads(void)
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
-int
-nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+static int
+nfp_vdpa_vf_config(struct nfp_hw *hw,
 		int vid)
 {
 	int ret;
@@ -114,24 +114,8 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	uint32_t new_ctrl;
 	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
-	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0), rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
-
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0), rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
-
-	rte_wmb();
-
-	nfp_disable_queues(hw);
-	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -177,6 +161,40 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[1].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[0].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
+
+	rte_wmb();
+}
+
+int
+nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw);
+
+	nfp_disable_queues(hw);
+	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
+
+	return nfp_vdpa_vf_config(hw, vid);
+}
+
 void
 nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw)
 {
-- 
2.39.1

Re: [PATCH 04/10] vdpa/nfp: refactor the logic of datapath update

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> In order to add the new configuration logic of software live
> migration, split the datapath update logic into two parts,
> queue configuration and VF configuration.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
>   1 file changed, 36 insertions(+), 18 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH 05/10] vdpa/nfp: add the live migration logic

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the basic logic of software live migration.

Unset the ring notify area to stop the direct IO datapath if the
device support, then we can setup the vring relay to help the
live migration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
 drivers/vdpa/nfp/nfp_vdpa_core.c |  3 ++
 drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index cef80b5476..45092cb0af 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	int vid = device->vid;
+
+	/* Stop the direct IO data path */
+	nfp_vdpa_unset_notify_relay(device);
+	nfp_vdpa_disable_vfio_intr(device);
+
+	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
+	if ((ret != 0) && (ret != -ENOTSUP)) {
+		DRV_VDPA_LOG(ERR, "Unset the host notifier failed.");
+		goto error;
+	}
+
+	device->hw.sw_fallback_running = true;
+
+	return 0;
+
+error:
+	return ret;
+}
+
 static int
 nfp_vdpa_dev_config(int vid)
 {
@@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid)
 	}
 
 	device = node->device;
-	rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed);
-	update_datapath(device);
+	if (device->hw.sw_fallback_running) {
+		device->hw.sw_fallback_running = false;
+
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		rte_atomic_store_explicit(&device->running, 0,
+				rte_memory_order_relaxed);
+	} else {
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		update_datapath(device);
+	}
 
 	return 0;
 }
@@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused,
 static int
 nfp_vdpa_set_features(int32_t vid)
 {
+	int ret;
+	uint64_t features = 0;
+	struct nfp_vdpa_dev *device;
+	struct rte_vdpa_device *vdev;
+	struct nfp_vdpa_dev_node *node;
+
 	DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid);
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	node = nfp_vdpa_find_node_by_vdev(vdev);
+	if (node == NULL) {
+		DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -ENODEV;
+	}
+
+	rte_vhost_get_negotiated_features(vid, &features);
+
+	if (RTE_VHOST_NEED_LOG(features) == 0)
+		return 0;
+
+	device = node->device;
+	if (device->hw.sw_lm) {
+		ret = nfp_vdpa_sw_fallback(device);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "Software fallback start failed");
+			return -1;
+		}
+	}
+
 	return 0;
 }
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 79ecd2b4fc..82a323a6d0 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -91,8 +91,11 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
 	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 
+	vdpa_hw->sw_lm = true;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
+			(1ULL << VHOST_F_LOG_ALL) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
 
 	return 0;
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a8e0d6dd70..0f880fc0c6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 	uint8_t notify_region;
 	uint8_t nr_vring;
+
+	/** Software Live Migration */
+	bool sw_lm;
+	bool sw_fallback_running;
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
-- 
2.39.1

Re: [PATCH 05/10] vdpa/nfp: add the live migration logic

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the basic logic of software live migration.
> 
> Unset the ring notify area to stop the direct IO datapath if the
> device support, then we can setup the vring relay to help the
> live migration.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
>   drivers/vdpa/nfp/nfp_vdpa_core.c |  3 ++
>   drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
>   3 files changed, 71 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
> index cef80b5476..45092cb0af 100644
> --- a/drivers/vdpa/nfp/nfp_vdpa.c
> +++ b/drivers/vdpa/nfp/nfp_vdpa.c
> @@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device)
>   	return ret;
>   }
>   
> +static int
> +nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
> +{
> +	int ret;
> +	int vid = device->vid;
> +
> +	/* Stop the direct IO data path */
> +	nfp_vdpa_unset_notify_relay(device);
> +	nfp_vdpa_disable_vfio_intr(device);
> +
> +	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
> +	if ((ret != 0) && (ret != -ENOTSUP)) {
> +		DRV_VDPA_LOG(ERR, "Unset the host notifier failed.");
> +		goto error;
> +	}
> +
> +	device->hw.sw_fallback_running = true;
> +
> +	return 0;
> +
> +error:
> +	return ret;
> +}
> +
>   static int
>   nfp_vdpa_dev_config(int vid)
>   {
> @@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid)
>   	}
>   
>   	device = node->device;
> -	rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed);
> -	update_datapath(device);
> +	if (device->hw.sw_fallback_running) {
> +		device->hw.sw_fallback_running = false;
> +
> +		rte_atomic_store_explicit(&device->dev_attached, 0,
> +				rte_memory_order_relaxed);
> +		rte_atomic_store_explicit(&device->running, 0,
> +				rte_memory_order_relaxed);
> +	} else {
> +		rte_atomic_store_explicit(&device->dev_attached, 0,
> +				rte_memory_order_relaxed);
> +		update_datapath(device);
> +	}
>   
>   	return 0;
>   }
> @@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused,
>   static int
>   nfp_vdpa_set_features(int32_t vid)
>   {
> +	int ret;
> +	uint64_t features = 0;
> +	struct nfp_vdpa_dev *device;
> +	struct rte_vdpa_device *vdev;
> +	struct nfp_vdpa_dev_node *node;
> +
>   	DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid);
> +
> +	vdev = rte_vhost_get_vdpa_device(vid);
> +	node = nfp_vdpa_find_node_by_vdev(vdev);
> +	if (node == NULL) {
> +		DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev);
> +		return -ENODEV;
> +	}
> +
> +	rte_vhost_get_negotiated_features(vid, &features);
> +
> +	if (RTE_VHOST_NEED_LOG(features) == 0)
> +		return 0;
> +
> +	device = node->device;
> +	if (device->hw.sw_lm) {
> +		ret = nfp_vdpa_sw_fallback(device);
> +		if (ret != 0) {
> +			DRV_VDPA_LOG(ERR, "Software fallback start failed");
> +			return -1;
> +		}
> +	}
> +
>   	return 0;
>   }
>   
> diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
> index 79ecd2b4fc..82a323a6d0 100644
> --- a/drivers/vdpa/nfp/nfp_vdpa_core.c
> +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
> @@ -91,8 +91,11 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
>   	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
>   	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
>   
> +	vdpa_hw->sw_lm = true;
> +
>   	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
>   			(1ULL << VIRTIO_F_IN_ORDER) |
> +			(1ULL << VHOST_F_LOG_ALL) |
>   			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);

Is the live-migration works if we stop at patch 5?
If not, I would suggest to enable the feature only at the end of the
series.

>   
>   	return 0;
> diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
> index a8e0d6dd70..0f880fc0c6 100644
> --- a/drivers/vdpa/nfp/nfp_vdpa_core.h
> +++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
> @@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
>   	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
>   	uint8_t notify_region;
>   	uint8_t nr_vring;
> +
> +	/** Software Live Migration */
> +	bool sw_lm;
> +	bool sw_fallback_running;
>   };
>   
>   int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);

RE: [PATCH 05/10] vdpa/nfp: add the live migration logic

[Chaoyong He]

> On 4/26/24 09:48, Chaoyong He wrote:
> > From: Xinying Yu <xinying.yu@corigine.com>
> >
> > Add the basic logic of software live migration.
> >
> > Unset the ring notify area to stop the direct IO datapath if the
> > device support, then we can setup the vring relay to help the live
> > migration.
> >
> > Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > Reviewed-by: Long Wu <long.wu@corigine.com>
> > Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> > ---
> >   drivers/vdpa/nfp/nfp_vdpa.c      | 66
> +++++++++++++++++++++++++++++++-
> >   drivers/vdpa/nfp/nfp_vdpa_core.c |  3 ++
> >   drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
> >   3 files changed, 71 insertions(+), 2 deletions(-)
> >
> >
> > diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c
> > b/drivers/vdpa/nfp/nfp_vdpa_core.c
> > index 79ecd2b4fc..82a323a6d0 100644
> > --- a/drivers/vdpa/nfp/nfp_vdpa_core.c
> > +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
> > @@ -91,8 +91,11 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
> >   	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
> >   	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
> >
> > +	vdpa_hw->sw_lm = true;
> > +
> >   	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
> >   			(1ULL << VIRTIO_F_IN_ORDER) |
> > +			(1ULL << VHOST_F_LOG_ALL) |
> >   			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
> 
> Is the live-migration works if we stop at patch 5?
> If not, I would suggest to enable the feature only at the end of the series.
> 

The live-migration not work if we stop at patch 5.
We will modify just as your advice in the v2 patch, and thanks for your review.

> >
> >   	return 0;
> > diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h
> > b/drivers/vdpa/nfp/nfp_vdpa_core.h
> > index a8e0d6dd70..0f880fc0c6 100644
> > --- a/drivers/vdpa/nfp/nfp_vdpa_core.h
> > +++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
> > @@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
> >   	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
> >   	uint8_t notify_region;
> >   	uint8_t nr_vring;
> > +
> > +	/** Software Live Migration */
> > +	bool sw_lm;
> > +	bool sw_fallback_running;
> >   };
> >
> >   int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct
> > rte_pci_device *dev);

[PATCH 06/10] vdpa/nfp: add the interrupt logic of vring relay

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the interrupt setup logic of vring relay.

The epoll fd is provided here so host can get the interrupt from device
on Rx direction, all other operations on vring relay are based on this.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 45092cb0af..1643ebbb8c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -336,8 +336,10 @@ nfp_vdpa_stop(struct nfp_vdpa_dev *device)
 }
 
 static int
-nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
+nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device,
+		bool relay)
 {
+	int fd;
 	int ret;
 	uint16_t i;
 	int *fd_ptr;
@@ -366,6 +368,19 @@ nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
 	}
 
+	if (relay) {
+		for (i = 0; i < nr_vring; i += 2) {
+			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+			if (fd < 0) {
+				DRV_VDPA_LOG(ERR, "Can't setup eventfd");
+				return -EINVAL;
+			}
+
+			device->intr_fd[i] = fd;
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
+		}
+	}
+
 	ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
 	if (ret != 0) {
 		DRV_VDPA_LOG(ERR, "Error enabling MSI-X interrupts.");
@@ -556,7 +571,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto unlock_exit;
 
-		ret = nfp_vdpa_enable_vfio_intr(device);
+		ret = nfp_vdpa_enable_vfio_intr(device, false);
 		if (ret != 0)
 			goto dma_map_rollback;
 
@@ -619,6 +634,11 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 		goto error;
 	}
 
+	/* Setup interrupt for vring relay */
+	ret = nfp_vdpa_enable_vfio_intr(device, true);
+	if (ret != 0)
+		goto error;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
-- 
2.39.1

Re: [PATCH 06/10] vdpa/nfp: add the interrupt logic of vring relay

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the interrupt setup logic of vring relay.
> 
> The epoll fd is provided here so host can get the interrupt from device
> on Rx direction, all other operations on vring relay are based on this.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
>   1 file changed, 22 insertions(+), 2 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH 07/10] vdpa/nfp: setup the VF configure

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Create the relay vring on host and then set the address of Rx
used ring to the VF config bar. So the device can DMA the
used ring information to host rather than directly to VM.

Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
enable the MSIX interrupt on device.

Tx ring address is no need to change since the relay vring only
assistant Rx ring to do the dirty page logging.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |   3 +
 drivers/vdpa/nfp/nfp_vdpa.c          | 202 ++++++++++++++++++++++++---
 drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
 4 files changed, 238 insertions(+), 30 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index a0e62b063d..9311d01590 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -186,6 +186,9 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
 #define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
+#define NFP_NET_CFG_CTRL_LM_RELAY         (0x1 << 12) /**< Virtio live migration relay start */
+#define NFP_NET_CFG_CTRL_NOTIFY_DATA      (0x1 << 13) /**< Virtio notification data flag */
+#define NFP_NET_CFG_CTRL_SWLM             (0x1 << 14) /**< Virtio SW live migration enable */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 1643ebbb8c..65f7144671 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -11,6 +11,8 @@
 #include <nfp_common_pci.h>
 #include <nfp_dev.h>
 #include <rte_vfio.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
 #include <vdpa_driver.h>
 
 #include "nfp_vdpa_core.h"
@@ -21,6 +23,9 @@
 #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
 		sizeof(int) * (NFP_VDPA_MAX_QUEUES * 2 + 1))
 
+#define NFP_VDPA_USED_RING_LEN(size) \
+		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -261,15 +266,85 @@ nfp_vdpa_qva_to_gpa(int vid,
 	return gpa;
 }
 
+static void
+nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
+		uint16_t vring_index)
+{
+	uint16_t i;
+	uint64_t size;
+	struct rte_vhost_vring vring;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	for (i = 0; i < vring_index; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
+				rte_mem_page_size());
+		rte_vfio_container_dma_unmap(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
+				m_vring_iova, size);
+
+		rte_free(device->hw.m_vring[i].desc);
+		m_vring_iova += size;
+	}
+}
+
 static int
-nfp_vdpa_start(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint16_t i;
+	uint64_t size;
+	void *vring_buf;
+	uint64_t page_size;
+	struct rte_vhost_vring vring;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	page_size = rte_mem_page_size();
+
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, page_size), page_size);
+		vring_buf = rte_zmalloc("nfp_vdpa_relay", size, page_size);
+		if (vring_buf == NULL)
+			goto vring_free_all;
+
+		vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
+
+		ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
+			goto vring_free_one;
+		}
+
+		m_vring_iova += size;
+	}
+
+	return 0;
+
+vring_free_one:
+	rte_free(device->hw.m_vring[i].desc);
+vring_free_all:
+	nfp_vdpa_relay_vring_free(device, i);
+
+	return -ENOSPC;
+}
+
+static int
+nfp_vdpa_start(struct nfp_vdpa_dev *device,
+		bool relay)
 {
 	int ret;
 	int vid;
 	uint16_t i;
 	uint64_t gpa;
+	uint16_t size;
 	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
 
 	vid = device->vid;
 	vdpa_hw->nr_vring = rte_vhost_get_vring_num(vid);
@@ -278,15 +353,21 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		return ret;
 
+	if (relay) {
+		ret = nfp_vdpa_relay_vring_alloc(device);
+		if (ret != 0)
+			return ret;
+	}
+
 	for (i = 0; i < vdpa_hw->nr_vring; i++) {
 		ret = rte_vhost_get_vhost_vring(vid, i, &vring);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.desc);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for descriptor ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].desc = gpa;
@@ -294,45 +375,122 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.avail);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for available ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].avail = gpa;
 
-		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
-		if (gpa == 0) {
-			DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
-			return -1;
-		}
+		/* Direct I/O for Tx queue, relay for Rx queue */
+		if (relay && ((i & 1) == 0)) {
+			vdpa_hw->vring[i].used = m_vring_iova +
+					(char *)vdpa_hw->m_vring[i].used -
+					(char *)vdpa_hw->m_vring[i].desc;
+
+			ret = rte_vhost_get_vring_base(vid, i,
+					&vdpa_hw->m_vring[i].avail->idx,
+					&vdpa_hw->m_vring[i].used->idx);
+			if (ret != 0)
+				goto relay_vring_free;
+		} else {
+			gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
+			if (gpa == 0) {
+				DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
+				goto relay_vring_free;
+			}
 
-		vdpa_hw->vring[i].used = gpa;
+			vdpa_hw->vring[i].used = gpa;
+		}
 
 		vdpa_hw->vring[i].size = vring.size;
 
+		if (relay) {
+			size = RTE_ALIGN_CEIL(vring_size(vring.size,
+					rte_mem_page_size()), rte_mem_page_size());
+			m_vring_iova += size;
+		}
+
 		ret = rte_vhost_get_vring_base(vid, i,
 				&vdpa_hw->vring[i].last_avail_idx,
 				&vdpa_hw->vring[i].last_used_idx);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 	}
 
-	return nfp_vdpa_hw_start(&device->hw, vid);
+	if (relay)
+		return nfp_vdpa_relay_hw_start(&device->hw, vid);
+	else
+		return nfp_vdpa_hw_start(&device->hw, vid);
+
+relay_vring_free:
+	if (relay)
+		nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+
+	return -EFAULT;
+}
+
+static void
+nfp_vdpa_update_used_ring(struct nfp_vdpa_dev *dev,
+		uint16_t qid)
+{
+	rte_vdpa_relay_vring_used(dev->vid, qid, &dev->hw.m_vring[qid]);
+	rte_vhost_vring_call(dev->vid, qid);
 }
 
 static void
-nfp_vdpa_stop(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_stop(struct nfp_vdpa_dev *device)
 {
 	int vid;
 	uint32_t i;
+	uint64_t len;
+	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
 
 	nfp_vdpa_hw_stop(vdpa_hw);
 
 	vid = device->vid;
-	for (i = 0; i < vdpa_hw->nr_vring; i++)
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		/* Synchronize remaining new used entries if any */
+		if ((i & 1) == 0)
+			nfp_vdpa_update_used_ring(device, i);
+
+		rte_vhost_get_vhost_vring(vid, i, &vring);
+		len = NFP_VDPA_USED_RING_LEN(vring.size);
+		vdpa_hw->vring[i].last_avail_idx = vring.avail->idx;
+		vdpa_hw->vring[i].last_used_idx = vring.used->idx;
+
 		rte_vhost_set_vring_base(vid, i,
 				vdpa_hw->vring[i].last_avail_idx,
 				vdpa_hw->vring[i].last_used_idx);
+
+		rte_vhost_log_used_vring(vid, i, 0, len);
+
+		if (vring.used->idx != vring.avail->idx)
+			rte_atomic_store_explicit(&vring.used->idx, vring.avail->idx,
+					rte_memory_order_relaxed);
+	}
+
+	nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+}
+
+static void
+nfp_vdpa_stop(struct nfp_vdpa_dev *device,
+		bool relay)
+{
+	int vid;
+	uint32_t i;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+
+	nfp_vdpa_hw_stop(vdpa_hw);
+
+	vid = device->vid;
+	if (relay)
+		nfp_vdpa_relay_stop(device);
+	else
+		for (i = 0; i < vdpa_hw->nr_vring; i++)
+			rte_vhost_set_vring_base(vid, i,
+					vdpa_hw->vring[i].last_avail_idx,
+					vdpa_hw->vring[i].last_used_idx);
+
 }
 
 static int
@@ -575,7 +733,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto dma_map_rollback;
 
-		ret = nfp_vdpa_start(device);
+		ret = nfp_vdpa_start(device, false);
 		if (ret != 0)
 			goto disable_vfio_intr;
 
@@ -591,7 +749,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 					rte_memory_order_relaxed) != 0))) {
 		nfp_vdpa_unset_notify_relay(device);
 
-		nfp_vdpa_stop(device);
+		nfp_vdpa_stop(device, false);
 
 		ret = nfp_vdpa_disable_vfio_intr(device);
 		if (ret != 0)
@@ -608,7 +766,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return 0;
 
 vdpa_stop:
-	nfp_vdpa_stop(device);
+	nfp_vdpa_stop(device, false);
 disable_vfio_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 dma_map_rollback:
@@ -639,10 +797,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto error;
 
+	/* Config the VF */
+	ret = nfp_vdpa_start(device, true);
+	if (ret != 0)
+		goto unset_intr;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+unset_intr:
+	nfp_vdpa_disable_vfio_intr(device);
 error:
 	return ret;
 }
@@ -691,6 +856,9 @@ nfp_vdpa_dev_close(int vid)
 
 	device = node->device;
 	if (device->hw.sw_fallback_running) {
+		/* Reset VF */
+		nfp_vdpa_stop(device, true);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 82a323a6d0..63a69aaf36 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -110,7 +110,8 @@ nfp_vdpa_check_offloads(void)
 
 static int
 nfp_vdpa_vf_config(struct nfp_hw *hw,
-		int vid)
+		int vid,
+		bool relay)
 {
 	int ret;
 	uint32_t update;
@@ -134,6 +135,10 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
 	new_ext_ctrl = nfp_vdpa_check_offloads();
+	if (relay)
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_LM_RELAY;
+	else
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_SWLM;
 
 	update = NFP_NET_CFG_UPDATE_GEN;
 	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
@@ -150,6 +155,15 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 			NFP_NET_CFG_UPDATE_GEN |
 			NFP_NET_CFG_UPDATE_RING;
 
+	if (relay) {
+		update |= NFP_NET_CFG_UPDATE_MSIX;
+
+		/* Enable misx interrupt for vdpa relay */
+		new_ctrl |= NFP_NET_CFG_CTRL_MSIX_TX_OFF;
+
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_VEC(0), 1);
+	}
+
 	ret = nfp_reconfig(hw, new_ctrl, update);
 	if (ret < 0)
 		return -EIO;
@@ -165,20 +179,24 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 }
 
 static void
-nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
+		bool relay)
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+	if (!relay) {
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[1].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[0].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	}
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
 	rte_wmb();
@@ -190,12 +208,23 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nfp_vdpa_queue_config(vdpa_hw);
+	nfp_vdpa_queue_config(vdpa_hw, false);
 
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	return nfp_vdpa_vf_config(hw, vid);
+	return nfp_vdpa_vf_config(hw, vid, false);
+}
+
+int
+nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw, true);
+
+	return nfp_vdpa_vf_config(hw, vid, true);
 }
 
 void
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index 0f880fc0c6..a339ace601 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -9,12 +9,15 @@
 #include <bus_pci_driver.h>
 #include <nfp_common.h>
 #include <rte_ether.h>
+#include <rte_vhost.h>
 
 #define NFP_VDPA_MAX_QUEUES         1
 
 #define NFP_VDPA_NOTIFY_ADDR_BASE        0x4000
 #define NFP_VDPA_NOTIFY_ADDR_INTERVAL    0x1000
 
+#define NFP_VDPA_RELAY_VRING             0xd0000000
+
 struct nfp_vdpa_vring {
 	uint64_t desc;
 	uint64_t avail;
@@ -40,12 +43,17 @@ struct nfp_vdpa_hw {
 	/** Software Live Migration */
 	bool sw_lm;
 	bool sw_fallback_running;
+
+	/** Mediated vring for SW fallback */
+	struct vring m_vring[NFP_VDPA_MAX_QUEUES * 2];
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
 
 int nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
 
+int nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
+
 void nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw);
 
 void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
-- 
2.39.1

Re: [PATCH 07/10] vdpa/nfp: setup the VF configure

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Create the relay vring on host and then set the address of Rx
> used ring to the VF config bar. So the device can DMA the
> used ring information to host rather than directly to VM.
> 
> Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
> enable the MSIX interrupt on device.
> 
> Tx ring address is no need to change since the relay vring only

not needed*

> assistant Rx ring to do the dirty page logging.

assists*

> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |   3 +
>   drivers/vdpa/nfp/nfp_vdpa.c          | 202 ++++++++++++++++++++++++---
>   drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
>   4 files changed, 238 insertions(+), 30 deletions(-)


Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

RE: [PATCH 07/10] vdpa/nfp: setup the VF configure

[Chaoyong He]

> On 4/26/24 09:48, Chaoyong He wrote:
> > From: Xinying Yu <xinying.yu@corigine.com>
> >
> > Create the relay vring on host and then set the address of Rx used
> > ring to the VF config bar. So the device can DMA the used ring
> > information to host rather than directly to VM.
> >
> > Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And enable the
> > MSIX interrupt on device.
> >
> > Tx ring address is no need to change since the relay vring only
> 
> not needed*
> 
> > assistant Rx ring to do the dirty page logging.
> 
> assists*
> 

Will fix in the v2 patch series.
Thanks for your review.

> >
> > Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > Reviewed-by: Long Wu <long.wu@corigine.com>
> > Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> > ---
> >   drivers/common/nfp/nfp_common_ctrl.h |   3 +
> >   drivers/vdpa/nfp/nfp_vdpa.c          | 202 ++++++++++++++++++++++++---
> >   drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
> >   drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
> >   4 files changed, 238 insertions(+), 30 deletions(-)
> 
> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Thanks,
> Maxime

[PATCH 08/10] vdpa/nfp: recover the ring index on new host

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

After migrate to new host, the vring information was
recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
and 'NFP_NET_CFG_RX_USED_INDEX'.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 9311d01590..4b273372a8 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -193,8 +193,11 @@ struct nfp_net_fw_ver {
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
 
-/* 16B reserved for future use (0x00b0 - 0x00c0). */
-#define NFP_NET_CFG_RESERVED            0x00b0
+#define NFP_NET_CFG_TX_USED_INDEX       0x00b0
+#define NFP_NET_CFG_RX_USED_INDEX       0x00b4
+
+/* 16B reserved for future use (0x00b8 - 0x0010). */
+#define NFP_NET_CFG_RESERVED            0x00b8
 #define NFP_NET_CFG_RESERVED_SZ         0x0010
 
 /*
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 63a69aaf36..8f9aba9519 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -101,6 +101,16 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_hw_queue_init(struct nfp_vdpa_hw *vdpa_hw)
+{
+	/* Distribute ring information to firmware */
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_TX_USED_INDEX,
+			vdpa_hw->vring[1].last_used_idx);
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_RX_USED_INDEX,
+			vdpa_hw->vring[0].last_used_idx);
+}
+
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
@@ -199,6 +209,9 @@ nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
 
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
+	if (!relay)
+		nfp_vdpa_hw_queue_init(vdpa_hw);
+
 	rte_wmb();
 }
 
-- 
2.39.1

Re: [PATCH 08/10] vdpa/nfp: recover the ring index on new host

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> After migrate to new host, the vring information was

migrating*

is recovered*

> recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
> and 'NFP_NET_CFG_RX_USED_INDEX'.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
>   2 files changed, 18 insertions(+), 2 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

RE: [PATCH 08/10] vdpa/nfp: recover the ring index on new host

[Chaoyong He]

> On 4/26/24 09:48, Chaoyong He wrote:
> > From: Xinying Yu <xinying.yu@corigine.com>
> >
> > After migrate to new host, the vring information was
> 
> migrating*
> 
> is recovered*
> 

Will fix in the v2 patch series.
Thanks for your review.

> > recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
> > and 'NFP_NET_CFG_RX_USED_INDEX'.
> >
> > Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> > Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> > Reviewed-by: Long Wu <long.wu@corigine.com>
> > Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> > ---
> >   drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
> >   drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
> >   2 files changed, 18 insertions(+), 2 deletions(-)
> >
> 
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Thanks,
> Maxime

[PATCH 09/10] vdpa/nfp: setup vring relay thread

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Setup the vring relay thread to monitor the interruption from
device. And do the dirty page logging or notify device according
to event data.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
 drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
 drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
 3 files changed, 159 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 65f7144671..e57765eb1a 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -26,6 +26,8 @@
 #define NFP_VDPA_USED_RING_LEN(size) \
 		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
 
+#define EPOLL_DATA_INTR        1
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -776,6 +778,139 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_vring_epoll_ctl(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint32_t qid;
+	struct epoll_event ev;
+	struct rte_vhost_vring vring;
+
+	for (qid = 0; qid < queue_num; qid++) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		rte_vhost_get_vhost_vring(device->vid, qid, &vring);
+		ev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD, vring.kickfd, &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+	}
+
+	/* vDPA driver interrupt */
+	for (qid = 0; qid < queue_num; qid += 2) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		/* Leave a flag to mark it's for interrupt */
+		ev.data.u64 = EPOLL_DATA_INTR | qid << 1 |
+				(uint64_t)device->intr_fd[qid] << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD,
+				device->intr_fd[qid], &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+
+		nfp_vdpa_update_used_ring(device, qid);
+	}
+
+	return 0;
+}
+
+static int
+nfp_vdpa_vring_epoll_wait(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int i;
+	int fds;
+	int kickfd;
+	uint32_t qid;
+	struct epoll_event events[NFP_VDPA_MAX_QUEUES * 2];
+
+	for (;;) {
+		fds = epoll_wait(device->epoll_fd, events, queue_num * 2, -1);
+		if (fds < 0) {
+			if (errno == EINTR)
+				continue;
+
+			DRV_VDPA_LOG(ERR, "Epoll wait fail");
+			return -EACCES;
+		}
+
+		for (i = 0; i < fds; i++) {
+			qid = events[i].data.u32 >> 1;
+			kickfd = (uint32_t)(events[i].data.u64 >> 32);
+
+			nfp_vdpa_read_kickfd(kickfd);
+			if ((events[i].data.u32 & EPOLL_DATA_INTR) != 0) {
+				nfp_vdpa_update_used_ring(device, qid);
+				nfp_vdpa_irq_unmask(&device->hw);
+			} else {
+				nfp_vdpa_notify_queue(&device->hw, qid);
+			}
+		}
+	}
+
+	return 0;
+}
+
+static uint32_t
+nfp_vdpa_vring_relay(void *arg)
+{
+	int ret;
+	int epoll_fd;
+	uint16_t queue_id;
+	uint32_t queue_num;
+	struct nfp_vdpa_dev *device = arg;
+
+	epoll_fd = epoll_create(NFP_VDPA_MAX_QUEUES * 2);
+	if (epoll_fd < 0) {
+		DRV_VDPA_LOG(ERR, "failed to create epoll instance.");
+		return 1;
+	}
+
+	device->epoll_fd = epoll_fd;
+
+	queue_num = rte_vhost_get_vring_num(device->vid);
+
+	ret = nfp_vdpa_vring_epoll_ctl(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	/* Start relay with a first kick */
+	for (queue_id = 0; queue_id < queue_num; queue_id++)
+		nfp_vdpa_notify_queue(&device->hw, queue_id);
+
+	ret = nfp_vdpa_vring_epoll_wait(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	return 0;
+
+notify_exit:
+	close(device->epoll_fd);
+	device->epoll_fd = -1;
+
+	return 1;
+}
+
+static int
+nfp_vdpa_setup_vring_relay(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	char name[RTE_THREAD_INTERNAL_NAME_SIZE];
+
+	snprintf(name, sizeof(name), "nfp_vring%d", device->vid);
+	ret = rte_thread_create_internal_control(&device->tid, name,
+			nfp_vdpa_vring_relay, (void *)device);
+	if (ret != 0) {
+		DRV_VDPA_LOG(ERR, "Failed to create vring relay pthread.");
+		return -EPERM;
+	}
+
+	return 0;
+}
+
 static int
 nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 {
@@ -802,10 +937,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto unset_intr;
 
+	/* Setup vring relay thread */
+	ret = nfp_vdpa_setup_vring_relay(device);
+	if (ret != 0)
+		goto stop_vf;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+stop_vf:
+	nfp_vdpa_stop(device, true);
 unset_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 error:
@@ -859,6 +1001,12 @@ nfp_vdpa_dev_close(int vid)
 		/* Reset VF */
 		nfp_vdpa_stop(device, true);
 
+		/* Remove interrupt setting */
+		nfp_vdpa_disable_vfio_intr(device);
+
+		/* Unset DMA map for guest memory */
+		nfp_vdpa_dma_map(device, false);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 8f9aba9519..70aeb4a3ac 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -271,3 +271,12 @@ nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_qcp_notify_ptr_add(vdpa_hw->notify_addr[qid],
 			NFP_QCP_NOTIFY_WRITE_PTR, qid);
 }
+
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	/* Make sure all updates are written before un-masking */
+	rte_wmb();
+	nn_cfg_writeb(hw, NFP_NET_CFG_ICR(1), NFP_NET_CFG_ICR_UNMASKED);
+}
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a339ace601..bc4db556a2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -60,4 +60,6 @@ void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
 
 uint64_t nfp_vdpa_get_queue_notify_offset(struct nfp_vdpa_hw *vdpa_hw, int qid);
 
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw);
+
 #endif /* __NFP_VDPA_CORE_H__ */
-- 
2.39.1

Re: [PATCH 09/10] vdpa/nfp: setup vring relay thread

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Setup the vring relay thread to monitor the interruption from
> device. And do the dirty page logging or notify device according
> to event data.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
>   drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
>   drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
>   3 files changed, 159 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH 10/10] doc: update nfp document

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the software assisted vDPA live migration feature
into NFP document.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 doc/guides/vdpadevs/nfp.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
index dc9e94dbc8..e4736d9f61 100644
--- a/doc/guides/vdpadevs/nfp.rst
+++ b/doc/guides/vdpadevs/nfp.rst
@@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
 
 This PMD uses (common/nfp) code to access the device firmware.
 
+Software Live Migration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Now the NFP vDPA driver only support software assisted live migration mode.
+In this mode, the driver will setup a software relay thread when live migration
+happens, this thread will help device to log dirty pages. Although this mode
+does not require hardware to implement a dirty page logging function block, it
+will consume percentage of CPU resource depending on the network throughput.
+
 Per-Device Parameters
 ~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.39.1

Re: [PATCH 10/10] doc: update nfp document

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Recheck-request: iol-compile-amd64-testing

The DPDK Community Lab updated to the latest Alpine image yesterday, which
resulted in all Alpine builds failing. The failure is unrelated to your
patch, and this recheck should remove the fail on Patchwork, as we have
disabled Alpine testing for now.

[-- Attachment #2: Type: text/html, Size: 361 bytes --]

Re: [PATCH 10/10] doc: update nfp document

[Maxime Coquelin]

On 4/26/24 09:48, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the software assisted vDPA live migration feature
> into NFP document.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   doc/guides/vdpadevs/nfp.rst | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
> index dc9e94dbc8..e4736d9f61 100644
> --- a/doc/guides/vdpadevs/nfp.rst
> +++ b/doc/guides/vdpadevs/nfp.rst
> @@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
>   
>   This PMD uses (common/nfp) code to access the device firmware.
>   
> +Software Live Migration
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Now the NFP vDPA driver only support software assisted live migration mode.
> +In this mode, the driver will setup a software relay thread when live migration
> +happens, this thread will help device to log dirty pages. Although this mode
> +does not require hardware to implement a dirty page logging function block, it
> +will consume percentage of CPU resource depending on the network throughput.
> +
>   Per-Device Parameters
>   ~~~~~~~~~~~~~~~~~~~~~
>   


Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

RE: [PATCH 00/10] support software live migration

[Chaoyong He]

A gentle ping~

> -----Original Message-----
> From: Chaoyong He
> Sent: Friday, April 26, 2024 3:49 PM
> To: dev@dpdk.org
> Cc: oss-drivers <oss-drivers@corigine.com>; Chaoyong He
> <chaoyong.he@corigine.com>
> Subject: [PATCH 00/10] support software live migration
> 
> This patch series aims to add the support of software live migration feature for
> NFP vDPA device.
> 
> Xinying Yu (10):
>   mailmap: add new contributor
>   vdpa/nfp: fix logic in hardware init
>   vdpa/nfp: fix the logic of reconfiguration
>   vdpa/nfp: refactor the logic of datapath update
>   vdpa/nfp: add the live migration logic
>   vdpa/nfp: add the interrupt logic of vring relay
>   vdpa/nfp: setup the VF configure
>   vdpa/nfp: recover the ring index on new host
>   vdpa/nfp: setup vring relay thread
>   doc: update nfp document
> 
>  .mailmap                             |   1 +
>  doc/guides/vdpadevs/nfp.rst          |   9 +
>  drivers/common/nfp/nfp_common_ctrl.h |  11 +-
>  drivers/vdpa/nfp/nfp_vdpa.c          | 440 +++++++++++++++++++++++++--
>  drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
>  drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
>  6 files changed, 564 insertions(+), 46 deletions(-)
> 
> --
> 2.39.1

[PATCH v2 00/11] support software live migration

[Chaoyong He]

This patch series aims to add the support of software live migration
feature for NFP vDPA device.

---
v2:
* Adjust some spell in the commit message.
* Split out a commit to enable this feature.
---

Xinying Yu (11):
  mailmap: add new contributor
  vdpa/nfp: fix logic in hardware init
  vdpa/nfp: fix the logic of reconfiguration
  vdpa/nfp: refactor the logic of datapath update
  vdpa/nfp: add the live migration logic
  vdpa/nfp: add the interrupt logic of vring relay
  vdpa/nfp: setup the VF configure
  vdpa/nfp: recover the ring index on new host
  vdpa/nfp: setup vring relay thread
  vdpa/nfp: enable feature bits of live migration
  doc: update nfp document

 .mailmap                             |   1 +
 doc/guides/vdpadevs/nfp.rst          |   9 +
 drivers/common/nfp/nfp_common_ctrl.h |  11 +-
 drivers/vdpa/nfp/nfp_vdpa.c          | 440 +++++++++++++++++++++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
 6 files changed, 564 insertions(+), 46 deletions(-)

-- 
2.39.1

[PATCH v2 01/11] mailmap: add new contributor

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add new contributor.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 .mailmap | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.mailmap b/.mailmap
index 6b396107d0..01c31b8c70 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1598,6 +1598,7 @@ Xieming Katty <katty.xieming@huawei.com>
 Xinfeng Zhao <xinfengx.zhao@intel.com>
 Xingguang He <xingguang.he@intel.com>
 Xingyou Chen <niatlantice@gmail.com>
+Xinying Yu <xinying.yu@corigine.com>
 Xin Long <longxin.xl@alibaba-inc.com>
 Xi Zhang <xix.zhang@intel.com>
 Xuan Ding <xuan.ding@intel.com>
-- 
2.39.1

[PATCH v2 02/11] vdpa/nfp: fix logic in hardware init

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Reconfigure the NIC will fail because lack of the
initialization logic of queue configuration pointer.
Fix this by adding the correct initialization logic.

Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 7b877605e4..291798196c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -55,7 +55,10 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 		struct rte_pci_device *pci_dev)
 {
 	uint32_t queue;
+	uint8_t *tx_bar;
+	uint32_t start_q;
 	struct nfp_hw *hw;
+	uint32_t tx_bar_off;
 	uint8_t *notify_base;
 
 	hw = &vdpa_hw->super;
@@ -82,6 +85,12 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 				idx + 1, vdpa_hw->notify_addr[idx + 1]);
 	}
 
+	/* NFP vDPA cfg queue setup */
+	start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
+	tx_bar_off = start_q * NFP_QCP_QUEUE_ADDR_SZ;
+	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
+	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
-- 
2.39.1

[PATCH v2 03/11] vdpa/nfp: fix the logic of reconfiguration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

The ctrl words of vDPA is located on the extended word, so it
should use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.

Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
with 'NFP_NET_CFG_CTRL_VIRTIO'.

Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  1 +
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 6badf769fc..a0e62b063d 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -184,6 +184,7 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_IPSEC_LM_LOOKUP  (0x1 << 4) /**< SA long match lookup */
 #define NFP_NET_CFG_CTRL_MULTI_PF         (0x1 << 5)
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
+#define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 291798196c..6d07356581 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -101,7 +101,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
-	return NFP_NET_CFG_CTRL_SCATTER |
+	return NFP_NET_CFG_CTRL_VIRTIO  |
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
@@ -112,6 +112,7 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	int ret;
 	uint32_t update;
 	uint32_t new_ctrl;
+	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
 	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
@@ -131,8 +132,6 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	new_ctrl = nfp_vdpa_check_offloads();
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -147,8 +146,17 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	/* Writing new MAC to the specific port BAR address */
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
+	new_ext_ctrl = nfp_vdpa_check_offloads();
+
+	update = NFP_NET_CFG_UPDATE_GEN;
+	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
+	if (ret != 0)
+		return -EIO;
+
+	hw->ctrl_ext = new_ext_ctrl;
+
 	/* Enable device */
-	new_ctrl |= NFP_NET_CFG_CTRL_ENABLE;
+	new_ctrl = NFP_NET_CFG_CTRL_ENABLE;
 
 	/* Signal the NIC about the change */
 	update = NFP_NET_CFG_UPDATE_MACADDR |
-- 
2.39.1

[PATCH v2 04/11] vdpa/nfp: refactor the logic of datapath update

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

In order to add the new configuration logic of software live
migration, split the datapath update logic into two parts,
queue configuration and VF configuration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 18 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 6d07356581..79ecd2b4fc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -105,8 +105,8 @@ nfp_vdpa_check_offloads(void)
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
-int
-nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+static int
+nfp_vdpa_vf_config(struct nfp_hw *hw,
 		int vid)
 {
 	int ret;
@@ -114,24 +114,8 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	uint32_t new_ctrl;
 	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
-	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0), rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
-
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0), rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
-
-	rte_wmb();
-
-	nfp_disable_queues(hw);
-	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -177,6 +161,40 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[1].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[0].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
+
+	rte_wmb();
+}
+
+int
+nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw);
+
+	nfp_disable_queues(hw);
+	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
+
+	return nfp_vdpa_vf_config(hw, vid);
+}
+
 void
 nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw)
 {
-- 
2.39.1

[PATCH v2 05/11] vdpa/nfp: add the live migration logic

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the basic logic of software live migration.

Unset the ring notify area to stop the direct IO datapath if the
device support, then we can setup the vring relay to help the
live migration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
 drivers/vdpa/nfp/nfp_vdpa_core.c |  2 +
 drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
 3 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index cef80b5476..45092cb0af 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	int vid = device->vid;
+
+	/* Stop the direct IO data path */
+	nfp_vdpa_unset_notify_relay(device);
+	nfp_vdpa_disable_vfio_intr(device);
+
+	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
+	if ((ret != 0) && (ret != -ENOTSUP)) {
+		DRV_VDPA_LOG(ERR, "Unset the host notifier failed.");
+		goto error;
+	}
+
+	device->hw.sw_fallback_running = true;
+
+	return 0;
+
+error:
+	return ret;
+}
+
 static int
 nfp_vdpa_dev_config(int vid)
 {
@@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid)
 	}
 
 	device = node->device;
-	rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed);
-	update_datapath(device);
+	if (device->hw.sw_fallback_running) {
+		device->hw.sw_fallback_running = false;
+
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		rte_atomic_store_explicit(&device->running, 0,
+				rte_memory_order_relaxed);
+	} else {
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		update_datapath(device);
+	}
 
 	return 0;
 }
@@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused,
 static int
 nfp_vdpa_set_features(int32_t vid)
 {
+	int ret;
+	uint64_t features = 0;
+	struct nfp_vdpa_dev *device;
+	struct rte_vdpa_device *vdev;
+	struct nfp_vdpa_dev_node *node;
+
 	DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid);
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	node = nfp_vdpa_find_node_by_vdev(vdev);
+	if (node == NULL) {
+		DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -ENODEV;
+	}
+
+	rte_vhost_get_negotiated_features(vid, &features);
+
+	if (RTE_VHOST_NEED_LOG(features) == 0)
+		return 0;
+
+	device = node->device;
+	if (device->hw.sw_lm) {
+		ret = nfp_vdpa_sw_fallback(device);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "Software fallback start failed");
+			return -1;
+		}
+	}
+
 	return 0;
 }
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 79ecd2b4fc..50eda4cb2c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -91,6 +91,8 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
 	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 
+	vdpa_hw->sw_lm = true;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a8e0d6dd70..0f880fc0c6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 	uint8_t notify_region;
 	uint8_t nr_vring;
+
+	/** Software Live Migration */
+	bool sw_lm;
+	bool sw_fallback_running;
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
-- 
2.39.1

[PATCH v2 06/11] vdpa/nfp: add the interrupt logic of vring relay

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the interrupt setup logic of vring relay.

The epoll fd is provided here so host can get the interrupt from device
on Rx direction, all other operations on vring relay are based on this.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 45092cb0af..1643ebbb8c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -336,8 +336,10 @@ nfp_vdpa_stop(struct nfp_vdpa_dev *device)
 }
 
 static int
-nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
+nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device,
+		bool relay)
 {
+	int fd;
 	int ret;
 	uint16_t i;
 	int *fd_ptr;
@@ -366,6 +368,19 @@ nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
 	}
 
+	if (relay) {
+		for (i = 0; i < nr_vring; i += 2) {
+			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+			if (fd < 0) {
+				DRV_VDPA_LOG(ERR, "Can't setup eventfd");
+				return -EINVAL;
+			}
+
+			device->intr_fd[i] = fd;
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
+		}
+	}
+
 	ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
 	if (ret != 0) {
 		DRV_VDPA_LOG(ERR, "Error enabling MSI-X interrupts.");
@@ -556,7 +571,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto unlock_exit;
 
-		ret = nfp_vdpa_enable_vfio_intr(device);
+		ret = nfp_vdpa_enable_vfio_intr(device, false);
 		if (ret != 0)
 			goto dma_map_rollback;
 
@@ -619,6 +634,11 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 		goto error;
 	}
 
+	/* Setup interrupt for vring relay */
+	ret = nfp_vdpa_enable_vfio_intr(device, true);
+	if (ret != 0)
+		goto error;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
-- 
2.39.1

[PATCH v2 07/11] vdpa/nfp: setup the VF configure

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Create the relay vring on host and then set the address of Rx
used ring to the VF config bar. So the device can DMA the
used ring information to host rather than directly to VM.

Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
enable the MSIX interrupt on device.

Tx ring address is not needed to change since the relay vring
only assists Rx ring to do the dirty page logging.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |   3 +
 drivers/vdpa/nfp/nfp_vdpa.c          | 202 ++++++++++++++++++++++++---
 drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
 4 files changed, 238 insertions(+), 30 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index a0e62b063d..9311d01590 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -186,6 +186,9 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
 #define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
+#define NFP_NET_CFG_CTRL_LM_RELAY         (0x1 << 12) /**< Virtio live migration relay start */
+#define NFP_NET_CFG_CTRL_NOTIFY_DATA      (0x1 << 13) /**< Virtio notification data flag */
+#define NFP_NET_CFG_CTRL_SWLM             (0x1 << 14) /**< Virtio SW live migration enable */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 1643ebbb8c..65f7144671 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -11,6 +11,8 @@
 #include <nfp_common_pci.h>
 #include <nfp_dev.h>
 #include <rte_vfio.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
 #include <vdpa_driver.h>
 
 #include "nfp_vdpa_core.h"
@@ -21,6 +23,9 @@
 #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
 		sizeof(int) * (NFP_VDPA_MAX_QUEUES * 2 + 1))
 
+#define NFP_VDPA_USED_RING_LEN(size) \
+		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -261,15 +266,85 @@ nfp_vdpa_qva_to_gpa(int vid,
 	return gpa;
 }
 
+static void
+nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
+		uint16_t vring_index)
+{
+	uint16_t i;
+	uint64_t size;
+	struct rte_vhost_vring vring;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	for (i = 0; i < vring_index; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
+				rte_mem_page_size());
+		rte_vfio_container_dma_unmap(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
+				m_vring_iova, size);
+
+		rte_free(device->hw.m_vring[i].desc);
+		m_vring_iova += size;
+	}
+}
+
 static int
-nfp_vdpa_start(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint16_t i;
+	uint64_t size;
+	void *vring_buf;
+	uint64_t page_size;
+	struct rte_vhost_vring vring;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	page_size = rte_mem_page_size();
+
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, page_size), page_size);
+		vring_buf = rte_zmalloc("nfp_vdpa_relay", size, page_size);
+		if (vring_buf == NULL)
+			goto vring_free_all;
+
+		vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
+
+		ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
+			goto vring_free_one;
+		}
+
+		m_vring_iova += size;
+	}
+
+	return 0;
+
+vring_free_one:
+	rte_free(device->hw.m_vring[i].desc);
+vring_free_all:
+	nfp_vdpa_relay_vring_free(device, i);
+
+	return -ENOSPC;
+}
+
+static int
+nfp_vdpa_start(struct nfp_vdpa_dev *device,
+		bool relay)
 {
 	int ret;
 	int vid;
 	uint16_t i;
 	uint64_t gpa;
+	uint16_t size;
 	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
 
 	vid = device->vid;
 	vdpa_hw->nr_vring = rte_vhost_get_vring_num(vid);
@@ -278,15 +353,21 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		return ret;
 
+	if (relay) {
+		ret = nfp_vdpa_relay_vring_alloc(device);
+		if (ret != 0)
+			return ret;
+	}
+
 	for (i = 0; i < vdpa_hw->nr_vring; i++) {
 		ret = rte_vhost_get_vhost_vring(vid, i, &vring);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.desc);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for descriptor ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].desc = gpa;
@@ -294,45 +375,122 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.avail);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for available ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].avail = gpa;
 
-		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
-		if (gpa == 0) {
-			DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
-			return -1;
-		}
+		/* Direct I/O for Tx queue, relay for Rx queue */
+		if (relay && ((i & 1) == 0)) {
+			vdpa_hw->vring[i].used = m_vring_iova +
+					(char *)vdpa_hw->m_vring[i].used -
+					(char *)vdpa_hw->m_vring[i].desc;
+
+			ret = rte_vhost_get_vring_base(vid, i,
+					&vdpa_hw->m_vring[i].avail->idx,
+					&vdpa_hw->m_vring[i].used->idx);
+			if (ret != 0)
+				goto relay_vring_free;
+		} else {
+			gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
+			if (gpa == 0) {
+				DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
+				goto relay_vring_free;
+			}
 
-		vdpa_hw->vring[i].used = gpa;
+			vdpa_hw->vring[i].used = gpa;
+		}
 
 		vdpa_hw->vring[i].size = vring.size;
 
+		if (relay) {
+			size = RTE_ALIGN_CEIL(vring_size(vring.size,
+					rte_mem_page_size()), rte_mem_page_size());
+			m_vring_iova += size;
+		}
+
 		ret = rte_vhost_get_vring_base(vid, i,
 				&vdpa_hw->vring[i].last_avail_idx,
 				&vdpa_hw->vring[i].last_used_idx);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 	}
 
-	return nfp_vdpa_hw_start(&device->hw, vid);
+	if (relay)
+		return nfp_vdpa_relay_hw_start(&device->hw, vid);
+	else
+		return nfp_vdpa_hw_start(&device->hw, vid);
+
+relay_vring_free:
+	if (relay)
+		nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+
+	return -EFAULT;
+}
+
+static void
+nfp_vdpa_update_used_ring(struct nfp_vdpa_dev *dev,
+		uint16_t qid)
+{
+	rte_vdpa_relay_vring_used(dev->vid, qid, &dev->hw.m_vring[qid]);
+	rte_vhost_vring_call(dev->vid, qid);
 }
 
 static void
-nfp_vdpa_stop(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_stop(struct nfp_vdpa_dev *device)
 {
 	int vid;
 	uint32_t i;
+	uint64_t len;
+	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
 
 	nfp_vdpa_hw_stop(vdpa_hw);
 
 	vid = device->vid;
-	for (i = 0; i < vdpa_hw->nr_vring; i++)
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		/* Synchronize remaining new used entries if any */
+		if ((i & 1) == 0)
+			nfp_vdpa_update_used_ring(device, i);
+
+		rte_vhost_get_vhost_vring(vid, i, &vring);
+		len = NFP_VDPA_USED_RING_LEN(vring.size);
+		vdpa_hw->vring[i].last_avail_idx = vring.avail->idx;
+		vdpa_hw->vring[i].last_used_idx = vring.used->idx;
+
 		rte_vhost_set_vring_base(vid, i,
 				vdpa_hw->vring[i].last_avail_idx,
 				vdpa_hw->vring[i].last_used_idx);
+
+		rte_vhost_log_used_vring(vid, i, 0, len);
+
+		if (vring.used->idx != vring.avail->idx)
+			rte_atomic_store_explicit(&vring.used->idx, vring.avail->idx,
+					rte_memory_order_relaxed);
+	}
+
+	nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+}
+
+static void
+nfp_vdpa_stop(struct nfp_vdpa_dev *device,
+		bool relay)
+{
+	int vid;
+	uint32_t i;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+
+	nfp_vdpa_hw_stop(vdpa_hw);
+
+	vid = device->vid;
+	if (relay)
+		nfp_vdpa_relay_stop(device);
+	else
+		for (i = 0; i < vdpa_hw->nr_vring; i++)
+			rte_vhost_set_vring_base(vid, i,
+					vdpa_hw->vring[i].last_avail_idx,
+					vdpa_hw->vring[i].last_used_idx);
+
 }
 
 static int
@@ -575,7 +733,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto dma_map_rollback;
 
-		ret = nfp_vdpa_start(device);
+		ret = nfp_vdpa_start(device, false);
 		if (ret != 0)
 			goto disable_vfio_intr;
 
@@ -591,7 +749,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 					rte_memory_order_relaxed) != 0))) {
 		nfp_vdpa_unset_notify_relay(device);
 
-		nfp_vdpa_stop(device);
+		nfp_vdpa_stop(device, false);
 
 		ret = nfp_vdpa_disable_vfio_intr(device);
 		if (ret != 0)
@@ -608,7 +766,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return 0;
 
 vdpa_stop:
-	nfp_vdpa_stop(device);
+	nfp_vdpa_stop(device, false);
 disable_vfio_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 dma_map_rollback:
@@ -639,10 +797,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto error;
 
+	/* Config the VF */
+	ret = nfp_vdpa_start(device, true);
+	if (ret != 0)
+		goto unset_intr;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+unset_intr:
+	nfp_vdpa_disable_vfio_intr(device);
 error:
 	return ret;
 }
@@ -691,6 +856,9 @@ nfp_vdpa_dev_close(int vid)
 
 	device = node->device;
 	if (device->hw.sw_fallback_running) {
+		/* Reset VF */
+		nfp_vdpa_stop(device, true);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 50eda4cb2c..2b609dddc2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -109,7 +109,8 @@ nfp_vdpa_check_offloads(void)
 
 static int
 nfp_vdpa_vf_config(struct nfp_hw *hw,
-		int vid)
+		int vid,
+		bool relay)
 {
 	int ret;
 	uint32_t update;
@@ -133,6 +134,10 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
 	new_ext_ctrl = nfp_vdpa_check_offloads();
+	if (relay)
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_LM_RELAY;
+	else
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_SWLM;
 
 	update = NFP_NET_CFG_UPDATE_GEN;
 	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
@@ -149,6 +154,15 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 			NFP_NET_CFG_UPDATE_GEN |
 			NFP_NET_CFG_UPDATE_RING;
 
+	if (relay) {
+		update |= NFP_NET_CFG_UPDATE_MSIX;
+
+		/* Enable misx interrupt for vdpa relay */
+		new_ctrl |= NFP_NET_CFG_CTRL_MSIX_TX_OFF;
+
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_VEC(0), 1);
+	}
+
 	ret = nfp_reconfig(hw, new_ctrl, update);
 	if (ret < 0)
 		return -EIO;
@@ -164,20 +178,24 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 }
 
 static void
-nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
+		bool relay)
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+	if (!relay) {
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[1].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[0].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	}
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
 	rte_wmb();
@@ -189,12 +207,23 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nfp_vdpa_queue_config(vdpa_hw);
+	nfp_vdpa_queue_config(vdpa_hw, false);
 
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	return nfp_vdpa_vf_config(hw, vid);
+	return nfp_vdpa_vf_config(hw, vid, false);
+}
+
+int
+nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw, true);
+
+	return nfp_vdpa_vf_config(hw, vid, true);
 }
 
 void
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index 0f880fc0c6..a339ace601 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -9,12 +9,15 @@
 #include <bus_pci_driver.h>
 #include <nfp_common.h>
 #include <rte_ether.h>
+#include <rte_vhost.h>
 
 #define NFP_VDPA_MAX_QUEUES         1
 
 #define NFP_VDPA_NOTIFY_ADDR_BASE        0x4000
 #define NFP_VDPA_NOTIFY_ADDR_INTERVAL    0x1000
 
+#define NFP_VDPA_RELAY_VRING             0xd0000000
+
 struct nfp_vdpa_vring {
 	uint64_t desc;
 	uint64_t avail;
@@ -40,12 +43,17 @@ struct nfp_vdpa_hw {
 	/** Software Live Migration */
 	bool sw_lm;
 	bool sw_fallback_running;
+
+	/** Mediated vring for SW fallback */
+	struct vring m_vring[NFP_VDPA_MAX_QUEUES * 2];
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
 
 int nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
 
+int nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
+
 void nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw);
 
 void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
-- 
2.39.1

[PATCH v2 08/11] vdpa/nfp: recover the ring index on new host

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

After migrating to new host, the vring information is
recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
and 'NFP_NET_CFG_RX_USED_INDEX'.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 9311d01590..4b273372a8 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -193,8 +193,11 @@ struct nfp_net_fw_ver {
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
 
-/* 16B reserved for future use (0x00b0 - 0x00c0). */
-#define NFP_NET_CFG_RESERVED            0x00b0
+#define NFP_NET_CFG_TX_USED_INDEX       0x00b0
+#define NFP_NET_CFG_RX_USED_INDEX       0x00b4
+
+/* 16B reserved for future use (0x00b8 - 0x0010). */
+#define NFP_NET_CFG_RESERVED            0x00b8
 #define NFP_NET_CFG_RESERVED_SZ         0x0010
 
 /*
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 2b609dddc2..d7c48e2490 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -100,6 +100,16 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_hw_queue_init(struct nfp_vdpa_hw *vdpa_hw)
+{
+	/* Distribute ring information to firmware */
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_TX_USED_INDEX,
+			vdpa_hw->vring[1].last_used_idx);
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_RX_USED_INDEX,
+			vdpa_hw->vring[0].last_used_idx);
+}
+
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
@@ -198,6 +208,9 @@ nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
 
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
+	if (!relay)
+		nfp_vdpa_hw_queue_init(vdpa_hw);
+
 	rte_wmb();
 }
 
-- 
2.39.1

[PATCH v2 09/11] vdpa/nfp: setup vring relay thread

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Setup the vring relay thread to monitor the interruption from
device. And do the dirty page logging or notify device according
to event data.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
 drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
 drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
 3 files changed, 159 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 65f7144671..e57765eb1a 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -26,6 +26,8 @@
 #define NFP_VDPA_USED_RING_LEN(size) \
 		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
 
+#define EPOLL_DATA_INTR        1
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -776,6 +778,139 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_vring_epoll_ctl(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint32_t qid;
+	struct epoll_event ev;
+	struct rte_vhost_vring vring;
+
+	for (qid = 0; qid < queue_num; qid++) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		rte_vhost_get_vhost_vring(device->vid, qid, &vring);
+		ev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD, vring.kickfd, &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+	}
+
+	/* vDPA driver interrupt */
+	for (qid = 0; qid < queue_num; qid += 2) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		/* Leave a flag to mark it's for interrupt */
+		ev.data.u64 = EPOLL_DATA_INTR | qid << 1 |
+				(uint64_t)device->intr_fd[qid] << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD,
+				device->intr_fd[qid], &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+
+		nfp_vdpa_update_used_ring(device, qid);
+	}
+
+	return 0;
+}
+
+static int
+nfp_vdpa_vring_epoll_wait(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int i;
+	int fds;
+	int kickfd;
+	uint32_t qid;
+	struct epoll_event events[NFP_VDPA_MAX_QUEUES * 2];
+
+	for (;;) {
+		fds = epoll_wait(device->epoll_fd, events, queue_num * 2, -1);
+		if (fds < 0) {
+			if (errno == EINTR)
+				continue;
+
+			DRV_VDPA_LOG(ERR, "Epoll wait fail");
+			return -EACCES;
+		}
+
+		for (i = 0; i < fds; i++) {
+			qid = events[i].data.u32 >> 1;
+			kickfd = (uint32_t)(events[i].data.u64 >> 32);
+
+			nfp_vdpa_read_kickfd(kickfd);
+			if ((events[i].data.u32 & EPOLL_DATA_INTR) != 0) {
+				nfp_vdpa_update_used_ring(device, qid);
+				nfp_vdpa_irq_unmask(&device->hw);
+			} else {
+				nfp_vdpa_notify_queue(&device->hw, qid);
+			}
+		}
+	}
+
+	return 0;
+}
+
+static uint32_t
+nfp_vdpa_vring_relay(void *arg)
+{
+	int ret;
+	int epoll_fd;
+	uint16_t queue_id;
+	uint32_t queue_num;
+	struct nfp_vdpa_dev *device = arg;
+
+	epoll_fd = epoll_create(NFP_VDPA_MAX_QUEUES * 2);
+	if (epoll_fd < 0) {
+		DRV_VDPA_LOG(ERR, "failed to create epoll instance.");
+		return 1;
+	}
+
+	device->epoll_fd = epoll_fd;
+
+	queue_num = rte_vhost_get_vring_num(device->vid);
+
+	ret = nfp_vdpa_vring_epoll_ctl(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	/* Start relay with a first kick */
+	for (queue_id = 0; queue_id < queue_num; queue_id++)
+		nfp_vdpa_notify_queue(&device->hw, queue_id);
+
+	ret = nfp_vdpa_vring_epoll_wait(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	return 0;
+
+notify_exit:
+	close(device->epoll_fd);
+	device->epoll_fd = -1;
+
+	return 1;
+}
+
+static int
+nfp_vdpa_setup_vring_relay(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	char name[RTE_THREAD_INTERNAL_NAME_SIZE];
+
+	snprintf(name, sizeof(name), "nfp_vring%d", device->vid);
+	ret = rte_thread_create_internal_control(&device->tid, name,
+			nfp_vdpa_vring_relay, (void *)device);
+	if (ret != 0) {
+		DRV_VDPA_LOG(ERR, "Failed to create vring relay pthread.");
+		return -EPERM;
+	}
+
+	return 0;
+}
+
 static int
 nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 {
@@ -802,10 +937,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto unset_intr;
 
+	/* Setup vring relay thread */
+	ret = nfp_vdpa_setup_vring_relay(device);
+	if (ret != 0)
+		goto stop_vf;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+stop_vf:
+	nfp_vdpa_stop(device, true);
 unset_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 error:
@@ -859,6 +1001,12 @@ nfp_vdpa_dev_close(int vid)
 		/* Reset VF */
 		nfp_vdpa_stop(device, true);
 
+		/* Remove interrupt setting */
+		nfp_vdpa_disable_vfio_intr(device);
+
+		/* Unset DMA map for guest memory */
+		nfp_vdpa_dma_map(device, false);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index d7c48e2490..3b3481a99c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -270,3 +270,12 @@ nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_qcp_notify_ptr_add(vdpa_hw->notify_addr[qid],
 			NFP_QCP_NOTIFY_WRITE_PTR, qid);
 }
+
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	/* Make sure all updates are written before un-masking */
+	rte_wmb();
+	nn_cfg_writeb(hw, NFP_NET_CFG_ICR(1), NFP_NET_CFG_ICR_UNMASKED);
+}
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a339ace601..bc4db556a2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -60,4 +60,6 @@ void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
 
 uint64_t nfp_vdpa_get_queue_notify_offset(struct nfp_vdpa_hw *vdpa_hw, int qid);
 
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw);
+
 #endif /* __NFP_VDPA_CORE_H__ */
-- 
2.39.1

[PATCH v2 10/11] vdpa/nfp: enable feature bits of live migration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the 'VHOST_F_LOG_ALL' feature bits inorder to enable the
live migration function.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 3b3481a99c..70aeb4a3ac 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -95,6 +95,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
+			(1ULL << VHOST_F_LOG_ALL) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
 
 	return 0;
-- 
2.39.1

[PATCH v2 11/11] doc: update nfp document

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the software assisted vDPA live migration feature
into NFP document.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 doc/guides/vdpadevs/nfp.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
index dc9e94dbc8..e4736d9f61 100644
--- a/doc/guides/vdpadevs/nfp.rst
+++ b/doc/guides/vdpadevs/nfp.rst
@@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
 
 This PMD uses (common/nfp) code to access the device firmware.
 
+Software Live Migration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Now the NFP vDPA driver only support software assisted live migration mode.
+In this mode, the driver will setup a software relay thread when live migration
+happens, this thread will help device to log dirty pages. Although this mode
+does not require hardware to implement a dirty page logging function block, it
+will consume percentage of CPU resource depending on the network throughput.
+
 Per-Device Parameters
 ~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.39.1

[PATCH v3 00/11] support software live migration

[Chaoyong He]

This patch series aims to add the support of software live migration
feature for NFP vDPA device.

---
v3:
* Fix one compile error when using standard atomic.
v2:
* Adjust some spell in the commit message.
* Split out a commit to enable this feature.
---

Xinying Yu (11):
  mailmap: add new contributor
  vdpa/nfp: fix logic in hardware init
  vdpa/nfp: fix the logic of reconfiguration
  vdpa/nfp: refactor the logic of datapath update
  vdpa/nfp: add the live migration logic
  vdpa/nfp: add the interrupt logic of vring relay
  vdpa/nfp: setup the VF configure
  vdpa/nfp: recover the ring index on new host
  vdpa/nfp: setup vring relay thread
  vdpa/nfp: enable feature bits of live migration
  doc: update nfp document

 .mailmap                             |   1 +
 doc/guides/vdpadevs/nfp.rst          |   9 +
 drivers/common/nfp/nfp_common_ctrl.h |  11 +-
 drivers/vdpa/nfp/nfp_vdpa.c          | 441 +++++++++++++++++++++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
 6 files changed, 565 insertions(+), 46 deletions(-)

-- 
2.39.1

[PATCH v3 01/11] mailmap: add new contributor

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add new contributor.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 .mailmap | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.mailmap b/.mailmap
index 6b396107d0..01c31b8c70 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1598,6 +1598,7 @@ Xieming Katty <katty.xieming@huawei.com>
 Xinfeng Zhao <xinfengx.zhao@intel.com>
 Xingguang He <xingguang.he@intel.com>
 Xingyou Chen <niatlantice@gmail.com>
+Xinying Yu <xinying.yu@corigine.com>
 Xin Long <longxin.xl@alibaba-inc.com>
 Xi Zhang <xix.zhang@intel.com>
 Xuan Ding <xuan.ding@intel.com>
-- 
2.39.1

Re: [PATCH v3 01/11] mailmap: add new contributor

[Maxime Coquelin]

On 6/17/24 08:26, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add new contributor.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   .mailmap | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/.mailmap b/.mailmap
> index 6b396107d0..01c31b8c70 100644
> --- a/.mailmap
> +++ b/.mailmap
> @@ -1598,6 +1598,7 @@ Xieming Katty <katty.xieming@huawei.com>
>   Xinfeng Zhao <xinfengx.zhao@intel.com>
>   Xingguang He <xingguang.he@intel.com>
>   Xingyou Chen <niatlantice@gmail.com>
> +Xinying Yu <xinying.yu@corigine.com>
>   Xin Long <longxin.xl@alibaba-inc.com>
>   Xi Zhang <xix.zhang@intel.com>
>   Xuan Ding <xuan.ding@intel.com>

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 02/11] vdpa/nfp: fix logic in hardware init

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Reconfigure the NIC will fail because lack of the
initialization logic of queue configuration pointer.
Fix this by adding the correct initialization logic.

Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 7b877605e4..291798196c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -55,7 +55,10 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 		struct rte_pci_device *pci_dev)
 {
 	uint32_t queue;
+	uint8_t *tx_bar;
+	uint32_t start_q;
 	struct nfp_hw *hw;
+	uint32_t tx_bar_off;
 	uint8_t *notify_base;
 
 	hw = &vdpa_hw->super;
@@ -82,6 +85,12 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 				idx + 1, vdpa_hw->notify_addr[idx + 1]);
 	}
 
+	/* NFP vDPA cfg queue setup */
+	start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
+	tx_bar_off = start_q * NFP_QCP_QUEUE_ADDR_SZ;
+	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
+	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
-- 
2.39.1

Re: [PATCH v3 02/11] vdpa/nfp: fix logic in hardware init

[Maxime Coquelin]

On 6/17/24 08:26, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Reconfigure the NIC will fail because lack of the
> initialization logic of queue configuration pointer.
> Fix this by adding the correct initialization logic.
> 
> Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
> Cc: chaoyong.he@corigine.com
> Cc: stable@dpdk.org
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 03/11] vdpa/nfp: fix the logic of reconfiguration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

The ctrl words of vDPA is located on the extended word, so it
should use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.

Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
with 'NFP_NET_CFG_CTRL_VIRTIO'.

Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  1 +
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 6badf769fc..a0e62b063d 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -184,6 +184,7 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_IPSEC_LM_LOOKUP  (0x1 << 4) /**< SA long match lookup */
 #define NFP_NET_CFG_CTRL_MULTI_PF         (0x1 << 5)
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
+#define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 291798196c..6d07356581 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -101,7 +101,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
-	return NFP_NET_CFG_CTRL_SCATTER |
+	return NFP_NET_CFG_CTRL_VIRTIO  |
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
@@ -112,6 +112,7 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	int ret;
 	uint32_t update;
 	uint32_t new_ctrl;
+	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
 	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
@@ -131,8 +132,6 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	new_ctrl = nfp_vdpa_check_offloads();
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -147,8 +146,17 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	/* Writing new MAC to the specific port BAR address */
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
+	new_ext_ctrl = nfp_vdpa_check_offloads();
+
+	update = NFP_NET_CFG_UPDATE_GEN;
+	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
+	if (ret != 0)
+		return -EIO;
+
+	hw->ctrl_ext = new_ext_ctrl;
+
 	/* Enable device */
-	new_ctrl |= NFP_NET_CFG_CTRL_ENABLE;
+	new_ctrl = NFP_NET_CFG_CTRL_ENABLE;
 
 	/* Signal the NIC about the change */
 	update = NFP_NET_CFG_UPDATE_MACADDR |
-- 
2.39.1

Re: [PATCH v3 03/11] vdpa/nfp: fix the logic of reconfiguration

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> The ctrl words of vDPA is located on the extended word, so it
> should use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.
> 
> Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
> with 'NFP_NET_CFG_CTRL_VIRTIO'.
> 
> Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
> Cc: chaoyong.he@corigine.com
> Cc: stable@dpdk.org
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |  1 +
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
>   2 files changed, 13 insertions(+), 4 deletions(-)
> 
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 04/11] vdpa/nfp: refactor the logic of datapath update

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

In order to add the new configuration logic of software live
migration, split the datapath update logic into two parts,
queue configuration and VF configuration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 18 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 6d07356581..79ecd2b4fc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -105,8 +105,8 @@ nfp_vdpa_check_offloads(void)
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
-int
-nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+static int
+nfp_vdpa_vf_config(struct nfp_hw *hw,
 		int vid)
 {
 	int ret;
@@ -114,24 +114,8 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	uint32_t new_ctrl;
 	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
-	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0), rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
-
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0), rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
-
-	rte_wmb();
-
-	nfp_disable_queues(hw);
-	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -177,6 +161,40 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[1].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[0].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
+
+	rte_wmb();
+}
+
+int
+nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw);
+
+	nfp_disable_queues(hw);
+	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
+
+	return nfp_vdpa_vf_config(hw, vid);
+}
+
 void
 nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw)
 {
-- 
2.39.1

Re: [PATCH v3 04/11] vdpa/nfp: refactor the logic of datapath update

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> In order to add the new configuration logic of software live
> migration, split the datapath update logic into two parts,
> queue configuration and VF configuration.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
>   1 file changed, 36 insertions(+), 18 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 05/11] vdpa/nfp: add the live migration logic

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the basic logic of software live migration.

Unset the ring notify area to stop the direct IO datapath if the
device support, then we can setup the vring relay to help the
live migration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
 drivers/vdpa/nfp/nfp_vdpa_core.c |  2 +
 drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
 3 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index cef80b5476..45092cb0af 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	int vid = device->vid;
+
+	/* Stop the direct IO data path */
+	nfp_vdpa_unset_notify_relay(device);
+	nfp_vdpa_disable_vfio_intr(device);
+
+	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
+	if ((ret != 0) && (ret != -ENOTSUP)) {
+		DRV_VDPA_LOG(ERR, "Unset the host notifier failed.");
+		goto error;
+	}
+
+	device->hw.sw_fallback_running = true;
+
+	return 0;
+
+error:
+	return ret;
+}
+
 static int
 nfp_vdpa_dev_config(int vid)
 {
@@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid)
 	}
 
 	device = node->device;
-	rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed);
-	update_datapath(device);
+	if (device->hw.sw_fallback_running) {
+		device->hw.sw_fallback_running = false;
+
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		rte_atomic_store_explicit(&device->running, 0,
+				rte_memory_order_relaxed);
+	} else {
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		update_datapath(device);
+	}
 
 	return 0;
 }
@@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused,
 static int
 nfp_vdpa_set_features(int32_t vid)
 {
+	int ret;
+	uint64_t features = 0;
+	struct nfp_vdpa_dev *device;
+	struct rte_vdpa_device *vdev;
+	struct nfp_vdpa_dev_node *node;
+
 	DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid);
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	node = nfp_vdpa_find_node_by_vdev(vdev);
+	if (node == NULL) {
+		DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -ENODEV;
+	}
+
+	rte_vhost_get_negotiated_features(vid, &features);
+
+	if (RTE_VHOST_NEED_LOG(features) == 0)
+		return 0;
+
+	device = node->device;
+	if (device->hw.sw_lm) {
+		ret = nfp_vdpa_sw_fallback(device);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "Software fallback start failed");
+			return -1;
+		}
+	}
+
 	return 0;
 }
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 79ecd2b4fc..50eda4cb2c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -91,6 +91,8 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
 	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 
+	vdpa_hw->sw_lm = true;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a8e0d6dd70..0f880fc0c6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 	uint8_t notify_region;
 	uint8_t nr_vring;
+
+	/** Software Live Migration */
+	bool sw_lm;
+	bool sw_fallback_running;
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
-- 
2.39.1

Re: [PATCH v3 05/11] vdpa/nfp: add the live migration logic

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the basic logic of software live migration.
> 
> Unset the ring notify area to stop the direct IO datapath if the
> device support, then we can setup the vring relay to help the
> live migration.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
>   drivers/vdpa/nfp/nfp_vdpa_core.c |  2 +
>   drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
>   3 files changed, 70 insertions(+), 2 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 06/11] vdpa/nfp: add the interrupt logic of vring relay

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the interrupt setup logic of vring relay.

The epoll fd is provided here so host can get the interrupt from device
on Rx direction, all other operations on vring relay are based on this.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 45092cb0af..1643ebbb8c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -336,8 +336,10 @@ nfp_vdpa_stop(struct nfp_vdpa_dev *device)
 }
 
 static int
-nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
+nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device,
+		bool relay)
 {
+	int fd;
 	int ret;
 	uint16_t i;
 	int *fd_ptr;
@@ -366,6 +368,19 @@ nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
 	}
 
+	if (relay) {
+		for (i = 0; i < nr_vring; i += 2) {
+			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+			if (fd < 0) {
+				DRV_VDPA_LOG(ERR, "Can't setup eventfd");
+				return -EINVAL;
+			}
+
+			device->intr_fd[i] = fd;
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
+		}
+	}
+
 	ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
 	if (ret != 0) {
 		DRV_VDPA_LOG(ERR, "Error enabling MSI-X interrupts.");
@@ -556,7 +571,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto unlock_exit;
 
-		ret = nfp_vdpa_enable_vfio_intr(device);
+		ret = nfp_vdpa_enable_vfio_intr(device, false);
 		if (ret != 0)
 			goto dma_map_rollback;
 
@@ -619,6 +634,11 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 		goto error;
 	}
 
+	/* Setup interrupt for vring relay */
+	ret = nfp_vdpa_enable_vfio_intr(device, true);
+	if (ret != 0)
+		goto error;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
-- 
2.39.1

Re: [PATCH v3 06/11] vdpa/nfp: add the interrupt logic of vring relay

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the interrupt setup logic of vring relay.
> 
> The epoll fd is provided here so host can get the interrupt from device
> on Rx direction, all other operations on vring relay are based on this.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
>   1 file changed, 22 insertions(+), 2 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 07/11] vdpa/nfp: setup the VF configure

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Create the relay vring on host and then set the address of Rx
used ring to the VF config bar. So the device can DMA the
used ring information to host rather than directly to VM.

Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
enable the MSIX interrupt on device.

Tx ring address is not needed to change since the relay vring
only assists Rx ring to do the dirty page logging.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |   3 +
 drivers/vdpa/nfp/nfp_vdpa.c          | 203 ++++++++++++++++++++++++---
 drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
 4 files changed, 239 insertions(+), 30 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index a0e62b063d..9311d01590 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -186,6 +186,9 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
 #define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
+#define NFP_NET_CFG_CTRL_LM_RELAY         (0x1 << 12) /**< Virtio live migration relay start */
+#define NFP_NET_CFG_CTRL_NOTIFY_DATA      (0x1 << 13) /**< Virtio notification data flag */
+#define NFP_NET_CFG_CTRL_SWLM             (0x1 << 14) /**< Virtio SW live migration enable */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 1643ebbb8c..983123ba08 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -11,6 +11,8 @@
 #include <nfp_common_pci.h>
 #include <nfp_dev.h>
 #include <rte_vfio.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
 #include <vdpa_driver.h>
 
 #include "nfp_vdpa_core.h"
@@ -21,6 +23,9 @@
 #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
 		sizeof(int) * (NFP_VDPA_MAX_QUEUES * 2 + 1))
 
+#define NFP_VDPA_USED_RING_LEN(size) \
+		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -261,15 +266,85 @@ nfp_vdpa_qva_to_gpa(int vid,
 	return gpa;
 }
 
+static void
+nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
+		uint16_t vring_index)
+{
+	uint16_t i;
+	uint64_t size;
+	struct rte_vhost_vring vring;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	for (i = 0; i < vring_index; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
+				rte_mem_page_size());
+		rte_vfio_container_dma_unmap(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
+				m_vring_iova, size);
+
+		rte_free(device->hw.m_vring[i].desc);
+		m_vring_iova += size;
+	}
+}
+
 static int
-nfp_vdpa_start(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint16_t i;
+	uint64_t size;
+	void *vring_buf;
+	uint64_t page_size;
+	struct rte_vhost_vring vring;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	page_size = rte_mem_page_size();
+
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, page_size), page_size);
+		vring_buf = rte_zmalloc("nfp_vdpa_relay", size, page_size);
+		if (vring_buf == NULL)
+			goto vring_free_all;
+
+		vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
+
+		ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
+			goto vring_free_one;
+		}
+
+		m_vring_iova += size;
+	}
+
+	return 0;
+
+vring_free_one:
+	rte_free(device->hw.m_vring[i].desc);
+vring_free_all:
+	nfp_vdpa_relay_vring_free(device, i);
+
+	return -ENOSPC;
+}
+
+static int
+nfp_vdpa_start(struct nfp_vdpa_dev *device,
+		bool relay)
 {
 	int ret;
 	int vid;
 	uint16_t i;
 	uint64_t gpa;
+	uint16_t size;
 	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
 
 	vid = device->vid;
 	vdpa_hw->nr_vring = rte_vhost_get_vring_num(vid);
@@ -278,15 +353,21 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		return ret;
 
+	if (relay) {
+		ret = nfp_vdpa_relay_vring_alloc(device);
+		if (ret != 0)
+			return ret;
+	}
+
 	for (i = 0; i < vdpa_hw->nr_vring; i++) {
 		ret = rte_vhost_get_vhost_vring(vid, i, &vring);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.desc);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for descriptor ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].desc = gpa;
@@ -294,45 +375,123 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.avail);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for available ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].avail = gpa;
 
-		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
-		if (gpa == 0) {
-			DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
-			return -1;
-		}
+		/* Direct I/O for Tx queue, relay for Rx queue */
+		if (relay && ((i & 1) == 0)) {
+			vdpa_hw->vring[i].used = m_vring_iova +
+					(char *)vdpa_hw->m_vring[i].used -
+					(char *)vdpa_hw->m_vring[i].desc;
+
+			ret = rte_vhost_get_vring_base(vid, i,
+					&vdpa_hw->m_vring[i].avail->idx,
+					&vdpa_hw->m_vring[i].used->idx);
+			if (ret != 0)
+				goto relay_vring_free;
+		} else {
+			gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
+			if (gpa == 0) {
+				DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
+				goto relay_vring_free;
+			}
 
-		vdpa_hw->vring[i].used = gpa;
+			vdpa_hw->vring[i].used = gpa;
+		}
 
 		vdpa_hw->vring[i].size = vring.size;
 
+		if (relay) {
+			size = RTE_ALIGN_CEIL(vring_size(vring.size,
+					rte_mem_page_size()), rte_mem_page_size());
+			m_vring_iova += size;
+		}
+
 		ret = rte_vhost_get_vring_base(vid, i,
 				&vdpa_hw->vring[i].last_avail_idx,
 				&vdpa_hw->vring[i].last_used_idx);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 	}
 
-	return nfp_vdpa_hw_start(&device->hw, vid);
+	if (relay)
+		return nfp_vdpa_relay_hw_start(&device->hw, vid);
+	else
+		return nfp_vdpa_hw_start(&device->hw, vid);
+
+relay_vring_free:
+	if (relay)
+		nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+
+	return -EFAULT;
+}
+
+static void
+nfp_vdpa_update_used_ring(struct nfp_vdpa_dev *dev,
+		uint16_t qid)
+{
+	rte_vdpa_relay_vring_used(dev->vid, qid, &dev->hw.m_vring[qid]);
+	rte_vhost_vring_call(dev->vid, qid);
 }
 
 static void
-nfp_vdpa_stop(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_stop(struct nfp_vdpa_dev *device)
 {
 	int vid;
 	uint32_t i;
+	uint64_t len;
+	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
 
 	nfp_vdpa_hw_stop(vdpa_hw);
 
 	vid = device->vid;
-	for (i = 0; i < vdpa_hw->nr_vring; i++)
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		/* Synchronize remaining new used entries if any */
+		if ((i & 1) == 0)
+			nfp_vdpa_update_used_ring(device, i);
+
+		rte_vhost_get_vhost_vring(vid, i, &vring);
+		len = NFP_VDPA_USED_RING_LEN(vring.size);
+		vdpa_hw->vring[i].last_avail_idx = vring.avail->idx;
+		vdpa_hw->vring[i].last_used_idx = vring.used->idx;
+
 		rte_vhost_set_vring_base(vid, i,
 				vdpa_hw->vring[i].last_avail_idx,
 				vdpa_hw->vring[i].last_used_idx);
+
+		rte_vhost_log_used_vring(vid, i, 0, len);
+
+		if (vring.used->idx != vring.avail->idx)
+			rte_atomic_store_explicit(
+					(unsigned short __rte_atomic *)&vring.used->idx,
+					vring.avail->idx, rte_memory_order_release);
+	}
+
+	nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+}
+
+static void
+nfp_vdpa_stop(struct nfp_vdpa_dev *device,
+		bool relay)
+{
+	int vid;
+	uint32_t i;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+
+	nfp_vdpa_hw_stop(vdpa_hw);
+
+	vid = device->vid;
+	if (relay)
+		nfp_vdpa_relay_stop(device);
+	else
+		for (i = 0; i < vdpa_hw->nr_vring; i++)
+			rte_vhost_set_vring_base(vid, i,
+					vdpa_hw->vring[i].last_avail_idx,
+					vdpa_hw->vring[i].last_used_idx);
+
 }
 
 static int
@@ -575,7 +734,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto dma_map_rollback;
 
-		ret = nfp_vdpa_start(device);
+		ret = nfp_vdpa_start(device, false);
 		if (ret != 0)
 			goto disable_vfio_intr;
 
@@ -591,7 +750,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 					rte_memory_order_relaxed) != 0))) {
 		nfp_vdpa_unset_notify_relay(device);
 
-		nfp_vdpa_stop(device);
+		nfp_vdpa_stop(device, false);
 
 		ret = nfp_vdpa_disable_vfio_intr(device);
 		if (ret != 0)
@@ -608,7 +767,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return 0;
 
 vdpa_stop:
-	nfp_vdpa_stop(device);
+	nfp_vdpa_stop(device, false);
 disable_vfio_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 dma_map_rollback:
@@ -639,10 +798,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto error;
 
+	/* Config the VF */
+	ret = nfp_vdpa_start(device, true);
+	if (ret != 0)
+		goto unset_intr;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+unset_intr:
+	nfp_vdpa_disable_vfio_intr(device);
 error:
 	return ret;
 }
@@ -691,6 +857,9 @@ nfp_vdpa_dev_close(int vid)
 
 	device = node->device;
 	if (device->hw.sw_fallback_running) {
+		/* Reset VF */
+		nfp_vdpa_stop(device, true);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 50eda4cb2c..2b609dddc2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -109,7 +109,8 @@ nfp_vdpa_check_offloads(void)
 
 static int
 nfp_vdpa_vf_config(struct nfp_hw *hw,
-		int vid)
+		int vid,
+		bool relay)
 {
 	int ret;
 	uint32_t update;
@@ -133,6 +134,10 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
 	new_ext_ctrl = nfp_vdpa_check_offloads();
+	if (relay)
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_LM_RELAY;
+	else
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_SWLM;
 
 	update = NFP_NET_CFG_UPDATE_GEN;
 	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
@@ -149,6 +154,15 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 			NFP_NET_CFG_UPDATE_GEN |
 			NFP_NET_CFG_UPDATE_RING;
 
+	if (relay) {
+		update |= NFP_NET_CFG_UPDATE_MSIX;
+
+		/* Enable misx interrupt for vdpa relay */
+		new_ctrl |= NFP_NET_CFG_CTRL_MSIX_TX_OFF;
+
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_VEC(0), 1);
+	}
+
 	ret = nfp_reconfig(hw, new_ctrl, update);
 	if (ret < 0)
 		return -EIO;
@@ -164,20 +178,24 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 }
 
 static void
-nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
+		bool relay)
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+	if (!relay) {
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[1].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[0].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	}
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
 	rte_wmb();
@@ -189,12 +207,23 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nfp_vdpa_queue_config(vdpa_hw);
+	nfp_vdpa_queue_config(vdpa_hw, false);
 
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	return nfp_vdpa_vf_config(hw, vid);
+	return nfp_vdpa_vf_config(hw, vid, false);
+}
+
+int
+nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw, true);
+
+	return nfp_vdpa_vf_config(hw, vid, true);
 }
 
 void
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index 0f880fc0c6..a339ace601 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -9,12 +9,15 @@
 #include <bus_pci_driver.h>
 #include <nfp_common.h>
 #include <rte_ether.h>
+#include <rte_vhost.h>
 
 #define NFP_VDPA_MAX_QUEUES         1
 
 #define NFP_VDPA_NOTIFY_ADDR_BASE        0x4000
 #define NFP_VDPA_NOTIFY_ADDR_INTERVAL    0x1000
 
+#define NFP_VDPA_RELAY_VRING             0xd0000000
+
 struct nfp_vdpa_vring {
 	uint64_t desc;
 	uint64_t avail;
@@ -40,12 +43,17 @@ struct nfp_vdpa_hw {
 	/** Software Live Migration */
 	bool sw_lm;
 	bool sw_fallback_running;
+
+	/** Mediated vring for SW fallback */
+	struct vring m_vring[NFP_VDPA_MAX_QUEUES * 2];
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
 
 int nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
 
+int nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
+
 void nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw);
 
 void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
-- 
2.39.1

Re: [PATCH v3 07/11] vdpa/nfp: setup the VF configure

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Create the relay vring on host and then set the address of Rx
> used ring to the VF config bar. So the device can DMA the
> used ring information to host rather than directly to VM.
> 
> Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
> enable the MSIX interrupt on device.
> 
> Tx ring address is not needed to change since the relay vring
> only assists Rx ring to do the dirty page logging.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |   3 +
>   drivers/vdpa/nfp/nfp_vdpa.c          | 203 ++++++++++++++++++++++++---
>   drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
>   4 files changed, 239 insertions(+), 30 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 08/11] vdpa/nfp: recover the ring index on new host

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

After migrating to new host, the vring information is
recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
and 'NFP_NET_CFG_RX_USED_INDEX'.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 9311d01590..4b273372a8 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -193,8 +193,11 @@ struct nfp_net_fw_ver {
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
 
-/* 16B reserved for future use (0x00b0 - 0x00c0). */
-#define NFP_NET_CFG_RESERVED            0x00b0
+#define NFP_NET_CFG_TX_USED_INDEX       0x00b0
+#define NFP_NET_CFG_RX_USED_INDEX       0x00b4
+
+/* 16B reserved for future use (0x00b8 - 0x0010). */
+#define NFP_NET_CFG_RESERVED            0x00b8
 #define NFP_NET_CFG_RESERVED_SZ         0x0010
 
 /*
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 2b609dddc2..d7c48e2490 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -100,6 +100,16 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_hw_queue_init(struct nfp_vdpa_hw *vdpa_hw)
+{
+	/* Distribute ring information to firmware */
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_TX_USED_INDEX,
+			vdpa_hw->vring[1].last_used_idx);
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_RX_USED_INDEX,
+			vdpa_hw->vring[0].last_used_idx);
+}
+
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
@@ -198,6 +208,9 @@ nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
 
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
+	if (!relay)
+		nfp_vdpa_hw_queue_init(vdpa_hw);
+
 	rte_wmb();
 }
 
-- 
2.39.1

Re: [PATCH v3 08/11] vdpa/nfp: recover the ring index on new host

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> After migrating to new host, the vring information is
> recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
> and 'NFP_NET_CFG_RX_USED_INDEX'.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
>   2 files changed, 18 insertions(+), 2 deletions(-)
> 
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 09/11] vdpa/nfp: setup vring relay thread

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Setup the vring relay thread to monitor the interruption from
device. And do the dirty page logging or notify device according
to event data.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
 drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
 drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
 3 files changed, 159 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 983123ba08..91f1b8d779 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -26,6 +26,8 @@
 #define NFP_VDPA_USED_RING_LEN(size) \
 		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
 
+#define EPOLL_DATA_INTR        1
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -777,6 +779,139 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_vring_epoll_ctl(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint32_t qid;
+	struct epoll_event ev;
+	struct rte_vhost_vring vring;
+
+	for (qid = 0; qid < queue_num; qid++) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		rte_vhost_get_vhost_vring(device->vid, qid, &vring);
+		ev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD, vring.kickfd, &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+	}
+
+	/* vDPA driver interrupt */
+	for (qid = 0; qid < queue_num; qid += 2) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		/* Leave a flag to mark it's for interrupt */
+		ev.data.u64 = EPOLL_DATA_INTR | qid << 1 |
+				(uint64_t)device->intr_fd[qid] << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD,
+				device->intr_fd[qid], &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+
+		nfp_vdpa_update_used_ring(device, qid);
+	}
+
+	return 0;
+}
+
+static int
+nfp_vdpa_vring_epoll_wait(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int i;
+	int fds;
+	int kickfd;
+	uint32_t qid;
+	struct epoll_event events[NFP_VDPA_MAX_QUEUES * 2];
+
+	for (;;) {
+		fds = epoll_wait(device->epoll_fd, events, queue_num * 2, -1);
+		if (fds < 0) {
+			if (errno == EINTR)
+				continue;
+
+			DRV_VDPA_LOG(ERR, "Epoll wait fail");
+			return -EACCES;
+		}
+
+		for (i = 0; i < fds; i++) {
+			qid = events[i].data.u32 >> 1;
+			kickfd = (uint32_t)(events[i].data.u64 >> 32);
+
+			nfp_vdpa_read_kickfd(kickfd);
+			if ((events[i].data.u32 & EPOLL_DATA_INTR) != 0) {
+				nfp_vdpa_update_used_ring(device, qid);
+				nfp_vdpa_irq_unmask(&device->hw);
+			} else {
+				nfp_vdpa_notify_queue(&device->hw, qid);
+			}
+		}
+	}
+
+	return 0;
+}
+
+static uint32_t
+nfp_vdpa_vring_relay(void *arg)
+{
+	int ret;
+	int epoll_fd;
+	uint16_t queue_id;
+	uint32_t queue_num;
+	struct nfp_vdpa_dev *device = arg;
+
+	epoll_fd = epoll_create(NFP_VDPA_MAX_QUEUES * 2);
+	if (epoll_fd < 0) {
+		DRV_VDPA_LOG(ERR, "failed to create epoll instance.");
+		return 1;
+	}
+
+	device->epoll_fd = epoll_fd;
+
+	queue_num = rte_vhost_get_vring_num(device->vid);
+
+	ret = nfp_vdpa_vring_epoll_ctl(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	/* Start relay with a first kick */
+	for (queue_id = 0; queue_id < queue_num; queue_id++)
+		nfp_vdpa_notify_queue(&device->hw, queue_id);
+
+	ret = nfp_vdpa_vring_epoll_wait(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	return 0;
+
+notify_exit:
+	close(device->epoll_fd);
+	device->epoll_fd = -1;
+
+	return 1;
+}
+
+static int
+nfp_vdpa_setup_vring_relay(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	char name[RTE_THREAD_INTERNAL_NAME_SIZE];
+
+	snprintf(name, sizeof(name), "nfp_vring%d", device->vid);
+	ret = rte_thread_create_internal_control(&device->tid, name,
+			nfp_vdpa_vring_relay, (void *)device);
+	if (ret != 0) {
+		DRV_VDPA_LOG(ERR, "Failed to create vring relay pthread.");
+		return -EPERM;
+	}
+
+	return 0;
+}
+
 static int
 nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 {
@@ -803,10 +938,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto unset_intr;
 
+	/* Setup vring relay thread */
+	ret = nfp_vdpa_setup_vring_relay(device);
+	if (ret != 0)
+		goto stop_vf;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+stop_vf:
+	nfp_vdpa_stop(device, true);
 unset_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 error:
@@ -860,6 +1002,12 @@ nfp_vdpa_dev_close(int vid)
 		/* Reset VF */
 		nfp_vdpa_stop(device, true);
 
+		/* Remove interrupt setting */
+		nfp_vdpa_disable_vfio_intr(device);
+
+		/* Unset DMA map for guest memory */
+		nfp_vdpa_dma_map(device, false);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index d7c48e2490..3b3481a99c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -270,3 +270,12 @@ nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_qcp_notify_ptr_add(vdpa_hw->notify_addr[qid],
 			NFP_QCP_NOTIFY_WRITE_PTR, qid);
 }
+
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	/* Make sure all updates are written before un-masking */
+	rte_wmb();
+	nn_cfg_writeb(hw, NFP_NET_CFG_ICR(1), NFP_NET_CFG_ICR_UNMASKED);
+}
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a339ace601..bc4db556a2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -60,4 +60,6 @@ void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
 
 uint64_t nfp_vdpa_get_queue_notify_offset(struct nfp_vdpa_hw *vdpa_hw, int qid);
 
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw);
+
 #endif /* __NFP_VDPA_CORE_H__ */
-- 
2.39.1

Re: [PATCH v3 09/11] vdpa/nfp: setup vring relay thread

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Setup the vring relay thread to monitor the interruption from
> device. And do the dirty page logging or notify device according
> to event data.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
>   drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
>   drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
>   3 files changed, 159 insertions(+)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 10/11] vdpa/nfp: enable feature bits of live migration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the 'VHOST_F_LOG_ALL' feature bits inorder to enable the
live migration function.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 3b3481a99c..70aeb4a3ac 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -95,6 +95,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
+			(1ULL << VHOST_F_LOG_ALL) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
 
 	return 0;
-- 
2.39.1

Re: [PATCH v3 10/11] vdpa/nfp: enable feature bits of live migration

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the 'VHOST_F_LOG_ALL' feature bits inorder to enable the
> live migration function.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   drivers/vdpa/nfp/nfp_vdpa_core.c | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
> index 3b3481a99c..70aeb4a3ac 100644
> --- a/drivers/vdpa/nfp/nfp_vdpa_core.c
> +++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
> @@ -95,6 +95,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
>   
>   	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
>   			(1ULL << VIRTIO_F_IN_ORDER) |
> +			(1ULL << VHOST_F_LOG_ALL) |
>   			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
>   
>   	return 0;

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

[PATCH v3 11/11] doc: update nfp document

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the software assisted vDPA live migration feature
into NFP document.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 doc/guides/vdpadevs/nfp.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
index dc9e94dbc8..e4736d9f61 100644
--- a/doc/guides/vdpadevs/nfp.rst
+++ b/doc/guides/vdpadevs/nfp.rst
@@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
 
 This PMD uses (common/nfp) code to access the device firmware.
 
+Software Live Migration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Now the NFP vDPA driver only support software assisted live migration mode.
+In this mode, the driver will setup a software relay thread when live migration
+happens, this thread will help device to log dirty pages. Although this mode
+does not require hardware to implement a dirty page logging function block, it
+will consume percentage of CPU resource depending on the network throughput.
+
 Per-Device Parameters
 ~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.39.1

Re: [PATCH v3 11/11] doc: update nfp document

[Maxime Coquelin]

On 6/17/24 08:27, Chaoyong He wrote:
> From: Xinying Yu <xinying.yu@corigine.com>
> 
> Add the software assisted vDPA live migration feature
> into NFP document.
> 
> Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
> Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
> Reviewed-by: Long Wu <long.wu@corigine.com>
> Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
> ---
>   doc/guides/vdpadevs/nfp.rst | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
> index dc9e94dbc8..e4736d9f61 100644
> --- a/doc/guides/vdpadevs/nfp.rst
> +++ b/doc/guides/vdpadevs/nfp.rst
> @@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
>   
>   This PMD uses (common/nfp) code to access the device firmware.
>   
> +Software Live Migration
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Now the NFP vDPA driver only support software assisted live migration mode.
> +In this mode, the driver will setup a software relay thread when live migration
> +happens, this thread will help device to log dirty pages. Although this mode
> +does not require hardware to implement a dirty page logging function block, it
> +will consume percentage of CPU resource depending on the network throughput.
> +
>   Per-Device Parameters
>   ~~~~~~~~~~~~~~~~~~~~~
>   
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v3 00/11] support software live migration

[Maxime Coquelin]

Hi,

The series looks good to me, will apply it shortly.

For next time, when sending a new revision, could you please apply
all the R-By tags provided by reviewers on previous revision if the
patch did not change?

Thanks,
Maxime

On 6/17/24 08:26, Chaoyong He wrote:
> This patch series aims to add the support of software live migration
> feature for NFP vDPA device.
> 
> ---
> v3:
> * Fix one compile error when using standard atomic.
> v2:
> * Adjust some spell in the commit message.
> * Split out a commit to enable this feature.
> ---
> 
> Xinying Yu (11):
>    mailmap: add new contributor
>    vdpa/nfp: fix logic in hardware init
>    vdpa/nfp: fix the logic of reconfiguration
>    vdpa/nfp: refactor the logic of datapath update
>    vdpa/nfp: add the live migration logic
>    vdpa/nfp: add the interrupt logic of vring relay
>    vdpa/nfp: setup the VF configure
>    vdpa/nfp: recover the ring index on new host
>    vdpa/nfp: setup vring relay thread
>    vdpa/nfp: enable feature bits of live migration
>    doc: update nfp document
> 
>   .mailmap                             |   1 +
>   doc/guides/vdpadevs/nfp.rst          |   9 +
>   drivers/common/nfp/nfp_common_ctrl.h |  11 +-
>   drivers/vdpa/nfp/nfp_vdpa.c          | 441 +++++++++++++++++++++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
>   6 files changed, 565 insertions(+), 46 deletions(-)
> 

RE: [PATCH v3 00/11] support software live migration

[Chaoyong He]

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Tuesday, June 18, 2024 3:46 PM
> To: Chaoyong He <chaoyong.he@corigine.com>; dev@dpdk.org
> Cc: oss-drivers <oss-drivers@corigine.com>
> Subject: Re: [PATCH v3 00/11] support software live migration
> 
> Hi,
> 
> The series looks good to me, will apply it shortly.
> 
> For next time, when sending a new revision, could you please apply all the R-
> By tags provided by reviewers on previous revision if the patch did not
> change?

Oh, I got it and will do just as you said for the next time.
Thanks for your kind remind!

> 
> Thanks,
> Maxime
> 
> On 6/17/24 08:26, Chaoyong He wrote:
> > This patch series aims to add the support of software live migration
> > feature for NFP vDPA device.
> >
> > ---
> > v3:
> > * Fix one compile error when using standard atomic.
> > v2:
> > * Adjust some spell in the commit message.
> > * Split out a commit to enable this feature.
> > ---
> >
> > Xinying Yu (11):
> >    mailmap: add new contributor
> >    vdpa/nfp: fix logic in hardware init
> >    vdpa/nfp: fix the logic of reconfiguration
> >    vdpa/nfp: refactor the logic of datapath update
> >    vdpa/nfp: add the live migration logic
> >    vdpa/nfp: add the interrupt logic of vring relay
> >    vdpa/nfp: setup the VF configure
> >    vdpa/nfp: recover the ring index on new host
> >    vdpa/nfp: setup vring relay thread
> >    vdpa/nfp: enable feature bits of live migration
> >    doc: update nfp document
> >
> >   .mailmap                             |   1 +
> >   doc/guides/vdpadevs/nfp.rst          |   9 +
> >   drivers/common/nfp/nfp_common_ctrl.h |  11 +-
> >   drivers/vdpa/nfp/nfp_vdpa.c          | 441 +++++++++++++++++++++++++--
> >   drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
> >   drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
> >   6 files changed, 565 insertions(+), 46 deletions(-)
> >

[PATCH v4 00/11] support software live migration

[Chaoyong He]

This patch series aims to add the support of software live migration
feature for NFP vDPA device.

---
v4:
* Rebase to the newest main branch.
* Add the 'Review-by' tag of the external reviewer.
v3:
* Fix one compile error when using standard atomic.
v2:
* Adjust some spell in the commit message.
* Split out a commit to enable this feature.
---

Xinying Yu (11):
  mailmap: add new contributor
  vdpa/nfp: fix logic in hardware init
  vdpa/nfp: fix the logic of reconfiguration
  vdpa/nfp: refactor the logic of datapath update
  vdpa/nfp: add the live migration logic
  vdpa/nfp: add the interrupt logic of vring relay
  vdpa/nfp: setup the VF configure
  vdpa/nfp: recover the ring index on new host
  vdpa/nfp: setup vring relay thread
  vdpa/nfp: enable feature bits of live migration
  doc: update nfp document

 .mailmap                             |   1 +
 doc/guides/vdpadevs/nfp.rst          |   9 +
 drivers/common/nfp/nfp_common_ctrl.h |  11 +-
 drivers/vdpa/nfp/nfp_vdpa.c          | 441 +++++++++++++++++++++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
 6 files changed, 565 insertions(+), 46 deletions(-)

-- 
2.39.1

[PATCH v4 01/11] mailmap: add new contributor

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add new contributor.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
---
 .mailmap | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.mailmap b/.mailmap
index 4a508bafad..dc3a202a41 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1645,6 +1645,7 @@ Xieming Katty <katty.xieming@huawei.com>
 Xinfeng Zhao <xinfengx.zhao@intel.com>
 Xingguang He <xingguang.he@intel.com>
 Xingyou Chen <niatlantice@gmail.com>
+Xinying Yu <xinying.yu@corigine.com>
 Xin Long <longxin.xl@alibaba-inc.com>
 Xi Zhang <xix.zhang@intel.com>
 Xuan Ding <xuan.ding@intel.com>
-- 
2.39.1

[PATCH v4 02/11] vdpa/nfp: fix logic in hardware init

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Reconfigure the NIC will fail because lack of the
initialization logic of queue configuration pointer.
Fix this by adding the correct initialization logic.

Fixes: d89f4990c14e ("vdpa/nfp: add hardware init")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 7b877605e4..291798196c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -55,7 +55,10 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 		struct rte_pci_device *pci_dev)
 {
 	uint32_t queue;
+	uint8_t *tx_bar;
+	uint32_t start_q;
 	struct nfp_hw *hw;
+	uint32_t tx_bar_off;
 	uint8_t *notify_base;
 
 	hw = &vdpa_hw->super;
@@ -82,6 +85,12 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 				idx + 1, vdpa_hw->notify_addr[idx + 1]);
 	}
 
+	/* NFP vDPA cfg queue setup */
+	start_q = nn_cfg_readl(hw, NFP_NET_CFG_START_TXQ);
+	tx_bar_off = start_q * NFP_QCP_QUEUE_ADDR_SZ;
+	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
+	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
-- 
2.39.1

[PATCH v4 03/11] vdpa/nfp: fix the logic of reconfiguration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

The ctrl words of vDPA is located on the extended word, so it
should use the 'nfp_ext_reconfig()' rather than 'nfp_reconfig()'.

Also replace the misuse of 'NFP_NET_CFG_CTRL_SCATTER' macro
with 'NFP_NET_CFG_CTRL_VIRTIO'.

Fixes: b47a0373903f ("vdpa/nfp: add datapath update")
Cc: chaoyong.he@corigine.com
Cc: stable@dpdk.org

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  1 +
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 16 ++++++++++++----
 2 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 69596dd6f5..1b30f81fdb 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -205,6 +205,7 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_IPSEC_LM_LOOKUP  (0x1 << 4) /**< SA long match lookup */
 #define NFP_NET_CFG_CTRL_MULTI_PF         (0x1 << 5)
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
+#define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 291798196c..6d07356581 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -101,7 +101,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
-	return NFP_NET_CFG_CTRL_SCATTER |
+	return NFP_NET_CFG_CTRL_VIRTIO  |
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
@@ -112,6 +112,7 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	int ret;
 	uint32_t update;
 	uint32_t new_ctrl;
+	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
 	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
@@ -131,8 +132,6 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	new_ctrl = nfp_vdpa_check_offloads();
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -147,8 +146,17 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	/* Writing new MAC to the specific port BAR address */
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
+	new_ext_ctrl = nfp_vdpa_check_offloads();
+
+	update = NFP_NET_CFG_UPDATE_GEN;
+	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
+	if (ret != 0)
+		return -EIO;
+
+	hw->ctrl_ext = new_ext_ctrl;
+
 	/* Enable device */
-	new_ctrl |= NFP_NET_CFG_CTRL_ENABLE;
+	new_ctrl = NFP_NET_CFG_CTRL_ENABLE;
 
 	/* Signal the NIC about the change */
 	update = NFP_NET_CFG_UPDATE_MACADDR |
-- 
2.39.1

[PATCH v4 04/11] vdpa/nfp: refactor the logic of datapath update

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

In order to add the new configuration logic of software live
migration, split the datapath update logic into two parts,
queue configuration and VF configuration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 54 +++++++++++++++++++++-----------
 1 file changed, 36 insertions(+), 18 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 6d07356581..79ecd2b4fc 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -105,8 +105,8 @@ nfp_vdpa_check_offloads(void)
 			NFP_NET_CFG_CTRL_IN_ORDER;
 }
 
-int
-nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+static int
+nfp_vdpa_vf_config(struct nfp_hw *hw,
 		int vid)
 {
 	int ret;
@@ -114,24 +114,8 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	uint32_t new_ctrl;
 	uint32_t new_ext_ctrl;
 	struct timespec wait_tst;
-	struct nfp_hw *hw = &vdpa_hw->super;
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0), rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
-
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0), rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
-
-	rte_wmb();
-
-	nfp_disable_queues(hw);
-	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
-
 	nn_cfg_writel(hw, NFP_NET_CFG_MTU, 9216);
 	nn_cfg_writel(hw, NFP_NET_CFG_FLBUFSZ, 10240);
 
@@ -177,6 +161,40 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[1].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+			rte_log2_u32(vdpa_hw->vring[0].size));
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
+
+	rte_wmb();
+}
+
+int
+nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw);
+
+	nfp_disable_queues(hw);
+	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
+
+	return nfp_vdpa_vf_config(hw, vid);
+}
+
 void
 nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw)
 {
-- 
2.39.1

[PATCH v4 05/11] vdpa/nfp: add the live migration logic

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the basic logic of software live migration.

Unset the ring notify area to stop the direct IO datapath if the
device support, then we can setup the vring relay to help the
live migration.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 66 +++++++++++++++++++++++++++++++-
 drivers/vdpa/nfp/nfp_vdpa_core.c |  2 +
 drivers/vdpa/nfp/nfp_vdpa_core.h |  4 ++
 3 files changed, 70 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index cef80b5476..45092cb0af 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -603,6 +603,30 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	int vid = device->vid;
+
+	/* Stop the direct IO data path */
+	nfp_vdpa_unset_notify_relay(device);
+	nfp_vdpa_disable_vfio_intr(device);
+
+	ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false);
+	if ((ret != 0) && (ret != -ENOTSUP)) {
+		DRV_VDPA_LOG(ERR, "Unset the host notifier failed.");
+		goto error;
+	}
+
+	device->hw.sw_fallback_running = true;
+
+	return 0;
+
+error:
+	return ret;
+}
+
 static int
 nfp_vdpa_dev_config(int vid)
 {
@@ -646,8 +670,18 @@ nfp_vdpa_dev_close(int vid)
 	}
 
 	device = node->device;
-	rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed);
-	update_datapath(device);
+	if (device->hw.sw_fallback_running) {
+		device->hw.sw_fallback_running = false;
+
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		rte_atomic_store_explicit(&device->running, 0,
+				rte_memory_order_relaxed);
+	} else {
+		rte_atomic_store_explicit(&device->dev_attached, 0,
+				rte_memory_order_relaxed);
+		update_datapath(device);
+	}
 
 	return 0;
 }
@@ -770,7 +804,35 @@ nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused,
 static int
 nfp_vdpa_set_features(int32_t vid)
 {
+	int ret;
+	uint64_t features = 0;
+	struct nfp_vdpa_dev *device;
+	struct rte_vdpa_device *vdev;
+	struct nfp_vdpa_dev_node *node;
+
 	DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid);
+
+	vdev = rte_vhost_get_vdpa_device(vid);
+	node = nfp_vdpa_find_node_by_vdev(vdev);
+	if (node == NULL) {
+		DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev);
+		return -ENODEV;
+	}
+
+	rte_vhost_get_negotiated_features(vid, &features);
+
+	if (RTE_VHOST_NEED_LOG(features) == 0)
+		return 0;
+
+	device = node->device;
+	if (device->hw.sw_lm) {
+		ret = nfp_vdpa_sw_fallback(device);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "Software fallback start failed");
+			return -1;
+		}
+	}
+
 	return 0;
 }
 
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 79ecd2b4fc..50eda4cb2c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -91,6 +91,8 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	tx_bar = (uint8_t *)pci_dev->mem_resource[2].addr + tx_bar_off;
 	hw->qcp_cfg = tx_bar + NFP_QCP_QUEUE_ADDR_SZ;
 
+	vdpa_hw->sw_lm = true;
+
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a8e0d6dd70..0f880fc0c6 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -36,6 +36,10 @@ struct nfp_vdpa_hw {
 	uint8_t mac_addr[RTE_ETHER_ADDR_LEN];
 	uint8_t notify_region;
 	uint8_t nr_vring;
+
+	/** Software Live Migration */
+	bool sw_lm;
+	bool sw_fallback_running;
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
-- 
2.39.1

[PATCH v4 06/11] vdpa/nfp: add the interrupt logic of vring relay

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the interrupt setup logic of vring relay.

The epoll fd is provided here so host can get the interrupt from device
on Rx direction, all other operations on vring relay are based on this.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 45092cb0af..1643ebbb8c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -336,8 +336,10 @@ nfp_vdpa_stop(struct nfp_vdpa_dev *device)
 }
 
 static int
-nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
+nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device,
+		bool relay)
 {
+	int fd;
 	int ret;
 	uint16_t i;
 	int *fd_ptr;
@@ -366,6 +368,19 @@ nfp_vdpa_enable_vfio_intr(struct nfp_vdpa_dev *device)
 		fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = vring.callfd;
 	}
 
+	if (relay) {
+		for (i = 0; i < nr_vring; i += 2) {
+			fd = eventfd(0, EFD_NONBLOCK | EFD_CLOEXEC);
+			if (fd < 0) {
+				DRV_VDPA_LOG(ERR, "Can't setup eventfd");
+				return -EINVAL;
+			}
+
+			device->intr_fd[i] = fd;
+			fd_ptr[RTE_INTR_VEC_RXTX_OFFSET + i] = fd;
+		}
+	}
+
 	ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_set);
 	if (ret != 0) {
 		DRV_VDPA_LOG(ERR, "Error enabling MSI-X interrupts.");
@@ -556,7 +571,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto unlock_exit;
 
-		ret = nfp_vdpa_enable_vfio_intr(device);
+		ret = nfp_vdpa_enable_vfio_intr(device, false);
 		if (ret != 0)
 			goto dma_map_rollback;
 
@@ -619,6 +634,11 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 		goto error;
 	}
 
+	/* Setup interrupt for vring relay */
+	ret = nfp_vdpa_enable_vfio_intr(device, true);
+	if (ret != 0)
+		goto error;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
-- 
2.39.1

[PATCH v4 07/11] vdpa/nfp: setup the VF configure

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Create the relay vring on host and then set the address of Rx
used ring to the VF config bar. So the device can DMA the
used ring information to host rather than directly to VM.

Use 'NFP_NET_CFG_CTRL_LM_RELAY' notify the device side. And
enable the MSIX interrupt on device.

Tx ring address is not needed to change since the relay vring
only assists Rx ring to do the dirty page logging.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |   3 +
 drivers/vdpa/nfp/nfp_vdpa.c          | 203 ++++++++++++++++++++++++---
 drivers/vdpa/nfp/nfp_vdpa_core.c     |  55 ++++++--
 drivers/vdpa/nfp/nfp_vdpa_core.h     |   8 ++
 4 files changed, 239 insertions(+), 30 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 1b30f81fdb..8a760ddb4b 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -207,6 +207,9 @@ struct nfp_net_fw_ver {
 #define NFP_NET_CFG_CTRL_FLOW_STEER       (0x1 << 8) /**< Flow Steering */
 #define NFP_NET_CFG_CTRL_VIRTIO           (0x1 << 10) /**< Virtio offload */
 #define NFP_NET_CFG_CTRL_IN_ORDER         (0x1 << 11) /**< Virtio in-order flag */
+#define NFP_NET_CFG_CTRL_LM_RELAY         (0x1 << 12) /**< Virtio live migration relay start */
+#define NFP_NET_CFG_CTRL_NOTIFY_DATA      (0x1 << 13) /**< Virtio notification data flag */
+#define NFP_NET_CFG_CTRL_SWLM             (0x1 << 14) /**< Virtio SW live migration enable */
 #define NFP_NET_CFG_CTRL_USO              (0x1 << 16) /**< UDP segmentation offload */
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 1643ebbb8c..983123ba08 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -11,6 +11,8 @@
 #include <nfp_common_pci.h>
 #include <nfp_dev.h>
 #include <rte_vfio.h>
+#include <rte_eal_paging.h>
+#include <rte_malloc.h>
 #include <vdpa_driver.h>
 
 #include "nfp_vdpa_core.h"
@@ -21,6 +23,9 @@
 #define MSIX_IRQ_SET_BUF_LEN (sizeof(struct vfio_irq_set) + \
 		sizeof(int) * (NFP_VDPA_MAX_QUEUES * 2 + 1))
 
+#define NFP_VDPA_USED_RING_LEN(size) \
+		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -261,15 +266,85 @@ nfp_vdpa_qva_to_gpa(int vid,
 	return gpa;
 }
 
+static void
+nfp_vdpa_relay_vring_free(struct nfp_vdpa_dev *device,
+		uint16_t vring_index)
+{
+	uint16_t i;
+	uint64_t size;
+	struct rte_vhost_vring vring;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	for (i = 0; i < vring_index; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, rte_mem_page_size()),
+				rte_mem_page_size());
+		rte_vfio_container_dma_unmap(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)device->hw.m_vring[i].desc,
+				m_vring_iova, size);
+
+		rte_free(device->hw.m_vring[i].desc);
+		m_vring_iova += size;
+	}
+}
+
 static int
-nfp_vdpa_start(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_vring_alloc(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint16_t i;
+	uint64_t size;
+	void *vring_buf;
+	uint64_t page_size;
+	struct rte_vhost_vring vring;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
+
+	page_size = rte_mem_page_size();
+
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		rte_vhost_get_vhost_vring(device->vid, i, &vring);
+
+		size = RTE_ALIGN_CEIL(vring_size(vring.size, page_size), page_size);
+		vring_buf = rte_zmalloc("nfp_vdpa_relay", size, page_size);
+		if (vring_buf == NULL)
+			goto vring_free_all;
+
+		vring_init(&vdpa_hw->m_vring[i], vring.size, vring_buf, page_size);
+
+		ret = rte_vfio_container_dma_map(device->vfio_container_fd,
+				(uint64_t)(uintptr_t)vring_buf, m_vring_iova, size);
+		if (ret != 0) {
+			DRV_VDPA_LOG(ERR, "vDPA vring relay dma map failed.");
+			goto vring_free_one;
+		}
+
+		m_vring_iova += size;
+	}
+
+	return 0;
+
+vring_free_one:
+	rte_free(device->hw.m_vring[i].desc);
+vring_free_all:
+	nfp_vdpa_relay_vring_free(device, i);
+
+	return -ENOSPC;
+}
+
+static int
+nfp_vdpa_start(struct nfp_vdpa_dev *device,
+		bool relay)
 {
 	int ret;
 	int vid;
 	uint16_t i;
 	uint64_t gpa;
+	uint16_t size;
 	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+	uint64_t m_vring_iova = NFP_VDPA_RELAY_VRING;
 
 	vid = device->vid;
 	vdpa_hw->nr_vring = rte_vhost_get_vring_num(vid);
@@ -278,15 +353,21 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		return ret;
 
+	if (relay) {
+		ret = nfp_vdpa_relay_vring_alloc(device);
+		if (ret != 0)
+			return ret;
+	}
+
 	for (i = 0; i < vdpa_hw->nr_vring; i++) {
 		ret = rte_vhost_get_vhost_vring(vid, i, &vring);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.desc);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for descriptor ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].desc = gpa;
@@ -294,45 +375,123 @@ nfp_vdpa_start(struct nfp_vdpa_dev *device)
 		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.avail);
 		if (gpa == 0) {
 			DRV_VDPA_LOG(ERR, "Fail to get GPA for available ring.");
-			return -1;
+			goto relay_vring_free;
 		}
 
 		vdpa_hw->vring[i].avail = gpa;
 
-		gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
-		if (gpa == 0) {
-			DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
-			return -1;
-		}
+		/* Direct I/O for Tx queue, relay for Rx queue */
+		if (relay && ((i & 1) == 0)) {
+			vdpa_hw->vring[i].used = m_vring_iova +
+					(char *)vdpa_hw->m_vring[i].used -
+					(char *)vdpa_hw->m_vring[i].desc;
+
+			ret = rte_vhost_get_vring_base(vid, i,
+					&vdpa_hw->m_vring[i].avail->idx,
+					&vdpa_hw->m_vring[i].used->idx);
+			if (ret != 0)
+				goto relay_vring_free;
+		} else {
+			gpa = nfp_vdpa_qva_to_gpa(vid, (uint64_t)(uintptr_t)vring.used);
+			if (gpa == 0) {
+				DRV_VDPA_LOG(ERR, "Fail to get GPA for used ring.");
+				goto relay_vring_free;
+			}
 
-		vdpa_hw->vring[i].used = gpa;
+			vdpa_hw->vring[i].used = gpa;
+		}
 
 		vdpa_hw->vring[i].size = vring.size;
 
+		if (relay) {
+			size = RTE_ALIGN_CEIL(vring_size(vring.size,
+					rte_mem_page_size()), rte_mem_page_size());
+			m_vring_iova += size;
+		}
+
 		ret = rte_vhost_get_vring_base(vid, i,
 				&vdpa_hw->vring[i].last_avail_idx,
 				&vdpa_hw->vring[i].last_used_idx);
 		if (ret != 0)
-			return ret;
+			goto relay_vring_free;
 	}
 
-	return nfp_vdpa_hw_start(&device->hw, vid);
+	if (relay)
+		return nfp_vdpa_relay_hw_start(&device->hw, vid);
+	else
+		return nfp_vdpa_hw_start(&device->hw, vid);
+
+relay_vring_free:
+	if (relay)
+		nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+
+	return -EFAULT;
+}
+
+static void
+nfp_vdpa_update_used_ring(struct nfp_vdpa_dev *dev,
+		uint16_t qid)
+{
+	rte_vdpa_relay_vring_used(dev->vid, qid, &dev->hw.m_vring[qid]);
+	rte_vhost_vring_call(dev->vid, qid);
 }
 
 static void
-nfp_vdpa_stop(struct nfp_vdpa_dev *device)
+nfp_vdpa_relay_stop(struct nfp_vdpa_dev *device)
 {
 	int vid;
 	uint32_t i;
+	uint64_t len;
+	struct rte_vhost_vring vring;
 	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
 
 	nfp_vdpa_hw_stop(vdpa_hw);
 
 	vid = device->vid;
-	for (i = 0; i < vdpa_hw->nr_vring; i++)
+	for (i = 0; i < vdpa_hw->nr_vring; i++) {
+		/* Synchronize remaining new used entries if any */
+		if ((i & 1) == 0)
+			nfp_vdpa_update_used_ring(device, i);
+
+		rte_vhost_get_vhost_vring(vid, i, &vring);
+		len = NFP_VDPA_USED_RING_LEN(vring.size);
+		vdpa_hw->vring[i].last_avail_idx = vring.avail->idx;
+		vdpa_hw->vring[i].last_used_idx = vring.used->idx;
+
 		rte_vhost_set_vring_base(vid, i,
 				vdpa_hw->vring[i].last_avail_idx,
 				vdpa_hw->vring[i].last_used_idx);
+
+		rte_vhost_log_used_vring(vid, i, 0, len);
+
+		if (vring.used->idx != vring.avail->idx)
+			rte_atomic_store_explicit(
+					(unsigned short __rte_atomic *)&vring.used->idx,
+					vring.avail->idx, rte_memory_order_release);
+	}
+
+	nfp_vdpa_relay_vring_free(device, vdpa_hw->nr_vring);
+}
+
+static void
+nfp_vdpa_stop(struct nfp_vdpa_dev *device,
+		bool relay)
+{
+	int vid;
+	uint32_t i;
+	struct nfp_vdpa_hw *vdpa_hw = &device->hw;
+
+	nfp_vdpa_hw_stop(vdpa_hw);
+
+	vid = device->vid;
+	if (relay)
+		nfp_vdpa_relay_stop(device);
+	else
+		for (i = 0; i < vdpa_hw->nr_vring; i++)
+			rte_vhost_set_vring_base(vid, i,
+					vdpa_hw->vring[i].last_avail_idx,
+					vdpa_hw->vring[i].last_used_idx);
+
 }
 
 static int
@@ -575,7 +734,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 		if (ret != 0)
 			goto dma_map_rollback;
 
-		ret = nfp_vdpa_start(device);
+		ret = nfp_vdpa_start(device, false);
 		if (ret != 0)
 			goto disable_vfio_intr;
 
@@ -591,7 +750,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 					rte_memory_order_relaxed) != 0))) {
 		nfp_vdpa_unset_notify_relay(device);
 
-		nfp_vdpa_stop(device);
+		nfp_vdpa_stop(device, false);
 
 		ret = nfp_vdpa_disable_vfio_intr(device);
 		if (ret != 0)
@@ -608,7 +767,7 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return 0;
 
 vdpa_stop:
-	nfp_vdpa_stop(device);
+	nfp_vdpa_stop(device, false);
 disable_vfio_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 dma_map_rollback:
@@ -639,10 +798,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto error;
 
+	/* Config the VF */
+	ret = nfp_vdpa_start(device, true);
+	if (ret != 0)
+		goto unset_intr;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+unset_intr:
+	nfp_vdpa_disable_vfio_intr(device);
 error:
 	return ret;
 }
@@ -691,6 +857,9 @@ nfp_vdpa_dev_close(int vid)
 
 	device = node->device;
 	if (device->hw.sw_fallback_running) {
+		/* Reset VF */
+		nfp_vdpa_stop(device, true);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 50eda4cb2c..2b609dddc2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -109,7 +109,8 @@ nfp_vdpa_check_offloads(void)
 
 static int
 nfp_vdpa_vf_config(struct nfp_hw *hw,
-		int vid)
+		int vid,
+		bool relay)
 {
 	int ret;
 	uint32_t update;
@@ -133,6 +134,10 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 	nfp_write_mac(hw, (uint8_t *)mac_addr);
 
 	new_ext_ctrl = nfp_vdpa_check_offloads();
+	if (relay)
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_LM_RELAY;
+	else
+		new_ext_ctrl |= NFP_NET_CFG_CTRL_SWLM;
 
 	update = NFP_NET_CFG_UPDATE_GEN;
 	ret = nfp_ext_reconfig(hw, new_ext_ctrl, update);
@@ -149,6 +154,15 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 			NFP_NET_CFG_UPDATE_GEN |
 			NFP_NET_CFG_UPDATE_RING;
 
+	if (relay) {
+		update |= NFP_NET_CFG_UPDATE_MSIX;
+
+		/* Enable misx interrupt for vdpa relay */
+		new_ctrl |= NFP_NET_CFG_CTRL_MSIX_TX_OFF;
+
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_VEC(0), 1);
+	}
+
 	ret = nfp_reconfig(hw, new_ctrl, update);
 	if (ret < 0)
 		return -EIO;
@@ -164,20 +178,24 @@ nfp_vdpa_vf_config(struct nfp_hw *hw,
 }
 
 static void
-nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw)
+nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
+		bool relay)
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[1].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
-	nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+	if (!relay) {
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(0), vdpa_hw->vring[1].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_TXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[1].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(1), vdpa_hw->vring[1].avail);
+		nn_cfg_writeq(hw, NFP_NET_CFG_TXR_ADDR(2), vdpa_hw->vring[1].used);
+
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
+		nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
+				rte_log2_u32(vdpa_hw->vring[0].size));
+		nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
+	}
 
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(0), vdpa_hw->vring[0].desc);
-	nn_cfg_writeb(hw, NFP_NET_CFG_RXR_SZ(0),
-			rte_log2_u32(vdpa_hw->vring[0].size));
-	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(1), vdpa_hw->vring[0].avail);
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
 	rte_wmb();
@@ -189,12 +207,23 @@ nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw,
 {
 	struct nfp_hw *hw = &vdpa_hw->super;
 
-	nfp_vdpa_queue_config(vdpa_hw);
+	nfp_vdpa_queue_config(vdpa_hw, false);
 
 	nfp_disable_queues(hw);
 	nfp_enable_queues(hw, NFP_VDPA_MAX_QUEUES, NFP_VDPA_MAX_QUEUES);
 
-	return nfp_vdpa_vf_config(hw, vid);
+	return nfp_vdpa_vf_config(hw, vid, false);
+}
+
+int
+nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw,
+		int vid)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	nfp_vdpa_queue_config(vdpa_hw, true);
+
+	return nfp_vdpa_vf_config(hw, vid, true);
 }
 
 void
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index 0f880fc0c6..a339ace601 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -9,12 +9,15 @@
 #include <bus_pci_driver.h>
 #include <nfp_common.h>
 #include <rte_ether.h>
+#include <rte_vhost.h>
 
 #define NFP_VDPA_MAX_QUEUES         1
 
 #define NFP_VDPA_NOTIFY_ADDR_BASE        0x4000
 #define NFP_VDPA_NOTIFY_ADDR_INTERVAL    0x1000
 
+#define NFP_VDPA_RELAY_VRING             0xd0000000
+
 struct nfp_vdpa_vring {
 	uint64_t desc;
 	uint64_t avail;
@@ -40,12 +43,17 @@ struct nfp_vdpa_hw {
 	/** Software Live Migration */
 	bool sw_lm;
 	bool sw_fallback_running;
+
+	/** Mediated vring for SW fallback */
+	struct vring m_vring[NFP_VDPA_MAX_QUEUES * 2];
 };
 
 int nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw, struct rte_pci_device *dev);
 
 int nfp_vdpa_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
 
+int nfp_vdpa_relay_hw_start(struct nfp_vdpa_hw *vdpa_hw, int vid);
+
 void nfp_vdpa_hw_stop(struct nfp_vdpa_hw *vdpa_hw);
 
 void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
-- 
2.39.1

[PATCH v4 08/11] vdpa/nfp: recover the ring index on new host

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

After migrating to new host, the vring information is
recovered by the value in offset 'NFP_NET_CFG_TX_USED_INDEX'
and 'NFP_NET_CFG_RX_USED_INDEX'.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/common/nfp/nfp_common_ctrl.h |  7 +++++--
 drivers/vdpa/nfp/nfp_vdpa_core.c     | 13 +++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/nfp/nfp_common_ctrl.h b/drivers/common/nfp/nfp_common_ctrl.h
index 8a760ddb4b..b64bb1dd2d 100644
--- a/drivers/common/nfp/nfp_common_ctrl.h
+++ b/drivers/common/nfp/nfp_common_ctrl.h
@@ -214,8 +214,11 @@ struct nfp_net_fw_ver {
 
 #define NFP_NET_CFG_CAP_WORD1           0x00a4
 
-/* 16B reserved for future use (0x00b0 - 0x00c0). */
-#define NFP_NET_CFG_RESERVED            0x00b0
+#define NFP_NET_CFG_TX_USED_INDEX       0x00b0
+#define NFP_NET_CFG_RX_USED_INDEX       0x00b4
+
+/* 16B reserved for future use (0x00b8 - 0x0010). */
+#define NFP_NET_CFG_RESERVED            0x00b8
 #define NFP_NET_CFG_RESERVED_SZ         0x0010
 
 /*
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 2b609dddc2..d7c48e2490 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -100,6 +100,16 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 	return 0;
 }
 
+static void
+nfp_vdpa_hw_queue_init(struct nfp_vdpa_hw *vdpa_hw)
+{
+	/* Distribute ring information to firmware */
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_TX_USED_INDEX,
+			vdpa_hw->vring[1].last_used_idx);
+	nn_cfg_writel(&vdpa_hw->super, NFP_NET_CFG_RX_USED_INDEX,
+			vdpa_hw->vring[0].last_used_idx);
+}
+
 static uint32_t
 nfp_vdpa_check_offloads(void)
 {
@@ -198,6 +208,9 @@ nfp_vdpa_queue_config(struct nfp_vdpa_hw *vdpa_hw,
 
 	nn_cfg_writeq(hw, NFP_NET_CFG_RXR_ADDR(2), vdpa_hw->vring[0].used);
 
+	if (!relay)
+		nfp_vdpa_hw_queue_init(vdpa_hw);
+
 	rte_wmb();
 }
 
-- 
2.39.1

[PATCH v4 09/11] vdpa/nfp: setup vring relay thread

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Setup the vring relay thread to monitor the interruption from
device. And do the dirty page logging or notify device according
to event data.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa.c      | 148 +++++++++++++++++++++++++++++++
 drivers/vdpa/nfp/nfp_vdpa_core.c |   9 ++
 drivers/vdpa/nfp/nfp_vdpa_core.h |   2 +
 3 files changed, 159 insertions(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c
index 983123ba08..91f1b8d779 100644
--- a/drivers/vdpa/nfp/nfp_vdpa.c
+++ b/drivers/vdpa/nfp/nfp_vdpa.c
@@ -26,6 +26,8 @@
 #define NFP_VDPA_USED_RING_LEN(size) \
 		((size) * sizeof(struct vring_used_elem) + sizeof(struct vring_used))
 
+#define EPOLL_DATA_INTR        1
+
 struct nfp_vdpa_dev {
 	struct rte_pci_device *pci_dev;
 	struct rte_vdpa_device *vdev;
@@ -777,6 +779,139 @@ update_datapath(struct nfp_vdpa_dev *device)
 	return ret;
 }
 
+static int
+nfp_vdpa_vring_epoll_ctl(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int ret;
+	uint32_t qid;
+	struct epoll_event ev;
+	struct rte_vhost_vring vring;
+
+	for (qid = 0; qid < queue_num; qid++) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		rte_vhost_get_vhost_vring(device->vid, qid, &vring);
+		ev.data.u64 = qid << 1 | (uint64_t)vring.kickfd << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD, vring.kickfd, &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+	}
+
+	/* vDPA driver interrupt */
+	for (qid = 0; qid < queue_num; qid += 2) {
+		ev.events = EPOLLIN | EPOLLPRI;
+		/* Leave a flag to mark it's for interrupt */
+		ev.data.u64 = EPOLL_DATA_INTR | qid << 1 |
+				(uint64_t)device->intr_fd[qid] << 32;
+		ret = epoll_ctl(device->epoll_fd, EPOLL_CTL_ADD,
+				device->intr_fd[qid], &ev);
+		if (ret < 0) {
+			DRV_VDPA_LOG(ERR, "Epoll add error for queue %u", qid);
+			return ret;
+		}
+
+		nfp_vdpa_update_used_ring(device, qid);
+	}
+
+	return 0;
+}
+
+static int
+nfp_vdpa_vring_epoll_wait(uint32_t queue_num,
+		struct nfp_vdpa_dev *device)
+{
+	int i;
+	int fds;
+	int kickfd;
+	uint32_t qid;
+	struct epoll_event events[NFP_VDPA_MAX_QUEUES * 2];
+
+	for (;;) {
+		fds = epoll_wait(device->epoll_fd, events, queue_num * 2, -1);
+		if (fds < 0) {
+			if (errno == EINTR)
+				continue;
+
+			DRV_VDPA_LOG(ERR, "Epoll wait fail");
+			return -EACCES;
+		}
+
+		for (i = 0; i < fds; i++) {
+			qid = events[i].data.u32 >> 1;
+			kickfd = (uint32_t)(events[i].data.u64 >> 32);
+
+			nfp_vdpa_read_kickfd(kickfd);
+			if ((events[i].data.u32 & EPOLL_DATA_INTR) != 0) {
+				nfp_vdpa_update_used_ring(device, qid);
+				nfp_vdpa_irq_unmask(&device->hw);
+			} else {
+				nfp_vdpa_notify_queue(&device->hw, qid);
+			}
+		}
+	}
+
+	return 0;
+}
+
+static uint32_t
+nfp_vdpa_vring_relay(void *arg)
+{
+	int ret;
+	int epoll_fd;
+	uint16_t queue_id;
+	uint32_t queue_num;
+	struct nfp_vdpa_dev *device = arg;
+
+	epoll_fd = epoll_create(NFP_VDPA_MAX_QUEUES * 2);
+	if (epoll_fd < 0) {
+		DRV_VDPA_LOG(ERR, "failed to create epoll instance.");
+		return 1;
+	}
+
+	device->epoll_fd = epoll_fd;
+
+	queue_num = rte_vhost_get_vring_num(device->vid);
+
+	ret = nfp_vdpa_vring_epoll_ctl(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	/* Start relay with a first kick */
+	for (queue_id = 0; queue_id < queue_num; queue_id++)
+		nfp_vdpa_notify_queue(&device->hw, queue_id);
+
+	ret = nfp_vdpa_vring_epoll_wait(queue_num, device);
+	if (ret != 0)
+		goto notify_exit;
+
+	return 0;
+
+notify_exit:
+	close(device->epoll_fd);
+	device->epoll_fd = -1;
+
+	return 1;
+}
+
+static int
+nfp_vdpa_setup_vring_relay(struct nfp_vdpa_dev *device)
+{
+	int ret;
+	char name[RTE_THREAD_INTERNAL_NAME_SIZE];
+
+	snprintf(name, sizeof(name), "nfp_vring%d", device->vid);
+	ret = rte_thread_create_internal_control(&device->tid, name,
+			nfp_vdpa_vring_relay, (void *)device);
+	if (ret != 0) {
+		DRV_VDPA_LOG(ERR, "Failed to create vring relay pthread.");
+		return -EPERM;
+	}
+
+	return 0;
+}
+
 static int
 nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 {
@@ -803,10 +938,17 @@ nfp_vdpa_sw_fallback(struct nfp_vdpa_dev *device)
 	if (ret != 0)
 		goto unset_intr;
 
+	/* Setup vring relay thread */
+	ret = nfp_vdpa_setup_vring_relay(device);
+	if (ret != 0)
+		goto stop_vf;
+
 	device->hw.sw_fallback_running = true;
 
 	return 0;
 
+stop_vf:
+	nfp_vdpa_stop(device, true);
 unset_intr:
 	nfp_vdpa_disable_vfio_intr(device);
 error:
@@ -860,6 +1002,12 @@ nfp_vdpa_dev_close(int vid)
 		/* Reset VF */
 		nfp_vdpa_stop(device, true);
 
+		/* Remove interrupt setting */
+		nfp_vdpa_disable_vfio_intr(device);
+
+		/* Unset DMA map for guest memory */
+		nfp_vdpa_dma_map(device, false);
+
 		device->hw.sw_fallback_running = false;
 
 		rte_atomic_store_explicit(&device->dev_attached, 0,
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index d7c48e2490..3b3481a99c 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -270,3 +270,12 @@ nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw,
 	nfp_qcp_notify_ptr_add(vdpa_hw->notify_addr[qid],
 			NFP_QCP_NOTIFY_WRITE_PTR, qid);
 }
+
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw)
+{
+	struct nfp_hw *hw = &vdpa_hw->super;
+
+	/* Make sure all updates are written before un-masking */
+	rte_wmb();
+	nn_cfg_writeb(hw, NFP_NET_CFG_ICR(1), NFP_NET_CFG_ICR_UNMASKED);
+}
diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.h b/drivers/vdpa/nfp/nfp_vdpa_core.h
index a339ace601..bc4db556a2 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.h
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.h
@@ -60,4 +60,6 @@ void nfp_vdpa_notify_queue(struct nfp_vdpa_hw *vdpa_hw, uint16_t qid);
 
 uint64_t nfp_vdpa_get_queue_notify_offset(struct nfp_vdpa_hw *vdpa_hw, int qid);
 
+void nfp_vdpa_irq_unmask(struct nfp_vdpa_hw *vdpa_hw);
+
 #endif /* __NFP_VDPA_CORE_H__ */
-- 
2.39.1

[PATCH v4 10/11] vdpa/nfp: enable feature bits of live migration

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the 'VHOST_F_LOG_ALL' feature bits inorder to enable the
live migration function.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 drivers/vdpa/nfp/nfp_vdpa_core.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vdpa/nfp/nfp_vdpa_core.c b/drivers/vdpa/nfp/nfp_vdpa_core.c
index 3b3481a99c..70aeb4a3ac 100644
--- a/drivers/vdpa/nfp/nfp_vdpa_core.c
+++ b/drivers/vdpa/nfp/nfp_vdpa_core.c
@@ -95,6 +95,7 @@ nfp_vdpa_hw_init(struct nfp_vdpa_hw *vdpa_hw,
 
 	vdpa_hw->features = (1ULL << VIRTIO_F_VERSION_1) |
 			(1ULL << VIRTIO_F_IN_ORDER) |
+			(1ULL << VHOST_F_LOG_ALL) |
 			(1ULL << VHOST_USER_F_PROTOCOL_FEATURES);
 
 	return 0;
-- 
2.39.1

[PATCH v4 11/11] doc: update nfp document

[Chaoyong He]

From: Xinying Yu <xinying.yu@corigine.com>

Add the software assisted vDPA live migration feature
into NFP document.

Signed-off-by: Xinying Yu <xinying.yu@corigine.com>
Reviewed-by: Chaoyong He <chaoyong.he@corigine.com>
Reviewed-by: Long Wu <long.wu@corigine.com>
Reviewed-by: Peng Zhang <peng.zhang@corigine.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 doc/guides/vdpadevs/nfp.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/doc/guides/vdpadevs/nfp.rst b/doc/guides/vdpadevs/nfp.rst
index dc9e94dbc8..e4736d9f61 100644
--- a/doc/guides/vdpadevs/nfp.rst
+++ b/doc/guides/vdpadevs/nfp.rst
@@ -19,6 +19,15 @@ device will be probed by net/nfp driver and will used as a VF net device.
 
 This PMD uses (common/nfp) code to access the device firmware.
 
+Software Live Migration
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Now the NFP vDPA driver only support software assisted live migration mode.
+In this mode, the driver will setup a software relay thread when live migration
+happens, this thread will help device to log dirty pages. Although this mode
+does not require hardware to implement a dirty page logging function block, it
+will consume percentage of CPU resource depending on the network throughput.
+
 Per-Device Parameters
 ~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.39.1

Re: [PATCH v4 00/11] support software live migration

[Maxime Coquelin]

On 8/5/24 04:12, Chaoyong He wrote:
> This patch series aims to add the support of software live migration
> feature for NFP vDPA device.
> 
> ---
> v4:
> * Rebase to the newest main branch.
> * Add the 'Review-by' tag of the external reviewer.
> v3:
> * Fix one compile error when using standard atomic.
> v2:
> * Adjust some spell in the commit message.
> * Split out a commit to enable this feature.
> ---
> 
> Xinying Yu (11):
>    mailmap: add new contributor
>    vdpa/nfp: fix logic in hardware init
>    vdpa/nfp: fix the logic of reconfiguration
>    vdpa/nfp: refactor the logic of datapath update
>    vdpa/nfp: add the live migration logic
>    vdpa/nfp: add the interrupt logic of vring relay
>    vdpa/nfp: setup the VF configure
>    vdpa/nfp: recover the ring index on new host
>    vdpa/nfp: setup vring relay thread
>    vdpa/nfp: enable feature bits of live migration
>    doc: update nfp document
> 
>   .mailmap                             |   1 +
>   doc/guides/vdpadevs/nfp.rst          |   9 +
>   drivers/common/nfp/nfp_common_ctrl.h |  11 +-
>   drivers/vdpa/nfp/nfp_vdpa.c          | 441 +++++++++++++++++++++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.c     | 135 ++++++--
>   drivers/vdpa/nfp/nfp_vdpa_core.h     |  14 +
>   6 files changed, 565 insertions(+), 46 deletions(-)
> 

Applied to next-virtio/for-next-net.

Thanks,
Maxime